🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(api)!: remove unstable evaluation endpoints (#17056)
Tobias Wochinger619dbe32b940
diff --git a/fern/apis/server/definition/dataset-run-items.yml b/fern/apis/server/definition/dataset-run-items.ymlindex 4cd54717a7..79c8fe02db 100644--- a/fern/apis/server/definition/dataset-run-items.yml+++ b/fern/apis/server/definition/dataset-run-items.yml@@ -54,7 +54,7 @@ types:type: optional<datetime>docs: |ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g., "2026-01-21T14:35:42Z").- Specifies the dataset version to use for this experiment run.+ Specifies the dataset version to use for this experiment run.If provided, the experiment will use dataset items as they existed at or before this timestamp.If not provided, uses the latest version of dataset items.createdAt:diff --git a/fern/apis/server/definition/unstable/commons.yml b/fern/apis/server/definition/unstable/commons.ymldeleted file mode 100644index 48e0c0a72a..0000000000--- a/fern/apis/server/definition/unstable/commons.yml+++ /dev/null@@ -1,616 +0,0 @@-# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json-types:- EvaluatorType:- docs: |- The evaluator type.-- The unstable public API supports LLM-as-a-judge and code evaluators.- enum:- - llm_as_judge- - code-- CodeEvaluatorSourceCodeLanguage:- docs: Code evaluator runtime language.- enum:- - PYTHON- - TYPESCRIPT-- EvaluationRuleTarget:- docs: |- The ingestion object type that should trigger evaluation runs.-- Choose the target first, because it changes both the valid filter columns and the valid variable-mapping sources:- - `observation` evaluates live-ingested observations such as generations, spans, and events.- It supports mapping from `input`, `output`, `metadata`, and `tool_calls`.- - `experiment` evaluates live experiment executions and can additionally map `expected_output` and `experiment_item_metadata`.- It currently supports filtering by `datasetId`.- Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter.- enum:- - observation- - experiment-- EvaluationRuleStatus:- docs: |- Effective runtime status of the evaluation rule.-- - `active`: enabled and currently runnable.- - `inactive`: disabled by configuration.- - `paused`: enabled, but Langfuse has blocked execution until the underlying issue is resolved.- enum:- - active- - inactive- - paused-- PromptVariableMappingSource:- docs: |- Source field used to populate a prompt variable.-- Use these values when mapping evaluator prompt variables to live data.-- Target-specific rules:- - `target=observation` supports `input`, `output`, `metadata`, and `tool_calls`- - `target=experiment` supports `input`, `output`, `metadata`, `tool_calls`, `expected_output`, and `experiment_item_metadata`-- Source semantics:- - `input`: the observation or experiment input payload- - `output`: the observation or experiment output payload- - `metadata`: the metadata object for the target. Combine with `jsonPath` when you need one nested field instead of the whole object.- - `tool_calls`: the tool calls recorded on the observation, as an array of `{id, name, arguments, type, index}` objects in the order the model emitted them. Combine with `jsonPath` (for example `$[*].name`) to select parts of each call.- - `expected_output`: the experiment item's expected output. Only valid for `target=experiment`.- - `experiment_item_metadata`: the experiment item's metadata object. Only valid for `target=experiment`.- enum:- - input- - output- - metadata- - tool_calls- - expected_output- - experiment_item_metadata-- EvaluatorModelConfig:- docs: |- Optional explicit model configuration for an evaluator.-- If omitted, Langfuse uses the project's default evaluation model.- If provided, the model must be available to the project when the evaluator or evaluation rule is enabled.-- To discover valid configured `provider` values for a project, call `GET /api/public/llm-connections` and read the `provider` field from the returned connections.- Use a `provider` value that matches one of the connections already configured in the same project.-- Recovery guidance:- - If evaluator creation returns `422` with `code=evaluator_preflight_failed`, either provide a valid explicit `modelConfig` here or configure the project's default evaluation model, then retry the same request.- properties:- provider:- type: string- docs: |- Provider identifier to use for this evaluator, for example `openai` or `anthropic`.-- To discover valid values for the current project, call `GET /api/public/llm-connections` and use one of the returned `provider` values.- model:- type: string- docs: Model identifier exposed by the provider, for example `gpt-4.1-mini`.- examples:- - value:- provider: openai- model: gpt-4.1-mini-- EvaluatorOutputDataType:- docs: |- Structured score type returned by an evaluator.-- This controls the type of score value Langfuse stores for evaluation results:- - `NUMERIC`: a numeric score such as `0.82`- - `BOOLEAN`: a boolean score such as `true`- - `CATEGORICAL`: one or more category labels from a fixed list- enum:- - NUMERIC- - BOOLEAN- - CATEGORICAL-- EvaluatorOutputFieldDefinition:- properties:- description:- type: string- docs: Human-readable instructions for what the evaluator should return in this field.-- EvaluatorOutputDefinition:- docs: |- Structured output definition to send when creating an evaluator.-- Agent guidance:- - `dataType` is required.- - Do not send `version`; that is an internal storage detail and is not part of the public request contract.- - For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and `score.description`.- - For `NUMERIC`, `score.minValue` and `score.maxValue` optionally define inclusive bounds. If both are set, `minValue` must not exceed `maxValue`.- - For `CATEGORICAL`, also provide `score.categories` and `score.shouldAllowMultipleMatches`.- discriminant: dataType- union:- NUMERIC:- type: PublicNumericEvaluatorOutputDefinition- BOOLEAN:- type: PublicBooleanEvaluatorOutputDefinition- CATEGORICAL:- type: PublicCategoricalEvaluatorOutputDefinition- examples:- - name: Numeric- value:- dataType: NUMERIC- reasoning:- description: Explain why the answer is correct or incorrect.- score:- description: Return a score between 0 and 1.- minValue: 0- maxValue: 1- - name: Boolean- value:- dataType: BOOLEAN- reasoning:- description: Explain why the output satisfies the requirement.- score:- description: Return true if the output satisfies the requirement, otherwise false.- - name: Categorical- value:- dataType: CATEGORICAL- reasoning:- description: Explain which category best fits the output.- score:- description: Choose the best category.- categories:- - correct- - partially_correct- - incorrect- shouldAllowMultipleMatches: false-- PublicNumericEvaluatorOutputScoreDefinition:- properties:- description:- type: string- minValue:- type: optional<double>- docs: Inclusive minimum value for numeric evaluator scores.- maxValue:- type: optional<double>- docs: Inclusive maximum value for numeric evaluator scores.-- PublicNumericEvaluatorOutputDefinition:- properties:- dataType:- type: EvaluatorOutputDataType- docs: Always `NUMERIC`.- reasoning:- type: EvaluatorOutputFieldDefinition- score:- type: PublicNumericEvaluatorOutputScoreDefinition-- PublicBooleanEvaluatorOutputDefinition:- properties:- dataType:- type: EvaluatorOutputDataType- docs: Always `BOOLEAN`.- reasoning:- type: EvaluatorOutputFieldDefinition- score:- type: EvaluatorOutputFieldDefinition-- PublicCategoricalEvaluatorOutputScoreDefinition:- properties:- description:- type: string- categories:- type: list<string>- shouldAllowMultipleMatches:- type: boolean-- PublicCategoricalEvaluatorOutputDefinition:- properties:- dataType:- type: EvaluatorOutputDataType- docs: Always `CATEGORICAL`.- reasoning:- type: EvaluatorOutputFieldDefinition- score:- type: PublicCategoricalEvaluatorOutputScoreDefinition-- PublicEvaluatorOutputDefinition:- docs: |- Evaluator output definition returned by the public API.-- This response always includes `dataType` and never includes an internal output-definition `version`.- Legacy stored evaluator definitions are normalized into this shape before they are returned.-- Use this response shape when deciding how to interpret future evaluation scores:- - `NUMERIC`: expect numeric score values- - `BOOLEAN`: expect `true` / `false`- - `CATEGORICAL`: expect one or more values from `score.categories`- discriminant: dataType- union:- NUMERIC:- type: PublicNumericEvaluatorOutputDefinition- BOOLEAN:- type: PublicBooleanEvaluatorOutputDefinition- CATEGORICAL:- type: PublicCategoricalEvaluatorOutputDefinition- examples:- - name: PublicNumeric- value:- dataType: NUMERIC- reasoning:- description: Explain why the answer is correct or incorrect.- score:- description: Return a score between 0 and 1.- minValue: 0- maxValue: 1- - name: PublicCategorical- value:- dataType: CATEGORICAL- reasoning:- description: Explain which label best fits the output.- score:- description: Choose the best label.- categories:- - correct- - partially_correct- - incorrect- shouldAllowMultipleMatches: false-- EvaluationRuleStringFilterOperator:- enum:- - name: Equals- value: "="- - name: Contains- value: contains- - name: DoesNotContain- value: does not contain- - name: StartsWith- value: starts with- - name: EndsWith- value: ends with-- EvaluationRuleNumberFilterOperator:- enum:- - name: Equals- value: "="- - name: GreaterThan- value: ">"- - name: LessThan- value: "<"- - name: GreaterThanOrEqual- value: ">="- - name: LessThanOrEqual- value: "<="-- EvaluationRuleOptionsFilterOperator:- enum:- - name: AnyOf- value: any of- - name: NoneOf- value: none of-- EvaluationRuleArrayOptionsFilterOperator:- enum:- - name: AnyOf- value: any of- - name: NoneOf- value: none of- - name: AllOf- value: all of-- EvaluationRuleBooleanFilterOperator:- enum:- - name: Equals- value: "="- - name: NotEquals- value: "<>"-- EvaluationRuleNullFilterOperator:- enum:- - name: IsNull- value: is null- - name: IsNotNull- value: is not null-- DateTimeEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on.- operator:- type: EvaluationRuleNumberFilterOperator- docs: Comparison operator for datetime values.- value:- type: datetime- docs: Datetime value to compare against.-- StringEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on.- operator:- type: EvaluationRuleStringFilterOperator- value:- type: string-- NumberEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on.- operator:- type: EvaluationRuleNumberFilterOperator- value:- type: double-- StringOptionsEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on.- operator:- type: EvaluationRuleOptionsFilterOperator- value:- type: list<string>- docs: One or more allowed string values.-- ArrayOptionsEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on.- operator:- type: EvaluationRuleArrayOptionsFilterOperator- value:- type: list<string>- docs: One or more array elements to match.-- StringObjectEvaluationRuleFilter:- properties:- column:- type: string- docs: Object-valued column to filter on. In the unstable public API this is currently `metadata`.- key:- type: string- docs: Top-level key inside the object-valued column to filter on.- operator:- type: EvaluationRuleStringFilterOperator- value:- type: string-- NumberObjectEvaluationRuleFilter:- properties:- column:- type: string- docs: Object-valued column to filter on.- key:- type: string- docs: Key inside the object-valued column to filter on.- operator:- type: EvaluationRuleNumberFilterOperator- value:- type: double-- CategoryOptionsEvaluationRuleFilter:- properties:- column:- type: string- docs: Object-valued column to filter on.- key:- type: string- docs: Key inside the object-valued column to filter on.- operator:- type: EvaluationRuleOptionsFilterOperator- value:- type: list<string>-- BooleanEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on.- operator:- type: EvaluationRuleBooleanFilterOperator- value:- type: boolean-- NullEvaluationRuleFilter:- properties:- column:- type: string- docs: Column to filter on. In the unstable public API this is currently `parentObservationId`.- operator:- type: EvaluationRuleNullFilterOperator- value:- type: optional<string>- docs: Ignored placeholder value. Clients may omit it or send an empty string.-- PromptVariableMappingInput:- docs: |- Connects one prompt variable to the available evaluation data.-- Manual mappings are used for `llm_as_judge` evaluators. `code` evaluators use a fixed runtime mapping managed by Langfuse.-- How to build a valid mapping list:- 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.- 2. Read the evaluator `variables` array.- 3. Add exactly one mapping object for each variable in that array.- 4. Use the variable name exactly as returned, without braces such as `{{` or `}}`.- 5. Choose a `source` that is valid for the selected `target`.-- `jsonPath` is optional. Use it only when the selected source is a JSON object and you want to extract one nested field before inserting it into the evaluator prompt.-- Recovery guidance:- - `invalid_variable_mapping`: the variable name is unknown for this evaluator, or the selected `source` is not valid for the chosen `target`- - `missing_variable_mapping`: one or more LLM-as-judge prompt variables are not mapped yet- - `duplicate_variable_mapping`: the same prompt variable appears more than once- - `invalid_json_path`: the JSONPath expression is malformed. Remove it or correct it.- properties:- variable:- type: string- docs: |- Prompt variable name without braces.-- Example: for the prompt `Judge {{input}} against {{output}}`, use `input` and `output`.- source:- type: PromptVariableMappingSource- docs: |- Source field that should populate the prompt variable.-- Quick reference:- - `target=observation`: `input`, `output`, `metadata`, `tool_calls`- - `target=experiment`: `input`, `output`, `metadata`, `tool_calls`, `expected_output`, `experiment_item_metadata`- jsonPath:- type: optional<string>- docs: |- Optional JSONPath selector applied to the selected source before it is passed to the evaluator prompt.-- Requirements:- - Must start with `$`- - Must be a syntactically valid JSONPath expression- - Most useful with `source=metadata`- examples:- - name: BasicObservationMapping- value:- variable: input- source: input- - name: MetadataProjectionMapping- value:- variable: customer_tier- source: metadata- jsonPath: "$.customer.tier"- - name: ExperimentExpectedOutputMapping- value:- variable: expected_output- source: expected_output-- PromptVariableMappingRead:- docs: |- Connects one prompt variable to data from a target.-- `source` is `null` when the mapping is incomplete.- properties:- variable:- type: string- docs: Prompt variable name without braces.- source:- type: nullable<PromptVariableMappingSource>- docs: Source field populating the variable, or `null` when the mapping is incomplete.- jsonPath:- type: optional<string>- docs: Optional JSONPath selector applied to the selected source.- examples:- - name: ConfiguredReadMapping- value:- variable: input- source: input- - name: IncompleteReadMapping- value:- variable: input- source: null-- EvaluationRuleFilter:- docs: |- One filter condition used to decide whether a live-ingested target should be evaluated.-- An evaluation rule can include zero or more filter objects. All filters must be satisfied for the target to run.-- How to build a valid filter object:- - Pick the `target` first, because it changes the supported columns.- - Pick the filter `type`. That determines which fields are required.- - Use `key` only for object filters such as `metadata`.- - Use the correct `value` shape for the chosen filter `type`.-- Operator quick reference by filter `type`:- - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends with`- - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`- - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`- - `stringOptions`: `any of`, `none of`- - `arrayOptions`: `any of`, `none of`, `all of`- - `stringObject`: same operators as `string`- - `boolean`: `"="`, `"<>"`- - `null`: `is null`, `is not null`-- Supported columns by target:- - `target=observation`- - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`- - `name`: `stringOptions`, operators `any of` / `none of`- - `environment`: `stringOptions`, operators `any of` / `none of`- - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`- - `version`: `string`- - `traceName`: `stringOptions`, operators `any of` / `none of`- - `userId`: `string`- - `sessionId`: `string`- - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`- - `metadata`: `stringObject` with `key`- - `isRootObservation`: `boolean`, operators `=` / `<>`; true when the observation has no parent or is explicitly marked as an application root- - `parentObservationId`: `null`, operators `is null` / `is not null`- - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`- - `toolCalls`: `number`- - `target=experiment`- - `datasetId`: `stringOptions`, operators `any of` / `none of`- Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.-- Recovery guidance:- - `invalid_filter_value` with `details.column` but no `invalidValues`: the selected `column` is not supported for the chosen `target`- - `invalid_filter_value` with `details.invalidValues`: the selected values are not allowed for that column. Replace them with one of `details.allowedValues` when provided.- - `invalid_filter_value` for `column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response.- discriminant: type- union:- "datetime":- type: DateTimeEvaluationRuleFilter- "string":- type: StringEvaluationRuleFilter- "number":- type: NumberEvaluationRuleFilter- "stringOptions":- type: StringOptionsEvaluationRuleFilter- "categoryOptions":- type: CategoryOptionsEvaluationRuleFilter- "arrayOptions":- type: ArrayOptionsEvaluationRuleFilter- "stringObject":- type: StringObjectEvaluationRuleFilter- "numberObject":- type: NumberObjectEvaluationRuleFilter- "boolean":- type: BooleanEvaluationRuleFilter- "null":- type: NullEvaluationRuleFilter- examples:- - name: ObservationTypeFilter- value:- type: stringOptions- column: type- operator: any of- value:- - GENERATION- - name: ObservationMetadataFilter- value:- type: stringObject- column: metadata- key: customerTier- operator: "="- value: enterprise- - name: ObservationRootOnlyFilter- value:- type: "null"- column: parentObservationId- operator: is null- - name: ObservationLogicalRootFilter- value:- type: boolean- column: isRootObservation- operator: "="- value: true- - name: ObservationTagsFilter- value:- type: arrayOptions- column: tags- operator: any of- value:- - production- - name: ExperimentDatasetFilter- value:- type: stringOptions- column: datasetId- operator: any of- value:- - "550e8400-e29b-41d4-a716-446655440000"diff --git a/fern/apis/server/definition/unstable/errors.yml b/fern/apis/server/definition/unstable/errors.ymlindex 180da7af55..74af0edfa0 100644--- a/fern/apis/server/definition/unstable/errors.yml+++ b/fern/apis/server/definition/unstable/errors.yml@@ -1,232 +1,68 @@# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.jsontypes:PublicApiErrorCode:- docs: |- Machine-readable error code returned by the unstable evaluators API.-- SDKs, CLIs, and agents should branch on `code` rather than parsing the human-readable `message`.- The HTTP status still indicates the broad error class, while `code` gives the specific failure reason.+ docs: Machine-readable error code returned by unstable public API endpoints.enum:- authentication_failed- access_denied- invalid_request- invalid_query- invalid_body- - invalid_filter_value- - invalid_json_path- - invalid_variable_mapping- - missing_variable_mapping- - duplicate_variable_mapping- resource_not_found- - name_conflict- - evaluator_preflight_failed- conflict- - unprocessable_content- rate_limited- method_not_allowed- internal_errorPublicApiValidationIssue:- docs: |- One validation issue returned for malformed request bodies or query parameters.-- This mirrors the most important parts of a Zod issue: a machine-readable `code`,- a human-readable `message`, and a structured `path`.+ docs: A validation issue returned for malformed request bodies or query parameters.properties:- code:- type: string- docs: Machine-readable validation issue code emitted by the server validator.- message:- type: string- docs: Human-readable explanation of the validation failure.- path:- type: list<unknown>- docs: Path to the invalid field, for example `["mapping", 0, "jsonPath"]`.+ code: string+ message: string+ path: list<unknown>PublicApiErrorDetails:- docs: |- Optional structured context attached to an unstable-evals error.-- The populated fields depend on the error `code`:- - request parsing failures populate `issues`- - filter validation failures populate `field`, `column`, `invalidValues`, and `allowedValues`- - variable mapping failures populate `field`, `variable`, or `variables`- - JSONPath validation failures populate `field`, `variable`, and `value`- - evaluator preflight failures populate `evaluatorName`, `provider`, and `model`- - rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and `resetAt`+ docs: Optional structured context attached to an error.properties:- issues:- type: optional<list<PublicApiValidationIssue>>- docs: Validation issues for malformed request bodies or query parameters.- field:- type: optional<string>- docs: Path-like reference to the failing field, for example `mapping[1].jsonPath`.- column:- type: optional<string>- docs: Filter column that failed validation.- invalidValues:- type: optional<list<string>>- docs: Unsupported values supplied by the caller.- allowedValues:- type: optional<list<string>>- docs: Allowed values for the failing filter column.- variable:- type: optional<string>- docs: Evaluator variable involved in the failure.- variables:- type: optional<list<string>>- docs: Multiple evaluator variables involved in the failure, for example missing mappings.- value:- type: optional<string>- docs: Raw invalid value supplied by the caller.- evaluatorName:- type: optional<string>- docs: Evaluator name used during preflight validation.- provider:- type: optional<nullable<string>>- docs: Provider resolved during evaluator preflight, if any.- model:- type: optional<nullable<string>>- docs: Model resolved during evaluator preflight, if any.- retryAfterSeconds:- type: optional<integer>- docs: Suggested retry delay for rate-limited requests.- limit:- type: optional<integer>- docs: Numeric limit associated with the failure, for example the active evaluation-rule cap or the current rate-limit window.- remaining:- type: optional<integer>- docs: Remaining requests in the current rate-limit window.- resetAt:- type: optional<string>- docs: ISO-8601 timestamp when the current rate-limit window resets.+ issues: optional<list<PublicApiValidationIssue>>+ field: optional<string>+ column: optional<string>+ invalidValues: optional<list<string>>+ allowedValues: optional<list<string>>+ value: optional<string>+ retryAfterSeconds: optional<integer>+ limit: optional<integer>+ remaining: optional<integer>+ resetAt: optional<string>PublicApiError:- docs: |- Standard error envelope for the unstable evaluators API.-- Response handling guidance:- - Use the HTTP status code for the broad class of failure.- - Use `code` for precise branching in SDKs, CLIs, or agents.- - Inspect `details` for field-level validation context such as invalid filter values, malformed JSONPath expressions, or missing variable mappings.- - Retry only after fixing the specific issue described by `code` and `details`.+ docs: Standard error envelope for unstable public API endpoints.properties:message:type: stringdocs: Human-readable description of the failure.code:type: PublicApiErrorCode- docs: Stable machine-readable error code.+ docs: Machine-readable error code.details:type: optional<PublicApiErrorDetails>- docs: Optional structured error context. Inspect the populated fields based on `code`.- examples:- - name: InvalidFilterValue- value:- message: 'Filter column "type" contains unsupported value(s): INVALID'- code: invalid_filter_value- details:- field: filter[0].value- column: type- invalidValues:- - INVALID- allowedValues:- - GENERATION- - SPAN- - EVENT- - name: InvalidExperimentDatasetFilterValue- value:- message: 'Filter column "datasetId" contains dataset id(s) that do not exist in this project: dataset-prod'- code: invalid_filter_value- details:- field: filter[0].value- column: datasetId- invalidValues:- - dataset-prod- - name: EvaluatorPreflightFailed- value:- message: 'No valid LLM model found for evaluator "answer-correctness". No default model or custom model configured for project project_123'- code: evaluator_preflight_failed- details:- evaluatorName: answer-correctness- provider: null- model: null- - name: MissingVariableMapping- value:- message: "Missing mappings for evaluator variable(s): output"- code: missing_variable_mapping- details:- field: mapping- variables:- - output- - name: InvalidJsonPath- value:- message: 'Mapping for variable "customer_tier" has an invalid jsonPath "$[". JSONPath expressions must use balanced quotes, brackets, and parentheses.'- code: invalid_json_path- details:- field: mapping[0].jsonPath- variable: customer_tier- value: "$["- - name: NameConflict- value:- message: 'An evaluation rule named "answer-quality-live" already exists in this project. Use PATCH /api/public/unstable/evaluation-rules/erule_123 to update it instead of creating a duplicate.'- code: name_conflict- details:- field: name- - name: ActiveEvaluationRuleLimit- value:- message: This project already has the maximum number of active evaluation rules (500). Disable an existing active evaluation rule before enabling another one.- code: conflict- details:- limit: 500- - name: RateLimited- value:- message: Rate limit exceeded- code: rate_limited- details:- retryAfterSeconds: 60- limit: 1000- remaining: 0- resetAt: "2026-03-30T10:00:00.000Z"+ docs: Optional structured error context.errors:BadRequestError:- docs: |- Request parsing or validation failed.-- Typical unstable-eval examples:- - malformed request body or query parameters- - unsupported filter value- - invalid JSONPath selector- - missing or duplicate variable mappings-- Recovery guidance:- - read `details.issues` for malformed bodies or queries- - read `details.column`, `details.invalidValues`, and `details.allowedValues` for filter problems- - for `details.column=datasetId`, call `GET /api/public/v2/datasets` and retry with dataset `id` values from that response- - read `details.variable` or `details.variables` for mapping problems+ docs: Request parsing or validation failed.status-code: 400type: PublicApiErrorUnauthorizedError:- docs: |- Authentication failed.-- Typical unstable-eval examples:- - the API key or secret key is invalid- - the request uses the wrong host or stale credentials+ docs: Authentication failed.status-code: 401type: PublicApiErrorAccessDeniedError:- docs: |- Authenticated, but the caller is not allowed to access the requested resource.-- Typical unstable-eval examples:- - using an organization-scoped key against project-scoped evaluator endpoints- - using a valid key that lacks the required access level for this resource+ docs: The caller is not allowed to access the requested resource.status-code: 403type: PublicApiErrorNotFoundError:- docs: The requested evaluator or evaluation rule does not exist within the authorized project.+ docs: The requested resource does not exist within the authorized project.status-code: 404type: PublicApiErrorMethodNotAllowedError:@@ -234,31 +70,9 @@ errors:status-code: 405type: PublicApiErrorConflictError:- docs: |- The request conflicts with existing state.-- Typical unstable-eval examples:- - evaluator version changed during concurrent creation- - an evaluation rule name already exists in the project, so the caller should PATCH the existing resource instead- - enabling another evaluation rule would exceed the project limit of 500 active evaluation rules- - the underlying state changed between read and write operations, so the client should retry-- Recovery guidance:- - for `name_conflict`, PATCH the existing evaluation rule instead of creating another one- - for the active-limit conflict, disable another active evaluation rule before enabling a new one+ docs: The request conflicts with existing state.status-code: 409type: PublicApiError- UnprocessableContentError:- docs: |- The request is syntactically valid, but Langfuse cannot accept it in its current form.-- The most important unstable-eval case is `code=evaluator_preflight_failed`, which means the evaluator cannot currently run with the resolved model configuration.-- Recovery guidance:- - configure the project's default evaluation model, or send a valid explicit evaluator `modelConfig`- - once the model configuration is valid, retry the same request- status-code: 422- type: PublicApiErrorTooManyRequestsError:docs: The project is rate limited. Retry after the interval described in the response headers and error details.status-code: 429diff --git a/fern/apis/server/definition/unstable/evaluation-rules.yml b/fern/apis/server/definition/unstable/evaluation-rules.ymldeleted file mode 100644index a347e1e506..0000000000--- a/fern/apis/server/definition/unstable/evaluation-rules.yml+++ /dev/null@@ -1,822 +0,0 @@-# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json-imports:- commons: ./commons.yml- errors: ./errors.yml- evals: ../evaluation-commons.yml- pagination: ../utils/pagination.yml--service:- auth: true- base-path: /api/public/unstable- endpoints:- create:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Create an evaluation rule.-- An evaluation rule defines **what** incoming data should be evaluated and **how prompt variables should be populated** from that data.-- Use this resource after choosing an evaluator from the evaluator endpoints.-- Key rules:- - `name` must be unique within the project for public evaluation rules- - `target` must be `observation` or `experiment`- - provide either the compatibility `evaluator` field or the new `evaluators` array, never both- - each evaluator `name` + `type` must identify an existing evaluator family returned by the evaluator endpoints- - Langfuse resolves that family to its latest version before saving the evaluation rule- - for `target=experiment`, use dataset `id` values from `GET /api/public/v2/datasets` when filtering by `datasetId`- - an omitted LLM-as-judge assignment mapping inherits the evaluator version's default mapping- - the effective mapping must map every evaluator prompt variable exactly once- - for `code` evaluators, Langfuse uses the fixed code runtime mapping; omit `mapping` in create and update requests- - for user-provided `llm_as_judge` mappings, `expected_output` and `experiment_item_metadata` are only valid for `target=experiment`- - if `enabled=true`, Langfuse validates that the referenced evaluator can currently run- - at most 500 evaluation rules can be effectively active in one project at the same time (enforced identically by the API, the MCP tools, and the app)-- If an evaluation rule with the same `name` already exists in the project, the API returns `409`.- In that case, update the existing resource with `PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of creating a second one.-- If enabling this resource would exceed the 500-active limit, the API also returns `409`.- In that case, disable or pause another active evaluation rule before enabling a new one.-- Current scope:- - evaluation rules are live-ingestion rules only- - they do not trigger historical backfills-- Recovery guidance:- - `400 invalid_filter_value`: fix the filter `column` or `value` using `details.column`, `details.invalidValues`, and `details.allowedValues`- - `400 invalid_filter_value` with `details.column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response- - `400 missing_variable_mapping`: for `llm_as_judge` evaluators, fetch the evaluator again and make sure every variable in `variables` appears exactly once in `mapping`- - `400 duplicate_variable_mapping`: remove repeated mappings for the same variable- - `400 invalid_variable_mapping`: for `llm_as_judge`, switch to a valid `source` for the selected `target`, or fix the variable name- - `400 invalid_json_path`: remove or correct the `jsonPath`- - `422 evaluator_preflight_failed`: the selected evaluator cannot run with the resolved model configuration. Fix the evaluator/default model setup, then retry the create request.- method: POST- path: /evaluation-rules- request: CreateEvaluationRuleRequest- response: EvaluationRule- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.NotFoundError- - errors.ConflictError- - errors.MethodNotAllowedError- - errors.UnprocessableContentError- - errors.TooManyRequestsError- - errors.InternalServerError- examples:- - name: CreateObservationEvaluationRule- docs: Deploy an evaluator to score live generations only.- request:- name: answer-correctness-live- evaluator:- name: answer-correctness- type: llm_as_judge- target: observation- enabled: true- sampling: 1- filter:- - type: stringOptions- column: type- operator: any of- value:- - GENERATION- mapping:- - variable: input- source: input- - variable: output- source: output- response:- body:- id: erule_123- name: answer-correctness-live- evaluator:- id: evaltmpl_123- name: answer-correctness- type: llm_as_judge- evaluators:- - evaluator:- id: evaltmpl_123- name: answer-correctness- type: llm_as_judge- mapping:- - variable: input- source: input- - variable: output- source: output- target: observation- enabled: true- status: active- pausedReason: null- pausedMessage: null- sampling: 1- filter:- - type: stringOptions- column: type- operator: any of- value:- - GENERATION- mapping:- - variable: input- source: input- - variable: output- source: output- createdAt: "2026-03-30T09:20:00.000Z"- updatedAt: "2026-03-30T09:20:00.000Z"- - name: CreateCodeObservationEvaluationRule- docs: Deploy a code evaluator to score live generation outputs.- request:- name: toxicity-code-live- evaluator:- name: toxicity-detector- type: code- target: observation- enabled: true- sampling: 1- filter:- - type: stringOptions- column: type- operator: any of- value:- - GENERATION- response:- body:- id: erule_234- name: toxicity-code-live- evaluator:- id: evaltmpl_234- name: toxicity-detector- type: code- evaluators:- - evaluator:- id: evaltmpl_234- name: toxicity-detector- type: code- mapping: null- target: observation- enabled: true- status: active- pausedReason: null- pausedMessage: null- sampling: 1- filter:- - type: stringOptions- column: type- operator: any of- value:- - GENERATION- mapping:- - variable: input- source: input- - variable: output- source: output- - variable: metadata- source: metadata- - variable: experimentItemExpectedOutput- source: expected_output- - variable: experimentItemMetadata- source: experiment_item_metadata- createdAt: "2026-03-30T09:25:00.000Z"- updatedAt: "2026-03-30T09:25:00.000Z"- - name: CreateExperimentEvaluationRule- docs: Deploy an evaluator to compare experiment outputs against expected outputs. Discover valid dataset IDs with `GET /api/public/v2/datasets` first.- request:- name: experiment-expected-output-match- evaluator:- name: expected-output-match- type: llm_as_judge- target: experiment- enabled: true- sampling: 0.5- filter:- - type: stringOptions- column: datasetId- operator: any of- value:- - "550e8400-e29b-41d4-a716-446655440000"- mapping:- - variable: output- source: output- - variable: expected_output- source: expected_output- response:- body:- id: erule_456- name: experiment-expected-output-match- evaluator:- id: evaltmpl_456- name: expected-output-match- type: llm_as_judge- evaluators:- - evaluator:- id: evaltmpl_456- name: expected-output-match- type: llm_as_judge- mapping:- - variable: output- source: output- - variable: expected_output- source: expected_output- target: experiment- enabled: true- status: active- pausedReason: null- pausedMessage: null- sampling: 0.5- filter:- - type: stringOptions- column: datasetId- operator: any of- value:- - "550e8400-e29b-41d4-a716-446655440000"- mapping:- - variable: output- source: output- - variable: expected_output- source: expected_output- createdAt: "2026-03-30T09:30:00.000Z"- updatedAt: "2026-03-30T09:30:00.000Z"-- list:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- List evaluation rules in the authenticated project.-- This includes legacy `trace` and `dataset` rules so they can be inspected and migrated to v4 rules. Legacy rules are read-only through this API; create, update, and delete continue to support only `observation` and `experiment` rules.- method: GET- path: /evaluation-rules- request:- name: ListEvaluationRulesRequest- query-parameters:- page:- type: optional<integer>- docs: 1-based page number. Defaults to `1`.- limit:- type: optional<integer>- docs: Maximum number of items per page. Defaults to `50`.- response: EvaluationRules- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.MethodNotAllowedError- - errors.TooManyRequestsError- - errors.InternalServerError-- get:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Get one evaluation rule by its identifier.-- Use this endpoint to inspect the current evaluator, target, mapping, filters, execution timing, and effective runtime status. Legacy `trace` and `dataset` rules are returned for migration and are read-only through this API.- method: GET- path: /evaluation-rules/{evaluationRuleId}- path-parameters:- evaluationRuleId:- type: string- docs: Evaluation rule identifier returned by the evaluation rule endpoints.- response: ReadableEvaluationRule- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.NotFoundError- - errors.MethodNotAllowedError- - errors.TooManyRequestsError- - errors.InternalServerError-- update:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Update an evaluation rule.-- Typical uses:- - enable or disable live execution- - switch to another evaluator- - adjust sampling- - change filters- - update LLM-as-judge variable mappings-- Important behavior:- - provide only the fields you want to change- - if you provide `evaluator`, Langfuse resolves that evaluator family to its latest version before saving- - changing `target`, `filter`, or an LLM-as-judge `mapping` must still produce a valid target-specific configuration- - if you change `target` for an LLM-as-judge rule, also send a compatible `filter` and `mapping` in the same request unless the existing ones are still valid for the new target- - for `code` evaluator rules, omit `mapping`; Langfuse stores the fixed code runtime mapping automatically- - if the resulting config is enabled, Langfuse re-validates that the selected evaluator can run- - if the update would move a non-active evaluation rule into the active state and the project already has 500 active evaluation rules, the API returns `409`-- Recovery guidance:- - if an LLM-as-judge update fails with `missing_variable_mapping` or `invalid_variable_mapping` after changing `evaluator` or `target`, resend the request with a complete new `mapping`- - if the update fails with `invalid_filter_value` after changing `target`, resend the request with a target-compatible `filter`- method: PATCH- path: /evaluation-rules/{evaluationRuleId}- path-parameters:- evaluationRuleId:- type: string- docs: Evaluation rule identifier.- request: UpdateEvaluationRuleRequest- response: ReadableV2EvaluationRule- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.NotFoundError- - errors.MethodNotAllowedError- - errors.UnprocessableContentError- - errors.TooManyRequestsError- - errors.InternalServerError-- delete:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Delete an evaluation rule.-- This removes the live-ingestion rule only. It does not delete the referenced evaluator.- method: DELETE- path: /evaluation-rules/{evaluationRuleId}- path-parameters:- evaluationRuleId:- type: string- docs: Evaluation rule identifier.- response: DeleteEvaluationRuleResponse- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.NotFoundError- - errors.MethodNotAllowedError- - errors.TooManyRequestsError- - errors.InternalServerError--types:- EvaluationRuleBase:- docs: |- Live evaluation rule for incoming data.-- An evaluation rule answers:- - which evaluator should be used- - which target objects should trigger scoring- - how often scoring should run- - which target fields should populate each prompt variable- - whether the deployment is active, inactive, or paused-- Important status semantics:- - `enabled` is the desired on/off setting from the client- - `status` is the effective runtime state after Langfuse applies validation and blocking rules- - `enabled=true` with `status=paused` means the rule should run, but Langfuse has paused it until the underlying problem is fixed- properties:- id:- type: string- docs: Stable evaluation rule identifier.- name:- type: string- docs: Human-readable deployment name. This is independent from the evaluator name.- evaluator:- type: nullable<EvaluationRuleEvaluator>- docs: |- Deprecated compatibility alias for `evaluators[0].evaluator`, or `null`- when the rule currently has no evaluator assignments.-- `id` identifies the evaluator family. The rule automatically uses the- latest available version of that evaluator.- enabled:- type: boolean- docs: Desired enabled state configured by the client.- status:- type: commons.EvaluationRuleStatus- docs: Effective runtime status after Langfuse applies validation and blocking rules.- pausedReason:- type: nullable<string>- docs: Machine-readable reason when `status=paused`, otherwise `null`.- pausedMessage:- type: nullable<string>- docs: Human-readable explanation when `status=paused`, otherwise `null`.- sampling:- type: double- docs: |- Fraction of matching target objects that should be evaluated.-- Must be greater than `0` and less than or equal to `1`.- - `1` means evaluate every matching target.- - `0.25` means evaluate approximately 25% of matching targets.- createdAt:- type: datetime- docs: Timestamp when the evaluation rule was created.- updatedAt:- type: datetime- docs: Timestamp when the evaluation rule was last updated.-- EvaluationRule:- extends: EvaluationRuleBase- properties:- evaluators:- type: list<EvaluationRuleEvaluatorAssignment>- docs: Evaluators attached to this rule in deterministic assignment order. A `null` mapping inherits the evaluator version's default mapping.- target:- type: commons.EvaluationRuleTarget- docs: Target object type that should trigger scoring.- filter:- type: list<commons.EvaluationRuleFilter>- docs: List of filter conditions used to decide whether a target should be evaluated.- mapping:- type: list<commons.PromptVariableMappingRead>- docs: Deprecated compatibility alias containing the effective mapping for `evaluators[0]`.-- ReadableV2EvaluationRule:- extends: EvaluationRuleBase- properties:- evaluators:- type: list<EvaluationRuleEvaluatorAssignment>- docs: Evaluators attached to this rule in deterministic assignment order. A `null` mapping inherits the evaluator version's default mapping.- target:- type: commons.EvaluationRuleTarget- docs: Target object type that should trigger scoring.- filter:- type: list<evals.EvaluationRuleReadFilter>- docs: List of stored filter conditions returned verbatim. Historical filters remain readable even when the current write contract no longer accepts their shape.- mapping:- type: list<commons.PromptVariableMappingRead>- docs: Deprecated compatibility alias containing the effective mapping for `evaluators[0]`.- examples:- - value:- id: erule_123- name: answer-correctness-live- evaluator:- id: evaltmpl_123- name: answer-correctness- type: llm_as_judge- evaluators:- - evaluator:- id: evaltmpl_123- name: answer-correctness- type: llm_as_judge- mapping:- - variable: input- source: input- - variable: output- source: output- target: observation- enabled: true- status: active- pausedReason: null- pausedMessage: null- sampling: 1- filter:- - type: stringOptions- column: type- operator: any of- value:- - GENERATION- mapping:- - variable: input- source: input- - variable: output- source: output- createdAt: "2026-03-30T09:20:00.000Z"- updatedAt: "2026-03-30T09:20:00.000Z"-- LegacyEvaluationRule:- docs: |- **Deprecated:** Legacy trace- or dataset-level evaluation rule returned by list and get for migration.-- This resource is read-only through the unstable public API. Its mapping preserves the trace, dataset item, or named observation selected for each prompt variable.- extends: EvaluationRuleBase- properties:- evaluators:- type: list<LegacyEvaluationRuleEvaluatorAssignment>- docs: Evaluators attached to this rule in deterministic assignment order.- target:- type: LegacyEvaluationRuleTarget- delay:- type: integer- docs: Delay in milliseconds before the legacy evaluation job runs.- timeScope:- type: list<EvaluationRuleTimeScope>- docs: Whether the legacy rule evaluates newly ingested data, existing data, or both.- filter:- type: list<commons.EvaluationRuleFilter>- docs: Stored filters used by the legacy trace or dataset rule.- mapping:- type: list<LegacyPromptVariableMapping>- docs: Stored variable mappings, including the trace, dataset item, or named observation selected for each variable.-- ReadableEvaluationRule:- docs: Evaluation rule returned by list and get, including read-only legacy trace and dataset rules.- discriminated: false- union:- - ReadableV2EvaluationRule- - LegacyEvaluationRule-- EvaluationRules:- docs: Paginated list of evaluation rules.- properties:- data:- type: list<ReadableEvaluationRule>- docs: Evaluation rules in the current page.- meta:- type: pagination.MetaResponse- docs: Standard pagination metadata.-- CreateEvaluationRuleRequest:- docs: |- Request body for creating an evaluation rule.-- Checklist for agents and SDK clients:- - reference an existing evaluator family by `evaluator.name` and `evaluator.type`- - choose `target=observation` or `target=experiment`- - if `target=experiment` and you want a dataset filter, call `GET /api/public/v2/datasets` first and use dataset `id` values in `filter[].value`- - for `llm_as_judge`, fetch or inspect the evaluator first and provide a complete mapping for every prompt variable- - for `code`, do not send variables or mappings; Langfuse stores the fixed code runtime mapping automatically- - optionally narrow execution with `filter`- - set `enabled=true` only when you want live execution immediately- discriminated: false- union:- - CreateLlmAsJudgeEvaluationRuleRequest- - CreateCodeEvaluationRuleRequest- - CreateEvaluationRuleWithEvaluatorsRequest-- CreateEvaluationRuleWithEvaluatorsRequest:- properties:- name:- type: string- docs: Human-readable deployment name.- evaluators:- type: list<CreateEvaluationRuleEvaluatorAssignment>- docs: |- One or more evaluator assignments. Providing the deprecated top-level `evaluator` or `mapping` fields alongside this is rejected with `400`.-- Multiple assignments are supported on writable targets.- target:- type: commons.EvaluationRuleTarget- enabled:- type: boolean- sampling:- type: optional<double>- filter:- type: optional<list<commons.EvaluationRuleFilter>>-- CreateLlmAsJudgeEvaluationRuleRequest:- properties:- name:- type: string- docs: Human-readable deployment name.- evaluator:- type: LlmAsJudgeEvaluationRuleEvaluatorReference- docs: |- LLM-as-judge evaluator family to use.-- Use `name` and `type` from the evaluator endpoints. If `type` is omitted, Langfuse defaults it to `llm_as_judge` for backwards compatibility.- Langfuse resolves that family to its latest version before saving the rule.- target:- type: commons.EvaluationRuleTarget- docs: Target object type to evaluate.- enabled:- type: boolean- docs: Whether the deployment should be active immediately after creation.- sampling:- type: optional<double>- docs: Optional sampling fraction. Defaults to `1`.- filter:- type: optional<list<commons.EvaluationRuleFilter>>- docs: |- Optional filter list.-- Omit or pass an empty list to evaluate all matching targets for the selected `target`.- Each filter object must use a column that is valid for that `target`.- For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.- mapping:- type: list<commons.PromptVariableMappingInput>- docs: |- LLM-as-judge variable mappings.-- Every prompt variable must appear exactly once.- Build this list from the evaluator's `variables` array.-- CreateCodeEvaluationRuleRequest:- properties:- name:- type: string- docs: Human-readable deployment name.- evaluator:- type: CodeEvaluationRuleEvaluatorReference- docs: |- Code evaluator family to use.-- Use `name` and `type` from the evaluator endpoints.- Langfuse resolves that family to its latest version before saving the rule.- target:- type: commons.EvaluationRuleTarget- docs: Target object type to evaluate.- enabled:- type: boolean- docs: Whether the deployment should be active immediately after creation.- sampling:- type: optional<double>- docs: Optional sampling fraction. Defaults to `1`.- filter:- type: optional<list<commons.EvaluationRuleFilter>>- docs: |- Optional filter list.-- Omit or pass an empty list to evaluate all matching targets for the selected `target`.- Each filter object must use a column that is valid for that `target`.- For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.-- UpdateEvaluationRuleRequest:- docs: |- Partial update body for an evaluation rule.-- Provide only the fields you want to change.- An empty body is rejected.-- Practical guidance:- - If you only want to rename the rule or change sampling, send just those fields.- - To add, remove, or remap evaluators, send `evaluators`. It replaces the whole assignment set, so include every evaluator the rule should keep.- - `evaluators` cannot be combined with the deprecated `evaluator`/`mapping` pair, which only ever addressed the first assignment.- - If you change to an LLM-as-judge `evaluator`, send a fresh `mapping` unless you are certain the existing mapping still matches the prompt variables.- - If you change `target` for an LLM-as-judge rule, usually send both `filter` and `mapping` in the same request.- - For code evaluator rules, omit `mapping`; Langfuse stores the fixed code runtime mapping automatically.- - If you change an experiment `datasetId` filter, call `GET /api/public/v2/datasets` and use dataset `id` values from that response.- properties:- name:- type: optional<string>- docs: Updated deployment name.- evaluators:- type: optional<list<CreateEvaluationRuleEvaluatorAssignment>>- docs: |- Full replacement of the rule's evaluator assignments: entries that are- not listed are detached.-- Mutually exclusive with the deprecated `evaluator` and `mapping` fields.- evaluator:- type: optional<EvaluationRuleEvaluatorReference>- docs: |- Deprecated single-evaluator alias: updates the first assignment only. Prefer `evaluators`.-- Langfuse resolves the provided evaluator family to its latest version before saving the rule.- A rule's evaluator type cannot be changed: provide `name` for an evaluator family of the rule's current type. To use a different evaluator type, create a new rule.- target:- type: optional<commons.EvaluationRuleTarget>- docs: Updated target object type.- enabled:- type: optional<boolean>- docs: Updated desired enabled state.- sampling:- type: optional<double>- docs: Updated sampling fraction.- filter:- type: optional<list<commons.EvaluationRuleFilter>>- docs: |- Updated filter list.-- For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.- mapping:- type: optional<list<commons.PromptVariableMappingInput>>- docs: |- Updated LLM-as-judge variable mappings.-- Do not send this field for code evaluator rules. Langfuse stores the fixed code runtime mapping automatically and returns it in the response.-- DeleteEvaluationRuleResponse:- docs: Confirmation response returned after successful deletion.- properties:- message:- type: string- docs: Always `Evaluation rule successfully deleted`.-- EvaluationRuleEvaluatorReference:- docs: |- Evaluator family reference used when updating an evaluation rule.-- `name` identifies the evaluator family in the authenticated project context.- A rule's evaluator type cannot be changed, so this reference does not accept a `type`; the family must match the rule's current evaluator type.- properties:- name:- type: string- docs: Evaluator family name.-- LlmAsJudgeEvaluationRuleEvaluatorReference:- docs: LLM-as-judge evaluator family reference used when creating an evaluation rule.- properties:- name:- type: string- docs: Evaluator family name.- type:- type: optional<LlmAsJudgeEvaluatorType>- docs: Evaluator type. Defaults to `llm_as_judge` when omitted.-- CodeEvaluationRuleEvaluatorReference:- docs: Code evaluator family reference used when creating an evaluation rule.- properties:- name:- type: string- docs: Evaluator family name.- type:- type: literal<"code">- docs: Must be `code`.-- LlmAsJudgeEvaluatorType:- enum:- - llm_as_judge-- EvaluationRuleEvaluator:- docs: |- Resolved evaluator currently used by the evaluation rule.-- `id` identifies the evaluator family. Evaluation runs automatically use- the latest available evaluator version.- properties:- id:- type: string- docs: Identifier of the exact evaluator version currently used by the rule.- name:- type: string- docs: Evaluator family name.- type:- type: commons.EvaluatorType- docs: Evaluator type.-- EvaluationRuleEvaluatorAssignment:- properties:- evaluator:- type: EvaluationRuleEvaluator- mapping:- type: nullable<list<commons.PromptVariableMappingRead>>- docs: Rule-specific override, or `null` to inherit the evaluator version's default mapping.-- LegacyEvaluationRuleEvaluatorAssignment:- docs: "**Deprecated:** Evaluator assignment on a legacy trace or dataset rule."- properties:- evaluator:- type: EvaluationRuleEvaluator- mapping:- type: nullable<list<LegacyPromptVariableMapping>>-- EvaluationRuleTimeScope:- enum:- - NEW- - EXISTING-- LegacyEvaluationRuleTarget:- docs: "**Deprecated:** Legacy trace or dataset evaluation target."- enum:- - trace- - dataset-- LegacyPromptVariableMapping:- docs: "**Deprecated:** Connects one prompt variable to data from a trace, dataset item, or named observation in a legacy rule."- properties:- variable:- type: string- langfuseObject:- type: LegacyEvaluationObject- objectName:- type: nullable<string>- docs: Observation name to match, or `null` when `langfuseObject` is `trace` or `dataset_item`.- source:- type: string- jsonPath:- type: optional<string>-- LegacyEvaluationObject:- docs: "**Deprecated:** Legacy Langfuse object used by a trace or dataset evaluation rule."- enum:- - trace- - span- - generation- - event- - agent- - tool- - chain- - retriever- - evaluator- - embedding- - guardrail- - dataset_item-- CreateEvaluationRuleEvaluatorAssignment:- properties:- evaluator:- type: CreateEvaluationRuleEvaluatorReference- mapping:- type: optional<list<commons.PromptVariableMappingInput>>- docs: Optional rule-specific override. Omit to inherit the evaluator version's default mapping. Omit for code evaluators.-- CreateEvaluationRuleEvaluatorReference:- properties:- name:- type: string- type:- type: optional<commons.EvaluatorType>- docs: Defaults to `llm_as_judge`.diff --git a/fern/apis/server/definition/unstable/evaluators.yml b/fern/apis/server/definition/unstable/evaluators.ymldeleted file mode 100644index 86f64bee17..0000000000--- a/fern/apis/server/definition/unstable/evaluators.yml+++ /dev/null@@ -1,490 +0,0 @@-# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json-imports:- commons: ./commons.yml- errors: ./errors.yml- pagination: ../utils/pagination.yml--service:- auth: true- base-path: /api/public/unstable- endpoints:- create:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Create an evaluator in the authenticated project.-- Use evaluators to define **how** Langfuse should score data.- LLM-as-a-judge evaluators define a prompt, expected structured output, and optional model configuration.- Code evaluators define source code and a runtime language.-- Naming behavior:- - If this is a new evaluator name in your project, Langfuse creates version `1`.- - If the name already exists in your project, Langfuse creates the next version and returns it.- - The evaluator `id` remains stable across versions.- - Existing evaluation rules automatically use the latest evaluator version; no rule update is required.-- Recommended workflow:- 1. Create the evaluator.- 2. Read the returned `variables` array.- 3. Read the returned `outputDefinition.dataType` so the client knows whether future scores will be numeric, boolean, or categorical.- 4. Create one or more evaluation rules that reference the returned evaluator family using `name` and `type`.-- Code evaluator validation:- - At creation, Langfuse only validates the request shape- - The `sourceCode` itself is not executed here. It is first run (preflight-tested against a sample observation) when you link the evaluator to an evaluation rule, so runtime errors in the code surface at evaluation-rule creation, not at evaluator creation.-- Recovery guidance:- - `422` with `code=evaluator_preflight_failed`: the evaluator cannot run with the resolved model configuration. Add a valid explicit `modelConfig`, or configure the project's default evaluation model, then retry the same request.- - `400` with `code=invalid_body`: the request shape is malformed. Use the structured `details.issues` array to fix the specific fields and retry.- - `400` with `code=invalid_body` on `outputDefinition`: for `type=llm_as_judge`, send `dataType`, `reasoning.description`, and `score.description`. Do not send `version`; it is not part of the public request shape.- - If `type` is omitted, Langfuse treats the request as `type=llm_as_judge` for backwards compatibility. New clients should send `type` explicitly.-- Unstable API note:- - This surface may evolve while the underlying evaluation data model is being redesigned.- method: POST- path: /evaluators- request: CreateEvaluatorRequest- response: Evaluator- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.MethodNotAllowedError- - errors.ConflictError- - errors.UnprocessableContentError- - errors.TooManyRequestsError- - errors.InternalServerError- examples:- - name: CreateEvaluatorVersion- docs: Create a new version of an evaluator named `answer-correctness`.- request:- name: answer-correctness- type: llm_as_judge- prompt: |- You are grading an answer.-- Input:- {{input}}-- Output:- {{output}}-- Return a score between 0 and 1.- outputDefinition:- dataType: NUMERIC- reasoning:- description: Explain why the score was assigned.- score:- description: Correctness score between 0 and 1.- modelConfig:- provider: openai- model: gpt-4.1-mini- mapping:- - variable: input- source: input- - variable: output- source: output- response:- body:- id: evaltmpl_123- name: answer-correctness- version: 2- type: llm_as_judge- prompt: |- You are grading an answer.-- Input:- {{input}}-- Output:- {{output}}-- Return a score between 0 and 1.- variables:- - input- - output- mapping:- - variable: input- source: input- - variable: output- source: output- outputDefinition:- dataType: NUMERIC- reasoning:- description: Explain why the score was assigned.- score:- description: Correctness score between 0 and 1.- modelConfig:- provider: openai- model: gpt-4.1-mini- evaluationRuleCount: 0- createdAt: "2026-03-30T09:00:00.000Z"- updatedAt: "2026-03-30T09:00:00.000Z"- - name: CreateTypeScriptCodeEvaluator- docs: |- Create a TypeScript code evaluator named `exact-match`.-- Send only the `evaluate` function; Langfuse injects the `EvaluationContext`,- `EvaluationResult`, and `Score` type definitions at runtime.- Code evaluators always expose the fixed runtime payload variables and do not take a `mapping`.- request:- name: exact-match- type: code- sourceCode: |- function evaluate(ctx: EvaluationContext): EvaluationResult {- const input = ctx.observation.input;- const matchesOutput =- input !== undefined && ctx.observation.output === input;-- return {- scores: [- {- name: "Exact match",- value: matchesOutput,- dataType: "BOOLEAN",- comment: matchesOutput- ? "Output exactly matches the input."- : "Output does not match the input.",- },- ],- };- }- sourceCodeLanguage: TYPESCRIPT- response:- body:- id: evaltmpl_ts_123- name: exact-match- version: 1- type: code- variables:- - input- - output- - metadata- - toolCalls- - experimentItemExpectedOutput- - experimentItemMetadata- mapping: null- sourceCode: |- function evaluate(ctx: EvaluationContext): EvaluationResult {- const input = ctx.observation.input;- const matchesOutput =- input !== undefined && ctx.observation.output === input;-- return {- scores: [- {- name: "Exact match",- value: matchesOutput,- dataType: "BOOLEAN",- comment: matchesOutput- ? "Output exactly matches the input."- : "Output does not match the input.",- },- ],- };- }- sourceCodeLanguage: TYPESCRIPT- evaluationRuleCount: 0- createdAt: "2026-03-30T09:10:00.000Z"- updatedAt: "2026-03-30T09:10:00.000Z"- - name: CreatePythonCodeEvaluator- docs: |- Create a Python code evaluator named `exact-match`.-- Send only the `evaluate` function; Langfuse injects the `EvaluationContext`,- `EvaluationResult`, and `Score` definitions at runtime.- Code evaluators always expose the fixed runtime payload variables and do not take a `mapping`.- request:- name: exact-match- type: code- sourceCode: |- def evaluate(ctx: EvaluationContext) -> EvaluationResult:- """Evaluates one observation and returns one or more Langfuse scores."""- input = ctx.observation.input- matches_output = input is not None and ctx.observation.output == input-- return EvaluationResult(- scores=[- Score(- name="Exact match",- value=matches_output,- data_type="BOOLEAN",- comment=(- "Output exactly matches the input."- if matches_output- else "Output does not match the input."- ),- )- ]- )- sourceCodeLanguage: PYTHON- response:- body:- id: evaltmpl_py_123- name: exact-match- version: 1- type: code- variables:- - input- - output- - metadata- - toolCalls- - experimentItemExpectedOutput- - experimentItemMetadata- mapping: null- sourceCode: |- def evaluate(ctx: EvaluationContext) -> EvaluationResult:- """Evaluates one observation and returns one or more Langfuse scores."""- input = ctx.observation.input- matches_output = input is not None and ctx.observation.output == input-- return EvaluationResult(- scores=[- Score(- name="Exact match",- value=matches_output,- data_type="BOOLEAN",- comment=(- "Output exactly matches the input."- if matches_output- else "Output does not match the input."- ),- )- ]- )- sourceCodeLanguage: PYTHON- evaluationRuleCount: 0- createdAt: "2026-03-30T09:15:00.000Z"- updatedAt: "2026-03-30T09:15:00.000Z"-- list:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- List the evaluators available to the authenticated project.-- Important behavior:- - This endpoint returns the latest version of each available evaluator.- - Every evaluator is owned by the authenticated project.- method: GET- path: /evaluators- request:- name: ListEvaluatorsRequest- query-parameters:- page:- type: optional<integer>- docs: 1-based page number. Defaults to `1`.- limit:- type: optional<integer>- docs: Maximum number of items per page. Defaults to `50`.- response: Evaluators- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.MethodNotAllowedError- - errors.TooManyRequestsError- - errors.InternalServerError-- get:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Get one evaluator by `id`.-- This endpoint always returns the evaluator's latest version. Use it when you want the current prompt, output definition, model configuration, and derived variables for the evaluator you plan to use in an evaluation rule.- method: GET- path: /evaluators/{evaluatorId}- path-parameters:- evaluatorId:- type: string- docs: Evaluator identifier returned by the evaluator endpoints.- response: Evaluator- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.NotFoundError- - errors.MethodNotAllowedError- - errors.TooManyRequestsError- - errors.InternalServerError-- delete:- availability:- status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."- docs: |- Delete an evaluator.-- Important behavior:- - This deletes the evaluator including all of its stored versions.- - Evaluation rule assignments referencing the evaluator are also deleted.- - Scores already produced by the evaluator are not deleted.- method: DELETE- path: /evaluators/{evaluatorId}- path-parameters:- evaluatorId:- type: string- docs: Evaluator identifier returned by the evaluator endpoints.- response: DeleteEvaluatorResponse- errors:- - errors.BadRequestError- - errors.UnauthorizedError- - errors.AccessDeniedError- - errors.NotFoundError- - errors.MethodNotAllowedError- - errors.TooManyRequestsError- - errors.InternalServerError--types:- DeleteEvaluatorResponse:- docs: Confirmation response returned after successful deletion.- properties:- message:- type: string- docs: Always `Evaluator successfully deleted`.-- Evaluator:- docs: |- One evaluator that can be used for scoring.-- An evaluator describes **how** to score data.-- It does not define **which** live objects are evaluated. That is the job of `evaluation-rules`.-- For agent clients, the most important fields are:- - `type`: determines which evaluator fields are present- - `variables`: for LLM evaluators, use these exact names when building the evaluation-rule `mapping` array. LLM evaluators require every variable to be mapped. Code evaluators always expose the fixed runtime payload fields and Langfuse maps them automatically.-- Versioning behavior:- - `GET /evaluators` returns the latest version of each available evaluator.- - `GET /evaluators/{id}` returns the latest version.- - `id` identifies the evaluator family and remains stable when a new version is created.- - Evaluation rules always run against the latest version for the selected evaluator name within the same source (`project` or `managed`).- discriminant: type- union:- llm_as_judge:- type: LlmAsJudgeEvaluator- code:- type: CodeEvaluator-- EvaluatorBase:- properties:- id:- type: string- docs: Stable identifier of this evaluator across all versions.- name:- type: string- docs: Evaluator name.- version:- type: integer- docs: Version number of this evaluator.- variables:- type: list<string>- docs: |- Variables that can be mapped when creating an evaluation rule.-- LLM evaluators require every variable to be mapped exactly once. Code evaluators always expose the fixed runtime payload fields and Langfuse maps them automatically.- mapping:- type: nullable<list<commons.PromptVariableMappingRead>>- docs: |- Default variable mapping for this evaluator version, or `null` when no default is configured.-- An entry's `source` is `null` when that variable was never fully configured, and sources- are not restricted by rule `target` here, because the default is stored on the evaluator- rather than on any one rule.- evaluationRuleCount:- type: integer- docs: Number of evaluation rules in the project that currently use this evaluator.- createdAt:- type: datetime- docs: Timestamp when this evaluator was created.- updatedAt:- type: datetime- docs: Timestamp when this evaluator was last updated.-- LlmAsJudgeEvaluator:- extends: EvaluatorBase- properties:- prompt:- type: string- docs: Prompt template used during evaluation.- outputDefinition:- type: commons.PublicEvaluatorOutputDefinition- docs: |- Structured output schema returned by this evaluator.-- Responses always include `dataType` and omit the internal output-definition `version`.- Use `dataType` to decide how future scores should be interpreted.- modelConfig:- type: nullable<commons.EvaluatorModelConfig>- docs: Explicit model configuration, or `null` when the project default evaluation model is used.-- CodeEvaluator:- extends: EvaluatorBase- properties:- sourceCode:- type: string- docs: Source code executed for each matched observation.- sourceCodeLanguage:- type: commons.CodeEvaluatorSourceCodeLanguage- docs: Runtime language for `sourceCode`.-- Evaluators:- properties:- data:- type: list<Evaluator>- meta:- type: pagination.MetaResponse-- CreateEvaluatorRequest:- docs: |- Request body for creating an evaluator.-- If the same `name` already exists in your project, Langfuse creates the next version and returns it.- Existing evaluation rules automatically use the latest evaluator version.- If `type` is omitted, Langfuse defaults it to `llm_as_judge` for backwards compatibility.- discriminant: type- union:- llm_as_judge:- type: CreateLlmAsJudgeEvaluatorRequest- code:- type: CreateCodeEvaluatorRequest-- CreateLlmAsJudgeEvaluatorRequest:- properties:- name:- type: string- docs: Evaluator name within the authenticated project.- prompt:- type: string- docs: Prompt template used by the evaluator.- outputDefinition:- type: commons.EvaluatorOutputDefinition- docs: |- Structured output schema the evaluator must return.-- Always send `dataType`.- Do not send `version`; it is an internal storage detail and not part of the public request contract.- modelConfig:- type: optional<nullable<commons.EvaluatorModelConfig>>- docs: Optional explicit model configuration. Omit or set to `null` to use the project default evaluation model.- mapping:- type: optional<list<commons.PromptVariableMappingInput>>- docs: Optional default variable mapping inherited by rule assignments that do not provide an override.-- CreateCodeEvaluatorRequest:- properties:- name:- type: string- docs: Evaluator name within the authenticated project.- sourceCode:- type: string- docs: Code executed for each matched observation.- sourceCodeLanguage:- type: commons.CodeEvaluatorSourceCodeLanguage- docs: Runtime language for `sourceCode`.diff --git a/knip.jsonc b/knip.jsoncindex 93e2568e41..1cb5d48f1d 100644--- a/knip.jsonc+++ b/knip.jsonc@@ -7,8 +7,6 @@"workspaces": {"packages/shared": {"ignoreFiles": [- // Standalone renderer invoked by ClickHouse migration shell scripts and Docker builds.- "clickhouse/scripts/prepare-migrations.mjs",// Standalone helper invoked manually as documented in the adjacent framework-traces README."scripts/seeder/utils/framework-traces/merge-observations.ts",],diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex d1b1951f8c..e9e5dcc725 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -760,9 +760,6 @@ importers:json-schema-faker:specifier: 0.6.1version: 0.6.1- jsonpath-plus:- specifier: 10.3.0- version: 10.3.0langfuse:specifier: 3.38.4version: 3.38.4@@ -19318,17 +19315,6 @@ snapshots:transitivePeerDependencies:- supports-color- [email protected](@typescript-eslint/[email protected]([email protected]([email protected])([email protected]))([email protected])([email protected]))([email protected]([email protected]))([email protected])([email protected]([email protected])([email protected]))([email protected]):- dependencies:- debug: 3.2.7([email protected])- optionalDependencies:- '@typescript-eslint/parser': 8.67.0([email protected]([email protected])([email protected]))([email protected])([email protected])- eslint: 10.8.0([email protected])([email protected])- eslint-import-resolver-node: 0.3.10([email protected])- eslint-import-resolver-typescript: 3.10.1([email protected])([email protected]([email protected])([email protected]))([email protected])- transitivePeerDependencies:- - supports-color-[email protected](@typescript-eslint/[email protected]([email protected]([email protected])([email protected]))([email protected])([email protected]))([email protected]([email protected]))([email protected])([email protected]([email protected])([email protected]))([email protected]):dependencies:debug: 3.2.7([email protected])@@ -19384,35 +19370,6 @@ snapshots:- eslint-import-resolver-webpack- supports-color- [email protected](@typescript-eslint/[email protected]([email protected]([email protected])([email protected]))([email protected])([email protected]))([email protected])([email protected]([email protected])([email protected]))([email protected]):- dependencies:- '@rtsao/scc': 1.1.0- array-includes: 3.1.9- array.prototype.findlastindex: 1.2.6- array.prototype.flat: 1.3.3- array.prototype.flatmap: 1.3.3- debug: 3.2.7([email protected])- doctrine: 2.1.0- eslint: 10.8.0([email protected])([email protected])- eslint-import-resolver-node: 0.3.10([email protected])- eslint-module-utils: 2.12.1(@typescript-eslint/[email protected]([email protected]([email protected])([email protected]))([email protected])([email protected]))([email protected]([email protected]))([email protected])([email protected]([email protected])([email protected]))([email protected])- hasown: 2.0.4- is-core-module: 2.16.2- is-glob: 4.0.3- minimatch: 3.1.5- object.fromentries: 2.0.8- object.groupby: 1.0.3- object.values: 1.2.1- semver: 6.3.1- string.prototype.trimend: 1.0.9- tsconfig-paths: 3.15.0- optionalDependencies:- '@typescript-eslint/parser': 8.67.0([email protected]([email protected])([email protected]))([email protected])([email protected])- transitivePeerDependencies:- - eslint-import-resolver-typescript- - eslint-import-resolver-webpack- - supports-color-dependencies:aria-query: 5.3.2diff --git a/web/.structure-baseline.json b/web/.structure-baseline.jsonindex adb40eeb2d..ae4a050343 100644--- a/web/.structure-baseline.json+++ b/web/.structure-baseline.json@@ -969,17 +969,6 @@"src/features/evals/server/evalTemplateDeletion.ts -> src/features/audit-logs/auditLog.ts","src/features/evals/server/router.ts -> src/features/audit-logs/auditLog.ts","src/features/evals/server/router.ts -> src/features/rbac/utils/checkProjectAccess.ts",- "src/features/evals/server/unstable-public-api/adapters.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",- "src/features/evals/server/unstable-public-api/evaluation-rule-service.ts -> src/features/audit-logs/auditLog.ts",- "src/features/evals/server/unstable-public-api/evaluation-rule-service.ts -> src/features/public-api/server/unstable-public-api-error-contract.ts",- "src/features/evals/server/unstable-public-api/evaluation-rule-service.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/audit-logs/auditLog.ts",- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/public-api/server/unstable-public-api-error-contract.ts",- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/public-api/types/unstable-evaluators.ts",- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",- "src/features/evals/server/unstable-public-api/types.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",- "src/features/evals/server/unstable-public-api/validation.ts -> src/features/public-api/server/unstable-public-api-error-contract.ts",- "src/features/evals/server/unstable-public-api/validation.ts -> src/features/public-api/types/unstable-public-evals-contract.ts","src/features/events/components/CategoryPresetChips.tsx -> src/features/posthog-analytics/usePostHogClientCapture.ts","src/features/events/components/EventsTable.tsx -> src/features/batch-actions/components/AddObservationsToDatasetDialog/index.tsx","src/features/events/components/EventsTable.tsx -> src/features/batch-actions/components/RunEvaluationDialog/index.tsx",@@ -1198,28 +1187,6 @@"src/features/mcp/features/datasets/tools/upsertDataset.ts -> src/features/public-api/types/datasets.ts","src/features/mcp/features/datasets/tools/upsertDatasetItem.ts -> src/features/datasets/server/publicDatasetService.ts","src/features/mcp/features/datasets/tools/upsertDatasetItem.ts -> src/features/public-api/types/datasets.ts",- "src/features/mcp/features/evals/schema.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",- "src/features/mcp/features/evals/tools/deleteEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/deleteEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",- "src/features/mcp/features/evals/tools/deleteEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/deleteEvaluator.ts -> src/features/public-api/types/unstable-evaluators.ts",- "src/features/mcp/features/evals/tools/getEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/getEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",- "src/features/mcp/features/evals/tools/getEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/getEvaluator.ts -> src/features/public-api/types/unstable-evaluators.ts",- "src/features/mcp/features/evals/tools/listEvaluationRules.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/listEvaluationRules.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",- "src/features/mcp/features/evals/tools/listEvaluators.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/listEvaluators.ts -> src/features/public-api/types/unstable-evaluators.ts",- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/public-api/types/unstable-evaluators.ts",- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/public-api/types/unstable-public-evals-contract.ts","src/features/mcp/features/experiments/schema.ts -> src/features/public-api/types/experiments.ts","src/features/mcp/features/experiments/tools/listExperimentItems.ts -> src/features/experiments/server/public/service.ts","src/features/mcp/features/experiments/tools/listExperimentItems.ts -> src/features/public-api/types/experiments.ts",@@ -1830,15 +1797,6 @@"src/features/mcp/features/datasets/tools/listDatasets.ts -> src/features/datasets/server/publicDatasetService.ts","src/features/mcp/features/datasets/tools/upsertDataset.ts -> src/features/datasets/server/publicDatasetService.ts","src/features/mcp/features/datasets/tools/upsertDatasetItem.ts -> src/features/datasets/server/publicDatasetService.ts",- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/deleteEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/deleteEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/getEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/getEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/listEvaluationRules.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/listEvaluators.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts","src/features/mcp/features/experiments/tools/listExperimentItems.ts -> src/features/experiments/server/public/service.ts","src/features/mcp/features/experiments/tools/listExperiments.ts -> src/features/experiments/server/public/service.ts","src/features/mcp/features/feedback/tools/submitFeedback.ts -> src/features/feedback/server/FeedbackService.ts",diff --git a/web/AGENTS.md b/web/AGENTS.mdindex 7e914bbc19..2b3a15e144 100644--- a/web/AGENTS.md+++ b/web/AGENTS.md@@ -19,7 +19,7 @@- tRPC router registry: `src/server/api/root.ts`- tRPC routers: `src/server/api/routers/*`, `src/features/*/server/*`- Public REST API routes: `src/pages/api/public/*`-- Unstable public eval APIs: `src/pages/api/public/unstable/{evaluators,evaluation-rules}/*`+- Public eval APIs: `src/pages/api/public/v2/{evaluators,evaluation-rules}/*`- Feature modules: `src/features/*`- Reusable UI components: `src/components/*`- Tests:diff --git a/web/package.json b/web/package.jsonindex 043ecfc929..41f055d906 100644--- a/web/package.json+++ b/web/package.json@@ -128,7 +128,6 @@"https-proxy-agent": "^7.0.6","ioredis": "^5.10.1","json-schema-faker": "0.6.1",- "jsonpath-plus": "10.3.0","langfuse": "3.38.4","lodash": "^4.18.1","lucide-react": "^0.552.0",diff --git a/web/public/generated/api/openapi.yml b/web/public/generated/api/openapi.ymlindex ad7c162133..819726ea60 100644--- a/web/public/generated/api/openapi.yml+++ b/web/public/generated/api/openapi.yml@@ -8571,2893 +8571,36 @@ paths:schema:$ref: '#/components/schemas/unstablePublicApiError'security: *ref_0- /api/public/unstable/evaluation-rules:- post:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments- are unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Create an evaluation rule.--- An evaluation rule defines **what** incoming data should be evaluated- and **how prompt variables should be populated** from that data.--- Use this resource after choosing an evaluator from the evaluator- endpoints.--- Key rules:-- - `name` must be unique within the project for public evaluation rules-- - `target` must be `observation` or `experiment`-- - provide either the compatibility `evaluator` field or the new- `evaluators` array, never both-- - each evaluator `name` + `type` must identify an existing evaluator- family returned by the evaluator endpoints-- - Langfuse resolves that family to its latest version before saving the- evaluation rule-- - for `target=experiment`, use dataset `id` values from `GET- /api/public/v2/datasets` when filtering by `datasetId`-- - an omitted LLM-as-judge assignment mapping inherits the evaluator- version's default mapping-- - the effective mapping must map every evaluator prompt variable exactly- once-- - for `code` evaluators, Langfuse uses the fixed code runtime mapping;- omit `mapping` in create and update requests-- - for user-provided `llm_as_judge` mappings, `expected_output` and- `experiment_item_metadata` are only valid for `target=experiment`-- - if `enabled=true`, Langfuse validates that the referenced evaluator- can currently run-- - at most 500 evaluation rules can be effectively active in one project- at the same time (enforced identically by the API, the MCP tools, and- the app)--- If an evaluation rule with the same `name` already exists in the- project, the API returns `409`.-- In that case, update the existing resource with `PATCH- /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of- creating a second one.--- If enabling this resource would exceed the 500-active limit, the API- also returns `409`.-- In that case, disable or pause another active evaluation rule before- enabling a new one.--- Current scope:-- - evaluation rules are live-ingestion rules only-- - they do not trigger historical backfills--- Recovery guidance:-- - `400 invalid_filter_value`: fix the filter `column` or `value` using- `details.column`, `details.invalidValues`, and `details.allowedValues`-- - `400 invalid_filter_value` with `details.column=datasetId`: call `GET- /api/public/v2/datasets`, then retry with dataset `id` values from that- response-- - `400 missing_variable_mapping`: for `llm_as_judge` evaluators, fetch- the evaluator again and make sure every variable in `variables` appears- exactly once in `mapping`-- - `400 duplicate_variable_mapping`: remove repeated mappings for the- same variable-- - `400 invalid_variable_mapping`: for `llm_as_judge`, switch to a valid- `source` for the selected `target`, or fix the variable name-- - `400 invalid_json_path`: remove or correct the `jsonPath`-- - `422 evaluator_preflight_failed`: the selected evaluator cannot run- with the resolved model configuration. Fix the evaluator/default model- setup, then retry the create request.- operationId: unstable_evaluationRules_create- tags:- - UnstableEvaluationRules- parameters: []- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableEvaluationRule'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '409':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '422':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- requestBody:- required: true- content:- application/json:- schema:- $ref: '#/components/schemas/unstableCreateEvaluationRuleRequest'- deprecated: true- get:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments- are unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- List evaluation rules in the authenticated project.--- This includes legacy `trace` and `dataset` rules so they can be- inspected and migrated to v4 rules. Legacy rules are read-only through- this API; create, update, and delete continue to support only- `observation` and `experiment` rules.- operationId: unstable_evaluationRules_list- tags:- - UnstableEvaluationRules- parameters:- - name: page- in: query- description: 1-based page number. Defaults to `1`.- required: false- schema:- type: integer- nullable: true- - name: limit- in: query- description: Maximum number of items per page. Defaults to `50`.- required: false- schema:- type: integer- nullable: true- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableEvaluationRules'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- deprecated: true- /api/public/unstable/evaluation-rules/{evaluationRuleId}:- get:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments- are unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Get one evaluation rule by its identifier.--- Use this endpoint to inspect the current evaluator, target, mapping,- filters, execution timing, and effective runtime status. Legacy `trace`- and `dataset` rules are returned for migration and are read-only through- this API.- operationId: unstable_evaluationRules_get- tags:- - UnstableEvaluationRules- parameters:- - name: evaluationRuleId- in: path- description: >-- Evaluation rule identifier returned by the evaluation rule endpoints.- required: true- schema:- type: string- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableReadableEvaluationRule'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- deprecated: true- patch:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments- are unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Update an evaluation rule.--- Typical uses:-- - enable or disable live execution-- - switch to another evaluator-- - adjust sampling-- - change filters-- - update LLM-as-judge variable mappings--- Important behavior:-- - provide only the fields you want to change-- - if you provide `evaluator`, Langfuse resolves that evaluator family to- its latest version before saving-- - changing `target`, `filter`, or an LLM-as-judge `mapping` must still- produce a valid target-specific configuration-- - if you change `target` for an LLM-as-judge rule, also send a- compatible `filter` and `mapping` in the same request unless the- existing ones are still valid for the new target-- - for `code` evaluator rules, omit `mapping`; Langfuse stores the fixed- code runtime mapping automatically-- - if the resulting config is enabled, Langfuse re-validates that the- selected evaluator can run-- - if the update would move a non-active evaluation rule into the active- state and the project already has 500 active evaluation rules, the API- returns `409`--- Recovery guidance:-- - if an LLM-as-judge update fails with `missing_variable_mapping` or- `invalid_variable_mapping` after changing `evaluator` or `target`,- resend the request with a complete new `mapping`-- - if the update fails with `invalid_filter_value` after changing- `target`, resend the request with a target-compatible `filter`- operationId: unstable_evaluationRules_update- tags:- - UnstableEvaluationRules- parameters:- - name: evaluationRuleId- in: path- description: Evaluation rule identifier.- required: true- schema:- type: string- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableReadableV2EvaluationRule'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '422':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- requestBody:- required: true- content:- application/json:- schema:- $ref: '#/components/schemas/unstableUpdateEvaluationRuleRequest'- deprecated: true- delete:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments- are unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Delete an evaluation rule.--- This removes the live-ingestion rule only. It does not delete the- referenced evaluator.- operationId: unstable_evaluationRules_delete- tags:- - UnstableEvaluationRules- parameters:- - name: evaluationRuleId- in: path- description: Evaluation rule identifier.- required: true- schema:- type: string- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableDeleteEvaluationRuleResponse'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- deprecated: true- /api/public/unstable/evaluators:- post:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluators` API instead. Self-hosted deployments are- unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Create an evaluator in the authenticated project.--- Use evaluators to define **how** Langfuse should score data.-- LLM-as-a-judge evaluators define a prompt, expected structured output,- and optional model configuration.-- Code evaluators define source code and a runtime language.--- Naming behavior:-- - If this is a new evaluator name in your project, Langfuse creates- version `1`.-- - If the name already exists in your project, Langfuse creates the next- version and returns it.-- - The evaluator `id` remains stable across versions.-- - Existing evaluation rules automatically use the latest evaluator- version; no rule update is required.--- Recommended workflow:-- 1. Create the evaluator.-- 2. Read the returned `variables` array.-- 3. Read the returned `outputDefinition.dataType` so the client knows- whether future scores will be numeric, boolean, or categorical.-- 4. Create one or more evaluation rules that reference the returned- evaluator family using `name` and `type`.--- Code evaluator validation:-- - At creation, Langfuse only validates the request shape-- - The `sourceCode` itself is not executed here. It is first run- (preflight-tested against a sample observation) when you link the- evaluator to an evaluation rule, so runtime errors in the code surface- at evaluation-rule creation, not at evaluator creation.--- Recovery guidance:-- - `422` with `code=evaluator_preflight_failed`: the evaluator cannot run- with the resolved model configuration. Add a valid explicit- `modelConfig`, or configure the project's default evaluation model, then- retry the same request.-- - `400` with `code=invalid_body`: the request shape is malformed. Use- the structured `details.issues` array to fix the specific fields and- retry.-- - `400` with `code=invalid_body` on `outputDefinition`: for- `type=llm_as_judge`, send `dataType`, `reasoning.description`, and- `score.description`. Do not send `version`; it is not part of the public- request shape.-- - If `type` is omitted, Langfuse treats the request as- `type=llm_as_judge` for backwards compatibility. New clients should send- `type` explicitly.--- Unstable API note:-- - This surface may evolve while the underlying evaluation data model is- being redesigned.- operationId: unstable_evaluators_create- tags:- - UnstableEvaluators- parameters: []- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableEvaluator'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '409':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '422':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- requestBody:- required: true- content:- application/json:- schema:- $ref: '#/components/schemas/unstableCreateEvaluatorRequest'- deprecated: true- get:- description: |-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).-- List the evaluators available to the authenticated project.-- Important behavior:- - This endpoint returns the latest version of each available evaluator.- - Every evaluator is owned by the authenticated project.- operationId: unstable_evaluators_list- tags:- - UnstableEvaluators- parameters:- - name: page- in: query- description: 1-based page number. Defaults to `1`.- required: false- schema:- type: integer- nullable: true- - name: limit- in: query- description: Maximum number of items per page. Defaults to `50`.- required: false- schema:- type: integer- nullable: true- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableEvaluators'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- deprecated: true- /api/public/unstable/evaluators/{evaluatorId}:- get:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluators` API instead. Self-hosted deployments are- unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Get one evaluator by `id`.--- This endpoint always returns the evaluator's latest version. Use it when- you want the current prompt, output definition, model configuration, and- derived variables for the evaluator you plan to use in an evaluation- rule.- operationId: unstable_evaluators_get- tags:- - UnstableEvaluators- parameters:- - name: evaluatorId- in: path- description: Evaluator identifier returned by the evaluator endpoints.- required: true- schema:- type: string- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableEvaluator'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- deprecated: true- delete:- description: >-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on September 4, 2026. Use the stable- `/api/public/v2/evaluators` API instead. Self-hosted deployments are- unaffected by this date; the endpoint becomes unavailable when they- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).--- Delete an evaluator.--- Important behavior:-- - This deletes the evaluator including all of its stored versions.-- - Evaluation rule assignments referencing the evaluator are also- deleted.-- - Scores already produced by the evaluator are not deleted.- operationId: unstable_evaluators_delete- tags:- - UnstableEvaluators- parameters:- - name: evaluatorId- in: path- description: Evaluator identifier returned by the evaluator endpoints.- required: true- schema:- type: string- responses:- '200':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstableDeleteEvaluatorResponse'- '400':- description: ''- content:- application/json:- schema: {}- '401':- description: ''- content:- application/json:- schema: {}- '403':- description: ''- content:- application/json:- schema: {}- '404':- description: ''- content:- application/json:- schema: {}- '405':- description: ''- content:- application/json:- schema: {}- '429':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- '500':- description: ''- content:- application/json:- schema:- $ref: '#/components/schemas/unstablePublicApiError'- security: *ref_0- deprecated: true-components:- schemas:- AnnotationQueueStatus:- title: AnnotationQueueStatus- type: string- enum:- - PENDING- - COMPLETED- AnnotationQueueObjectType:- title: AnnotationQueueObjectType- type: string- enum:- - TRACE- - OBSERVATION- - SESSION- AnnotationQueue:- title: AnnotationQueue- type: object- properties:- id:- type: string- name:- type: string- description:- type: string- nullable: true- scoreConfigIds:- type: array- items:- type: string- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- required:- - id- - name- - description- - scoreConfigIds- - createdAt- - updatedAt- AnnotationQueueItem:- title: AnnotationQueueItem- type: object- properties:- id:- type: string- queueId:- type: string- objectId:- type: string- objectType:- $ref: '#/components/schemas/AnnotationQueueObjectType'- status:- $ref: '#/components/schemas/AnnotationQueueStatus'- completedAt:- type: string- format: date-time- nullable: true- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- required:- - id- - queueId- - objectId- - objectType- - status- - createdAt- - updatedAt- PaginatedAnnotationQueues:- title: PaginatedAnnotationQueues- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/AnnotationQueue'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- required:- - data- - meta- PaginatedAnnotationQueueItems:- title: PaginatedAnnotationQueueItems- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/AnnotationQueueItem'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- required:- - data- - meta- CreateAnnotationQueueRequest:- title: CreateAnnotationQueueRequest- type: object- properties:- name:- type: string- description:- type: string- nullable: true- scoreConfigIds:- type: array- items:- type: string- required:- - name- - scoreConfigIds- CreateAnnotationQueueItemRequest:- title: CreateAnnotationQueueItemRequest- type: object- properties:- objectId:- type: string- objectType:- $ref: '#/components/schemas/AnnotationQueueObjectType'- status:- $ref: '#/components/schemas/AnnotationQueueStatus'- nullable: true- description: Defaults to PENDING for new queue items- required:- - objectId- - objectType- UpdateAnnotationQueueItemRequest:- title: UpdateAnnotationQueueItemRequest- type: object- properties:- status:- $ref: '#/components/schemas/AnnotationQueueStatus'- nullable: true- DeleteAnnotationQueueItemResponse:- title: DeleteAnnotationQueueItemResponse- type: object- properties:- success:- type: boolean- message:- type: string- required:- - success- - message- AnnotationQueueAssignmentRequest:- title: AnnotationQueueAssignmentRequest- type: object- properties:- userId:- type: string- required:- - userId- DeleteAnnotationQueueAssignmentResponse:- title: DeleteAnnotationQueueAssignmentResponse- type: object- properties:- success:- type: boolean- required:- - success- CreateAnnotationQueueAssignmentResponse:- title: CreateAnnotationQueueAssignmentResponse- type: object- properties:- userId:- type: string- queueId:- type: string- projectId:- type: string- required:- - userId- - queueId- - projectId- BlobStorageIntegrationType:- title: BlobStorageIntegrationType- type: string- enum:- - S3- - S3_COMPATIBLE- - AZURE_BLOB_STORAGE- BlobStorageIntegrationFileType:- title: BlobStorageIntegrationFileType- type: string- enum:- - JSON- - CSV- - JSONL- - PARQUET- description: >-- File format for exported data. `PARQUET` is a columnar binary format- encoded and compressed by the storage engine; gzip compression does not- apply to it. Note that the model-price columns (`input_price`,- `output_price`, `total_price`) are not included in Parquet observation- exports.- BlobStorageIntegrationFileTypeResponse:- title: BlobStorageIntegrationFileTypeResponse- type: string- enum:- - JSON- - CSV- - JSONL- - PARQUET- description: File type reported for an existing integration.- BlobStorageExportMode:- title: BlobStorageExportMode- type: string- enum:- - FULL_HISTORY- - FROM_TODAY- - FROM_CUSTOM_DATE- BlobStorageExportFrequency:- title: BlobStorageExportFrequency- type: string- enum:- - every_20_minutes- - hourly- - daily- - weekly- BlobStorageExportSource:- title: BlobStorageExportSource- type: string- enum:- - LEGACY_TRACES_OBSERVATIONS- - OBSERVATIONS_V2- - LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS- description: >-- What data the integration exports.-- - `LEGACY_TRACES_OBSERVATIONS`: traces, observations, and scores tables.- Observation columns are controlled by `exportFieldGroups`; field groups- without a counterpart in this data model (e.g. `trace_context`) are- omitted.-- - `OBSERVATIONS_V2`: same data model as the- `/api/public/v2/observations` endpoint, plus scores. Columns are- controlled by `exportFieldGroups`.-- - `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`: both sets. Observation- columns of both portions are controlled by `exportFieldGroups`.--- **Note:** which sources a deployment accepts depends on how far it has- moved to the v4 data model. `OBSERVATIONS_V2` and the- enriched-observations portion of- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the enriched observations- table, so they require a deployment that already populates it.- `LEGACY_TRACES_OBSERVATIONS` and the legacy portion of- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the legacy traces and- observations tables, so they require a deployment that still populates- those. A deployment part-way through the migration populates both and- accepts every source. Selecting a source the deployment cannot serve is- rejected with `400`, rather than exporting an empty result. See- https://langfuse.com/docs/v4.- BlobStorageExportFieldGroup:- title: BlobStorageExportFieldGroup- type: string- enum:- - core- - basic- - time- - io- - metadata- - model- - usage- - prompt- - metrics- - tools- - trace_context- description: >-- Field group selecting which observation columns are included in the- export. Applies to all export sources; groups without a counterpart in- the legacy data model (e.g. `trace_context`) are omitted from the legacy- observations export.- CreateBlobStorageIntegrationRequest:- title: CreateBlobStorageIntegrationRequest- type: object- properties:- projectId:- type: string- description: ID of the project in which to configure the blob storage integration- type:- $ref: '#/components/schemas/BlobStorageIntegrationType'- bucketName:- type: string- description: >-- Name of the storage bucket. For AZURE_BLOB_STORAGE, must be a valid- Azure container name (3-63 chars, lowercase letters, numbers, and- hyphens only, must start and end with a letter or number, no- consecutive hyphens).- endpoint:- type: string- nullable: true- description: Custom endpoint URL (required for S3_COMPATIBLE type)- region:- type: string- description: >-- Storage region used by S3-compatible clients (AWS, GCS, Cloudflare- R2, MinIO, Azure location IDs such as eastus, OCI). Leading and- trailing whitespace is removed. The remaining value must be 1-63- letters, numbers, or hyphens, and cannot start or end with a hyphen.- Examples: us-east-1, europe-west1, eastus, auto.- accessKeyId:- type: string- nullable: true- description: Access key ID for authentication- secretAccessKey:- type: string- nullable: true- description: Secret access key for authentication (will be encrypted when stored)- prefix:- type: string- nullable: true- description: >-- Path prefix for exported files (must end with forward slash if- provided)- exportFrequency:- $ref: '#/components/schemas/BlobStorageExportFrequency'- enabled:- type: boolean- description: Whether the integration is active- forcePathStyle:- type: boolean- description: Use path-style URLs for S3 requests- fileType:- $ref: '#/components/schemas/BlobStorageIntegrationFileType'- exportMode:- $ref: '#/components/schemas/BlobStorageExportMode'- exportStartDate:- type: string- format: date-time- nullable: true- description: >-- Custom start date for exports (required when exportMode is- FROM_CUSTOM_DATE). Must not be in the future (27 h tolerance for- timezone differences).- compressed:- type: boolean- nullable: true- description: >-- Enable gzip compression for exported files (.csv.gz, .json.gz,- .jsonl.gz). Defaults to true.- exportSource:- $ref: '#/components/schemas/BlobStorageExportSource'- nullable: true- description: >-- Data to export. When omitted on update, the existing value is- preserved. When omitted on create, the default is `OBSERVATIONS_V2`- on Langfuse Cloud, and on self-hosted deployments- `LEGACY_TRACES_OBSERVATIONS` — or `OBSERVATIONS_V2` where the- deployment no longer populates the legacy tables. The default is- never a source the deployment cannot serve. Required when- `exportFieldGroups` is provided.--- **Cloud-only project deprecation gate (effective 2026-05-20):** For- projects created on or after 2026-05-20 on Langfuse Cloud,- `LEGACY_TRACES_OBSERVATIONS` and- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP- 400. Use `OBSERVATIONS_V2` for all new integrations. Self-hosted- deployments are unaffected.--- **Cloud-only integration deprecation gate (effective 2026-06-22):**- On Langfuse Cloud, legacy export sources are only accepted for blob- storage integrations created before 2026-06-22, regardless of- project age. Requests that would create a new integration with- `LEGACY_TRACES_OBSERVATIONS` or- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP- 400. Use `OBSERVATIONS_V2` instead. Self-hosted deployments are- unaffected.- exportFieldGroups:- type: array- items:- $ref: '#/components/schemas/BlobStorageExportFieldGroup'- nullable: true- description: >-- Field groups to include in each exported observation row. Applies to- all export sources; must include `core` if provided. When omitted on- create, the column default (all groups) applies. When omitted on- update, the existing value is preserved.--- `exportFieldGroups` requires `exportSource` to be provided in the- same request.- required:- - projectId- - type- - bucketName- - region- - exportFrequency- - enabled- - forcePathStyle- - fileType- - exportMode- BlobStorageIntegrationResponse:- title: BlobStorageIntegrationResponse- type: object- properties:- id:- type: string- projectId:- type: string- type:- $ref: '#/components/schemas/BlobStorageIntegrationType'- bucketName:- type: string- endpoint:- type: string- nullable: true- region:- type: string- accessKeyId:- type: string- nullable: true- prefix:- type: string- exportFrequency:- $ref: '#/components/schemas/BlobStorageExportFrequency'- enabled:- type: boolean- forcePathStyle:- type: boolean- fileType:- $ref: '#/components/schemas/BlobStorageIntegrationFileTypeResponse'- exportMode:- $ref: '#/components/schemas/BlobStorageExportMode'- exportStartDate:- type: string- format: date-time- nullable: true- compressed:- type: boolean- exportSource:- $ref: '#/components/schemas/BlobStorageExportSource'- exportFieldGroups:- type: array- items:- $ref: '#/components/schemas/BlobStorageExportFieldGroup'- nullable: true- description: >-- Field groups included in each exported observation row. An empty- list is treated as all groups during export.- nextSyncAt:- type: string- format: date-time- nullable: true- lastSyncAt:- type: string- format: date-time- nullable: true- lastError:- type: string- nullable: true- lastErrorAt:- type: string- format: date-time- nullable: true- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- required:- - id- - projectId- - type- - bucketName- - endpoint- - region- - accessKeyId- - prefix- - exportFrequency- - enabled- - forcePathStyle- - fileType- - exportMode- - exportStartDate- - compressed- - exportSource- - exportFieldGroups- - nextSyncAt- - lastSyncAt- - lastError- - lastErrorAt- - createdAt- - updatedAt- BlobStorageIntegrationsResponse:- title: BlobStorageIntegrationsResponse- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/BlobStorageIntegrationResponse'- required:- - data- BlobStorageSyncStatus:- title: BlobStorageSyncStatus- type: string- enum:- - idle- - running- - queued- - up_to_date- - disabled- - error- description: >-- Sync status of the blob storage integration:-- - `disabled` — integration is not enabled-- - `error` — last export failed (see `lastError` for details)-- - `running` — an export job is currently being processed-- - `queued` — next export is overdue (`nextSyncAt` is in the past) and- waiting to be picked up by the worker-- - `idle` — enabled but has never exported yet and no export is queued-- - `up_to_date` — all available data has been exported; next export is- scheduled for the future--- **ETL usage**: poll this endpoint and check for `up_to_date` status.- Compare `lastSyncAt` against your-- ETL bookmark to determine if new data is available. Note that exports- run with a 20-minute lag buffer,-- so `lastSyncAt` will always be at least 20 minutes behind real-time.- BlobStorageIntegrationStatusResponse:- title: BlobStorageIntegrationStatusResponse- type: object- properties:- id:- type: string- projectId:- type: string- syncStatus:- $ref: '#/components/schemas/BlobStorageSyncStatus'- enabled:- type: boolean- lastSyncAt:- type: string- format: date-time- nullable: true- description: >-- End of the last successfully exported time window. Compare against- your ETL bookmark to determine if new data is available. Null if the- integration has never synced.- nextSyncAt:- type: string- format: date-time- nullable: true- description: When the next export is scheduled. Null if no sync has occurred yet.- lastError:- type: string- nullable: true- description: >-- Raw error message from the storage provider (S3/Azure/GCS) if the- last export failed. Cleared on successful export.- lastErrorAt:- type: string- format: date-time- nullable: true- description: When the last error occurred. Cleared on successful export.- required:- - id- - projectId- - syncStatus- - enabled- - lastSyncAt- - nextSyncAt- - lastError- - lastErrorAt- BlobStorageIntegrationDeletionResponse:- title: BlobStorageIntegrationDeletionResponse- type: object- properties:- message:- type: string- required:- - message- CreateCommentRequest:- title: CreateCommentRequest- type: object- properties:- projectId:- type: string- description: The id of the project to attach the comment to.- objectType:- type: string- description: >-- The type of the object to attach the comment to (trace, observation,- session, prompt).- objectId:- type: string- description: >-- The id of the object to attach the comment to. If this does not- reference a valid existing object, an error will be thrown.- content:- type: string- description: >-- The content of the comment. May include markdown. Currently limited- to 5000 characters.- authorUserId:- type: string- nullable: true- description: >-- The id of the user who created the comment. Must be a member of the- organization that owns the project, otherwise an error will be- thrown.- required:- - projectId- - objectType- - objectId- - content- CreateCommentResponse:- title: CreateCommentResponse- type: object- properties:- id:- type: string- description: The id of the created object in Langfuse- required:- - id- GetCommentsResponse:- title: GetCommentsResponse- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/Comment'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- required:- - data- - meta- Deprecation:- title: Deprecation- type: object- description: >-- Migration signal returned by deprecated endpoints. Optional fields are- omitted when they have no value.- properties:- message:- type: string- description: >-- Human- and agent-readable summary of the deprecation and its- replacement.- replacement:- type: string- nullable: true- description: >-- The replacement endpoint, e.g. "GET /api/public/v2/observations".- Omitted when the endpoint is being removed without a direct- replacement.- docsUrl:- type: string- nullable: true- description: Link to the migration documentation (markdown), when available.- sunsetAt:- type: string- nullable: true- description: >-- ISO date after which the endpoint may stop working, when a removal- date is committed.- required:- - message- Trace:- title: Trace- type: object- properties:- id:- type: string- description: The unique identifier of a trace- timestamp:- type: string- format: date-time- description: The timestamp when the trace was created- name:- type: string- nullable: true- description: The name of the trace- input:- nullable: true- description: The input data of the trace. Can be any JSON.- output:- nullable: true- description: The output data of the trace. Can be any JSON.- sessionId:- type: string- nullable: true- description: The session identifier associated with the trace- release:- type: string- nullable: true- description: The release version of the application when the trace was created- version:- type: string- nullable: true- description: The version of the trace- userId:- type: string- nullable: true- description: The user identifier associated with the trace- metadata:- nullable: true- description: The metadata associated with the trace. Can be any JSON.- tags:- type: array- items:- type: string- description: The tags associated with the trace.- public:- type: boolean- description: Public traces are accessible via url without login- environment:- type: string- description: >-- The environment from which this trace originated. Can be any- lowercase alphanumeric string with hyphens and underscores that does- not start with 'langfuse'.- required:- - id- - timestamp- - name- - sessionId- - release- - version- - userId- - tags- - public- - environment- TraceWithDetails:- title: TraceWithDetails- type: object- properties:- htmlPath:- type: string- description: Path of trace in Langfuse UI- latency:- type: number- format: double- nullable: true- description: Latency of trace in seconds- totalCost:- type: number- format: double- nullable: true- description: Cost of trace in USD- observations:- type: array- items:- type: string- nullable: true- description: List of observation ids- scores:- type: array- items:- type: string- nullable: true- description: List of score ids- required:- - htmlPath- allOf:- - $ref: '#/components/schemas/Trace'- TraceWithFullDetails:- title: TraceWithFullDetails- type: object- properties:- htmlPath:- type: string- description: Path of trace in Langfuse UI- latency:- type: number- format: double- nullable: true- description: Latency of trace in seconds- totalCost:- type: number- format: double- nullable: true- description: Cost of trace in USD- observations:- type: array- items:- $ref: '#/components/schemas/ObservationsView'- description: List of observations- scores:- type: array- items:- $ref: '#/components/schemas/ScoreV1'- description: List of scores- _deprecation:- $ref: '#/components/schemas/Deprecation'- nullable: true- required:- - htmlPath- - observations- - scores- allOf:- - $ref: '#/components/schemas/Trace'- Session:- title: Session- type: object- properties:- id:- type: string- createdAt:- type: string- format: date-time- projectId:- type: string- environment:- type: string- description: The environment from which this session originated.- required:- - id- - createdAt- - projectId- - environment- SessionWithTraces:- title: SessionWithTraces- type: object- properties:- traces:- type: array- items:- $ref: '#/components/schemas/Trace'- _deprecation:- $ref: '#/components/schemas/Deprecation'- nullable: true- required:- - traces- allOf:- - $ref: '#/components/schemas/Session'- Observation:- title: Observation- type: object- properties:- id:- type: string- description: The unique identifier of the observation- traceId:- type: string- nullable: true- description: The trace ID associated with the observation- type:- type: string- description: The type of the observation- name:- type: string- nullable: true- description: The name of the observation- startTime:- type: string- format: date-time- description: The start time of the observation- endTime:- type: string- format: date-time- nullable: true- description: The end time of the observation.- completionStartTime:- type: string- format: date-time- nullable: true- description: The completion start time of the observation- model:- type: string- nullable: true- description: The model used for the observation- modelParameters:- description: The parameters of the model used for the observation- input:- description: The input data of the observation- version:- type: string- nullable: true- description: The version of the observation- metadata:- description: Additional metadata of the observation- output:- description: The output data of the observation- usage:- $ref: '#/components/schemas/Usage'- description: >-- (Deprecated. Use usageDetails and costDetails instead.) The usage- data of the observation- level:- $ref: '#/components/schemas/ObservationLevel'- description: The level of the observation- statusMessage:- type: string- nullable: true- description: The status message of the observation- parentObservationId:- type: string- nullable: true- description: The parent observation ID- promptId:- type: string- nullable: true- description: The prompt ID associated with the observation- usageDetails:- type: object- additionalProperties:- type: integer- description: >-- The usage details of the observation. Key is the name of the usage- metric, value is the number of units consumed. The total key is the- sum of all (non-total) usage metrics or the total value ingested.- costDetails:- type: object- additionalProperties:- type: number- format: double- description: >-- The cost details of the observation. Key is the name of the cost- metric, value is the cost in USD. The total key is the sum of all- (non-total) cost metrics or the total value ingested.- environment:- type: string- description: >-- The environment from which this observation originated. Can be any- lowercase alphanumeric string with hyphens and underscores that does- not start with 'langfuse'.- required:- - id- - traceId- - type- - name- - startTime- - endTime- - completionStartTime- - model- - modelParameters- - input- - version- - metadata- - output- - usage- - level- - statusMessage- - parentObservationId- - promptId- - usageDetails- - costDetails- - environment- ObservationsView:- title: ObservationsView- type: object- properties:- promptName:- type: string- nullable: true- description: The name of the prompt associated with the observation- promptVersion:- type: integer- nullable: true- description: The version of the prompt associated with the observation- modelId:- type: string- nullable: true- description: The unique identifier of the model- inputPrice:- type: number- format: double- nullable: true- description: The price of the input in USD- outputPrice:- type: number- format: double- nullable: true- description: The price of the output in USD.- totalPrice:- type: number- format: double- nullable: true- description: The total price in USD.- calculatedInputCost:- type: number- format: double- nullable: true- description: >-- (Deprecated. Use usageDetails and costDetails instead.) The- calculated cost of the input in USD- calculatedOutputCost:- type: number- format: double- nullable: true- description: >-- (Deprecated. Use usageDetails and costDetails instead.) The- calculated cost of the output in USD- calculatedTotalCost:- type: number- format: double- nullable: true- description: >-- (Deprecated. Use usageDetails and costDetails instead.) The- calculated total cost in USD- latency:- type: number- format: double- nullable: true- description: The latency in seconds.- timeToFirstToken:- type: number- format: double- nullable: true- description: The time to the first token in seconds- required:- - promptName- - promptVersion- - modelId- - inputPrice- - outputPrice- - totalPrice- - calculatedInputCost- - calculatedOutputCost- - calculatedTotalCost- - latency- - timeToFirstToken- allOf:- - $ref: '#/components/schemas/Observation'- ObservationsViewSingle:- title: ObservationsViewSingle- type: object- properties:- _deprecation:- $ref: '#/components/schemas/Deprecation'- nullable: true- allOf:- - $ref: '#/components/schemas/ObservationsView'- ObservationV2:- title: ObservationV2- type: object- description: >-- An observation from the v2 API with field-group-based selection.-- Core fields are always present. Other fields are included only when- their field group is requested.- properties:- id:- type: string- description: The unique identifier of the observation- traceId:- type: string- nullable: true- description: The trace ID associated with the observation- startTime:- type: string- format: date-time- description: The start time of the observation- endTime:- type: string- format: date-time- nullable: true- description: The end time of the observation- projectId:- type: string- description: The project ID this observation belongs to- parentObservationId:- type: string- nullable: true- description: >-- The physical parent observation ID, if present.-- Observations marked as app roots by the SDK may retain a non-null- parent ID.- type:- type: string- description: The type of the observation (e.g. GENERATION, SPAN, EVENT)- isRootObservation:- type: boolean- nullable: true- description: >-- Whether this observation is a logical root.-- This is true for observations without a physical parent and- observations marked as app roots by the SDK.- name:- type: string- nullable: true- description: The name of the observation- level:- $ref: '#/components/schemas/ObservationLevel'- nullable: true- description: The level of the observation- statusMessage:- type: string- nullable: true- description: The status message of the observation- version:- type: string- nullable: true- description: The version of the observation- environment:- type: string- nullable: true- description: The environment from which this observation originated- bookmarked:- type: boolean- nullable: true- description: Whether the observation is bookmarked- public:- type: boolean- nullable: true- description: Whether the observation is public- userId:- type: string- nullable: true- description: The user ID associated with the observation- sessionId:- type: string- nullable: true- description: The session ID associated with the observation- completionStartTime:- type: string- format: date-time- nullable: true- description: The completion start time of the observation- createdAt:- type: string- format: date-time- nullable: true- description: The creation timestamp of the observation- updatedAt:- type: string- format: date-time- nullable: true- description: The last update timestamp of the observation- input:- nullable: true- description: The input data of the observation- output:- nullable: true- description: The output data of the observation- metadata:- nullable: true- description: Additional metadata of the observation- model:- type: string- nullable: true- description: The model name as provided by the user- internalModelId:- type: string- nullable: true- description: The internal model ID matched by Langfuse- modelParameters:- nullable: true- description: The parameters of the model used for the observation- usageDetails:- type: object- additionalProperties:- type: integer- nullable: true- description: >-- The usage details of the observation. Key is the usage metric name,- value is the number of units consumed.- costDetails:- type: object- additionalProperties:- type: number- format: double- nullable: true- description: >-- The cost details of the observation. Key is the cost metric name,- value is the cost in USD.- totalCost:- type: number- format: double- nullable: true- description: The total cost of the observation in USD- usagePricingTierName:- type: string- nullable: true- description: >-- The name of the pricing tier applied to this observation's usage- costs- promptId:- type: string- nullable: true- description: The prompt ID associated with the observation- promptName:- type: string- nullable: true- description: The prompt name associated with the observation- promptVersion:- type: integer- nullable: true- description: The prompt version associated with the observation- latency:- type: number- format: double- nullable: true- description: The latency in seconds- timeToFirstToken:- type: number- format: double- nullable: true- description: The time to first token in seconds- modelId:- type: string- nullable: true- description: >-- The matched model ID. Null when the `model` field group is not- requested.- inputPrice:- type: string- nullable: true- description: >-- The input token price (USD per unit) from the matched model,- serialized as a decimal string (e.g. "0.0001"). Null when the- `model` field group is not requested.- outputPrice:- type: string- nullable: true- description: >-- The output token price (USD per unit) from the matched model,- serialized as a decimal string (e.g. "0.0001"). Null when the- `model` field group is not requested.- totalPrice:- type: string- nullable: true- description: >-- The total token price (USD per unit) from the matched model,- serialized as a decimal string (e.g. "0.0001"). Null when the- `model` field group is not requested.- traceName:- type: string- nullable: true- description: The name of the parent trace- tags:- type: array- items:- type: string- nullable: true- description: Tags from the parent trace (denormalized onto the observation)- release:- type: string- nullable: true- description: The release version of the parent trace- required:- - id- - traceId- - startTime- - endTime- - projectId- - parentObservationId- - type- - modelId- - inputPrice- - outputPrice- - totalPrice- Usage:- title: Usage- type: object- description: >-- (Deprecated. Use usageDetails and costDetails instead.) Standard- interface for usage and cost- properties:- input:- type: integer- description: Number of input units (e.g. tokens)- output:- type: integer- description: Number of output units (e.g. tokens)- total:- type: integer- description: Defaults to input+output if not set- unit:- type: string- nullable: true- description: Unit of measurement- inputCost:- type: number- format: double- nullable: true- description: USD input cost- outputCost:- type: number- format: double- nullable: true- description: USD output cost- totalCost:- type: number- format: double- nullable: true- description: USD total cost, defaults to input+output- required:- - input- - output- - total- - unit- ScoreConfig:- title: ScoreConfig- type: object- description: Configuration for a score- properties:- id:- type: string- name:- type: string- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- projectId:- type: string- dataType:- $ref: '#/components/schemas/ScoreConfigDataType'- isArchived:- type: boolean- description: Whether the score config is archived. Defaults to false- minValue:- type: number- format: double- nullable: true- description: >-- Sets minimum value for numerical scores. If not set, the minimum- value defaults to -∞- maxValue:- type: number- format: double- nullable: true- description: >-- Sets maximum value for numerical scores. If not set, the maximum- value defaults to +∞- categories:- type: array- items:- $ref: '#/components/schemas/ConfigCategory'- nullable: true- description: Configures custom categories for categorical scores- description:- type: string- nullable: true- description: Description of the score config- required:- - id- - name- - createdAt- - updatedAt- - projectId- - dataType- - isArchived- ConfigCategory:- title: ConfigCategory- type: object- properties:- value:- type: number- format: double- label:- type: string- required:- - value- - label- BaseScoreV1:- title: BaseScoreV1- type: object- properties:- id:- type: string- traceId:- type: string- name:- type: string- source:- $ref: '#/components/schemas/ScoreSource'- observationId:- type: string- nullable: true- description: The observation ID associated with the score- timestamp:- type: string- format: date-time- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- authorUserId:- type: string- nullable: true- description: The user ID of the author- comment:- type: string- nullable: true- description: Comment on the score- metadata:- description: Metadata associated with the score- configId:- type: string- nullable: true- description: >-- Reference a score config on a score. When set, config and score name- must be equal and value must comply to optionally defined numerical- range- queueId:- type: string- nullable: true- description: >-- The annotation queue referenced by the score. Indicates if score was- initially created while processing annotation queue.- environment:- type: string- description: >-- The environment from which this score originated. Can be any- lowercase alphanumeric string with hyphens and underscores that does- not start with 'langfuse'.- required:- - id- - traceId- - name- - source- - timestamp- - createdAt- - updatedAt- - authorUserId- - comment- - metadata- - configId- - queueId- - environment- NumericScoreV1:- title: NumericScoreV1- type: object- properties:- value:- type: number- format: double- description: The numeric value of the score- required:- - value- allOf:- - $ref: '#/components/schemas/BaseScoreV1'- BooleanScoreV1:- title: BooleanScoreV1- type: object- properties:- value:- type: number- format: double- description: >-- The numeric value of the score. Equals 1 for "True" and 0 for "False"- stringValue:- type: string- description: >-- The string representation of the score value. Is inferred from the- numeric value and equals "True" or "False"- required:- - value- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScoreV1'- CategoricalScoreV1:- title: CategoricalScoreV1- type: object- properties:- value:- type: number- format: double- description: >-- Represents the numeric category mapping of the stringValue. If no- config is linked, defaults to 0.- stringValue:- type: string- description: >-- The string representation of the score value. If no config is- linked, can be any string. Otherwise, must map to a config category- required:- - value- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScoreV1'- TextScoreV1:- title: TextScoreV1- type: object- properties:- stringValue:- type: string- description: The text content of the score (1-500 characters)- required:- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScoreV1'- ScoreV1:- title: ScoreV1- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: '#/components/schemas/NumericScoreV1'- required:- - dataType- title: NumericScoreV1- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: '#/components/schemas/CategoricalScoreV1'- required:- - dataType- title: CategoricalScoreV1- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: '#/components/schemas/BooleanScoreV1'- required:- - dataType- title: BooleanScoreV1- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - TEXT- - $ref: '#/components/schemas/TextScoreV1'- required:- - dataType- title: TextScoreV1- BaseScore:- title: BaseScore- type: object- properties:- id:- type: string- traceId:- type: string- nullable: true- description: The trace ID associated with the score- sessionId:- type: string- nullable: true- description: The session ID associated with the score- observationId:- type: string- nullable: true- description: The observation ID associated with the score- datasetRunId:- type: string- nullable: true- description: The dataset run ID associated with the score- name:- type: string- source:- $ref: '#/components/schemas/ScoreSource'- timestamp:- type: string- format: date-time- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- authorUserId:- type: string- nullable: true- description: The user ID of the author- comment:- type: string- nullable: true- description: Comment on the score- metadata:- description: Metadata associated with the score- configId:- type: string- nullable: true- description: >-- Reference a score config on a score. When set, config and score name- must be equal and value must comply to optionally defined numerical- range- queueId:- type: string- nullable: true- description: >-- The annotation queue referenced by the score. Indicates if score was- initially created while processing annotation queue.- environment:- type: string- description: >-- The environment from which this score originated. Can be any- lowercase alphanumeric string with hyphens and underscores that does- not start with 'langfuse'.- required:- - id- - name- - source- - timestamp- - createdAt- - updatedAt- - authorUserId- - comment- - metadata- - configId- - queueId- - environment- NumericScore:- title: NumericScore- type: object- properties:- value:- type: number- format: double- description: The numeric value of the score- required:- - value- allOf:- - $ref: '#/components/schemas/BaseScore'- BooleanScore:- title: BooleanScore- type: object- properties:- value:- type: number- format: double- description: >-- The numeric value of the score. Equals 1 for "True" and 0 for "False"- stringValue:- type: string- description: >-- The string representation of the score value. Is inferred from the- numeric value and equals "True" or "False"- required:- - value- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScore'- CategoricalScore:- title: CategoricalScore- type: object- properties:- value:- type: number- format: double- description: >-- Represents the numeric category mapping of the stringValue. If no- config is linked, defaults to 0.- stringValue:- type: string- description: >-- The string representation of the score value. If no config is- linked, can be any string. Otherwise, must map to a config category- required:- - value- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScore'- CorrectionScore:- title: CorrectionScore- type: object- properties:- value:- type: number- format: double- description: The numeric value of the score. Always 0 for correction scores.- stringValue:- type: string- description: The string representation of the correction content- required:- - value- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScore'- TextScore:- title: TextScore- type: object- properties:- stringValue:- type: string- description: The text content of the score (1-500 characters)- required:- - stringValue- allOf:- - $ref: '#/components/schemas/BaseScore'- Score:- title: Score- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: '#/components/schemas/NumericScore'- required:- - dataType- title: NumericScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: '#/components/schemas/CategoricalScore'- required:- - dataType- title: CategoricalScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: '#/components/schemas/BooleanScore'- required:- - dataType- title: BooleanScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CORRECTION- - $ref: '#/components/schemas/CorrectionScore'- required:- - dataType- title: CorrectionScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - TEXT- - $ref: '#/components/schemas/TextScore'- required:- - dataType- title: TextScore- properties:- _deprecation:- $ref: '#/components/schemas/Deprecation'- nullable: true- CreateScoreValue:- title: CreateScoreValue- oneOf:- - type: number- format: double- title: DoubleNumber- - type: string- title: String- description: >-- The value of the score. Must be passed as string for categorical and- text scores, and numeric for boolean and numeric scores- Comment:- title: Comment- type: object- properties:- id:- type: string- projectId:- type: string- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- objectType:- $ref: '#/components/schemas/CommentObjectType'- objectId:- type: string- content:- type: string- authorUserId:- type: string- nullable: true- description: The user ID of the comment author- required:- - id- - projectId- - createdAt- - updatedAt- - objectType- - objectId- - content- Dataset:- title: Dataset- type: object- properties:- id:- type: string- name:- type: string- description:- type: string- nullable: true- description: Description of the dataset- metadata:- description: Metadata associated with the dataset- inputSchema:- nullable: true- description: JSON Schema for validating dataset item inputs- expectedOutputSchema:- nullable: true- description: JSON Schema for validating dataset item expected outputs- projectId:- type: string- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- required:- - id- - name- - description- - metadata- - inputSchema- - expectedOutputSchema- - projectId- - createdAt- - updatedAt- DatasetItem:- title: DatasetItem- type: object- properties:- id:- type: string- status:- $ref: '#/components/schemas/DatasetStatus'- input:- description: Input data for the dataset item- expectedOutput:- description: Expected output for the dataset item- metadata:- description: Metadata associated with the dataset item- sourceTraceId:- type: string- nullable: true- description: The trace ID that sourced this dataset item- sourceObservationId:- type: string- nullable: true- description: The observation ID that sourced this dataset item- datasetId:- type: string- datasetName:- type: string- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- mediaReferences:- type: array- items:- $ref: '#/components/schemas/DatasetItemMediaReference'- description: >-- Resolved Langfuse media references found in input, expectedOutput,- and metadata.- required:- - id- - status- - input- - expectedOutput- - metadata- - sourceTraceId- - sourceObservationId- - datasetId- - datasetName- - createdAt- - updatedAt- - mediaReferences- DatasetItemMediaReference:- title: DatasetItemMediaReference- type: object- properties:- field:- $ref: '#/components/schemas/DatasetItemMediaReferenceField'- description: The dataset item field containing the reference- referenceString:- type: string- description: >-- The Langfuse media reference string, e.g.- `@@@langfuseMedia:type=image/png|id=...|source=bytes@@@`- jsonPath:- type: string- description: >-- JSONPath of the string holding the reference within the field, e.g.- `$['image']`- media:- $ref: '#/components/schemas/DatasetItemMediaReferenceMedia'- description: The resolved media record.- required:- - field- - referenceString- - jsonPath- - media- DatasetItemMediaReferenceField:- title: DatasetItemMediaReferenceField+components:+ schemas:+ AnnotationQueueStatus:+ title: AnnotationQueueStatustype: stringenum:- - input- - expectedOutput- - metadata- DatasetItemMediaReferenceMedia:- title: DatasetItemMediaReferenceMedia- type: object- properties:- mediaId:- type: string- description: The unique langfuse identifier of the media record- contentType:- type: string- description: The MIME type of the media record- contentLength:- type: integer- description: The size of the media record in bytes- url:- type: string- description: The signed download URL of the media record- urlExpiry:- type: string- description: The expiry date and time of the download URL- required:- - mediaId- - contentType- - contentLength- - url- - urlExpiry- DatasetRunItem:- title: DatasetRunItem+ - PENDING+ - COMPLETED+ AnnotationQueueObjectType:+ title: AnnotationQueueObjectType+ type: string+ enum:+ - TRACE+ - OBSERVATION+ - SESSION+ AnnotationQueue:+ title: AnnotationQueuetype: objectproperties:id:type: string- datasetRunId:- type: string- datasetRunName:- type: string- datasetItemId:- type: string- traceId:+ name:type: string- observationId:+ description:type: stringnullable: true- description: The observation ID associated with this run item+ scoreConfigIds:+ type: array+ items:+ type: stringcreatedAt:type: stringformat: date-time@@ -11466,6608 +8609,6618 @@ components:format: date-timerequired:- id- - datasetRunId- - datasetRunName- - datasetItemId- - traceId- - observationId+ - name+ - description+ - scoreConfigIds- createdAt- updatedAt- DatasetRun:- title: DatasetRun+ AnnotationQueueItem:+ title: AnnotationQueueItemtype: objectproperties:id:type: string- description: Unique identifier of the dataset run- name:- type: string- description: Name of the dataset run- description:+ queueId:type: string- nullable: true- description: Description of the run- metadata:- description: Metadata of the dataset run- datasetId:+ objectId:type: string- description: Id of the associated dataset- datasetName:+ objectType:+ $ref: '#/components/schemas/AnnotationQueueObjectType'+ status:+ $ref: '#/components/schemas/AnnotationQueueStatus'+ completedAt:type: string- description: Name of the associated dataset+ format: date-time+ nullable: truecreatedAt:type: stringformat: date-time- description: The date and time when the dataset run was createdupdatedAt:type: stringformat: date-time- description: The date and time when the dataset run was last updatedrequired:- id- - name- - description- - metadata- - datasetId- - datasetName+ - queueId+ - objectId+ - objectType+ - status- createdAt- updatedAt- DatasetRunWithItems:- title: DatasetRunWithItems+ PaginatedAnnotationQueues:+ title: PaginatedAnnotationQueuestype: objectproperties:- datasetRunItems:+ data:type: arrayitems:- $ref: '#/components/schemas/DatasetRunItem'- _deprecation:- $ref: '#/components/schemas/Deprecation'- nullable: true+ $ref: '#/components/schemas/AnnotationQueue'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'required:- - datasetRunItems- allOf:- - $ref: '#/components/schemas/DatasetRun'- Model:- title: Model+ - data+ - meta+ PaginatedAnnotationQueueItems:+ title: PaginatedAnnotationQueueItemstype: object- description: >-- Model definition used for transforming usage into USD cost and/or- tokenization.--- Models can have either simple flat pricing or tiered pricing:-- - Flat pricing: Single price per usage type (legacy, but still- supported)-- - Tiered pricing: Multiple pricing tiers with conditional matching based- on usage patterns or observation attributes--- The pricing tiers approach is recommended for models with usage-based- pricing variations.-- When using tiered pricing, the flat price fields (inputPrice,- outputPrice, prices) are populated-- from the default tier for backward compatibility.properties:- id:- type: string- modelName:- type: string- description: >-- Name of the model definition. If multiple with the same name exist,- they are applied in the following order: (1) custom over built-in,- (2) newest according to startTime where- model.startTime<observation.startTime- matchPattern:- type: string- description: >-- Regex pattern which matches this model definition to- generation.model. Useful in case of fine-tuned models. If you want- to exact match, use `(?i)^modelname$`- startDate:- type: string- format: date-time- nullable: true- description: Apply only to generations which are newer than this ISO date.- unit:- $ref: '#/components/schemas/ModelUsageUnit'- nullable: true- description: Unit used by this model.- inputPrice:- type: number- format: double- nullable: true- description: Deprecated. See 'prices' instead. Price (USD) per input unit- outputPrice:- type: number- format: double- nullable: true- description: Deprecated. See 'prices' instead. Price (USD) per output unit- totalPrice:- type: number- format: double- nullable: true- description: >-- Deprecated. See 'prices' instead. Price (USD) per total unit. Cannot- be set if input or output price is set.- tokenizerId:- type: string- nullable: true- description: >-- Optional. Tokenizer to be applied to observations which match to- this model. See docs for more details.- tokenizerConfig:- description: >-- Optional. Configuration for the selected tokenizer. Needs to be- JSON. See docs for more details.- isLangfuseManaged:- type: boolean- createdAt:- type: string- format: date-time- description: Timestamp when the model was created- prices:- type: object- additionalProperties:- $ref: '#/components/schemas/ModelPrice'- description: >-- Deprecated. Use 'pricingTiers' instead for models with usage-based- pricing variations.--- This field shows prices by usage type from the default pricing tier.- Maintained for backward compatibility.-- If the model uses tiered pricing, this field will be populated from- the default tier's prices.- pricingTiers:+ data:type: arrayitems:- $ref: '#/components/schemas/PricingTier'- description: >-- Array of pricing tiers with conditional pricing based on usage- thresholds or observation attributes.--- Pricing tiers enable accurate cost tracking for models that charge- different rates based on usage patterns-- or request attributes (e.g., high-volume usage, large context- windows, cached tokens, or service tiers).--- Each model must have exactly one default tier (isDefault=true,- priority=0) that serves as a fallback.-- Additional conditional tiers can be defined with specific matching- criteria.--- If this array is empty, the model uses legacy flat pricing from the- inputPrice/outputPrice/totalPrice fields.- required:- - id- - modelName- - matchPattern- - startDate- - inputPrice- - outputPrice- - totalPrice- - tokenizerId- - tokenizerConfig- - isLangfuseManaged- - createdAt- - prices- - pricingTiers- ModelPrice:- title: ModelPrice- type: object- properties:- price:- type: number- format: double- required:- - price- PricingTierCondition:- title: PricingTierCondition- oneOf:- - $ref: '#/components/schemas/PricingTierUsageCondition'- - $ref: '#/components/schemas/PricingTierAttributeCondition'- description: >-- Condition for matching a pricing tier against usage details or- observation attributes.--- Usage-detail conditions treat usageDetailPattern as a regex, sum all- matching usage values, and compare the sum to the numeric value.- Model-parameter and metadata conditions match an exact top-level key- against one or more string values.- PricingTierUsageCondition:- title: PricingTierUsageCondition- type: object- description: Condition that sums usage details whose keys match a regex.- properties:- usageDetailPattern:- type: string- description: Regex pattern matched against usage detail keys.- operator:- $ref: '#/components/schemas/PricingTierOperator'- value:- type: number- format: double- description: Numeric threshold for the summed matching usage values.- caseSensitive:- type: boolean- description: Whether the usage-detail regex is case-sensitive.+ $ref: '#/components/schemas/AnnotationQueueItem'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'required:- - usageDetailPattern- - operator- - value- - caseSensitive- PricingTierAttributeCondition:- title: PricingTierAttributeCondition+ - data+ - meta+ CreateAnnotationQueueRequest:+ title: CreateAnnotationQueueRequesttype: object- description: >-- Condition that matches any configured value for a top-level observation- attribute.properties:- source:- $ref: '#/components/schemas/PricingTierAttributeSource'- description: Observation attribute object evaluated by this condition.- key:+ name:type: string- description: Exact top-level attribute key.- operator:+ description:type: string- const: in- description: Membership operator.- values:+ nullable: true+ scoreConfigIds:type: arrayitems:type: string- description: Accepted string attribute values. At least one value is required.required:- - source- - key- - operator- - values- PricingTierAttributeSource:- title: PricingTierAttributeSource- type: string- enum:- - model_parameters- - metadata- PricingTierConditionInput:- title: PricingTierConditionInput- oneOf:- - $ref: '#/components/schemas/PricingTierUsageConditionInput'- - $ref: '#/components/schemas/PricingTierAttributeCondition'- description: Condition input for matching a pricing tier.- PricingTierUsageConditionInput:- title: PricingTierUsageConditionInput+ - name+ - scoreConfigIds+ CreateAnnotationQueueItemRequest:+ title: CreateAnnotationQueueItemRequesttype: object- description: Input condition that sums usage details whose keys match a regex.properties:- usageDetailPattern:+ objectId:type: string- description: Regex pattern matched against usage detail keys.- operator:- $ref: '#/components/schemas/PricingTierOperator'- value:- type: number- format: double- description: Numeric threshold for the summed matching usage values.- caseSensitive:- type: boolean+ objectType:+ $ref: '#/components/schemas/AnnotationQueueObjectType'+ status:+ $ref: '#/components/schemas/AnnotationQueueStatus'nullable: true- description: Whether the usage-detail regex is case-sensitive. Defaults to false.+ description: Defaults to PENDING for new queue itemsrequired:- - usageDetailPattern- - operator- - value- PricingTier:- title: PricingTier+ - objectId+ - objectType+ UpdateAnnotationQueueItemRequest:+ title: UpdateAnnotationQueueItemRequest+ type: object+ properties:+ status:+ $ref: '#/components/schemas/AnnotationQueueStatus'+ nullable: true+ DeleteAnnotationQueueItemResponse:+ title: DeleteAnnotationQueueItemResponse+ type: object+ properties:+ success:+ type: boolean+ message:+ type: string+ required:+ - success+ - message+ AnnotationQueueAssignmentRequest:+ title: AnnotationQueueAssignmentRequest+ type: object+ properties:+ userId:+ type: string+ required:+ - userId+ DeleteAnnotationQueueAssignmentResponse:+ title: DeleteAnnotationQueueAssignmentResponse+ type: object+ properties:+ success:+ type: boolean+ required:+ - success+ CreateAnnotationQueueAssignmentResponse:+ title: CreateAnnotationQueueAssignmentResponsetype: object+ properties:+ userId:+ type: string+ queueId:+ type: string+ projectId:+ type: string+ required:+ - userId+ - queueId+ - projectId+ BlobStorageIntegrationType:+ title: BlobStorageIntegrationType+ type: string+ enum:+ - S3+ - S3_COMPATIBLE+ - AZURE_BLOB_STORAGE+ BlobStorageIntegrationFileType:+ title: BlobStorageIntegrationFileType+ type: string+ enum:+ - JSON+ - CSV+ - JSONL+ - PARQUETdescription: >-- Pricing tier definition with conditional pricing based on usage- thresholds or observation attributes.--- Pricing tiers enable accurate cost tracking for LLM providers that- charge different rates based on usage patterns or request attributes.-- For example, some providers charge higher rates when context size- exceeds certain thresholds or when an accelerated service tier is- selected.--- How tier matching works:-- 1. Tiers are evaluated in ascending priority order (priority 1 before- priority 2, etc.)-- 2. The first tier where ALL conditions match is selected-- 3. If no conditional tiers match, the default tier is used as a fallback-- 4. The default tier has priority 0 and no conditions--- Why priorities matter:+ File format for exported data. `PARQUET` is a columnar binary format+ encoded and compressed by the storage engine; gzip compression does not+ apply to it. Note that the model-price columns (`input_price`,+ `output_price`, `total_price`) are not included in Parquet observation+ exports.+ BlobStorageIntegrationFileTypeResponse:+ title: BlobStorageIntegrationFileTypeResponse+ type: string+ enum:+ - JSON+ - CSV+ - JSONL+ - PARQUET+ description: File type reported for an existing integration.+ BlobStorageExportMode:+ title: BlobStorageExportMode+ type: string+ enum:+ - FULL_HISTORY+ - FROM_TODAY+ - FROM_CUSTOM_DATE+ BlobStorageExportFrequency:+ title: BlobStorageExportFrequency+ type: string+ enum:+ - every_20_minutes+ - hourly+ - daily+ - weekly+ BlobStorageExportSource:+ title: BlobStorageExportSource+ type: string+ enum:+ - LEGACY_TRACES_OBSERVATIONS+ - OBSERVATIONS_V2+ - LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS+ description: >-+ What data the integration exports.- - Lower priority numbers are evaluated first, allowing you to define- specific cases before general ones+ - `LEGACY_TRACES_OBSERVATIONS`: traces, observations, and scores tables.+ Observation columns are controlled by `exportFieldGroups`; field groups+ without a counterpart in this data model (e.g. `trace_context`) are+ omitted.- - Example: Priority 1 for "high usage" (>200K tokens), Priority 2 for- "medium usage" (>100K tokens), Priority 0 for default+ - `OBSERVATIONS_V2`: same data model as the+ `/api/public/v2/observations` endpoint, plus scores. Columns are+ controlled by `exportFieldGroups`.- - Without proper ordering, a less specific condition might match before- a more specific one+ - `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`: both sets. Observation+ columns of both portions are controlled by `exportFieldGroups`.- Every model must have exactly one default tier to ensure cost- calculation always succeeds.+ **Note:** which sources a deployment accepts depends on how far it has+ moved to the v4 data model. `OBSERVATIONS_V2` and the+ enriched-observations portion of+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the enriched observations+ table, so they require a deployment that already populates it.+ `LEGACY_TRACES_OBSERVATIONS` and the legacy portion of+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the legacy traces and+ observations tables, so they require a deployment that still populates+ those. A deployment part-way through the migration populates both and+ accepts every source. Selecting a source the deployment cannot serve is+ rejected with `400`, rather than exporting an empty result. See+ https://langfuse.com/docs/v4.+ BlobStorageExportFieldGroup:+ title: BlobStorageExportFieldGroup+ type: string+ enum:+ - core+ - basic+ - time+ - io+ - metadata+ - model+ - usage+ - prompt+ - metrics+ - tools+ - trace_context+ description: >-+ Field group selecting which observation columns are included in the+ export. Applies to all export sources; groups without a counterpart in+ the legacy data model (e.g. `trace_context`) are omitted from the legacy+ observations export.+ CreateBlobStorageIntegrationRequest:+ title: CreateBlobStorageIntegrationRequest+ type: objectproperties:- id:+ projectId:+ type: string+ description: ID of the project in which to configure the blob storage integration+ type:+ $ref: '#/components/schemas/BlobStorageIntegrationType'+ bucketName:type: string- description: Unique identifier for the pricing tier- name:+ description: >-+ Name of the storage bucket. For AZURE_BLOB_STORAGE, must be a valid+ Azure container name (3-63 chars, lowercase letters, numbers, and+ hyphens only, must start and end with a letter or number, no+ consecutive hyphens).+ endpoint:+ type: string+ nullable: true+ description: Custom endpoint URL (required for S3_COMPATIBLE type)+ region:type: stringdescription: >-- Name of the pricing tier for display and identification purposes.--- Examples: "Standard", "High Volume Tier", "Large Context", "Extended- Context Tier"- isDefault:+ Storage region used by S3-compatible clients (AWS, GCS, Cloudflare+ R2, MinIO, Azure location IDs such as eastus, OCI). Leading and+ trailing whitespace is removed. The remaining value must be 1-63+ letters, numbers, or hyphens, and cannot start or end with a hyphen.+ Examples: us-east-1, europe-west1, eastus, auto.+ accessKeyId:+ type: string+ nullable: true+ description: Access key ID for authentication+ secretAccessKey:+ type: string+ nullable: true+ description: Secret access key for authentication (will be encrypted when stored)+ prefix:+ type: string+ nullable: true+ description: >-+ Path prefix for exported files (must end with forward slash if+ provided)+ exportFrequency:+ $ref: '#/components/schemas/BlobStorageExportFrequency'+ enabled:+ type: boolean+ description: Whether the integration is active+ forcePathStyle:type: boolean+ description: Use path-style URLs for S3 requests+ fileType:+ $ref: '#/components/schemas/BlobStorageIntegrationFileType'+ exportMode:+ $ref: '#/components/schemas/BlobStorageExportMode'+ exportStartDate:+ type: string+ format: date-time+ nullable: truedescription: >-- Whether this is the default tier. Every model must have exactly one- default tier with priority 0 and no conditions.--- The default tier serves as a fallback when no conditional tiers- match, ensuring cost calculation always succeeds.-- It typically represents the base pricing for standard usage- patterns.- priority:- type: integer+ Custom start date for exports (required when exportMode is+ FROM_CUSTOM_DATE). Must not be in the future (27 h tolerance for+ timezone differences).+ compressed:+ type: boolean+ nullable: truedescription: >-- Priority for tier matching evaluation. Lower numbers = higher- priority (evaluated first).--- The default tier must always have priority 0. Conditional tiers- should have priority 1, 2, 3, etc.--- Example ordering:-- - Priority 0: Default tier (no conditions, always matches as- fallback)+ Enable gzip compression for exported files (.csv.gz, .json.gz,+ .jsonl.gz). Defaults to true.+ exportSource:+ $ref: '#/components/schemas/BlobStorageExportSource'+ nullable: true+ description: >-+ Data to export. When omitted on update, the existing value is+ preserved. When omitted on create, the default is `OBSERVATIONS_V2`+ on Langfuse Cloud, and on self-hosted deployments+ `LEGACY_TRACES_OBSERVATIONS` — or `OBSERVATIONS_V2` where the+ deployment no longer populates the legacy tables. The default is+ never a source the deployment cannot serve. Required when+ `exportFieldGroups` is provided.- - Priority 1: High usage tier (e.g., >200K tokens)- - Priority 2: Medium usage tier (e.g., >100K tokens)+ **Cloud-only project deprecation gate (effective 2026-05-20):** For+ projects created on or after 2026-05-20 on Langfuse Cloud,+ `LEGACY_TRACES_OBSERVATIONS` and+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP+ 400. Use `OBSERVATIONS_V2` for all new integrations. Self-hosted+ deployments are unaffected.- This ensures more specific conditions are checked before general- ones.- conditions:+ **Cloud-only integration deprecation gate (effective 2026-06-22):**+ On Langfuse Cloud, legacy export sources are only accepted for blob+ storage integrations created before 2026-06-22, regardless of+ project age. Requests that would create a new integration with+ `LEGACY_TRACES_OBSERVATIONS` or+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP+ 400. Use `OBSERVATIONS_V2` instead. Self-hosted deployments are+ unaffected.+ exportFieldGroups:type: arrayitems:- $ref: '#/components/schemas/PricingTierCondition'+ $ref: '#/components/schemas/BlobStorageExportFieldGroup'+ nullable: truedescription: >-- Array of conditions that must ALL be met for this tier to match (AND- logic).--- The default tier must have an empty conditions array. Conditional- tiers should have one or more conditions-- that define when this tier's pricing applies.+ Field groups to include in each exported observation row. Applies to+ all export sources; must include `core` if provided. When omitted on+ create, the column default (all groups) applies. When omitted on+ update, the existing value is preserved.- Multiple conditions enable complex matching scenarios (e.g., "high- input tokens AND low output tokens").- prices:- type: object- additionalProperties:- type: number- format: double+ `exportFieldGroups` requires `exportSource` to be provided in the+ same request.+ required:+ - projectId+ - type+ - bucketName+ - region+ - exportFrequency+ - enabled+ - forcePathStyle+ - fileType+ - exportMode+ BlobStorageIntegrationResponse:+ title: BlobStorageIntegrationResponse+ type: object+ properties:+ id:+ type: string+ projectId:+ type: string+ type:+ $ref: '#/components/schemas/BlobStorageIntegrationType'+ bucketName:+ type: string+ endpoint:+ type: string+ nullable: true+ region:+ type: string+ accessKeyId:+ type: string+ nullable: true+ prefix:+ type: string+ exportFrequency:+ $ref: '#/components/schemas/BlobStorageExportFrequency'+ enabled:+ type: boolean+ forcePathStyle:+ type: boolean+ fileType:+ $ref: '#/components/schemas/BlobStorageIntegrationFileTypeResponse'+ exportMode:+ $ref: '#/components/schemas/BlobStorageExportMode'+ exportStartDate:+ type: string+ format: date-time+ nullable: true+ compressed:+ type: boolean+ exportSource:+ $ref: '#/components/schemas/BlobStorageExportSource'+ exportFieldGroups:+ type: array+ items:+ $ref: '#/components/schemas/BlobStorageExportFieldGroup'+ nullable: truedescription: >-- Prices (USD) by usage type for this tier.--- Common usage types: "input", "output", "total", "request", "image"-- Prices are specified in USD per unit (e.g., per token, per request,- per second).--- Example: {"input": 0.000003, "output": 0.000015} means $3 per- million input tokens and $15 per million output tokens.+ Field groups included in each exported observation row. An empty+ list is treated as all groups during export.+ nextSyncAt:+ type: string+ format: date-time+ nullable: true+ lastSyncAt:+ type: string+ format: date-time+ nullable: true+ lastError:+ type: string+ nullable: true+ lastErrorAt:+ type: string+ format: date-time+ nullable: true+ createdAt:+ type: string+ format: date-time+ updatedAt:+ type: string+ format: date-timerequired:- id- - name- - isDefault- - priority- - conditions- - prices- PricingTierInput:- title: PricingTierInput+ - projectId+ - type+ - bucketName+ - endpoint+ - region+ - accessKeyId+ - prefix+ - exportFrequency+ - enabled+ - forcePathStyle+ - fileType+ - exportMode+ - exportStartDate+ - compressed+ - exportSource+ - exportFieldGroups+ - nextSyncAt+ - lastSyncAt+ - lastError+ - lastErrorAt+ - createdAt+ - updatedAt+ BlobStorageIntegrationsResponse:+ title: BlobStorageIntegrationsResponsetype: object+ properties:+ data:+ type: array+ items:+ $ref: '#/components/schemas/BlobStorageIntegrationResponse'+ required:+ - data+ BlobStorageSyncStatus:+ title: BlobStorageSyncStatus+ type: string+ enum:+ - idle+ - running+ - queued+ - up_to_date+ - disabled+ - errordescription: >-- Input schema for creating a pricing tier. The tier ID will be- automatically generated server-side.+ Sync status of the blob storage integration:+ - `disabled` — integration is not enabled- When creating a model with pricing tiers:+ - `error` — last export failed (see `lastError` for details)- - Exactly one tier must have isDefault=true (the fallback tier)+ - `running` — an export job is currently being processed- - The default tier must have priority=0 and conditions=[]+ - `queued` — next export is overdue (`nextSyncAt` is in the past) and+ waiting to be picked up by the worker- - All tier names and priorities must be unique within the model+ - `idle` — enabled but has never exported yet and no export is queued- - Each tier must define at least one price+ - `up_to_date` — all available data has been exported; next export is+ scheduled for the future- See PricingTier for detailed information about how tiers work and why- they're useful.- properties:- name:- type: string- description: >-- Name of the pricing tier for display and identification purposes.+ **ETL usage**: poll this endpoint and check for `up_to_date` status.+ Compare `lastSyncAt` against your+ ETL bookmark to determine if new data is available. Note that exports+ run with a 20-minute lag buffer,- Must be unique within the model. Common patterns: "Standard", "High- Volume Tier", "Extended Context"- isDefault:+ so `lastSyncAt` will always be at least 20 minutes behind real-time.+ BlobStorageIntegrationStatusResponse:+ title: BlobStorageIntegrationStatusResponse+ type: object+ properties:+ id:+ type: string+ projectId:+ type: string+ syncStatus:+ $ref: '#/components/schemas/BlobStorageSyncStatus'+ enabled:type: boolean+ lastSyncAt:+ type: string+ format: date-timenullable: truedescription: >-- Whether this is the default tier. Exactly one tier per model must be- marked as default.--- Requirements for default tier:-- - Must have isDefault=true-- - Must have priority=0-- - Must have empty conditions array (conditions=[])--- The default tier acts as a fallback when no conditional tiers match.- Defaults to false when omitted.- priority:- type: integer+ End of the last successfully exported time window. Compare against+ your ETL bookmark to determine if new data is available. Null if the+ integration has never synced.+ nextSyncAt:+ type: string+ format: date-time+ nullable: true+ description: When the next export is scheduled. Null if no sync has occurred yet.+ lastError:+ type: string+ nullable: truedescription: >-- Priority for tier matching evaluation. Lower numbers = higher- priority (evaluated first).--- Must be unique within the model. The default tier must have- priority=0.-- Conditional tiers should use priority 1, 2, 3, etc. based on their- specificity.- conditions:+ Raw error message from the storage provider (S3/Azure/GCS) if the+ last export failed. Cleared on successful export.+ lastErrorAt:+ type: string+ format: date-time+ nullable: true+ description: When the last error occurred. Cleared on successful export.+ required:+ - id+ - projectId+ - syncStatus+ - enabled+ - lastSyncAt+ - nextSyncAt+ - lastError+ - lastErrorAt+ BlobStorageIntegrationDeletionResponse:+ title: BlobStorageIntegrationDeletionResponse+ type: object+ properties:+ message:+ type: string+ required:+ - message+ CreateCommentRequest:+ title: CreateCommentRequest+ type: object+ properties:+ projectId:+ type: string+ description: The id of the project to attach the comment to.+ objectType:+ type: string+ description: >-+ The type of the object to attach the comment to (trace, observation,+ session, prompt).+ objectId:+ type: string+ description: >-+ The id of the object to attach the comment to. If this does not+ reference a valid existing object, an error will be thrown.+ content:+ type: string+ description: >-+ The content of the comment. May include markdown. Currently limited+ to 5000 characters.+ authorUserId:+ type: string+ nullable: true+ description: >-+ The id of the user who created the comment. Must be a member of the+ organization that owns the project, otherwise an error will be+ thrown.+ required:+ - projectId+ - objectType+ - objectId+ - content+ CreateCommentResponse:+ title: CreateCommentResponse+ type: object+ properties:+ id:+ type: string+ description: The id of the created object in Langfuse+ required:+ - id+ GetCommentsResponse:+ title: GetCommentsResponse+ type: object+ properties:+ data:type: arrayitems:- $ref: '#/components/schemas/PricingTierConditionInput'+ $ref: '#/components/schemas/Comment'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ required:+ - data+ - meta+ Deprecation:+ title: Deprecation+ type: object+ description: >-+ Migration signal returned by deprecated endpoints. Optional fields are+ omitted when they have no value.+ properties:+ message:+ type: stringdescription: >-- Array of conditions that must ALL be met for this tier to match (AND- logic).--- The default tier must have an empty array (conditions=[]).-- Conditional tiers should define one or more conditions that specify- when this tier's pricing applies.--- Conditions can compare summed matching usage details to a numeric- threshold, or exactly match a top-level model parameter or metadata- value.- prices:- type: object- additionalProperties:- type: number- format: double+ Human- and agent-readable summary of the deprecation and its+ replacement.+ replacement:+ type: string+ nullable: true+ description: >-+ The replacement endpoint, e.g. "GET /api/public/v2/observations".+ Omitted when the endpoint is being removed without a direct+ replacement.+ docsUrl:+ type: string+ nullable: true+ description: Link to the migration documentation (markdown), when available.+ sunsetAt:+ type: string+ nullable: true+ description: >-+ ISO date after which the endpoint may stop working, when a removal+ date is committed.+ required:+ - message+ Trace:+ title: Trace+ type: object+ properties:+ id:+ type: string+ description: The unique identifier of a trace+ timestamp:+ type: string+ format: date-time+ description: The timestamp when the trace was created+ name:+ type: string+ nullable: true+ description: The name of the trace+ input:+ nullable: true+ description: The input data of the trace. Can be any JSON.+ output:+ nullable: true+ description: The output data of the trace. Can be any JSON.+ sessionId:+ type: string+ nullable: true+ description: The session identifier associated with the trace+ release:+ type: string+ nullable: true+ description: The release version of the application when the trace was created+ version:+ type: string+ nullable: true+ description: The version of the trace+ userId:+ type: string+ nullable: true+ description: The user identifier associated with the trace+ metadata:+ nullable: true+ description: The metadata associated with the trace. Can be any JSON.+ tags:+ type: array+ items:+ type: string+ description: The tags associated with the trace.+ public:+ type: boolean+ description: Public traces are accessible via url without login+ environment:+ type: stringdescription: >-- Prices (USD) by usage type for this tier. At least one price must be- defined.--- Common usage types: "input", "output", "total", "request", "image"-- Prices are in USD per unit (e.g., per token).--- Example: {"input": 0.000003, "output": 0.000015} represents $3 per- million input tokens and $15 per million output tokens.+ The environment from which this trace originated. Can be any+ lowercase alphanumeric string with hyphens and underscores that does+ not start with 'langfuse'.required:+ - id+ - timestamp- name- - priority- - conditions- - prices- PricingTierOperator:- title: PricingTierOperator- type: string- enum:- - gt- - gte- - lt- - lte- - eq- - neq- description: Comparison operators for pricing tier conditions- ModelUsageUnit:- title: ModelUsageUnit- type: string- enum:- - CHARACTERS- - TOKENS- - MILLISECONDS- - SECONDS- - IMAGES- - REQUESTS- description: Unit of usage in Langfuse- ObservationLevel:- title: ObservationLevel- type: string- enum:- - DEBUG- - DEFAULT- - WARNING- - ERROR- MapValue:- title: MapValue- oneOf:- - type: string- nullable: true- title: String- - type: integer+ - sessionId+ - release+ - version+ - userId+ - tags+ - public+ - environment+ TraceWithDetails:+ title: TraceWithDetails+ type: object+ properties:+ htmlPath:+ type: string+ description: Path of trace in Langfuse UI+ latency:+ type: number+ format: doublenullable: true- title: Integer- - type: number- format: float+ description: Latency of trace in seconds+ totalCost:+ type: number+ format: doublenullable: true- title: FloatNumber- - type: boolean+ description: Cost of trace in USD+ observations:+ type: array+ items:+ type: stringnullable: true- title: Boolean- - type: array+ description: List of observation ids+ scores:+ type: arrayitems:type: stringnullable: true- title: Array- CommentObjectType:- title: CommentObjectType- type: string- enum:- - TRACE- - OBSERVATION- - SESSION- - PROMPT- DatasetStatus:- title: DatasetStatus- type: string- enum:- - ACTIVE- - ARCHIVED- ScoreSource:- title: ScoreSource- type: string- enum:- - ANNOTATION- - API- - EVAL- ScoreConfigDataType:- title: ScoreConfigDataType- type: string- enum:- - NUMERIC- - BOOLEAN- - CATEGORICAL- - TEXT- ScoreDataType:- title: ScoreDataType- type: string- enum:- - NUMERIC- - BOOLEAN- - CATEGORICAL- - CORRECTION- - TEXT- DeleteDatasetItemResponse:- title: DeleteDatasetItemResponse+ description: List of score ids+ required:+ - htmlPath+ allOf:+ - $ref: '#/components/schemas/Trace'+ TraceWithFullDetails:+ title: TraceWithFullDetailstype: objectproperties:- message:+ htmlPath:+ type: string+ description: Path of trace in Langfuse UI+ latency:+ type: number+ format: double+ nullable: true+ description: Latency of trace in seconds+ totalCost:+ type: number+ format: double+ nullable: true+ description: Cost of trace in USD+ observations:+ type: array+ items:+ $ref: '#/components/schemas/ObservationsView'+ description: List of observations+ scores:+ type: array+ items:+ $ref: '#/components/schemas/ScoreV1'+ description: List of scores+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: true+ required:+ - htmlPath+ - observations+ - scores+ allOf:+ - $ref: '#/components/schemas/Trace'+ Session:+ title: Session+ type: object+ properties:+ id:+ type: string+ createdAt:+ type: string+ format: date-time+ projectId:+ type: string+ environment:+ type: string+ description: The environment from which this session originated.+ required:+ - id+ - createdAt+ - projectId+ - environment+ SessionWithTraces:+ title: SessionWithTraces+ type: object+ properties:+ traces:+ type: array+ items:+ $ref: '#/components/schemas/Trace'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: true+ required:+ - traces+ allOf:+ - $ref: '#/components/schemas/Session'+ Observation:+ title: Observation+ type: object+ properties:+ id:+ type: string+ description: The unique identifier of the observation+ traceId:+ type: string+ nullable: true+ description: The trace ID associated with the observation+ type:+ type: string+ description: The type of the observation+ name:+ type: string+ nullable: true+ description: The name of the observation+ startTime:+ type: string+ format: date-time+ description: The start time of the observation+ endTime:+ type: string+ format: date-time+ nullable: true+ description: The end time of the observation.+ completionStartTime:type: string- description: Success message after deletion- required:- - message- CreateDatasetItemRequest:- title: CreateDatasetItemRequest- type: object- properties:- datasetName:+ format: date-time+ nullable: true+ description: The completion start time of the observation+ model:type: string- input:nullable: true- expectedOutput:+ description: The model used for the observation+ modelParameters:+ description: The parameters of the model used for the observation+ input:+ description: The input data of the observation+ version:+ type: stringnullable: true+ description: The version of the observationmetadata:+ description: Additional metadata of the observation+ output:+ description: The output data of the observation+ usage:+ $ref: '#/components/schemas/Usage'+ description: >-+ (Deprecated. Use usageDetails and costDetails instead.) The usage+ data of the observation+ level:+ $ref: '#/components/schemas/ObservationLevel'+ description: The level of the observation+ statusMessage:+ type: stringnullable: true- sourceTraceId:+ description: The status message of the observation+ parentObservationId:type: stringnullable: true- sourceObservationId:+ description: The parent observation ID+ promptId:type: stringnullable: true- id:+ description: The prompt ID associated with the observation+ usageDetails:+ type: object+ additionalProperties:+ type: integer+ description: >-+ The usage details of the observation. Key is the name of the usage+ metric, value is the number of units consumed. The total key is the+ sum of all (non-total) usage metrics or the total value ingested.+ costDetails:+ type: object+ additionalProperties:+ type: number+ format: double+ description: >-+ The cost details of the observation. Key is the name of the cost+ metric, value is the cost in USD. The total key is the sum of all+ (non-total) cost metrics or the total value ingested.+ environment:+ type: string+ description: >-+ The environment from which this observation originated. Can be any+ lowercase alphanumeric string with hyphens and underscores that does+ not start with 'langfuse'.+ required:+ - id+ - traceId+ - type+ - name+ - startTime+ - endTime+ - completionStartTime+ - model+ - modelParameters+ - input+ - version+ - metadata+ - output+ - usage+ - level+ - statusMessage+ - parentObservationId+ - promptId+ - usageDetails+ - costDetails+ - environment+ ObservationsView:+ title: ObservationsView+ type: object+ properties:+ promptName:+ type: string+ nullable: true+ description: The name of the prompt associated with the observation+ promptVersion:+ type: integer+ nullable: true+ description: The version of the prompt associated with the observation+ modelId:type: stringnullable: true+ description: The unique identifier of the model+ inputPrice:+ type: number+ format: double+ nullable: true+ description: The price of the input in USD+ outputPrice:+ type: number+ format: double+ nullable: true+ description: The price of the output in USD.+ totalPrice:+ type: number+ format: double+ nullable: true+ description: The total price in USD.+ calculatedInputCost:+ type: number+ format: double+ nullable: truedescription: >-- Dataset items are upserted on their id. Id needs to be unique- (project-level), cannot be reused across datasets, and must be at- most 255 characters.- status:- $ref: '#/components/schemas/DatasetStatus'+ (Deprecated. Use usageDetails and costDetails instead.) The+ calculated cost of the input in USD+ calculatedOutputCost:+ type: number+ format: doublenullable: true- description: Defaults to ACTIVE for newly created items+ description: >-+ (Deprecated. Use usageDetails and costDetails instead.) The+ calculated cost of the output in USD+ calculatedTotalCost:+ type: number+ format: double+ nullable: true+ description: >-+ (Deprecated. Use usageDetails and costDetails instead.) The+ calculated total cost in USD+ latency:+ type: number+ format: double+ nullable: true+ description: The latency in seconds.+ timeToFirstToken:+ type: number+ format: double+ nullable: true+ description: The time to the first token in secondsrequired:- - datasetName- PaginatedDatasetItems:- title: PaginatedDatasetItems+ - promptName+ - promptVersion+ - modelId+ - inputPrice+ - outputPrice+ - totalPrice+ - calculatedInputCost+ - calculatedOutputCost+ - calculatedTotalCost+ - latency+ - timeToFirstToken+ allOf:+ - $ref: '#/components/schemas/Observation'+ ObservationsViewSingle:+ title: ObservationsViewSingletype: objectproperties:- data:- type: array- items:- $ref: '#/components/schemas/DatasetItem'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- required:- - data- - meta- CreateDatasetRunItemRequest:- title: CreateDatasetRunItemRequest+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: true+ allOf:+ - $ref: '#/components/schemas/ObservationsView'+ ObservationV2:+ title: ObservationV2type: object+ description: >-+ An observation from the v2 API with field-group-based selection.++ Core fields are always present. Other fields are included only when+ their field group is requested.properties:- runName:+ id:type: string- runDescription:+ description: The unique identifier of the observation+ traceId:+ type: string+ nullable: true+ description: The trace ID associated with the observation+ startTime:+ type: string+ format: date-time+ description: The start time of the observation+ endTime:+ type: string+ format: date-time+ nullable: true+ description: The end time of the observation+ projectId:+ type: string+ description: The project ID this observation belongs to+ parentObservationId:+ type: string+ nullable: true+ description: >-+ The physical parent observation ID, if present.++ Observations marked as app roots by the SDK may retain a non-null+ parent ID.+ type:+ type: string+ description: The type of the observation (e.g. GENERATION, SPAN, EVENT)+ isRootObservation:+ type: boolean+ nullable: true+ description: >-+ Whether this observation is a logical root.++ This is true for observations without a physical parent and+ observations marked as app roots by the SDK.+ name:+ type: string+ nullable: true+ description: The name of the observation+ level:+ $ref: '#/components/schemas/ObservationLevel'+ nullable: true+ description: The level of the observation+ statusMessage:+ type: string+ nullable: true+ description: The status message of the observation+ version:+ type: string+ nullable: true+ description: The version of the observation+ environment:type: stringnullable: true- description: Description of the run. If run exists, description will be updated.- metadata:+ description: The environment from which this observation originated+ bookmarked:+ type: boolean+ nullable: true+ description: Whether the observation is bookmarked+ public:+ type: booleannullable: true- description: Metadata of the dataset run, updates run if run already exists- datasetItemId:- type: string- observationId:+ description: Whether the observation is public+ userId:type: stringnullable: true- traceId:+ description: The user ID associated with the observation+ sessionId:type: stringnullable: true- description: >-- traceId should always be provided. For compatibility with older SDK- versions it can also be inferred from the provided observationId.- datasetVersion:+ description: The session ID associated with the observation+ completionStartTime:type: stringformat: date-timenullable: true- description: >-- ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g.,- "2026-01-21T14:35:42Z").-- Specifies the dataset version to use for this experiment run.-- If provided, the experiment will use dataset items as they existed- at or before this timestamp.-- If not provided, uses the latest version of dataset items.+ description: The completion start time of the observationcreatedAt:type: stringformat: date-timenullable: true- description: >-- Optional timestamp to set the createdAt field of the dataset run- item. If not provided or null, defaults to current timestamp.- required:- - runName- - datasetItemId- PaginatedDatasetRunItems:- title: PaginatedDatasetRunItems- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/DatasetRunItem'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- _deprecation:- $ref: '#/components/schemas/Deprecation'- nullable: true- required:- - data- - meta- PaginatedDatasets:- title: PaginatedDatasets- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/Dataset'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- required:- - data- - meta- CreateDatasetRequest:- title: CreateDatasetRequest- type: object- properties:- name:- type: string- description:+ description: The creation timestamp of the observation+ updatedAt:type: string+ format: date-timenullable: true- metadata:+ description: The last update timestamp of the observation+ input:nullable: true- inputSchema:+ description: The input data of the observation+ output:nullable: true- description: >-- JSON Schema for validating dataset item inputs. When set, all new- and existing dataset items will be validated against this schema.- expectedOutputSchema:+ description: The output data of the observation+ metadata:nullable: true- description: >-- JSON Schema for validating dataset item expected outputs. When set,- all new and existing dataset items will be validated against this- schema.- required:- - name- PaginatedDatasetRuns:- title: PaginatedDatasetRuns- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/DatasetRun'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- _deprecation:- $ref: '#/components/schemas/Deprecation'+ description: Additional metadata of the observation+ model:+ type: stringnullable: true- required:- - data- - meta- DeleteDatasetRunResponse:- title: DeleteDatasetRunResponse- type: object- properties:- message:+ description: The model name as provided by the user+ internalModelId:type: string- required:- - message- EvaluatorType:- title: EvaluatorType- type: string- enum:- - llm_as_judge- - code- description: |-- The evaluator type.-- The public API supports LLM-as-a-judge and code evaluators.- CodeEvaluatorSourceCodeLanguage:- title: CodeEvaluatorSourceCodeLanguage- type: string- enum:- - PYTHON- - TYPESCRIPT- description: Code evaluator runtime language.- PromptVariableMappingSource:- title: PromptVariableMappingSource- type: string- enum:- - input- - output- - metadata- - tool_calls- - expected_output- - experiment_item_metadata- description: >-- Source field used to populate a prompt variable.--- Use these values when mapping evaluator prompt variables to live data.--- Source semantics:-- - `input`: the observation input payload-- - `output`: the observation output payload-- - `metadata`: the observation metadata object. Combine with `jsonPath`- when you need one nested field instead of the whole object.-- - `tool_calls`: the tool calls recorded on the observation, as an array- of `{id, name, arguments, type, index}` objects in the order the model- emitted them. Combine with `jsonPath` (for example `$[*].name`) to- select parts of each call.-- - `expected_output`: the experiment item's expected output when the- observation belongs to an experiment.-- - `experiment_item_metadata`: the experiment item's metadata when the- observation belongs to an experiment.- EvaluatorOutputScoreType:- title: EvaluatorOutputScoreType- type: string- enum:- - NUMERIC- - BOOLEAN- - CATEGORICAL- description: >-- Structured score type returned by an evaluator.--- This controls the type of score value Langfuse stores for evaluation- results:-- - `NUMERIC`: a numeric score such as `0.82`-- - `BOOLEAN`: a boolean score such as `true`-- - `CATEGORICAL`: one or more category labels from a fixed list- EvaluatorOutputDefinition:- title: EvaluatorOutputDefinition- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: '#/components/schemas/PublicEvaluatorNumericScore'- required:- - dataType- title: PublicEvaluatorNumericScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'- required:- - dataType- title: PublicEvaluatorBooleanScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'- required:- - dataType- title: PublicEvaluatorCategoricalScore- description: >-- Flat structured output definition used when creating or updating an- evaluator.--- - `dataType` is required.-- - `scoreReasoningInstructions` and `scoreValueInstructions` are optional- instructions.-- - `minValue` and `maxValue` apply only to `NUMERIC` outputs. If both are- set, `minValue` must not exceed `maxValue`.-- - `categories` and `shouldAllowMultipleMatches` apply only to- `CATEGORICAL` outputs.-- - Do not send `version`; that is an internal storage detail.- EvaluatorOutputDefinitionBase:- title: EvaluatorOutputDefinitionBase- type: object- properties:- scoreReasoningInstructions:+ nullable: true+ description: The internal model ID matched by Langfuse+ modelParameters:+ nullable: true+ description: The parameters of the model used for the observation+ usageDetails:+ type: object+ additionalProperties:+ type: integer+ nullable: true+ description: >-+ The usage details of the observation. Key is the usage metric name,+ value is the number of units consumed.+ costDetails:+ type: object+ additionalProperties:+ type: number+ format: double+ nullable: true+ description: >-+ The cost details of the observation. Key is the cost metric name,+ value is the cost in USD.+ totalCost:+ type: number+ format: double+ nullable: true+ description: The total cost of the observation in USD+ usagePricingTierName:type: stringnullable: truedescription: >-- Optional instructions for deriving the reasoning returned with the- score.- scoreValueInstructions:+ The name of the pricing tier applied to this observation's usage+ costs+ promptId:type: stringnullable: true- description: Optional instructions for deriving the score value.- PublicEvaluatorNumericScore:- title: PublicEvaluatorNumericScore- type: object- properties:- dataType:+ description: The prompt ID associated with the observation+ promptName:type: string- const: NUMERIC- description: Numeric score output.- minValue:+ nullable: true+ description: The prompt name associated with the observation+ promptVersion:+ type: integer+ nullable: true+ description: The prompt version associated with the observation+ latency:type: numberformat: doublenullable: true- description: Optional inclusive minimum value.- maxValue:+ description: The latency in seconds+ timeToFirstToken:type: numberformat: doublenullable: true- description: Optional inclusive maximum value.- required:- - dataType- allOf:- - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'- PublicEvaluatorBooleanScore:- title: PublicEvaluatorBooleanScore- type: object- properties:- dataType:- type: string- const: BOOLEAN- description: Boolean score output.- required:- - dataType- allOf:- - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'- PublicEvaluatorCategoricalScore:- title: PublicEvaluatorCategoricalScore- type: object- properties:- dataType:- type: string- const: CATEGORICAL- description: Categorical score output.- categories:- type: array- items:- type: string- description: Allowed category values. At least two unique values are required.- shouldAllowMultipleMatches:- type: boolean- description: Whether the evaluator may return more than one category.- required:- - dataType- - categories- - shouldAllowMultipleMatches- allOf:- - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'- PublicEvaluatorOutputDefinition:- title: PublicEvaluatorOutputDefinition- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: '#/components/schemas/PublicEvaluatorNumericScore'- required:- - dataType- title: PublicEvaluatorNumericScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'- required:- - dataType- title: PublicEvaluatorBooleanScore- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'- required:- - dataType- title: PublicEvaluatorCategoricalScore- description: >-- Flat evaluator output definition returned by the public API.--- This response always includes `dataType` and never includes an internal- output-definition `version`.-- Optional empty descriptions from legacy definitions are omitted.- EvaluationRuleStringFilterOperator:- title: EvaluationRuleStringFilterOperator- type: string- enum:- - '='- - contains- - does not contain- - starts with- - ends with- EvaluationRuleNumberFilterOperator:- title: EvaluationRuleNumberFilterOperator- type: string- enum:- - '='- - '>'- - <- - '>='- - <=- EvaluationRuleOptionsFilterOperator:- title: EvaluationRuleOptionsFilterOperator- type: string- enum:- - any of- - none of- EvaluationRuleArrayOptionsFilterOperator:- title: EvaluationRuleArrayOptionsFilterOperator- type: string- enum:- - any of- - none of- - all of- EvaluationRuleBooleanFilterOperator:- title: EvaluationRuleBooleanFilterOperator- type: string- enum:- - '='- - <>- EvaluationRuleNullFilterOperator:- title: EvaluationRuleNullFilterOperator- type: string- enum:- - is null- - is not null- DateTimeEvaluationRuleFilter:- title: DateTimeEvaluationRuleFilter- type: object- properties:- column:- type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'- description: Comparison operator for datetime values.- value:+ description: The time to first token in seconds+ modelId:type: string- format: date-time- description: Datetime value to compare against.- required:- - column- - operator- - value- StringEvaluationRuleFilter:- title: StringEvaluationRuleFilter- type: object- properties:- column:+ nullable: true+ description: >-+ The matched model ID. Null when the `model` field group is not+ requested.+ inputPrice:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'- value:+ nullable: true+ description: >-+ The input token price (USD per unit) from the matched model,+ serialized as a decimal string (e.g. "0.0001"). Null when the+ `model` field group is not requested.+ outputPrice:type: string- required:- - column- - operator- - value- NumberEvaluationRuleFilter:- title: NumberEvaluationRuleFilter- type: object- properties:- column:+ nullable: true+ description: >-+ The output token price (USD per unit) from the matched model,+ serialized as a decimal string (e.g. "0.0001"). Null when the+ `model` field group is not requested.+ totalPrice:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'- value:- type: number- format: double- required:- - column- - operator- - value- StringOptionsEvaluationRuleFilter:- title: StringOptionsEvaluationRuleFilter- type: object- properties:- column:+ nullable: true+ description: >-+ The total token price (USD per unit) from the matched model,+ serialized as a decimal string (e.g. "0.0001"). Null when the+ `model` field group is not requested.+ traceName:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'- value:+ nullable: true+ description: The name of the parent trace+ tags:type: arrayitems:type: string- description: One or more allowed string values.+ nullable: true+ description: Tags from the parent trace (denormalized onto the observation)+ release:+ type: string+ nullable: true+ description: The release version of the parent tracerequired:- - column- - operator- - value- ArrayOptionsEvaluationRuleFilter:- title: ArrayOptionsEvaluationRuleFilter+ - id+ - traceId+ - startTime+ - endTime+ - projectId+ - parentObservationId+ - type+ - modelId+ - inputPrice+ - outputPrice+ - totalPrice+ Usage:+ title: Usagetype: object+ description: >-+ (Deprecated. Use usageDetails and costDetails instead.) Standard+ interface for usage and costproperties:- column:+ input:+ type: integer+ description: Number of input units (e.g. tokens)+ output:+ type: integer+ description: Number of output units (e.g. tokens)+ total:+ type: integer+ description: Defaults to input+output if not set+ unit:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleArrayOptionsFilterOperator'- value:- type: array- items:- type: string- description: One or more array elements to match.+ nullable: true+ description: Unit of measurement+ inputCost:+ type: number+ format: double+ nullable: true+ description: USD input cost+ outputCost:+ type: number+ format: double+ nullable: true+ description: USD output cost+ totalCost:+ type: number+ format: double+ nullable: true+ description: USD total cost, defaults to input+outputrequired:- - column- - operator- - value- StringObjectEvaluationRuleFilter:- title: StringObjectEvaluationRuleFilter+ - input+ - output+ - total+ - unit+ ScoreConfig:+ title: ScoreConfigtype: object+ description: Configuration for a scoreproperties:- column:+ id:type: string- description: >-- Object-valued column to filter on. Currently only `metadata` is- supported.- key:+ name:type: string- description: Top-level key inside the object-valued column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'- value:+ createdAt:type: string- required:- - column- - key- - operator- - value- NumberObjectEvaluationRuleFilter:- title: NumberObjectEvaluationRuleFilter- type: object- properties:- column:+ format: date-time+ updatedAt:type: string- description: Object-valued column to filter on.- key:+ format: date-time+ projectId:type: string- description: Key inside the object-valued column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'- value:+ dataType:+ $ref: '#/components/schemas/ScoreConfigDataType'+ isArchived:+ type: boolean+ description: Whether the score config is archived. Defaults to false+ minValue:type: numberformat: double- required:- - column- - key- - operator- - value- CategoryOptionsEvaluationRuleFilter:- title: CategoryOptionsEvaluationRuleFilter- type: object- properties:- column:- type: string- description: Object-valued column to filter on.- key:- type: string- description: Key inside the object-valued column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'- value:+ nullable: true+ description: >-+ Sets minimum value for numerical scores. If not set, the minimum+ value defaults to -∞+ maxValue:+ type: number+ format: double+ nullable: true+ description: >-+ Sets maximum value for numerical scores. If not set, the maximum+ value defaults to +∞+ categories:type: arrayitems:- type: string- required:- - column- - key- - operator- - value- BooleanEvaluationRuleFilter:- title: BooleanEvaluationRuleFilter- type: object- properties:- column:+ $ref: '#/components/schemas/ConfigCategory'+ nullable: true+ description: Configures custom categories for categorical scores+ description:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/EvaluationRuleBooleanFilterOperator'- value:- type: boolean+ nullable: true+ description: Description of the score configrequired:- - column- - operator- - value- NullEvaluationRuleFilter:- title: NullEvaluationRuleFilter+ - id+ - name+ - createdAt+ - updatedAt+ - projectId+ - dataType+ - isArchived+ ConfigCategory:+ title: ConfigCategorytype: objectproperties:- column:- type: string- description: Column to filter on, for example `parentObservationId`.- operator:- $ref: '#/components/schemas/EvaluationRuleNullFilterOperator'value:+ type: number+ format: double+ label:type: string- const: ''- description: >-- Required empty-string placeholder used by the runtime filter- contract.- required:- - column- - operator- - value- PromptVariableMappingInput:- title: PromptVariableMappingInput- type: object- description: >-- Connects one prompt variable to data from an observation or experiment.--- Manual mappings are used for `llm_as_judge` evaluators. `code`- evaluators use a fixed runtime mapping managed by Langfuse.--- How to build a valid mapping list:-- 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.-- 2. Read the evaluator `variables` array.-- 3. Add exactly one mapping object for each variable in that array.-- 4. Use the variable name exactly as returned, without braces such as- `{{` or `}}`.-- 5. Choose the source field that should populate the variable.--- `jsonPath` is optional. Use it only when the selected source is a JSON- object and you want to extract one nested field before inserting it into- the evaluator prompt.--- Invalid, missing, or duplicate mappings return a validation error.- Malformed JSONPath expressions are also rejected.- properties:- variable:- type: string- description: >-- Prompt variable name without braces.--- Example: for the prompt `Judge {{input}} against {{output}}`, use- `input` and `output`.- source:- $ref: '#/components/schemas/PromptVariableMappingSource'- description: >-- Source field that should populate the prompt variable.--- Available sources are `input`, `output`, `metadata`, `tool_calls`,- `expected_output`, and `experiment_item_metadata`.- jsonPath:- type: string- nullable: true- description: >-- Optional JSONPath selector applied to the selected source before it- is passed to the evaluator prompt.--- Requirements:-- - Must start with `$`-- - Must be a syntactically valid JSONPath expression-- - Most useful with `source=metadata`required:- - variable- - source- PromptVariableMappingRead:- title: PromptVariableMappingRead+ - value+ - label+ BaseScoreV1:+ title: BaseScoreV1type: object- description: |-- Connects one prompt variable to source data.-- `source` is `null` when the mapping is incomplete.properties:- variable:+ id:+ type: string+ traceId:+ type: string+ name:type: string- description: Prompt variable name without braces.source:+ $ref: '#/components/schemas/ScoreSource'+ observationId:type: stringnullable: true- description: >-- Stored source field populating the variable, or `null` when the- mapping is incomplete.- jsonPath:+ description: The observation ID associated with the score+ timestamp:type: string- nullable: true- description: Optional JSONPath selector applied to the selected source.- required:- - variable- - source- PromptVariableMapping:- title: PromptVariableMapping- oneOf:- - $ref: '#/components/schemas/PromptVariableMappingRead'- - $ref: '#/components/schemas/LegacyPromptVariableMapping'- description: Connects one prompt variable to source data.- LegacyPromptVariableMapping:- title: LegacyPromptVariableMapping- type: object- description: >-- **Deprecated:** Connects one prompt variable to data from a legacy trace- or dataset evaluation rule.--- `langfuseObject` selects the object kind. `objectName` separately- selects a named observation and is `null` for trace and dataset-item- mappings.- properties:- mappingType:+ format: date-time+ createdAt:type: string- const: legacy- description: Explicitly marks this as a legacy mapping.- variable:+ format: date-time+ updatedAt:type: string- description: Prompt variable name without braces.- langfuseObject:- $ref: '#/components/schemas/LegacyEvaluationObject'- description: Legacy object kind selected as the mapping source.- objectName:+ format: date-time+ authorUserId:type: stringnullable: true- description: >-- Observation name to match, or `null` when `langfuseObject` is- `trace` or `dataset_item`.- source:+ description: The user ID of the author+ comment:type: string- description: Field selected from the legacy object.- jsonPath:+ nullable: true+ description: Comment on the score+ metadata:+ description: Metadata associated with the score+ configId:type: stringnullable: true- description: Optional JSONPath selector applied to the selected field.+ description: >-+ Reference a score config on a score. When set, config and score name+ must be equal and value must comply to optionally defined numerical+ range+ queueId:+ type: string+ nullable: true+ description: >-+ The annotation queue referenced by the score. Indicates if score was+ initially created while processing annotation queue.+ environment:+ type: string+ description: >-+ The environment from which this score originated. Can be any+ lowercase alphanumeric string with hyphens and underscores that does+ not start with 'langfuse'.required:- - mappingType- - variable- - langfuseObject- - objectName+ - id+ - traceId+ - name- source- LegacyEvaluationObject:- title: LegacyEvaluationObject- type: string- enum:- - trace- - span- - generation- - event- - agent- - tool- - chain- - retriever- - evaluator- - embedding- - guardrail- - dataset_item- description: >-- **Deprecated:** Legacy Langfuse object kind used by trace and dataset- evaluation rules.- EvaluationRuleFilter:- title: EvaluationRuleFilter- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - datetime- - $ref: '#/components/schemas/DateTimeEvaluationRuleFilter'- required:- - type- title: DateTimeEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - string- - $ref: '#/components/schemas/StringEvaluationRuleFilter'- required:- - type- title: StringEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - number- - $ref: '#/components/schemas/NumberEvaluationRuleFilter'- required:- - type- title: NumberEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - stringOptions- - $ref: '#/components/schemas/StringOptionsEvaluationRuleFilter'- required:- - type- title: StringOptionsEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - categoryOptions- - $ref: '#/components/schemas/CategoryOptionsEvaluationRuleFilter'- required:- - type- title: CategoryOptionsEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - arrayOptions- - $ref: '#/components/schemas/ArrayOptionsEvaluationRuleFilter'- required:- - type- title: ArrayOptionsEvaluationRuleFilter+ - timestamp+ - createdAt+ - updatedAt+ - authorUserId+ - comment+ - metadata+ - configId+ - queueId+ - environment+ NumericScoreV1:+ title: NumericScoreV1+ type: object+ properties:+ value:+ type: number+ format: double+ description: The numeric value of the score+ required:+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScoreV1'+ BooleanScoreV1:+ title: BooleanScoreV1+ type: object+ properties:+ value:+ type: number+ format: double+ description: >-+ The numeric value of the score. Equals 1 for "True" and 0 for "False"+ stringValue:+ type: string+ description: >-+ The string representation of the score value. Is inferred from the+ numeric value and equals "True" or "False"+ required:+ - value+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScoreV1'+ CategoricalScoreV1:+ title: CategoricalScoreV1+ type: object+ properties:+ value:+ type: number+ format: double+ description: >-+ Represents the numeric category mapping of the stringValue. If no+ config is linked, defaults to 0.+ stringValue:+ type: string+ description: >-+ The string representation of the score value. If no config is+ linked, can be any string. Otherwise, must map to a config category+ required:+ - value+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScoreV1'+ TextScoreV1:+ title: TextScoreV1+ type: object+ properties:+ stringValue:+ type: string+ description: The text content of the score (1-500 characters)+ required:+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScoreV1'+ ScoreV1:+ title: ScoreV1+ oneOf:- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - stringObject- - $ref: '#/components/schemas/StringObjectEvaluationRuleFilter'+ - NUMERIC+ - $ref: '#/components/schemas/NumericScoreV1'required:- - type- title: StringObjectEvaluationRuleFilter+ - dataType+ title: NumericScoreV1- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - numberObject- - $ref: '#/components/schemas/NumberObjectEvaluationRuleFilter'+ - CATEGORICAL+ - $ref: '#/components/schemas/CategoricalScoreV1'required:- - type- title: NumberObjectEvaluationRuleFilter+ - dataType+ title: CategoricalScoreV1- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - boolean- - $ref: '#/components/schemas/BooleanEvaluationRuleFilter'+ - BOOLEAN+ - $ref: '#/components/schemas/BooleanScoreV1'required:- - type- title: BooleanEvaluationRuleFilter+ - dataType+ title: BooleanScoreV1- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - 'null'- - $ref: '#/components/schemas/NullEvaluationRuleFilter'+ - TEXT+ - $ref: '#/components/schemas/TextScoreV1'required:- - type- title: NullEvaluationRuleFilter- description: >-- One filter condition used to decide whether a live-ingested observation- should be evaluated.--- All filters must be satisfied for the rule to run. Experiment scope is- expressed with filters rather than a separate target field:-- - `isExperimentItemRootSpan = true` limits execution to experiment item- roots.-- - `datasetId` limits execution to experiments for the selected datasets.- Use dataset IDs from `GET /api/public/v2/datasets`.--- Pick the filter `type` first, as it determines the required fields and- value shape. Use `key` only for object filters such as `metadata`.--- Operator quick reference by filter `type`:-- - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends- with`-- - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`-- - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`-- - `stringOptions`: `any of`, `none of`-- - `arrayOptions`: `any of`, `none of`, `all of`-- - `stringObject`: same operators as `string`-- - `boolean`: `"="`, `"<>"`-- - `null`: `is null`, `is not null`--- Supported columns. Each column accepts exactly one filter `type`;- sending a column with a different `type` is rejected with `400`.-- - `type`: `stringOptions`, values `SPAN`, `EVENT`, `GENERATION`,- `AGENT`, `TOOL`, `CHAIN`, `RETRIEVER`, `EVALUATOR`, `EMBEDDING`,- `GUARDRAIL`-- - `name`: `stringOptions`-- - `environment`: `stringOptions`-- - `level`: `stringOptions`, values `DEBUG`, `DEFAULT`, `WARNING`,- `ERROR`-- - `version`: `string`-- - `traceName`: `stringOptions`-- - `userId`: `string`-- - `sessionId`: `string`-- - `tags`: `arrayOptions`-- - `metadata`: `stringObject`, requires `key`-- - `isRootObservation`: `boolean`; true when the observation has no- parent or is explicitly marked as an application root-- - `parentObservationId`: `null`-- - `experimentId`: `stringOptions`-- - `isExperimentItemRootSpan`: `boolean`-- - `calledToolNames`: `arrayOptions`-- - `toolCalls`: `number`, the number of tool calls on the observation-- - `datasetId`: `stringOptions`. Use dataset `id` values from `GET- /api/public/v2/datasets`, not dataset names.- EvaluationRuleReadFilterBase:- title: EvaluationRuleReadFilterBase+ - dataType+ title: TextScoreV1+ BaseScore:+ title: BaseScoretype: object- description: Stored filter returned verbatim by evaluation-rule read endpoints.properties:- type:- type: string- description: >-- Stored filter type. This is not broken down into separate public- response types.- column:+ id:type: string- operator:+ traceId:type: string- value:nullable: true- description: Stored filter value. Its shape depends on the filter type.- required:- - type- - column- - operator- EvaluationRuleReadFilterWithKey:- title: EvaluationRuleReadFilterWithKey- type: object- properties:- key:+ description: The trace ID associated with the score+ sessionId:type: string- description: Stored object or legacy filter key.- required:- - key- allOf:- - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'- EvaluationRuleReadFilter:- title: EvaluationRuleReadFilter- oneOf:- - $ref: '#/components/schemas/EvaluationRuleReadFilterWithKey'- - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'- description: >-- Filter returned verbatim by evaluation-rule read endpoints.--- Filters with a stored `key` use the keyed shape. All other filters use- the base shape. Response filters are intentionally not broken down by- their internal `type` value.- PublicApiErrorCode:- title: PublicApiErrorCode- type: string- enum:- - authentication_failed- - access_denied- - invalid_request- - invalid_query- - invalid_body- - resource_not_found- - conflict- - rate_limited- - method_not_allowed- - internal_error- description: >-- Stable machine-readable error code returned by the evaluators and- evaluation-rules API.- PublicApiValidationIssue:- title: PublicApiValidationIssue- type: object- description: One request validation issue.- properties:- code:+ nullable: true+ description: The session ID associated with the score+ observationId:type: string- description: Validator issue code.- message:+ nullable: true+ description: The observation ID associated with the score+ datasetRunId:type: string- description: Human-readable explanation.- path:- type: array- items: {}- description: Path to the invalid request field.- required:- - code- - message- - path- PublicApiErrorDetails:- title: PublicApiErrorDetails- type: object- description: Optional structured context for validation and rate-limit errors.- properties:- issues:- type: array- items:- $ref: '#/components/schemas/PublicApiValidationIssue'nullable: true- description: Validation issues for an invalid request body or query.- retryAfterSeconds:- type: integer+ description: The dataset run ID associated with the score+ name:+ type: string+ source:+ $ref: '#/components/schemas/ScoreSource'+ timestamp:+ type: string+ format: date-time+ createdAt:+ type: string+ format: date-time+ updatedAt:+ type: string+ format: date-time+ authorUserId:+ type: stringnullable: true- description: Number of seconds to wait before retrying a rate-limited request.- limit:- type: integer+ description: The user ID of the author+ comment:+ type: stringnullable: true- description: Rate-limit request allowance.- remaining:- type: integer+ description: Comment on the score+ metadata:+ description: Metadata associated with the score+ configId:+ type: stringnullable: true- description: Remaining requests in the current rate-limit window.- resetAt:+ description: >-+ Reference a score config on a score. When set, config and score name+ must be equal and value must comply to optionally defined numerical+ range+ queueId:type: string- format: date-timenullable: true- description: Time when the current rate-limit window resets.- PublicApiError:- title: PublicApiError+ description: >-+ The annotation queue referenced by the score. Indicates if score was+ initially created while processing annotation queue.+ environment:+ type: string+ description: >-+ The environment from which this score originated. Can be any+ lowercase alphanumeric string with hyphens and underscores that does+ not start with 'langfuse'.+ required:+ - id+ - name+ - source+ - timestamp+ - createdAt+ - updatedAt+ - authorUserId+ - comment+ - metadata+ - configId+ - queueId+ - environment+ NumericScore:+ title: NumericScoretype: object- description: >-- Standard error response for the stable evaluators and evaluation-rules- API.--- Use the HTTP status for the broad failure class and `code` for- programmatic handling. `details` is included when field-level validation- or retry information is available.properties:- message:- type: string- description: Human-readable description of the failure.- code:- $ref: '#/components/schemas/PublicApiErrorCode'- description: Stable machine-readable error code.- details:- $ref: '#/components/schemas/PublicApiErrorDetails'- nullable: true- description: Optional structured validation or rate-limit context.+ value:+ type: number+ format: double+ description: The numeric value of the scorerequired:- - message- - code- EvaluationRuleEvaluatorAssignmentInput:- title: EvaluationRuleEvaluatorAssignmentInput+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScore'+ BooleanScore:+ title: BooleanScoretype: objectproperties:- evaluatorId:- type: string+ value:+ type: number+ format: doubledescription: >-- Stable evaluator identifier. The rule automatically uses that- evaluator's latest version.- variableMapping:- type: array- items:- $ref: '#/components/schemas/PromptVariableMappingInput'- nullable: true+ The numeric value of the score. Equals 1 for "True" and 0 for "False"+ stringValue:+ type: stringdescription: >-- Rule-specific prompt-variable mapping.--- Set to `null` or omit to inherit the evaluator's latest default- mapping. Code evaluators use the fixed runtime mapping and should- use `null`.+ The string representation of the score value. Is inferred from the+ numeric value and equals "True" or "False"required:- - evaluatorId- EvaluatorAssignment:- title: EvaluatorAssignment+ - value+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScore'+ CategoricalScore:+ title: CategoricalScoretype: objectproperties:- evaluatorId:+ value:+ type: number+ format: double+ description: >-+ Represents the numeric category mapping of the stringValue. If no+ config is linked, defaults to 0.+ stringValue:type: string- description: Stable identifier of the evaluator associated with this rule.- variableMapping:- type: array- items:- $ref: '#/components/schemas/PromptVariableMapping'- nullable: truedescription: >-- Stored rule-specific override, or `null` when the evaluator's latest- default mapping is inherited. Legacy mappings are returned as an- explicit `legacy` variant.+ The string representation of the score value. If no config is+ linked, can be any string. Otherwise, must map to a config categoryrequired:- - evaluatorId- - variableMapping- EvaluationRule:- title: EvaluationRule+ - value+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScore'+ CorrectionScore:+ title: CorrectionScoretype: object- description: >-- Live evaluation rule for incoming observations.--- A rule determines which evaluators should be used, which observations- should trigger scoring, how often scoring should run, and which- observation fields should populate prompt variables.properties:- id:+ value:+ type: number+ format: double+ description: The numeric value of the score. Always 0 for correction scores.+ stringValue:type: string- description: Stable evaluation-rule identifier.- name:+ description: The string representation of the correction content+ required:+ - value+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScore'+ TextScore:+ title: TextScore+ type: object+ properties:+ stringValue:type: string- description: >-- Human-readable rule name. This is independent from evaluator names- and does not need to be unique.- createdBy:- $ref: '#/components/schemas/Creator'+ description: The text content of the score (1-500 characters)+ required:+ - stringValue+ allOf:+ - $ref: '#/components/schemas/BaseScore'+ Score:+ title: Score+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - NUMERIC+ - $ref: '#/components/schemas/NumericScore'+ required:+ - dataType+ title: NumericScore+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CATEGORICAL+ - $ref: '#/components/schemas/CategoricalScore'+ required:+ - dataType+ title: CategoricalScore+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - BOOLEAN+ - $ref: '#/components/schemas/BooleanScore'+ required:+ - dataType+ title: BooleanScore+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CORRECTION+ - $ref: '#/components/schemas/CorrectionScore'+ required:+ - dataType+ title: CorrectionScore+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - TEXT+ - $ref: '#/components/schemas/TextScore'+ required:+ - dataType+ title: TextScore+ properties:+ _deprecation:+ $ref: '#/components/schemas/Deprecation'nullable: true- description: User who created this rule, or `null` when no user can be resolved.- enabled:- type: boolean- description: Whether live execution is enabled for this rule.- sampling:- type: number+ CreateScoreValue:+ title: CreateScoreValue+ oneOf:+ - type: numberformat: double- description: |-- Fraction of matching observations that should be evaluated.-- Must be between `0` and `1`.- - `1` evaluates every matching observation.- - `0.25` evaluates approximately 25% of matching observations.- filter:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleReadFilter'- description: >-- List of stored filter conditions returned verbatim. Filters with a- `key` use the keyed response shape; all others use the base shape.- These response shapes are not broken down by internal filter type.- An empty list matches every incoming object.- evaluatorAssignments:- type: array- items:- $ref: '#/components/schemas/EvaluatorAssignment'- description: Evaluators attached to this rule in deterministic assignment order.+ title: DoubleNumber+ - type: string+ title: String+ description: >-+ The value of the score. Must be passed as string for categorical and+ text scores, and numeric for boolean and numeric scores+ Comment:+ title: Comment+ type: object+ properties:+ id:+ type: string+ projectId:+ type: stringcreatedAt:type: stringformat: date-time- description: Timestamp when the evaluation rule was created.updatedAt:type: stringformat: date-time- description: Timestamp when the evaluation rule was last updated.+ objectType:+ $ref: '#/components/schemas/CommentObjectType'+ objectId:+ type: string+ content:+ type: string+ authorUserId:+ type: string+ nullable: true+ description: The user ID of the comment authorrequired:- id- - name- - createdBy- - enabled- - sampling- - filter- - evaluatorAssignments+ - projectId- createdAt- updatedAt- CreateEvaluationRuleRequest:- title: CreateEvaluationRuleRequest+ - objectType+ - objectId+ - content+ Dataset:+ title: Datasettype: objectproperties:- name:+ id:type: string- description: >-- Human-readable rule name. Names are not identifiers and do not need- to be unique.- enabled:- type: boolean- description: >-- Whether live execution should start immediately. Enabled rules- require at least one evaluator assignment.- sampling:- type: number- format: double- nullable: true- description: >-- Fraction of matching observations to evaluate. Omit this field to- use the default of `1`, which evaluates every match. `null` is not- accepted.- filter:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleFilter'- nullable: true- description: >-- Conditions used to select observations. Defaults to an empty list,- which matches every incoming observation.- evaluatorAssignments:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'- description: >-- Evaluators to attach to this rule. Disabled rules may use an empty- list as a draft.- required:- - name- - enabled- - evaluatorAssignments- UpdateEvaluationRuleRequest:- title: UpdateEvaluationRuleRequest- type: object- description: At least one field must be provided.- properties:name:type: string+ description:+ type: stringnullable: true- description: New human-readable rule name.- enabled:- type: boolean- nullable: true- description: New desired live-execution state.- sampling:- type: number- format: double- nullable: true- description: >-- New fraction of matching observations to evaluate. Omit to keep the- current value.- filter:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleFilter'+ description: Description of the dataset+ metadata:+ description: Metadata associated with the dataset+ inputSchema:nullable: true- description: >-- Complete replacement filter list. An empty list matches every- incoming observation.- evaluatorAssignments:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'+ description: JSON Schema for validating dataset item inputs+ expectedOutputSchema:nullable: true- description: >-- Complete replacement assignment list. An empty list disables the- rule.- EvaluationRulesPage:- title: EvaluationRulesPage- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/EvaluationRule'- description: Evaluation rules for this page.- meta:- $ref: '#/components/schemas/CursorMeta'- description: Cursor pagination metadata.- required:- - data- - meta- DeletedEvaluationRule:- title: DeletedEvaluationRule- type: object- description: Confirmation returned after successful evaluation-rule deletion.- properties:- id:+ description: JSON Schema for validating dataset item expected outputs+ projectId:type: string- description: Identifier of the deleted evaluation rule.- required:- - id- Creator:- title: Creator- type: object- description: User who created the resource, or `null` when no user can be resolved.- properties:- id:+ createdAt:type: string- description: User identifier.- name:+ format: date-time+ updatedAt:type: string- nullable: true- description: User name, or `null` when unavailable.+ format: date-timerequired:- id- name- EvaluatorModelConfig:- title: EvaluatorModelConfig- type: object- description: >-- Explicit model configuration for an evaluator.--- If the complete `modelConfig` is `null`, Langfuse uses the project's- default evaluation model.-- If provided, the model must be available to the project when the- evaluator or evaluation rule is enabled.- properties:- provider:- type: string- description: >-- Provider identifier, for example `openai` or `anthropic`.--- To discover valid values for the current project, call `GET- /api/public/llm-connections` and use one of the returned `provider`- values.- model:- type: string- description: >-- Model identifier exposed by the provider, for example `gpt-4.1-mini`.- required:- - provider- - model- EvaluatorChatMessageRole:- title: EvaluatorChatMessageRole- type: string- enum:- - system- - user- - assistant- description: Role of an evaluator prompt message.- EvaluatorChatMessage:- title: EvaluatorChatMessage- type: object- description: One chat message in an evaluator prompt.- properties:- role:- $ref: '#/components/schemas/EvaluatorChatMessageRole'- content:- type: string- description: Message content. Evaluator variables use `{{variable}}` syntax.- required:- - role- - content- EvaluatorChatPrompt:- title: EvaluatorChatPrompt- type: array- items:- $ref: '#/components/schemas/EvaluatorChatMessage'- description: >-- An ordered list of chat messages. A system message is only allowed as- the first message.- EvaluatorChatPromptInput:- title: EvaluatorChatPromptInput- oneOf:- - type: string- title: String- - $ref: '#/components/schemas/EvaluatorChatPrompt'- description: A user prompt string shortcut, or an ordered list of chat messages.- EvaluatorVersionBase:- title: EvaluatorVersionBase+ - description+ - metadata+ - inputSchema+ - expectedOutputSchema+ - projectId+ - createdAt+ - updatedAt+ DatasetItem:+ title: DatasetItemtype: objectproperties:id:type: string- description: Stable identifier of this evaluator version.- version:- type: integer- description: Monotonically increasing evaluator version number.+ status:+ $ref: '#/components/schemas/DatasetStatus'+ input:+ description: Input data for the dataset item+ expectedOutput:+ description: Expected output for the dataset item+ metadata:+ description: Metadata associated with the dataset item+ sourceTraceId:+ type: string+ nullable: true+ description: The trace ID that sourced this dataset item+ sourceObservationId:+ type: string+ nullable: true+ description: The observation ID that sourced this dataset item+ datasetId:+ type: string+ datasetName:+ type: stringcreatedAt:type: stringformat: date-time- description: Timestamp when this evaluator version was created.- createdBy:- $ref: '#/components/schemas/Creator'- nullable: true+ updatedAt:+ type: string+ format: date-time+ mediaReferences:+ type: array+ items:+ $ref: '#/components/schemas/DatasetItemMediaReference'description: >-- User who created this version, or `null` when no user can be- resolved.+ Resolved Langfuse media references found in input, expectedOutput,+ and metadata.required:- id- - version+ - status+ - input+ - expectedOutput+ - metadata+ - sourceTraceId+ - sourceObservationId+ - datasetId+ - datasetName- createdAt- - createdBy- LlmAsJudgeEvaluatorVersion:- title: LlmAsJudgeEvaluatorVersion+ - updatedAt+ - mediaReferences+ DatasetItemMediaReference:+ title: DatasetItemMediaReferencetype: objectproperties:- type:+ field:+ $ref: '#/components/schemas/DatasetItemMediaReferenceField'+ description: The dataset item field containing the reference+ referenceString:type: string- const: llm_as_judge- description: Evaluator type.- prompt:- $ref: '#/components/schemas/EvaluatorChatPrompt'- description: Ordered chat messages used during evaluation.- variables:- type: array- items:- type: string- description: >-- Variables extracted from the prompt and available for- evaluation-rule mappings.--- Every variable must be mapped exactly once when a rule provides an- explicit mapping.- variableMapping:- type: array- items:- $ref: '#/components/schemas/PromptVariableMappingRead'- nullable: true- description: >-- Default variable mapping for this evaluator version, or `null` when- no default is configured.- modelConfig:- $ref: '#/components/schemas/EvaluatorModelConfig'- nullable: truedescription: >-- Explicit model configuration, or `null` when the project's default- evaluation model is used.- outputDefinition:- $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'+ The Langfuse media reference string, e.g.+ `@@@langfuseMedia:type=image/png|id=...|source=bytes@@@`+ jsonPath:+ type: stringdescription: >-- Structured output schema returned by this evaluator.--- Responses include `dataType` and omit the internal output-definition- persistence version.+ JSONPath of the string holding the reference within the field, e.g.+ `$['image']`+ media:+ $ref: '#/components/schemas/DatasetItemMediaReferenceMedia'+ description: The resolved media record.required:- - type- - prompt- - variables- - variableMapping- - modelConfig- - outputDefinition- allOf:- - $ref: '#/components/schemas/EvaluatorVersionBase'- CodeEvaluatorVersion:- title: CodeEvaluatorVersion+ - field+ - referenceString+ - jsonPath+ - media+ DatasetItemMediaReferenceField:+ title: DatasetItemMediaReferenceField+ type: string+ enum:+ - input+ - expectedOutput+ - metadata+ DatasetItemMediaReferenceMedia:+ title: DatasetItemMediaReferenceMediatype: objectproperties:- type:+ mediaId:type: string- const: code- description: Evaluator type.- sourceCode:+ description: The unique langfuse identifier of the media record+ contentType:type: string- description: Source code executed for each matched observation.- sourceCodeLanguage:- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'- description: Runtime language used to execute the source code.+ description: The MIME type of the media record+ contentLength:+ type: integer+ description: The size of the media record in bytes+ url:+ type: string+ description: The signed download URL of the media record+ urlExpiry:+ type: string+ description: The expiry date and time of the download URLrequired:- - type- - sourceCode- - sourceCodeLanguage- allOf:- - $ref: '#/components/schemas/EvaluatorVersionBase'- EvaluatorVersion:- title: EvaluatorVersion- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - llm_as_judge- - $ref: '#/components/schemas/LlmAsJudgeEvaluatorVersion'- required:- - type- title: LlmAsJudgeEvaluatorVersion- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - code- - $ref: '#/components/schemas/CodeEvaluatorVersion'- required:- - type- title: CodeEvaluatorVersion- EvaluationRuleAssignment:- title: EvaluationRuleAssignment+ - mediaId+ - contentType+ - contentLength+ - url+ - urlExpiry+ DatasetRunItem:+ title: DatasetRunItemtype: object- description: Evaluation-rule assignment associated with this evaluator.properties:- evaluationRuleId:+ id:+ type: string+ datasetRunId:+ type: string+ datasetRunName:+ type: string+ datasetItemId:+ type: string+ traceId:+ type: string+ observationId:type: string- description: Stable identifier of the assigned evaluation rule.- variableMappingOverride:- type: array- items:- $ref: '#/components/schemas/PromptVariableMapping'nullable: true- description: >-- Rule-specific variable mapping override. Omitted when the- evaluator's latest default mapping is inherited. Legacy mappings use- the explicit `legacy` variant.+ description: The observation ID associated with this run item+ createdAt:+ type: string+ format: date-time+ updatedAt:+ type: string+ format: date-timerequired:- - evaluationRuleId- EvaluatorBase:- title: EvaluatorBase+ - id+ - datasetRunId+ - datasetRunName+ - datasetItemId+ - traceId+ - observationId+ - createdAt+ - updatedAt+ DatasetRun:+ title: DatasetRuntype: objectproperties:id:type: string- description: Stable identifier of this evaluator across all versions.+ description: Unique identifier of the dataset runname:type: string- description: >-- Human-readable evaluator name. Names are not identifiers and do not- need to be unique.+ description: Name of the dataset rundescription:type: stringnullable: true- description: Optional human-readable evaluator description.- createdBy:- $ref: '#/components/schemas/Creator'- nullable: true- description: >-- User who created this evaluator, or `null` when no user can be- resolved.- status:- $ref: '#/components/schemas/EvaluatorStatus'- description: >-- Effective evaluator status after Langfuse validates its runtime- configuration.- pausedAt:- type: string- format: date-time- nullable: true- description: Timestamp when the evaluator was paused, otherwise `null`.- pausedReason:+ description: Description of the run+ metadata:+ description: Metadata of the dataset run+ datasetId:type: string- nullable: true- description: Machine-readable reason when `status=paused`, otherwise `null`.- pausedMessage:+ description: Id of the associated dataset+ datasetName:type: string- nullable: true- description: Human-readable explanation when `status=paused`, otherwise `null`.- evaluationRuleAssignments:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleAssignment'- description: >-- All modern and legacy evaluation-rule assignments in- newest-assignment-first order. Rule-specific mappings are exposed as- `variableMappingOverride`; inherited defaults are omitted.+ description: Name of the associated datasetcreatedAt:type: stringformat: date-time- description: Timestamp when the evaluator was created.+ description: The date and time when the dataset run was createdupdatedAt:type: stringformat: date-time- description: Timestamp when the evaluator was last updated.- versionId:- type: string- description: Stable identifier of the latest evaluator version.- version:- type: integer- description: Monotonically increasing latest evaluator version number.- versionCreatedAt:- type: string- format: date-time- description: Timestamp when the latest evaluator version was created.- versionCreatedBy:- $ref: '#/components/schemas/Creator'- nullable: true- description: >-- User who created the latest evaluator version, or `null` when no- user can be resolved.+ description: The date and time when the dataset run was last updatedrequired:- id- name- description- - createdBy- - status- - pausedAt- - pausedReason- - pausedMessage- - evaluationRuleAssignments+ - metadata+ - datasetId+ - datasetName- createdAt- updatedAt- - versionId- - version- - versionCreatedAt- - versionCreatedBy- LlmAsJudgeEvaluator:- title: LlmAsJudgeEvaluator+ DatasetRunWithItems:+ title: DatasetRunWithItemstype: objectproperties:- type:- type: string- const: llm_as_judge- description: Evaluator type.- prompt:- $ref: '#/components/schemas/EvaluatorChatPrompt'- description: Ordered chat messages used by the latest evaluator version.- variables:- type: array- items:- type: string- description: >-- Variables extracted from the latest prompt and available for- evaluation-rule mappings.- variableMapping:+ datasetRunItems:type: arrayitems:- $ref: '#/components/schemas/PromptVariableMappingRead'- nullable: true- description: >-- Default variable mapping for the latest version, or `null` when no- default is configured.- modelConfig:- $ref: '#/components/schemas/EvaluatorModelConfig'+ $ref: '#/components/schemas/DatasetRunItem'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'nullable: true- description: >-- Explicit model configuration for the latest version, or `null` when- the project's default evaluation model is used.- outputDefinition:- $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'- description: Structured output schema returned by the latest evaluator version.required:- - type- - prompt- - variables- - variableMapping- - modelConfig- - outputDefinition+ - datasetRunItemsallOf:- - $ref: '#/components/schemas/EvaluatorBase'- CodeEvaluator:- title: CodeEvaluator+ - $ref: '#/components/schemas/DatasetRun'+ Model:+ title: Modeltype: object+ description: >-+ Model definition used for transforming usage into USD cost and/or+ tokenization.+++ Models can have either simple flat pricing or tiered pricing:++ - Flat pricing: Single price per usage type (legacy, but still+ supported)++ - Tiered pricing: Multiple pricing tiers with conditional matching based+ on usage patterns or observation attributes+++ The pricing tiers approach is recommended for models with usage-based+ pricing variations.++ When using tiered pricing, the flat price fields (inputPrice,+ outputPrice, prices) are populated++ from the default tier for backward compatibility.properties:- type:+ id:type: string- const: code- description: Evaluator type.- sourceCode:+ modelName:type: string- description: Source code executed by the latest evaluator version.- sourceCodeLanguage:- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'- description: Runtime language used to execute the latest source code.- required:- - type- - sourceCode- - sourceCodeLanguage- allOf:- - $ref: '#/components/schemas/EvaluatorBase'- Evaluator:- title: Evaluator- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - llm_as_judge- - $ref: '#/components/schemas/LlmAsJudgeEvaluator'- required:- - type- title: LlmAsJudgeEvaluator- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - code- - $ref: '#/components/schemas/CodeEvaluator'- required:- - type- title: CodeEvaluator- description: >-- One evaluator that can be used for scoring.+ description: >-+ Name of the model definition. If multiple with the same name exist,+ they are applied in the following order: (1) custom over built-in,+ (2) newest according to startTime where+ model.startTime<observation.startTime+ matchPattern:+ type: string+ description: >-+ Regex pattern which matches this model definition to+ generation.model. Useful in case of fine-tuned models. If you want+ to exact match, use `(?i)^modelname$`+ startDate:+ type: string+ format: date-time+ nullable: true+ description: Apply only to generations which are newer than this ISO date.+ unit:+ $ref: '#/components/schemas/ModelUsageUnit'+ nullable: true+ description: Unit used by this model.+ inputPrice:+ type: number+ format: double+ nullable: true+ description: Deprecated. See 'prices' instead. Price (USD) per input unit+ outputPrice:+ type: number+ format: double+ nullable: true+ description: Deprecated. See 'prices' instead. Price (USD) per output unit+ totalPrice:+ type: number+ format: double+ nullable: true+ description: >-+ Deprecated. See 'prices' instead. Price (USD) per total unit. Cannot+ be set if input or output price is set.+ tokenizerId:+ type: string+ nullable: true+ description: >-+ Optional. Tokenizer to be applied to observations which match to+ this model. See docs for more details.+ tokenizerConfig:+ description: >-+ Optional. Configuration for the selected tokenizer. Needs to be+ JSON. See docs for more details.+ isLangfuseManaged:+ type: boolean+ createdAt:+ type: string+ format: date-time+ description: Timestamp when the model was created+ prices:+ type: object+ additionalProperties:+ $ref: '#/components/schemas/ModelPrice'+ description: >-+ Deprecated. Use 'pricingTiers' instead for models with usage-based+ pricing variations.- An evaluator describes **how** to score data. Associated evaluation- rules describe **which** live objects should be evaluated.- EvaluatorStatus:- title: EvaluatorStatus- type: string- enum:- - active- - paused- description: >-- Effective evaluator runtime status.+ This field shows prices by usage type from the default pricing tier.+ Maintained for backward compatibility.++ If the model uses tiered pricing, this field will be populated from+ the default tier's prices.+ pricingTiers:+ type: array+ items:+ $ref: '#/components/schemas/PricingTier'+ description: >-+ Array of pricing tiers with conditional pricing based on usage+ thresholds or observation attributes.+++ Pricing tiers enable accurate cost tracking for models that charge+ different rates based on usage patterns++ or request attributes (e.g., high-volume usage, large context+ windows, cached tokens, or service tiers).+++ Each model must have exactly one default tier (isDefault=true,+ priority=0) that serves as a fallback.+ Additional conditional tiers can be defined with specific matching+ criteria.- - `active`: the evaluator can run.- - `paused`: Langfuse paused execution until the underlying issue is- resolved.- CreateLlmAsJudgeEvaluatorRequest:- title: CreateLlmAsJudgeEvaluatorRequest+ If this array is empty, the model uses legacy flat pricing from the+ inputPrice/outputPrice/totalPrice fields.+ required:+ - id+ - modelName+ - matchPattern+ - startDate+ - inputPrice+ - outputPrice+ - totalPrice+ - tokenizerId+ - tokenizerConfig+ - isLangfuseManaged+ - createdAt+ - prices+ - pricingTiers+ ModelPrice:+ title: ModelPricetype: objectproperties:- name:- type: string- description: >-- Human-readable evaluator name. Names are not identifiers and do not- need to be unique.- description:- type: string- nullable: true- description: Optional human-readable evaluator description.- type:- type: string- const: llm_as_judge- description: Evaluator type.- prompt:- $ref: '#/components/schemas/EvaluatorChatPromptInput'- description: User prompt string shortcut or an ordered list of chat messages.- modelConfig:- $ref: '#/components/schemas/EvaluatorModelConfig'- nullable: true- description: >-- Explicit model configuration. Set to `null` or omit to use the- project's default evaluation model.- variableMapping:- type: array- items:- $ref: '#/components/schemas/PromptVariableMappingInput'- nullable: true- description: >-- Default prompt-variable mapping, or `null` when no default is- configured.- outputDefinition:- $ref: '#/components/schemas/EvaluatorOutputDefinition'- description: >-- Structured output schema returned by this evaluator. Do not include- an internal persistence version.+ price:+ type: number+ format: doublerequired:- - name- - type- - prompt- - outputDefinition- CreateCodeEvaluatorRequest:- title: CreateCodeEvaluatorRequest+ - price+ PricingTierCondition:+ title: PricingTierCondition+ oneOf:+ - $ref: '#/components/schemas/PricingTierUsageCondition'+ - $ref: '#/components/schemas/PricingTierAttributeCondition'+ description: >-+ Condition for matching a pricing tier against usage details or+ observation attributes.+++ Usage-detail conditions treat usageDetailPattern as a regex, sum all+ matching usage values, and compare the sum to the numeric value.+ Model-parameter and metadata conditions match an exact top-level key+ against one or more string values.+ PricingTierUsageCondition:+ title: PricingTierUsageConditiontype: object+ description: Condition that sums usage details whose keys match a regex.properties:- name:- type: string- description: >-- Human-readable evaluator name. Names are not identifiers and do not- need to be unique.- description:+ usageDetailPattern:type: string- nullable: true- description: Optional human-readable evaluator description.- type:+ description: Regex pattern matched against usage detail keys.+ operator:+ $ref: '#/components/schemas/PricingTierOperator'+ value:+ type: number+ format: double+ description: Numeric threshold for the summed matching usage values.+ caseSensitive:+ type: boolean+ description: Whether the usage-detail regex is case-sensitive.+ required:+ - usageDetailPattern+ - operator+ - value+ - caseSensitive+ PricingTierAttributeCondition:+ title: PricingTierAttributeCondition+ type: object+ description: >-+ Condition that matches any configured value for a top-level observation+ attribute.+ properties:+ source:+ $ref: '#/components/schemas/PricingTierAttributeSource'+ description: Observation attribute object evaluated by this condition.+ key:type: string- const: code- description: Evaluator type.- sourceCode:+ description: Exact top-level attribute key.+ operator:type: string- description: Source code executed for each matched observation.- sourceCodeLanguage:- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'- description: Runtime language used to execute the source code.+ const: in+ description: Membership operator.+ values:+ type: array+ items:+ type: string+ description: Accepted string attribute values. At least one value is required.required:- - name- - type- - sourceCode- - sourceCodeLanguage- CreateEvaluatorRequest:- title: CreateEvaluatorRequest+ - source+ - key+ - operator+ - values+ PricingTierAttributeSource:+ title: PricingTierAttributeSource+ type: string+ enum:+ - model_parameters+ - metadata+ PricingTierConditionInput:+ title: PricingTierConditionInputoneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - llm_as_judge- - $ref: '#/components/schemas/CreateLlmAsJudgeEvaluatorRequest'- required:- - type- title: CreateLlmAsJudgeEvaluatorRequest- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - code- - $ref: '#/components/schemas/CreateCodeEvaluatorRequest'- required:- - type- title: CreateCodeEvaluatorRequest- description: >-- Initial evaluator definition with metadata and definition fields at the- same level. The returned evaluator starts at version `1`.- UpdateEvaluatorMetadataRequest:- title: UpdateEvaluatorMetadataRequest+ - $ref: '#/components/schemas/PricingTierUsageConditionInput'+ - $ref: '#/components/schemas/PricingTierAttributeCondition'+ description: Condition input for matching a pricing tier.+ PricingTierUsageConditionInput:+ title: PricingTierUsageConditionInputtype: object- description: Metadata-only update. At least one field must be provided.+ description: Input condition that sums usage details whose keys match a regex.properties:- name:- type: string- nullable: true- description: New human-readable evaluator name.- description:+ usageDetailPattern:type: string+ description: Regex pattern matched against usage detail keys.+ operator:+ $ref: '#/components/schemas/PricingTierOperator'+ value:+ type: number+ format: double+ description: Numeric threshold for the summed matching usage values.+ caseSensitive:+ type: booleannullable: true- description: New description. Set to `null` to clear it.- UpdateLlmAsJudgeEvaluatorRequest:- title: UpdateLlmAsJudgeEvaluatorRequest+ description: Whether the usage-detail regex is case-sensitive. Defaults to false.+ required:+ - usageDetailPattern+ - operator+ - value+ PricingTier:+ title: PricingTiertype: object+ description: >-+ Pricing tier definition with conditional pricing based on usage+ thresholds or observation attributes.+++ Pricing tiers enable accurate cost tracking for LLM providers that+ charge different rates based on usage patterns or request attributes.++ For example, some providers charge higher rates when context size+ exceeds certain thresholds or when an accelerated service tier is+ selected.+++ How tier matching works:++ 1. Tiers are evaluated in ascending priority order (priority 1 before+ priority 2, etc.)++ 2. The first tier where ALL conditions match is selected++ 3. If no conditional tiers match, the default tier is used as a fallback++ 4. The default tier has priority 0 and no conditions+++ Why priorities matter:++ - Lower priority numbers are evaluated first, allowing you to define+ specific cases before general ones++ - Example: Priority 1 for "high usage" (>200K tokens), Priority 2 for+ "medium usage" (>100K tokens), Priority 0 for default++ - Without proper ordering, a less specific condition might match before+ a more specific one+++ Every model must have exactly one default tier to ensure cost+ calculation always succeeds.properties:- name:- type: string- nullable: true- description: New human-readable evaluator name.- description:+ id:type: string- nullable: true- description: New description. Set to `null` to clear it.- type:+ description: Unique identifier for the pricing tier+ name:type: string- const: llm_as_judge- description: Evaluator type. The type of an existing evaluator cannot change.- prompt:- $ref: '#/components/schemas/EvaluatorChatPromptInput'description: >-- Complete replacement user prompt string or an ordered list of chat- messages.- modelConfig:- $ref: '#/components/schemas/EvaluatorModelConfig'- nullable: true+ Name of the pricing tier for display and identification purposes.+++ Examples: "Standard", "High Volume Tier", "Large Context", "Extended+ Context Tier"+ isDefault:+ type: booleandescription: >-- Explicit model configuration. Set to `null` or omit to use the- project's default evaluation model.- variableMapping:+ Whether this is the default tier. Every model must have exactly one+ default tier with priority 0 and no conditions.+++ The default tier serves as a fallback when no conditional tiers+ match, ensuring cost calculation always succeeds.++ It typically represents the base pricing for standard usage+ patterns.+ priority:+ type: integer+ description: >-+ Priority for tier matching evaluation. Lower numbers = higher+ priority (evaluated first).+++ The default tier must always have priority 0. Conditional tiers+ should have priority 1, 2, 3, etc.+++ Example ordering:++ - Priority 0: Default tier (no conditions, always matches as+ fallback)++ - Priority 1: High usage tier (e.g., >200K tokens)++ - Priority 2: Medium usage tier (e.g., >100K tokens)+++ This ensures more specific conditions are checked before general+ ones.+ conditions:type: arrayitems:- $ref: '#/components/schemas/PromptVariableMappingInput'- nullable: true+ $ref: '#/components/schemas/PricingTierCondition'description: >-- Complete replacement default variable mapping, or `null` when no- default is configured.- outputDefinition:- $ref: '#/components/schemas/EvaluatorOutputDefinition'+ Array of conditions that must ALL be met for this tier to match (AND+ logic).+++ The default tier must have an empty conditions array. Conditional+ tiers should have one or more conditions++ that define when this tier's pricing applies.+++ Multiple conditions enable complex matching scenarios (e.g., "high+ input tokens AND low output tokens").+ prices:+ type: object+ additionalProperties:+ type: number+ format: doubledescription: >-- Complete replacement output schema. Do not include an internal- persistence version.+ Prices (USD) by usage type for this tier.+++ Common usage types: "input", "output", "total", "request", "image"++ Prices are specified in USD per unit (e.g., per token, per request,+ per second).+++ Example: {"input": 0.000003, "output": 0.000015} means $3 per+ million input tokens and $15 per million output tokens.required:- - type- - prompt- - outputDefinition- UpdateCodeEvaluatorRequest:- title: UpdateCodeEvaluatorRequest+ - id+ - name+ - isDefault+ - priority+ - conditions+ - prices+ PricingTierInput:+ title: PricingTierInputtype: object+ description: >-+ Input schema for creating a pricing tier. The tier ID will be+ automatically generated server-side.+++ When creating a model with pricing tiers:++ - Exactly one tier must have isDefault=true (the fallback tier)++ - The default tier must have priority=0 and conditions=[]++ - All tier names and priorities must be unique within the model++ - Each tier must define at least one price+++ See PricingTier for detailed information about how tiers work and why+ they're useful.properties:name:type: string+ description: >-+ Name of the pricing tier for display and identification purposes.+++ Must be unique within the model. Common patterns: "Standard", "High+ Volume Tier", "Extended Context"+ isDefault:+ type: booleannullable: true- description: New human-readable evaluator name.- description:- type: string- nullable: true- description: New description. Set to `null` to clear it.- type:- type: string- const: code- description: Evaluator type. The type of an existing evaluator cannot change.- sourceCode:- type: string- description: Complete replacement source code.- sourceCodeLanguage:- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'- description: Runtime language used to execute the source code.+ description: >-+ Whether this is the default tier. Exactly one tier per model must be+ marked as default.+++ Requirements for default tier:++ - Must have isDefault=true++ - Must have priority=0++ - Must have empty conditions array (conditions=[])+++ The default tier acts as a fallback when no conditional tiers match.+ Defaults to false when omitted.+ priority:+ type: integer+ description: >-+ Priority for tier matching evaluation. Lower numbers = higher+ priority (evaluated first).+++ Must be unique within the model. The default tier must have+ priority=0.++ Conditional tiers should use priority 1, 2, 3, etc. based on their+ specificity.+ conditions:+ type: array+ items:+ $ref: '#/components/schemas/PricingTierConditionInput'+ description: >-+ Array of conditions that must ALL be met for this tier to match (AND+ logic).+++ The default tier must have an empty array (conditions=[]).++ Conditional tiers should define one or more conditions that specify+ when this tier's pricing applies.+++ Conditions can compare summed matching usage details to a numeric+ threshold, or exactly match a top-level model parameter or metadata+ value.+ prices:+ type: object+ additionalProperties:+ type: number+ format: double+ description: >-+ Prices (USD) by usage type for this tier. At least one price must be+ defined.+++ Common usage types: "input", "output", "total", "request", "image"++ Prices are in USD per unit (e.g., per token).+++ Example: {"input": 0.000003, "output": 0.000015} represents $3 per+ million input tokens and $15 per million output tokens.required:- - type- - sourceCode- - sourceCodeLanguage- UpdateEvaluatorRequest:- title: UpdateEvaluatorRequest+ - name+ - priority+ - conditions+ - prices+ PricingTierOperator:+ title: PricingTierOperator+ type: string+ enum:+ - gt+ - gte+ - lt+ - lte+ - eq+ - neq+ description: Comparison operators for pricing tier conditions+ ModelUsageUnit:+ title: ModelUsageUnit+ type: string+ enum:+ - CHARACTERS+ - TOKENS+ - MILLISECONDS+ - SECONDS+ - IMAGES+ - REQUESTS+ description: Unit of usage in Langfuse+ ObservationLevel:+ title: ObservationLevel+ type: string+ enum:+ - DEBUG+ - DEFAULT+ - WARNING+ - ERROR+ MapValue:+ title: MapValueoneOf:- - $ref: '#/components/schemas/UpdateEvaluatorMetadataRequest'- - $ref: '#/components/schemas/UpdateLlmAsJudgeEvaluatorRequest'- - $ref: '#/components/schemas/UpdateCodeEvaluatorRequest'- description: >-- Metadata-only update or complete flattened definition replacement. At- least one field must be provided.- EvaluatorsPage:- title: EvaluatorsPage- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/Evaluator'- description: Evaluators for this page.- meta:- $ref: '#/components/schemas/CursorMeta'- description: Cursor pagination metadata.- required:- - data- - meta- EvaluatorVersionsPage:- title: EvaluatorVersionsPage- type: object- properties:- data:- type: array+ - type: string+ nullable: true+ title: String+ - type: integer+ nullable: true+ title: Integer+ - type: number+ format: float+ nullable: true+ title: FloatNumber+ - type: boolean+ nullable: true+ title: Boolean+ - type: arrayitems:- $ref: '#/components/schemas/EvaluatorVersion'- description: Evaluator versions for this page in newest-first order.- meta:- $ref: '#/components/schemas/CursorMeta'- description: Cursor pagination metadata.- required:- - data- - meta- CursorMeta:- title: CursorMeta- type: object- properties:- cursor:- type: string+ type: stringnullable: true- description: Opaque cursor for the next page. Omitted when there is no next page.- DeletedEvaluator:- title: DeletedEvaluator+ title: Array+ CommentObjectType:+ title: CommentObjectType+ type: string+ enum:+ - TRACE+ - OBSERVATION+ - SESSION+ - PROMPT+ DatasetStatus:+ title: DatasetStatus+ type: string+ enum:+ - ACTIVE+ - ARCHIVED+ ScoreSource:+ title: ScoreSource+ type: string+ enum:+ - ANNOTATION+ - API+ - EVAL+ ScoreConfigDataType:+ title: ScoreConfigDataType+ type: string+ enum:+ - NUMERIC+ - BOOLEAN+ - CATEGORICAL+ - TEXT+ ScoreDataType:+ title: ScoreDataType+ type: string+ enum:+ - NUMERIC+ - BOOLEAN+ - CATEGORICAL+ - CORRECTION+ - TEXT+ DeleteDatasetItemResponse:+ title: DeleteDatasetItemResponsetype: object- description: Confirmation returned after successful evaluator deletion.properties:- id:+ message:type: string- description: Identifier of the deleted evaluator.- required:- - id- ExperimentsResponse:- title: ExperimentsResponse- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/Experiment'- meta:- $ref: '#/components/schemas/ExperimentsResponseMeta'+ description: Success message after deletionrequired:- - data- - meta- ExperimentsResponseMeta:- title: ExperimentsResponseMeta+ - message+ CreateDatasetItemRequest:+ title: CreateDatasetItemRequesttype: objectproperties:- cursor:+ datasetName:type: string+ input:nullable: true- description: >-- Versioned base64url cursor for retrieving the next page. Absent when- there are no more results.- Experiment:- title: Experiment- type: object- properties:- id:- type: string- name:- type: string- description:- type: string+ expectedOutput:nullable: true- startTime:- type: string- format: date-time- description: |-- Start of the experiment, i.e. the earliest event within the- requested time range. Clipped to `fromStartTime` when the- experiment started before the requested range.- endTime:- type: string- format: date-time- description: |-- End of the experiment, i.e. the latest event end within the- requested time range.- itemCount:- type: integer- description: Number of experiment items within the requested time range.- datasetId:+ metadata:+ nullable: true+ sourceTraceId:type: stringnullable: true- description: Null when the experiment is not associated with a dataset.- metadata:- type: object- additionalProperties: true+ sourceObservationId:+ type: stringnullable: true- description: Included only when `fields=metadata` is requested.- scores:- type: array- items:- $ref: '#/components/schemas/ScoreV3'+ id:+ type: stringnullable: truedescription: >-- Included only when `fields=scores` is requested. Contains scores- directly attached to the experiment.+ Dataset items are upserted on their id. Id needs to be unique+ (project-level), cannot be reused across datasets, and must be at+ most 255 characters.+ status:+ $ref: '#/components/schemas/DatasetStatus'+ nullable: true+ description: Defaults to ACTIVE for newly created itemsrequired:- - id- - name- - description- - startTime- - endTime- - itemCount- - datasetId- ExperimentItemsResponse:- title: ExperimentItemsResponse+ - datasetName+ PaginatedDatasetItems:+ title: PaginatedDatasetItemstype: objectproperties:data:type: arrayitems:- $ref: '#/components/schemas/ExperimentItem'+ $ref: '#/components/schemas/DatasetItem'meta:- $ref: '#/components/schemas/ExperimentsResponseMeta'+ $ref: '#/components/schemas/utilsMetaResponse'required:- data- meta- ExperimentItem:- title: ExperimentItem+ CreateDatasetRunItemRequest:+ title: CreateDatasetRunItemRequesttype: objectproperties:- id:- type: string- traceId:- type: string- startTime:- type: string- format: date-time- endTime:- type: string- format: date-time- nullable: true- level:- $ref: '#/components/schemas/ObservationLevel'- environment:- type: string- experimentId:- type: string- experimentName:- type: string- experimentItemId:- type: string- experimentDatasetId:+ runName:type: string- nullable: true- description: Included when `fields=dataset` is requested.- experimentItemVersion:+ runDescription:type: string- format: date-time- nullable: true- description: Included when `fields=dataset` is requested.- input:- nullable: true- description: Included when `fields=io` is requested.- output:- nullable: true- description: Included when `fields=io` is requested.- expectedOutput:nullable: true- description: Included when `fields=io` is requested.+ description: Description of the run. If run exists, description will be updated.metadata:- type: object- additionalProperties: truenullable: true- description: Included when `fields=metadata` is requested.- experimentItemMetadata:- type: object- additionalProperties: true+ description: Metadata of the dataset run, updates run if run already exists+ datasetItemId:+ type: string+ observationId:+ type: stringnullable: true- description: Included when `fields=itemMetadata` is requested.- experimentMetadata:- type: object- additionalProperties: true+ traceId:+ type: stringnullable: true- description: Included when `fields=experimentMetadata` is requested.- experimentDescription:+ description: >-+ traceId should always be provided. For compatibility with older SDK+ versions it can also be inferred from the provided observationId.+ datasetVersion:type: string+ format: date-timenullable: true- description: Included when `fields=experimentMetadata` is requested.- scores:+ description: >-+ ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g.,+ "2026-01-21T14:35:42Z").++ Specifies the dataset version to use for this experiment run.++ If provided, the experiment will use dataset items as they existed+ at or before this timestamp.++ If not provided, uses the latest version of dataset items.+ createdAt:+ type: string+ format: date-time+ nullable: true+ description: >-+ Optional timestamp to set the createdAt field of the dataset run+ item. If not provided or null, defaults to current timestamp.+ required:+ - runName+ - datasetItemId+ PaginatedDatasetRunItems:+ title: PaginatedDatasetRunItems+ type: object+ properties:+ data:type: arrayitems:- $ref: '#/components/schemas/ScoreV3'+ $ref: '#/components/schemas/DatasetRunItem'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'nullable: true- description: >-- Included only when `fields=scores` is requested. Contains item and- trace scores only; experiment-level scores are returned by the- experiments endpoint.required:- - id- - traceId- - startTime- - endTime- - level- - environment- - experimentId- - experimentName- - experimentItemId- FeedbackTargetType:- title: FeedbackTargetType- type: string- enum:- - skill- - mcp-tool- - cli- - docs- - public-api- - other- SubmitFeedbackRequest:- title: SubmitFeedbackRequest+ - data+ - meta+ PaginatedDatasets:+ title: PaginatedDatasetstype: objectproperties:- targetType:- $ref: '#/components/schemas/FeedbackTargetType'- description: Category of the thing the feedback is about.- target:- type: string- description: >-- The specific instance within targetType: the skill name, MCP tool- name, CLI command, API endpoint path, or docs page path (e.g.- 'queryMetrics', '/docs/mcp'). An identifier, not a sentence. Must be- between 1 and 200 characters.- feedback:+ data:+ type: array+ items:+ $ref: '#/components/schemas/Dataset'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ required:+ - data+ - meta+ CreateDatasetRequest:+ title: CreateDatasetRequest+ type: object+ properties:+ name:type: string- description: >-- Concise feedback text approved by the user. Must be between 1 and- 3000 characters.- goal:+ description:type: stringnullable: true+ metadata:+ nullable: true+ inputSchema:+ nullable: truedescription: >-- Optional user-approved goal or use case they were trying to achieve.- Must be between 1 and 1500 characters when provided. Do not include- secrets, customer data, trace payloads, or broad unrelated context.- referenceUrl:- type: string+ JSON Schema for validating dataset item inputs. When set, all new+ and existing dataset items will be validated against this schema.+ expectedOutputSchema:nullable: truedescription: >-- Optional HTTP(S) reference URL. Langfuse stores it as text for- triage and does not fetch it.+ JSON Schema for validating dataset item expected outputs. When set,+ all new and existing dataset items will be validated against this+ schema.required:- - targetType- - target- - feedback- SubmitFeedbackResponse:- title: SubmitFeedbackResponse+ - name+ PaginatedDatasetRuns:+ title: PaginatedDatasetRunstype: objectproperties:- id:- type: string- description: Correlation ID for the submitted feedback.+ data:+ type: array+ items:+ $ref: '#/components/schemas/DatasetRun'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: truerequired:- - id- HealthResponse:- title: HealthResponse+ - data+ - meta+ DeleteDatasetRunResponse:+ title: DeleteDatasetRunResponsetype: objectproperties:- version:- type: string- description: Langfuse server version- status:+ message:type: stringrequired:- - version- - status- IngestionEvent:- title: IngestionEvent- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - trace-create- - $ref: '#/components/schemas/TraceEvent'- required:- - type- title: TraceEvent- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - score-create- - $ref: '#/components/schemas/ScoreEvent'- required:- - type- title: ScoreEvent- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - span-create- - $ref: '#/components/schemas/CreateSpanEvent'- required:- - type- title: CreateSpanEvent- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - span-update- - $ref: '#/components/schemas/UpdateSpanEvent'- required:- - type- title: UpdateSpanEvent- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - generation-create- - $ref: '#/components/schemas/CreateGenerationEvent'- required:- - type- title: CreateGenerationEvent- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - generation-update- - $ref: '#/components/schemas/UpdateGenerationEvent'- required:- - type- title: UpdateGenerationEvent- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - event-create- - $ref: '#/components/schemas/CreateEventEvent'- required:- - type- title: CreateEventEvent+ - message+ EvaluatorType:+ title: EvaluatorType+ type: string+ enum:+ - llm_as_judge+ - code+ description: |-+ The evaluator type.++ The public API supports LLM-as-a-judge and code evaluators.+ CodeEvaluatorSourceCodeLanguage:+ title: CodeEvaluatorSourceCodeLanguage+ type: string+ enum:+ - PYTHON+ - TYPESCRIPT+ description: Code evaluator runtime language.+ PromptVariableMappingSource:+ title: PromptVariableMappingSource+ type: string+ enum:+ - input+ - output+ - metadata+ - tool_calls+ - expected_output+ - experiment_item_metadata+ description: >-+ Source field used to populate a prompt variable.+++ Use these values when mapping evaluator prompt variables to live data.+++ Source semantics:++ - `input`: the observation input payload++ - `output`: the observation output payload++ - `metadata`: the observation metadata object. Combine with `jsonPath`+ when you need one nested field instead of the whole object.++ - `tool_calls`: the tool calls recorded on the observation, as an array+ of `{id, name, arguments, type, index}` objects in the order the model+ emitted them. Combine with `jsonPath` (for example `$[*].name`) to+ select parts of each call.++ - `expected_output`: the experiment item's expected output when the+ observation belongs to an experiment.++ - `experiment_item_metadata`: the experiment item's metadata when the+ observation belongs to an experiment.+ EvaluatorOutputScoreType:+ title: EvaluatorOutputScoreType+ type: string+ enum:+ - NUMERIC+ - BOOLEAN+ - CATEGORICAL+ description: >-+ Structured score type returned by an evaluator.+++ This controls the type of score value Langfuse stores for evaluation+ results:++ - `NUMERIC`: a numeric score such as `0.82`++ - `BOOLEAN`: a boolean score such as `true`++ - `CATEGORICAL`: one or more category labels from a fixed list+ EvaluatorOutputDefinition:+ title: EvaluatorOutputDefinition+ oneOf:- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - sdk-log- - $ref: '#/components/schemas/SDKLogEvent'+ - NUMERIC+ - $ref: '#/components/schemas/PublicEvaluatorNumericScore'required:- - type- title: SDKLogEvent+ - dataType+ title: PublicEvaluatorNumericScore- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - observation-create- - $ref: '#/components/schemas/CreateObservationEvent'+ - BOOLEAN+ - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'required:- - type- title: CreateObservationEvent+ - dataType+ title: PublicEvaluatorBooleanScore- type: objectallOf:- type: objectproperties:- type:+ dataType:type: stringenum:- - observation-update- - $ref: '#/components/schemas/UpdateObservationEvent'+ - CATEGORICAL+ - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'required:- - type- title: UpdateObservationEvent- ObservationType:- title: ObservationType- type: string- enum:- - SPAN- - GENERATION- - EVENT- - AGENT- - TOOL- - CHAIN- - RETRIEVER- - EVALUATOR- - EMBEDDING- - GUARDRAIL- IngestionUsage:- title: IngestionUsage- oneOf:- - $ref: '#/components/schemas/Usage'- - $ref: '#/components/schemas/OpenAIUsage'- OpenAIUsage:- title: OpenAIUsage- type: object- description: Usage interface of OpenAI for improved compatibility.- properties:- promptTokens:- type: integer- nullable: true- completionTokens:- type: integer- nullable: true- totalTokens:- type: integer- nullable: true- OptionalObservationBody:- title: OptionalObservationBody- type: object- properties:- traceId:- type: string- nullable: true- name:- type: string- nullable: true- startTime:- type: string- format: date-time- nullable: true- metadata:- nullable: true- input:- nullable: true- output:- nullable: true- level:- $ref: '#/components/schemas/ObservationLevel'- nullable: true- statusMessage:- type: string- nullable: true- parentObservationId:- type: string- nullable: true- version:- type: string- nullable: true- environment:- type: string- nullable: true- CreateEventBody:- title: CreateEventBody- type: object- properties:- id:- type: string- nullable: true- allOf:- - $ref: '#/components/schemas/OptionalObservationBody'- UpdateEventBody:- title: UpdateEventBody- type: object- properties:- id:- type: string- required:- - id- allOf:- - $ref: '#/components/schemas/OptionalObservationBody'- CreateSpanBody:- title: CreateSpanBody- type: object- properties:- endTime:- type: string- format: date-time- nullable: true- allOf:- - $ref: '#/components/schemas/CreateEventBody'- UpdateSpanBody:- title: UpdateSpanBody- type: object- properties:- endTime:- type: string- format: date-time- nullable: true- allOf:- - $ref: '#/components/schemas/UpdateEventBody'- CreateGenerationBody:- title: CreateGenerationBody- type: object- properties:- completionStartTime:- type: string- format: date-time- nullable: true- model:- type: string- nullable: true- modelParameters:- type: object- additionalProperties:- $ref: '#/components/schemas/MapValue'- nullable: true- usage:- $ref: '#/components/schemas/IngestionUsage'- nullable: true- usageDetails:- $ref: '#/components/schemas/UsageDetails'- nullable: true- costDetails:- type: object- additionalProperties:- type: number- format: double- nullable: true- promptName:- type: string- nullable: true- promptVersion:- type: integer- nullable: true- allOf:- - $ref: '#/components/schemas/CreateSpanBody'- UpdateGenerationBody:- title: UpdateGenerationBody+ - dataType+ title: PublicEvaluatorCategoricalScore+ description: >-+ Flat structured output definition used when creating or updating an+ evaluator.+++ - `dataType` is required.++ - `scoreReasoningInstructions` and `scoreValueInstructions` are optional+ instructions.++ - `minValue` and `maxValue` apply only to `NUMERIC` outputs. If both are+ set, `minValue` must not exceed `maxValue`.++ - `categories` and `shouldAllowMultipleMatches` apply only to+ `CATEGORICAL` outputs.++ - Do not send `version`; that is an internal storage detail.+ EvaluatorOutputDefinitionBase:+ title: EvaluatorOutputDefinitionBasetype: objectproperties:- completionStartTime:+ scoreReasoningInstructions:type: string- format: date-timenullable: true- model:+ description: >-+ Optional instructions for deriving the reasoning returned with the+ score.+ scoreValueInstructions:type: stringnullable: true- modelParameters:- type: object- additionalProperties:- $ref: '#/components/schemas/MapValue'- nullable: true- usage:- $ref: '#/components/schemas/IngestionUsage'- nullable: true- promptName:+ description: Optional instructions for deriving the score value.+ PublicEvaluatorNumericScore:+ title: PublicEvaluatorNumericScore+ type: object+ properties:+ dataType:type: string+ const: NUMERIC+ description: Numeric score output.+ minValue:+ type: number+ format: doublenullable: true- usageDetails:- $ref: '#/components/schemas/UsageDetails'- nullable: true- costDetails:- type: object- additionalProperties:- type: number- format: double- nullable: true- promptVersion:- type: integer+ description: Optional inclusive minimum value.+ maxValue:+ type: number+ format: doublenullable: true+ description: Optional inclusive maximum value.+ required:+ - dataTypeallOf:- - $ref: '#/components/schemas/UpdateSpanBody'- ObservationBody:- title: ObservationBody+ - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'+ PublicEvaluatorBooleanScore:+ title: PublicEvaluatorBooleanScoretype: objectproperties:- id:- type: string- nullable: true- traceId:- type: string- nullable: true- type:- $ref: '#/components/schemas/ObservationType'- name:+ dataType:type: string- nullable: true- startTime:+ const: BOOLEAN+ description: Boolean score output.+ required:+ - dataType+ allOf:+ - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'+ PublicEvaluatorCategoricalScore:+ title: PublicEvaluatorCategoricalScore+ type: object+ properties:+ dataType:type: string- format: date-time- nullable: true- endTime:+ const: CATEGORICAL+ description: Categorical score output.+ categories:+ type: array+ items:+ type: string+ description: Allowed category values. At least two unique values are required.+ shouldAllowMultipleMatches:+ type: boolean+ description: Whether the evaluator may return more than one category.+ required:+ - dataType+ - categories+ - shouldAllowMultipleMatches+ allOf:+ - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'+ PublicEvaluatorOutputDefinition:+ title: PublicEvaluatorOutputDefinition+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - NUMERIC+ - $ref: '#/components/schemas/PublicEvaluatorNumericScore'+ required:+ - dataType+ title: PublicEvaluatorNumericScore+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - BOOLEAN+ - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'+ required:+ - dataType+ title: PublicEvaluatorBooleanScore+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CATEGORICAL+ - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'+ required:+ - dataType+ title: PublicEvaluatorCategoricalScore+ description: >-+ Flat evaluator output definition returned by the public API.+++ This response always includes `dataType` and never includes an internal+ output-definition `version`.++ Optional empty descriptions from legacy definitions are omitted.+ EvaluationRuleStringFilterOperator:+ title: EvaluationRuleStringFilterOperator+ type: string+ enum:+ - '='+ - contains+ - does not contain+ - starts with+ - ends with+ EvaluationRuleNumberFilterOperator:+ title: EvaluationRuleNumberFilterOperator+ type: string+ enum:+ - '='+ - '>'+ - <+ - '>='+ - <=+ EvaluationRuleOptionsFilterOperator:+ title: EvaluationRuleOptionsFilterOperator+ type: string+ enum:+ - any of+ - none of+ EvaluationRuleArrayOptionsFilterOperator:+ title: EvaluationRuleArrayOptionsFilterOperator+ type: string+ enum:+ - any of+ - none of+ - all of+ EvaluationRuleBooleanFilterOperator:+ title: EvaluationRuleBooleanFilterOperator+ type: string+ enum:+ - '='+ - <>+ EvaluationRuleNullFilterOperator:+ title: EvaluationRuleNullFilterOperator+ type: string+ enum:+ - is null+ - is not null+ DateTimeEvaluationRuleFilter:+ title: DateTimeEvaluationRuleFilter+ type: object+ properties:+ column:type: string- format: date-time- nullable: true- completionStartTime:+ description: Column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'+ description: Comparison operator for datetime values.+ value:type: stringformat: date-time- nullable: true- model:+ description: Datetime value to compare against.+ required:+ - column+ - operator+ - value+ StringEvaluationRuleFilter:+ title: StringEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true- modelParameters:- type: object- additionalProperties:- $ref: '#/components/schemas/MapValue'- nullable: true- input:- nullable: true- version:+ description: Column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'+ value:type: string- nullable: true- metadata:- nullable: true- output:- nullable: true- usage:- $ref: '#/components/schemas/Usage'- nullable: true- level:- $ref: '#/components/schemas/ObservationLevel'- nullable: true- statusMessage:+ required:+ - column+ - operator+ - value+ NumberEvaluationRuleFilter:+ title: NumberEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true- parentObservationId:+ description: Column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'+ value:+ type: number+ format: double+ required:+ - column+ - operator+ - value+ StringOptionsEvaluationRuleFilter:+ title: StringOptionsEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true- environment:+ description: Column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'+ value:+ type: array+ items:+ type: string+ description: One or more allowed string values.+ required:+ - column+ - operator+ - value+ ArrayOptionsEvaluationRuleFilter:+ title: ArrayOptionsEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true+ description: Column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleArrayOptionsFilterOperator'+ value:+ type: array+ items:+ type: string+ description: One or more array elements to match.required:- - type- TraceBody:- title: TraceBody+ - column+ - operator+ - value+ StringObjectEvaluationRuleFilter:+ title: StringObjectEvaluationRuleFiltertype: objectproperties:- id:+ column:type: string- nullable: true- timestamp:+ description: >-+ Object-valued column to filter on. Currently only `metadata` is+ supported.+ key:type: string- format: date-time- nullable: true- name:+ description: Top-level key inside the object-valued column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'+ value:type: string- nullable: true- userId:+ required:+ - column+ - key+ - operator+ - value+ NumberObjectEvaluationRuleFilter:+ title: NumberObjectEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true- input:- nullable: true- output:- nullable: true- sessionId:+ description: Object-valued column to filter on.+ key:type: string- nullable: true- release:+ description: Key inside the object-valued column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'+ value:+ type: number+ format: double+ required:+ - column+ - key+ - operator+ - value+ CategoryOptionsEvaluationRuleFilter:+ title: CategoryOptionsEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true- version:+ description: Object-valued column to filter on.+ key:type: string- nullable: true- metadata:- nullable: true- tags:+ description: Key inside the object-valued column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'+ value:type: arrayitems:type: string- nullable: true- environment:+ required:+ - column+ - key+ - operator+ - value+ BooleanEvaluationRuleFilter:+ title: BooleanEvaluationRuleFilter+ type: object+ properties:+ column:type: string- nullable: true- public:+ description: Column to filter on.+ operator:+ $ref: '#/components/schemas/EvaluationRuleBooleanFilterOperator'+ value:type: boolean- nullable: true- description: Make trace publicly accessible via url- SDKLogBody:- title: SDKLogBody+ required:+ - column+ - operator+ - value+ NullEvaluationRuleFilter:+ title: NullEvaluationRuleFiltertype: objectproperties:- log: {}+ column:+ type: string+ description: Column to filter on, for example `parentObservationId`.+ operator:+ $ref: '#/components/schemas/EvaluationRuleNullFilterOperator'+ value:+ type: string+ const: ''+ description: >-+ Required empty-string placeholder used by the runtime filter+ contract.required:- - log- ScoreBody:- title: ScoreBody+ - column+ - operator+ - value+ PromptVariableMappingInput:+ title: PromptVariableMappingInputtype: object+ description: >-+ Connects one prompt variable to data from an observation or experiment.+++ Manual mappings are used for `llm_as_judge` evaluators. `code`+ evaluators use a fixed runtime mapping managed by Langfuse.+++ How to build a valid mapping list:++ 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.++ 2. Read the evaluator `variables` array.++ 3. Add exactly one mapping object for each variable in that array.++ 4. Use the variable name exactly as returned, without braces such as+ `{{` or `}}`.++ 5. Choose the source field that should populate the variable.+++ `jsonPath` is optional. Use it only when the selected source is a JSON+ object and you want to extract one nested field before inserting it into+ the evaluator prompt.+++ Invalid, missing, or duplicate mappings return a validation error.+ Malformed JSONPath expressions are also rejected.properties:- id:+ variable:type: string- nullable: true- traceId:+ description: >-+ Prompt variable name without braces.+++ Example: for the prompt `Judge {{input}} against {{output}}`, use+ `input` and `output`.+ source:+ $ref: '#/components/schemas/PromptVariableMappingSource'+ description: >-+ Source field that should populate the prompt variable.+++ Available sources are `input`, `output`, `metadata`, `tool_calls`,+ `expected_output`, and `experiment_item_metadata`.+ jsonPath:type: stringnullable: true- sessionId:+ description: >-+ Optional JSONPath selector applied to the selected source before it+ is passed to the evaluator prompt.+++ Requirements:++ - Must start with `$`++ - Must be a syntactically valid JSONPath expression++ - Most useful with `source=metadata`+ required:+ - variable+ - source+ PromptVariableMappingRead:+ title: PromptVariableMappingRead+ type: object+ description: |-+ Connects one prompt variable to source data.++ `source` is `null` when the mapping is incomplete.+ properties:+ variable:type: string- nullable: true- observationId:+ description: Prompt variable name without braces.+ source:type: stringnullable: true- datasetRunId:+ description: >-+ Stored source field populating the variable, or `null` when the+ mapping is incomplete.+ jsonPath:type: stringnullable: true- name:+ description: Optional JSONPath selector applied to the selected source.+ required:+ - variable+ - source+ PromptVariableMapping:+ title: PromptVariableMapping+ oneOf:+ - $ref: '#/components/schemas/PromptVariableMappingRead'+ - $ref: '#/components/schemas/LegacyPromptVariableMapping'+ description: Connects one prompt variable to source data.+ LegacyPromptVariableMapping:+ title: LegacyPromptVariableMapping+ type: object+ description: >-+ **Deprecated:** Connects one prompt variable to data from a legacy trace+ or dataset evaluation rule.+++ `langfuseObject` selects the object kind. `objectName` separately+ selects a named observation and is `null` for trace and dataset-item+ mappings.+ properties:+ mappingType:type: string- description: >-- The name of the score. Always overrides "output" for correction- scores.- environment:+ const: legacy+ description: Explicitly marks this as a legacy mapping.+ variable:type: string- nullable: true- queueId:+ description: Prompt variable name without braces.+ langfuseObject:+ $ref: '#/components/schemas/LegacyEvaluationObject'+ description: Legacy object kind selected as the mapping source.+ objectName:type: stringnullable: truedescription: >-- The annotation queue referenced by the score. Indicates if score was- initially created while processing annotation queue.- value:- $ref: '#/components/schemas/CreateScoreValue'- description: >-- The value of the score. Must be passed as string for categorical and- text scores, and numeric for boolean and numeric scores. Boolean- score values must equal either 1 or 0 (true or false). Text score- values must be between 1 and 500 characters.- comment:+ Observation name to match, or `null` when `langfuseObject` is+ `trace` or `dataset_item`.+ source:type: string- nullable: true- metadata:- nullable: true- dataType:- $ref: '#/components/schemas/ScoreDataType'- nullable: true- description: >-- When set, must match the score value's type. If not set, will be- inferred from the score value or config- configId:+ description: Field selected from the legacy object.+ jsonPath:type: stringnullable: true- description: >-- Reference a score config on a score. When set, the score name must- equal the config name and scores must comply with the config's range- and data type. For categorical scores, the value must map to a- config category. Numeric scores might be constrained by the score- config's max and min values+ description: Optional JSONPath selector applied to the selected field.required:- - name- - value- BaseEvent:- title: BaseEvent+ - mappingType+ - variable+ - langfuseObject+ - objectName+ - source+ LegacyEvaluationObject:+ title: LegacyEvaluationObject+ type: string+ enum:+ - trace+ - span+ - generation+ - event+ - agent+ - tool+ - chain+ - retriever+ - evaluator+ - embedding+ - guardrail+ - dataset_item+ description: >-+ **Deprecated:** Legacy Langfuse object kind used by trace and dataset+ evaluation rules.+ EvaluationRuleFilter:+ title: EvaluationRuleFilter+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - datetime+ - $ref: '#/components/schemas/DateTimeEvaluationRuleFilter'+ required:+ - type+ title: DateTimeEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - string+ - $ref: '#/components/schemas/StringEvaluationRuleFilter'+ required:+ - type+ title: StringEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - number+ - $ref: '#/components/schemas/NumberEvaluationRuleFilter'+ required:+ - type+ title: NumberEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - stringOptions+ - $ref: '#/components/schemas/StringOptionsEvaluationRuleFilter'+ required:+ - type+ title: StringOptionsEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - categoryOptions+ - $ref: '#/components/schemas/CategoryOptionsEvaluationRuleFilter'+ required:+ - type+ title: CategoryOptionsEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - arrayOptions+ - $ref: '#/components/schemas/ArrayOptionsEvaluationRuleFilter'+ required:+ - type+ title: ArrayOptionsEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - stringObject+ - $ref: '#/components/schemas/StringObjectEvaluationRuleFilter'+ required:+ - type+ title: StringObjectEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - numberObject+ - $ref: '#/components/schemas/NumberObjectEvaluationRuleFilter'+ required:+ - type+ title: NumberObjectEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - boolean+ - $ref: '#/components/schemas/BooleanEvaluationRuleFilter'+ required:+ - type+ title: BooleanEvaluationRuleFilter+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - 'null'+ - $ref: '#/components/schemas/NullEvaluationRuleFilter'+ required:+ - type+ title: NullEvaluationRuleFilter+ description: >-+ One filter condition used to decide whether a live-ingested observation+ should be evaluated.+++ All filters must be satisfied for the rule to run. Experiment scope is+ expressed with filters rather than a separate target field:++ - `isExperimentItemRootSpan = true` limits execution to experiment item+ roots.++ - `datasetId` limits execution to experiments for the selected datasets.+ Use dataset IDs from `GET /api/public/v2/datasets`.+++ Pick the filter `type` first, as it determines the required fields and+ value shape. Use `key` only for object filters such as `metadata`.+++ Operator quick reference by filter `type`:++ - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends+ with`++ - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`++ - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`++ - `stringOptions`: `any of`, `none of`++ - `arrayOptions`: `any of`, `none of`, `all of`++ - `stringObject`: same operators as `string`++ - `boolean`: `"="`, `"<>"`++ - `null`: `is null`, `is not null`+++ Supported columns. Each column accepts exactly one filter `type`;+ sending a column with a different `type` is rejected with `400`.++ - `type`: `stringOptions`, values `SPAN`, `EVENT`, `GENERATION`,+ `AGENT`, `TOOL`, `CHAIN`, `RETRIEVER`, `EVALUATOR`, `EMBEDDING`,+ `GUARDRAIL`++ - `name`: `stringOptions`++ - `environment`: `stringOptions`++ - `level`: `stringOptions`, values `DEBUG`, `DEFAULT`, `WARNING`,+ `ERROR`++ - `version`: `string`++ - `traceName`: `stringOptions`++ - `userId`: `string`++ - `sessionId`: `string`++ - `tags`: `arrayOptions`++ - `metadata`: `stringObject`, requires `key`++ - `isRootObservation`: `boolean`; true when the observation has no+ parent or is explicitly marked as an application root++ - `parentObservationId`: `null`++ - `experimentId`: `stringOptions`++ - `isExperimentItemRootSpan`: `boolean`++ - `calledToolNames`: `arrayOptions`++ - `toolCalls`: `number`, the number of tool calls on the observation++ - `datasetId`: `stringOptions`. Use dataset `id` values from `GET+ /api/public/v2/datasets`, not dataset names.+ EvaluationRuleReadFilterBase:+ title: EvaluationRuleReadFilterBasetype: object+ description: Stored filter returned verbatim by evaluation-rule read endpoints.properties:- id:- type: string- description: UUID v4 that identifies the event- timestamp:+ type:type: stringdescription: >-- Datetime (ISO 8601) of event creation in client. Should be as close- to actual event creation in client as possible, this timestamp will- be used for ordering of events in future release. Resolution:- milliseconds (required), microseconds (optimal).- metadata:+ Stored filter type. This is not broken down into separate public+ response types.+ column:+ type: string+ operator:+ type: string+ value:nullable: true- description: Optional. Metadata field used by the Langfuse SDKs for debugging.- required:- - id- - timestamp- TraceEvent:- title: TraceEvent- type: object- properties:- body:- $ref: '#/components/schemas/TraceBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- CreateObservationEvent:- title: CreateObservationEvent- type: object- properties:- body:- $ref: '#/components/schemas/ObservationBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- UpdateObservationEvent:- title: UpdateObservationEvent- type: object- properties:- body:- $ref: '#/components/schemas/ObservationBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- ScoreEvent:- title: ScoreEvent- type: object- properties:- body:- $ref: '#/components/schemas/ScoreBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- SDKLogEvent:- title: SDKLogEvent- type: object- properties:- body:- $ref: '#/components/schemas/SDKLogBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- CreateGenerationEvent:- title: CreateGenerationEvent- type: object- properties:- body:- $ref: '#/components/schemas/CreateGenerationBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- UpdateGenerationEvent:- title: UpdateGenerationEvent- type: object- properties:- body:- $ref: '#/components/schemas/UpdateGenerationBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- CreateSpanEvent:- title: CreateSpanEvent- type: object- properties:- body:- $ref: '#/components/schemas/CreateSpanBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- UpdateSpanEvent:- title: UpdateSpanEvent- type: object- properties:- body:- $ref: '#/components/schemas/UpdateSpanBody'- required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- CreateEventEvent:- title: CreateEventEvent- type: object- properties:- body:- $ref: '#/components/schemas/CreateEventBody'+ description: Stored filter value. Its shape depends on the filter type.required:- - body- allOf:- - $ref: '#/components/schemas/BaseEvent'- IngestionSuccess:- title: IngestionSuccess+ - type+ - column+ - operator+ EvaluationRuleReadFilterWithKey:+ title: EvaluationRuleReadFilterWithKeytype: objectproperties:- id:+ key:type: string- status:- type: integer+ description: Stored object or legacy filter key.required:- - id- - status- IngestionError:- title: IngestionError+ - key+ allOf:+ - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'+ EvaluationRuleReadFilter:+ title: EvaluationRuleReadFilter+ oneOf:+ - $ref: '#/components/schemas/EvaluationRuleReadFilterWithKey'+ - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'+ description: >-+ Filter returned verbatim by evaluation-rule read endpoints.+++ Filters with a stored `key` use the keyed shape. All other filters use+ the base shape. Response filters are intentionally not broken down by+ their internal `type` value.+ PublicApiErrorCode:+ title: PublicApiErrorCode+ type: string+ enum:+ - authentication_failed+ - access_denied+ - invalid_request+ - invalid_query+ - invalid_body+ - resource_not_found+ - conflict+ - rate_limited+ - method_not_allowed+ - internal_error+ description: >-+ Stable machine-readable error code returned by the evaluators and+ evaluation-rules API.+ PublicApiValidationIssue:+ title: PublicApiValidationIssuetype: object+ description: One request validation issue.properties:- id:+ code:type: string- status:- type: integer+ description: Validator issue code.message:type: string- nullable: true- error:- nullable: true- required:- - id- - status- IngestionResponse:- title: IngestionResponse- type: object- properties:- successes:- type: array- items:- $ref: '#/components/schemas/IngestionSuccess'- errors:+ description: Human-readable explanation.+ path:type: array- items:- $ref: '#/components/schemas/IngestionError'- required:- - successes- - errors- OpenAICompletionUsageSchema:- title: OpenAICompletionUsageSchema- type: object- description: OpenAI Usage schema from (Chat-)Completion APIs- properties:- prompt_tokens:- type: integer- completion_tokens:- type: integer- total_tokens:- type: integer- prompt_tokens_details:- type: object- additionalProperties:- type: integer- nullable: true- nullable: true- completion_tokens_details:- type: object- additionalProperties:- type: integer- nullable: true- nullable: true- required:- - prompt_tokens- - completion_tokens- - total_tokens- OpenAIResponseUsageSchema:- title: OpenAIResponseUsageSchema+ items: {}+ description: Path to the invalid request field.+ required:+ - code+ - message+ - path+ PublicApiErrorDetails:+ title: PublicApiErrorDetailstype: object- description: OpenAI Usage schema from Response API+ description: Optional structured context for validation and rate-limit errors.properties:- input_tokens:+ issues:+ type: array+ items:+ $ref: '#/components/schemas/PublicApiValidationIssue'+ nullable: true+ description: Validation issues for an invalid request body or query.+ retryAfterSeconds:type: integer- output_tokens:+ nullable: true+ description: Number of seconds to wait before retrying a rate-limited request.+ limit:type: integer- total_tokens:+ nullable: true+ description: Rate-limit request allowance.+ remaining:type: integer- input_tokens_details:- type: object- additionalProperties:- type: integer- nullable: truenullable: true- output_tokens_details:- type: object- additionalProperties:- type: integer- nullable: true+ description: Remaining requests in the current rate-limit window.+ resetAt:+ type: string+ format: date-timenullable: true- required:- - input_tokens- - output_tokens- - total_tokens- UsageDetails:- title: UsageDetails- oneOf:- - type: object- additionalProperties:- type: integer- title: Object- - $ref: '#/components/schemas/OpenAICompletionUsageSchema'- - $ref: '#/components/schemas/OpenAIResponseUsageSchema'- legacyMetricsResponse:- title: legacyMetricsResponse+ description: Time when the current rate-limit window resets.+ PublicApiError:+ title: PublicApiErrortype: object- properties:- data:- type: array- items:- type: object- additionalProperties: true- description: >-- The metrics data. Each item in the list contains the metric values- and dimensions requested in the query.+ description: >-+ Standard error response for the stable evaluators and evaluation-rules+ API.- Format varies based on the query parameters.- Histograms will return an array with [lower, upper, height] tuples.- _deprecation:- $ref: '#/components/schemas/Deprecation'+ Use the HTTP status for the broad failure class and `code` for+ programmatic handling. `details` is included when field-level validation+ or retry information is available.+ properties:+ message:+ type: string+ description: Human-readable description of the failure.+ code:+ $ref: '#/components/schemas/PublicApiErrorCode'+ description: Stable machine-readable error code.+ details:+ $ref: '#/components/schemas/PublicApiErrorDetails'nullable: true+ description: Optional structured validation or rate-limit context.required:- - data- legacyObservations:- title: legacyObservations+ - message+ - code+ EvaluationRuleEvaluatorAssignmentInput:+ title: EvaluationRuleEvaluatorAssignmentInputtype: objectproperties:- data:+ evaluatorId:+ type: string+ description: >-+ Stable evaluator identifier. The rule automatically uses that+ evaluator's latest version.+ variableMapping:type: arrayitems:- $ref: '#/components/schemas/Observation'- meta:- $ref: '#/components/schemas/utilsMetaResponse'+ $ref: '#/components/schemas/PromptVariableMappingInput'+ nullable: true+ description: >-+ Rule-specific prompt-variable mapping.+++ Set to `null` or omit to inherit the evaluator's latest default+ mapping. Code evaluators use the fixed runtime mapping and should+ use `null`.required:- - data- - meta- legacyObservationsViews:- title: legacyObservationsViews+ - evaluatorId+ EvaluatorAssignment:+ title: EvaluatorAssignmenttype: objectproperties:- data:+ evaluatorId:+ type: string+ description: Stable identifier of the evaluator associated with this rule.+ variableMapping:type: arrayitems:- $ref: '#/components/schemas/ObservationsView'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- _deprecation:- $ref: '#/components/schemas/Deprecation'+ $ref: '#/components/schemas/PromptVariableMapping'nullable: true+ description: >-+ Stored rule-specific override, or `null` when the evaluator's latest+ default mapping is inherited. Legacy mappings are returned as an+ explicit `legacy` variant.required:- - data- - meta- LlmConnection:- title: LlmConnection+ - evaluatorId+ - variableMapping+ EvaluationRule:+ title: EvaluationRuletype: object- description: LLM API connection configuration (secrets excluded)+ description: >-+ Live evaluation rule for incoming observations.+++ A rule determines which evaluators should be used, which observations+ should trigger scoring, how often scoring should run, and which+ observation fields should populate prompt variables.properties:id:type: string- provider:+ description: Stable evaluation-rule identifier.+ name:type: stringdescription: >-- Provider name (e.g., 'openai', 'my-gateway'). Must be unique in- project, used for upserting.- adapter:+ Human-readable rule name. This is independent from evaluator names+ and does not need to be unique.+ createdBy:+ $ref: '#/components/schemas/Creator'+ nullable: true+ description: User who created this rule, or `null` when no user can be resolved.+ enabled:+ type: boolean+ description: Whether live execution is enabled for this rule.+ sampling:+ type: number+ format: double+ description: |-+ Fraction of matching observations that should be evaluated.++ Must be between `0` and `1`.+ - `1` evaluates every matching observation.+ - `0.25` evaluates approximately 25% of matching observations.+ filter:+ type: array+ items:+ $ref: '#/components/schemas/EvaluationRuleReadFilter'+ description: >-+ List of stored filter conditions returned verbatim. Filters with a+ `key` use the keyed response shape; all others use the base shape.+ These response shapes are not broken down by internal filter type.+ An empty list matches every incoming object.+ evaluatorAssignments:+ type: array+ items:+ $ref: '#/components/schemas/EvaluatorAssignment'+ description: Evaluators attached to this rule in deterministic assignment order.+ createdAt:type: string- description: The adapter used to interface with the LLM- displaySecretKey:+ format: date-time+ description: Timestamp when the evaluation rule was created.+ updatedAt:type: string- description: Masked version of the secret key for display purposes- baseURL:+ format: date-time+ description: Timestamp when the evaluation rule was last updated.+ required:+ - id+ - name+ - createdBy+ - enabled+ - sampling+ - filter+ - evaluatorAssignments+ - createdAt+ - updatedAt+ CreateEvaluationRuleRequest:+ title: CreateEvaluationRuleRequest+ type: object+ properties:+ name:+ type: string+ description: >-+ Human-readable rule name. Names are not identifiers and do not need+ to be unique.+ enabled:+ type: boolean+ description: >-+ Whether live execution should start immediately. Enabled rules+ require at least one evaluator assignment.+ sampling:+ type: number+ format: double+ nullable: true+ description: >-+ Fraction of matching observations to evaluate. Omit this field to+ use the default of `1`, which evaluates every match. `null` is not+ accepted.+ filter:+ type: array+ items:+ $ref: '#/components/schemas/EvaluationRuleFilter'+ nullable: true+ description: >-+ Conditions used to select observations. Defaults to an empty list,+ which matches every incoming observation.+ evaluatorAssignments:+ type: array+ items:+ $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'+ description: >-+ Evaluators to attach to this rule. Disabled rules may use an empty+ list as a draft.+ required:+ - name+ - enabled+ - evaluatorAssignments+ UpdateEvaluationRuleRequest:+ title: UpdateEvaluationRuleRequest+ type: object+ description: At least one field must be provided.+ properties:+ name:type: stringnullable: true- description: Custom base URL for the LLM API- customModels:- type: array- items:- type: string- description: List of custom model names available for this connection- withDefaultModels:+ description: New human-readable rule name.+ enabled:type: boolean- description: Whether to include default models for this adapter- extraHeaderKeys:+ nullable: true+ description: New desired live-execution state.+ sampling:+ type: number+ format: double+ nullable: true+ description: >-+ New fraction of matching observations to evaluate. Omit to keep the+ current value.+ filter:type: arrayitems:- type: string+ $ref: '#/components/schemas/EvaluationRuleFilter'+ nullable: truedescription: >-- Keys of extra headers sent with requests (values excluded for- security)- config:- type: object- additionalProperties: true+ Complete replacement filter list. An empty list matches every+ incoming observation.+ evaluatorAssignments:+ type: array+ items:+ $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'nullable: truedescription: >-- Adapter-specific configuration. Required for Bedrock- (`{"region":"us-east-1"}`), optional for OpenAI- (`{"useResponsesApi":true}`), optional for VertexAI- (`{"location":"us-central1"}`), not used by other adapters.- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- required:- - id- - provider- - adapter- - displaySecretKey- - customModels- - withDefaultModels- - extraHeaderKeys- - createdAt- - updatedAt- PaginatedLlmConnections:- title: PaginatedLlmConnections+ Complete replacement assignment list. An empty list disables the+ rule.+ EvaluationRulesPage:+ title: EvaluationRulesPagetype: objectproperties:data:type: arrayitems:- $ref: '#/components/schemas/LlmConnection'+ $ref: '#/components/schemas/EvaluationRule'+ description: Evaluation rules for this page.meta:- $ref: '#/components/schemas/utilsMetaResponse'+ $ref: '#/components/schemas/CursorMeta'+ description: Cursor pagination metadata.required:- data- meta- UpsertLlmConnectionRequest:- title: UpsertLlmConnectionRequest+ DeletedEvaluationRule:+ title: DeletedEvaluationRuletype: object- description: Request to create or update an LLM connection (upsert)+ description: Confirmation returned after successful evaluation-rule deletion.properties:- provider:+ id:type: string- description: >-- Provider name (e.g., 'openai', 'my-gateway'). Must be unique in- project, used for upserting.- adapter:- $ref: '#/components/schemas/LlmAdapter'- description: The adapter used to interface with the LLM- secretKey:+ description: Identifier of the deleted evaluation rule.+ required:+ - id+ Creator:+ title: Creator+ type: object+ description: User who created the resource, or `null` when no user can be resolved.+ properties:+ id:type: string- description: Secret key for the LLM API.- baseURL:+ description: User identifier.+ name:type: stringnullable: true- description: Custom base URL for the LLM API- customModels:- type: array- items:- type: string- nullable: true- description: List of custom model names- withDefaultModels:- type: boolean- nullable: true- description: Whether to include default models. Default is true.- extraHeaders:- type: object- additionalProperties:- type: string- nullable: true- description: Extra headers to send with requests- config:- type: object- additionalProperties: true- nullable: true- description: >-- Adapter-specific configuration. Validation rules: - **Bedrock**:- Required. Must be `{"region": "<aws-region>"}` (e.g.,- `{"region":"us-east-1"}`) - **OpenAI**: Optional. If provided, must- be `{"useResponsesApi": <boolean>}` to control whether Langfuse- routes calls through OpenAI's Responses API. - **VertexAI**:- Optional. If provided, must be `{"location": "<gcp-location>"}`- (e.g., `{"location":"us-central1"}`) - **Other adapters**: Not- supported. Omit this field or set to null.+ description: User name, or `null` when unavailable.required:- - provider- - adapter- - secretKey- DeleteLlmConnectionResponse:- title: DeleteLlmConnectionResponse+ - id+ - name+ EvaluatorModelConfig:+ title: EvaluatorModelConfigtype: object+ description: >-+ Explicit model configuration for an evaluator.+++ If the complete `modelConfig` is `null`, Langfuse uses the project's+ default evaluation model.++ If provided, the model must be available to the project when the+ evaluator or evaluation rule is enabled.properties:- message:+ provider:+ type: string+ description: >-+ Provider identifier, for example `openai` or `anthropic`.+++ To discover valid values for the current project, call `GET+ /api/public/llm-connections` and use one of the returned `provider`+ values.+ model:type: string+ description: >-+ Model identifier exposed by the provider, for example `gpt-4.1-mini`.required:- - message- LlmAdapter:- title: LlmAdapter+ - provider+ - model+ EvaluatorChatMessageRole:+ title: EvaluatorChatMessageRoletype: stringenum:- - anthropic- - openai- - azure- - bedrock- - google-vertex-ai- - google-ai-studio- GetMediaResponse:- title: GetMediaResponse+ - system+ - user+ - assistant+ description: Role of an evaluator prompt message.+ EvaluatorChatMessage:+ title: EvaluatorChatMessagetype: object+ description: One chat message in an evaluator prompt.properties:- mediaId:- type: string- description: The unique langfuse identifier of a media record- contentType:- type: string- description: The MIME type of the media record- contentLength:- type: integer- description: The size of the media record in bytes- uploadedAt:- type: string- format: date-time- description: The date and time when the media record was uploaded- url:- type: string- description: The download URL of the media record- urlExpiry:+ role:+ $ref: '#/components/schemas/EvaluatorChatMessageRole'+ content:type: string- description: The expiry date and time of the media record download URL+ description: Message content. Evaluator variables use `{{variable}}` syntax.required:- - mediaId- - contentType- - contentLength- - uploadedAt- - url- - urlExpiry- PatchMediaBody:- title: PatchMediaBody+ - role+ - content+ EvaluatorChatPrompt:+ title: EvaluatorChatPrompt+ type: array+ items:+ $ref: '#/components/schemas/EvaluatorChatMessage'+ description: >-+ An ordered list of chat messages. A system message is only allowed as+ the first message.+ EvaluatorChatPromptInput:+ title: EvaluatorChatPromptInput+ oneOf:+ - type: string+ title: String+ - $ref: '#/components/schemas/EvaluatorChatPrompt'+ description: A user prompt string shortcut, or an ordered list of chat messages.+ EvaluatorVersionBase:+ title: EvaluatorVersionBasetype: objectproperties:- uploadedAt:- type: string- format: date-time- description: The date and time when the media record was uploaded- uploadHttpStatus:- type: integer- description: The HTTP status code of the upload- uploadHttpError:+ id:type: string- nullable: true- description: The HTTP error message of the upload- uploadTimeMs:+ description: Stable identifier of this evaluator version.+ version:type: integer- nullable: true- description: The time in milliseconds it took to upload the media record- required:- - uploadedAt- - uploadHttpStatus- GetMediaUploadUrlRequest:- title: GetMediaUploadUrlRequest- type: object- description: >-- Request a presigned media upload URL. Provide exactly one context: a- trace (traceId, optionally observationId) or a dataset item (datasetId +- datasetItemId). field is required and must match the chosen context.- properties:- traceId:+ description: Monotonically increasing evaluator version number.+ createdAt:type: string+ format: date-time+ description: Timestamp when this evaluator version was created.+ createdBy:+ $ref: '#/components/schemas/Creator'nullable: truedescription: >-- The trace the media is associated with. Null for dataset item media- uploads.- observationId:+ User who created this version, or `null` when no user can be+ resolved.+ required:+ - id+ - version+ - createdAt+ - createdBy+ LlmAsJudgeEvaluatorVersion:+ title: LlmAsJudgeEvaluatorVersion+ type: object+ properties:+ type:type: string- nullable: true+ const: llm_as_judge+ description: Evaluator type.+ prompt:+ $ref: '#/components/schemas/EvaluatorChatPrompt'+ description: Ordered chat messages used during evaluation.+ variables:+ type: array+ items:+ type: stringdescription: >-- The observation ID associated with the media record. If the media- record is associated directly with a trace, this will be null.- datasetId:- type: string+ Variables extracted from the prompt and available for+ evaluation-rule mappings.+++ Every variable must be mapped exactly once when a rule provides an+ explicit mapping.+ variableMapping:+ type: array+ items:+ $ref: '#/components/schemas/PromptVariableMappingRead'nullable: truedescription: >-- The dataset the media belongs to. Null for trace/observation media- uploads.- datasetItemId:- type: string+ Default variable mapping for this evaluator version, or `null` when+ no default is configured.+ modelConfig:+ $ref: '#/components/schemas/EvaluatorModelConfig'nullable: truedescription: >-- The dataset item the media is associated with (need not exist yet).- Null for trace/observation media uploads.- contentType:- $ref: '#/components/schemas/MediaContentType'- contentLength:- type: integer- description: The size of the media record in bytes- sha256Hash:- type: string- description: The SHA-256 hash of the media record- field:- type: string+ Explicit model configuration, or `null` when the project's default+ evaluation model is used.+ outputDefinition:+ $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'description: >-- The item field the media is in: `input`/`output`/`metadata` (trace)- or `input`/`expectedOutput`/`metadata` (dataset item).+ Structured output schema returned by this evaluator.+++ Responses include `dataType` and omit the internal output-definition+ persistence version.required:- - contentType- - contentLength- - sha256Hash- - field- GetMediaUploadUrlResponse:- title: GetMediaUploadUrlResponse+ - type+ - prompt+ - variables+ - variableMapping+ - modelConfig+ - outputDefinition+ allOf:+ - $ref: '#/components/schemas/EvaluatorVersionBase'+ CodeEvaluatorVersion:+ title: CodeEvaluatorVersiontype: objectproperties:- uploadUrl:+ type:type: string- nullable: true- description: >-- The presigned upload URL. If the asset is already uploaded, this- will be null- mediaId:+ const: code+ description: Evaluator type.+ sourceCode:type: string- description: The unique langfuse identifier of a media record+ description: Source code executed for each matched observation.+ sourceCodeLanguage:+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'+ description: Runtime language used to execute the source code.required:- - mediaId- MediaContentType:- title: MediaContentType- type: string- enum:- - image/png- - image/jpeg- - image/jpg- - image/webp- - image/gif- - image/svg+xml- - image/tiff- - image/bmp- - image/avif- - image/heic- - audio/mpeg- - audio/mp3- - audio/wav- - audio/ogg- - audio/oga- - audio/aac- - audio/mp4- - audio/flac- - audio/opus- - audio/webm- - video/mp4- - video/webm- - video/ogg- - video/mpeg- - video/quicktime- - video/x-msvideo- - video/x-matroska- - text/plain- - text/html- - text/css- - text/csv- - text/markdown- - text/x-python- - application/javascript- - text/x-typescript- - application/x-yaml- - application/pdf- - application/msword- - application/vnd.ms-excel- - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet- - application/zip- - application/json- - application/xml- - application/octet-stream- - >-- application/vnd.openxmlformats-officedocument.wordprocessingml.document- - >-- application/vnd.openxmlformats-officedocument.presentationml.presentation- - application/rtf- - application/x-ndjson- - application/vnd.apache.parquet- - application/gzip- - application/x-tar- - application/x-7z-compressed- description: The MIME type of the media record- MetricsV2Response:- title: MetricsV2Response+ - type+ - sourceCode+ - sourceCodeLanguage+ allOf:+ - $ref: '#/components/schemas/EvaluatorVersionBase'+ EvaluatorVersion:+ title: EvaluatorVersion+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - llm_as_judge+ - $ref: '#/components/schemas/LlmAsJudgeEvaluatorVersion'+ required:+ - type+ title: LlmAsJudgeEvaluatorVersion+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - code+ - $ref: '#/components/schemas/CodeEvaluatorVersion'+ required:+ - type+ title: CodeEvaluatorVersion+ EvaluationRuleAssignment:+ title: EvaluationRuleAssignmenttype: object+ description: Evaluation-rule assignment associated with this evaluator.properties:- data:+ evaluationRuleId:+ type: string+ description: Stable identifier of the assigned evaluation rule.+ variableMappingOverride:type: arrayitems:- type: object- additionalProperties: true+ $ref: '#/components/schemas/PromptVariableMapping'+ nullable: truedescription: >-- The metrics data. Each item in the list contains the metric values- and dimensions requested in the query.-- Format varies based on the query parameters.-- Histograms will return an array with [lower, upper, height] tuples.- required:- - data- ModelTokenizerId:- title: ModelTokenizerId- type: string- enum:- - openai- - claude- description: Tokenizer supported by Langfuse for model usage inference.- PaginatedModels:- title: PaginatedModels- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/Model'- meta:- $ref: '#/components/schemas/utilsMetaResponse'+ Rule-specific variable mapping override. Omitted when the+ evaluator's latest default mapping is inherited. Legacy mappings use+ the explicit `legacy` variant.required:- - data- - meta- CreateModelRequest:- title: CreateModelRequest+ - evaluationRuleId+ EvaluatorBase:+ title: EvaluatorBasetype: objectproperties:- modelName:+ id:+ type: string+ description: Stable identifier of this evaluator across all versions.+ name:+ type: string+ description: >-+ Human-readable evaluator name. Names are not identifiers and do not+ need to be unique.+ description:type: string+ nullable: true+ description: Optional human-readable evaluator description.+ createdBy:+ $ref: '#/components/schemas/Creator'+ nullable: truedescription: >-- Name of the model definition. If multiple with the same name exist,- they are applied in the following order: (1) custom over built-in,- (2) newest according to startTime where- model.startTime<observation.startTime- matchPattern:- type: string+ User who created this evaluator, or `null` when no user can be+ resolved.+ status:+ $ref: '#/components/schemas/EvaluatorStatus'description: >-- Regex pattern which matches this model definition to- generation.model. Useful in case of fine-tuned models. If you want- to exact match, use `(?i)^modelname$`- startDate:+ Effective evaluator status after Langfuse validates its runtime+ configuration.+ pausedAt:type: stringformat: date-timenullable: true- description: Apply only to generations which are newer than this ISO date.- unit:- $ref: '#/components/schemas/ModelUsageUnit'- description: Unit used by this model.- inputPrice:- type: number- format: double+ description: Timestamp when the evaluator was paused, otherwise `null`.+ pausedReason:+ type: stringnullable: true- description: >-- Deprecated. Use 'pricingTiers' instead. Price (USD) per input unit.- Creates a default tier if pricingTiers not provided.- outputPrice:- type: number- format: double+ description: Machine-readable reason when `status=paused`, otherwise `null`.+ pausedMessage:+ type: stringnullable: true+ description: Human-readable explanation when `status=paused`, otherwise `null`.+ evaluationRuleAssignments:+ type: array+ items:+ $ref: '#/components/schemas/EvaluationRuleAssignment'description: >-- Deprecated. Use 'pricingTiers' instead. Price (USD) per output unit.- Creates a default tier if pricingTiers not provided.- totalPrice:- type: number- format: double+ All modern and legacy evaluation-rule assignments in+ newest-assignment-first order. Rule-specific mappings are exposed as+ `variableMappingOverride`; inherited defaults are omitted.+ createdAt:+ type: string+ format: date-time+ description: Timestamp when the evaluator was created.+ updatedAt:+ type: string+ format: date-time+ description: Timestamp when the evaluator was last updated.+ versionId:+ type: string+ description: Stable identifier of the latest evaluator version.+ version:+ type: integer+ description: Monotonically increasing latest evaluator version number.+ versionCreatedAt:+ type: string+ format: date-time+ description: Timestamp when the latest evaluator version was created.+ versionCreatedBy:+ $ref: '#/components/schemas/Creator'nullable: truedescription: >-- Deprecated. Use 'pricingTiers' instead. Price (USD) per total units.- Cannot be set if input or output price is set. Creates a default- tier if pricingTiers not provided.- pricingTiers:+ User who created the latest evaluator version, or `null` when no+ user can be resolved.+ required:+ - id+ - name+ - description+ - createdBy+ - status+ - pausedAt+ - pausedReason+ - pausedMessage+ - evaluationRuleAssignments+ - createdAt+ - updatedAt+ - versionId+ - version+ - versionCreatedAt+ - versionCreatedBy+ LlmAsJudgeEvaluator:+ title: LlmAsJudgeEvaluator+ type: object+ properties:+ type:+ type: string+ const: llm_as_judge+ description: Evaluator type.+ prompt:+ $ref: '#/components/schemas/EvaluatorChatPrompt'+ description: Ordered chat messages used by the latest evaluator version.+ variables:type: arrayitems:- $ref: '#/components/schemas/PricingTierInput'- nullable: true+ type: stringdescription: >-- Optional. Array of pricing tiers for this model.--- Use pricing tiers for all models - both those with threshold-based- pricing variations and those with simple flat pricing:--- - For models with standard flat pricing: Create a single default- tier with your prices- (e.g., one tier with isDefault=true, priority=0, conditions=[], and your standard prices)-- - For models with threshold-based pricing: Create a default tier- plus additional conditional tiers- (e.g., default tier for standard usage + high-volume tier for usage above certain thresholds)-- Requirements:-- - Cannot be provided with flat prices- (inputPrice/outputPrice/totalPrice) - use one approach or the other-- - Must include exactly one default tier with isDefault=true,- priority=0, and conditions=[]-- - All tier names and priorities must be unique within the model-- - Each tier must define at least one price--- If omitted, you must provide flat prices instead- (inputPrice/outputPrice/totalPrice),-- which will automatically create a single default tier named- "Standard".- tokenizerId:- $ref: '#/components/schemas/ModelTokenizerId'+ Variables extracted from the latest prompt and available for+ evaluation-rule mappings.+ variableMapping:+ type: array+ items:+ $ref: '#/components/schemas/PromptVariableMappingRead'nullable: truedescription: >-- Optional. Tokenizer to be applied to observations which match to- this model. See docs for more details.- tokenizerConfig:+ Default variable mapping for the latest version, or `null` when no+ default is configured.+ modelConfig:+ $ref: '#/components/schemas/EvaluatorModelConfig'nullable: truedescription: >-- Optional. Configuration for the selected tokenizer. Needs to be- JSON. See docs for more details.+ Explicit model configuration for the latest version, or `null` when+ the project's default evaluation model is used.+ outputDefinition:+ $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'+ description: Structured output schema returned by the latest evaluator version.required:- - modelName- - matchPattern- - unit- ObservationsV2Response:- title: ObservationsV2Response+ - type+ - prompt+ - variables+ - variableMapping+ - modelConfig+ - outputDefinition+ allOf:+ - $ref: '#/components/schemas/EvaluatorBase'+ CodeEvaluator:+ title: CodeEvaluatortype: object+ properties:+ type:+ type: string+ const: code+ description: Evaluator type.+ sourceCode:+ type: string+ description: Source code executed by the latest evaluator version.+ sourceCodeLanguage:+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'+ description: Runtime language used to execute the latest source code.+ required:+ - type+ - sourceCode+ - sourceCodeLanguage+ allOf:+ - $ref: '#/components/schemas/EvaluatorBase'+ Evaluator:+ title: Evaluator+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - llm_as_judge+ - $ref: '#/components/schemas/LlmAsJudgeEvaluator'+ required:+ - type+ title: LlmAsJudgeEvaluator+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - code+ - $ref: '#/components/schemas/CodeEvaluator'+ required:+ - type+ title: CodeEvaluatordescription: >-- Response containing observations with field-group-based filtering and- cursor-based pagination.-+ One evaluator that can be used for scoring.- The `data` array contains observation objects with only the requested- field groups included.- Use the `cursor` in `meta` to retrieve the next page of results.- properties:- data:- type: array- items:- $ref: '#/components/schemas/ObservationV2'- description: >-- Array of observation objects. Fields included depend on the `fields`- parameter in the request.- meta:- $ref: '#/components/schemas/ObservationsV2Meta'- required:- - data- - meta- ObservationsV2Meta:- title: ObservationsV2Meta+ An evaluator describes **how** to score data. Associated evaluation+ rules describe **which** live objects should be evaluated.+ EvaluatorStatus:+ title: EvaluatorStatus+ type: string+ enum:+ - active+ - paused+ description: >-+ Effective evaluator runtime status.+++ - `active`: the evaluator can run.++ - `paused`: Langfuse paused execution until the underlying issue is+ resolved.+ CreateLlmAsJudgeEvaluatorRequest:+ title: CreateLlmAsJudgeEvaluatorRequesttype: object- description: Metadata for cursor-based paginationproperties:- cursor:+ name:type: string- nullable: truedescription: >-- Base64-encoded cursor to use for retrieving the next page. If not- present, there are no more results.- OtelResourceSpan:- title: OtelResourceSpan- type: object- description: >-- Represents a collection of spans from a single resource as per OTLP- specification- properties:- resource:- $ref: '#/components/schemas/OtelResource'+ Human-readable evaluator name. Names are not identifiers and do not+ need to be unique.+ description:+ type: stringnullable: true- description: Resource information- scopeSpans:+ description: Optional human-readable evaluator description.+ type:+ type: string+ const: llm_as_judge+ description: Evaluator type.+ prompt:+ $ref: '#/components/schemas/EvaluatorChatPromptInput'+ description: User prompt string shortcut or an ordered list of chat messages.+ modelConfig:+ $ref: '#/components/schemas/EvaluatorModelConfig'+ nullable: true+ description: >-+ Explicit model configuration. Set to `null` or omit to use the+ project's default evaluation model.+ variableMapping:type: arrayitems:- $ref: '#/components/schemas/OtelScopeSpan'+ $ref: '#/components/schemas/PromptVariableMappingInput'nullable: true- description: Array of scope spans- OtelResource:- title: OtelResource+ description: >-+ Default prompt-variable mapping, or `null` when no default is+ configured.+ outputDefinition:+ $ref: '#/components/schemas/EvaluatorOutputDefinition'+ description: >-+ Structured output schema returned by this evaluator. Do not include+ an internal persistence version.+ required:+ - name+ - type+ - prompt+ - outputDefinition+ CreateCodeEvaluatorRequest:+ title: CreateCodeEvaluatorRequesttype: object- description: Resource attributes identifying the source of telemetryproperties:- attributes:- type: array- items:- $ref: '#/components/schemas/OtelAttribute'+ name:+ type: string+ description: >-+ Human-readable evaluator name. Names are not identifiers and do not+ need to be unique.+ description:+ type: stringnullable: true- description: Resource attributes like service.name, service.version, etc.- OtelScopeSpan:- title: OtelScopeSpan+ description: Optional human-readable evaluator description.+ type:+ type: string+ const: code+ description: Evaluator type.+ sourceCode:+ type: string+ description: Source code executed for each matched observation.+ sourceCodeLanguage:+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'+ description: Runtime language used to execute the source code.+ required:+ - name+ - type+ - sourceCode+ - sourceCodeLanguage+ CreateEvaluatorRequest:+ title: CreateEvaluatorRequest+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - llm_as_judge+ - $ref: '#/components/schemas/CreateLlmAsJudgeEvaluatorRequest'+ required:+ - type+ title: CreateLlmAsJudgeEvaluatorRequest+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - code+ - $ref: '#/components/schemas/CreateCodeEvaluatorRequest'+ required:+ - type+ title: CreateCodeEvaluatorRequest+ description: >-+ Initial evaluator definition with metadata and definition fields at the+ same level. The returned evaluator starts at version `1`.+ UpdateEvaluatorMetadataRequest:+ title: UpdateEvaluatorMetadataRequesttype: object- description: Collection of spans from a single instrumentation scope+ description: Metadata-only update. At least one field must be provided.properties:- scope:- $ref: '#/components/schemas/OtelScope'+ name:+ type: stringnullable: true- description: Instrumentation scope information- spans:- type: array- items:- $ref: '#/components/schemas/OtelSpan'+ description: New human-readable evaluator name.+ description:+ type: stringnullable: true- description: Array of spans- OtelScope:- title: OtelScope+ description: New description. Set to `null` to clear it.+ UpdateLlmAsJudgeEvaluatorRequest:+ title: UpdateLlmAsJudgeEvaluatorRequesttype: object- description: Instrumentation scope informationproperties:name:type: stringnullable: true- description: Instrumentation scope name- version:+ description: New human-readable evaluator name.+ description:type: stringnullable: true- description: Instrumentation scope version- attributes:+ description: New description. Set to `null` to clear it.+ type:+ type: string+ const: llm_as_judge+ description: Evaluator type. The type of an existing evaluator cannot change.+ prompt:+ $ref: '#/components/schemas/EvaluatorChatPromptInput'+ description: >-+ Complete replacement user prompt string or an ordered list of chat+ messages.+ modelConfig:+ $ref: '#/components/schemas/EvaluatorModelConfig'+ nullable: true+ description: >-+ Explicit model configuration. Set to `null` or omit to use the+ project's default evaluation model.+ variableMapping:type: arrayitems:- $ref: '#/components/schemas/OtelAttribute'+ $ref: '#/components/schemas/PromptVariableMappingInput'nullable: true- description: Additional scope attributes- OtelSpan:- title: OtelSpan+ description: >-+ Complete replacement default variable mapping, or `null` when no+ default is configured.+ outputDefinition:+ $ref: '#/components/schemas/EvaluatorOutputDefinition'+ description: >-+ Complete replacement output schema. Do not include an internal+ persistence version.+ required:+ - type+ - prompt+ - outputDefinition+ UpdateCodeEvaluatorRequest:+ title: UpdateCodeEvaluatorRequesttype: object- description: Individual span representing a unit of work or operationproperties:- traceId:- nullable: true- description: Trace ID (16 bytes, hex-encoded string in JSON or Buffer in binary)- spanId:- nullable: true- description: Span ID (8 bytes, hex-encoded string in JSON or Buffer in binary)- parentSpanId:- nullable: true- description: Parent span ID if this is a child spanname:type: stringnullable: true- description: Span name describing the operation- kind:- type: integer- nullable: true- description: Span kind (1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER)- startTimeUnixNano:- nullable: true- description: Start time in nanoseconds since Unix epoch- endTimeUnixNano:+ description: New human-readable evaluator name.+ description:+ type: stringnullable: true- description: End time in nanoseconds since Unix epoch- attributes:+ description: New description. Set to `null` to clear it.+ type:+ type: string+ const: code+ description: Evaluator type. The type of an existing evaluator cannot change.+ sourceCode:+ type: string+ description: Complete replacement source code.+ sourceCodeLanguage:+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'+ description: Runtime language used to execute the source code.+ required:+ - type+ - sourceCode+ - sourceCodeLanguage+ UpdateEvaluatorRequest:+ title: UpdateEvaluatorRequest+ oneOf:+ - $ref: '#/components/schemas/UpdateEvaluatorMetadataRequest'+ - $ref: '#/components/schemas/UpdateLlmAsJudgeEvaluatorRequest'+ - $ref: '#/components/schemas/UpdateCodeEvaluatorRequest'+ description: >-+ Metadata-only update or complete flattened definition replacement. At+ least one field must be provided.+ EvaluatorsPage:+ title: EvaluatorsPage+ type: object+ properties:+ data:+ type: array+ items:+ $ref: '#/components/schemas/Evaluator'+ description: Evaluators for this page.+ meta:+ $ref: '#/components/schemas/CursorMeta'+ description: Cursor pagination metadata.+ required:+ - data+ - meta+ EvaluatorVersionsPage:+ title: EvaluatorVersionsPage+ type: object+ properties:+ data:type: arrayitems:- $ref: '#/components/schemas/OtelAttribute'- nullable: true- description: >-- Span attributes including Langfuse-specific attributes- (langfuse.observation.*)- status:- nullable: true- description: Span status object- OtelAttribute:- title: OtelAttribute+ $ref: '#/components/schemas/EvaluatorVersion'+ description: Evaluator versions for this page in newest-first order.+ meta:+ $ref: '#/components/schemas/CursorMeta'+ description: Cursor pagination metadata.+ required:+ - data+ - meta+ CursorMeta:+ title: CursorMetatype: object- description: Key-value attribute pair for resources, scopes, or spansproperties:- key:+ cursor:type: stringnullable: true- description: Attribute key (e.g., "service.name", "langfuse.observation.type")- value:- $ref: '#/components/schemas/OtelAttributeValue'- nullable: true- description: Attribute value- OtelAttributeValue:- title: OtelAttributeValue+ description: Opaque cursor for the next page. Omitted when there is no next page.+ DeletedEvaluator:+ title: DeletedEvaluatortype: object- description: Attribute value wrapper supporting different value types+ description: Confirmation returned after successful evaluator deletion.properties:- stringValue:+ id:type: string- nullable: true- description: String value- intValue:- type: integer- nullable: true- description: Integer value- doubleValue:- type: number- format: double- nullable: true- description: Double value- boolValue:- type: boolean- nullable: true- description: Boolean value- OtelTraceResponse:- title: OtelTraceResponse- type: object- description: Response from trace export request. Empty object indicates success.- properties: {}- MembershipRole:- title: MembershipRole- type: string- enum:- - OWNER- - ADMIN- - MEMBER- - VIEWER- MembershipRequest:- title: MembershipRequest+ description: Identifier of the deleted evaluator.+ required:+ - id+ ExperimentsResponse:+ title: ExperimentsResponsetype: objectproperties:- userId:- type: string- role:- $ref: '#/components/schemas/MembershipRole'+ data:+ type: array+ items:+ $ref: '#/components/schemas/Experiment'+ meta:+ $ref: '#/components/schemas/ExperimentsResponseMeta'required:- - userId- - role- DeleteMembershipRequest:- title: DeleteMembershipRequest+ - data+ - meta+ ExperimentsResponseMeta:+ title: ExperimentsResponseMetatype: objectproperties:- userId:+ cursor:type: string- required:- - userId- MembershipResponse:- title: MembershipResponse+ nullable: true+ description: >-+ Versioned base64url cursor for retrieving the next page. Absent when+ there are no more results.+ Experiment:+ title: Experimenttype: objectproperties:- userId:- type: string- role:- $ref: '#/components/schemas/MembershipRole'- email:+ id:type: stringname:type: string- required:- - userId- - role- - name- MembershipDeletionResponse:- title: MembershipDeletionResponse- type: object- properties:- message:+ description:type: string- userId:+ nullable: true+ startTime:+ type: string+ format: date-time+ description: |-+ Start of the experiment, i.e. the earliest event within the+ requested time range. Clipped to `fromStartTime` when the+ experiment started before the requested range.+ endTime:+ type: string+ format: date-time+ description: |-+ End of the experiment, i.e. the latest event end within the+ requested time range.+ itemCount:+ type: integer+ description: Number of experiment items within the requested time range.+ datasetId:type: string+ nullable: true+ description: Null when the experiment is not associated with a dataset.+ metadata:+ type: object+ additionalProperties: true+ nullable: true+ description: Included only when `fields=metadata` is requested.+ scores:+ type: array+ items:+ $ref: '#/components/schemas/ScoreV3'+ nullable: true+ description: >-+ Included only when `fields=scores` is requested. Contains scores+ directly attached to the experiment.required:- - message- - userId- MembershipsResponse:- title: MembershipsResponse+ - id+ - name+ - description+ - startTime+ - endTime+ - itemCount+ - datasetId+ ExperimentItemsResponse:+ title: ExperimentItemsResponsetype: objectproperties:- memberships:+ data:type: arrayitems:- $ref: '#/components/schemas/MembershipResponse'+ $ref: '#/components/schemas/ExperimentItem'+ meta:+ $ref: '#/components/schemas/ExperimentsResponseMeta'required:- - memberships- OrganizationProject:- title: OrganizationProject+ - data+ - meta+ ExperimentItem:+ title: ExperimentItemtype: objectproperties:id:type: string- name:+ traceId:+ type: string+ startTime:+ type: string+ format: date-time+ endTime:+ type: string+ format: date-time+ nullable: true+ level:+ $ref: '#/components/schemas/ObservationLevel'+ environment:+ type: string+ experimentId:+ type: string+ experimentName:+ type: string+ experimentItemId:+ type: string+ experimentDatasetId:+ type: string+ nullable: true+ description: Included when `fields=dataset` is requested.+ experimentItemVersion:type: string+ format: date-time+ nullable: true+ description: Included when `fields=dataset` is requested.+ input:+ nullable: true+ description: Included when `fields=io` is requested.+ output:+ nullable: true+ description: Included when `fields=io` is requested.+ expectedOutput:+ nullable: true+ description: Included when `fields=io` is requested.metadata:type: objectadditionalProperties: truenullable: true- createdAt:- type: string- format: date-time- updatedAt:+ description: Included when `fields=metadata` is requested.+ experimentItemMetadata:+ type: object+ additionalProperties: true+ nullable: true+ description: Included when `fields=itemMetadata` is requested.+ experimentMetadata:+ type: object+ additionalProperties: true+ nullable: true+ description: Included when `fields=experimentMetadata` is requested.+ experimentDescription:type: string- format: date-time- required:- - id- - name- - createdAt- - updatedAt- OrganizationProjectsResponse:- title: OrganizationProjectsResponse- type: object- properties:- projects:+ nullable: true+ description: Included when `fields=experimentMetadata` is requested.+ scores:type: arrayitems:- $ref: '#/components/schemas/OrganizationProject'+ $ref: '#/components/schemas/ScoreV3'+ nullable: true+ description: >-+ Included only when `fields=scores` is requested. Contains item and+ trace scores only; experiment-level scores are returned by the+ experiments endpoint.required:- - projects- OrganizationApiKey:- title: OrganizationApiKey+ - id+ - traceId+ - startTime+ - endTime+ - level+ - environment+ - experimentId+ - experimentName+ - experimentItemId+ FeedbackTargetType:+ title: FeedbackTargetType+ type: string+ enum:+ - skill+ - mcp-tool+ - cli+ - docs+ - public-api+ - other+ SubmitFeedbackRequest:+ title: SubmitFeedbackRequesttype: objectproperties:- id:- type: string- createdAt:+ targetType:+ $ref: '#/components/schemas/FeedbackTargetType'+ description: Category of the thing the feedback is about.+ target:type: string- format: date-time- expiresAt:+ description: >-+ The specific instance within targetType: the skill name, MCP tool+ name, CLI command, API endpoint path, or docs page path (e.g.+ 'queryMetrics', '/docs/mcp'). An identifier, not a sentence. Must be+ between 1 and 200 characters.+ feedback:type: string- format: date-time- nullable: true- lastUsedAt:+ description: >-+ Concise feedback text approved by the user. Must be between 1 and+ 3000 characters.+ goal:type: string- format: date-timenullable: true- note:+ description: >-+ Optional user-approved goal or use case they were trying to achieve.+ Must be between 1 and 1500 characters when provided. Do not include+ secrets, customer data, trace payloads, or broad unrelated context.+ referenceUrl:type: stringnullable: true- publicKey:- type: string- displaySecretKey:+ description: >-+ Optional HTTP(S) reference URL. Langfuse stores it as text for+ triage and does not fetch it.+ required:+ - targetType+ - target+ - feedback+ SubmitFeedbackResponse:+ title: SubmitFeedbackResponse+ type: object+ properties:+ id:type: string+ description: Correlation ID for the submitted feedback.required:- id- - createdAt- - publicKey- - displaySecretKey- OrganizationApiKeysResponse:- title: OrganizationApiKeysResponse+ HealthResponse:+ title: HealthResponsetype: objectproperties:- apiKeys:- type: array- items:- $ref: '#/components/schemas/OrganizationApiKey'+ version:+ type: string+ description: Langfuse server version+ status:+ type: stringrequired:- - apiKeys- Projects:- title: Projects+ - version+ - status+ IngestionEvent:+ title: IngestionEvent+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - trace-create+ - $ref: '#/components/schemas/TraceEvent'+ required:+ - type+ title: TraceEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - score-create+ - $ref: '#/components/schemas/ScoreEvent'+ required:+ - type+ title: ScoreEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - span-create+ - $ref: '#/components/schemas/CreateSpanEvent'+ required:+ - type+ title: CreateSpanEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - span-update+ - $ref: '#/components/schemas/UpdateSpanEvent'+ required:+ - type+ title: UpdateSpanEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - generation-create+ - $ref: '#/components/schemas/CreateGenerationEvent'+ required:+ - type+ title: CreateGenerationEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - generation-update+ - $ref: '#/components/schemas/UpdateGenerationEvent'+ required:+ - type+ title: UpdateGenerationEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - event-create+ - $ref: '#/components/schemas/CreateEventEvent'+ required:+ - type+ title: CreateEventEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - sdk-log+ - $ref: '#/components/schemas/SDKLogEvent'+ required:+ - type+ title: SDKLogEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - observation-create+ - $ref: '#/components/schemas/CreateObservationEvent'+ required:+ - type+ title: CreateObservationEvent+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - observation-update+ - $ref: '#/components/schemas/UpdateObservationEvent'+ required:+ - type+ title: UpdateObservationEvent+ ObservationType:+ title: ObservationType+ type: string+ enum:+ - SPAN+ - GENERATION+ - EVENT+ - AGENT+ - TOOL+ - CHAIN+ - RETRIEVER+ - EVALUATOR+ - EMBEDDING+ - GUARDRAIL+ IngestionUsage:+ title: IngestionUsage+ oneOf:+ - $ref: '#/components/schemas/Usage'+ - $ref: '#/components/schemas/OpenAIUsage'+ OpenAIUsage:+ title: OpenAIUsagetype: object+ description: Usage interface of OpenAI for improved compatibility.properties:- data:- type: array- items:- $ref: '#/components/schemas/Project'- required:- - data- Organization:- title: Organization+ promptTokens:+ type: integer+ nullable: true+ completionTokens:+ type: integer+ nullable: true+ totalTokens:+ type: integer+ nullable: true+ OptionalObservationBody:+ title: OptionalObservationBodytype: objectproperties:- id:+ traceId:type: string- description: The unique identifier of the organization+ nullable: truename:type: string- description: The name of the organization- required:- - id- - name- Project:- title: Project- type: object- properties:- id:- type: string- name:+ nullable: true+ startTime:type: string- organization:- $ref: '#/components/schemas/Organization'- description: The organization this project belongs to+ format: date-time+ nullable: truemetadata:- type: object- additionalProperties: true- description: Metadata for the project- retentionDays:- type: integernullable: true- description: >-- Number of days to retain data. Null or 0 means no retention. Omitted- if no retention is configured.- required:- - id- - name- - organization- - metadata- ProjectDeletionResponse:- title: ProjectDeletionResponse+ input:+ nullable: true+ output:+ nullable: true+ level:+ $ref: '#/components/schemas/ObservationLevel'+ nullable: true+ statusMessage:+ type: string+ nullable: true+ parentObservationId:+ type: string+ nullable: true+ version:+ type: string+ nullable: true+ environment:+ type: string+ nullable: true+ CreateEventBody:+ title: CreateEventBodytype: objectproperties:- success:- type: boolean- message:+ id:type: string- required:- - success- - message- ApiKeyList:- title: ApiKeyList+ nullable: true+ allOf:+ - $ref: '#/components/schemas/OptionalObservationBody'+ UpdateEventBody:+ title: UpdateEventBodytype: object- description: List of API keys for a projectproperties:- apiKeys:- type: array- items:- $ref: '#/components/schemas/ApiKeySummary'+ id:+ type: stringrequired:- - apiKeys- ApiKeySummary:- title: ApiKeySummary+ - id+ allOf:+ - $ref: '#/components/schemas/OptionalObservationBody'+ CreateSpanBody:+ title: CreateSpanBodytype: object- description: Summary of an API keyproperties:- id:- type: string- createdAt:+ endTime:type: stringformat: date-time- expiresAt:+ nullable: true+ allOf:+ - $ref: '#/components/schemas/CreateEventBody'+ UpdateSpanBody:+ title: UpdateSpanBody+ type: object+ properties:+ endTime:type: stringformat: date-timenullable: true- lastUsedAt:+ allOf:+ - $ref: '#/components/schemas/UpdateEventBody'+ CreateGenerationBody:+ title: CreateGenerationBody+ type: object+ properties:+ completionStartTime:type: stringformat: date-timenullable: true- note:+ model:type: stringnullable: true- publicKey:- type: string- displaySecretKey:+ modelParameters:+ type: object+ additionalProperties:+ $ref: '#/components/schemas/MapValue'+ nullable: true+ usage:+ $ref: '#/components/schemas/IngestionUsage'+ nullable: true+ usageDetails:+ $ref: '#/components/schemas/UsageDetails'+ nullable: true+ costDetails:+ type: object+ additionalProperties:+ type: number+ format: double+ nullable: true+ promptName:type: string- required:- - id- - createdAt- - publicKey- - displaySecretKey- ApiKeyResponse:- title: ApiKeyResponse+ nullable: true+ promptVersion:+ type: integer+ nullable: true+ allOf:+ - $ref: '#/components/schemas/CreateSpanBody'+ UpdateGenerationBody:+ title: UpdateGenerationBodytype: object- description: Response for API key creationproperties:- id:- type: string- createdAt:+ completionStartTime:type: stringformat: date-time- publicKey:- type: string- secretKey:- type: string- displaySecretKey:- type: string- note:- type: stringnullable: true- required:- - id- - createdAt- - publicKey- - secretKey- - displaySecretKey- ApiKeyDeletionResponse:- title: ApiKeyDeletionResponse- type: object- description: Response for API key deletion- properties:- success:- type: boolean- required:- - success- PromptMetaListResponse:- title: PromptMetaListResponse- type: object- properties:- data:- type: array- items:- $ref: '#/components/schemas/PromptMeta'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- required:- - data- - meta- PromptMeta:- title: PromptMeta- type: object- properties:- name:+ model:type: string- type:- $ref: '#/components/schemas/PromptType'- description: Indicates whether the prompt is a text or chat prompt.- versions:- type: array- items:- type: integer- labels:- type: array- items:- type: string- tags:- type: array- items:- type: string- lastUpdatedAt:+ nullable: true+ modelParameters:+ type: object+ additionalProperties:+ $ref: '#/components/schemas/MapValue'+ nullable: true+ usage:+ $ref: '#/components/schemas/IngestionUsage'+ nullable: true+ promptName:type: string- format: date-time- lastConfig:- description: >-- Config object of the most recent prompt version that matches the- filters (if any are provided)- required:- - name- - type- - versions- - labels- - tags- - lastUpdatedAt- - lastConfig- CreatePromptRequest:- title: CreatePromptRequest- oneOf:- - $ref: '#/components/schemas/CreateChatPromptRequest'- - $ref: '#/components/schemas/CreateTextPromptRequest'- CreateChatPromptRequest:- title: CreateChatPromptRequest+ nullable: true+ usageDetails:+ $ref: '#/components/schemas/UsageDetails'+ nullable: true+ costDetails:+ type: object+ additionalProperties:+ type: number+ format: double+ nullable: true+ promptVersion:+ type: integer+ nullable: true+ allOf:+ - $ref: '#/components/schemas/UpdateSpanBody'+ ObservationBody:+ title: ObservationBodytype: objectproperties:- name:+ id:+ type: string+ nullable: true+ traceId:type: string- prompt:- type: array- items:- $ref: '#/components/schemas/ChatMessageWithPlaceholders'- config:nullable: truetype:- $ref: '#/components/schemas/CreateChatPromptType'- labels:- type: array- items:- type: string+ $ref: '#/components/schemas/ObservationType'+ name:+ type: stringnullable: true- description: List of deployment labels of this prompt version.- tags:- type: array- items:- type: string+ startTime:+ type: string+ format: date-timenullable: true- description: List of tags to apply to all versions of this prompt.- commitMessage:+ endTime:+ type: string+ format: date-time+ nullable: true+ completionStartTime:+ type: string+ format: date-time+ nullable: true+ model:+ type: string+ nullable: true+ modelParameters:+ type: object+ additionalProperties:+ $ref: '#/components/schemas/MapValue'+ nullable: true+ input:+ nullable: true+ version:+ type: string+ nullable: true+ metadata:+ nullable: true+ output:+ nullable: true+ usage:+ $ref: '#/components/schemas/Usage'+ nullable: true+ level:+ $ref: '#/components/schemas/ObservationLevel'+ nullable: true+ statusMessage:+ type: string+ nullable: true+ parentObservationId:+ type: string+ nullable: true+ environment:type: stringnullable: true- description: Commit message for this prompt version.required:- - name- - prompt- type- CreateTextPromptRequest:- title: CreateTextPromptRequest+ TraceBody:+ title: TraceBodytype: objectproperties:+ id:+ type: string+ nullable: true+ timestamp:+ type: string+ format: date-time+ nullable: truename:type: string- prompt:+ nullable: true+ userId:type: string- config:nullable: true- type:- $ref: '#/components/schemas/CreateTextPromptType'+ input:nullable: true- labels:- type: array- items:- type: string+ output:+ nullable: true+ sessionId:+ type: string+ nullable: true+ release:+ type: string+ nullable: true+ version:+ type: string+ nullable: true+ metadata:nullable: true- description: List of deployment labels of this prompt version.tags:type: arrayitems:type: stringnullable: true- description: List of tags to apply to all versions of this prompt.- commitMessage:+ environment:type: stringnullable: true- description: Commit message for this prompt version.+ public:+ type: boolean+ nullable: true+ description: Make trace publicly accessible via url+ SDKLogBody:+ title: SDKLogBody+ type: object+ properties:+ log: {}required:- - name- - prompt- Prompt:- title: Prompt- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - chat- - $ref: '#/components/schemas/ChatPrompt'- required:- - type- title: ChatPrompt- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - text- - $ref: '#/components/schemas/TextPrompt'- required:- - type- title: TextPrompt- PromptType:- title: PromptType- type: string- enum:- - chat- - text- BasePrompt:- title: BasePrompt+ - log+ ScoreBody:+ title: ScoreBodytype: objectproperties:+ id:+ type: string+ nullable: true+ traceId:+ type: string+ nullable: true+ sessionId:+ type: string+ nullable: true+ observationId:+ type: string+ nullable: true+ datasetRunId:+ type: string+ nullable: truename:type: string- version:- type: integer- config: {}- labels:- type: array- items:- type: string- description: List of deployment labels of this prompt version.- tags:- type: array- items:- type: stringdescription: >-- List of tags. Used to filter via UI and API. The same across- versions of a prompt.- commitMessage:+ The name of the score. Always overrides "output" for correction+ scores.+ environment:type: stringnullable: true- description: Commit message for this prompt version.- resolutionGraph:- type: object- additionalProperties: true+ queueId:+ type: stringnullable: truedescription: >-- The dependency resolution graph for the current prompt. Null if the- prompt has no dependencies or if `resolve=false` was used.- required:- - name- - version- - config- - labels- - tags- ChatMessageWithPlaceholders:- title: ChatMessageWithPlaceholders- oneOf:- - $ref: '#/components/schemas/ChatMessage'- - $ref: '#/components/schemas/PlaceholderMessage'- ChatMessage:- title: ChatMessage- type: object- properties:- role:+ The annotation queue referenced by the score. Indicates if score was+ initially created while processing annotation queue.+ value:+ $ref: '#/components/schemas/CreateScoreValue'+ description: >-+ The value of the score. Must be passed as string for categorical and+ text scores, and numeric for boolean and numeric scores. Boolean+ score values must equal either 1 or 0 (true or false). Text score+ values must be between 1 and 500 characters.+ comment:type: string- content:+ nullable: true+ metadata:+ nullable: true+ dataType:+ $ref: '#/components/schemas/ScoreDataType'+ nullable: true+ description: >-+ When set, must match the score value's type. If not set, will be+ inferred from the score value or config+ configId:type: string- type:- $ref: '#/components/schemas/ChatMessageType'nullable: true+ description: >-+ Reference a score config on a score. When set, the score name must+ equal the config name and scores must comply with the config's range+ and data type. For categorical scores, the value must map to a+ config category. Numeric scores might be constrained by the score+ config's max and min valuesrequired:- - role- - content- ChatMessageType:- title: ChatMessageType- type: string- enum:- - chatmessage- PlaceholderMessage:- title: PlaceholderMessage+ - name+ - value+ BaseEvent:+ title: BaseEventtype: objectproperties:- name:+ id:type: string- type:- $ref: '#/components/schemas/PlaceholderMessageType'+ description: UUID v4 that identifies the event+ timestamp:+ type: string+ description: >-+ Datetime (ISO 8601) of event creation in client. Should be as close+ to actual event creation in client as possible, this timestamp will+ be used for ordering of events in future release. Resolution:+ milliseconds (required), microseconds (optimal).+ metadata:nullable: true+ description: Optional. Metadata field used by the Langfuse SDKs for debugging.required:- - name- PlaceholderMessageType:- title: PlaceholderMessageType- type: string- enum:- - placeholder- TextPrompt:- title: TextPrompt+ - id+ - timestamp+ TraceEvent:+ title: TraceEventtype: objectproperties:- prompt:- type: string+ body:+ $ref: '#/components/schemas/TraceBody'required:- - prompt+ - bodyallOf:- - $ref: '#/components/schemas/BasePrompt'- ChatPrompt:- title: ChatPrompt+ - $ref: '#/components/schemas/BaseEvent'+ CreateObservationEvent:+ title: CreateObservationEventtype: objectproperties:- prompt:- type: array- items:- $ref: '#/components/schemas/ChatMessageWithPlaceholders'+ body:+ $ref: '#/components/schemas/ObservationBody'required:- - prompt+ - bodyallOf:- - $ref: '#/components/schemas/BasePrompt'- CreateChatPromptType:- title: CreateChatPromptType- type: string- enum:- - chat- CreateTextPromptType:- title: CreateTextPromptType- type: string- enum:- - text- ServiceProviderConfig:- title: ServiceProviderConfig+ - $ref: '#/components/schemas/BaseEvent'+ UpdateObservationEvent:+ title: UpdateObservationEventtype: objectproperties:- schemas:- type: array- items:- type: string- documentationUri:- type: string- patch:- $ref: '#/components/schemas/ScimFeatureSupport'- bulk:- $ref: '#/components/schemas/BulkConfig'- filter:- $ref: '#/components/schemas/FilterConfig'- changePassword:- $ref: '#/components/schemas/ScimFeatureSupport'- sort:- $ref: '#/components/schemas/ScimFeatureSupport'- etag:- $ref: '#/components/schemas/ScimFeatureSupport'- authenticationSchemes:- type: array- items:- $ref: '#/components/schemas/AuthenticationScheme'- meta:- $ref: '#/components/schemas/ResourceMeta'+ body:+ $ref: '#/components/schemas/ObservationBody'required:- - schemas- - documentationUri- - patch- - bulk- - filter- - changePassword- - sort- - etag- - authenticationSchemes- - meta- ScimFeatureSupport:- title: ScimFeatureSupport+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ ScoreEvent:+ title: ScoreEventtype: objectproperties:- supported:- type: boolean+ body:+ $ref: '#/components/schemas/ScoreBody'required:- - supported- BulkConfig:- title: BulkConfig+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ SDKLogEvent:+ title: SDKLogEventtype: objectproperties:- supported:- type: boolean- maxOperations:- type: integer- maxPayloadSize:- type: integer+ body:+ $ref: '#/components/schemas/SDKLogBody'required:- - supported- - maxOperations- - maxPayloadSize- FilterConfig:- title: FilterConfig+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ CreateGenerationEvent:+ title: CreateGenerationEventtype: objectproperties:- supported:- type: boolean- maxResults:- type: integer+ body:+ $ref: '#/components/schemas/CreateGenerationBody'required:- - supported- - maxResults- ResourceMeta:- title: ResourceMeta+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ UpdateGenerationEvent:+ title: UpdateGenerationEventtype: objectproperties:- resourceType:- type: string- location:- type: string+ body:+ $ref: '#/components/schemas/UpdateGenerationBody'required:- - resourceType- - location- AuthenticationScheme:- title: AuthenticationScheme+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ CreateSpanEvent:+ title: CreateSpanEventtype: objectproperties:- name:- type: string- description:- type: string- specUri:- type: string- type:- type: string- primary:- type: boolean+ body:+ $ref: '#/components/schemas/CreateSpanBody'required:- - name- - description- - specUri- - type- - primary- ResourceTypesResponse:- title: ResourceTypesResponse+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ UpdateSpanEvent:+ title: UpdateSpanEventtype: objectproperties:- schemas:- type: array- items:- type: string- totalResults:- type: integer- Resources:- type: array- items:- $ref: '#/components/schemas/ResourceType'+ body:+ $ref: '#/components/schemas/UpdateSpanBody'required:- - schemas- - totalResults- - Resources- ResourceType:- title: ResourceType+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ CreateEventEvent:+ title: CreateEventEvent+ type: object+ properties:+ body:+ $ref: '#/components/schemas/CreateEventBody'+ required:+ - body+ allOf:+ - $ref: '#/components/schemas/BaseEvent'+ IngestionSuccess:+ title: IngestionSuccesstype: objectproperties:- schemas:- type: array- items:- type: string- nullable: trueid:type: string- name:- type: string- endpoint:- type: string- description:- type: string- schema:- type: string- schemaExtensions:- type: array- items:- $ref: '#/components/schemas/SchemaExtension'- meta:- $ref: '#/components/schemas/ResourceMeta'+ status:+ type: integerrequired:- id- - name- - endpoint- - description- - schema- - schemaExtensions- - meta- SchemaExtension:- title: SchemaExtension+ - status+ IngestionError:+ title: IngestionErrortype: objectproperties:- schema:+ id:type: string- required:- type: boolean+ status:+ type: integer+ message:+ type: string+ nullable: true+ error:+ nullable: truerequired:- - schema- - required- SchemasResponse:- title: SchemasResponse+ - id+ - status+ IngestionResponse:+ title: IngestionResponsetype: objectproperties:- schemas:+ successes:type: arrayitems:- type: string- totalResults:- type: integer- Resources:+ $ref: '#/components/schemas/IngestionSuccess'+ errors:type: arrayitems:- $ref: '#/components/schemas/SchemaResource'+ $ref: '#/components/schemas/IngestionError'required:- - schemas- - totalResults- - Resources- SchemaResource:- title: SchemaResource+ - successes+ - errors+ OpenAICompletionUsageSchema:+ title: OpenAICompletionUsageSchematype: object+ description: OpenAI Usage schema from (Chat-)Completion APIsproperties:- id:- type: string- name:- type: string- description:- type: string- attributes:- type: array- items: {}- meta:- $ref: '#/components/schemas/ResourceMeta'+ prompt_tokens:+ type: integer+ completion_tokens:+ type: integer+ total_tokens:+ type: integer+ prompt_tokens_details:+ type: object+ additionalProperties:+ type: integer+ nullable: true+ nullable: true+ completion_tokens_details:+ type: object+ additionalProperties:+ type: integer+ nullable: true+ nullable: truerequired:- - id- - name- - description- - attributes- - meta- ScimUsersListResponse:- title: ScimUsersListResponse+ - prompt_tokens+ - completion_tokens+ - total_tokens+ OpenAIResponseUsageSchema:+ title: OpenAIResponseUsageSchematype: object+ description: OpenAI Usage schema from Response APIproperties:- schemas:- type: array- items:- type: string- totalResults:+ input_tokens:type: integer- startIndex:+ output_tokens:type: integer- itemsPerPage:+ total_tokens:type: integer- Resources:+ input_tokens_details:+ type: object+ additionalProperties:+ type: integer+ nullable: true+ nullable: true+ output_tokens_details:+ type: object+ additionalProperties:+ type: integer+ nullable: true+ nullable: true+ required:+ - input_tokens+ - output_tokens+ - total_tokens+ UsageDetails:+ title: UsageDetails+ oneOf:+ - type: object+ additionalProperties:+ type: integer+ title: Object+ - $ref: '#/components/schemas/OpenAICompletionUsageSchema'+ - $ref: '#/components/schemas/OpenAIResponseUsageSchema'+ legacyMetricsResponse:+ title: legacyMetricsResponse+ type: object+ properties:+ data:type: arrayitems:- $ref: '#/components/schemas/ScimUser'+ type: object+ additionalProperties: true+ description: >-+ The metrics data. Each item in the list contains the metric values+ and dimensions requested in the query.++ Format varies based on the query parameters.++ Histograms will return an array with [lower, upper, height] tuples.+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: truerequired:- - schemas- - totalResults- - startIndex- - itemsPerPage- - Resources- ScimUser:- title: ScimUser+ - data+ legacyObservations:+ title: legacyObservationstype: objectproperties:- schemas:+ data:type: arrayitems:- type: string- id:- type: string- userName:- type: string- name:- $ref: '#/components/schemas/ScimName'- emails:+ $ref: '#/components/schemas/Observation'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ required:+ - data+ - meta+ legacyObservationsViews:+ title: legacyObservationsViews+ type: object+ properties:+ data:type: arrayitems:- $ref: '#/components/schemas/ScimEmail'+ $ref: '#/components/schemas/ObservationsView'meta:- $ref: '#/components/schemas/UserMeta'+ $ref: '#/components/schemas/utilsMetaResponse'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: truerequired:- - schemas- - id- - userName- - name- - emails+ - data- meta- UserMeta:- title: UserMeta+ LlmConnection:+ title: LlmConnectiontype: object+ description: LLM API connection configuration (secrets excluded)properties:- resourceType:+ id:type: string- created:+ provider:type: string- nullable: true- lastModified:+ description: >-+ Provider name (e.g., 'openai', 'my-gateway'). Must be unique in+ project, used for upserting.+ adapter:type: string- nullable: true- required:- - resourceType- ScimName:- title: ScimName- type: object- properties:- formatted:+ description: The adapter used to interface with the LLM+ displaySecretKey:+ type: string+ description: Masked version of the secret key for display purposes+ baseURL:type: stringnullable: true- ScimEmail:- title: ScimEmail- type: object- properties:- primary:+ description: Custom base URL for the LLM API+ customModels:+ type: array+ items:+ type: string+ description: List of custom model names available for this connection+ withDefaultModels:type: boolean- value:+ description: Whether to include default models for this adapter+ extraHeaderKeys:+ type: array+ items:+ type: string+ description: >-+ Keys of extra headers sent with requests (values excluded for+ security)+ config:+ type: object+ additionalProperties: true+ nullable: true+ description: >-+ Adapter-specific configuration. Required for Bedrock+ (`{"region":"us-east-1"}`), optional for OpenAI+ (`{"useResponsesApi":true}`), optional for VertexAI+ (`{"location":"us-central1"}`), not used by other adapters.+ createdAt:type: string- type:+ format: date-time+ updatedAt:type: string+ format: date-timerequired:- - primary- - value- - type- EmptyResponse:- title: EmptyResponse- type: object- description: Empty response for 204 No Content responses- properties: {}- ScoreConfigs:- title: ScoreConfigs+ - id+ - provider+ - adapter+ - displaySecretKey+ - customModels+ - withDefaultModels+ - extraHeaderKeys+ - createdAt+ - updatedAt+ PaginatedLlmConnections:+ title: PaginatedLlmConnectionstype: objectproperties:data:type: arrayitems:- $ref: '#/components/schemas/ScoreConfig'+ $ref: '#/components/schemas/LlmConnection'meta:$ref: '#/components/schemas/utilsMetaResponse'required:- data- meta- CreateScoreConfigRequest:- title: CreateScoreConfigRequest+ UpsertLlmConnectionRequest:+ title: UpsertLlmConnectionRequesttype: object+ description: Request to create or update an LLM connection (upsert)properties:- name:+ provider:type: stringdescription: >-- Name of the score config. Max 35 characters. Only letters, numbers,- underscores, spaces, periods, parentheses, and hyphens are allowed.- dataType:- $ref: '#/components/schemas/ScoreConfigDataType'- categories:- type: array- items:- $ref: '#/components/schemas/ConfigCategory'- nullable: true- description: >-- Configure custom categories for categorical scores. Pass a list of- objects with `label` and `value` properties. Categories are- autogenerated for boolean configs and cannot be passed- minValue:- type: number- format: double- nullable: true- description: >-- Configure a minimum value for numerical scores. If not set, the- minimum value defaults to -∞- maxValue:- type: number- format: double- nullable: true- description: >-- Configure a maximum value for numerical scores. If not set, the- maximum value defaults to +∞- description:+ Provider name (e.g., 'openai', 'my-gateway'). Must be unique in+ project, used for upserting.+ adapter:+ $ref: '#/components/schemas/LlmAdapter'+ description: The adapter used to interface with the LLM+ secretKey:type: string- nullable: true- description: >-- Description is shown across the Langfuse UI and can be used to e.g.- explain the config categories in detail, why a numeric range was- set, or provide additional context on config name or usage- required:- - name- - dataType- UpdateScoreConfigRequest:- title: UpdateScoreConfigRequest- type: object- properties:- isArchived:- type: boolean- nullable: true- description: The status of the score config showing if it is archived or not- name:+ description: Secret key for the LLM API.+ baseURL:type: stringnullable: true- description: >-- Name of the score config. Max 35 characters. Only letters, numbers,- underscores, spaces, periods, parentheses, and hyphens are allowed.- categories:+ description: Custom base URL for the LLM API+ customModels:type: arrayitems:- $ref: '#/components/schemas/ConfigCategory'+ type: stringnullable: true- description: >-- Configure custom categories for categorical scores. Pass a list of- objects with `label` and `value` properties. Categories are- autogenerated for boolean configs and cannot be passed- minValue:- type: number- format: double+ description: List of custom model names+ withDefaultModels:+ type: booleannullable: true- description: >-- Configure a minimum value for numerical scores. If not set, the- minimum value defaults to -∞- maxValue:- type: number- format: double+ description: Whether to include default models. Default is true.+ extraHeaders:+ type: object+ additionalProperties:+ type: stringnullable: true- description: >-- Configure a maximum value for numerical scores. If not set, the- maximum value defaults to +∞- description:- type: string+ description: Extra headers to send with requests+ config:+ type: object+ additionalProperties: truenullable: truedescription: >-- Description is shown across the Langfuse UI and can be used to e.g.- explain the config categories in detail, why a numeric range was- set, or provide additional context on config name or usage- ScoreSubjectTraceV3:- title: ScoreSubjectTraceV3- type: object- properties:- id:- type: string- description: The trace ID.- required:- - id- ScoreSubjectObservationV3:- title: ScoreSubjectObservationV3- type: object- properties:- id:- type: string- description: The observation ID.- traceId:- type: string- nullable: true- description: The parent trace ID, if available.- required:- - id- ScoreSubjectSessionV3:- title: ScoreSubjectSessionV3- type: object- properties:- id:- type: string- description: The session ID.+ Adapter-specific configuration. Validation rules: - **Bedrock**:+ Required. Must be `{"region": "<aws-region>"}` (e.g.,+ `{"region":"us-east-1"}`) - **OpenAI**: Optional. If provided, must+ be `{"useResponsesApi": <boolean>}` to control whether Langfuse+ routes calls through OpenAI's Responses API. - **VertexAI**:+ Optional. If provided, must be `{"location": "<gcp-location>"}`+ (e.g., `{"location":"us-central1"}`) - **Other adapters**: Not+ supported. Omit this field or set to null.required:- - id- ScoreSubjectExperimentV3:- title: ScoreSubjectExperimentV3+ - provider+ - adapter+ - secretKey+ DeleteLlmConnectionResponse:+ title: DeleteLlmConnectionResponsetype: objectproperties:- id:+ message:type: string- description: The dataset run ID (experiment ID).required:- - id- ScoreSubjectV3:- title: ScoreSubjectV3- oneOf:- - type: object- allOf:- - type: object- properties:- kind:- type: string- enum:- - trace- - $ref: '#/components/schemas/ScoreSubjectTraceV3'- required:- - kind- title: ScoreSubjectTraceV3- - type: object- allOf:- - type: object- properties:- kind:- type: string- enum:- - observation- - $ref: '#/components/schemas/ScoreSubjectObservationV3'- required:- - kind- title: ScoreSubjectObservationV3- - type: object- allOf:- - type: object- properties:- kind:- type: string- enum:- - session- - $ref: '#/components/schemas/ScoreSubjectSessionV3'- required:- - kind- title: ScoreSubjectSessionV3- - type: object- allOf:- - type: object- properties:- kind:- type: string- enum:- - experiment- - $ref: '#/components/schemas/ScoreSubjectExperimentV3'- required:- - kind- title: ScoreSubjectExperimentV3- description: >-- A reference to the entity this score is attached to. Discriminated by- "kind" — one of trace, observation, session, or experiment.- BaseScoreV3:- title: BaseScoreV3+ - message+ LlmAdapter:+ title: LlmAdapter+ type: string+ enum:+ - anthropic+ - openai+ - azure+ - bedrock+ - google-vertex-ai+ - google-ai-studio+ GetMediaResponse:+ title: GetMediaResponsetype: objectproperties:- id:- type: string- projectId:+ mediaId:type: string- name:+ description: The unique langfuse identifier of a media record+ contentType:type: string- source:- $ref: '#/components/schemas/ScoreSource'- timestamp:+ description: The MIME type of the media record+ contentLength:+ type: integer+ description: The size of the media record in bytes+ uploadedAt:type: stringformat: date-time- environment:+ description: The date and time when the media record was uploaded+ url:type: string- description: The environment from which this score originated.- createdAt:+ description: The download URL of the media record+ urlExpiry:type: string- format: date-time- updatedAt:+ description: The expiry date and time of the media record download URL+ required:+ - mediaId+ - contentType+ - contentLength+ - uploadedAt+ - url+ - urlExpiry+ PatchMediaBody:+ title: PatchMediaBody+ type: object+ properties:+ uploadedAt:type: stringformat: date-time- comment:+ description: The date and time when the media record was uploaded+ uploadHttpStatus:+ type: integer+ description: The HTTP status code of the upload+ uploadHttpError:type: stringnullable: true- description: >-- Optional comment attached to the score. Present when "details" is- included in the fields parameter.- configId:- type: string+ description: The HTTP error message of the upload+ uploadTimeMs:+ type: integernullable: true- description: >-- The score config ID, if this score was created from a config.- Present when "details" is included in the fields parameter.- metadata:- type: object- additionalProperties: true+ description: The time in milliseconds it took to upload the media record+ required:+ - uploadedAt+ - uploadHttpStatus+ GetMediaUploadUrlRequest:+ title: GetMediaUploadUrlRequest+ type: object+ description: >-+ Request a presigned media upload URL. Provide exactly one context: a+ trace (traceId, optionally observationId) or a dataset item (datasetId ++ datasetItemId). field is required and must match the chosen context.+ properties:+ traceId:+ type: stringnullable: truedescription: >-- Arbitrary metadata attached to the score. Present when "details" is- included in the fields parameter.- authorUserId:+ The trace the media is associated with. Null for dataset item media+ uploads.+ observationId:type: stringnullable: truedescription: >-- The user who created this score, if available. Present when- "annotation" is included in the fields parameter.- queueId:+ The observation ID associated with the media record. If the media+ record is associated directly with a trace, this will be null.+ datasetId:type: stringnullable: truedescription: >-- The annotation queue this score belongs to, if any. Present when- "annotation" is included in the fields parameter.- subject:- $ref: '#/components/schemas/ScoreSubjectV3'+ The dataset the media belongs to. Null for trace/observation media+ uploads.+ datasetItemId:+ type: stringnullable: truedescription: >-- The entity this score is attached to (trace, observation, session,- or experiment). Present when "subject" is included in the fields- parameter.- required:- - id- - projectId- - name- - source- - timestamp- - environment- - createdAt- - updatedAt- NumericScoreV3:- title: NumericScoreV3- type: object- properties:- value:- type: number- format: double- description: The numeric value of the score.- required:- - value- allOf:- - $ref: '#/components/schemas/BaseScoreV3'- BooleanScoreV3:- title: BooleanScoreV3- type: object- properties:- value:- type: boolean- description: The boolean value of the score.- required:- - value- allOf:- - $ref: '#/components/schemas/BaseScoreV3'- CategoricalScoreV3:- title: CategoricalScoreV3- type: object- properties:- value:+ The dataset item the media is associated with (need not exist yet).+ Null for trace/observation media uploads.+ contentType:+ $ref: '#/components/schemas/MediaContentType'+ contentLength:+ type: integer+ description: The size of the media record in bytes+ sha256Hash:type: string- description: The string category value of the score.- required:- - value- allOf:- - $ref: '#/components/schemas/BaseScoreV3'- TextScoreV3:- title: TextScoreV3- type: object- properties:- value:+ description: The SHA-256 hash of the media record+ field:type: string- description: The text content of the score.+ description: >-+ The item field the media is in: `input`/`output`/`metadata` (trace)+ or `input`/`expectedOutput`/`metadata` (dataset item).required:- - value- allOf:- - $ref: '#/components/schemas/BaseScoreV3'- CorrectionScoreV3:- title: CorrectionScoreV3+ - contentType+ - contentLength+ - sha256Hash+ - field+ GetMediaUploadUrlResponse:+ title: GetMediaUploadUrlResponsetype: objectproperties:- value:+ uploadUrl:type: string- description: The correction content of the score. Empty string if not set.+ nullable: true+ description: >-+ The presigned upload URL. If the asset is already uploaded, this+ will be null+ mediaId:+ type: string+ description: The unique langfuse identifier of a media recordrequired:- - value- allOf:- - $ref: '#/components/schemas/BaseScoreV3'- ScoreV3:- title: ScoreV3- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: '#/components/schemas/NumericScoreV3'- required:- - dataType- title: NumericScoreV3- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: '#/components/schemas/BooleanScoreV3'- required:- - dataType- title: BooleanScoreV3- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: '#/components/schemas/CategoricalScoreV3'- required:- - dataType- title: CategoricalScoreV3- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - TEXT- - $ref: '#/components/schemas/TextScoreV3'- required:- - dataType- title: TextScoreV3- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CORRECTION- - $ref: '#/components/schemas/CorrectionScoreV3'- required:- - dataType- title: CorrectionScoreV3- GetScoresV3Meta:- title: GetScoresV3Meta+ - mediaId+ MediaContentType:+ title: MediaContentType+ type: string+ enum:+ - image/png+ - image/jpeg+ - image/jpg+ - image/webp+ - image/gif+ - image/svg+xml+ - image/tiff+ - image/bmp+ - image/avif+ - image/heic+ - audio/mpeg+ - audio/mp3+ - audio/wav+ - audio/ogg+ - audio/oga+ - audio/aac+ - audio/mp4+ - audio/flac+ - audio/opus+ - audio/webm+ - video/mp4+ - video/webm+ - video/ogg+ - video/mpeg+ - video/quicktime+ - video/x-msvideo+ - video/x-matroska+ - text/plain+ - text/html+ - text/css+ - text/csv+ - text/markdown+ - text/x-python+ - application/javascript+ - text/x-typescript+ - application/x-yaml+ - application/pdf+ - application/msword+ - application/vnd.ms-excel+ - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet+ - application/zip+ - application/json+ - application/xml+ - application/octet-stream+ - >-+ application/vnd.openxmlformats-officedocument.wordprocessingml.document+ - >-+ application/vnd.openxmlformats-officedocument.presentationml.presentation+ - application/rtf+ - application/x-ndjson+ - application/vnd.apache.parquet+ - application/gzip+ - application/x-tar+ - application/x-7z-compressed+ description: The MIME type of the media record+ MetricsV2Response:+ title: MetricsV2Responsetype: objectproperties:- limit:- type: integer- cursor:- type: string- nullable: true+ data:+ type: array+ items:+ type: object+ additionalProperties: truedescription: >-- URL-safe base64 (base64url) cursor for the next page. Absent when- there are no more results.+ The metrics data. Each item in the list contains the metric values+ and dimensions requested in the query.++ Format varies based on the query parameters.++ Histograms will return an array with [lower, upper, height] tuples.required:- - limit- GetScoresV3Response:- title: GetScoresV3Response+ - data+ ModelTokenizerId:+ title: ModelTokenizerId+ type: string+ enum:+ - openai+ - claude+ description: Tokenizer supported by Langfuse for model usage inference.+ PaginatedModels:+ title: PaginatedModelstype: objectproperties:data:type: arrayitems:- $ref: '#/components/schemas/ScoreV3'+ $ref: '#/components/schemas/Model'meta:- $ref: '#/components/schemas/GetScoresV3Meta'+ $ref: '#/components/schemas/utilsMetaResponse'required:- data- meta- CreateScoreRequest:- title: CreateScoreRequest+ CreateModelRequest:+ title: CreateModelRequesttype: objectproperties:- id:- type: string- nullable: true- traceId:+ modelName:type: string- nullable: true- sessionId:+ description: >-+ Name of the model definition. If multiple with the same name exist,+ they are applied in the following order: (1) custom over built-in,+ (2) newest according to startTime where+ model.startTime<observation.startTime+ matchPattern:type: string- nullable: true- observationId:+ description: >-+ Regex pattern which matches this model definition to+ generation.model. Useful in case of fine-tuned models. If you want+ to exact match, use `(?i)^modelname$`+ startDate:type: string+ format: date-timenullable: true- datasetRunId:- type: string+ description: Apply only to generations which are newer than this ISO date.+ unit:+ $ref: '#/components/schemas/ModelUsageUnit'+ description: Unit used by this model.+ inputPrice:+ type: number+ format: doublenullable: true- name:- type: string- value:- $ref: '#/components/schemas/CreateScoreValue'description: >-- The value of the score. Must be passed as string for categorical and- text scores, and numeric for boolean and numeric scores. Boolean- score values must equal either 1 or 0 (true or false). Text score- values must be between 1 and 500 characters.- comment:- type: string+ Deprecated. Use 'pricingTiers' instead. Price (USD) per input unit.+ Creates a default tier if pricingTiers not provided.+ outputPrice:+ type: number+ format: doublenullable: true- metadata:- type: object- additionalProperties: true+ description: >-+ Deprecated. Use 'pricingTiers' instead. Price (USD) per output unit.+ Creates a default tier if pricingTiers not provided.+ totalPrice:+ type: number+ format: doublenullable: true- environment:- type: string+ description: >-+ Deprecated. Use 'pricingTiers' instead. Price (USD) per total units.+ Cannot be set if input or output price is set. Creates a default+ tier if pricingTiers not provided.+ pricingTiers:+ type: array+ items:+ $ref: '#/components/schemas/PricingTierInput'nullable: truedescription: >-- The environment of the score. Can be any lowercase alphanumeric- string with hyphens and underscores that does not start with- 'langfuse'.- queueId:- type: string+ Optional. Array of pricing tiers for this model.+++ Use pricing tiers for all models - both those with threshold-based+ pricing variations and those with simple flat pricing:+++ - For models with standard flat pricing: Create a single default+ tier with your prices+ (e.g., one tier with isDefault=true, priority=0, conditions=[], and your standard prices)++ - For models with threshold-based pricing: Create a default tier+ plus additional conditional tiers+ (e.g., default tier for standard usage + high-volume tier for usage above certain thresholds)++ Requirements:++ - Cannot be provided with flat prices+ (inputPrice/outputPrice/totalPrice) - use one approach or the other++ - Must include exactly one default tier with isDefault=true,+ priority=0, and conditions=[]++ - All tier names and priorities must be unique within the model++ - Each tier must define at least one price+++ If omitted, you must provide flat prices instead+ (inputPrice/outputPrice/totalPrice),++ which will automatically create a single default tier named+ "Standard".+ tokenizerId:+ $ref: '#/components/schemas/ModelTokenizerId'nullable: truedescription: >-- The annotation queue referenced by the score. Indicates if score was- initially created while processing annotation queue.- dataType:- $ref: '#/components/schemas/ScoreDataType'+ Optional. Tokenizer to be applied to observations which match to+ this model. See docs for more details.+ tokenizerConfig:nullable: truedescription: >-- The data type of the score. When passing a configId this field is- inferred. Otherwise, this field must be passed or will default to- numeric.- configId:+ Optional. Configuration for the selected tokenizer. Needs to be+ JSON. See docs for more details.+ required:+ - modelName+ - matchPattern+ - unit+ ObservationsV2Response:+ title: ObservationsV2Response+ type: object+ description: >-+ Response containing observations with field-group-based filtering and+ cursor-based pagination.+++ The `data` array contains observation objects with only the requested+ field groups included.++ Use the `cursor` in `meta` to retrieve the next page of results.+ properties:+ data:+ type: array+ items:+ $ref: '#/components/schemas/ObservationV2'+ description: >-+ Array of observation objects. Fields included depend on the `fields`+ parameter in the request.+ meta:+ $ref: '#/components/schemas/ObservationsV2Meta'+ required:+ - data+ - meta+ ObservationsV2Meta:+ title: ObservationsV2Meta+ type: object+ description: Metadata for cursor-based pagination+ properties:+ cursor:type: stringnullable: truedescription: >-- Reference a score config on a score. The unique langfuse identifier- of a score config. When passing this field, the dataType and- stringValue fields are automatically populated.- source:- $ref: '#/components/schemas/CreateScoreSource'+ Base64-encoded cursor to use for retrieving the next page. If not+ present, there are no more results.+ OtelResourceSpan:+ title: OtelResourceSpan+ type: object+ description: >-+ Represents a collection of spans from a single resource as per OTLP+ specification+ properties:+ resource:+ $ref: '#/components/schemas/OtelResource'nullable: true- description: >-- The source of the score. Defaults to API. Set to ANNOTATION to- prefill scores (e.g. from an LLM) for a human reviewer to verify in- an annotation queue. When source is ANNOTATION, a configId is- required unless dataType is CORRECTION. EVAL is reserved for- internal evaluator outputs and is not accepted on this endpoint.- required:- - name- - value- CreateScoreSource:- title: CreateScoreSource- type: string- enum:- - API- - ANNOTATION- description: |-- Source values accepted when creating a score via the public REST API.- EVAL is reserved for internal evaluator outputs and is intentionally not- exposed here — use commons.ScoreSource when reading scores.- CreateScoreResponse:- title: CreateScoreResponse+ description: Resource information+ scopeSpans:+ type: array+ items:+ $ref: '#/components/schemas/OtelScopeSpan'+ nullable: true+ description: Array of scope spans+ OtelResource:+ title: OtelResourcetype: object+ description: Resource attributes identifying the source of telemetryproperties:- id:- type: string- description: The id of the created object in Langfuse- required:- - id- GetScoresResponseTraceData:- title: GetScoresResponseTraceData+ attributes:+ type: array+ items:+ $ref: '#/components/schemas/OtelAttribute'+ nullable: true+ description: Resource attributes like service.name, service.version, etc.+ OtelScopeSpan:+ title: OtelScopeSpantype: object+ description: Collection of spans from a single instrumentation scopeproperties:- userId:- type: string+ scope:+ $ref: '#/components/schemas/OtelScope'nullable: true- description: The user ID associated with the trace referenced by score- tags:+ description: Instrumentation scope information+ spans:type: arrayitems:- type: string+ $ref: '#/components/schemas/OtelSpan'nullable: true- description: A list of tags associated with the trace referenced by score- environment:+ description: Array of spans+ OtelScope:+ title: OtelScope+ type: object+ description: Instrumentation scope information+ properties:+ name:type: stringnullable: true- description: The environment of the trace referenced by score- sessionId:+ description: Instrumentation scope name+ version:type: stringnullable: true- description: The session ID associated with the trace referenced by score- GetScoresResponseDataNumeric:- title: GetScoresResponseDataNumeric- type: object- properties:- trace:- $ref: '#/components/schemas/GetScoresResponseTraceData'+ description: Instrumentation scope version+ attributes:+ type: array+ items:+ $ref: '#/components/schemas/OtelAttribute'nullable: true- allOf:- - $ref: '#/components/schemas/NumericScore'- GetScoresResponseDataCategorical:- title: GetScoresResponseDataCategorical+ description: Additional scope attributes+ OtelSpan:+ title: OtelSpantype: object+ description: Individual span representing a unit of work or operationproperties:- trace:- $ref: '#/components/schemas/GetScoresResponseTraceData'+ traceId:nullable: true- allOf:- - $ref: '#/components/schemas/CategoricalScore'- GetScoresResponseDataBoolean:- title: GetScoresResponseDataBoolean- type: object- properties:- trace:- $ref: '#/components/schemas/GetScoresResponseTraceData'+ description: Trace ID (16 bytes, hex-encoded string in JSON or Buffer in binary)+ spanId:nullable: true- allOf:- - $ref: '#/components/schemas/BooleanScore'- GetScoresResponseDataCorrection:- title: GetScoresResponseDataCorrection- type: object- properties:- trace:- $ref: '#/components/schemas/GetScoresResponseTraceData'+ description: Span ID (8 bytes, hex-encoded string in JSON or Buffer in binary)+ parentSpanId:nullable: true- allOf:- - $ref: '#/components/schemas/CorrectionScore'- GetScoresResponseDataText:- title: GetScoresResponseDataText- type: object- properties:- trace:- $ref: '#/components/schemas/GetScoresResponseTraceData'+ description: Parent span ID if this is a child span+ name:+ type: stringnullable: true- allOf:- - $ref: '#/components/schemas/TextScore'- GetScoresResponseData:- title: GetScoresResponseData- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: '#/components/schemas/GetScoresResponseDataNumeric'- required:- - dataType- title: GetScoresResponseDataNumeric- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: '#/components/schemas/GetScoresResponseDataCategorical'- required:- - dataType- title: GetScoresResponseDataCategorical- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: '#/components/schemas/GetScoresResponseDataBoolean'- required:- - dataType- title: GetScoresResponseDataBoolean- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CORRECTION- - $ref: '#/components/schemas/GetScoresResponseDataCorrection'- required:- - dataType- title: GetScoresResponseDataCorrection- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - TEXT- - $ref: '#/components/schemas/GetScoresResponseDataText'- required:- - dataType- title: GetScoresResponseDataText- GetScoresResponse:- title: GetScoresResponse- type: object- properties:- data:+ description: Span name describing the operation+ kind:+ type: integer+ nullable: true+ description: Span kind (1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER)+ startTimeUnixNano:+ nullable: true+ description: Start time in nanoseconds since Unix epoch+ endTimeUnixNano:+ nullable: true+ description: End time in nanoseconds since Unix epoch+ attributes:type: arrayitems:- $ref: '#/components/schemas/GetScoresResponseData'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- _deprecation:- $ref: '#/components/schemas/Deprecation'+ $ref: '#/components/schemas/OtelAttribute'nullable: true- required:- - data- - meta- PaginatedSessions:- title: PaginatedSessions+ description: >-+ Span attributes including Langfuse-specific attributes+ (langfuse.observation.*)+ status:+ nullable: true+ description: Span status object+ OtelAttribute:+ title: OtelAttributetype: object+ description: Key-value attribute pair for resources, scopes, or spansproperties:- data:- type: array- items:- $ref: '#/components/schemas/Session'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- _deprecation:- $ref: '#/components/schemas/Deprecation'+ key:+ type: stringnullable: true- required:- - data- - meta- Traces:- title: Traces+ description: Attribute key (e.g., "service.name", "langfuse.observation.type")+ value:+ $ref: '#/components/schemas/OtelAttributeValue'+ nullable: true+ description: Attribute value+ OtelAttributeValue:+ title: OtelAttributeValuetype: object+ description: Attribute value wrapper supporting different value typesproperties:- data:- type: array- items:- $ref: '#/components/schemas/TraceWithDetails'- meta:- $ref: '#/components/schemas/utilsMetaResponse'- _deprecation:- $ref: '#/components/schemas/Deprecation'+ stringValue:+ type: stringnullable: true- required:- - data- - meta- DeleteTraceResponse:- title: DeleteTraceResponse+ description: String value+ intValue:+ type: integer+ nullable: true+ description: Integer value+ doubleValue:+ type: number+ format: double+ nullable: true+ description: Double value+ boolValue:+ type: boolean+ nullable: true+ description: Boolean value+ OtelTraceResponse:+ title: OtelTraceResponse+ type: object+ description: Response from trace export request. Empty object indicates success.+ properties: {}+ MembershipRole:+ title: MembershipRole+ type: string+ enum:+ - OWNER+ - ADMIN+ - MEMBER+ - VIEWER+ MembershipRequest:+ title: MembershipRequesttype: objectproperties:- message:+ userId:type: string+ role:+ $ref: '#/components/schemas/MembershipRole'required:- - message- Sort:- title: Sort+ - userId+ - role+ DeleteMembershipRequest:+ title: DeleteMembershipRequesttype: objectproperties:- id:+ userId:type: stringrequired:- - id- unstableEvaluatorType:- title: unstableEvaluatorType- type: string- enum:- - llm_as_judge- - code- description: |-- The evaluator type.-- The unstable public API supports LLM-as-a-judge and code evaluators.- unstableCodeEvaluatorSourceCodeLanguage:- title: unstableCodeEvaluatorSourceCodeLanguage- type: string- enum:- - PYTHON- - TYPESCRIPT- description: Code evaluator runtime language.- unstableEvaluationRuleTarget:- title: unstableEvaluationRuleTarget- type: string- enum:- - observation- - experiment- description: >-- The ingestion object type that should trigger evaluation runs.--- Choose the target first, because it changes both the valid filter- columns and the valid variable-mapping sources:-- - `observation` evaluates live-ingested observations such as- generations, spans, and events.- It supports mapping from `input`, `output`, `metadata`, and `tool_calls`.- - `experiment` evaluates live experiment executions and can additionally- map `expected_output` and `experiment_item_metadata`.- It currently supports filtering by `datasetId`.- Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter.- unstableEvaluationRuleStatus:- title: unstableEvaluationRuleStatus- type: string- enum:- - active- - inactive- - paused- description: >-- Effective runtime status of the evaluation rule.--- - `active`: enabled and currently runnable.-- - `inactive`: disabled by configuration.-- - `paused`: enabled, but Langfuse has blocked execution until the- underlying issue is resolved.- unstablePromptVariableMappingSource:- title: unstablePromptVariableMappingSource- type: string- enum:- - input- - output- - metadata- - tool_calls- - expected_output- - experiment_item_metadata- description: >-- Source field used to populate a prompt variable.--- Use these values when mapping evaluator prompt variables to live data.--- Target-specific rules:-- - `target=observation` supports `input`, `output`, `metadata`, and- `tool_calls`-- - `target=experiment` supports `input`, `output`, `metadata`,- `tool_calls`, `expected_output`, and `experiment_item_metadata`--- Source semantics:-- - `input`: the observation or experiment input payload-- - `output`: the observation or experiment output payload-- - `metadata`: the metadata object for the target. Combine with- `jsonPath` when you need one nested field instead of the whole object.-- - `tool_calls`: the tool calls recorded on the observation, as an array- of `{id, name, arguments, type, index}` objects in the order the model- emitted them. Combine with `jsonPath` (for example `$[*].name`) to- select parts of each call.-- - `expected_output`: the experiment item's expected output. Only valid- for `target=experiment`.-- - `experiment_item_metadata`: the experiment item's metadata object.- Only valid for `target=experiment`.- unstableEvaluatorModelConfig:- title: unstableEvaluatorModelConfig+ - userId+ MembershipResponse:+ title: MembershipResponsetype: object- description: >-- Optional explicit model configuration for an evaluator.--- If omitted, Langfuse uses the project's default evaluation model.-- If provided, the model must be available to the project when the- evaluator or evaluation rule is enabled.--- To discover valid configured `provider` values for a project, call `GET- /api/public/llm-connections` and read the `provider` field from the- returned connections.-- Use a `provider` value that matches one of the connections already- configured in the same project.--- Recovery guidance:-- - If evaluator creation returns `422` with- `code=evaluator_preflight_failed`, either provide a valid explicit- `modelConfig` here or configure the project's default evaluation model,- then retry the same request.properties:- provider:+ userId:type: string- description: >-- Provider identifier to use for this evaluator, for example `openai`- or `anthropic`.--- To discover valid values for the current project, call `GET- /api/public/llm-connections` and use one of the returned `provider`- values.- model:+ role:+ $ref: '#/components/schemas/MembershipRole'+ email:+ type: string+ name:type: string- description: >-- Model identifier exposed by the provider, for example `gpt-4.1-mini`.required:- - provider- - model- unstableEvaluatorOutputDataType:- title: unstableEvaluatorOutputDataType- type: string- enum:- - NUMERIC- - BOOLEAN- - CATEGORICAL- description: >-- Structured score type returned by an evaluator.--- This controls the type of score value Langfuse stores for evaluation- results:-- - `NUMERIC`: a numeric score such as `0.82`-- - `BOOLEAN`: a boolean score such as `true`-- - `CATEGORICAL`: one or more category labels from a fixed list- unstableEvaluatorOutputFieldDefinition:- title: unstableEvaluatorOutputFieldDefinition+ - userId+ - role+ - name+ MembershipDeletionResponse:+ title: MembershipDeletionResponsetype: objectproperties:- description:+ message:+ type: string+ userId:type: string- description: >-- Human-readable instructions for what the evaluator should return in- this field.required:- - description- unstableEvaluatorOutputDefinition:- title: unstableEvaluatorOutputDefinition- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: >-- #/components/schemas/unstablePublicNumericEvaluatorOutputDefinition- required:- - dataType- title: UnstablePublicNumericEvaluatorOutputDefinition- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: >-- #/components/schemas/unstablePublicBooleanEvaluatorOutputDefinition- required:- - dataType- title: UnstablePublicBooleanEvaluatorOutputDefinition- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: >-- #/components/schemas/unstablePublicCategoricalEvaluatorOutputDefinition- required:- - dataType- title: UnstablePublicCategoricalEvaluatorOutputDefinition- description: >-- Structured output definition to send when creating an evaluator.--- Agent guidance:-- - `dataType` is required.-- - Do not send `version`; that is an internal storage detail and is not- part of the public request contract.-- - For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and- `score.description`.-- - For `NUMERIC`, `score.minValue` and `score.maxValue` optionally define- inclusive bounds. If both are set, `minValue` must not exceed- `maxValue`.-- - For `CATEGORICAL`, also provide `score.categories` and- `score.shouldAllowMultipleMatches`.- unstablePublicNumericEvaluatorOutputScoreDefinition:- title: unstablePublicNumericEvaluatorOutputScoreDefinition+ - message+ - userId+ MembershipsResponse:+ title: MembershipsResponsetype: objectproperties:- description:+ memberships:+ type: array+ items:+ $ref: '#/components/schemas/MembershipResponse'+ required:+ - memberships+ OrganizationProject:+ title: OrganizationProject+ type: object+ properties:+ id:type: string- minValue:- type: number- format: double- nullable: true- description: Inclusive minimum value for numeric evaluator scores.- maxValue:- type: number- format: double+ name:+ type: string+ metadata:+ type: object+ additionalProperties: truenullable: true- description: Inclusive maximum value for numeric evaluator scores.+ createdAt:+ type: string+ format: date-time+ updatedAt:+ type: string+ format: date-timerequired:- - description- unstablePublicNumericEvaluatorOutputDefinition:- title: unstablePublicNumericEvaluatorOutputDefinition+ - id+ - name+ - createdAt+ - updatedAt+ OrganizationProjectsResponse:+ title: OrganizationProjectsResponsetype: objectproperties:- dataType:- $ref: '#/components/schemas/unstableEvaluatorOutputDataType'- description: Always `NUMERIC`.- reasoning:- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'- score:- $ref: >-- #/components/schemas/unstablePublicNumericEvaluatorOutputScoreDefinition+ projects:+ type: array+ items:+ $ref: '#/components/schemas/OrganizationProject'required:- - dataType- - reasoning- - score- unstablePublicBooleanEvaluatorOutputDefinition:- title: unstablePublicBooleanEvaluatorOutputDefinition+ - projects+ OrganizationApiKey:+ title: OrganizationApiKeytype: objectproperties:- dataType:- $ref: '#/components/schemas/unstableEvaluatorOutputDataType'- description: Always `BOOLEAN`.- reasoning:- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'- score:- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'+ id:+ type: string+ createdAt:+ type: string+ format: date-time+ expiresAt:+ type: string+ format: date-time+ nullable: true+ lastUsedAt:+ type: string+ format: date-time+ nullable: true+ note:+ type: string+ nullable: true+ publicKey:+ type: string+ displaySecretKey:+ type: stringrequired:- - dataType- - reasoning- - score- unstablePublicCategoricalEvaluatorOutputScoreDefinition:- title: unstablePublicCategoricalEvaluatorOutputScoreDefinition+ - id+ - createdAt+ - publicKey+ - displaySecretKey+ OrganizationApiKeysResponse:+ title: OrganizationApiKeysResponsetype: objectproperties:- description:- type: string- categories:+ apiKeys:type: arrayitems:- type: string- shouldAllowMultipleMatches:- type: boolean+ $ref: '#/components/schemas/OrganizationApiKey'required:- - description- - categories- - shouldAllowMultipleMatches- unstablePublicCategoricalEvaluatorOutputDefinition:- title: unstablePublicCategoricalEvaluatorOutputDefinition+ - apiKeys+ Projects:+ title: Projectstype: objectproperties:- dataType:- $ref: '#/components/schemas/unstableEvaluatorOutputDataType'- description: Always `CATEGORICAL`.- reasoning:- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'- score:- $ref: >-- #/components/schemas/unstablePublicCategoricalEvaluatorOutputScoreDefinition+ data:+ type: array+ items:+ $ref: '#/components/schemas/Project'required:- - dataType- - reasoning- - score- unstablePublicEvaluatorOutputDefinition:- title: unstablePublicEvaluatorOutputDefinition- oneOf:- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - NUMERIC- - $ref: >-- #/components/schemas/unstablePublicNumericEvaluatorOutputDefinition- required:- - dataType- title: UnstablePublicNumericEvaluatorOutputDefinition- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - BOOLEAN- - $ref: >-- #/components/schemas/unstablePublicBooleanEvaluatorOutputDefinition- required:- - dataType- title: UnstablePublicBooleanEvaluatorOutputDefinition- - type: object- allOf:- - type: object- properties:- dataType:- type: string- enum:- - CATEGORICAL- - $ref: >-- #/components/schemas/unstablePublicCategoricalEvaluatorOutputDefinition- required:- - dataType- title: UnstablePublicCategoricalEvaluatorOutputDefinition- description: >-- Evaluator output definition returned by the public API.--- This response always includes `dataType` and never includes an internal- output-definition `version`.-- Legacy stored evaluator definitions are normalized into this shape- before they are returned.--- Use this response shape when deciding how to interpret future evaluation- scores:-- - `NUMERIC`: expect numeric score values-- - `BOOLEAN`: expect `true` / `false`-- - `CATEGORICAL`: expect one or more values from `score.categories`- unstableEvaluationRuleStringFilterOperator:- title: unstableEvaluationRuleStringFilterOperator- type: string- enum:- - '='- - contains- - does not contain- - starts with- - ends with- unstableEvaluationRuleNumberFilterOperator:- title: unstableEvaluationRuleNumberFilterOperator- type: string- enum:- - '='- - '>'- - <- - '>='- - <=- unstableEvaluationRuleOptionsFilterOperator:- title: unstableEvaluationRuleOptionsFilterOperator- type: string- enum:- - any of- - none of- unstableEvaluationRuleArrayOptionsFilterOperator:- title: unstableEvaluationRuleArrayOptionsFilterOperator- type: string- enum:- - any of- - none of- - all of- unstableEvaluationRuleBooleanFilterOperator:- title: unstableEvaluationRuleBooleanFilterOperator- type: string- enum:- - '='- - <>- unstableEvaluationRuleNullFilterOperator:- title: unstableEvaluationRuleNullFilterOperator- type: string- enum:- - is null- - is not null- unstableDateTimeEvaluationRuleFilter:- title: unstableDateTimeEvaluationRuleFilter+ - data+ Organization:+ title: Organizationtype: objectproperties:- column:+ id:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'- description: Comparison operator for datetime values.- value:+ description: The unique identifier of the organization+ name:type: string- format: date-time- description: Datetime value to compare against.+ description: The name of the organizationrequired:- - column- - operator- - value- unstableStringEvaluationRuleFilter:- title: unstableStringEvaluationRuleFilter+ - id+ - name+ Project:+ title: Projecttype: objectproperties:- column:+ id:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleStringFilterOperator'- value:+ name:type: string+ organization:+ $ref: '#/components/schemas/Organization'+ description: The organization this project belongs to+ metadata:+ type: object+ additionalProperties: true+ description: Metadata for the project+ retentionDays:+ type: integer+ nullable: true+ description: >-+ Number of days to retain data. Null or 0 means no retention. Omitted+ if no retention is configured.required:- - column- - operator- - value- unstableNumberEvaluationRuleFilter:- title: unstableNumberEvaluationRuleFilter+ - id+ - name+ - organization+ - metadata+ ProjectDeletionResponse:+ title: ProjectDeletionResponsetype: objectproperties:- column:+ success:+ type: boolean+ message:type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'- value:- type: number- format: doublerequired:- - column- - operator- - value- unstableStringOptionsEvaluationRuleFilter:- title: unstableStringOptionsEvaluationRuleFilter+ - success+ - message+ ApiKeyList:+ title: ApiKeyListtype: object+ description: List of API keys for a projectproperties:- column:- type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleOptionsFilterOperator'- value:+ apiKeys:type: arrayitems:- type: string- description: One or more allowed string values.+ $ref: '#/components/schemas/ApiKeySummary'required:- - column- - operator- - value- unstableArrayOptionsEvaluationRuleFilter:- title: unstableArrayOptionsEvaluationRuleFilter+ - apiKeys+ ApiKeySummary:+ title: ApiKeySummarytype: object+ description: Summary of an API keyproperties:- column:+ id:type: string- description: Column to filter on.- operator:- $ref: >-- #/components/schemas/unstableEvaluationRuleArrayOptionsFilterOperator- value:- type: array- items:- type: string- description: One or more array elements to match.- required:- - column- - operator- - value- unstableStringObjectEvaluationRuleFilter:- title: unstableStringObjectEvaluationRuleFilter- type: object- properties:- column:+ createdAt:type: string- description: >-- Object-valued column to filter on. In the unstable public API this- is currently `metadata`.- key:+ format: date-time+ expiresAt:type: string- description: Top-level key inside the object-valued column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleStringFilterOperator'- value:+ format: date-time+ nullable: true+ lastUsedAt:type: string- required:- - column- - key- - operator- - value- unstableNumberObjectEvaluationRuleFilter:- title: unstableNumberObjectEvaluationRuleFilter- type: object- properties:- column:+ format: date-time+ nullable: true+ note:type: string- description: Object-valued column to filter on.- key:+ nullable: true+ publicKey:+ type: string+ displaySecretKey:type: string- description: Key inside the object-valued column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'- value:- type: number- format: doublerequired:- - column- - key- - operator- - value- unstableCategoryOptionsEvaluationRuleFilter:- title: unstableCategoryOptionsEvaluationRuleFilter+ - id+ - createdAt+ - publicKey+ - displaySecretKey+ ApiKeyResponse:+ title: ApiKeyResponsetype: object+ description: Response for API key creationproperties:- column:+ id:type: string- description: Object-valued column to filter on.- key:+ createdAt:type: string- description: Key inside the object-valued column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleOptionsFilterOperator'- value:- type: array- items:- type: string+ format: date-time+ publicKey:+ type: string+ secretKey:+ type: string+ displaySecretKey:+ type: string+ note:+ type: string+ nullable: truerequired:- - column- - key- - operator- - value- unstableBooleanEvaluationRuleFilter:- title: unstableBooleanEvaluationRuleFilter+ - id+ - createdAt+ - publicKey+ - secretKey+ - displaySecretKey+ ApiKeyDeletionResponse:+ title: ApiKeyDeletionResponsetype: object+ description: Response for API key deletionproperties:- column:- type: string- description: Column to filter on.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleBooleanFilterOperator'- value:+ success:type: booleanrequired:- - column- - operator- - value- unstableNullEvaluationRuleFilter:- title: unstableNullEvaluationRuleFilter+ - success+ PromptMetaListResponse:+ title: PromptMetaListResponsetype: objectproperties:- column:- type: string- description: >-- Column to filter on. In the unstable public API this is currently- `parentObservationId`.- operator:- $ref: '#/components/schemas/unstableEvaluationRuleNullFilterOperator'- value:- type: string- nullable: true- description: >-- Ignored placeholder value. Clients may omit it or send an empty- string.+ data:+ type: array+ items:+ $ref: '#/components/schemas/PromptMeta'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'required:- - column- - operator- unstablePromptVariableMappingInput:- title: unstablePromptVariableMappingInput+ - data+ - meta+ PromptMeta:+ title: PromptMetatype: object- description: >-- Connects one prompt variable to the available evaluation data.--- Manual mappings are used for `llm_as_judge` evaluators. `code`- evaluators use a fixed runtime mapping managed by Langfuse.--- How to build a valid mapping list:-- 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.-- 2. Read the evaluator `variables` array.-- 3. Add exactly one mapping object for each variable in that array.-- 4. Use the variable name exactly as returned, without braces such as- `{{` or `}}`.-- 5. Choose a `source` that is valid for the selected `target`.--- `jsonPath` is optional. Use it only when the selected source is a JSON- object and you want to extract one nested field before inserting it into- the evaluator prompt.--- Recovery guidance:-- - `invalid_variable_mapping`: the variable name is unknown for this- evaluator, or the selected `source` is not valid for the chosen `target`-- - `missing_variable_mapping`: one or more LLM-as-judge prompt variables- are not mapped yet-- - `duplicate_variable_mapping`: the same prompt variable appears more- than once-- - `invalid_json_path`: the JSONPath expression is malformed. Remove it- or correct it.properties:- variable:+ name:type: string- description: >-- Prompt variable name without braces.--- Example: for the prompt `Judge {{input}} against {{output}}`, use- `input` and `output`.- source:- $ref: '#/components/schemas/unstablePromptVariableMappingSource'- description: >-- Source field that should populate the prompt variable.--- Quick reference:-- - `target=observation`: `input`, `output`, `metadata`, `tool_calls`-- - `target=experiment`: `input`, `output`, `metadata`, `tool_calls`,- `expected_output`, `experiment_item_metadata`- jsonPath:+ type:+ $ref: '#/components/schemas/PromptType'+ description: Indicates whether the prompt is a text or chat prompt.+ versions:+ type: array+ items:+ type: integer+ labels:+ type: array+ items:+ type: string+ tags:+ type: array+ items:+ type: string+ lastUpdatedAt:type: string- nullable: true+ format: date-time+ lastConfig:description: >-- Optional JSONPath selector applied to the selected source before it- is passed to the evaluator prompt.--- Requirements:-- - Must start with `$`-- - Must be a syntactically valid JSONPath expression-- - Most useful with `source=metadata`+ Config object of the most recent prompt version that matches the+ filters (if any are provided)required:- - variable- - source- unstablePromptVariableMappingRead:- title: unstablePromptVariableMappingRead+ - name+ - type+ - versions+ - labels+ - tags+ - lastUpdatedAt+ - lastConfig+ CreatePromptRequest:+ title: CreatePromptRequest+ oneOf:+ - $ref: '#/components/schemas/CreateChatPromptRequest'+ - $ref: '#/components/schemas/CreateTextPromptRequest'+ CreateChatPromptRequest:+ title: CreateChatPromptRequesttype: object- description: |-- Connects one prompt variable to data from a target.-- `source` is `null` when the mapping is incomplete.properties:- variable:+ name:type: string- description: Prompt variable name without braces.- source:- $ref: '#/components/schemas/unstablePromptVariableMappingSource'+ prompt:+ type: array+ items:+ $ref: '#/components/schemas/ChatMessageWithPlaceholders'+ config:nullable: true- description: >-- Source field populating the variable, or `null` when the mapping is- incomplete.- jsonPath:- type: string+ type:+ $ref: '#/components/schemas/CreateChatPromptType'+ labels:+ type: array+ items:+ type: stringnullable: true- description: Optional JSONPath selector applied to the selected source.- required:- - variable- - source- unstableEvaluationRuleFilter:- title: unstableEvaluationRuleFilter- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - datetime- - $ref: '#/components/schemas/unstableDateTimeEvaluationRuleFilter'- required:- - type- title: UnstableDateTimeEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - string- - $ref: '#/components/schemas/unstableStringEvaluationRuleFilter'- required:- - type- title: UnstableStringEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - number- - $ref: '#/components/schemas/unstableNumberEvaluationRuleFilter'- required:- - type- title: UnstableNumberEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - stringOptions- - $ref: '#/components/schemas/unstableStringOptionsEvaluationRuleFilter'- required:- - type- title: UnstableStringOptionsEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - categoryOptions- - $ref: '#/components/schemas/unstableCategoryOptionsEvaluationRuleFilter'- required:- - type- title: UnstableCategoryOptionsEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - arrayOptions- - $ref: '#/components/schemas/unstableArrayOptionsEvaluationRuleFilter'- required:- - type- title: UnstableArrayOptionsEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - stringObject- - $ref: '#/components/schemas/unstableStringObjectEvaluationRuleFilter'- required:- - type- title: UnstableStringObjectEvaluationRuleFilter- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - numberObject- - $ref: '#/components/schemas/unstableNumberObjectEvaluationRuleFilter'- required:- - type- title: UnstableNumberObjectEvaluationRuleFilter+ description: List of deployment labels of this prompt version.+ tags:+ type: array+ items:+ type: string+ nullable: true+ description: List of tags to apply to all versions of this prompt.+ commitMessage:+ type: string+ nullable: true+ description: Commit message for this prompt version.+ required:+ - name+ - prompt+ - type+ CreateTextPromptRequest:+ title: CreateTextPromptRequest+ type: object+ properties:+ name:+ type: string+ prompt:+ type: string+ config:+ nullable: true+ type:+ $ref: '#/components/schemas/CreateTextPromptType'+ nullable: true+ labels:+ type: array+ items:+ type: string+ nullable: true+ description: List of deployment labels of this prompt version.+ tags:+ type: array+ items:+ type: string+ nullable: true+ description: List of tags to apply to all versions of this prompt.+ commitMessage:+ type: string+ nullable: true+ description: Commit message for this prompt version.+ required:+ - name+ - prompt+ Prompt:+ title: Prompt+ oneOf:- type: objectallOf:- type: object@@ -18075,11 +15228,11 @@ components:type:type: stringenum:- - boolean- - $ref: '#/components/schemas/unstableBooleanEvaluationRuleFilter'+ - chat+ - $ref: '#/components/schemas/ChatPrompt'required:- type- title: UnstableBooleanEvaluationRuleFilter+ title: ChatPrompt- type: objectallOf:- type: object@@ -18087,1641 +15240,1528 @@ components:type:type: stringenum:- - 'null'- - $ref: '#/components/schemas/unstableNullEvaluationRuleFilter'+ - text+ - $ref: '#/components/schemas/TextPrompt'required:- type- title: UnstableNullEvaluationRuleFilter- description: >-- One filter condition used to decide whether a live-ingested target- should be evaluated.--- An evaluation rule can include zero or more filter objects. All filters- must be satisfied for the target to run.--- How to build a valid filter object:-- - Pick the `target` first, because it changes the supported columns.-- - Pick the filter `type`. That determines which fields are required.-- - Use `key` only for object filters such as `metadata`.-- - Use the correct `value` shape for the chosen filter `type`.--- Operator quick reference by filter `type`:-- - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends- with`-- - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`-- - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`-- - `stringOptions`: `any of`, `none of`-- - `arrayOptions`: `any of`, `none of`, `all of`-- - `stringObject`: same operators as `string`-- - `boolean`: `"="`, `"<>"`-- - `null`: `is null`, `is not null`--- Supported columns by target:-- - `target=observation`- - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`- - `name`: `stringOptions`, operators `any of` / `none of`- - `environment`: `stringOptions`, operators `any of` / `none of`- - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`- - `version`: `string`- - `traceName`: `stringOptions`, operators `any of` / `none of`- - `userId`: `string`- - `sessionId`: `string`- - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`- - `metadata`: `stringObject` with `key`- - `isRootObservation`: `boolean`, operators `=` / `<>`; true when the observation has no parent or is explicitly marked as an application root- - `parentObservationId`: `null`, operators `is null` / `is not null`- - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`- - `toolCalls`: `number`- - `target=experiment`- - `datasetId`: `stringOptions`, operators `any of` / `none of`- Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.-- Recovery guidance:-- - `invalid_filter_value` with `details.column` but no `invalidValues`:- the selected `column` is not supported for the chosen `target`-- - `invalid_filter_value` with `details.invalidValues`: the selected- values are not allowed for that column. Replace them with one of- `details.allowedValues` when provided.-- - `invalid_filter_value` for `column=datasetId`: call `GET- /api/public/v2/datasets`, then retry with dataset `id` values from that- response.- unstableDashboardWidgetView:- title: unstableDashboardWidgetView+ title: TextPrompt+ PromptType:+ title: PromptTypetype: stringenum:- - observations- - scores-numeric- - scores-boolean- - scores-categorical- unstableDashboardWidgetViewWithLegacy:- title: unstableDashboardWidgetViewWithLegacy+ - chat+ - text+ BasePrompt:+ title: BasePrompt+ type: object+ properties:+ name:+ type: string+ version:+ type: integer+ config: {}+ labels:+ type: array+ items:+ type: string+ description: List of deployment labels of this prompt version.+ tags:+ type: array+ items:+ type: string+ description: >-+ List of tags. Used to filter via UI and API. The same across+ versions of a prompt.+ commitMessage:+ type: string+ nullable: true+ description: Commit message for this prompt version.+ resolutionGraph:+ type: object+ additionalProperties: true+ nullable: true+ description: >-+ The dependency resolution graph for the current prompt. Null if the+ prompt has no dependencies or if `resolve=false` was used.+ required:+ - name+ - version+ - config+ - labels+ - tags+ ChatMessageWithPlaceholders:+ title: ChatMessageWithPlaceholders+ oneOf:+ - $ref: '#/components/schemas/ChatMessage'+ - $ref: '#/components/schemas/PlaceholderMessage'+ ChatMessage:+ title: ChatMessage+ type: object+ properties:+ role:+ type: string+ content:+ type: string+ type:+ $ref: '#/components/schemas/ChatMessageType'+ nullable: true+ required:+ - role+ - content+ ChatMessageType:+ title: ChatMessageTypetype: stringenum:- - observations- - scores-numeric- - scores-boolean- - scores-categorical- - traces- description: |-- Widget data view. Responses may include the legacy `traces` value for- widgets created before this API existed.- unstableDashboardWidgetChartType:- title: unstableDashboardWidgetChartType+ - chatmessage+ PlaceholderMessage:+ title: PlaceholderMessage+ type: object+ properties:+ name:+ type: string+ type:+ $ref: '#/components/schemas/PlaceholderMessageType'+ nullable: true+ required:+ - name+ PlaceholderMessageType:+ title: PlaceholderMessageTypetype: stringenum:- - LINE_TIME_SERIES- - AREA_TIME_SERIES- - BAR_TIME_SERIES- - HORIZONTAL_BAR- - VERTICAL_BAR- - PIE- - NUMBER- - HISTOGRAM- - PIVOT_TABLE- unstableDashboardWidgetMetricAggregation:- title: unstableDashboardWidgetMetricAggregation+ - placeholder+ TextPrompt:+ title: TextPrompt+ type: object+ properties:+ prompt:+ type: string+ required:+ - prompt+ allOf:+ - $ref: '#/components/schemas/BasePrompt'+ ChatPrompt:+ title: ChatPrompt+ type: object+ properties:+ prompt:+ type: array+ items:+ $ref: '#/components/schemas/ChatMessageWithPlaceholders'+ required:+ - prompt+ allOf:+ - $ref: '#/components/schemas/BasePrompt'+ CreateChatPromptType:+ title: CreateChatPromptTypetype: stringenum:- - sum- - avg- - count- - max- - min- - p50- - p75- - p90- - p95- - p99- - histogram- - uniq- unstableDashboardWidgetDimension:- title: unstableDashboardWidgetDimension+ - chat+ CreateTextPromptType:+ title: CreateTextPromptType+ type: string+ enum:+ - text+ ServiceProviderConfig:+ title: ServiceProviderConfig+ type: object+ properties:+ schemas:+ type: array+ items:+ type: string+ documentationUri:+ type: string+ patch:+ $ref: '#/components/schemas/ScimFeatureSupport'+ bulk:+ $ref: '#/components/schemas/BulkConfig'+ filter:+ $ref: '#/components/schemas/FilterConfig'+ changePassword:+ $ref: '#/components/schemas/ScimFeatureSupport'+ sort:+ $ref: '#/components/schemas/ScimFeatureSupport'+ etag:+ $ref: '#/components/schemas/ScimFeatureSupport'+ authenticationSchemes:+ type: array+ items:+ $ref: '#/components/schemas/AuthenticationScheme'+ meta:+ $ref: '#/components/schemas/ResourceMeta'+ required:+ - schemas+ - documentationUri+ - patch+ - bulk+ - filter+ - changePassword+ - sort+ - etag+ - authenticationSchemes+ - meta+ ScimFeatureSupport:+ title: ScimFeatureSupporttype: objectproperties:- field:+ supported:+ type: boolean+ required:+ - supported+ BulkConfig:+ title: BulkConfig+ type: object+ properties:+ supported:+ type: boolean+ maxOperations:+ type: integer+ maxPayloadSize:+ type: integer+ required:+ - supported+ - maxOperations+ - maxPayloadSize+ FilterConfig:+ title: FilterConfig+ type: object+ properties:+ supported:+ type: boolean+ maxResults:+ type: integer+ required:+ - supported+ - maxResults+ ResourceMeta:+ title: ResourceMeta+ type: object+ properties:+ resourceType:+ type: string+ location:type: stringrequired:- - field- unstableDashboardWidgetMetric:- title: unstableDashboardWidgetMetric+ - resourceType+ - location+ AuthenticationScheme:+ title: AuthenticationSchemetype: objectproperties:- measure:+ name:type: string- agg:- $ref: '#/components/schemas/unstableDashboardWidgetMetricAggregation'+ description:+ type: string+ specUri:+ type: string+ type:+ type: string+ primary:+ type: booleanrequired:- - measure- - agg- unstableDashboardWidgetFilter:- title: unstableDashboardWidgetFilter+ - name+ - description+ - specUri+ - type+ - primary+ ResourceTypesResponse:+ title: ResourceTypesResponse+ type: object+ properties:+ schemas:+ type: array+ items:+ type: string+ totalResults:+ type: integer+ Resources:+ type: array+ items:+ $ref: '#/components/schemas/ResourceType'+ required:+ - schemas+ - totalResults+ - Resources+ ResourceType:+ title: ResourceType+ type: object+ properties:+ schemas:+ type: array+ items:+ type: string+ nullable: true+ id:+ type: string+ name:+ type: string+ endpoint:+ type: string+ description:+ type: string+ schema:+ type: string+ schemaExtensions:+ type: array+ items:+ $ref: '#/components/schemas/SchemaExtension'+ meta:+ $ref: '#/components/schemas/ResourceMeta'+ required:+ - id+ - name+ - endpoint+ - description+ - schema+ - schemaExtensions+ - meta+ SchemaExtension:+ title: SchemaExtension+ type: object+ properties:+ schema:+ type: string+ required:+ type: boolean+ required:+ - schema+ - required+ SchemasResponse:+ title: SchemasResponse+ type: object+ properties:+ schemas:+ type: array+ items:+ type: string+ totalResults:+ type: integer+ Resources:+ type: array+ items:+ $ref: '#/components/schemas/SchemaResource'+ required:+ - schemas+ - totalResults+ - Resources+ SchemaResource:+ title: SchemaResource+ type: object+ properties:+ id:+ type: string+ name:+ type: string+ description:+ type: string+ attributes:+ type: array+ items: {}+ meta:+ $ref: '#/components/schemas/ResourceMeta'+ required:+ - id+ - name+ - description+ - attributes+ - meta+ ScimUsersListResponse:+ title: ScimUsersListResponse+ type: object+ properties:+ schemas:+ type: array+ items:+ type: string+ totalResults:+ type: integer+ startIndex:+ type: integer+ itemsPerPage:+ type: integer+ Resources:+ type: array+ items:+ $ref: '#/components/schemas/ScimUser'+ required:+ - schemas+ - totalResults+ - startIndex+ - itemsPerPage+ - Resources+ ScimUser:+ title: ScimUser+ type: object+ properties:+ schemas:+ type: array+ items:+ type: string+ id:+ type: string+ userName:+ type: string+ name:+ $ref: '#/components/schemas/ScimName'+ emails:+ type: array+ items:+ $ref: '#/components/schemas/ScimEmail'+ meta:+ $ref: '#/components/schemas/UserMeta'+ required:+ - schemas+ - id+ - userName+ - name+ - emails+ - meta+ UserMeta:+ title: UserMetatype: object- description: >-- A filter in Langfuse filter-state shape. The `value` shape and the-- allowed operators depend on `type`:--- | `type` | `value` | operators |-- |---|---|---|-- | `string` | string | `=`, `contains`, `does not contain`, `starts- with`, `ends with` |-- | `number` | number | `=`, `>`, `<`, `>=`, `<=` |-- | `datetime` | ISO datetime string | `>`, `<`, `>=`, `<=` |-- | `boolean` | boolean | `=`, `<>` |-- | `null` | `""` | `is null`, `is not null` |-- | `stringOptions` | list of strings | `any of`, `none of` |-- | `arrayOptions` | list of strings | `any of`, `none of`, `all of` |-- | `categoryOptions` | list of strings (requires `key`) | `any of`, `none- of` |-- | `stringObject` | string (requires `key`, e.g. a metadata key) | same- as `string` |-- | `numberObject` | number (requires `key`, e.g. a score name) | same as- `number` |-- | `booleanObject` | boolean (requires `key`) | `=`, `<>` |properties:- column:- type: string- operator:+ resourceType:type: string- type:+ created:type: string- value:nullable: true- key:+ lastModified:type: stringnullable: truerequired:- - column- - operator- - type- unstableDashboardWidgetChartConfig:- title: unstableDashboardWidgetChartConfig+ - resourceType+ ScimName:+ title: ScimNametype: object- description: |-- Chart-specific widget configuration.-- `type` must match the top-level `chartType`.- `row_limit` applies to total-value charts and pivot tables.- `bins` applies to histograms.- `defaultSort` applies to pivot tables.properties:- type:- $ref: '#/components/schemas/unstableDashboardWidgetChartType'- row_limit:- type: integer+ formatted:+ type: stringnullable: true- show_value_labels:+ ScimEmail:+ title: ScimEmail+ type: object+ properties:+ primary:type: boolean- nullable: true- bins:- type: integer- nullable: true- defaultSort:- $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'- nullable: true+ value:+ type: string+ type:+ type: stringrequired:+ - primary+ - value- type- unstableDashboardWidgetDefaultSort:- title: unstableDashboardWidgetDefaultSort+ EmptyResponse:+ title: EmptyResponse+ type: object+ description: Empty response for 204 No Content responses+ properties: {}+ ScoreConfigs:+ title: ScoreConfigstype: objectproperties:- column:- type: string- order:- $ref: '#/components/schemas/unstableDashboardWidgetSortOrder'+ data:+ type: array+ items:+ $ref: '#/components/schemas/ScoreConfig'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'required:- - column- - order- unstableDashboardWidgetSortOrder:- title: unstableDashboardWidgetSortOrder- type: string- enum:- - ASC- - DESC- unstableDashboardWidgetChartConfigInput:- title: unstableDashboardWidgetChartConfigInput+ - data+ - meta+ CreateScoreConfigRequest:+ title: CreateScoreConfigRequesttype: object- description: |-- Input-side chart config. `type` is optional and defaults to the- widget's `chartType`; when given it must match.properties:- type:- $ref: '#/components/schemas/unstableDashboardWidgetChartType'- nullable: true- row_limit:- type: integer- nullable: true- show_value_labels:- type: boolean+ name:+ type: string+ description: >-+ Name of the score config. Max 35 characters. Only letters, numbers,+ underscores, spaces, periods, parentheses, and hyphens are allowed.+ dataType:+ $ref: '#/components/schemas/ScoreConfigDataType'+ categories:+ type: array+ items:+ $ref: '#/components/schemas/ConfigCategory'nullable: true- bins:- type: integer+ description: >-+ Configure custom categories for categorical scores. Pass a list of+ objects with `label` and `value` properties. Categories are+ autogenerated for boolean configs and cannot be passed+ minValue:+ type: number+ format: doublenullable: true- defaultSort:- $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'+ description: >-+ Configure a minimum value for numerical scores. If not set, the+ minimum value defaults to -∞+ maxValue:+ type: number+ format: doublenullable: true- unstableCreateDashboardWidgetRequest:- title: unstableCreateDashboardWidgetRequest- type: object- properties:- name:- type: string+ description: >-+ Configure a maximum value for numerical scores. If not set, the+ maximum value defaults to +∞description:type: stringnullable: true- description: Defaults to an empty string.- view:- $ref: '#/components/schemas/unstableDashboardWidgetView'- dimensions:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetDimension'- metrics:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetMetric'- filters:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetFilter'- chartType:- $ref: '#/components/schemas/unstableDashboardWidgetChartType'- chartConfig:- $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'- nullable: true- description: Defaults to the plain config for `chartType`.+ description: >-+ Description is shown across the Langfuse UI and can be used to e.g.+ explain the config categories in detail, why a numeric range was+ set, or provide additional context on config name or usagerequired:- name- - view- - dimensions- - metrics- - filters- - chartType- unstableUpdateDashboardWidgetRequest:- title: unstableUpdateDashboardWidgetRequest+ - dataType+ UpdateScoreConfigRequest:+ title: UpdateScoreConfigRequesttype: objectproperties:- name:- type: string+ isArchived:+ type: booleannullable: true- description:+ description: The status of the score config showing if it is archived or not+ name:type: stringnullable: true- view:- $ref: '#/components/schemas/unstableDashboardWidgetView'- nullable: true- dimensions:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetDimension'- nullable: true- metrics:+ description: >-+ Name of the score config. Max 35 characters. Only letters, numbers,+ underscores, spaces, periods, parentheses, and hyphens are allowed.+ categories:type: arrayitems:- $ref: '#/components/schemas/unstableDashboardWidgetMetric'+ $ref: '#/components/schemas/ConfigCategory'nullable: true- filters:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetFilter'+ description: >-+ Configure custom categories for categorical scores. Pass a list of+ objects with `label` and `value` properties. Categories are+ autogenerated for boolean configs and cannot be passed+ minValue:+ type: number+ format: doublenullable: true- chartType:- $ref: '#/components/schemas/unstableDashboardWidgetChartType'+ description: >-+ Configure a minimum value for numerical scores. If not set, the+ minimum value defaults to -∞+ maxValue:+ type: number+ format: doublenullable: true- chartConfig:- $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'+ description: >-+ Configure a maximum value for numerical scores. If not set, the+ maximum value defaults to +∞+ description:+ type: stringnullable: true- unstableDashboardWidgetList:- title: unstableDashboardWidgetList+ description: >-+ Description is shown across the Langfuse UI and can be used to e.g.+ explain the config categories in detail, why a numeric range was+ set, or provide additional context on config name or usage+ ScoreSubjectTraceV3:+ title: ScoreSubjectTraceV3type: objectproperties:- data:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidget'- meta:- $ref: '#/components/schemas/utilsMetaResponse'+ id:+ type: string+ description: The trace ID.+ required:+ - id+ ScoreSubjectObservationV3:+ title: ScoreSubjectObservationV3+ type: object+ properties:+ id:+ type: string+ description: The observation ID.+ traceId:+ type: string+ nullable: true+ description: The parent trace ID, if available.required:- - data- - meta- unstableDeleteDashboardWidgetResponse:- title: unstableDeleteDashboardWidgetResponse+ - id+ ScoreSubjectSessionV3:+ title: ScoreSubjectSessionV3type: objectproperties:- message:+ id:type: string+ description: The session ID.required:- - message- unstableDashboardWidget:- title: unstableDashboardWidget+ - id+ ScoreSubjectExperimentV3:+ title: ScoreSubjectExperimentV3type: objectproperties:id:type: string- createdAt:- type: string- format: date-time- updatedAt:- type: string- format: date-time- name:- type: string- description:- type: string- view:- $ref: '#/components/schemas/unstableDashboardWidgetViewWithLegacy'- dimensions:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetDimension'- metrics:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetMetric'- filters:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetFilter'- chartType:- $ref: '#/components/schemas/unstableDashboardWidgetChartType'- chartConfig:- $ref: '#/components/schemas/unstableDashboardWidgetChartConfig'+ description: The dataset run ID (experiment ID).required:- id- - createdAt- - updatedAt- - name- - description- - view- - dimensions- - metrics- - filters- - chartType- - chartConfig- unstableDashboardPlacement:- title: unstableDashboardPlacement+ ScoreSubjectV3:+ title: ScoreSubjectV3oneOf:- type: objectallOf:- type: objectproperties:- type:+ kind:type: stringenum:- - widget- - $ref: '#/components/schemas/unstableWidgetPlacement'+ - trace+ - $ref: '#/components/schemas/ScoreSubjectTraceV3'required:- - type- title: UnstableWidgetPlacement+ - kind+ title: ScoreSubjectTraceV3- type: objectallOf:- type: objectproperties:- type:+ kind:type: stringenum:- - preset- - $ref: '#/components/schemas/unstablePresetPlacement'+ - observation+ - $ref: '#/components/schemas/ScoreSubjectObservationV3'required:- - type- title: UnstablePresetPlacement- description: |-- A tile on the dashboard's 12-column grid. `x`/`y` are the tile's- top-left cell (0-based; `y` grows downward), `width`/`height` its size- in cells. The UI default tile is 6x6 (half width). Overlapping tiles- are not rejected; prefer appending below existing tiles (or omit the- position on create to let the server do it).- unstableWidgetPlacement:- title: unstableWidgetPlacement- type: object- properties:- id:- type: string- widgetId:- type: string- x:- type: integer- 'y':- type: integer- width:- type: integer- height:- type: integer- required:- - id- - widgetId- - x- - 'y'- - width- - height- unstablePresetPlacement:- title: unstablePresetPlacement- type: object- properties:- id:- type: string- presetId:- type: string- x:- type: integer- 'y':- type: integer- width:- type: integer- height:- type: integer- required:- - id- - presetId- - x- - 'y'- - width- - height- unstableCreateDashboardPlacementRequest:- title: unstableCreateDashboardPlacementRequest- oneOf:+ - kind+ title: ScoreSubjectObservationV3- type: objectallOf:- type: objectproperties:- type:+ kind:type: stringenum:- - widget- - $ref: '#/components/schemas/unstableCreateWidgetPlacement'+ - session+ - $ref: '#/components/schemas/ScoreSubjectSessionV3'required:- - type- title: UnstableCreateWidgetPlacement+ - kind+ title: ScoreSubjectSessionV3- type: objectallOf:- type: objectproperties:- type:+ kind:type: stringenum:- - preset- - $ref: '#/components/schemas/unstableCreatePresetPlacement'+ - experiment+ - $ref: '#/components/schemas/ScoreSubjectExperimentV3'required:- - type- title: UnstableCreatePresetPlacement- unstableCreateWidgetPlacement:- title: unstableCreateWidgetPlacement- type: object- properties:- id:- type: string- nullable: true- description: Server-generated when omitted.- widgetId:- type: string- x:- type: integer- nullable: true- description: Grid column (12-column grid). Defaults to `0`.- 'y':- type: integer- nullable: true- description: Grid row. Defaults to the first row below all existing tiles.- width:- type: integer- nullable: true- description: Width in grid columns. Defaults to `6`.- height:- type: integer- nullable: true- description: Height in grid rows. Defaults to `6`.- required:- - widgetId- unstableCreatePresetPlacement:- title: unstableCreatePresetPlacement+ - kind+ title: ScoreSubjectExperimentV3+ description: >-+ A reference to the entity this score is attached to. Discriminated by+ "kind" — one of trace, observation, session, or experiment.+ BaseScoreV3:+ title: BaseScoreV3type: objectproperties:id:type: string- nullable: true- description: Server-generated when omitted.- presetId:- type: string- x:- type: integer- nullable: true- description: Grid column (12-column grid). Defaults to `0`.- 'y':- type: integer- nullable: true- description: Grid row. Defaults to the first row below all existing tiles.- width:- type: integer- nullable: true- description: Width in grid columns. Defaults to `6`.- height:- type: integer- nullable: true- description: Height in grid rows. Defaults to `6`.- required:- - presetId- unstableUpdateDashboardPlacementRequest:- title: unstableUpdateDashboardPlacementRequest- type: object- properties:- x:- type: integer- nullable: true- description: Grid column (12-column grid).- 'y':- type: integer- nullable: true- description: Grid row.- width:- type: integer- nullable: true- description: Width in grid columns.- height:- type: integer- nullable: true- description: Height in grid rows.- unstableDeleteDashboardPlacementResponse:- title: unstableDeleteDashboardPlacementResponse- type: object- properties:- message:+ projectId:type: string- required:- - message- unstableDashboardDefinition:- title: unstableDashboardDefinition- type: object- properties:- widgets:- type: array- items:- $ref: '#/components/schemas/unstableDashboardPlacement'- required:- - widgets- unstableDashboard:- title: unstableDashboard- type: object- properties:- id:+ name:+ type: string+ source:+ $ref: '#/components/schemas/ScoreSource'+ timestamp:+ type: string+ format: date-time+ environment:type: string+ description: The environment from which this score originated.createdAt:type: stringformat: date-timeupdatedAt:type: stringformat: date-time- name:+ comment:type: string- description:+ nullable: true+ description: >-+ Optional comment attached to the score. Present when "details" is+ included in the fields parameter.+ configId:type: string- definition:- $ref: '#/components/schemas/unstableDashboardDefinition'- filters:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetFilter'- description: Dashboard-level filters applied to all widgets on the dashboard.+ nullable: true+ description: >-+ The score config ID, if this score was created from a config.+ Present when "details" is included in the fields parameter.+ metadata:+ type: object+ additionalProperties: true+ nullable: true+ description: >-+ Arbitrary metadata attached to the score. Present when "details" is+ included in the fields parameter.+ authorUserId:+ type: string+ nullable: true+ description: >-+ The user who created this score, if available. Present when+ "annotation" is included in the fields parameter.+ queueId:+ type: string+ nullable: true+ description: >-+ The annotation queue this score belongs to, if any. Present when+ "annotation" is included in the fields parameter.+ subject:+ $ref: '#/components/schemas/ScoreSubjectV3'+ nullable: true+ description: >-+ The entity this score is attached to (trace, observation, session,+ or experiment). Present when "subject" is included in the fields+ parameter.required:- id+ - projectId+ - name+ - source+ - timestamp+ - environment- createdAt- updatedAt- - name- - description- - definition- - filters- unstableDashboardList:- title: unstableDashboardList+ NumericScoreV3:+ title: NumericScoreV3type: objectproperties:- data:- type: array- items:- $ref: '#/components/schemas/unstableDashboard'- meta:- $ref: '#/components/schemas/utilsMetaResponse'+ value:+ type: number+ format: double+ description: The numeric value of the score.required:- - data- - meta- unstableCreateDashboardRequest:- title: unstableCreateDashboardRequest+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScoreV3'+ BooleanScoreV3:+ title: BooleanScoreV3type: objectproperties:- name:- type: string- description:- type: string- nullable: true- definition:- $ref: '#/components/schemas/unstableDashboardDefinition'- nullable: true- filters:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetFilter'- nullable: true+ value:+ type: boolean+ description: The boolean value of the score.required:- - name- unstableUpdateDashboardRequest:- title: unstableUpdateDashboardRequest+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScoreV3'+ CategoricalScoreV3:+ title: CategoricalScoreV3type: objectproperties:- name:+ value:type: string- nullable: true- description:+ description: The string category value of the score.+ required:+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScoreV3'+ TextScoreV3:+ title: TextScoreV3+ type: object+ properties:+ value:type: string- nullable: true- definition:- $ref: '#/components/schemas/unstableDashboardDefinition'- nullable: true- filters:- type: array- items:- $ref: '#/components/schemas/unstableDashboardWidgetFilter'- nullable: true- unstableDeleteDashboardResponse:- title: unstableDeleteDashboardResponse+ description: The text content of the score.+ required:+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScoreV3'+ CorrectionScoreV3:+ title: CorrectionScoreV3type: objectproperties:- message:+ value:type: string+ description: The correction content of the score. Empty string if not set.required:- - message- unstablePublicApiErrorCode:- title: unstablePublicApiErrorCode- type: string- enum:- - authentication_failed- - access_denied- - invalid_request- - invalid_query- - invalid_body- - invalid_filter_value- - invalid_json_path- - invalid_variable_mapping- - missing_variable_mapping- - duplicate_variable_mapping- - resource_not_found- - name_conflict- - evaluator_preflight_failed- - conflict- - unprocessable_content- - rate_limited- - method_not_allowed- - internal_error- description: >-- Machine-readable error code returned by the unstable evaluators API.--- SDKs, CLIs, and agents should branch on `code` rather than parsing the- human-readable `message`.-- The HTTP status still indicates the broad error class, while `code`- gives the specific failure reason.- unstablePublicApiValidationIssue:- title: unstablePublicApiValidationIssue+ - value+ allOf:+ - $ref: '#/components/schemas/BaseScoreV3'+ ScoreV3:+ title: ScoreV3+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - NUMERIC+ - $ref: '#/components/schemas/NumericScoreV3'+ required:+ - dataType+ title: NumericScoreV3+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - BOOLEAN+ - $ref: '#/components/schemas/BooleanScoreV3'+ required:+ - dataType+ title: BooleanScoreV3+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CATEGORICAL+ - $ref: '#/components/schemas/CategoricalScoreV3'+ required:+ - dataType+ title: CategoricalScoreV3+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - TEXT+ - $ref: '#/components/schemas/TextScoreV3'+ required:+ - dataType+ title: TextScoreV3+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CORRECTION+ - $ref: '#/components/schemas/CorrectionScoreV3'+ required:+ - dataType+ title: CorrectionScoreV3+ GetScoresV3Meta:+ title: GetScoresV3Metatype: object- description: >-- One validation issue returned for malformed request bodies or query- parameters.--- This mirrors the most important parts of a Zod issue: a machine-readable- `code`,-- a human-readable `message`, and a structured `path`.properties:- code:+ limit:+ type: integer+ cursor:type: string+ nullable: truedescription: >-- Machine-readable validation issue code emitted by the server- validator.- message:- type: string- description: Human-readable explanation of the validation failure.- path:- type: array- items: {}- description: Path to the invalid field, for example `["mapping", 0, "jsonPath"]`.+ URL-safe base64 (base64url) cursor for the next page. Absent when+ there are no more results.required:- - code- - message- - path- unstablePublicApiErrorDetails:- title: unstablePublicApiErrorDetails+ - limit+ GetScoresV3Response:+ title: GetScoresV3Responsetype: object- description: >-- Optional structured context attached to an unstable-evals error.--- The populated fields depend on the error `code`:-- - request parsing failures populate `issues`-- - filter validation failures populate `field`, `column`,- `invalidValues`, and `allowedValues`-- - variable mapping failures populate `field`, `variable`, or `variables`-- - JSONPath validation failures populate `field`, `variable`, and `value`-- - evaluator preflight failures populate `evaluatorName`, `provider`, and- `model`-- - rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and- `resetAt`properties:- issues:+ data:type: arrayitems:- $ref: '#/components/schemas/unstablePublicApiValidationIssue'- nullable: true- description: Validation issues for malformed request bodies or query parameters.- field:+ $ref: '#/components/schemas/ScoreV3'+ meta:+ $ref: '#/components/schemas/GetScoresV3Meta'+ required:+ - data+ - meta+ CreateScoreRequest:+ title: CreateScoreRequest+ type: object+ properties:+ id:type: stringnullable: true- description: >-- Path-like reference to the failing field, for example- `mapping[1].jsonPath`.- column:+ traceId:type: stringnullable: true- description: Filter column that failed validation.- invalidValues:- type: array- items:- type: string- nullable: true- description: Unsupported values supplied by the caller.- allowedValues:- type: array- items:- type: string+ sessionId:+ type: stringnullable: true- description: Allowed values for the failing filter column.- variable:+ observationId:type: stringnullable: true- description: Evaluator variable involved in the failure.- variables:- type: array- items:- type: string+ datasetRunId:+ type: stringnullable: true- description: >-- Multiple evaluator variables involved in the failure, for example- missing mappings.+ name:+ type: stringvalue:+ $ref: '#/components/schemas/CreateScoreValue'+ description: >-+ The value of the score. Must be passed as string for categorical and+ text scores, and numeric for boolean and numeric scores. Boolean+ score values must equal either 1 or 0 (true or false). Text score+ values must be between 1 and 500 characters.+ comment:type: stringnullable: true- description: Raw invalid value supplied by the caller.- evaluatorName:- type: string+ metadata:+ type: object+ additionalProperties: truenullable: true- description: Evaluator name used during preflight validation.- provider:+ environment:type: stringnullable: true- description: Provider resolved during evaluator preflight, if any.- model:+ description: >-+ The environment of the score. Can be any lowercase alphanumeric+ string with hyphens and underscores that does not start with+ 'langfuse'.+ queueId:type: stringnullable: true- description: Model resolved during evaluator preflight, if any.- retryAfterSeconds:- type: integer- nullable: true- description: Suggested retry delay for rate-limited requests.- limit:- type: integer- nullable: truedescription: >-- Numeric limit associated with the failure, for example the active- evaluation-rule cap or the current rate-limit window.- remaining:- type: integer+ The annotation queue referenced by the score. Indicates if score was+ initially created while processing annotation queue.+ dataType:+ $ref: '#/components/schemas/ScoreDataType'nullable: true- description: Remaining requests in the current rate-limit window.- resetAt:+ description: >-+ The data type of the score. When passing a configId this field is+ inferred. Otherwise, this field must be passed or will default to+ numeric.+ configId:type: stringnullable: true- description: ISO-8601 timestamp when the current rate-limit window resets.- unstablePublicApiError:- title: unstablePublicApiError- type: object- description: >-- Standard error envelope for the unstable evaluators API.--- Response handling guidance:-- - Use the HTTP status code for the broad class of failure.-- - Use `code` for precise branching in SDKs, CLIs, or agents.-- - Inspect `details` for field-level validation context such as invalid- filter values, malformed JSONPath expressions, or missing variable- mappings.-- - Retry only after fixing the specific issue described by `code` and- `details`.- properties:- message:- type: string- description: Human-readable description of the failure.- code:- $ref: '#/components/schemas/unstablePublicApiErrorCode'- description: Stable machine-readable error code.- details:- $ref: '#/components/schemas/unstablePublicApiErrorDetails'+ description: >-+ Reference a score config on a score. The unique langfuse identifier+ of a score config. When passing this field, the dataType and+ stringValue fields are automatically populated.+ source:+ $ref: '#/components/schemas/CreateScoreSource'nullable: truedescription: >-- Optional structured error context. Inspect the populated fields- based on `code`.+ The source of the score. Defaults to API. Set to ANNOTATION to+ prefill scores (e.g. from an LLM) for a human reviewer to verify in+ an annotation queue. When source is ANNOTATION, a configId is+ required unless dataType is CORRECTION. EVAL is reserved for+ internal evaluator outputs and is not accepted on this endpoint.required:- - message- - code- unstableEvaluationRuleBase:- title: unstableEvaluationRuleBase+ - name+ - value+ CreateScoreSource:+ title: CreateScoreSource+ type: string+ enum:+ - API+ - ANNOTATION+ description: |-+ Source values accepted when creating a score via the public REST API.+ EVAL is reserved for internal evaluator outputs and is intentionally not+ exposed here — use commons.ScoreSource when reading scores.+ CreateScoreResponse:+ title: CreateScoreResponsetype: object- description: >-- Live evaluation rule for incoming data.--- An evaluation rule answers:-- - which evaluator should be used-- - which target objects should trigger scoring-- - how often scoring should run-- - which target fields should populate each prompt variable-- - whether the deployment is active, inactive, or paused--- Important status semantics:-- - `enabled` is the desired on/off setting from the client-- - `status` is the effective runtime state after Langfuse applies- validation and blocking rules-- - `enabled=true` with `status=paused` means the rule should run, but- Langfuse has paused it until the underlying problem is fixedproperties:id:type: string- description: Stable evaluation rule identifier.- name:+ description: The id of the created object in Langfuse+ required:+ - id+ GetScoresResponseTraceData:+ title: GetScoresResponseTraceData+ type: object+ properties:+ userId:type: string- description: >-- Human-readable deployment name. This is independent from the- evaluator name.- evaluator:- $ref: '#/components/schemas/unstableEvaluationRuleEvaluator'nullable: true- description: >-- Deprecated compatibility alias for `evaluators[0].evaluator`, or- `null`-- when the rule currently has no evaluator assignments.--- `id` identifies the evaluator family. The rule automatically uses- the-- latest available version of that evaluator.- enabled:- type: boolean- description: Desired enabled state configured by the client.- status:- $ref: '#/components/schemas/unstableEvaluationRuleStatus'- description: >-- Effective runtime status after Langfuse applies validation and- blocking rules.- pausedReason:- type: string+ description: The user ID associated with the trace referenced by score+ tags:+ type: array+ items:+ type: stringnullable: true- description: Machine-readable reason when `status=paused`, otherwise `null`.- pausedMessage:+ description: A list of tags associated with the trace referenced by score+ environment:type: stringnullable: true- description: Human-readable explanation when `status=paused`, otherwise `null`.- sampling:- type: number- format: double- description: |-- Fraction of matching target objects that should be evaluated.-- Must be greater than `0` and less than or equal to `1`.- - `1` means evaluate every matching target.- - `0.25` means evaluate approximately 25% of matching targets.- createdAt:- type: string- format: date-time- description: Timestamp when the evaluation rule was created.- updatedAt:+ description: The environment of the trace referenced by score+ sessionId:type: string- format: date-time- description: Timestamp when the evaluation rule was last updated.- required:- - id- - name- - evaluator- - enabled- - status- - pausedReason- - pausedMessage- - sampling- - createdAt- - updatedAt- unstableEvaluationRule:- title: unstableEvaluationRule+ nullable: true+ description: The session ID associated with the trace referenced by score+ GetScoresResponseDataNumeric:+ title: GetScoresResponseDataNumerictype: objectproperties:- evaluators:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleEvaluatorAssignment'- description: >-- Evaluators attached to this rule in deterministic assignment order.- A `null` mapping inherits the evaluator version's default mapping.- target:- $ref: '#/components/schemas/unstableEvaluationRuleTarget'- description: Target object type that should trigger scoring.- filter:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleFilter'- description: >-- List of filter conditions used to decide whether a target should be- evaluated.- mapping:- type: array- items:- $ref: '#/components/schemas/unstablePromptVariableMappingRead'- description: >-- Deprecated compatibility alias containing the effective mapping for- `evaluators[0]`.- required:- - evaluators- - target- - filter- - mapping+ trace:+ $ref: '#/components/schemas/GetScoresResponseTraceData'+ nullable: true+ allOf:+ - $ref: '#/components/schemas/NumericScore'+ GetScoresResponseDataCategorical:+ title: GetScoresResponseDataCategorical+ type: object+ properties:+ trace:+ $ref: '#/components/schemas/GetScoresResponseTraceData'+ nullable: true+ allOf:+ - $ref: '#/components/schemas/CategoricalScore'+ GetScoresResponseDataBoolean:+ title: GetScoresResponseDataBoolean+ type: object+ properties:+ trace:+ $ref: '#/components/schemas/GetScoresResponseTraceData'+ nullable: true+ allOf:+ - $ref: '#/components/schemas/BooleanScore'+ GetScoresResponseDataCorrection:+ title: GetScoresResponseDataCorrection+ type: object+ properties:+ trace:+ $ref: '#/components/schemas/GetScoresResponseTraceData'+ nullable: true+ allOf:+ - $ref: '#/components/schemas/CorrectionScore'+ GetScoresResponseDataText:+ title: GetScoresResponseDataText+ type: object+ properties:+ trace:+ $ref: '#/components/schemas/GetScoresResponseTraceData'+ nullable: trueallOf:- - $ref: '#/components/schemas/unstableEvaluationRuleBase'- unstableReadableV2EvaluationRule:- title: unstableReadableV2EvaluationRule+ - $ref: '#/components/schemas/TextScore'+ GetScoresResponseData:+ title: GetScoresResponseData+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - NUMERIC+ - $ref: '#/components/schemas/GetScoresResponseDataNumeric'+ required:+ - dataType+ title: GetScoresResponseDataNumeric+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CATEGORICAL+ - $ref: '#/components/schemas/GetScoresResponseDataCategorical'+ required:+ - dataType+ title: GetScoresResponseDataCategorical+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - BOOLEAN+ - $ref: '#/components/schemas/GetScoresResponseDataBoolean'+ required:+ - dataType+ title: GetScoresResponseDataBoolean+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - CORRECTION+ - $ref: '#/components/schemas/GetScoresResponseDataCorrection'+ required:+ - dataType+ title: GetScoresResponseDataCorrection+ - type: object+ allOf:+ - type: object+ properties:+ dataType:+ type: string+ enum:+ - TEXT+ - $ref: '#/components/schemas/GetScoresResponseDataText'+ required:+ - dataType+ title: GetScoresResponseDataText+ GetScoresResponse:+ title: GetScoresResponsetype: objectproperties:- evaluators:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleEvaluatorAssignment'- description: >-- Evaluators attached to this rule in deterministic assignment order.- A `null` mapping inherits the evaluator version's default mapping.- target:- $ref: '#/components/schemas/unstableEvaluationRuleTarget'- description: Target object type that should trigger scoring.- filter:- type: array- items:- $ref: '#/components/schemas/EvaluationRuleReadFilter'- description: >-- List of stored filter conditions returned verbatim. Historical- filters remain readable even when the current write contract no- longer accepts their shape.- mapping:+ data:type: arrayitems:- $ref: '#/components/schemas/unstablePromptVariableMappingRead'- description: >-- Deprecated compatibility alias containing the effective mapping for- `evaluators[0]`.+ $ref: '#/components/schemas/GetScoresResponseData'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: truerequired:- - evaluators- - target- - filter- - mapping- allOf:- - $ref: '#/components/schemas/unstableEvaluationRuleBase'- unstableLegacyEvaluationRule:- title: unstableLegacyEvaluationRule+ - data+ - meta+ PaginatedSessions:+ title: PaginatedSessionstype: object- description: >-- **Deprecated:** Legacy trace- or dataset-level evaluation rule returned- by list and get for migration.--- This resource is read-only through the unstable public API. Its mapping- preserves the trace, dataset item, or named observation selected for- each prompt variable.properties:- evaluators:- type: array- items:- $ref: >-- #/components/schemas/unstableLegacyEvaluationRuleEvaluatorAssignment- description: Evaluators attached to this rule in deterministic assignment order.- target:- $ref: '#/components/schemas/unstableLegacyEvaluationRuleTarget'- delay:- type: integer- description: Delay in milliseconds before the legacy evaluation job runs.- timeScope:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleTimeScope'- description: >-- Whether the legacy rule evaluates newly ingested data, existing- data, or both.- filter:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleFilter'- description: Stored filters used by the legacy trace or dataset rule.- mapping:+ data:type: arrayitems:- $ref: '#/components/schemas/unstableLegacyPromptVariableMapping'- description: >-- Stored variable mappings, including the trace, dataset item, or- named observation selected for each variable.+ $ref: '#/components/schemas/Session'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: truerequired:- - evaluators- - target- - delay- - timeScope- - filter- - mapping- allOf:- - $ref: '#/components/schemas/unstableEvaluationRuleBase'- unstableReadableEvaluationRule:- title: unstableReadableEvaluationRule- oneOf:- - $ref: '#/components/schemas/unstableReadableV2EvaluationRule'- - $ref: '#/components/schemas/unstableLegacyEvaluationRule'- description: >-- Evaluation rule returned by list and get, including read-only legacy- trace and dataset rules.- unstableEvaluationRules:- title: unstableEvaluationRules+ - data+ - meta+ Traces:+ title: Tracestype: object- description: Paginated list of evaluation rules.properties:data:type: arrayitems:- $ref: '#/components/schemas/unstableReadableEvaluationRule'- description: Evaluation rules in the current page.+ $ref: '#/components/schemas/TraceWithDetails'meta:$ref: '#/components/schemas/utilsMetaResponse'- description: Standard pagination metadata.+ _deprecation:+ $ref: '#/components/schemas/Deprecation'+ nullable: truerequired:- data- meta- unstableCreateEvaluationRuleRequest:- title: unstableCreateEvaluationRuleRequest- oneOf:- - $ref: '#/components/schemas/unstableCreateLlmAsJudgeEvaluationRuleRequest'- - $ref: '#/components/schemas/unstableCreateCodeEvaluationRuleRequest'- - $ref: >-- #/components/schemas/unstableCreateEvaluationRuleWithEvaluatorsRequest- description: >-- Request body for creating an evaluation rule.--- Checklist for agents and SDK clients:-- - reference an existing evaluator family by `evaluator.name` and- `evaluator.type`-- - choose `target=observation` or `target=experiment`-- - if `target=experiment` and you want a dataset filter, call `GET- /api/public/v2/datasets` first and use dataset `id` values in- `filter[].value`-- - for `llm_as_judge`, fetch or inspect the evaluator first and provide a- complete mapping for every prompt variable-- - for `code`, do not send variables or mappings; Langfuse stores the- fixed code runtime mapping automatically-- - optionally narrow execution with `filter`-- - set `enabled=true` only when you want live execution immediately- unstableCreateEvaluationRuleWithEvaluatorsRequest:- title: unstableCreateEvaluationRuleWithEvaluatorsRequest+ DeleteTraceResponse:+ title: DeleteTraceResponsetype: objectproperties:- name:+ message:type: string- description: Human-readable deployment name.- evaluators:- type: array- items:- $ref: >-- #/components/schemas/unstableCreateEvaluationRuleEvaluatorAssignment- description: >-- One or more evaluator assignments. Providing the deprecated- top-level `evaluator` or `mapping` fields alongside this is rejected- with `400`.--- Multiple assignments are supported on writable targets.- target:- $ref: '#/components/schemas/unstableEvaluationRuleTarget'- enabled:- type: boolean- sampling:- type: number- format: double- nullable: true- filter:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleFilter'- nullable: truerequired:- - name- - evaluators- - target- - enabled- unstableCreateLlmAsJudgeEvaluationRuleRequest:- title: unstableCreateLlmAsJudgeEvaluationRuleRequest+ - message+ Sort:+ title: Sorttype: objectproperties:- name:+ id:type: string- description: Human-readable deployment name.- evaluator:- $ref: >-- #/components/schemas/unstableLlmAsJudgeEvaluationRuleEvaluatorReference- description: >-- LLM-as-judge evaluator family to use.--- Use `name` and `type` from the evaluator endpoints. If `type` is- omitted, Langfuse defaults it to `llm_as_judge` for backwards- compatibility.-- Langfuse resolves that family to its latest version before saving- the rule.- target:- $ref: '#/components/schemas/unstableEvaluationRuleTarget'- description: Target object type to evaluate.- enabled:- type: boolean- description: Whether the deployment should be active immediately after creation.- sampling:- type: number- format: double- nullable: true- description: Optional sampling fraction. Defaults to `1`.- filter:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleFilter'- nullable: true- description: >-- Optional filter list.--- Omit or pass an empty list to evaluate all matching targets for the- selected `target`.-- Each filter object must use a column that is valid for that- `target`.-- For `target=experiment`, `column=datasetId` expects dataset `id`- values from `GET /api/public/v2/datasets`, not dataset names.- mapping:- type: array- items:- $ref: '#/components/schemas/unstablePromptVariableMappingInput'- description: |-- LLM-as-judge variable mappings.-- Every prompt variable must appear exactly once.- Build this list from the evaluator's `variables` array.required:- - name- - evaluator- - target- - enabled- - mapping- unstableCreateCodeEvaluationRuleRequest:- title: unstableCreateCodeEvaluationRuleRequest+ - id+ unstableDashboardWidgetView:+ title: unstableDashboardWidgetView+ type: string+ enum:+ - observations+ - scores-numeric+ - scores-boolean+ - scores-categorical+ unstableDashboardWidgetViewWithLegacy:+ title: unstableDashboardWidgetViewWithLegacy+ type: string+ enum:+ - observations+ - scores-numeric+ - scores-boolean+ - scores-categorical+ - traces+ description: |-+ Widget data view. Responses may include the legacy `traces` value for+ widgets created before this API existed.+ unstableDashboardWidgetChartType:+ title: unstableDashboardWidgetChartType+ type: string+ enum:+ - LINE_TIME_SERIES+ - AREA_TIME_SERIES+ - BAR_TIME_SERIES+ - HORIZONTAL_BAR+ - VERTICAL_BAR+ - PIE+ - NUMBER+ - HISTOGRAM+ - PIVOT_TABLE+ unstableDashboardWidgetMetricAggregation:+ title: unstableDashboardWidgetMetricAggregation+ type: string+ enum:+ - sum+ - avg+ - count+ - max+ - min+ - p50+ - p75+ - p90+ - p95+ - p99+ - histogram+ - uniq+ unstableDashboardWidgetDimension:+ title: unstableDashboardWidgetDimension+ type: object+ properties:+ field:+ type: string+ required:+ - field+ unstableDashboardWidgetMetric:+ title: unstableDashboardWidgetMetrictype: objectproperties:- name:+ measure:type: string- description: Human-readable deployment name.- evaluator:- $ref: '#/components/schemas/unstableCodeEvaluationRuleEvaluatorReference'- description: >-- Code evaluator family to use.--- Use `name` and `type` from the evaluator endpoints.-- Langfuse resolves that family to its latest version before saving- the rule.- target:- $ref: '#/components/schemas/unstableEvaluationRuleTarget'- description: Target object type to evaluate.- enabled:- type: boolean- description: Whether the deployment should be active immediately after creation.- sampling:- type: number- format: double- nullable: true- description: Optional sampling fraction. Defaults to `1`.- filter:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleFilter'- nullable: true- description: >-- Optional filter list.--- Omit or pass an empty list to evaluate all matching targets for the- selected `target`.-- Each filter object must use a column that is valid for that- `target`.-- For `target=experiment`, `column=datasetId` expects dataset `id`- values from `GET /api/public/v2/datasets`, not dataset names.+ agg:+ $ref: '#/components/schemas/unstableDashboardWidgetMetricAggregation'required:- - name- - evaluator- - target- - enabled- unstableUpdateEvaluationRuleRequest:- title: unstableUpdateEvaluationRuleRequest+ - measure+ - agg+ unstableDashboardWidgetFilter:+ title: unstableDashboardWidgetFiltertype: objectdescription: >-- Partial update body for an evaluation rule.+ A filter in Langfuse filter-state shape. The `value` shape and the+ allowed operators depend on `type`:- Provide only the fields you want to change.- An empty body is rejected.+ | `type` | `value` | operators |+ |---|---|---|- Practical guidance:+ | `string` | string | `=`, `contains`, `does not contain`, `starts+ with`, `ends with` |- - If you only want to rename the rule or change sampling, send just- those fields.+ | `number` | number | `=`, `>`, `<`, `>=`, `<=` |- - To add, remove, or remap evaluators, send `evaluators`. It replaces- the whole assignment set, so include every evaluator the rule should- keep.+ | `datetime` | ISO datetime string | `>`, `<`, `>=`, `<=` |- - `evaluators` cannot be combined with the deprecated- `evaluator`/`mapping` pair, which only ever addressed the first- assignment.+ | `boolean` | boolean | `=`, `<>` |- - If you change to an LLM-as-judge `evaluator`, send a fresh `mapping`- unless you are certain the existing mapping still matches the prompt- variables.+ | `null` | `""` | `is null`, `is not null` |- - If you change `target` for an LLM-as-judge rule, usually send both- `filter` and `mapping` in the same request.+ | `stringOptions` | list of strings | `any of`, `none of` |- - For code evaluator rules, omit `mapping`; Langfuse stores the fixed- code runtime mapping automatically.+ | `arrayOptions` | list of strings | `any of`, `none of`, `all of` |- - If you change an experiment `datasetId` filter, call `GET- /api/public/v2/datasets` and use dataset `id` values from that response.- properties:- name:- type: string- nullable: true- description: Updated deployment name.- evaluators:- type: array- items:- $ref: >-- #/components/schemas/unstableCreateEvaluationRuleEvaluatorAssignment- nullable: true- description: >-- Full replacement of the rule's evaluator assignments: entries that- are+ | `categoryOptions` | list of strings (requires `key`) | `any of`, `none+ of` |- not listed are detached.+ | `stringObject` | string (requires `key`, e.g. a metadata key) | same+ as `string` |+ | `numberObject` | number (requires `key`, e.g. a score name) | same as+ `number` |- Mutually exclusive with the deprecated `evaluator` and `mapping`- fields.- evaluator:- $ref: '#/components/schemas/unstableEvaluationRuleEvaluatorReference'+ | `booleanObject` | boolean (requires `key`) | `=`, `<>` |+ properties:+ column:+ type: string+ operator:+ type: string+ type:+ type: string+ value:nullable: true- description: >-- Deprecated single-evaluator alias: updates the first assignment- only. Prefer `evaluators`.--- Langfuse resolves the provided evaluator family to its latest- version before saving the rule.+ key:+ type: string+ nullable: true+ required:+ - column+ - operator+ - type+ unstableDashboardWidgetChartConfig:+ title: unstableDashboardWidgetChartConfig+ type: object+ description: |-+ Chart-specific widget configuration.- A rule's evaluator type cannot be changed: provide `name` for an- evaluator family of the rule's current type. To use a different- evaluator type, create a new rule.- target:- $ref: '#/components/schemas/unstableEvaluationRuleTarget'+ `type` must match the top-level `chartType`.+ `row_limit` applies to total-value charts and pivot tables.+ `bins` applies to histograms.+ `defaultSort` applies to pivot tables.+ properties:+ type:+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'+ row_limit:+ type: integernullable: true- description: Updated target object type.- enabled:+ show_value_labels:type: booleannullable: true- description: Updated desired enabled state.- sampling:- type: number- format: double- nullable: true- description: Updated sampling fraction.- filter:- type: array- items:- $ref: '#/components/schemas/unstableEvaluationRuleFilter'+ bins:+ type: integernullable: true- description: >-- Updated filter list.--- For `target=experiment`, `column=datasetId` expects dataset `id`- values from `GET /api/public/v2/datasets`, not dataset names.- mapping:- type: array- items:- $ref: '#/components/schemas/unstablePromptVariableMappingInput'+ defaultSort:+ $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'nullable: true- description: >-- Updated LLM-as-judge variable mappings.--- Do not send this field for code evaluator rules. Langfuse stores the- fixed code runtime mapping automatically and returns it in the- response.- unstableDeleteEvaluationRuleResponse:- title: unstableDeleteEvaluationRuleResponse- type: object- description: Confirmation response returned after successful deletion.- properties:- message:- type: string- description: Always `Evaluation rule successfully deleted`.required:- - message- unstableEvaluationRuleEvaluatorReference:- title: unstableEvaluationRuleEvaluatorReference+ - type+ unstableDashboardWidgetDefaultSort:+ title: unstableDashboardWidgetDefaultSorttype: object- description: >-- Evaluator family reference used when updating an evaluation rule.--- `name` identifies the evaluator family in the authenticated project- context.-- A rule's evaluator type cannot be changed, so this reference does not- accept a `type`; the family must match the rule's current evaluator- type.properties:- name:+ column:type: string- description: Evaluator family name.+ order:+ $ref: '#/components/schemas/unstableDashboardWidgetSortOrder'required:- - name- unstableLlmAsJudgeEvaluationRuleEvaluatorReference:- title: unstableLlmAsJudgeEvaluationRuleEvaluatorReference+ - column+ - order+ unstableDashboardWidgetSortOrder:+ title: unstableDashboardWidgetSortOrder+ type: string+ enum:+ - ASC+ - DESC+ unstableDashboardWidgetChartConfigInput:+ title: unstableDashboardWidgetChartConfigInputtype: object- description: >-- LLM-as-judge evaluator family reference used when creating an evaluation- rule.+ description: |-+ Input-side chart config. `type` is optional and defaults to the+ widget's `chartType`; when given it must match.properties:- name:- type: string- description: Evaluator family name.type:- $ref: '#/components/schemas/unstableLlmAsJudgeEvaluatorType'+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'nullable: true- description: Evaluator type. Defaults to `llm_as_judge` when omitted.- required:- - name- unstableCodeEvaluationRuleEvaluatorReference:- title: unstableCodeEvaluationRuleEvaluatorReference+ row_limit:+ type: integer+ nullable: true+ show_value_labels:+ type: boolean+ nullable: true+ bins:+ type: integer+ nullable: true+ defaultSort:+ $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'+ nullable: true+ unstableCreateDashboardWidgetRequest:+ title: unstableCreateDashboardWidgetRequesttype: object- description: Code evaluator family reference used when creating an evaluation rule.properties:name:type: string- description: Evaluator family name.- type:+ description:type: string- const: code- description: Must be `code`.+ nullable: true+ description: Defaults to an empty string.+ view:+ $ref: '#/components/schemas/unstableDashboardWidgetView'+ dimensions:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetDimension'+ metrics:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetMetric'+ filters:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'+ chartType:+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'+ chartConfig:+ $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'+ nullable: true+ description: Defaults to the plain config for `chartType`.required:- name- - type- unstableLlmAsJudgeEvaluatorType:- title: unstableLlmAsJudgeEvaluatorType- type: string- enum:- - llm_as_judge- unstableEvaluationRuleEvaluator:- title: unstableEvaluationRuleEvaluator+ - view+ - dimensions+ - metrics+ - filters+ - chartType+ unstableUpdateDashboardWidgetRequest:+ title: unstableUpdateDashboardWidgetRequesttype: object- description: |-- Resolved evaluator currently used by the evaluation rule.-- `id` identifies the evaluator family. Evaluation runs automatically use- the latest available evaluator version.properties:- id:- type: string- description: >-- Identifier of the exact evaluator version currently used by the rule.name:type: string- description: Evaluator family name.- type:- $ref: '#/components/schemas/unstableEvaluatorType'- description: Evaluator type.- required:- - id- - name- - type- unstableEvaluationRuleEvaluatorAssignment:- title: unstableEvaluationRuleEvaluatorAssignment- type: object- properties:- evaluator:- $ref: '#/components/schemas/unstableEvaluationRuleEvaluator'- mapping:+ nullable: true+ description:+ type: string+ nullable: true+ view:+ $ref: '#/components/schemas/unstableDashboardWidgetView'+ nullable: true+ dimensions:type: arrayitems:- $ref: '#/components/schemas/unstablePromptVariableMappingRead'+ $ref: '#/components/schemas/unstableDashboardWidgetDimension'nullable: true- description: >-- Rule-specific override, or `null` to inherit the evaluator version's- default mapping.- required:- - evaluator- - mapping- unstableLegacyEvaluationRuleEvaluatorAssignment:- title: unstableLegacyEvaluationRuleEvaluatorAssignment+ metrics:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetMetric'+ nullable: true+ filters:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'+ nullable: true+ chartType:+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'+ nullable: true+ chartConfig:+ $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'+ nullable: true+ unstableDashboardWidgetList:+ title: unstableDashboardWidgetListtype: object- description: '**Deprecated:** Evaluator assignment on a legacy trace or dataset rule.'properties:- evaluator:- $ref: '#/components/schemas/unstableEvaluationRuleEvaluator'- mapping:+ data:type: arrayitems:- $ref: '#/components/schemas/unstableLegacyPromptVariableMapping'- nullable: true+ $ref: '#/components/schemas/unstableDashboardWidget'+ meta:+ $ref: '#/components/schemas/utilsMetaResponse'+ required:+ - data+ - meta+ unstableDeleteDashboardWidgetResponse:+ title: unstableDeleteDashboardWidgetResponse+ type: object+ properties:+ message:+ type: stringrequired:- - evaluator- - mapping- unstableEvaluationRuleTimeScope:- title: unstableEvaluationRuleTimeScope- type: string- enum:- - NEW- - EXISTING- unstableLegacyEvaluationRuleTarget:- title: unstableLegacyEvaluationRuleTarget- type: string- enum:- - trace- - dataset- description: '**Deprecated:** Legacy trace or dataset evaluation target.'- unstableLegacyPromptVariableMapping:- title: unstableLegacyPromptVariableMapping+ - message+ unstableDashboardWidget:+ title: unstableDashboardWidgettype: object- description: >-- **Deprecated:** Connects one prompt variable to data from a trace,- dataset item, or named observation in a legacy rule.properties:- variable:+ id:type: string- langfuseObject:- $ref: '#/components/schemas/unstableLegacyEvaluationObject'- objectName:+ createdAt:type: string- nullable: true- description: >-- Observation name to match, or `null` when `langfuseObject` is- `trace` or `dataset_item`.- source:+ format: date-time+ updatedAt:type: string- jsonPath:+ format: date-time+ name:type: string- nullable: true- required:- - variable- - langfuseObject- - objectName- - source- unstableLegacyEvaluationObject:- title: unstableLegacyEvaluationObject- type: string- enum:- - trace- - span- - generation- - event- - agent- - tool- - chain- - retriever- - evaluator- - embedding- - guardrail- - dataset_item- description: >-- **Deprecated:** Legacy Langfuse object used by a trace or dataset- evaluation rule.- unstableCreateEvaluationRuleEvaluatorAssignment:- title: unstableCreateEvaluationRuleEvaluatorAssignment- type: object- properties:- evaluator:- $ref: '#/components/schemas/unstableCreateEvaluationRuleEvaluatorReference'- mapping:+ description:+ type: string+ view:+ $ref: '#/components/schemas/unstableDashboardWidgetViewWithLegacy'+ dimensions:type: arrayitems:- $ref: '#/components/schemas/unstablePromptVariableMappingInput'- nullable: true- description: >-- Optional rule-specific override. Omit to inherit the evaluator- version's default mapping. Omit for code evaluators.+ $ref: '#/components/schemas/unstableDashboardWidgetDimension'+ metrics:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetMetric'+ filters:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'+ chartType:+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'+ chartConfig:+ $ref: '#/components/schemas/unstableDashboardWidgetChartConfig'required:- - evaluator- unstableCreateEvaluationRuleEvaluatorReference:- title: unstableCreateEvaluationRuleEvaluatorReference+ - id+ - createdAt+ - updatedAt+ - name+ - description+ - view+ - dimensions+ - metrics+ - filters+ - chartType+ - chartConfig+ unstableDashboardPlacement:+ title: unstableDashboardPlacement+ oneOf:+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - widget+ - $ref: '#/components/schemas/unstableWidgetPlacement'+ required:+ - type+ title: UnstableWidgetPlacement+ - type: object+ allOf:+ - type: object+ properties:+ type:+ type: string+ enum:+ - preset+ - $ref: '#/components/schemas/unstablePresetPlacement'+ required:+ - type+ title: UnstablePresetPlacement+ description: |-+ A tile on the dashboard's 12-column grid. `x`/`y` are the tile's+ top-left cell (0-based; `y` grows downward), `width`/`height` its size+ in cells. The UI default tile is 6x6 (half width). Overlapping tiles+ are not rejected; prefer appending below existing tiles (or omit the+ position on create to let the server do it).+ unstableWidgetPlacement:+ title: unstableWidgetPlacementtype: objectproperties:- name:+ id:type: string- type:- $ref: '#/components/schemas/unstableEvaluatorType'- nullable: true- description: Defaults to `llm_as_judge`.+ widgetId:+ type: string+ x:+ type: integer+ 'y':+ type: integer+ width:+ type: integer+ height:+ type: integerrequired:- - name- unstableDeleteEvaluatorResponse:- title: unstableDeleteEvaluatorResponse+ - id+ - widgetId+ - x+ - 'y'+ - width+ - height+ unstablePresetPlacement:+ title: unstablePresetPlacementtype: object- description: Confirmation response returned after successful deletion.properties:- message:+ id:type: string- description: Always `Evaluator successfully deleted`.+ presetId:+ type: string+ x:+ type: integer+ 'y':+ type: integer+ width:+ type: integer+ height:+ type: integerrequired:- - message- unstableEvaluator:- title: unstableEvaluator+ - id+ - presetId+ - x+ - 'y'+ - width+ - height+ unstableCreateDashboardPlacementRequest:+ title: unstableCreateDashboardPlacementRequestoneOf:- type: objectallOf:@@ -19730,11 +16770,11 @@ components:type:type: stringenum:- - llm_as_judge- - $ref: '#/components/schemas/unstableLlmAsJudgeEvaluator'+ - widget+ - $ref: '#/components/schemas/unstableCreateWidgetPlacement'required:- type- title: UnstableLlmAsJudgeEvaluator+ title: UnstableCreateWidgetPlacement- type: objectallOf:- type: object@@ -19742,258 +16782,286 @@ components:type:type: stringenum:- - code- - $ref: '#/components/schemas/unstableCodeEvaluator'+ - preset+ - $ref: '#/components/schemas/unstableCreatePresetPlacement'required:- type- title: UnstableCodeEvaluator- description: >-- One evaluator that can be used for scoring.--- An evaluator describes **how** to score data.--- It does not define **which** live objects are evaluated. That is the job- of `evaluation-rules`.--- For agent clients, the most important fields are:-- - `type`: determines which evaluator fields are present-- - `variables`: for LLM evaluators, use these exact names when building- the evaluation-rule `mapping` array. LLM evaluators require every- variable to be mapped. Code evaluators always expose the fixed runtime- payload fields and Langfuse maps them automatically.--- Versioning behavior:-- - `GET /evaluators` returns the latest version of each available- evaluator.-- - `GET /evaluators/{id}` returns the latest version.-- - `id` identifies the evaluator family and remains stable when a new- version is created.-- - Evaluation rules always run against the latest version for the- selected evaluator name within the same source (`project` or `managed`).- unstableEvaluatorBase:- title: unstableEvaluatorBase+ title: UnstableCreatePresetPlacement+ unstableCreateWidgetPlacement:+ title: unstableCreateWidgetPlacement+ type: object+ properties:+ id:+ type: string+ nullable: true+ description: Server-generated when omitted.+ widgetId:+ type: string+ x:+ type: integer+ nullable: true+ description: Grid column (12-column grid). Defaults to `0`.+ 'y':+ type: integer+ nullable: true+ description: Grid row. Defaults to the first row below all existing tiles.+ width:+ type: integer+ nullable: true+ description: Width in grid columns. Defaults to `6`.+ height:+ type: integer+ nullable: true+ description: Height in grid rows. Defaults to `6`.+ required:+ - widgetId+ unstableCreatePresetPlacement:+ title: unstableCreatePresetPlacement+ type: object+ properties:+ id:+ type: string+ nullable: true+ description: Server-generated when omitted.+ presetId:+ type: string+ x:+ type: integer+ nullable: true+ description: Grid column (12-column grid). Defaults to `0`.+ 'y':+ type: integer+ nullable: true+ description: Grid row. Defaults to the first row below all existing tiles.+ width:+ type: integer+ nullable: true+ description: Width in grid columns. Defaults to `6`.+ height:+ type: integer+ nullable: true+ description: Height in grid rows. Defaults to `6`.+ required:+ - presetId+ unstableUpdateDashboardPlacementRequest:+ title: unstableUpdateDashboardPlacementRequest+ type: object+ properties:+ x:+ type: integer+ nullable: true+ description: Grid column (12-column grid).+ 'y':+ type: integer+ nullable: true+ description: Grid row.+ width:+ type: integer+ nullable: true+ description: Width in grid columns.+ height:+ type: integer+ nullable: true+ description: Height in grid rows.+ unstableDeleteDashboardPlacementResponse:+ title: unstableDeleteDashboardPlacementResponsetype: objectproperties:- id:- type: string- description: Stable identifier of this evaluator across all versions.- name:+ message:type: string- description: Evaluator name.- version:- type: integer- description: Version number of this evaluator.- variables:- type: array- items:- type: string- description: >-- Variables that can be mapped when creating an evaluation rule.--- LLM evaluators require every variable to be mapped exactly once.- Code evaluators always expose the fixed runtime payload fields and- Langfuse maps them automatically.- mapping:+ required:+ - message+ unstableDashboardDefinition:+ title: unstableDashboardDefinition+ type: object+ properties:+ widgets:type: arrayitems:- $ref: '#/components/schemas/unstablePromptVariableMappingRead'- nullable: true- description: >-- Default variable mapping for this evaluator version, or `null` when- no default is configured.--- An entry's `source` is `null` when that variable was never fully- configured, and sources-- are not restricted by rule `target` here, because the default is- stored on the evaluator-- rather than on any one rule.- evaluationRuleCount:- type: integer- description: >-- Number of evaluation rules in the project that currently use this- evaluator.+ $ref: '#/components/schemas/unstableDashboardPlacement'+ required:+ - widgets+ unstableDashboard:+ title: unstableDashboard+ type: object+ properties:+ id:+ type: stringcreatedAt:type: stringformat: date-time- description: Timestamp when this evaluator was created.updatedAt:type: stringformat: date-time- description: Timestamp when this evaluator was last updated.+ name:+ type: string+ description:+ type: string+ definition:+ $ref: '#/components/schemas/unstableDashboardDefinition'+ filters:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'+ description: Dashboard-level filters applied to all widgets on the dashboard.required:- id- - name- - version- - variables- - mapping- - evaluationRuleCount- createdAt- updatedAt- unstableLlmAsJudgeEvaluator:- title: unstableLlmAsJudgeEvaluator- type: object- properties:- prompt:- type: string- description: Prompt template used during evaluation.- outputDefinition:- $ref: '#/components/schemas/unstablePublicEvaluatorOutputDefinition'- description: >-- Structured output schema returned by this evaluator.--- Responses always include `dataType` and omit the internal- output-definition `version`.-- Use `dataType` to decide how future scores should be interpreted.- modelConfig:- $ref: '#/components/schemas/unstableEvaluatorModelConfig'- nullable: true- description: >-- Explicit model configuration, or `null` when the project default- evaluation model is used.- required:- - prompt- - outputDefinition- - modelConfig- allOf:- - $ref: '#/components/schemas/unstableEvaluatorBase'- unstableCodeEvaluator:- title: unstableCodeEvaluator- type: object- properties:- sourceCode:- type: string- description: Source code executed for each matched observation.- sourceCodeLanguage:- $ref: '#/components/schemas/unstableCodeEvaluatorSourceCodeLanguage'- description: Runtime language for `sourceCode`.- required:- - sourceCode- - sourceCodeLanguage- allOf:- - $ref: '#/components/schemas/unstableEvaluatorBase'- unstableEvaluators:- title: unstableEvaluators+ - name+ - description+ - definition+ - filters+ unstableDashboardList:+ title: unstableDashboardListtype: objectproperties:data:type: arrayitems:- $ref: '#/components/schemas/unstableEvaluator'+ $ref: '#/components/schemas/unstableDashboard'meta:$ref: '#/components/schemas/utilsMetaResponse'required:- data- meta- unstableCreateEvaluatorRequest:- title: unstableCreateEvaluatorRequest- oneOf:- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - llm_as_judge- - $ref: '#/components/schemas/unstableCreateLlmAsJudgeEvaluatorRequest'- required:- - type- title: UnstableCreateLlmAsJudgeEvaluatorRequest- - type: object- allOf:- - type: object- properties:- type:- type: string- enum:- - code- - $ref: '#/components/schemas/unstableCreateCodeEvaluatorRequest'- required:- - type- title: UnstableCreateCodeEvaluatorRequest- description: >-- Request body for creating an evaluator.--- If the same `name` already exists in your project, Langfuse creates the- next version and returns it.-- Existing evaluation rules automatically use the latest evaluator- version.-- If `type` is omitted, Langfuse defaults it to `llm_as_judge` for- backwards compatibility.- unstableCreateLlmAsJudgeEvaluatorRequest:- title: unstableCreateLlmAsJudgeEvaluatorRequest+ unstableCreateDashboardRequest:+ title: unstableCreateDashboardRequesttype: objectproperties:name:type: string- description: Evaluator name within the authenticated project.- prompt:+ description:type: string- description: Prompt template used by the evaluator.- outputDefinition:- $ref: '#/components/schemas/unstableEvaluatorOutputDefinition'- description: >-- Structured output schema the evaluator must return.--- Always send `dataType`.-- Do not send `version`; it is an internal storage detail and not part- of the public request contract.- modelConfig:- $ref: '#/components/schemas/unstableEvaluatorModelConfig'nullable: true- description: >-- Optional explicit model configuration. Omit or set to `null` to use- the project default evaluation model.- mapping:+ definition:+ $ref: '#/components/schemas/unstableDashboardDefinition'+ nullable: true+ filters:type: arrayitems:- $ref: '#/components/schemas/unstablePromptVariableMappingInput'+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'nullable: true- description: >-- Optional default variable mapping inherited by rule assignments that- do not provide an override.required:- name- - prompt- - outputDefinition- unstableCreateCodeEvaluatorRequest:- title: unstableCreateCodeEvaluatorRequest+ unstableUpdateDashboardRequest:+ title: unstableUpdateDashboardRequesttype: objectproperties:name:type: string- description: Evaluator name within the authenticated project.- sourceCode:+ nullable: true+ description:+ type: string+ nullable: true+ definition:+ $ref: '#/components/schemas/unstableDashboardDefinition'+ nullable: true+ filters:+ type: array+ items:+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'+ nullable: true+ unstableDeleteDashboardResponse:+ title: unstableDeleteDashboardResponse+ type: object+ properties:+ message:type: string- description: Code executed for each matched observation.- sourceCodeLanguage:- $ref: '#/components/schemas/unstableCodeEvaluatorSourceCodeLanguage'- description: Runtime language for `sourceCode`.required:- - name- - sourceCode- - sourceCodeLanguage+ - message+ unstablePublicApiErrorCode:+ title: unstablePublicApiErrorCode+ type: string+ enum:+ - authentication_failed+ - access_denied+ - invalid_request+ - invalid_query+ - invalid_body+ - resource_not_found+ - conflict+ - rate_limited+ - method_not_allowed+ - internal_error+ description: Machine-readable error code returned by unstable public API endpoints.+ unstablePublicApiValidationIssue:+ title: unstablePublicApiValidationIssue+ type: object+ description: >-+ A validation issue returned for malformed request bodies or query+ parameters.+ properties:+ code:+ type: string+ message:+ type: string+ path:+ type: array+ items: {}+ required:+ - code+ - message+ - path+ unstablePublicApiErrorDetails:+ title: unstablePublicApiErrorDetails+ type: object+ description: Optional structured context attached to an error.+ properties:+ issues:+ type: array+ items:+ $ref: '#/components/schemas/unstablePublicApiValidationIssue'+ nullable: true+ field:+ type: string+ nullable: true+ column:+ type: string+ nullable: true+ invalidValues:+ type: array+ items:+ type: string+ nullable: true+ allowedValues:+ type: array+ items:+ type: string+ nullable: true+ value:+ type: string+ nullable: true+ retryAfterSeconds:+ type: integer+ nullable: true+ limit:+ type: integer+ nullable: true+ remaining:+ type: integer+ nullable: true+ resetAt:+ type: string+ nullable: true+ unstablePublicApiError:+ title: unstablePublicApiError+ type: object+ description: Standard error envelope for unstable public API endpoints.+ properties:+ message:+ type: string+ description: Human-readable description of the failure.+ code:+ $ref: '#/components/schemas/unstablePublicApiErrorCode'+ description: Machine-readable error code.+ details:+ $ref: '#/components/schemas/unstablePublicApiErrorDetails'+ nullable: true+ description: Optional structured error context.+ required:+ - message+ - codeutilsMetaResponse:title: utilsMetaResponsetype: objectdiff --git a/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap b/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snapindex 2f0c41a22b..21f53d21e8 100644--- a/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap+++ b/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap@@ -86,20 +86,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1"DELETE unstable/dashboards/[dashboardId]/placements/[placementId] | org/bearer": 401,"DELETE unstable/dashboards/[dashboardId]/placements/[placementId] | project/basic": 404,"DELETE unstable/dashboards/[dashboardId]/placements/[placementId] | project/bearer": 403,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | admin/admin": 401,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | agent/basic": 401,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | org/basic": 403,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | org/bearer": 401,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | project/basic": 404,- "DELETE unstable/evaluation-rules/[evaluationRuleId] | project/bearer": 403,- "DELETE unstable/evaluators/[evaluatorId] | admin/admin": 401,- "DELETE unstable/evaluators/[evaluatorId] | agent/basic": 401,- "DELETE unstable/evaluators/[evaluatorId] | agent/bearer": 401,- "DELETE unstable/evaluators/[evaluatorId] | org/basic": 403,- "DELETE unstable/evaluators/[evaluatorId] | org/bearer": 401,- "DELETE unstable/evaluators/[evaluatorId] | project/basic": 404,- "DELETE unstable/evaluators/[evaluatorId] | project/bearer": 403,"DELETE v2/evaluation-rules/[evaluationRuleId] | admin/admin": 401,"DELETE v2/evaluation-rules/[evaluationRuleId] | agent/basic": 401,"DELETE v2/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,@@ -359,34 +345,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1"GET unstable/dashboards/index | org/bearer": 401,"GET unstable/dashboards/index | project/basic": 200,"GET unstable/dashboards/index | project/bearer": 403,- "GET unstable/evaluation-rules/[evaluationRuleId] | admin/admin": 401,- "GET unstable/evaluation-rules/[evaluationRuleId] | agent/basic": 401,- "GET unstable/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,- "GET unstable/evaluation-rules/[evaluationRuleId] | org/basic": 403,- "GET unstable/evaluation-rules/[evaluationRuleId] | org/bearer": 401,- "GET unstable/evaluation-rules/[evaluationRuleId] | project/basic": 404,- "GET unstable/evaluation-rules/[evaluationRuleId] | project/bearer": 403,- "GET unstable/evaluation-rules/index | admin/admin": 401,- "GET unstable/evaluation-rules/index | agent/basic": 401,- "GET unstable/evaluation-rules/index | agent/bearer": 401,- "GET unstable/evaluation-rules/index | org/basic": 403,- "GET unstable/evaluation-rules/index | org/bearer": 401,- "GET unstable/evaluation-rules/index | project/basic": 200,- "GET unstable/evaluation-rules/index | project/bearer": 403,- "GET unstable/evaluators/[evaluatorId] | admin/admin": 401,- "GET unstable/evaluators/[evaluatorId] | agent/basic": 401,- "GET unstable/evaluators/[evaluatorId] | agent/bearer": 401,- "GET unstable/evaluators/[evaluatorId] | org/basic": 403,- "GET unstable/evaluators/[evaluatorId] | org/bearer": 401,- "GET unstable/evaluators/[evaluatorId] | project/basic": 404,- "GET unstable/evaluators/[evaluatorId] | project/bearer": 403,- "GET unstable/evaluators/index | admin/admin": 401,- "GET unstable/evaluators/index | agent/basic": 401,- "GET unstable/evaluators/index | agent/bearer": 401,- "GET unstable/evaluators/index | org/basic": 403,- "GET unstable/evaluators/index | org/bearer": 401,- "GET unstable/evaluators/index | project/basic": 200,- "GET unstable/evaluators/index | project/bearer": 403,"GET v2/datasets/[datasetName]/index | admin/admin": 401,"GET v2/datasets/[datasetName]/index | agent/basic": 401,"GET v2/datasets/[datasetName]/index | agent/bearer": 401,@@ -527,13 +485,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1"PATCH unstable/dashboards/[dashboardId]/placements/[placementId] | org/bearer": 401,"PATCH unstable/dashboards/[dashboardId]/placements/[placementId] | project/basic": 400,"PATCH unstable/dashboards/[dashboardId]/placements/[placementId] | project/bearer": 403,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | admin/admin": 401,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | agent/basic": 401,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | org/basic": 403,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | org/bearer": 401,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | project/basic": 400,- "PATCH unstable/evaluation-rules/[evaluationRuleId] | project/bearer": 403,"PATCH v2/evaluation-rules/[evaluationRuleId] | admin/admin": 401,"PATCH v2/evaluation-rules/[evaluationRuleId] | agent/basic": 401,"PATCH v2/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,@@ -681,20 +632,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1"POST unstable/dashboards/index | org/bearer": 401,"POST unstable/dashboards/index | project/basic": 400,"POST unstable/dashboards/index | project/bearer": 403,- "POST unstable/evaluation-rules/index | admin/admin": 401,- "POST unstable/evaluation-rules/index | agent/basic": 401,- "POST unstable/evaluation-rules/index | agent/bearer": 401,- "POST unstable/evaluation-rules/index | org/basic": 403,- "POST unstable/evaluation-rules/index | org/bearer": 401,- "POST unstable/evaluation-rules/index | project/basic": 400,- "POST unstable/evaluation-rules/index | project/bearer": 403,- "POST unstable/evaluators/index | admin/admin": 401,- "POST unstable/evaluators/index | agent/basic": 401,- "POST unstable/evaluators/index | agent/bearer": 401,- "POST unstable/evaluators/index | org/basic": 403,- "POST unstable/evaluators/index | org/bearer": 401,- "POST unstable/evaluators/index | project/basic": 400,- "POST unstable/evaluators/index | project/bearer": 403,"POST v2/datasets/index | admin/admin": 401,"POST v2/datasets/index | agent/basic": 401,"POST v2/datasets/index | agent/bearer": 401,diff --git a/web/src/__tests__/server/project-route-auth-parity.servertest.ts b/web/src/__tests__/server/project-route-auth-parity.servertest.tsindex b92cc1dc03..b3280c6ffe 100644--- a/web/src/__tests__/server/project-route-auth-parity.servertest.ts+++ b/web/src/__tests__/server/project-route-auth-parity.servertest.ts@@ -97,13 +97,6 @@ const projectRoutes: Route[] = [route: "unstable/dashboard-widgets/[widgetId]",methods: ["GET", "PATCH", "DELETE"],},- { route: "unstable/evaluators/index", methods: ["GET", "POST"] },- { route: "unstable/evaluators/[evaluatorId]", methods: ["GET", "DELETE"] },- { route: "unstable/evaluation-rules/index", methods: ["GET", "POST"] },- {- route: "unstable/evaluation-rules/[evaluationRuleId]",- methods: ["GET", "PATCH", "DELETE"],- },{ route: "v2/evaluators/index", methods: ["GET", "POST"] },{route: "v2/evaluators/[evaluatorId]",diff --git a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts b/web/src/__tests__/server/unit/openapi-deprecations.servertest.tsindex 991af5cb04..e352d01c48 100644--- a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts+++ b/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts@@ -35,18 +35,6 @@ const openApiPath = path.resolve();const MESSAGE = "Use `GET /api/public/v2/observations?from=<from>` instead.";-const UNSTABLE_EVALS_SUNSET_HUMAN = "September 4, 2026";-const UNSTABLE_EVAL_OPERATIONS = new Set([- "POST /api/public/unstable/evaluators",- "GET /api/public/unstable/evaluators",- "GET /api/public/unstable/evaluators/{evaluatorId}",- "DELETE /api/public/unstable/evaluators/{evaluatorId}",- "POST /api/public/unstable/evaluation-rules",- "GET /api/public/unstable/evaluation-rules",- "GET /api/public/unstable/evaluation-rules/{evaluationRuleId}",- "PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}",- "DELETE /api/public/unstable/evaluation-rules/{evaluationRuleId}",-]);/** Spec fixture in the shape `fern export` produces, aliased `security` included. */const SPEC = `openapi: 3.0.1@@ -139,11 +127,10 @@ describe("OpenAPI deprecations", () => {definitionDirectory,)) {const operation = `${method.toUpperCase()} ${endpointPath}`;- const sunsetDate = UNSTABLE_EVAL_OPERATIONS.has(operation)- ? UNSTABLE_EVALS_SUNSET_HUMAN- : V3_SUNSET_HUMAN;- expect(message, operation).toContain(`will be removed on ${sunsetDate}.`);+ expect(message, operation).toContain(+ `will be removed on ${V3_SUNSET_HUMAN}.`,+ );}});diff --git a/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts b/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.tsindex 6e5538249e..6d92c99f44 100644--- a/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts+++ b/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts@@ -118,15 +118,6 @@ describe("Python SDK deprecations", () => {"sessions:get","trace:get","trace:list",- "unstable/evaluation_rules:create",- "unstable/evaluation_rules:list",- "unstable/evaluation_rules:get",- "unstable/evaluation_rules:update",- "unstable/evaluation_rules:delete",- "unstable/evaluators:create",- "unstable/evaluators:list",- "unstable/evaluators:get",- "unstable/evaluators:delete",]);});diff --git a/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts b/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.tsindex 43c9afd798..9ee7ac7ca1 100644--- a/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts+++ b/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts@@ -146,15 +146,6 @@ describe("TypeScript SDK deprecations", () => {"api/resources/sessions/client/Client.ts:get","api/resources/trace/client/Client.ts:get","api/resources/trace/client/Client.ts:list",- "api/resources/unstable/resources/evaluationRules/client/Client.ts:create",- "api/resources/unstable/resources/evaluationRules/client/Client.ts:list",- "api/resources/unstable/resources/evaluationRules/client/Client.ts:get",- "api/resources/unstable/resources/evaluationRules/client/Client.ts:update",- "api/resources/unstable/resources/evaluationRules/client/Client.ts:delete",- "api/resources/unstable/resources/evaluators/client/Client.ts:create",- "api/resources/unstable/resources/evaluators/client/Client.ts:list",- "api/resources/unstable/resources/evaluators/client/Client.ts:get",- "api/resources/unstable/resources/evaluators/client/Client.ts:delete",]);expect(deprecationsdiff --git a/web/src/__tests__/server/unit/unstable-evals-contract.servertest.ts b/web/src/__tests__/server/unit/unstable-evals-contract.servertest.tsdeleted file mode 100644index 63da9f8d13..0000000000--- a/web/src/__tests__/server/unit/unstable-evals-contract.servertest.ts+++ /dev/null@@ -1,949 +0,0 @@-import {- createNumericEvalOutputDefinition,- EvalTargetObject,- JobConfigState,- CODE_EVAL_TEMPLATE_VARIABLES,-} from "@langfuse/shared";-import { CODE_EVAL_SOURCE_MAX_BYTES } from "@langfuse/shared/src/server";-import { EvalTemplateType, Prisma } from "@langfuse/shared/src/db";-import {- toApiEvaluator,- toApiV2EvaluationRule,- toEvaluationRuleInput,-} from "@/src/features/evals/server/unstable-public-api/adapters";-import { StructuredPublicApiError } from "@/src/features/public-api";-import type {- StoredPublicEvaluatorTemplate,- StoredPublicV2EvaluationRule,-} from "@/src/features/evals/server/unstable-public-api/types";-import {- GetUnstableEvaluationRulesQuery,- PatchUnstableEvaluationRuleBody,- PostUnstableEvaluationRuleBody,-} from "@/src/features/public-api/types/unstable-evaluation-rules";-import {- GetUnstableEvaluatorsQuery,- PostUnstableEvaluatorBody,-} from "@/src/features/public-api/types/unstable-evaluators";-import {- PUBLIC_EVALUATOR_TYPE_CODE,- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,-} from "@/src/features/public-api/types/unstable-public-evals-contract";--const numericOutputDefinition = createNumericEvalOutputDefinition({- reasoningDescription: "Why the score was assigned",- scoreDescription: "A score between 0 and 1",- minValue: 0,- maxValue: 1,-});--const expectStructuredPublicApiError = (- fn: () => unknown,- params: {- code: StructuredPublicApiError["code"];- message?: string;- details?: Record<string, unknown>;- },-) => {- try {- fn();- throw new Error("Expected function to throw StructuredPublicApiError");- } catch (error) {- expect(error).toBeInstanceOf(StructuredPublicApiError);-- const unstableError = error as StructuredPublicApiError;- expect(unstableError.code).toBe(params.code);-- if (params.message) {- expect(unstableError.message).toContain(params.message);- }-- if (params.details) {- expect(unstableError.details).toMatchObject(params.details);- }- }-};--describe("unstable public eval contracts", () => {- it("ignores unknown evaluator creation fields for forward compatibility", () => {- const parsed = PostUnstableEvaluatorBody.parse({- name: "Answer correctness",- description: "future field",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- modelConfig: {- provider: "openai",- model: "gpt-4.1-mini",- futureSetting: "ignored",- },- });-- expect(parsed).toEqual({- name: "Answer correctness",- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- prompt: "Judge {{input}} against {{output}}",- outputDefinition: {- dataType: "NUMERIC",- reasoning: {- description: "Why the score was assigned",- },- score: {- description: "A score between 0 and 1",- minValue: 0,- maxValue: 1,- },- },- modelConfig: {- provider: "openai",- model: "gpt-4.1-mini",- },- });- });-- it("accepts evaluator default mappings", () => {- const parsed = PostUnstableEvaluatorBody.parse({- name: "Answer correctness",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- mapping: [- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ],- });-- expect(parsed.mapping).toEqual([- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ]);- });-- it("requires outputDefinition.dataType for evaluator creation", () => {- const parsed = PostUnstableEvaluatorBody.safeParse({- name: "Answer correctness",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: {- reasoning: "Explain why the answer is correct or incorrect.",- score: "Return a score between 0 and 1.",- },- });-- expect(parsed.success).toBe(false);- expect(parsed.error?.issues).toEqual(- expect.arrayContaining([- expect.objectContaining({- path: ["outputDefinition", "dataType"],- }),- ]),- );- });-- it("accepts explicitly typed code evaluator creation bodies", () => {- const parsed = PostUnstableEvaluatorBody.parse({- name: "Exact match",- type: PUBLIC_EVALUATOR_TYPE_CODE,- sourceCode:- 'function evaluate() { return { scores: [{ name: "match", value: true, dataType: "BOOLEAN" }] }; }',- sourceCodeLanguage: "TYPESCRIPT",- });-- expect(parsed).toEqual({- name: "Exact match",- type: PUBLIC_EVALUATOR_TYPE_CODE,- sourceCode:- 'function evaluate() { return { scores: [{ name: "match", value: true, dataType: "BOOLEAN" }] }; }',- sourceCodeLanguage: "TYPESCRIPT",- });- });-- it("rejects LLM-only fields on code evaluator creation bodies", () => {- const parsed = PostUnstableEvaluatorBody.safeParse({- name: "Exact match",- type: PUBLIC_EVALUATOR_TYPE_CODE,- prompt: "Judge {{input}}",- sourceCode:- 'function evaluate() { return { scores: [{ name: "match", value: true, dataType: "BOOLEAN" }] }; }',- sourceCodeLanguage: "TYPESCRIPT",- });-- expect(parsed.success).toBe(false);- expect(parsed.error?.issues).toEqual(- expect.arrayContaining([- expect.objectContaining({- path: ["prompt"],- }),- ]),- );- });-- it("rejects code evaluator creation bodies when source code is too large", () => {- const parsed = PostUnstableEvaluatorBody.safeParse({- name: "Oversized code",- type: PUBLIC_EVALUATOR_TYPE_CODE,- sourceCode: "a".repeat(CODE_EVAL_SOURCE_MAX_BYTES + 1),- sourceCodeLanguage: "TYPESCRIPT",- });-- expect(parsed.success).toBe(false);- expect(parsed.error?.issues).toEqual(- expect.arrayContaining([- expect.objectContaining({- path: ["sourceCode"],- message: `Source code must be ${CODE_EVAL_SOURCE_MAX_BYTES} bytes or less`,- }),- ]),- );- });-- it("rejects observation evaluation rules that use expected_output mappings", () => {- const parsed = PostUnstableEvaluationRuleBody.safeParse({- name: "answer_quality",- evaluator: {- name: "Answer correctness",- },- target: "observation",- enabled: true,- sampling: 1,- filter: [],- mapping: [- { variable: "output", source: "output" },- { variable: "expected_output", source: "expected_output" },- ],- });-- expect(parsed.success).toBe(false);- });-- it("accepts both root filters on code rule bodies without mappings", () => {- const rootFilters = [- {- type: "boolean",- column: "isRootObservation",- operator: "=",- value: true,- },- {- type: "null",- column: "parentObservationId",- operator: "is null",- value: "",- },- ] as const;- const parsed = PostUnstableEvaluationRuleBody.parse({- name: "toxicity-code-live",- evaluator: {- name: "Toxicity detector",- type: PUBLIC_EVALUATOR_TYPE_CODE,- },- target: "observation",- enabled: true,- sampling: 1,- filter: rootFilters,- });-- expect(parsed.filter).toEqual(rootFilters);- expect("mapping" in parsed).toBe(false);- expect("variableMapping" in parsed).toBe(false);- });-- it("keeps trace rules read-only", () => {- const parsed = PostUnstableEvaluationRuleBody.safeParse({- name: "trace-quality",- evaluator: {- name: "Quality",- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- },- target: "trace",- enabled: true,- filter: [],- mapping: [- {- variable: "input",- langfuseObject: "trace",- objectName: null,- source: "input",- },- {- variable: "output",- langfuseObject: "generation",- objectName: "answer-generation",- source: "output",- },- ],- });-- expect(parsed.success).toBe(false);- });-- it("rejects code evaluation rule create bodies with mappings", () => {- const parsed = PostUnstableEvaluationRuleBody.safeParse({- name: "toxicity-code-live",- evaluator: {- name: "Toxicity detector",- type: PUBLIC_EVALUATOR_TYPE_CODE,- },- target: "observation",- enabled: true,- sampling: 1,- filter: [],- mapping: [{ variable: "output", source: "output" }],- });-- expect(parsed.success).toBe(false);- });-- it("accepts LLM evaluation rule create bodies without mappings to inherit evaluator defaults", () => {- const parsed = PostUnstableEvaluationRuleBody.safeParse({- name: "answer-correctness-live",- evaluator: {- name: "Answer correctness",- },- target: "observation",- enabled: true,- sampling: 1,- filter: [],- });-- expect(parsed.success).toBe(true);- });-- it("accepts multiple evaluator assignments", () => {- const parsed = PostUnstableEvaluationRuleBody.parse({- name: "answer-quality-live",- evaluators: [- {- evaluator: { name: "Correctness" },- },- {- evaluator: { name: "Tone" },- mapping: [{ variable: "output", source: "output" }],- },- ],- target: "observation",- enabled: true,- filter: [],- });-- expect(parsed.evaluators).toHaveLength(2);- });-- it("rejects create bodies containing both compatibility and array evaluator fields", () => {- const parsed = PostUnstableEvaluationRuleBody.safeParse({- name: "answer-quality-live",- evaluator: { name: "Correctness" },- evaluators: [{ evaluator: { name: "Tone" } }],- target: "observation",- enabled: true,- filter: [],- });-- expect(parsed.success).toBe(false);- });-- it("rejects create bodies that pair the deprecated mapping alias with evaluators", () => {- const parsed = PostUnstableEvaluationRuleBody.safeParse({- name: "answer-quality-live",- evaluators: [{ evaluator: { name: "Tone" } }],- mapping: [{ variable: "output", source: "output" }],- target: "observation",- enabled: true,- filter: [],- });-- expect(parsed.success).toBe(false);- });-- it("accepts evaluators on patch bodies as a full assignment replacement", () => {- const parsed = PatchUnstableEvaluationRuleBody.parse({- evaluators: [- { evaluator: { name: "Correctness" } },- {- evaluator: { name: "Tone" },- mapping: [{ variable: "output", source: "output" }],- },- ],- });-- expect(parsed).toMatchObject({ evaluators: expect.any(Array) });- });-- it("rejects patch bodies pairing evaluators with the deprecated aliases", () => {- for (const conflicting of [- { evaluator: { name: "Correctness" } },- {- target: "observation" as const,- mapping: [{ variable: "output", source: "output" as const }],- },- ]) {- expect(- PatchUnstableEvaluationRuleBody.safeParse({- evaluators: [{ evaluator: { name: "Tone" } }],- ...conflicting,- }).success,- ).toBe(false);- }- });-- it("rejects experiment filter updates unless target is provided", () => {- const parsed = PatchUnstableEvaluationRuleBody.safeParse({- filter: [- {- type: "stringOptions",- column: "datasetId",- operator: "any of",- value: ["dataset-1"],- },- ],- });-- expect(parsed.success).toBe(false);- expect(parsed.error?.issues.length).toBeGreaterThan(0);- });-- it("rejects non-positive unstable pagination limits", () => {- expect(- GetUnstableEvaluatorsQuery.safeParse({- page: 1,- limit: 0,- }).success,- ).toBe(false);- expect(- GetUnstableEvaluationRulesQuery.safeParse({- page: 1,- limit: -1,- }).success,- ).toBe(false);- });-- it("rejects non-integer unstable pagination values", () => {- expect(- GetUnstableEvaluatorsQuery.safeParse({- page: 1.5,- limit: 10,- }).success,- ).toBe(false);- expect(- GetUnstableEvaluationRulesQuery.safeParse({- page: 1,- limit: 1.5,- }).success,- ).toBe(false);- });-- it("ignores unknown evaluation rule update fields for forward compatibility", () => {- const parsed = PatchUnstableEvaluationRuleBody.parse({- name: "answer_quality",- newFieldFromFutureVersion: true,- });-- expect(parsed).toEqual({- name: "answer_quality",- });- });-- it("preserves target-specific patch fields instead of dropping them into the untargeted schema", () => {- const parsed = PatchUnstableEvaluationRuleBody.parse({- name: "experiment-expected-output-match",- target: "experiment",- filter: [- {- type: "stringOptions",- column: "datasetId",- operator: "any of",- value: ["dataset-prod"],- },- ],- mapping: [- { variable: "output", source: "output" },- { variable: "expected_output", source: "expected_output" },- ],- });-- expect(parsed).toEqual({- name: "experiment-expected-output-match",- target: "experiment",- filter: [- {- type: "stringOptions",- column: "datasetId",- operator: "any of",- value: ["dataset-prod"],- },- ],- mapping: [- { variable: "output", source: "output" },- { variable: "expected_output", source: "expected_output" },- ],- });- });-- it("rejects invalid target-specific patch payloads instead of silently parsing a name-only subset", () => {- const parsed = PatchUnstableEvaluationRuleBody.safeParse({- name: "experiment-expected-output-match",- target: "experiment",- filter: [- // The experiment-root filter is what `target: "experiment"` means, so- // it is not separately addressable on either target.- {- type: "boolean",- column: "isExperimentItemRootSpan",- operator: "=",- value: true,- },- ],- });-- expect(parsed.success).toBe(false);- expect(parsed.error?.issues.length).toBeGreaterThan(0);- });-- it("accepts observation filters on the experiment target", () => {- // An experiment rule is an observation rule scoped to experiment root- // spans, so every observation filter stays available alongside datasetId.- const parsed = PatchUnstableEvaluationRuleBody.safeParse({- target: "experiment",- filter: [- {- type: "stringOptions",- column: "type",- operator: "any of",- value: ["GENERATION"],- },- {- type: "stringOptions",- column: "datasetId",- operator: "any of",- value: ["dataset-prod"],- },- ],- });-- expect(parsed.success).toBe(true);- });-- it("strips evaluator type from patch bodies so a rule's evaluator type cannot be changed", () => {- const parsed = PatchUnstableEvaluationRuleBody.parse({- evaluator: {- name: "toxicity-detector",- type: "llm_as_judge",- },- });-- // `type` is dropped: the service inherits the rule's current evaluator type,- // so a code rule cannot be retargeted to an LLM evaluator family.- expect(parsed).toEqual({- evaluator: { name: "toxicity-detector" },- });- expect(- (parsed as { evaluator: { type?: string } }).evaluator.type,- ).toBeUndefined();- });-});--describe("unstable public eval adapters", () => {- it("returns an empty effective mapping for zero-variable evaluators", () => {- const createdAt = new Date("2026-08-14T00:00:00.000Z");- const rule = {- id: "rule_zero_variables",- createdAt,- updatedAt: createdAt,- projectId: "project_123",- createdByUserId: null,- name: "Always pass",- status: JobConfigState.ACTIVE,- targetObject: EvalTargetObject.EVENT,- filter: [],- sampling: new Prisma.Decimal(1),- delay: 0,- timeScope: ["NEW"],- assignments: [- {- id: "assignment_zero_variables",- createdAt,- updatedAt: createdAt,- projectId: "project_123",- evaluationRuleId: "rule_zero_variables",- evaluatorId: "evaluator_zero_variables",- variableMapping: null,- evaluator: {- id: "evaluator_zero_variables",- createdAt,- updatedAt: createdAt,- projectId: "project_123",- name: "Always pass",- type: EvalTemplateType.LLM_AS_JUDGE,- description: null,- createdByUserId: null,- blockedAt: null,- blockReason: null,- blockMessage: null,- versions: [- {- id: "version_zero_variables",- createdAt,- evaluatorId: "evaluator_zero_variables",- version: 1,- createdByUserId: null,- prompt: "Always return a score of 1",- promptMessages: null,- partner: null,- model: null,- provider: null,- modelParams: null,- vars: [],- variableMapping: null,- outputDefinition: numericOutputDefinition,- sourceCode: null,- sourceCodeLanguage: null,- },- ],- },- },- ],- } satisfies StoredPublicV2EvaluationRule;-- expect(toApiV2EvaluationRule(rule)).toMatchObject({- mapping: [],- evaluators: [{ mapping: null }],- });- });-- it("translates evaluation rule writes into job configuration inputs", () => {- const writeModel = toEvaluationRuleInput({- input: {- name: "expected_output_match",- target: "experiment",- enabled: false,- sampling: 0.25,- filter: [- {- type: "stringOptions",- column: "datasetId",- operator: "any of",- value: ["dataset-1"],- },- ],- mapping: [- { variable: "output", source: "output" },- { variable: "expected_output", source: "expected_output" },- ],- },- evaluatorVariables: ["output", "expected_output"],- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- });-- expect(writeModel).toMatchObject({- scoreName: "expected_output_match",- targetObject: EvalTargetObject.EXPERIMENT,- sampling: 0.25,- status: JobConfigState.INACTIVE,- filter: [- {- type: "stringOptions",- column: "experimentDatasetId",- operator: "any of",- value: ["dataset-1"],- },- ],- variableMapping: [- {- templateVariable: "output",- selectedColumnId: "output",- jsonSelector: null,- },- {- templateVariable: "expected_output",- selectedColumnId: "experimentItemExpectedOutput",- jsonSelector: null,- },- ],- });- });-- it("rejects invalid static filter option values", () => {- expectStructuredPublicApiError(- () =>- toEvaluationRuleInput({- input: {- name: "answer_quality",- target: "observation",- enabled: true,- sampling: 1,- filter: [- {- type: "stringOptions",- column: "type",- operator: "any of",- value: ["NOT_A_REAL_TYPE"],- },- ],- mapping: [{ variable: "input", source: "input" }],- },- evaluatorVariables: ["input"],- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- }),- {- code: "invalid_filter_value",- details: {- field: "filter[0].value",- column: "type",- invalidValues: ["NOT_A_REAL_TYPE"],- },- },- );- });-- it("rejects malformed jsonPath selectors", () => {- expectStructuredPublicApiError(- () =>- toEvaluationRuleInput({- input: {- name: "metadata_projection",- target: "observation",- enabled: true,- sampling: 1,- filter: [],- mapping: [- {- variable: "input",- source: "metadata",- jsonPath: "$[",- },- ],- },- evaluatorVariables: ["input"],- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- }),- {- code: "invalid_json_path",- details: {- field: "mapping[0].jsonPath",- variable: "input",- value: "$[",- },- },- );- });-- it("rejects mapping sources that are incompatible with the selected target", () => {- expectStructuredPublicApiError(- () =>- toEvaluationRuleInput({- input: {- name: "answer_quality",- target: "observation",- enabled: true,- sampling: 1,- filter: [],- mapping: [- { variable: "expected_output", source: "expected_output" },- ],- },- evaluatorVariables: ["expected_output"],- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- }),- {- code: "invalid_variable_mapping",- details: {- field: "mapping[0].source",- variable: "expected_output",- },- },- );- });-- it("accepts tool_calls mappings for both targets", () => {- for (const target of ["observation", "experiment"] as const) {- const writeModel = toEvaluationRuleInput({- input: {- name: "tool_call_check",- target,- enabled: true,- sampling: 1,- filter: [],- mapping: [- { variable: "calls", source: "tool_calls", jsonPath: "$[*].name" },- ],- },- evaluatorVariables: ["calls"],- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- });-- expect(writeModel.variableMapping).toEqual([- {- templateVariable: "calls",- selectedColumnId: "toolCalls",- jsonSelector: "$[*].name",- },- ]);- }- });-- it("rejects missing evaluator variable mappings", () => {- expectStructuredPublicApiError(- () =>- toEvaluationRuleInput({- input: {- name: "answer_quality",- target: "observation",- enabled: true,- sampling: 1,- filter: [],- mapping: [{ variable: "input", source: "input" }],- },- evaluatorVariables: ["input", "output"],- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- }),- {- code: "missing_variable_mapping",- details: {- field: "mapping",- variables: ["output"],- },- },- );- });-- it("inherits canonical evaluator mappings for code evaluation rule writes", () => {- const writeModel = toEvaluationRuleInput({- input: {- name: "toxicity-code-live",- target: "observation",- enabled: true,- sampling: 1,- filter: [],- },- evaluatorVariables: [],- evaluatorType: PUBLIC_EVALUATOR_TYPE_CODE,- });-- expect(writeModel.variableMapping).toBeNull();- });-- it("maps evaluator records to exact template versions", () => {- const template: StoredPublicEvaluatorTemplate = {- id: "tmpl_latest",- projectId: "project_123",- name: "answer-correctness",- version: 7,- type: EvalTemplateType.LLM_AS_JUDGE,- prompt: "Judge {{input}} against {{output}}",- partner: "ragas",- provider: "openai",- model: "gpt-4.1-mini",- modelParams: { temperature: 0 },- vars: ["input", "output"],- variableMapping: null,- outputDefinition: numericOutputDefinition,- sourceCode: null,- sourceCodeLanguage: null,- createdAt: new Date("2026-03-30T08:00:00.000Z"),- updatedAt: new Date("2026-03-30T09:00:00.000Z"),- };-- expect(- toApiEvaluator({- template,- evaluationRuleCount: 2,- }),- ).toMatchObject({- id: "tmpl_latest",- name: "answer-correctness",- version: 7,- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- outputDefinition: {- dataType: "NUMERIC",- reasoning: {- description: "Why the score was assigned",- },- score: {- description: "A score between 0 and 1",- minValue: 0,- maxValue: 1,- },- },- modelConfig: {- provider: "openai",- model: "gpt-4.1-mini",- },- variables: ["input", "output"],- evaluationRuleCount: 2,- });- });-- it("maps code evaluator records with canonical runtime variables", () => {- const template: StoredPublicEvaluatorTemplate = {- id: "tmpl_code",- projectId: "project_123",- name: "toxicity-detector",- version: 1,- type: EvalTemplateType.CODE,- prompt: null,- partner: null,- provider: null,- model: null,- modelParams: null,- vars: ["stale"],- variableMapping: null,- outputDefinition: null,- sourceCode:- 'function evaluate() { return { scores: [{ name: "toxicity", value: false, dataType: "BOOLEAN" }] }; }',- sourceCodeLanguage: "TYPESCRIPT",- createdAt: new Date("2026-03-30T08:00:00.000Z"),- updatedAt: new Date("2026-03-30T09:00:00.000Z"),- };-- expect(- toApiEvaluator({- template,- evaluationRuleCount: 0,- }),- ).toMatchObject({- id: "tmpl_code",- type: PUBLIC_EVALUATOR_TYPE_CODE,- variables: [...CODE_EVAL_TEMPLATE_VARIABLES],- sourceCodeLanguage: "TYPESCRIPT",- });- });-- it("normalizes legacy evaluator output definitions into the public response shape", () => {- const template: StoredPublicEvaluatorTemplate = {- id: "tmpl_legacy",- projectId: "project_123",- name: "answer-correctness",- version: 1,- type: EvalTemplateType.LLM_AS_JUDGE,- prompt: "Judge {{input}} against {{output}}",- partner: null,- provider: null,- model: null,- modelParams: null,- vars: ["input", "output"],- variableMapping: null,- outputDefinition: {- reasoning: "Explain why the answer is correct or incorrect.",- score: "Return a score between 0 and 1.",- },- sourceCode: null,- sourceCodeLanguage: null,- createdAt: new Date("2026-03-30T08:00:00.000Z"),- updatedAt: new Date("2026-03-30T09:00:00.000Z"),- };-- expect(- toApiEvaluator({- template,- evaluationRuleCount: 0,- }),- ).toMatchObject({- outputDefinition: {- dataType: "NUMERIC",- reasoning: {- description: "Explain why the answer is correct or incorrect.",- },- score: {- description: "Return a score between 0 and 1.",- },- },- });- });-});diff --git a/web/src/__tests__/server/unit/unstable-evals-queries.servertest.ts b/web/src/__tests__/server/unit/unstable-evals-queries.servertest.tsdeleted file mode 100644index 1fde058e51..0000000000--- a/web/src/__tests__/server/unit/unstable-evals-queries.servertest.ts+++ /dev/null@@ -1,130 +0,0 @@-import type * as PrismaClientModule from "@prisma/client";-import type { Mock } from "vitest";--vi.mock("@langfuse/shared/src/db", async () => {- const { EvalTemplateType, JobConfigState } =- await vi.importActual<typeof PrismaClientModule>("@prisma/client");- return {- EvalTemplateType,- JobConfigState,- prisma: {- evaluationRule: {- count: vi.fn(),- findFirst: vi.fn(),- findMany: vi.fn(),- },- evaluator: { findFirst: vi.fn() },- },- };-});--import { prisma } from "@langfuse/shared/src/db";-import { countActiveEvaluationRules } from "@/src/features/evals/v2/server/rules/ruleErrors";-import {- findPublicV2EvaluatorById,- findPublicV2EvaluatorInFamily,- findPublicV2EvaluationRule,- listPublicEvaluationRulePage,-} from "@/src/features/evals/server/unstable-public-api/queries";--describe("unstable public eval queries", () => {- beforeEach(() => vi.clearAllMocks());-- it("lists rules exclusively from evaluation_rules", async () => {- (prisma.evaluationRule.findMany as Mock).mockResolvedValue([- { id: "rule" },- ]);- (prisma.evaluationRule.count as Mock).mockResolvedValue(1);-- await expect(- listPublicEvaluationRulePage({- projectId: "project",- page: 1,- limit: 20,- }),- ).resolves.toEqual({ records: [{ id: "rule" }], totalItems: 1 });-- expect(prisma.evaluationRule.findMany).toHaveBeenCalledWith(- expect.objectContaining({- where: expect.objectContaining({- projectId: "project",- targetObject: {- in: ["event", "experiment", "trace", "dataset"],- },- }),- }),- );- });-- it("counts active rules exclusively from evaluation_rules", async () => {- (prisma.evaluationRule.count as Mock).mockResolvedValue(4);-- await expect(- countActiveEvaluationRules({ prisma, projectId: "project" }),- ).resolves.toBe(4);- expect(prisma.evaluationRule.count).toHaveBeenCalledWith({- where: expect.objectContaining({- targetObject: { in: ["event", "experiment"] },- }),- });- });-- it("loads rules and assignments within the project", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue({ id: "rule" });-- await expect(- findPublicV2EvaluationRule({- projectId: "project",- evaluationRuleId: "rule",- }),- ).resolves.toEqual({ id: "rule" });- expect(prisma.evaluationRule.findFirst).toHaveBeenCalledWith(- expect.objectContaining({ where: { id: "rule", projectId: "project" } }),- );- });-- it("resolves project evaluators from the evaluators table", async () => {- (prisma.evaluator.findFirst as Mock).mockResolvedValue({- id: "evaluator",- versions: [{ version: 2 }],- });-- await expect(- findPublicV2EvaluatorInFamily({- projectId: "project",- evaluator: {- name: "Quality",- type: "llm_as_judge",- },- }),- ).resolves.toMatchObject({ id: "evaluator" });- await expect(- findPublicV2EvaluatorById({- projectId: "project",- evaluatorId: "evaluator",- }),- ).resolves.toMatchObject({ id: "evaluator" });- });-- it("resolves a family within the project by name and type", async () => {- (prisma.evaluator.findFirst as Mock).mockResolvedValue(null);-- await expect(- findPublicV2EvaluatorInFamily({- projectId: "project",- evaluator: {- name: "Missing",- type: "llm_as_judge",- },- }),- ).resolves.toBeNull();- expect(prisma.evaluator.findFirst).toHaveBeenCalledWith(- expect.objectContaining({- where: expect.objectContaining({- projectId: "project",- name: "Missing",- }),- }),- );- });-});diff --git a/web/src/__tests__/server/unit/unstable-evals-service.servertest.ts b/web/src/__tests__/server/unit/unstable-evals-service.servertest.tsdeleted file mode 100644index f1918047a1..0000000000--- a/web/src/__tests__/server/unit/unstable-evals-service.servertest.ts+++ /dev/null@@ -1,584 +0,0 @@-import type * as PrismaClientModule from "@prisma/client";-import type { Mock } from "vitest";--const { evaluator, codeEvaluator } = vi.hoisted(() => ({- codeEvaluator: {- id: "code-evaluator",- projectId: "project",- name: "Toxicity",- type: "CODE" as const,- description: null,- createdByUserId: null,- blockedAt: null,- blockReason: null,- blockMessage: null,- createdAt: new Date("2026-01-01"),- updatedAt: new Date("2026-01-01"),- versions: [- {- id: "code-version",- evaluatorId: "code-evaluator",- version: 1,- createdAt: new Date("2026-01-01"),- createdByUserId: null,- prompt: null,- partner: null,- provider: null,- model: null,- modelParams: null,- vars: [],- variableMapping: null,- outputDefinition: null,- sourceCode: "def evaluate(ctx): return []",- sourceCodeLanguage: "PYTHON" as const,- },- ],- },- evaluator: {- id: "evaluator",- projectId: "project",- name: "Quality",- type: "LLM_AS_JUDGE" as const,- description: null,- createdByUserId: null,- blockedAt: null,- blockReason: null,- blockMessage: null,- createdAt: new Date("2026-01-01"),- updatedAt: new Date("2026-01-01"),- versions: [- {- id: "version",- evaluatorId: "evaluator",- version: 1,- createdAt: new Date("2026-01-01"),- createdByUserId: null,- prompt: "Judge {{output}}",- partner: null,- provider: "openai",- model: "gpt-4o-mini",- modelParams: {},- vars: ["output"],- variableMapping: [- {- templateVariable: "output",- selectedColumnId: "output",- jsonSelector: null,- },- ],- outputDefinition: { score: { type: "NUMERIC", min: 0, max: 1 } },- sourceCode: null,- sourceCodeLanguage: null,- },- ],- },-}));--vi.mock("@langfuse/shared/src/db", async () => {- const actual =- await vi.importActual<typeof PrismaClientModule>("@prisma/client");- const prisma = {- evaluator: { count: vi.fn(), findMany: vi.fn() },- evaluationRule: {- count: vi.fn(),- findFirst: vi.fn(),- create: vi.fn(),- delete: vi.fn(),- deleteMany: vi.fn(),- update: vi.fn(),- },- jobExecution: { deleteMany: vi.fn() },- $transaction: vi.fn(),- evaluationRuleEvaluatorAssignment: { update: vi.fn(), create: vi.fn() },- };- prisma.$transaction.mockImplementation(- (callback: (client: typeof prisma) => unknown) => callback(prisma),- );- return {- EvalTemplateType: actual.EvalTemplateType,- JobConfigState: actual.JobConfigState,- Prisma: { DbNull: { dbNull: true } },- prisma,- };-});--vi.mock("@/src/features/evals/server/unstable-public-api/queries", () => ({- findPublicV2EvaluatorByIdOrThrow: vi.fn().mockResolvedValue(evaluator),- findPublicV2EvaluatorInFamilyOrThrow: vi.fn().mockResolvedValue(evaluator),- findPublicV2EvaluationRule: vi.fn(),- listPublicEvaluationRulePage: vi.fn(),-}));--vi.mock(- "@/src/features/evals/server/unstable-public-api/validation",- async () => ({- ...(await vi.importActual(- "@/src/features/evals/server/unstable-public-api/validation",- )),- assertEvaluationRuleFilterValuesExistForProject: vi.fn(),- assertEvaluatorDefinitionCanRunForPublicApi: vi.fn(),- }),-);--vi.mock("@/src/features/evals/server/codeEvalJobConfigValidation", () => ({- CodeEvalJobConfigError: class CodeEvalJobConfigError extends Error {- constructor(- message: string,- public readonly code = "preflight_failed",- ) {- super(message);- }- },- assertCodeEvalRuleCanRun: vi.fn(),-}));--vi.mock("@/src/features/evals/server/isCodeEvalEnabled", () => ({- isCodeEvalEnabled: vi.fn().mockReturnValue(true),- isCodeEvalSourceCodeLanguageSupported: vi.fn().mockReturnValue(true),-}));--vi.mock("@langfuse/shared/src/server", async () => ({- ...(await vi.importActual("@langfuse/shared/src/server")),- invalidateProjectEvalConfigCaches: vi.fn(),-}));--import { prisma } from "@langfuse/shared/src/db";-import {- CodeEvalJobConfigError,- assertCodeEvalRuleCanRun,-} from "@/src/features/evals/server/codeEvalJobConfigValidation";-import {- createPublicEvaluationRule,- deletePublicEvaluationRule,- listPublicEvaluationRules,- updatePublicEvaluationRule,-} from "@/src/features/evals/server/unstable-public-api/evaluation-rule-service";-import {- findPublicV2EvaluationRule,- findPublicV2EvaluatorInFamilyOrThrow,- listPublicEvaluationRulePage,-} from "@/src/features/evals/server/unstable-public-api/queries";-import { MAX_ACTIVE_EVALUATION_RULES } from "@/src/features/evals/v2/server/rules/ruleErrors";--function buildCreatedRule(data: {- projectId: string;- name: string;- status: string;- targetObject: string;- filter: unknown;- sampling: number;-}) {- return {- id: "rule",- createdAt: new Date("2026-01-01"),- updatedAt: new Date("2026-01-01"),- projectId: data.projectId,- createdByUserId: null,- name: data.name,- status: data.status,- targetObject: data.targetObject,- filter: data.filter,- sampling: {- toNumber: () => data.sampling,- valueOf: () => data.sampling,- },- delay: 0,- timeScope: ["NEW"],- assignments: [- {- id: "assignment",- createdAt: new Date("2026-01-01"),- updatedAt: new Date("2026-01-01"),- projectId: "project",- evaluationRuleId: "rule",- evaluatorId: evaluator.id,- variableMapping: null,- evaluator,- },- ],- };-}--function buildMigratedRuleWithLegacyFilter() {- const migratedRule = buildCreatedRule({- projectId: "project",- name: "Migrated rule",- status: "ACTIVE",- targetObject: "event",- filter: [- {- type: "string",- column: "environment",- operator: "does not contain",- value: "langfuse-",- },- {- type: "null",- column: "experimentId",- operator: "is null",- value: "",- },- ],- sampling: 1,- });- return migratedRule;-}--describe("unstable public evaluation-rule service", () => {- beforeEach(() => {- vi.clearAllMocks();- (prisma.evaluator.count as Mock).mockResolvedValue(1);- (prisma.evaluator.findMany as Mock).mockResolvedValue([evaluator]);- (prisma.evaluationRule.count as Mock).mockResolvedValue(0);- (findPublicV2EvaluatorInFamilyOrThrow as Mock).mockResolvedValue(evaluator);- (findPublicV2EvaluationRule as Mock).mockImplementation(- () => (prisma.evaluationRule.create as Mock).mock.results.at(-1)?.value,- );- });-- it("creates rules and assignments only in the new tables", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);- (prisma.evaluationRule.create as Mock).mockImplementation(({ data }) =>- buildCreatedRule(data),- );-- await expect(- createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "Quality rule",- evaluator: {- name: "Quality",- type: "llm_as_judge",- },- target: "observation",- enabled: false,- sampling: 1,- filter: [],- mapping: [{ variable: "output", source: "output" }],- },- }),- ).resolves.toMatchObject({ id: "rule", target: "observation" });- expect(prisma.evaluationRule.create).toHaveBeenCalledOnce();- });-- it("resolves the evaluator family by name and type within the project", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);- (prisma.evaluationRule.create as Mock).mockImplementation(({ data }) =>- buildCreatedRule(data),- );-- await createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "Quality rule",- evaluator: { name: "Quality", type: "llm_as_judge" },- target: "observation",- enabled: false,- sampling: 1,- filter: [],- mapping: [{ variable: "output", source: "output" }],- },- });-- expect(findPublicV2EvaluatorInFamilyOrThrow).toHaveBeenCalledWith({- projectId: "project",- evaluator: { name: "Quality", type: "llm_as_judge" },- });- });-- it("rejects creating more active evaluation rules than the shared cap", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);- (prisma.evaluationRule.count as Mock).mockResolvedValue(- MAX_ACTIVE_EVALUATION_RULES,- );-- await expect(- createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "One too many",- evaluator: {- name: "Quality",- type: "llm_as_judge",- },- target: "observation",- enabled: true,- sampling: 1,- filter: [],- mapping: [{ variable: "output", source: "output" }],- },- }),- ).rejects.toMatchObject({- httpCode: 409,- code: "conflict",- details: { limit: MAX_ACTIVE_EVALUATION_RULES },- });- expect(prisma.evaluationRule.create).not.toHaveBeenCalled();- });-- it("does not check the active limit for disabled evaluation rules", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);- (prisma.evaluationRule.create as Mock).mockImplementation(({ data }) =>- buildCreatedRule(data),- );-- await createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "Disabled rule",- evaluator: { name: "Quality", type: "llm_as_judge" },- target: "observation",- enabled: false,- sampling: 1,- filter: [],- mapping: [{ variable: "output", source: "output" }],- },- });-- expect(prisma.evaluationRule.count).not.toHaveBeenCalled();- });-- it("returns a name conflict when a rule with that name already exists", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue({- id: "existing-rule",- });-- await expect(- createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "Quality rule",- evaluator: {- name: "Quality",- type: "llm_as_judge",- },- target: "observation",- enabled: false,- sampling: 1,- filter: [],- mapping: [{ variable: "output", source: "output" }],- },- }),- ).rejects.toMatchObject({- httpCode: 409,- code: "name_conflict",- details: { field: "name" },- });- expect(prisma.evaluationRule.create).not.toHaveBeenCalled();- });-- it("translates code-eval preflight failures into structured public API errors", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);- (findPublicV2EvaluatorInFamilyOrThrow as Mock).mockResolvedValue(- codeEvaluator,- );- (assertCodeEvalRuleCanRun as Mock).mockRejectedValue(- new CodeEvalJobConfigError("Sandbox run failed", "preflight_failed"),- );-- await expect(- createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "Code rule",- evaluator: { name: "Toxicity", type: "code" },- target: "observation",- enabled: true,- sampling: 1,- filter: [],- },- }),- ).rejects.toMatchObject({- httpCode: 422,- code: "evaluator_preflight_failed",- message: "Sandbox run failed",- details: { evaluatorName: "Toxicity" },- });- expect(prisma.evaluationRule.create).not.toHaveBeenCalled();- });-- it("translates code-eval invalid-target failures into a 400", async () => {- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);- (findPublicV2EvaluatorInFamilyOrThrow as Mock).mockResolvedValue(- codeEvaluator,- );- (assertCodeEvalRuleCanRun as Mock).mockRejectedValue(- new CodeEvalJobConfigError("Unsupported target", "invalid_target"),- );-- await expect(- createPublicEvaluationRule({- orgId: "org",- projectId: "project",- input: {- name: "Code rule",- evaluator: { name: "Toxicity", type: "code" },- target: "observation",- enabled: true,- sampling: 1,- filter: [],- },- }),- ).rejects.toMatchObject({ httpCode: 400, code: "invalid_request" });- });-- it("lists migrated rules with filters that do not match their current target", async () => {- const migratedRule = buildMigratedRuleWithLegacyFilter();- (listPublicEvaluationRulePage as Mock).mockResolvedValue({- records: [migratedRule],- totalItems: 1,- });-- await expect(- listPublicEvaluationRules({ projectId: "project", page: 1, limit: 50 }),- ).resolves.toMatchObject({- data: [- {- id: "rule",- filter: [- {- type: "string",- column: "environment",- operator: "does not contain",- value: "langfuse-",- },- {- type: "null",- column: "experimentId",- operator: "is null",- value: "",- },- ],- },- ],- meta: { totalItems: 1 },- });- });-- it("preserves migrated filters on patches that do not replace them", async () => {- const migratedRule = buildMigratedRuleWithLegacyFilter();- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.update as Mock).mockImplementation(({ data }) => {- migratedRule.name = data.name ?? migratedRule.name;- return migratedRule;- });-- await expect(- updatePublicEvaluationRule({- orgId: "org",- projectId: "project",- evaluationRuleId: "rule",- input: { name: "Renamed rule", target: "observation" },- }),- ).resolves.toMatchObject({- name: "Renamed rule",- filter: [{ column: "environment" }, { column: "experimentId" }],- });- expect(prisma.evaluationRule.update).toHaveBeenCalledOnce();- });-- it("repairs migrated rules with a V2-compatible replacement filter", async () => {- const migratedRule = buildMigratedRuleWithLegacyFilter();- const replacementFilter = [- {- type: "boolean" as const,- column: "isRootObservation",- operator: "=" as const,- value: true,- },- ];- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.update as Mock).mockImplementation(({ data }) => {- migratedRule.filter = data.filter;- migratedRule.status = data.status;- return migratedRule;- });-- await expect(- updatePublicEvaluationRule({- orgId: "org",- projectId: "project",- evaluationRuleId: "rule",- input: {- target: "observation",- filter: replacementFilter,- enabled: false,- },- }),- ).resolves.toMatchObject({ filter: replacementFilter, enabled: false });- expect(prisma.evaluationRule.update).toHaveBeenCalledOnce();- });-- it("requires a replacement filter when changing the target of a migrated rule", async () => {- const migratedRule = buildMigratedRuleWithLegacyFilter();- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);-- await expect(- updatePublicEvaluationRule({- orgId: "org",- projectId: "project",- evaluationRuleId: "rule",- input: { target: "experiment" },- }),- ).rejects.toMatchObject({- httpCode: 400,- code: "invalid_body",- details: { field: "filter" },- });- expect(prisma.evaluationRule.update).not.toHaveBeenCalled();- });-- it.each([- {- operation: "disabled",- mutate: () =>- updatePublicEvaluationRule({- orgId: "org",- projectId: "project",- evaluationRuleId: "rule",- input: { enabled: false },- }),- },- {- operation: "deleted",- mutate: () =>- deletePublicEvaluationRule({- projectId: "project",- evaluationRuleId: "rule",- }),- },- ])(- "allows a migrated rule with an unsupported stored filter to be $operation",- async ({ mutate, operation }) => {- const migratedRule = buildMigratedRuleWithLegacyFilter();- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);- (prisma.evaluationRule.update as Mock).mockImplementation(({ data }) => {- migratedRule.status = data.status ?? migratedRule.status;- return migratedRule;- });- (prisma.evaluationRule.deleteMany as Mock).mockResolvedValue({- count: 1,- });- (prisma.jobExecution.deleteMany as Mock).mockResolvedValue({ count: 0 });-- const result = await mutate();- if (operation === "disabled") {- expect(result).toMatchObject({ enabled: false, status: "inactive" });- expect(prisma.evaluationRule.update).toHaveBeenCalledOnce();- } else {- expect(prisma.evaluationRule.deleteMany).toHaveBeenCalledOnce();- }- },- );-});diff --git a/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts b/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.tsindex fa3f361771..35cbf747e0 100644--- a/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts+++ b/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts@@ -8,7 +8,7 @@ import {PatchUnstableDashboardWidgetResponse,PostUnstableDashboardWidgetResponse,} from "@/src/features/public-api/types/unstable-dashboard-widgets";-import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/unstable-public-evals-contract";+import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/structuredPublicApiErrorSchema";import { DashboardWidgetViews, prisma } from "@langfuse/shared/src/db";import {createOrgProjectAndApiKey,diff --git a/web/src/__tests__/server/unstable-dashboards-api.servertest.ts b/web/src/__tests__/server/unstable-dashboards-api.servertest.tsindex e9459c8694..a749d0b61b 100644--- a/web/src/__tests__/server/unstable-dashboards-api.servertest.ts+++ b/web/src/__tests__/server/unstable-dashboards-api.servertest.ts@@ -19,7 +19,7 @@ import {PostDashboardPlacementResponse,PostUnstableDashboardResponse,} from "@/src/features/public-api/types/unstable-dashboards";-import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/unstable-public-evals-contract";+import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/structuredPublicApiErrorSchema";import { createOrgProjectAndApiKey } from "@langfuse/shared/src/server";import { prisma } from "@langfuse/shared/src/db";import { nanoid } from "nanoid";diff --git a/web/src/__tests__/server/unstable-evals-api.servertest.ts b/web/src/__tests__/server/unstable-evals-api.servertest.tsdeleted file mode 100644index 1b1c2d720a..0000000000--- a/web/src/__tests__/server/unstable-evals-api.servertest.ts+++ /dev/null@@ -1,1123 +0,0 @@-import {- makeAPICall,- makeZodVerifiedAPICall,-} from "@/src/__tests__/test-utils";-import {- DeleteUnstableEvaluationRuleResponse,- GetUnstableEvaluationRuleResponse,- GetUnstableEvaluationRulesResponse,- PostUnstableEvaluationRuleResponse,- PatchUnstableEvaluationRuleResponse,-} from "@/src/features/public-api/types/unstable-evaluation-rules";-import {- DeleteUnstableEvaluatorResponse,- GetUnstableEvaluatorResponse,- GetUnstableEvaluatorsResponse,- PostUnstableEvaluatorResponse,-} from "@/src/features/public-api/types/unstable-evaluators";-import {- createNumericEvalOutputDefinition,- LLMAdapter,-} from "@langfuse/shared";-import { prisma } from "@langfuse/shared/src/db";-import {- createAndAddApiKeysToDb,- createBasicAuthHeader,- createOrgProjectAndApiKey,-} from "@langfuse/shared/src/server";-import { ApiKeyScope } from "@prisma/client";-import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/unstable-public-evals-contract";-import type { z } from "zod";-import { encrypt } from "@langfuse/shared/encryption";--const __orgIds: string[] = [];-const __managedTemplateIds: string[] = [];--const numericOutputDefinition = createNumericEvalOutputDefinition({- reasoningDescription: "Why the score was assigned",- scoreDescription: "A score between 0 and 1",-});--const provisionDefaultEvalModel = async (projectId: string) => {- const provider = `openai-${projectId}`;-- const llmApiKey = await prisma.llmApiKeys.create({- data: {- projectId,- provider,- adapter: LLMAdapter.OpenAI,- secretKey: encrypt("sk-test"),- displaySecretKey: "...test",- baseURL: "https://api.openai.com/v1",- customModels: [],- withDefaultModels: true,- extraHeaders: null,- extraHeaderKeys: [],- },- });-- await prisma.defaultLlmModel.create({- data: {- projectId,- llmApiKeyId: llmApiKey.id,- provider,- adapter: LLMAdapter.OpenAI,- model: "gpt-4.1-mini",- },- });-};--const expectUnstableError = (- response: Awaited<ReturnType<typeof makeAPICall>>,- params: {- status: number;- code: z.infer<typeof StructuredPublicApiErrorResponse>["code"];- },-) => {- expect(response.status).toBe(params.status);- const body = StructuredPublicApiErrorResponse.parse(response.body);- expect(body.code).toBe(params.code);- return body;-};--const createManagedEvaluator = async (params: {- name: string;- version: number;- prompt?: string;-}) => {- const template = await prisma.evalTemplate.create({- data: {- projectId: null,- name: params.name,- version: params.version,- prompt: params.prompt ?? "Judge {{input}} against {{output}}",- partner: "ragas",- vars: ["input", "output"],- outputDefinition: numericOutputDefinition,- },- });-- __managedTemplateIds.push(template.id);- return template;-};--describe("/api/public/unstable evaluators API", () => {- let auth: string;- let projectId: string;-- beforeEach(async () => {- const result = await createOrgProjectAndApiKey();- auth = result.auth;- projectId = result.projectId;- __orgIds.push(result.orgId);- await provisionDefaultEvalModel(result.projectId);- });-- afterAll(async () => {- await prisma.jobConfiguration.deleteMany({- where: {- evalTemplateId: {- in: __managedTemplateIds,- },- },- });- await prisma.evalTemplate.deleteMany({- where: {- id: {- in: __managedTemplateIds,- },- },- });- await prisma.organization.deleteMany({- where: {- id: {- in: __orgIds,- },- },- });- });-- it("keeps one evaluator id across versions and returns only the latest version", async () => {- const v1 = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Answer correctness",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- const v2 = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Answer correctness",- prompt: "Judge {{input}} versus {{output}}",- outputDefinition: numericOutputDefinition,- mapping: [- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ],- },- auth,- );-- expect(v1.body).toMatchObject({- id: expect.any(String),- name: "Answer correctness",- version: 1,- });- expect(v2.body).toMatchObject({- id: expect.any(String),- name: "Answer correctness",- version: 2,- });- expect(v2.body.id).toBe(v1.body.id);-- const fetched = await makeZodVerifiedAPICall(- GetUnstableEvaluatorResponse,- "GET",- `/api/public/unstable/evaluators/${v1.body.id}`,- undefined,- auth,- );- expect(fetched.body).toMatchObject({- id: v1.body.id,- version: 2,- mapping: [- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ],- });-- const listed = await makeZodVerifiedAPICall(- GetUnstableEvaluatorsResponse,- "GET",- "/api/public/unstable/evaluators?page=1&limit=50",- undefined,- auth,- );-- const projectFamilyEntries = listed.body.data.filter(- (evaluator) => evaluator.name === "Answer correctness",- );-- expect(projectFamilyEntries).toEqual([- expect.objectContaining({- id: v2.body.id,- name: "Answer correctness",- version: 2,- }),- ]);- expect(listed.body.data).toContainEqual(- expect.objectContaining({ id: v1.body.id, version: 2 }),- );- });-- it("deletes an evaluator including all of its versions by evaluator id", async () => {- const v1 = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Deletable correctness",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- const v2 = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Deletable correctness",- prompt: "Judge {{input}} versus {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- expect(v2.body.id).toBe(v1.body.id);- const deleted = await makeZodVerifiedAPICall(- DeleteUnstableEvaluatorResponse,- "DELETE",- `/api/public/unstable/evaluators/${v1.body.id}`,- undefined,- auth,- );- expect(deleted.body.message).toBe("Evaluator successfully deleted");-- const fetchLatest = await makeAPICall(- "GET",- `/api/public/unstable/evaluators/${v2.body.id}`,- undefined,- auth,- );- expectUnstableError(fetchLatest, {- status: 404,- code: "resource_not_found",- });- });-- it("returns 404 when deleting an unknown evaluator", async () => {- const response = await makeAPICall(- "DELETE",- "/api/public/unstable/evaluators/unknown-evaluator-id",- undefined,- auth,- );- expectUnstableError(response, {- status: 404,- code: "resource_not_found",- });- });-- it("does not resolve evaluators from legacy managed templates", async () => {- await createManagedEvaluator({- name: "Answer relevance",- version: 3,- });-- const response = await makeAPICall(- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "answer_relevance_managed",- evaluator: {- name: "Answer relevance",- },- target: "observation",- enabled: false,- sampling: 1,- filter: [],- mapping: [- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ],- },- auth,- );- expectUnstableError(response, {- status: 404,- code: "resource_not_found",- });- });-- it("rejects unsupported filters without creating a rule", async () => {- const response = await makeAPICall(- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "unsupported_filter_rule",- evaluator: { name: "unused", type: "llm_as_judge" },- target: "observation",- enabled: false,- sampling: 1,- filter: [- {- column: "totalCost",- type: "number",- operator: ">",- value: 0.01,- },- ],- },- auth,- );-- expectUnstableError(response, { status: 400, code: "invalid_body" });- await expect(- prisma.evaluationRule.count({ where: { projectId } }),- ).resolves.toBe(0);- });-- it("uses public project evaluators in observation rules without changing the API contract", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Public project evaluator",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- const created = await makeZodVerifiedAPICall(- PostUnstableEvaluationRuleResponse,- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "public_project_rule",- evaluator: {- name: evaluator.body.name,- type: "llm_as_judge",- },- target: "observation",- enabled: false,- sampling: 0.5,- filter: [],- mapping: [- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ],- },- auth,- );- expect(created.body).toMatchObject({- name: "public_project_rule",- evaluator: { id: evaluator.body.id },- target: "observation",- enabled: false,- sampling: 0.5,- });-- await expect(- prisma.evaluationRule.findUniqueOrThrow({- where: { id: created.body.id },- include: { assignments: true },- }),- ).resolves.toMatchObject({- assignments: [{ evaluatorId: evaluator.body.id }],- });-- const fetched = await makeZodVerifiedAPICall(- GetUnstableEvaluationRuleResponse,- "GET",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- undefined,- auth,- );- expect(fetched.body).toEqual(created.body);-- const listed = await makeZodVerifiedAPICall(- GetUnstableEvaluationRulesResponse,- "GET",- "/api/public/unstable/evaluation-rules?page=1&limit=50",- undefined,- auth,- );- expect(listed.body.data).toContainEqual(created.body);-- const updated = await makeZodVerifiedAPICall(- PatchUnstableEvaluationRuleResponse,- "PATCH",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- { name: "public_project_rule_updated", enabled: false },- auth,- );- expect(updated.body).toMatchObject({- id: created.body.id,- name: "public_project_rule_updated",- enabled: false,- status: "inactive",- evaluator: { id: evaluator.body.id },- });-- const deleted = await makeZodVerifiedAPICall(- DeleteUnstableEvaluationRuleResponse,- "DELETE",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- undefined,- auth,- );- expect(deleted.body.message).toBe("Evaluation rule successfully deleted");- await expect(- prisma.evaluationRule.findUnique({ where: { id: created.body.id } }),- ).resolves.toBeNull();- });-- it("returns incomplete inherited legacy mappings from create and read endpoints", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Legacy mapping evaluator",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );- await prisma.evaluatorVersion.updateMany({- where: { evaluatorId: evaluator.body.id },- data: {- variableMapping: [- {- templateVariable: "input",- langfuseObject: "trace",- objectName: null,- selectedColumnId: "input",- jsonSelector: null,- },- {- templateVariable: "output",- langfuseObject: "trace",- objectName: null,- selectedColumnId: "output",- jsonSelector: null,- },- ],- },- });-- const created = await makeZodVerifiedAPICall(- PostUnstableEvaluationRuleResponse,- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "incomplete_inherited_mapping_rule",- evaluator: { name: evaluator.body.name },- target: "observation",- enabled: false,- sampling: 1,- filter: [],- },- auth,- );- const incompleteMapping = [- { variable: "input", source: null },- { variable: "output", source: null },- ];- expect(created.body).toMatchObject({- mapping: incompleteMapping,- evaluators: [{ mapping: incompleteMapping }],- });-- const fetched = await makeZodVerifiedAPICall(- GetUnstableEvaluationRuleResponse,- "GET",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- undefined,- auth,- );- expect(fetched.body).toEqual(created.body);-- const listed = await makeZodVerifiedAPICall(- GetUnstableEvaluationRulesResponse,- "GET",- "/api/public/unstable/evaluation-rules?page=1&limit=50",- undefined,- auth,- );- expect(listed.body.data).toContainEqual(created.body);- });-- it("creates and returns multiple evaluator assignments with compatibility aliases", async () => {- const createEvaluator = (name: string) =>- makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name,- prompt: "Judge {{input}}",- outputDefinition: numericOutputDefinition,- mapping: [{ variable: "input", source: "input" }],- },- auth,- );- const [correctness, tone] = await Promise.all([- createEvaluator("Multi-rule correctness"),- createEvaluator("Multi-rule tone"),- ]);-- const created = await makeZodVerifiedAPICall(- PostUnstableEvaluationRuleResponse,- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "multi_evaluator_rule",- evaluators: [- {- evaluator: {- name: correctness.body.name,- type: "llm_as_judge",- },- },- {- evaluator: {- name: tone.body.name,- type: "llm_as_judge",- },- mapping: [{ variable: "input", source: "output" }],- },- ],- target: "observation",- enabled: false,- filter: [],- },- auth,- );-- expect(created.body).toMatchObject({- evaluator: { id: correctness.body.id },- mapping: [{ variable: "input", source: "input" }],- evaluators: [- { evaluator: { id: correctness.body.id }, mapping: null },- {- evaluator: { id: tone.body.id },- mapping: [{ variable: "input", source: "output" }],- },- ],- });-- const updated = await makeZodVerifiedAPICall(- PatchUnstableEvaluationRuleResponse,- "PATCH",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- { name: "multi_evaluator_rule_updated" },- auth,- );- expect(updated.body.evaluators).toHaveLength(2);- await expect(- prisma.evaluationRuleEvaluatorAssignment.count({- where: { evaluationRuleId: created.body.id },- }),- ).resolves.toBe(2);- });-- it("replaces the whole assignment set through evaluators on patch", async () => {- const createEvaluator = (name: string) =>- makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name,- prompt: "Judge {{input}}",- outputDefinition: numericOutputDefinition,- mapping: [{ variable: "input", source: "input" }],- },- auth,- );- const [first, second, third] = await Promise.all([- createEvaluator("Patch set first"),- createEvaluator("Patch set second"),- createEvaluator("Patch set third"),- ]);-- const created = await makeZodVerifiedAPICall(- PostUnstableEvaluationRuleResponse,- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "patch_evaluator_set_rule",- evaluators: [- { evaluator: { name: first.body.name, type: "llm_as_judge" } },- { evaluator: { name: second.body.name, type: "llm_as_judge" } },- ],- target: "observation",- enabled: false,- filter: [],- },- auth,- );-- // Drops `second`, keeps `first`, attaches `third` with an override.- const updated = await makeZodVerifiedAPICall(- PatchUnstableEvaluationRuleResponse,- "PATCH",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- {- evaluators: [- { evaluator: { name: first.body.name, type: "llm_as_judge" } },- {- evaluator: { name: third.body.name, type: "llm_as_judge" },- mapping: [{ variable: "input", source: "output" }],- },- ],- },- auth,- );-- expect(updated.body.evaluators).toEqual([- expect.objectContaining({- evaluator: expect.objectContaining({ id: first.body.id }),- mapping: null,- }),- expect.objectContaining({- evaluator: expect.objectContaining({ id: third.body.id }),- mapping: [{ variable: "input", source: "output" }],- }),- ]);- await expect(- prisma.evaluationRuleEvaluatorAssignment.count({- where: { evaluationRuleId: created.body.id },- }),- ).resolves.toBe(2);- });-- it("keeps a rule readable and deletable after its last evaluator is detached", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Detached rule evaluator",- prompt: "Judge {{input}}",- outputDefinition: numericOutputDefinition,- mapping: [{ variable: "input", source: "input" }],- },- auth,- );- const created = await makeZodVerifiedAPICall(- PostUnstableEvaluationRuleResponse,- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "detached_assignments_rule",- evaluator: { name: evaluator.body.name, type: "llm_as_judge" },- target: "observation",- enabled: false,- filter: [],- },- auth,- );-- // Mirrors detaching the last evaluator from the app or MCP tools.- await prisma.evaluationRuleEvaluatorAssignment.deleteMany({- where: { evaluationRuleId: created.body.id },- });-- const fetched = await makeZodVerifiedAPICall(- GetUnstableEvaluationRuleResponse,- "GET",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- undefined,- auth,- );- expect(fetched.body).toMatchObject({- id: created.body.id,- evaluator: null,- evaluators: [],- mapping: [],- });-- const listed = await makeZodVerifiedAPICall(- GetUnstableEvaluationRulesResponse,- "GET",- "/api/public/unstable/evaluation-rules?page=1&limit=50",- undefined,- auth,- );- expect(listed.body.data.map((rule) => rule.id)).toContain(created.body.id);-- const deleted = await makeAPICall(- "DELETE",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- undefined,- auth,- );- expect(deleted.status).toBe(200);- });-- it("rejects create bodies that combine evaluators with the deprecated mapping alias", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Conflicting alias evaluator",- prompt: "Judge {{input}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- const response = await makeAPICall(- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "conflicting_alias_rule",- evaluators: [- { evaluator: { name: evaluator.body.name, type: "llm_as_judge" } },- ],- mapping: [{ variable: "input", source: "input" }],- target: "observation",- enabled: false,- filter: [],- },- auth,- );- expect(response.status).toBe(400);- });-- it("stores experiment rules and evaluator usage in the new tables", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Project experiment evaluator",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- const created = await makeZodVerifiedAPICall(- PostUnstableEvaluationRuleResponse,- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "project_experiment_rule",- evaluator: {- name: evaluator.body.name,- type: "llm_as_judge",- },- target: "experiment",- enabled: false,- filter: [],- mapping: [- { variable: "input", source: "input" },- { variable: "output", source: "output" },- ],- },- auth,- );- expect(created.body).toMatchObject({- evaluator: { id: evaluator.body.id },- target: "experiment",- enabled: false,- });- await expect(- prisma.evaluationRule.findUniqueOrThrow({- where: { id: created.body.id },- include: { assignments: true },- }),- ).resolves.toMatchObject({- targetObject: "event",- filter: [- {- column: "isExperimentItemRootSpan",- operator: "=",- value: true,- },- ],- assignments: [{ evaluatorId: evaluator.body.id }],- });- await expect(- prisma.jobConfiguration.findUnique({ where: { id: created.body.id } }),- ).resolves.toBeNull();-- const observationRule = await makeZodVerifiedAPICall(- PatchUnstableEvaluationRuleResponse,- "PATCH",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- { target: "observation" },- auth,- );- expect(observationRule.body).toMatchObject({ target: "observation" });- await expect(- prisma.evaluationRule.findUniqueOrThrow({- where: { id: created.body.id },- }),- ).resolves.toMatchObject({ targetObject: "event", filter: [] });-- const experimentRule = await makeZodVerifiedAPICall(- PatchUnstableEvaluationRuleResponse,- "PATCH",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- { target: "experiment" },- auth,- );- expect(experimentRule.body).toMatchObject({ target: "experiment" });- await expect(- prisma.evaluationRule.findUniqueOrThrow({- where: { id: created.body.id },- }),- ).resolves.toMatchObject({- targetObject: "event",- filter: [- {- column: "isExperimentItemRootSpan",- operator: "=",- value: true,- },- ],- });-- // An experiment rule is an observation rule scoped to experiment root- // spans, so observation filters survive the round trip alongside the- // implicit root filter rather than failing the experiment filter schema.- const scopedExperimentRule = await makeZodVerifiedAPICall(- PatchUnstableEvaluationRuleResponse,- "PATCH",- `/api/public/unstable/evaluation-rules/${created.body.id}`,- {- target: "experiment",- filter: [- {- type: "stringOptions",- column: "environment",- operator: "any of",- value: ["production"],- },- ],- },- auth,- );- expect(scopedExperimentRule.body).toMatchObject({- target: "experiment",- filter: [{ column: "environment", value: ["production"] }],- });- await expect(- prisma.evaluationRule.findUniqueOrThrow({- where: { id: created.body.id },- }),- ).resolves.toMatchObject({- targetObject: "event",- filter: [- { column: "environment" },- { column: "isExperimentItemRootSpan", operator: "=", value: true },- ],- });-- const fetchedEvaluator = await makeZodVerifiedAPICall(- GetUnstableEvaluatorResponse,- "GET",- `/api/public/unstable/evaluators/${evaluator.body.id}`,- undefined,- auth,- );- expect(fetchedEvaluator.body.evaluationRuleCount).toBe(1);-- await makeZodVerifiedAPICall(- DeleteUnstableEvaluatorResponse,- "DELETE",- `/api/public/unstable/evaluators/${evaluator.body.id}`,- undefined,- auth,- );- await expect(- prisma.evaluationRule.findUnique({- where: { id: created.body.id },- include: { assignments: true },- }),- ).resolves.toMatchObject({ id: created.body.id, assignments: [] });- });-- it("reads native trace and dataset rules but keeps them read-only", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Project trace evaluator",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );- const mapping = [- {- variable: "input",- langfuseObject: "trace",- objectName: null,- source: "input",- },- {- variable: "output",- langfuseObject: "generation",- objectName: "answer-generation",- source: "output",- jsonPath: "$.answer",- },- ];-- const nativeRule = await prisma.evaluationRule.create({- data: {- projectId,- name: "project_trace_rule",- targetObject: "trace",- status: "ACTIVE",- sampling: 0.5,- delay: 250,- timeScope: ["NEW"],- filter: [],- assignments: {- create: {- projectId,- evaluatorId: evaluator.body.id,- variableMapping: mapping.map((item) => ({- templateVariable: item.variable,- langfuseObject: item.langfuseObject,- objectName: item.objectName,- selectedColumnId: item.source,- jsonSelector: item.jsonPath ?? null,- })),- },- },- },- });-- const fetched = await makeZodVerifiedAPICall(- GetUnstableEvaluationRuleResponse,- "GET",- `/api/public/unstable/evaluation-rules/${nativeRule.id}`,- undefined,- auth,- );- expect(fetched.body).toMatchObject({- id: nativeRule.id,- name: "project_trace_rule",- evaluator: { id: evaluator.body.id },- target: "trace",- enabled: true,- sampling: 0.5,- delay: 250,- timeScope: ["NEW"],- mapping,- });-- const listed = await makeZodVerifiedAPICall(- GetUnstableEvaluationRulesResponse,- "GET",- "/api/public/unstable/evaluation-rules?page=1&limit=50",- undefined,- auth,- );- expect(listed.body.data).toContainEqual(fetched.body);-- const createResponse = await makeAPICall(- "POST",- "/api/public/unstable/evaluation-rules",- {- name: "new_trace_rule",- evaluator: {- name: evaluator.body.name,- type: "llm_as_judge",- },- target: "trace",- enabled: false,- filter: [],- mapping,- },- auth,- );- expectUnstableError(createResponse, { status: 400, code: "invalid_body" });-- const updateResponse = await makeAPICall(- "PATCH",- `/api/public/unstable/evaluation-rules/${nativeRule.id}`,- { name: "project_trace_rule_updated", enabled: false },- auth,- );- expectUnstableError(updateResponse, {- status: 404,- code: "resource_not_found",- });-- const deleteResponse = await makeAPICall(- "DELETE",- `/api/public/unstable/evaluation-rules/${nativeRule.id}`,- undefined,- auth,- );- expectUnstableError(deleteResponse, {- status: 404,- code: "resource_not_found",- });-- const nativeDatasetRule = await prisma.evaluationRule.create({- data: {- projectId,- name: "project_dataset_rule",- targetObject: "dataset",- status: "INACTIVE",- sampling: 1,- delay: 0,- timeScope: ["EXISTING"],- filter: [],- assignments: {- create: {- projectId,- evaluatorId: evaluator.body.id,- variableMapping: mapping.map((item) => ({- templateVariable: item.variable,- langfuseObject: "dataset_item",- objectName: null,- selectedColumnId: item.source,- jsonSelector: item.jsonPath ?? null,- })),- },- },- },- });- const fetchedDatasetRule = await makeZodVerifiedAPICall(- GetUnstableEvaluationRuleResponse,- "GET",- `/api/public/unstable/evaluation-rules/${nativeDatasetRule.id}`,- undefined,- auth,- );- expect(fetchedDatasetRule.body).toMatchObject({- id: nativeDatasetRule.id,- target: "dataset",- enabled: false,- timeScope: ["EXISTING"],- mapping: [- expect.objectContaining({ langfuseObject: "dataset_item" }),- expect.objectContaining({ langfuseObject: "dataset_item" }),- ],- });- });-- it("returns method_not_allowed for evaluator patch", async () => {- const evaluator = await makeZodVerifiedAPICall(- PostUnstableEvaluatorResponse,- "POST",- "/api/public/unstable/evaluators",- {- name: "Correctness",- prompt: "Judge {{input}} against {{output}}",- outputDefinition: numericOutputDefinition,- },- auth,- );-- const patchRes = await makeAPICall(- "PATCH",- `/api/public/unstable/evaluators/${evaluator.body.id}`,- {- prompt: "Updated",- },- auth,- );-- expectUnstableError(patchRes, {- status: 405,- code: "method_not_allowed",- });- });-- it("still rejects invalid auth with the unstable error envelope", async () => {- const result = await createOrgProjectAndApiKey();- __orgIds.push(result.orgId);- const orgApiKey = await createAndAddApiKeysToDb({- prisma,- entityId: result.orgId,- scope: ApiKeyScope.ORGANIZATION,- });-- const response = await makeAPICall(- "GET",- "/api/public/unstable/evaluators?page=1&limit=10",- undefined,- createBasicAuthHeader(orgApiKey.publicKey, orgApiKey.secretKey),- );-- expectUnstableError(response, {- status: 403,- code: "access_denied",- });- });-});diff --git a/web/src/__tests__/server/unstable-evaluator-v2-api.servertest.ts b/web/src/__tests__/server/unstable-evaluator-v2-api.servertest.tsdeleted file mode 100644index ee73cdbec8..0000000000--- a/web/src/__tests__/server/unstable-evaluator-v2-api.servertest.ts+++ /dev/null@@ -1,304 +0,0 @@-import {- afterAll,- afterEach,- beforeAll,- describe,- expect,- it,- vi,-} from "vitest";--vi.mock(- "@/src/features/evals/v2/server/evaluators/evaluatorValidation",- () => ({ assertEvaluatorConfigurationValid: vi.fn() }),-);--import { Prisma, prisma } from "@langfuse/shared/src/db";-import { createOrgProjectAndApiKey } from "@langfuse/shared/src/server";-import {- createPublicEvaluator,- deletePublicEvaluator,- getPublicEvaluator,- listPublicEvaluators,-} from "@/src/features/evals/server/unstable-public-api/evaluator-service";--const orgIds: string[] = [];-let projectId = "";-let otherProjectId = "";--const input = (name: string, prompt: string) => ({- name,- type: "llm_as_judge" as const,- prompt,- modelConfig: null,- outputDefinition: {- dataType: "NUMERIC" as const,- score: { description: "Quality" },- reasoning: { description: "Reasoning" },- },-});--beforeAll(async () => {- const [first, second] = await Promise.all([- createOrgProjectAndApiKey(),- createOrgProjectAndApiKey(),- ]);- projectId = first.project.id;- otherProjectId = second.project.id;- orgIds.push(first.org.id, second.org.id);-});--afterEach(async () => {- await prisma.evaluationRule.deleteMany({- where: { projectId: { in: [projectId, otherProjectId] } },- });- await prisma.evaluator.deleteMany({- where: { projectId: { in: [projectId, otherProjectId] } },- });-});--afterAll(async () => {- await prisma.organization.deleteMany({ where: { id: { in: orgIds } } });-});--describe("unstable evaluator API on stable evaluator storage", () => {- it("creates and versions an evaluator by name", async () => {- const first = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}}"),- });- const second = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}} strictly"),- });- const third = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}} strictly"),- });-- expect(first).toMatchObject({ name: "Public evaluator", version: 1 });- expect(second).toMatchObject({ name: first.name, version: 2 });- expect(third).toMatchObject({ name: first.name, version: 3 });- expect(second.id).toBe(first.id);- expect(third.id).toBe(first.id);- });-- it("returns the latest evaluator and its rule count", async () => {- const first = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}}"),- });- const latest = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}} strictly"),- });- await prisma.evaluationRule.create({- data: {- projectId,- name: "Public evaluator rule",- status: "INACTIVE",- targetObject: "EVENT",- filter: [],- sampling: 1,- delay: 0,- assignments: {- create: {- projectId,- evaluatorId: first.id,- variableMapping: Prisma.DbNull,- },- },- },- });-- await expect(- getPublicEvaluator({ projectId, evaluatorId: first.id }),- ).resolves.toMatchObject({- id: first.id,- version: latest.version,- evaluationRuleCount: 1,- });- });-- it("ignores stale legacy job configurations when counting evaluator rules", async () => {- const evaluator = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}}"),- });- await prisma.evalTemplate.create({- data: {- id: evaluator.id,- projectId,- name: evaluator.name,- version: 1,- prompt: "Judge {{output}}",- type: "LLM_AS_JUDGE",- vars: ["output"],- outputDefinition: Prisma.DbNull,- },- });- const legacyRule = await prisma.jobConfiguration.create({- data: {- projectId,- jobType: "EVAL",- status: "INACTIVE",- evalTemplateId: evaluator.id,- scoreName: "Stale legacy rule",- targetObject: "event",- filter: [],- variableMapping: [],- sampling: 1,- delay: 0,- },- });-- await expect(- getPublicEvaluator({ projectId, evaluatorId: evaluator.id }),- ).resolves.toMatchObject({ evaluationRuleCount: 0 });-- await prisma.jobConfiguration.delete({ where: { id: legacyRule.id } });- await prisma.evalTemplate.delete({ where: { id: evaluator.id } });- });-- it("rejects legacy version IDs and cross-project reads", async () => {- const evaluator = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}}"),- });- const oldVersion = await prisma.evaluatorVersion.findFirstOrThrow({- where: { evaluatorId: evaluator.id, version: 1 },- select: { id: true },- });- await expect(- getPublicEvaluator({ projectId, evaluatorId: oldVersion.id }),- ).rejects.toThrow("Evaluator not found");- await expect(- getPublicEvaluator({- projectId: otherProjectId,- evaluatorId: evaluator.id,- }),- ).rejects.toThrow("Evaluator not found");- });-- it("returns an empty evaluator list", async () => {- await expect(- listPublicEvaluators({ projectId, page: 1, limit: 50 }),- ).resolves.toMatchObject({- data: [],- meta: { page: 1, limit: 50, totalItems: 0, totalPages: 0 },- });- });-- it("lists only the latest evaluators from the requested project", async () => {- const first = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}}"),- });- const latest = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}} strictly"),- });- await createPublicEvaluator({- projectId: otherProjectId,- input: input("Public evaluator", "Other project {{output}}"),- });-- const projectList = await listPublicEvaluators({- projectId,- page: 1,- limit: 50,- });- expect(projectList.data).toEqual([- expect.objectContaining({ id: first.id, version: latest.version }),- ]);- expect(projectList.meta).toMatchObject({ totalItems: 1, totalPages: 1 });- });-- it("paginates evaluator lists", async () => {- const [older, newer] = await Promise.all([- createPublicEvaluator({- projectId,- input: input("Older evaluator", "Judge {{output}}"),- }),- createPublicEvaluator({- projectId,- input: input("Newer evaluator", "Judge {{output}}"),- }),- ]);- await Promise.all([- prisma.evaluator.update({- where: { id: older.id },- data: { updatedAt: new Date("2026-01-01T00:00:00.000Z") },- }),- prisma.evaluator.update({- where: { id: newer.id },- data: { updatedAt: new Date("2026-01-02T00:00:00.000Z") },- }),- ]);-- await expect(- listPublicEvaluators({ projectId, page: 1, limit: 1 }),- ).resolves.toMatchObject({- data: [{ id: newer.id }],- meta: { page: 1, limit: 1, totalItems: 2, totalPages: 2 },- });- await expect(- listPublicEvaluators({ projectId, page: 2, limit: 1 }),- ).resolves.toMatchObject({- data: [{ id: older.id }],- meta: { page: 2, limit: 1, totalItems: 2, totalPages: 2 },- });- });-- it("deletes the evaluator and its rule assignments", async () => {- const evaluator = await createPublicEvaluator({- projectId,- input: input("Public evaluator", "Judge {{output}}"),- });- const rule = await prisma.evaluationRule.create({- data: {- projectId,- name: "Public evaluator rule",- status: "INACTIVE",- targetObject: "EVENT",- filter: [],- sampling: 1,- delay: 0,- assignments: {- create: {- projectId,- evaluatorId: evaluator.id,- variableMapping: Prisma.DbNull,- },- },- },- });-- await deletePublicEvaluator({ projectId, evaluatorId: evaluator.id });-- await expect(- prisma.evaluationRuleEvaluatorAssignment.count({- where: { evaluationRuleId: rule.id, evaluatorId: evaluator.id },- }),- ).resolves.toBe(0);- await expect(- prisma.evaluator.findUnique({ where: { id: evaluator.id } }),- ).resolves.toBeNull();- });-- it("rejects ambiguous legacy name upserts", async () => {- await prisma.evaluator.createMany({- data: [- { projectId, name: "Duplicate", type: "LLM_AS_JUDGE" },- { projectId, name: "Duplicate", type: "LLM_AS_JUDGE" },- ],- });-- await expect(- createPublicEvaluator({- projectId,- input: input("Duplicate", "Judge {{output}}"),- }),- ).rejects.toMatchObject({ httpCode: 409 });- });-});diff --git a/web/src/features/evals/server/codeEvalJobConfigValidation.ts b/web/src/features/evals/server/codeEvalJobConfigValidation.tsindex c9c84c4262..c40ff7f1dd 100644--- a/web/src/features/evals/server/codeEvalJobConfigValidation.ts+++ b/web/src/features/evals/server/codeEvalJobConfigValidation.ts@@ -9,7 +9,6 @@ import type { PrismaClient } from "@prisma/client";import { env } from "@/src/env.mjs";import {CodeEvalTestRunSetupError,- runCodeEvalTestForEvaluationRule,runCodeEvalTestForJobConfig,} from "@/src/features/evals/server/codeEvalTestRun";import { assertUnreachable } from "@/src/utils/types";@@ -51,26 +50,6 @@ export async function assertCodeEvalJobConfigCanRun(params: {});}-export async function assertCodeEvalRuleCanRun(params: {- prisma: PrismaClient;- orgId: string;- projectId: string;- evaluatorId: string;- target: EvalTargetObjectType;- mapping: unknown;- scoreName: string;- filter: FilterCondition[] | null;-}): Promise<void> {- return assertCodeEvalCanRun({- ...params,- run: (mapping) =>- runCodeEvalTestForEvaluationRule({- ...params,- mapping,- }),- });-}-async function assertCodeEvalCanRun(params: {target: EvalTargetObjectType;mapping: unknown;diff --git a/web/src/features/evals/server/codeEvalTestRun.ts b/web/src/features/evals/server/codeEvalTestRun.tsindex e094161f09..18fd176d1b 100644--- a/web/src/features/evals/server/codeEvalTestRun.ts+++ b/web/src/features/evals/server/codeEvalTestRun.ts@@ -115,62 +115,6 @@ export async function runCodeEvalTestForJobConfig(params: {});}-export async function runCodeEvalTestForEvaluationRule(params: {- prisma: PrismaClient;- orgId: string;- projectId: string;- evaluatorId: string;- target: EvalTargetObject;- mapping: ObservationVariableMapping[];- scoreName: string;- filter: FilterCondition[] | null;-}): Promise<CodeEvalTestRunResult | null> {- const observation = await getObservationForEvalByFilter({- prisma: params.prisma,- projectId: params.projectId,- target: params.target,- filter: params.filter,- });-- if (!observation) return null;-- const evaluator = await params.prisma.evaluator.findFirst({- where: {- id: params.evaluatorId,- projectId: params.projectId,- type: EvalTemplateType.CODE,- },- include: {- versions: {- where: { sourceCode: { not: null }, sourceCodeLanguage: { not: null } },- orderBy: { version: "desc" },- take: 1,- },- },- });- const version = evaluator?.versions[0];- if (!evaluator || !version?.sourceCode || !version.sourceCodeLanguage) {- throw new CodeEvalTestRunSetupError(- "TEMPLATE_NOT_FOUND",- "Evaluator not found",- );- }-- return runCodeEvalTestForObservationWithEvaluator({- ...params,- observation,- evaluator,- version: {- ...version,- sourceCode: version.sourceCode,- sourceCodeLanguage: version.sourceCodeLanguage,- },- evaluatorMetadata: {- evaluator_version: version.version,- },- });-}-async function runCodeEvalTestForObservation(params: {prisma: PrismaClient;orgId: string;diff --git a/web/src/features/evals/server/unstable-public-api/adapters.ts b/web/src/features/evals/server/unstable-public-api/adapters.tsdeleted file mode 100644index 42f6f516c0..0000000000--- a/web/src/features/evals/server/unstable-public-api/adapters.ts+++ /dev/null@@ -1,690 +0,0 @@-import {- createBooleanEvalOutputDefinition,- createCategoricalEvalOutputDefinition,- createNumericEvalOutputDefinition,- EvalTargetObject,- extractVariables,- JobTimeScopeZod,- JobConfigState,- isExperimentEvaluationRule,- observationVariableMappingList,- PersistedEvalOutputDefinitionSchema,- resolvePersistedEvalOutputDefinition,- singleFilter,- stripExperimentRootFilter,- type FilterState,- type ObservationVariableMapping,- type PersistedEvalOutputDefinition,- variableMappingList,- InternalServerError,- CODE_EVAL_TEMPLATE_VARIABLES,- getCodeEvalVariableMapping,-} from "@langfuse/shared";-import { EvalTemplateType } from "@langfuse/shared/src/db";-import { logger } from "@langfuse/shared/src/server";-import { z } from "zod";-import {- getLegacyEvaluatorPrompt,- reconcileEvaluatorPromptMessages,-} from "@/src/features/evals/v2/server/evaluators/evaluatorService";-import {- LegacyPromptVariableMapping,- PUBLIC_EVALUATOR_TYPE_CODE,- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- PublicEvaluationRuleFilter,- PublicEvaluationRuleReadFilter,- type PublicEvaluationRuleFilterType,- type PromptVariableMappingInputType,- type PromptVariableMappingReadType,- type PublicEvaluationRuleReadTargetType,- type PublicEvaluationRuleStatusType,- type PublicEvaluationRuleTargetType,- type LegacyPromptVariableMappingType,- type PublicEvaluatorModelConfigType,- type PublicEvaluatorOutputDefinitionType,- type PublicEvaluatorTypeType,-} from "@/src/features/public-api/server";-import type {- ApiEvaluatorRecord,- ApiEvaluationRuleRecord,- ApiWritableEvaluationRuleRecord,- StoredPublicEvaluatorTemplate,- StoredPublicV2EvaluationRule,-} from "./types";-import {- validateEvaluationRuleFilters,- validateEvaluatorVariableMappings,-} from "./validation";--export function toPublicEvaluatorType(type: EvalTemplateType) {- return type === EvalTemplateType.CODE- ? PUBLIC_EVALUATOR_TYPE_CODE- : PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE;-}--export function toStoredEvaluatorType(- type: PublicEvaluatorTypeType,-): EvalTemplateType {- return type === PUBLIC_EVALUATOR_TYPE_CODE- ? EvalTemplateType.CODE- : EvalTemplateType.LLM_AS_JUDGE;-}--const PUBLIC_TARGET_TO_INTERNAL_TARGET_OBJECT = {- observation: EvalTargetObject.EVENT,- experiment: EvalTargetObject.EXPERIMENT,-} satisfies Record<PublicEvaluationRuleTargetType, EvalTargetObject>;--const INTERNAL_TARGET_OBJECT_TO_PUBLIC_TARGET: Record<- string,- PublicEvaluationRuleReadTargetType-> = {- [EvalTargetObject.EVENT]: "observation",- [EvalTargetObject.EXPERIMENT]: "experiment",- [EvalTargetObject.TRACE]: "trace",- [EvalTargetObject.DATASET]: "dataset",-};--const PUBLIC_MAPPING_SOURCE_TO_INTERNAL_COLUMN: Record<- PromptVariableMappingInputType["source"],- ObservationVariableMapping["selectedColumnId"]-> = {- input: "input",- output: "output",- metadata: "metadata",- tool_calls: "toolCalls",- expected_output: "experimentItemExpectedOutput",- experiment_item_metadata: "experimentItemMetadata",-};--const INTERNAL_MAPPING_COLUMN_TO_PUBLIC_SOURCE: Record<- string,- PromptVariableMappingInputType["source"]-> = {- input: "input",- output: "output",- metadata: "metadata",- // Only camelCase for tool calls: the column id is new with tool-call- // support, so unlike expected_output no legacy snake_case rows exist. An- // accidental "tool_calls" write should surface at the corrupted-mapping- // error boundary, not be absorbed here.- toolCalls: "tool_calls",- expected_output: "expected_output",- expectedOutput: "expected_output",- experiment_item_expected_output: "expected_output",- experimentItemExpectedOutput: "expected_output",- experimentItemMetadata: "experiment_item_metadata",- experiment_item_metadata: "experiment_item_metadata",-};--export function deriveEvaluatorVariables(- template: Pick<- StoredPublicEvaluatorTemplate,- "vars" | "prompt" | "promptMessages"- >,-) {- return template.vars.length > 0- ? template.vars- : reconcileEvaluatorPromptMessages(template).flatMap(({ content }) =>- extractVariables(content),- );-}--export function toStoredVariableMappings(params: {- mappings: PromptVariableMappingInputType[];- variables: string[];- target: PublicEvaluationRuleTargetType;-}) {- validateEvaluatorVariableMappings({- mappings: params.mappings,- variables: params.variables,- target: params.target,- });-- return toStoredMappingList(params.mappings);-}--/**- * Shape-only translation of the public mapping contract to storage. Use this- * where the evaluator's variables are not known and coverage is validated- * elsewhere; prefer `toStoredVariableMappings` when they are.- */-export function toStoredMappingList(- mappings: PromptVariableMappingInputType[],-) {- return observationVariableMappingList.parse(- mappings.map((mapping) => ({- templateVariable: mapping.variable,- selectedColumnId:- PUBLIC_MAPPING_SOURCE_TO_INTERNAL_COLUMN[mapping.source],- jsonSelector: mapping.jsonPath ?? null,- })),- );-}--function parseStoredOutputDefinition(- template: Pick<StoredPublicEvaluatorTemplate, "id" | "outputDefinition">,-): PublicEvaluatorOutputDefinitionType {- const parsed = PersistedEvalOutputDefinitionSchema.safeParse(- template.outputDefinition,- );-- if (!parsed.success) {- logger.error(- "Failed to parse unstable public evaluator output definition",- {- issues: parsed.error.issues,- templateId: "id" in template ? template.id : undefined,- },- );- throw new InternalServerError("Evaluator output definition is corrupted");- }-- const resolvedOutputDefinition = resolvePersistedEvalOutputDefinition(- parsed.data,- );-- if (resolvedOutputDefinition.dataType === "CATEGORICAL") {- return {- dataType: "CATEGORICAL",- reasoning: {- description: resolvedOutputDefinition.reasoningDescription,- },- score: {- description: resolvedOutputDefinition.scoreDescription,- categories: resolvedOutputDefinition.categories,- shouldAllowMultipleMatches:- resolvedOutputDefinition.shouldAllowMultipleMatches,- },- };- }-- if (resolvedOutputDefinition.dataType === "NUMERIC") {- const { minValue, maxValue } = resolvedOutputDefinition;- return {- dataType: "NUMERIC",- reasoning: {- description: resolvedOutputDefinition.reasoningDescription,- },- score: {- description: resolvedOutputDefinition.scoreDescription,- ...(minValue !== undefined ? { minValue } : {}),- ...(maxValue !== undefined ? { maxValue } : {}),- },- };- }-- return {- dataType: resolvedOutputDefinition.dataType,- reasoning: {- description: resolvedOutputDefinition.reasoningDescription,- },- score: {- description: resolvedOutputDefinition.scoreDescription,- },- };-}--export function toStoredOutputDefinition(- outputDefinition: PublicEvaluatorOutputDefinitionType,-): PersistedEvalOutputDefinition {- switch (outputDefinition.dataType) {- case "NUMERIC":- return createNumericEvalOutputDefinition({- reasoningDescription: outputDefinition.reasoning.description,- scoreDescription: outputDefinition.score.description,- minValue: outputDefinition.score.minValue,- maxValue: outputDefinition.score.maxValue,- });- case "BOOLEAN":- return createBooleanEvalOutputDefinition({- reasoningDescription: outputDefinition.reasoning.description,- scoreDescription: outputDefinition.score.description,- });- case "CATEGORICAL":- return createCategoricalEvalOutputDefinition({- reasoningDescription: outputDefinition.reasoning.description,- scoreDescription: outputDefinition.score.description,- categories: outputDefinition.score.categories,- shouldAllowMultipleMatches:- outputDefinition.score.shouldAllowMultipleMatches,- });- }-}--function toApiModelConfig(- template: Pick<- StoredPublicEvaluatorTemplate,- "provider" | "model" | "modelParams"- >,-): PublicEvaluatorModelConfigType | null {- if (!template.provider || !template.model) {- return null;- }-- return {- provider: template.provider,- model: template.model,- };-}--function assertPublicTarget(- targetObject: string,-): PublicEvaluationRuleReadTargetType {- const publicTarget = INTERNAL_TARGET_OBJECT_TO_PUBLIC_TARGET[targetObject];-- if (!publicTarget) {- throw new InternalServerError("Evaluation rule target is corrupted");- }-- return publicTarget;-}--function toStoredFilter(- filter: PublicEvaluationRuleFilterType,- target: PublicEvaluationRuleTargetType,-): PublicEvaluationRuleFilterType {- if (target === "experiment" && filter.column === "datasetId") {- return {- ...filter,- column: "experimentDatasetId",- };- }-- return filter;-}--function toApiFilter(filter: z.infer<typeof singleFilter>) {- if (filter.column === "experimentDatasetId") {- return {- ...filter,- column: "datasetId",- };- }-- return filter;-}--export function toApiReadMappings(- mappings: unknown,-): PromptVariableMappingReadType[] {- const parsed = observationVariableMappingList.safeParse(mappings);-- if (!parsed.success) {- logger.error("Failed to parse unstable public evaluation rule mappings", {- issues: parsed.error.issues,- });- throw new InternalServerError("Evaluation rule mapping is corrupted");- }-- return parsed.data.map((mapping) => {- const source =- INTERNAL_MAPPING_COLUMN_TO_PUBLIC_SOURCE[mapping.selectedColumnId];-- if (!source && mapping.selectedColumnId.trim()) {- throw new InternalServerError("Evaluation rule mapping is corrupted");- }-- return {- variable: mapping.templateVariable,- source: source ?? null,- ...(mapping.jsonSelector ? { jsonPath: mapping.jsonSelector } : {}),- };- });-}--export function toApiMappings(- mappings: unknown,-): PromptVariableMappingInputType[] {- return toApiReadMappings(mappings).map((mapping) => {- if (mapping.source === null) {- throw new InternalServerError("Evaluation rule mapping is corrupted");- }-- return { ...mapping, source: mapping.source };- });-}--function toApiLegacyMappings(- mappings: unknown,-): LegacyPromptVariableMappingType[] {- const parsed = variableMappingList.safeParse(mappings);-- if (!parsed.success) {- logger.error("Failed to parse unstable public legacy rule mappings", {- issues: parsed.error.issues,- });- throw new InternalServerError("Evaluation rule mapping is corrupted");- }-- const publicMappings = parsed.data.map((mapping) => ({- variable: mapping.templateVariable,- langfuseObject: mapping.langfuseObject,- objectName: mapping.objectName ?? null,- source: mapping.selectedColumnId,- ...(mapping.jsonSelector ? { jsonPath: mapping.jsonSelector } : {}),- }));- const parsedPublicMappings = z- .array(LegacyPromptVariableMapping)- .safeParse(publicMappings);- if (!parsedPublicMappings.success) {- logger.error("Failed to parse unstable public legacy rule mappings", {- issues: parsedPublicMappings.error.issues,- });- throw new InternalServerError("Evaluation rule mapping is corrupted");- }- return parsedPublicMappings.data;-}--function toDefaultLegacyMappings(- mappings: unknown,- target: "trace" | "dataset",-): LegacyPromptVariableMappingType[] {- return toApiMappings(mappings).map((mapping) => ({- ...mapping,- langfuseObject: target === "trace" ? "trace" : "dataset_item",- objectName: null,- }));-}--function getLegacyCodeEvalVariableMapping(target: "trace" | "dataset") {- return variableMappingList.parse(- CODE_EVAL_TEMPLATE_VARIABLES.map((variable) => ({- templateVariable: variable,- langfuseObject: target === "trace" ? "trace" : "dataset_item",- objectName: null,- selectedColumnId: variable,- jsonSelector: null,- })),- );-}--function toApiFilters(- filters: unknown,- target: PublicEvaluationRuleTargetType,-) {- const storedFilters = z.array(singleFilter).safeParse(filters);-- if (!storedFilters.success) {- logger.error("Failed to parse unstable public evaluation rule filters", {- issues: storedFilters.error.issues,- });- throw new InternalServerError("Evaluation rule filter is corrupted");- }-- const effectiveFilters =- target === "experiment"- ? stripExperimentRootFilter(storedFilters.data)- : storedFilters.data;- return z- .array(PublicEvaluationRuleReadFilter)- .parse(effectiveFilters.map(toApiFilter));-}--export function toApiEvaluator(params: {- template: StoredPublicEvaluatorTemplate;- evaluationRuleCount: number;-}): ApiEvaluatorRecord {- const template = params.template;- const base = {- id: template.id,- name: template.name,- version: template.version,- variables: deriveEvaluatorVariables(template),- mapping:- template.variableMapping == null- ? null- : toApiReadMappings(template.variableMapping),- evaluationRuleCount: params.evaluationRuleCount,- createdAt: template.createdAt,- updatedAt: template.updatedAt,- } as const;-- if (template.type === EvalTemplateType.CODE) {- if (!template.sourceCode || !template.sourceCodeLanguage) {- throw new InternalServerError("Code evaluator definition is corrupted");- }-- return {- ...base,- type: PUBLIC_EVALUATOR_TYPE_CODE,- variables: [...CODE_EVAL_TEMPLATE_VARIABLES],- sourceCode: template.sourceCode,- sourceCodeLanguage: template.sourceCodeLanguage,- };- }-- const promptMessages = reconcileEvaluatorPromptMessages(template);- if (promptMessages.every(({ content }) => content.length === 0)) {- throw new InternalServerError("Evaluator prompt messages are corrupted");- }-- return {- ...base,- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- prompt: getLegacyEvaluatorPrompt(promptMessages),- outputDefinition: parseStoredOutputDefinition(template),- modelConfig: toApiModelConfig(template),- };-}--// Blocking used to live on the job configuration, so the public contract reports it on the rule.-// In the v2 model it lives on the evaluator, so read it back off the rule's assignments to keep the-// contract intact. Every migrated rule has exactly one evaluator, which is also the only shape the-// pre-v2 API could produce, so those reproduce the old responses one for one. A rule with several-// evaluators reports paused as soon as one of them is blocked, since it is no longer writing every-// score it is configured to write.-function toApiRuleBlockState(rule: StoredPublicV2EvaluationRule) {- const blocked = rule.assignments- .map((candidate) => candidate.evaluator)- .filter(- (candidate): candidate is typeof candidate & { blockedAt: Date } =>- candidate.blockedAt !== null,- )- // most recently blocked wins, matching how the backfill picked a group's reason- .sort(- (left, right) => right.blockedAt.getTime() - left.blockedAt.getTime(),- );- const source = blocked[0];- return {- isBlocked: source !== undefined,- pausedReason: source?.blockReason ?? null,- pausedMessage: source?.blockMessage ?? null,- };-}--function toApiEvaluationRuleStatus(- status: JobConfigState,- isBlocked: boolean,-): PublicEvaluationRuleStatusType {- if (status !== JobConfigState.ACTIVE) {- return "inactive";- }- return isBlocked ? "paused" : "active";-}--export function toApiV2EvaluationRule(- rule: StoredPublicV2EvaluationRule,-): ApiEvaluationRuleRecord {- // A rule can legitimately have no assignments (e.g. its last evaluator was- // detached). The compatibility `evaluator`/`mapping` aliases then report- // null/empty rather than the resource 404ing.- const assignment = rule.assignments[0] ?? null;- const evaluator = assignment?.evaluator ?? null;- const block = toApiRuleBlockState(rule);-- const toEvaluator = (candidate: NonNullable<typeof assignment>) => ({- id: candidate.evaluator.id,- name: candidate.evaluator.name,- type: toPublicEvaluatorType(candidate.evaluator.type),- });- const storedFilters = z.array(singleFilter).safeParse(rule.filter);- const target =- storedFilters.success &&- isExperimentEvaluationRule({- targetObject: rule.targetObject as EvalTargetObject,- filter: storedFilters.data as FilterState,- })- ? "experiment"- : assertPublicTarget(rule.targetObject);- if (target === "trace" || target === "dataset") {- const evaluators = rule.assignments.map((candidate) => ({- evaluator: toEvaluator(candidate),- mapping:- candidate.variableMapping == null- ? null- : toApiLegacyMappings(candidate.variableMapping),- }));- const effectiveFirstMapping =- assignment === null || evaluator === null- ? []- : assignment.variableMapping != null- ? toApiLegacyMappings(assignment.variableMapping)- : evaluator.type === EvalTemplateType.CODE- ? toApiLegacyMappings(getLegacyCodeEvalVariableMapping(target))- : toDefaultLegacyMappings(- evaluator.versions[0]?.variableMapping,- target,- );- const timeScope = z.array(JobTimeScopeZod).safeParse(rule.timeScope);- if (!timeScope.success) {- throw new InternalServerError("Evaluation rule time scope is corrupted");- }-- return {- id: rule.id,- name: rule.name,- evaluator: assignment === null ? null : toEvaluator(assignment),- evaluators,- enabled: rule.status === JobConfigState.ACTIVE,- status: toApiEvaluationRuleStatus(rule.status, block.isBlocked),- pausedReason: block.pausedReason,- pausedMessage: block.pausedMessage,- sampling: Number(rule.sampling),- target,- delay: rule.delay,- timeScope: timeScope.data,- filter: z.array(singleFilter).parse(rule.filter),- mapping: effectiveFirstMapping,- createdAt: rule.createdAt,- updatedAt: rule.updatedAt,- };- }-- const evaluators = rule.assignments.map((candidate) => ({- evaluator: toEvaluator(candidate),- mapping:- candidate.variableMapping == null- ? null- : toApiReadMappings(candidate.variableMapping),- }));- const effectiveFirstMapping =- assignment === null || evaluator === null- ? []- : evaluator.type === EvalTemplateType.CODE- ? getCodeEvalVariableMapping()- : (assignment.variableMapping ??- evaluator.versions[0]?.variableMapping ??- []);-- return {- id: rule.id,- name: rule.name,- evaluator: assignment === null ? null : toEvaluator(assignment),- evaluators,- enabled: rule.status === JobConfigState.ACTIVE,- status: toApiEvaluationRuleStatus(rule.status, block.isBlocked),- pausedReason: block.pausedReason,- pausedMessage: block.pausedMessage,- sampling: Number(rule.sampling),- target,- filter: toApiFilters(rule.filter, target),- mapping: toApiReadMappings(effectiveFirstMapping),- createdAt: rule.createdAt,- updatedAt: rule.updatedAt,- };-}--export function toApiWritableV2EvaluationRule(- rule: StoredPublicV2EvaluationRule,-): ApiWritableEvaluationRuleRecord {- const evaluationRule = toApiV2EvaluationRule(rule);- if (- evaluationRule.target !== "observation" &&- evaluationRule.target !== "experiment"- ) {- throw new InternalServerError("Evaluation rule target is corrupted");- }-- const filter = z- .array(PublicEvaluationRuleFilter)- .safeParse(evaluationRule.filter);- if (!filter.success) {- throw new InternalServerError("Evaluation rule filter is corrupted");- }-- return { ...evaluationRule, filter: filter.data };-}--export function toEvaluationRuleInput(params: {- input: {- name: string;- target: PublicEvaluationRuleTargetType;- enabled: boolean;- sampling: number;- filter: PublicEvaluationRuleFilterType[];- mapping?: PromptVariableMappingInputType[];- };- evaluatorVariables: string[];- evaluatorType: PublicEvaluatorTypeType;-}) {- return {- ...toEvaluationRuleFields(params.input),- variableMapping: toStoredAssignmentMapping({- target: params.input.target,- mapping: params.input.mapping,- evaluatorVariables: params.evaluatorVariables,- evaluatorType: params.evaluatorType,- }),- };-}--/**- * Rule-level columns. Independent of any evaluator, so a rule with no- * assignments can still be written.- */-export function toEvaluationRuleFields(input: {- name: string;- target: PublicEvaluationRuleTargetType;- enabled: boolean;- sampling: number;- filter: PublicEvaluationRuleFilterType[];-}) {- const target = input.target;-- validateEvaluationRuleFilters({ target, filters: input.filter });-- return {- scoreName: input.name,- targetObject: PUBLIC_TARGET_TO_INTERNAL_TARGET_OBJECT[target],- filter: input.filter.map((filter) => toStoredFilter(filter, target)),- sampling: input.sampling,- status: input.enabled ? JobConfigState.ACTIVE : JobConfigState.INACTIVE,- };-}--/** Stored mapping for one assignment. Code evaluators inherit their fixed evaluator mapping. */-export function toStoredAssignmentMapping(params: {- target: PublicEvaluationRuleTargetType;- mapping?: PromptVariableMappingInputType[];- evaluatorVariables: string[];- evaluatorType: PublicEvaluatorTypeType;-}) {- return params.evaluatorType === PUBLIC_EVALUATOR_TYPE_CODE- ? null- : toStoredVariableMappings({- mappings: params.mapping ?? [],- variables: params.evaluatorVariables,- target: params.target,- });-}diff --git a/web/src/features/evals/server/unstable-public-api/evaluation-rule-service.ts b/web/src/features/evals/server/unstable-public-api/evaluation-rule-service.tsdeleted file mode 100644index be14fcdbd0..0000000000--- a/web/src/features/evals/server/unstable-public-api/evaluation-rule-service.ts+++ /dev/null@@ -1,882 +0,0 @@-import { type ApiAccessScope } from "@langfuse/shared/src/server";-import { EvalTemplateType, prisma } from "@langfuse/shared/src/db";-import {- JobConfigState,- InternalServerError,- LangfuseNotFoundError,- EvalTargetObject,- type FilterCondition,- type FilterState,- type ObservationVariableMapping,-} from "@langfuse/shared";-import {- assertCodeEvalRuleCanRun,- CodeEvalJobConfigError,-} from "@/src/features/evals/server/codeEvalJobConfigValidation";-import { auditLog } from "@/src/features/audit-logs/auditLog";-import { JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE } from "@/src/features/evals/server/audit-log-resource-types";-import {- isCodeEvalEnabled,- isCodeEvalSourceCodeLanguageSupported,-} from "@/src/features/evals/server/isCodeEvalEnabled";-import { createStructuredPublicApiError } from "@/src/features/public-api";-import {- PublicEvaluationRuleFilter,- type PatchUnstableEvaluationRuleBodyType,- type PostUnstableEvaluationRuleBodyType,- type PromptVariableMappingInputType,- type PromptVariableMappingReadType,- type PublicEvaluationRuleTargetType,- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,-} from "@/src/features/public-api/server";-import {- deriveEvaluatorVariables,- toApiMappings,- toApiV2EvaluationRule,- toApiWritableV2EvaluationRule,- toEvaluationRuleFields,- toEvaluationRuleInput,- toPublicEvaluatorType,- toStoredAssignmentMapping,-} from "./adapters";-import {- ActiveEvaluationRuleLimitError,- assertActiveRuleLimitNotExceeded,-} from "@/src/features/evals/v2/server/rules/ruleErrors";-import { RuleService } from "@/src/features/evals/v2/server/rules/ruleService";-import {- findPublicV2EvaluatorByIdOrThrow,- findPublicV2EvaluationRule,- findPublicV2EvaluatorInFamilyOrThrow,- listPublicEvaluationRulePage,-} from "./queries";-import type {- EvaluationRuleEvaluatorFamilyReference,- StoredPublicEvaluatorTemplate,- StoredPublicV2EvaluationRule,-} from "./types";-import {- assertEvaluationRuleFilterValuesExistForProject,- assertEvaluatorDefinitionCanRunForPublicApi,-} from "./validation";-import { assertUnreachable } from "@/src/utils/types";--const ruleService = new RuleService(prisma, async () => undefined);--function getRequestedEvaluatorAssignments(- input: PostUnstableEvaluationRuleBodyType,-) {- if (input.evaluators) return input.evaluators;- return [{ evaluator: input.evaluator!, mapping: input.mapping }];-}--type PublicV2Evaluator = Awaited<- ReturnType<typeof findPublicV2EvaluatorInFamilyOrThrow>->;--function toEvaluatorDefinition(- evaluator: PublicV2Evaluator,-): StoredPublicEvaluatorTemplate {- const version = evaluator.versions[0];- if (!version) {- throw createStructuredPublicApiError({- httpCode: 500,- code: "internal_error",- message: "Evaluator version is missing",- });- }- return {- id: evaluator.id,- projectId: evaluator.projectId,- name: evaluator.name,- version: version.version,- type: evaluator.type,- prompt: version.prompt,- promptMessages: version.promptMessages,- partner: version.partner,- provider: version.provider,- model: version.model,- modelParams: version.modelParams,- vars: version.vars,- outputDefinition: version.outputDefinition,- sourceCode: version.sourceCode,- sourceCodeLanguage: version.sourceCodeLanguage,- variableMapping: version.variableMapping,- createdAt: evaluator.createdAt,- updatedAt: evaluator.updatedAt,- };-}--async function resolveProjectEvaluator(params: {- projectId: string;- evaluator: EvaluationRuleEvaluatorFamilyReference;- evaluatorId?: string;-}) {- return params.evaluatorId- ? findPublicV2EvaluatorByIdOrThrow({- projectId: params.projectId,- evaluatorId: params.evaluatorId,- })- : findPublicV2EvaluatorInFamilyOrThrow({- projectId: params.projectId,- evaluator: params.evaluator,- });-}--async function assertEvaluationRuleCanRunForPublicApi(params: {- orgId: string;- projectId: string;- evaluatorId: string;- template: StoredPublicEvaluatorTemplate;- target: EvalTargetObject;- mapping: unknown;- scoreName: string;- filter: FilterCondition[] | null;-}) {- if (params.template.type !== EvalTemplateType.CODE) {- await assertEvaluatorDefinitionCanRunForPublicApi({- projectId: params.projectId,- template: {- name: params.template.name,- provider: params.template.provider,- model: params.template.model,- modelParams: params.template.modelParams,- outputDefinition: params.template.outputDefinition,- },- });- return;- }-- if (!isCodeEvalEnabled()) {- throw createStructuredPublicApiError({- httpCode: 403,- code: "access_denied",- message: "Code evals are not enabled",- details: { evaluatorName: params.template.name },- });- }- if (- !isCodeEvalSourceCodeLanguageSupported(params.template.sourceCodeLanguage)- ) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_request",- message:- "This code evaluator language is not supported by the configured dispatcher.",- details: { evaluatorName: params.template.name },- });- }-- try {- await assertCodeEvalRuleCanRun({- prisma,- orgId: params.orgId,- projectId: params.projectId,- evaluatorId: params.evaluatorId,- target: params.target,- mapping: params.mapping,- scoreName: params.scoreName,- filter: params.filter,- });- } catch (error) {- if (!(error instanceof CodeEvalJobConfigError)) throw error;- const details = { evaluatorName: params.template.name };- switch (error.code) {- case "invalid_target":- case "invalid_request":- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_request",- message: error.message,- details,- });- case "resource_not_found":- throw createStructuredPublicApiError({- httpCode: 404,- code: "resource_not_found",- message: error.message,- details,- });- case "preflight_failed":- throw createStructuredPublicApiError({- httpCode: 422,- code: "evaluator_preflight_failed",- message: error.message,- details,- });- default:- return assertUnreachable(error.code);- }- }-}--// The cap itself lives in the rule service so tRPC, MCP, and this API cannot-// drift; only the error envelope is public-API specific.-async function assertActivePublicApiEvaluationRuleLimitNotExceeded(- projectId: string,-) {- try {- await assertActiveRuleLimitNotExceeded({- prisma,- projectId,- additionalActiveRules: 1,- });- } catch (error) {- throwPublicRuleServiceError(error);- }-}--function throwPublicRuleServiceError(error: unknown): never {- if (error instanceof ActiveEvaluationRuleLimitError) {- throw createStructuredPublicApiError({- httpCode: 409,- code: "conflict",- message: error.message,- details: { limit: error.limit },- });- }- throw error;-}--async function runRuleServiceMutation<T>(mutation: () => Promise<T>) {- try {- return await mutation();- } catch (error) {- throwPublicRuleServiceError(error);- }-}--async function findPublicV2EvaluationRuleOrThrow(params: {- projectId: string;- evaluationRuleId: string;-}) {- const rule = await findPublicV2EvaluationRule(params);- if (!rule) {- throw new LangfuseNotFoundError(- "Evaluation rule not found within authorized project",- );- }- // A rule with no assignments is still a real, addressable resource: it must- // stay readable and deletable rather than 404 while consuming a slot.- return rule;-}--async function findPublicWritableV2EvaluationRuleOrThrow(params: {- projectId: string;- evaluationRuleId: string;-}) {- const rule = await findPublicV2EvaluationRuleOrThrow(params);- if (- rule.targetObject !== EvalTargetObject.EVENT &&- rule.targetObject !== EvalTargetObject.EXPERIMENT- ) {- throw new LangfuseNotFoundError(- "Evaluation rule not found within authorized project",- );- }- return rule;-}--function toApiReadableV2EvaluationRule(rule: StoredPublicV2EvaluationRule) {- const evaluationRule = toApiV2EvaluationRule(rule);- if (- evaluationRule.target !== "observation" &&- evaluationRule.target !== "experiment"- ) {- throw new InternalServerError("Evaluation rule target is corrupted");- }- return evaluationRule;-}--export async function listPublicEvaluationRules(params: {- projectId: string;- page: number;- limit: number;-}) {- const { records, totalItems } = await listPublicEvaluationRulePage(params);- return {- data: records.map(toApiV2EvaluationRule),- meta: {- page: params.page,- limit: params.limit,- totalItems,- totalPages: Math.ceil(totalItems / params.limit),- },- };-}--export async function getPublicEvaluationRule(params: {- projectId: string;- evaluationRuleId: string;-}) {- return toApiV2EvaluationRule(await findPublicV2EvaluationRuleOrThrow(params));-}--export async function createPublicEvaluationRule(params: {- orgId: string;- projectId: string;- input: PostUnstableEvaluationRuleBodyType;- evaluatorId?: string;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;-}) {- // Scoped to the targets this API can write. Rule names carry no uniqueness constraint, so this- // only exists to hand back the id to PATCH instead of a second rule -- and PATCH 404s on the- // trace/dataset rules the backfill brought in, so including those would be a dead end.- const existing = await prisma.evaluationRule.findFirst({- where: {- projectId: params.projectId,- name: params.input.name,- targetObject: {- in: [EvalTargetObject.EVENT, EvalTargetObject.EXPERIMENT],- },- },- select: { id: true },- });- if (existing) {- throw createStructuredPublicApiError({- httpCode: 409,- code: "name_conflict",- message: `An evaluation rule named "${params.input.name}" already exists in this project. Use PATCH /api/public/unstable/evaluation-rules/${existing.id} to update it instead of creating a duplicate.`,- details: { field: "name" },- });- }- if (params.input.enabled) {- await assertActivePublicApiEvaluationRuleLimitNotExceeded(params.projectId);- }- await assertEvaluationRuleFilterValuesExistForProject({- projectId: params.projectId,- target: params.input.target,- filters: params.input.filter,- });-- const requestedAssignments = getRequestedEvaluatorAssignments(params.input);- const evaluators = await Promise.all(- requestedAssignments.map(({ evaluator }, index) =>- resolveProjectEvaluator({- projectId: params.projectId,- evaluator,- evaluatorId:- params.evaluatorId && requestedAssignments.length === 1 && index === 0- ? params.evaluatorId- : undefined,- }),- ),- );- if (new Set(evaluators.map(({ id }) => id)).size !== evaluators.length) {- throw createStructuredPublicApiError({- httpCode: 409,- code: "conflict",- message: "An evaluator can only be attached once to an evaluation rule.",- });- }-- const preparedAssignments = requestedAssignments.map((assignment, index) => {- const evaluator = evaluators[index]!;- const template = toEvaluatorDefinition(evaluator);- const effectiveMapping =- assignment.mapping ??- (template.variableMapping == null- ? undefined- : toApiMappings(template.variableMapping));- const data = toEvaluationRuleInput({- input: {- name: params.input.name,- target: params.input.target,- enabled: params.input.enabled,- sampling: params.input.sampling,- filter: params.input.filter,- mapping: effectiveMapping,- },- evaluatorVariables: deriveEvaluatorVariables(template),- evaluatorType: toPublicEvaluatorType(template.type),- });- return { assignment, evaluator, template, data };- });-- await Promise.all(- preparedAssignments.map(({ evaluator, template, data }) =>- data.status === JobConfigState.ACTIVE- ? assertEvaluationRuleCanRunForPublicApi({- orgId: params.orgId,- projectId: params.projectId,- evaluatorId: evaluator.id,- template,- target: data.targetObject as EvalTargetObject,- mapping: data.variableMapping,- scoreName: data.scoreName,- filter: data.filter,- })- : Promise.resolve(),- ),- );-- const firstData = preparedAssignments[0]!.data;- const created = await createRuleWithRuleService({- projectId: params.projectId,- data: firstData,- assignments: preparedAssignments,- });- const evaluationRule = toApiWritableV2EvaluationRule(created);- if (params.auditScope) {- await auditLog({- action: "create",- resourceType: JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE,- resourceId: evaluationRule.id,- projectId: params.projectId,- orgId: params.auditScope.orgId,- apiKeyId: params.auditScope.apiKeyId,- after: evaluationRule,- });- }- return evaluationRule;-}--async function createRuleWithRuleService(params: {- projectId: string;- data: ReturnType<typeof toEvaluationRuleInput>;- assignments: Array<{- evaluator: PublicV2Evaluator;- assignment: { mapping?: PromptVariableMappingInputType[] };- data: ReturnType<typeof toEvaluationRuleInput>;- }>;-}) {- const created = await runRuleServiceMutation(() =>- ruleService.create(- {- projectId: params.projectId,- name: params.data.scoreName,- targetObject: params.data.targetObject,- enabled: params.data.status === JobConfigState.ACTIVE,- sampling: params.data.sampling,- filter: params.data.filter as FilterState,- evaluatorAssignments: params.assignments.map(- ({ assignment, evaluator, data }) => ({- evaluatorId: evaluator.id,- variableMapping:- assignment.mapping === undefined- ? null- : (data.variableMapping as ObservationVariableMapping[]),- }),- ),- },- null,- ),- );- return findPublicV2EvaluationRuleOrThrow({- projectId: params.projectId,- evaluationRuleId: created.id,- });-}--type PreparedAssignment = {- evaluatorId: string;- template: StoredPublicEvaluatorTemplate;- /** null stores "inherit the evaluator version's default mapping". */- storedMapping: unknown;- /** Mapping the rule would actually run with, for preflight validation. */- effectiveMapping: unknown;-};--function toWritableMappingsIfComplete(- mappings: PromptVariableMappingReadType[],-): PromptVariableMappingInputType[] | undefined {- const completeMappings: PromptVariableMappingInputType[] = [];- for (const mapping of mappings) {- if (mapping.source === null) return undefined;- completeMappings.push({ ...mapping, source: mapping.source });- }- return completeMappings;-}--/**- * Resolve one requested assignment (public shape) into its stored form.- * An omitted mapping inherits the evaluator version's default.- */-async function prepareAssignment(params: {- projectId: string;- target: PublicEvaluationRuleTargetType;- evaluator: EvaluationRuleEvaluatorFamilyReference;- mapping?: PromptVariableMappingInputType[];- evaluatorId?: string;-}): Promise<PreparedAssignment> {- const evaluator = await resolveProjectEvaluator({- projectId: params.projectId,- evaluator: params.evaluator,- evaluatorId: params.evaluatorId,- });- const template = toEvaluatorDefinition(evaluator);- const inherited =- template.variableMapping == null- ? undefined- : toApiMappings(template.variableMapping);- const storedMapping =- params.mapping === undefined- ? null- : toStoredAssignmentMapping({- target: params.target,- mapping: params.mapping,- evaluatorVariables: deriveEvaluatorVariables(template),- evaluatorType: toPublicEvaluatorType(template.type),- });- const effectiveMapping = toStoredAssignmentMapping({- target: params.target,- mapping: params.mapping ?? inherited,- evaluatorVariables: deriveEvaluatorVariables(template),- evaluatorType: toPublicEvaluatorType(template.type),- });- return {- evaluatorId: evaluator.id,- template,- storedMapping,- effectiveMapping,- };-}--function assertUniqueEvaluators(assignments: Array<{ evaluatorId: string }>) {- const ids = assignments.map(({ evaluatorId }) => evaluatorId);- if (new Set(ids).size !== ids.length) {- throw createStructuredPublicApiError({- httpCode: 409,- code: "conflict",- message: "An evaluator can only be attached once to an evaluation rule.",- });- }-}--/** Preflight every assignment the rule would run with once it is active. */-async function assertAssignmentsCanRun(params: {- orgId: string;- projectId: string;- assignments: PreparedAssignment[];- targetObject: EvalTargetObject;- scoreName: string;- filter: FilterCondition[] | null;-}) {- await Promise.all(- params.assignments.map((assignment) =>- assertEvaluationRuleCanRunForPublicApi({- orgId: params.orgId,- projectId: params.projectId,- evaluatorId: assignment.evaluatorId,- template: assignment.template,- target: params.targetObject,- mapping: assignment.effectiveMapping,- scoreName: params.scoreName,- filter: params.filter,- }),- ),- );-}--export async function updatePublicEvaluationRule(params: {- orgId: string;- projectId: string;- evaluationRuleId: string;- input: PatchUnstableEvaluationRuleBodyType;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;-}) {- const existing = await findPublicWritableV2EvaluationRuleOrThrow(params);- const existingPublic = toApiReadableV2EvaluationRule(existing);- const nextEnabled = params.input.enabled ?? existingPublic.enabled;- if (nextEnabled && existingPublic.status !== "active") {- await assertActivePublicApiEvaluationRuleLimitNotExceeded(params.projectId);- }- const suppliedTarget =- "target" in params.input ? params.input.target : undefined;- const suppliedFilter =- "filter" in params.input ? params.input.filter : undefined;- const nextTarget = suppliedTarget ?? existingPublic.target;- if (suppliedFilter !== undefined) {- await assertEvaluationRuleFilterValuesExistForProject({- projectId: params.projectId,- target: nextTarget,- filters: suppliedFilter,- });- }- const updatesFilterOrTarget =- suppliedFilter !== undefined ||- (suppliedTarget !== undefined && suppliedTarget !== existingPublic.target);- const parsedExistingFilter = PublicEvaluationRuleFilter.array().safeParse(- existingPublic.filter,- );- if (- updatesFilterOrTarget &&- suppliedFilter === undefined &&- !parsedExistingFilter.success- ) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_body",- message:- "The stored filter is no longer accepted for writes. Provide a target-compatible filter with this update.",- details: { field: "filter" },- });- }- const ruleFields = toEvaluationRuleFields({- name: params.input.name ?? existingPublic.name,- target: nextTarget,- enabled: nextEnabled,- sampling: params.input.sampling ?? existingPublic.sampling,- // Target changes must also validate the effective filter against the new target.- filter: updatesFilterOrTarget- ? (suppliedFilter ?? parsedExistingFilter.data ?? [])- : [],- });-- const firstAssignment = existing.assignments[0];- const patchesFirstAssignment =- params.input.evaluator !== undefined ||- ("mapping" in params.input && params.input.mapping !== undefined);-- if (- "mapping" in params.input &&- params.input.mapping !== undefined &&- firstAssignment?.evaluator.type === EvalTemplateType.CODE- ) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_body",- message:- "Code evaluator mappings are managed by Langfuse and cannot be provided in the request body.",- details: { field: "mapping" },- });- }- if (patchesFirstAssignment && !firstAssignment && !params.input.evaluator) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_body",- message:- "This evaluation rule has no evaluator assignments, so there is no mapping to update. Provide `evaluators` to attach one.",- details: { field: "mapping" },- });- }-- // Three shapes: replace the whole set, patch the first assignment (deprecated- // aliases), or leave assignments untouched.- const replacementAssignments = params.input.evaluators- ? await Promise.all(- params.input.evaluators.map((assignment) =>- prepareAssignment({- projectId: params.projectId,- target: nextTarget,- evaluator: assignment.evaluator,- mapping: assignment.mapping,- }),- ),- )- : null;- if (replacementAssignments) assertUniqueEvaluators(replacementAssignments);-- const patchedFirstAssignment =- replacementAssignments || !patchesFirstAssignment- ? null- : await prepareAssignment({- projectId: params.projectId,- target: nextTarget,- evaluator: params.input.evaluator- ? {- ...params.input.evaluator,- // No current assignment to inherit a type from: the reference- // schema's own default applies.- type:- existingPublic.evaluator?.type ??- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- }- : {- name: existingPublic.evaluator!.name,- type: existingPublic.evaluator!.type,- },- mapping:- "mapping" in params.input && params.input.mapping !== undefined- ? params.input.mapping- : toWritableMappingsIfComplete(existingPublic.mapping),- });- if (patchedFirstAssignment) {- assertUniqueEvaluators([- patchedFirstAssignment,- ...existing.assignments.slice(1),- ]);- }-- // Assignments the rule ends up running with, whether or not this call rewrites them.- const effectiveAssignments: PreparedAssignment[] = replacementAssignments ?? [- ...(patchedFirstAssignment- ? [patchedFirstAssignment]- : firstAssignment- ? [toPreparedExistingAssignment(firstAssignment, nextTarget)]- : []),- ...existing.assignments- .slice(1)- .map((assignment) =>- toPreparedExistingAssignment(assignment, nextTarget),- ),- ];-- if (ruleFields.status === JobConfigState.ACTIVE) {- await assertAssignmentsCanRun({- orgId: params.orgId,- projectId: params.projectId,- assignments: effectiveAssignments,- targetObject: ruleFields.targetObject,- scoreName: ruleFields.scoreName,- filter: updatesFilterOrTarget- ? ruleFields.filter- : (existing.filter as FilterCondition[]),- });- }-- const updated = await updateRuleWithRuleService({- projectId: params.projectId,- evaluationRuleId: params.evaluationRuleId,- input: params.input,- fields: ruleFields,- existing,- replacementAssignments,- patchedFirstAssignment,- effectiveAssignmentCount: effectiveAssignments.length,- updatesFilterOrTarget,- });- const evaluationRule = toApiReadableV2EvaluationRule(updated);- if (params.auditScope) {- await auditLog({- action: "update",- resourceType: JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE,- resourceId: evaluationRule.id,- projectId: params.projectId,- orgId: params.auditScope.orgId,- apiKeyId: params.auditScope.apiKeyId,- before: existingPublic,- after: evaluationRule,- });- }- return evaluationRule;-}--function getPatchedFirstStoredMapping(params: {- input: PatchUnstableEvaluationRuleBodyType;- firstAssignment:- | StoredPublicV2EvaluationRule["assignments"][number]- | undefined;- patchedFirstAssignment: PreparedAssignment;-}) {- return !("mapping" in params.input) || params.input.mapping === undefined- ? // Evaluator swapped without an explicit mapping: keep the stored- // override so an inherited mapping stays inherited.- (params.firstAssignment?.variableMapping ?? null)- : params.patchedFirstAssignment.storedMapping;-}--async function updateRuleWithRuleService(params: {- projectId: string;- evaluationRuleId: string;- input: PatchUnstableEvaluationRuleBodyType;- fields: ReturnType<typeof toEvaluationRuleFields>;- existing: StoredPublicV2EvaluationRule;- replacementAssignments: PreparedAssignment[] | null;- patchedFirstAssignment: PreparedAssignment | null;- effectiveAssignmentCount: number;- updatesFilterOrTarget: boolean;-}) {- const firstAssignment = params.existing.assignments[0];- const evaluatorMappings = params.replacementAssignments- ? params.replacementAssignments.map((assignment) => ({- evaluatorId: assignment.evaluatorId,- variableMapping: assignment.storedMapping as- | ObservationVariableMapping[]- | null,- }))- : params.patchedFirstAssignment- ? [- {- evaluatorId: params.patchedFirstAssignment.evaluatorId,- variableMapping: getPatchedFirstStoredMapping({- input: params.input,- firstAssignment,- patchedFirstAssignment: params.patchedFirstAssignment,- }) as ObservationVariableMapping[] | null,- },- ...params.existing.assignments.slice(1).map((assignment) => ({- evaluatorId: assignment.evaluatorId,- variableMapping: assignment.variableMapping as- | ObservationVariableMapping[]- | null,- })),- ]- : undefined;-- await runRuleServiceMutation(() =>- ruleService.update({- projectId: params.projectId,- ruleId: params.evaluationRuleId,- targetObject:- params.updatesFilterOrTarget && "target" in params.input- ? params.fields.targetObject- : undefined,- name: params.input.name,- enabled:- params.input.enabled === undefined- ? undefined- : params.effectiveAssignmentCount > 0 &&- params.fields.status === JobConfigState.ACTIVE,- sampling: params.input.sampling,- filter: params.updatesFilterOrTarget- ? (params.fields.filter as FilterState)- : undefined,- evaluatorMappings,- }),- );- return findPublicV2EvaluationRuleOrThrow({- projectId: params.projectId,- evaluationRuleId: params.evaluationRuleId,- });-}--/**- * Existing stored assignment in the shape the preflight needs, resolving an- * inherited (null) override against the evaluator version's default mapping.- */-function toPreparedExistingAssignment(- assignment: StoredPublicV2EvaluationRule["assignments"][number],- target: PublicEvaluationRuleTargetType,-): PreparedAssignment {- const template = toEvaluatorDefinition(assignment.evaluator);- const inherited = assignment.variableMapping ?? template.variableMapping;- return {- evaluatorId: assignment.evaluatorId,- template,- storedMapping: assignment.variableMapping,- effectiveMapping: toStoredAssignmentMapping({- target,- mapping:- template.type === EvalTemplateType.CODE || inherited == null- ? undefined- : toApiMappings(inherited),- evaluatorVariables: deriveEvaluatorVariables(template),- evaluatorType: toPublicEvaluatorType(template.type),- }),- };-}--export async function deletePublicEvaluationRule(params: {- projectId: string;- evaluationRuleId: string;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;-}) {- const existing = await findPublicWritableV2EvaluationRuleOrThrow(params);- const existingPublic = toApiReadableV2EvaluationRule(existing);- await runRuleServiceMutation(() =>- ruleService.delete(params.projectId, params.evaluationRuleId),- );- if (params.auditScope) {- await auditLog({- action: "delete",- resourceType: JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE,- resourceId: params.evaluationRuleId,- projectId: params.projectId,- orgId: params.auditScope.orgId,- apiKeyId: params.auditScope.apiKeyId,- before: existingPublic,- });- }- return existing;-}diff --git a/web/src/features/evals/server/unstable-public-api/evaluator-service.ts b/web/src/features/evals/server/unstable-public-api/evaluator-service.tsdeleted file mode 100644index 53b79e411a..0000000000--- a/web/src/features/evals/server/unstable-public-api/evaluator-service.ts+++ /dev/null@@ -1,296 +0,0 @@-import {- EvalTemplateType,- extractVariables,- LangfuseConflictError,-} from "@langfuse/shared";-import { prisma } from "@langfuse/shared/src/db";-import {- invalidateProjectEvalConfigCaches,- type ApiAccessScope,-} from "@langfuse/shared/src/server";-import { auditLog } from "@/src/features/audit-logs/auditLog";-import { EVAL_TEMPLATE_AUDIT_LOG_RESOURCE_TYPE } from "@/src/features/evals/server/audit-log-resource-types";-import {- isCodeEvalEnabled,- isCodeEvalSourceCodeLanguageSupported,-} from "@/src/features/evals/server/isCodeEvalEnabled";-import type { EvaluatorDefinition } from "@/src/features/evals/v2/server/evaluators/evaluatorTypes";-import {- EvaluatorService,- reconcileEvaluatorPromptMessages,-} from "@/src/features/evals/v2/server/evaluators/evaluatorService";-import {- EvaluatorConfigurationError,- EvaluatorVersionConflictError,-} from "@/src/features/evals/v2/server/evaluators/evaluatorErrors";-import { RuleService } from "@/src/features/evals/v2/server/rules/ruleService";-import { createStructuredPublicApiError } from "@/src/features/public-api";-import {- type PostUnstableEvaluatorBodyParsedType,- PUBLIC_EVALUATOR_TYPE_CODE,-} from "@/src/features/public-api/server";-import {- toApiEvaluator,- toStoredOutputDefinition,- toStoredVariableMappings,-} from "./adapters";-import type { StoredPublicEvaluatorTemplate } from "./types";--function readServiceForPublicApi() {- return new EvaluatorService(prisma, async () => undefined);-}--function serviceForPublicApi(params: {- projectId: string;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;-}) {- return new EvaluatorService(prisma, ({ action, evaluatorId }) =>- auditPublicEvaluator(params, action, evaluatorId),- );-}--function ruleServiceForPublicApi() {- return new RuleService(prisma, async () => undefined);-}--function auditPublicEvaluator(- params: {- projectId: string;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;- },- action: "create" | "update" | "delete",- evaluatorId: string,-) {- if (!params.auditScope) return Promise.resolve();- return auditLog({- action,- resourceType: EVAL_TEMPLATE_AUDIT_LOG_RESOURCE_TYPE,- resourceId: evaluatorId,- projectId: params.projectId,- orgId: params.auditScope.orgId,- apiKeyId: params.auditScope.apiKeyId,- });-}--function assertCodeEvaluatorDefinitionCanRunForPublicApi(- input: Extract<- PostUnstableEvaluatorBodyParsedType,- { type: typeof PUBLIC_EVALUATOR_TYPE_CODE }- >,-) {- if (!isCodeEvalEnabled()) {- throw createStructuredPublicApiError({- httpCode: 403,- code: "access_denied",- message: "Code evals are not enabled",- });- }-- if (!isCodeEvalSourceCodeLanguageSupported(input.sourceCodeLanguage)) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_request",- message:- "This code evaluator language is not supported by the configured dispatcher.",- details: {- field: "sourceCodeLanguage",- value: input.sourceCodeLanguage,- },- });- }-}--function toDefinition(- input: PostUnstableEvaluatorBodyParsedType,-): EvaluatorDefinition {- if (input.type === PUBLIC_EVALUATOR_TYPE_CODE) {- return {- type: EvalTemplateType.CODE,- sourceCode: input.sourceCode,- sourceCodeLanguage: input.sourceCodeLanguage,- };- }-- return {- type: EvalTemplateType.LLM_AS_JUDGE,- promptMessages: reconcileEvaluatorPromptMessages({ prompt: input.prompt }),- provider: input.modelConfig?.provider ?? null,- model: input.modelConfig?.model ?? null,- modelParams: null,- vars: extractVariables(input.prompt),- variableMapping:- input.mapping === undefined- ? null- : toStoredVariableMappings({- mappings: input.mapping,- variables: extractVariables(input.prompt),- target: "observation",- }),- outputDefinition: toStoredOutputDefinition(input.outputDefinition),- };-}--type StableEvaluator =- | Awaited<ReturnType<EvaluatorService["get"]>>- | Awaited<ReturnType<EvaluatorService["list"]>>["evaluators"][number];--function toLatestTemplate(- evaluator: StableEvaluator,-): StoredPublicEvaluatorTemplate {- const version = evaluator.versions[0];- if (!version) {- throw createStructuredPublicApiError({- httpCode: 500,- code: "internal_error",- message: "Evaluator version is missing",- });- }-- return {- id: evaluator.id,- projectId: evaluator.projectId,- name: evaluator.name,- version: version.version,- type: evaluator.type,- prompt: null,- promptMessages: version.promptMessages,- partner: version.partner,- provider: version.provider,- model: version.model,- modelParams: version.modelParams,- vars: version.vars,- outputDefinition: version.outputDefinition,- sourceCode: version.sourceCode,- sourceCodeLanguage: version.sourceCodeLanguage,- variableMapping: version.variableMapping,- createdAt: evaluator.createdAt,- updatedAt: evaluator.updatedAt,- };-}--function toPublicEvaluator(- evaluator: StableEvaluator,- evaluationRuleCount = 0,-) {- return toApiEvaluator({- template: toLatestTemplate(evaluator),- evaluationRuleCount,- });-}--export async function listPublicEvaluators(params: {- projectId: string;- page: number;- limit: number;-}) {- const service = readServiceForPublicApi();- const ruleService = ruleServiceForPublicApi();- const { evaluators, totalItems } = await service.list(params);- const evaluatorIds = evaluators.map((evaluator) => evaluator.id);- const assignmentCounts = await ruleService.countRulesForEvaluators(- params.projectId,- evaluatorIds,- );- return {- data: evaluators.map((evaluator) =>- toPublicEvaluator(evaluator, assignmentCounts[evaluator.id] ?? 0),- ),- meta: {- page: params.page,- limit: params.limit,- totalItems,- totalPages: Math.ceil(totalItems / params.limit),- },- };-}--export async function getPublicEvaluator(params: {- projectId: string;- evaluatorId: string;-}) {- const service = readServiceForPublicApi();- const evaluator = await service.get(params.projectId, params.evaluatorId);- const ruleCounts = await ruleServiceForPublicApi().countRulesForEvaluators(- params.projectId,- [evaluator.id],- );- return toPublicEvaluator(evaluator, ruleCounts[evaluator.id] ?? 0);-}--export async function createPublicEvaluator(params: {- projectId: string;- input: PostUnstableEvaluatorBodyParsedType;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;-}) {- const { input } = params;- const definition = toDefinition(input);-- if (input.type === PUBLIC_EVALUATOR_TYPE_CODE) {- assertCodeEvaluatorDefinitionCanRunForPublicApi(input);- }-- const service = serviceForPublicApi(params);- try {- const result = await service.upsertByName(- {- projectId: params.projectId,- name: input.name,- description: null,- definition,- },- null,- );- const { evaluator } = result;- const ruleCounts = await ruleServiceForPublicApi().countRulesForEvaluators(- params.projectId,- [evaluator.id],- );- return toPublicEvaluator(evaluator, ruleCounts[evaluator.id] ?? 0);- } catch (error) {- if (error instanceof EvaluatorConfigurationError) {- throw createStructuredPublicApiError({- httpCode: 422,- code: "evaluator_preflight_failed",- message: error.message,- details: {- evaluatorName: input.name,- provider:- input.type === PUBLIC_EVALUATOR_TYPE_CODE- ? null- : (input.modelConfig?.provider ?? null),- model:- input.type === PUBLIC_EVALUATOR_TYPE_CODE- ? null- : (input.modelConfig?.model ?? null),- },- });- }- if (error instanceof EvaluatorVersionConflictError) {- throw createStructuredPublicApiError({- httpCode: 409,- code: "conflict",- message: error.message,- });- }- if (error instanceof LangfuseConflictError) {- throw createStructuredPublicApiError({- httpCode: 409,- code: "name_conflict",- message: error.message,- details: { field: "name" },- });- }- throw error;- }-}--export async function deletePublicEvaluator(params: {- projectId: string;- evaluatorId: string;- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;-}) {- const service = serviceForPublicApi(params);- await service.get(params.projectId, params.evaluatorId);- await service.delete(params.projectId, params.evaluatorId);- await invalidateProjectEvalConfigCaches(params.projectId);-}diff --git a/web/src/features/evals/server/unstable-public-api/index.ts b/web/src/features/evals/server/unstable-public-api/index.tsdeleted file mode 100644index efec33996f..0000000000--- a/web/src/features/evals/server/unstable-public-api/index.ts+++ /dev/null@@ -1,13 +0,0 @@-export {- createPublicEvaluationRule,- deletePublicEvaluationRule,- getPublicEvaluationRule,- listPublicEvaluationRules,- updatePublicEvaluationRule,-} from "./evaluation-rule-service";-export {- createPublicEvaluator,- deletePublicEvaluator,- getPublicEvaluator,- listPublicEvaluators,-} from "./evaluator-service";diff --git a/web/src/features/evals/server/unstable-public-api/queries.ts b/web/src/features/evals/server/unstable-public-api/queries.tsdeleted file mode 100644index 8a291ad97f..0000000000--- a/web/src/features/evals/server/unstable-public-api/queries.ts+++ /dev/null@@ -1,125 +0,0 @@-import { EvalTargetObject, LangfuseNotFoundError } from "@langfuse/shared";-import { prisma } from "@langfuse/shared/src/db";-import type {- EvaluationRuleEvaluatorFamilyReference,- PrismaClientLike,- StoredPublicV2EvaluationRule,-} from "./types";-import { toStoredEvaluatorType } from "./adapters";--function getPrismaClient(client?: PrismaClientLike) {- return client ?? prisma;-}--const publicV2RuleInclude = (projectId: string) => ({- assignments: {- where: { projectId, evaluator: { projectId } },- orderBy: [{ createdAt: "asc" as const }, { id: "asc" as const }],- include: {- evaluator: {- include: {- versions: { orderBy: { version: "desc" as const }, take: 1 },- },- },- },- },-});--export async function findPublicV2EvaluationRule(params: {- client?: PrismaClientLike;- projectId: string;- evaluationRuleId: string;-}) {- const client = getPrismaClient(params.client);- return client.evaluationRule.findFirst({- where: { id: params.evaluationRuleId, projectId: params.projectId },- include: publicV2RuleInclude(params.projectId),- }) as Promise<StoredPublicV2EvaluationRule | null>;-}--export async function findPublicV2EvaluatorInFamily(params: {- client?: PrismaClientLike;- projectId: string;- evaluator: EvaluationRuleEvaluatorFamilyReference;-}) {- const client = getPrismaClient(params.client);- const evaluator = await client.evaluator.findFirst({- where: {- projectId: params.projectId,- name: params.evaluator.name,- type: toStoredEvaluatorType(params.evaluator.type),- },- include: { versions: { orderBy: { version: "desc" }, take: 1 } },- });- return evaluator?.versions.length ? evaluator : null;-}--export async function findPublicV2EvaluatorById(params: {- client?: PrismaClientLike;- projectId: string;- evaluatorId: string;-}) {- const client = getPrismaClient(params.client);- const evaluator = await client.evaluator.findFirst({- where: { id: params.evaluatorId, projectId: params.projectId },- include: { versions: { orderBy: { version: "desc" }, take: 1 } },- });- return evaluator?.versions.length ? evaluator : null;-}--export async function findPublicV2EvaluatorByIdOrThrow(- params: Parameters<typeof findPublicV2EvaluatorById>[0],-) {- const evaluator = await findPublicV2EvaluatorById(params);- if (!evaluator) {- throw new LangfuseNotFoundError(- "Latest evaluator version not found within authorized project",- );- }- return evaluator;-}--export async function findPublicV2EvaluatorInFamilyOrThrow(- params: Parameters<typeof findPublicV2EvaluatorInFamily>[0],-) {- const evaluator = await findPublicV2EvaluatorInFamily(params);- if (!evaluator) {- throw new LangfuseNotFoundError(- "Latest evaluator version not found within authorized project",- );- }- return evaluator;-}--export async function listPublicEvaluationRulePage(params: {- projectId: string;- page: number;- limit: number;-}) {- const where = {- projectId: params.projectId,- targetObject: {- in: [- EvalTargetObject.EVENT,- EvalTargetObject.EXPERIMENT,- EvalTargetObject.TRACE,- EvalTargetObject.DATASET,- ],- },- };- const [records, totalItems] = await Promise.all([- prisma.evaluationRule.findMany({- where,- include: publicV2RuleInclude(params.projectId),- orderBy: [{ createdAt: "desc" }, { id: "desc" }],- take: params.limit,- skip: (params.page - 1) * params.limit,- }),- prisma.evaluationRule.count({ where }),- ]);-- return {- records: records as StoredPublicV2EvaluationRule[],- totalItems,- };-}diff --git a/web/src/features/evals/server/unstable-public-api/types.ts b/web/src/features/evals/server/unstable-public-api/types.tsdeleted file mode 100644index 067c6d34e2..0000000000--- a/web/src/features/evals/server/unstable-public-api/types.ts+++ /dev/null@@ -1,141 +0,0 @@-import type {- EvaluationRule,- EvaluationRuleEvaluatorAssignment,- Evaluator,- EvaluatorVersion,- Prisma as PrismaNamespace,- prisma,-} from "@langfuse/shared/src/db";-import type {- PublicEvaluationRuleEvaluatorReferenceType,- PublicEvaluationRuleEvaluatorType,- PublicEvaluationRuleFilterType,- PublicEvaluationRuleReadFilterType,- PromptVariableMappingReadType,- PublicEvaluationRuleStatusType,- PublicEvaluationRuleTargetType,- LegacyPromptVariableMappingType,- PublicEvaluationRuleLegacyTargetType,- PublicEvaluatorModelConfigType,- PublicEvaluatorOutputDefinitionType,- PublicCodeEvaluatorSourceCodeLanguageType,- PUBLIC_EVALUATOR_TYPE_CODE,- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,-} from "@/src/features/public-api/server";-import type {- CODE_EVAL_TEMPLATE_VARIABLES,- FilterCondition,- JobTimeScope,-} from "@langfuse/shared";--export type PrismaClientLike =- | typeof prisma- | PrismaNamespace.TransactionClient;--type ApiEvaluatorRecordBase = {- id: string;- name: string;- version: number;- variables: string[];- // Read shape: an evaluator default can be incomplete or name experiment-only sources.- mapping: PromptVariableMappingReadType[] | null;- evaluationRuleCount: number;- createdAt: Date;- updatedAt: Date;-};--export type StoredPublicV2EvaluationRule = EvaluationRule & {- assignments: Array<- EvaluationRuleEvaluatorAssignment & {- evaluator: Evaluator & { versions: EvaluatorVersion[] };- }- >;-};--type ApiLlmAsJudgeEvaluatorRecord = ApiEvaluatorRecordBase & {- type: typeof PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE;- prompt: string;- outputDefinition: PublicEvaluatorOutputDefinitionType;- modelConfig: PublicEvaluatorModelConfigType | null;-};--type ApiCodeEvaluatorRecord = ApiEvaluatorRecordBase & {- type: typeof PUBLIC_EVALUATOR_TYPE_CODE;- variables: Array<(typeof CODE_EVAL_TEMPLATE_VARIABLES)[number]>;- sourceCode: string;- sourceCodeLanguage: PublicCodeEvaluatorSourceCodeLanguageType;-};--export type ApiEvaluatorRecord =- | ApiLlmAsJudgeEvaluatorRecord- | ApiCodeEvaluatorRecord;--type ApiEvaluationRuleRecordBase = {- id: string;- name: string;- evaluator: PublicEvaluationRuleEvaluatorType | null;- enabled: boolean;- status: PublicEvaluationRuleStatusType;- pausedReason: string | null;- pausedMessage: string | null;- sampling: number;- createdAt: Date;- updatedAt: Date;-};--export type ApiWritableEvaluationRuleRecord = ApiEvaluationRuleRecordBase & {- evaluators: Array<{- evaluator: PublicEvaluationRuleEvaluatorType;- mapping: PromptVariableMappingReadType[] | null;- }>;- target: PublicEvaluationRuleTargetType;- filter: PublicEvaluationRuleFilterType[];- mapping: PromptVariableMappingReadType[];-};--type ApiLegacyEvaluationRuleRecord = ApiEvaluationRuleRecordBase & {- evaluators: Array<{- evaluator: PublicEvaluationRuleEvaluatorType;- mapping: LegacyPromptVariableMappingType[] | null;- }>;- target: PublicEvaluationRuleLegacyTargetType;- delay: number;- timeScope: JobTimeScope[];- filter: FilterCondition[];- mapping: LegacyPromptVariableMappingType[];-};--type ApiReadableV2EvaluationRuleRecord = Omit<- ApiWritableEvaluationRuleRecord,- "filter"-> & {- filter: PublicEvaluationRuleReadFilterType[];-};--export type ApiEvaluationRuleRecord =- | ApiReadableV2EvaluationRuleRecord- | ApiLegacyEvaluationRuleRecord;--export type EvaluationRuleEvaluatorFamilyReference =- PublicEvaluationRuleEvaluatorReferenceType;--export type StoredPublicEvaluatorTemplate = Pick<- Evaluator,- "id" | "projectId" | "name" | "type" | "createdAt" | "updatedAt"-> &- Pick<- EvaluatorVersion,- | "version"- | "prompt"- | "partner"- | "provider"- | "model"- | "modelParams"- | "vars"- | "outputDefinition"- | "sourceCode"- | "sourceCodeLanguage"- | "variableMapping"- > & {- promptMessages?: EvaluatorVersion["promptMessages"];- };diff --git a/web/src/features/evals/server/unstable-public-api/validation.ts b/web/src/features/evals/server/unstable-public-api/validation.tsdeleted file mode 100644index 3d10ab35cc..0000000000--- a/web/src/features/evals/server/unstable-public-api/validation.ts+++ /dev/null@@ -1,393 +0,0 @@-import {- experimentEvalFilterColumns,- observationEvalFilterColumns,-} from "@langfuse/shared";-import { prisma } from "@langfuse/shared/src/db";-import { JSONPath } from "jsonpath-plus";-import { createStructuredPublicApiError } from "@/src/features/public-api";-import {- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS,- ExperimentPromptVariableMappingSource,- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,- ObservationPromptVariableMappingSource,- type PublicEvaluationRuleFilterType,- type PromptVariableMappingInputType,- type PublicEvaluationRuleTargetType,-} from "@/src/features/public-api/server";-import { getEvaluatorDefinitionPreflightError } from "@/src/features/evals/server/evaluator-preflight";--// Derived from the same column lists as the request schemas, so this runtime-// check cannot drift from the documented contract.-const toStaticFilterOptions = (columns: typeof observationEvalFilterColumns) =>- new Map(- columns.flatMap((column) => {- if (!("options" in column) || !Array.isArray(column.options)) {- return [];- }-- if (column.options.length === 0) {- return [];- }-- return [- [- column.id === "experimentDatasetId" ? "datasetId" : column.id,- new Set(- column.options.flatMap((option) =>- "value" in option ? [String(option.value)] : option.values,- ),- ),- ] as const,- ];- }),- );--const STATIC_FILTER_OPTIONS_BY_TARGET = {- observation: toStaticFilterOptions(observationEvalFilterColumns),- experiment: toStaticFilterOptions([- ...observationEvalFilterColumns,- ...experimentEvalFilterColumns,- ]),-} as const satisfies Record<- PublicEvaluationRuleTargetType,- Map<string, Set<string>>->;--const SUPPORTED_FILTER_COLUMNS_BY_TARGET = {- observation: new Set(- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS.map((column) => column.id),- ),- experiment: new Set(- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS.map((column) => column.id),- ),-} as const satisfies Record<PublicEvaluationRuleTargetType, Set<string>>;--// Derived from the per-target request schemas so this runtime check (which-// also guards paths that parse the target-agnostic mapping union) can never-// drift from the documented contract.-const SUPPORTED_MAPPING_SOURCES_BY_TARGET = {- observation: new Set<string>(ObservationPromptVariableMappingSource.options),- experiment: new Set<string>(ExperimentPromptVariableMappingSource.options),-} as const satisfies Record<PublicEvaluationRuleTargetType, Set<string>>;--export function validateEvaluationRuleFilters(params: {- target: PublicEvaluationRuleTargetType;- filters: PublicEvaluationRuleFilterType[];-}) {- const knownOptionValues = STATIC_FILTER_OPTIONS_BY_TARGET[params.target];- const supportedColumns = SUPPORTED_FILTER_COLUMNS_BY_TARGET[params.target];-- for (const [filterIndex, filter] of params.filters.entries()) {- if (!supportedColumns.has(filter.column)) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_filter_value",- message: `Filter column "${filter.column}" is not supported for target "${params.target}"`,- details: {- field: `filter[${filterIndex}].column`,- column: filter.column,- allowedValues: Array.from(supportedColumns),- },- });- }-- const allowedValues = knownOptionValues.get(filter.column);-- if (- !allowedValues ||- !("value" in filter) ||- !Array.isArray(filter.value)- ) {- continue;- }-- const invalidValues = filter.value.filter(- (value) => !allowedValues.has(value),- );-- if (invalidValues.length > 0) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_filter_value",- message: `Filter column "${filter.column}" contains unsupported value(s): ${invalidValues.join(", ")}`,- details: {- field: `filter[${filterIndex}].value`,- column: filter.column,- invalidValues,- allowedValues: Array.from(allowedValues),- },- });- }- }-}--export async function assertEvaluationRuleFilterValuesExistForProject(params: {- projectId: string;- target: PublicEvaluationRuleTargetType;- filters: PublicEvaluationRuleFilterType[];-}) {- if (params.target !== "experiment") {- return;- }-- const datasetFilters = params.filters.flatMap((filter, filterIndex) => {- if (- filter.column !== "datasetId" ||- !("value" in filter) ||- !Array.isArray(filter.value)- ) {- return [];- }-- return [{ filterIndex, values: filter.value }] as const;- });-- if (datasetFilters.length === 0) {- return;- }-- const requestedDatasetIds = Array.from(- new Set(datasetFilters.flatMap((filter) => filter.values)),- );- const existingDatasets = await prisma.dataset.findMany({- where: {- projectId: params.projectId,- id: {- in: requestedDatasetIds,- },- },- select: {- id: true,- },- });- const existingDatasetIds = new Set(- existingDatasets.map((dataset) => dataset.id),- );-- for (const filter of datasetFilters) {- const invalidValues = filter.values.filter(- (value) => !existingDatasetIds.has(value),- );-- if (invalidValues.length > 0) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_filter_value",- message: `Filter column "datasetId" contains dataset id(s) that do not exist in this project: ${invalidValues.join(", ")}`,- details: {- field: `filter[${filter.filterIndex}].value`,- column: "datasetId",- invalidValues,- },- });- }- }-}--function validateJsonPath(params: {- jsonPath: string;- variable: string;- mappingIndex: number;-}) {- const { jsonPath, variable, mappingIndex } = params;-- if (!jsonPath.startsWith("$")) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_json_path",- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". JSONPath expressions must start with "$".`,- details: {- field: `mapping[${mappingIndex}].jsonPath`,- variable,- value: jsonPath,- },- });- }-- let openQuote: "'" | '"' | null = null;- let isEscaped = false;- const delimiterStack: Array<"[" | "("> = [];-- for (const character of jsonPath) {- if (openQuote) {- if (isEscaped) {- isEscaped = false;- continue;- }-- if (character === "\\") {- isEscaped = true;- continue;- }-- if (character === openQuote) {- openQuote = null;- }-- continue;- }-- if (character === "'" || character === '"') {- openQuote = character;- continue;- }-- if (character === "[" || character === "(") {- delimiterStack.push(character);- continue;- }-- if (character === "]" || character === ")") {- const expectedDelimiter = character === "]" ? "[" : "(";-- if (delimiterStack.pop() !== expectedDelimiter) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_json_path",- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". JSONPath expressions must use balanced brackets and parentheses.`,- details: {- field: `mapping[${mappingIndex}].jsonPath`,- variable,- value: jsonPath,- },- });- }- }- }-- if (openQuote || delimiterStack.length > 0) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_json_path",- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". JSONPath expressions must use balanced quotes, brackets, and parentheses.`,- details: {- field: `mapping[${mappingIndex}].jsonPath`,- variable,- value: jsonPath,- },- });- }-- try {- JSONPath({- path: jsonPath,- json: {},- eval: false,- });- } catch (error) {- const message = error instanceof Error ? error.message : "Unknown error";- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_json_path",- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". ${message}`,- details: {- field: `mapping[${mappingIndex}].jsonPath`,- variable,- value: jsonPath,- },- });- }-}--export function validateEvaluatorVariableMappings(params: {- mappings: PromptVariableMappingInputType[];- variables: string[];- target: PublicEvaluationRuleTargetType;-}) {- const variableSet = new Set(params.variables);- const mappedVariables = new Set<string>();- const allowedSources = SUPPORTED_MAPPING_SOURCES_BY_TARGET[params.target];-- for (const [mappingIndex, mapping] of params.mappings.entries()) {- if (!allowedSources.has(mapping.source)) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_variable_mapping",- message: `Mapping source "${mapping.source}" is not supported for target "${params.target}"`,- details: {- field: `mapping[${mappingIndex}].source`,- variable: mapping.variable,- allowedValues: Array.from(allowedSources),- },- });- }-- if (!variableSet.has(mapping.variable)) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "invalid_variable_mapping",- message: `Mapping variable "${mapping.variable}" is not present in the evaluator prompt`,- details: {- field: `mapping[${mappingIndex}].variable`,- variable: mapping.variable,- },- });- }-- if (mappedVariables.has(mapping.variable)) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "duplicate_variable_mapping",- message: `Mapping variable "${mapping.variable}" can only be mapped once`,- details: {- field: "mapping",- variable: mapping.variable,- },- });- }-- mappedVariables.add(mapping.variable);- }-- const missingVariables = params.variables.filter(- (variable) => !mappedVariables.has(variable),- );-- if (missingVariables.length > 0) {- throw createStructuredPublicApiError({- httpCode: 400,- code: "missing_variable_mapping",- message: `Missing mappings for evaluator variables: ${missingVariables.join(", ")}`,- details: {- field: "mapping",- variables: missingVariables,- },- });- }-- for (const [mappingIndex, mapping] of params.mappings.entries()) {- if (mapping.jsonPath) {- validateJsonPath({- jsonPath: mapping.jsonPath,- variable: mapping.variable,- mappingIndex,- });- }- }-}--export async function assertEvaluatorDefinitionCanRunForPublicApi(params: {- projectId: string;- template: {- name: string;- provider?: string | null;- model?: string | null;- modelParams?: unknown;- outputDefinition: unknown;- };-}) {- const error = await getEvaluatorDefinitionPreflightError(params);-- if (error) {- throw createStructuredPublicApiError({- httpCode: 422,- code: "evaluator_preflight_failed",- message: error,- details: {- evaluatorName: params.template.name,- provider: params.template.provider ?? null,- model: params.template.model ?? null,- },- });- }-}diff --git a/web/src/features/mcp/server/evals/rule-schema.ts b/web/src/features/mcp/server/evals/rule-schema.tsindex 4b4ef80005..6a3cd074ce 100644--- a/web/src/features/mcp/server/evals/rule-schema.ts+++ b/web/src/features/mcp/server/evals/rule-schema.ts@@ -12,7 +12,7 @@ import {ObservationPromptVariableMappingInput,PromptVariableMappingRead,PublicEvaluatorType,-} from "@/src/features/public-api/server";+} from "@/src/features/public-api/types/evaluation/publicEvalsContract";import { McpAdvancedFilterBaseSchema } from "../../core/filter-schema";const RuleFilterBaseSchema = McpAdvancedFilterBaseSchema.describe(diff --git a/web/src/features/mcp/server/evals/rule-service.ts b/web/src/features/mcp/server/evals/rule-service.tsindex aec4809a7b..ef26147a58 100644--- a/web/src/features/mcp/server/evals/rule-service.ts+++ b/web/src/features/mcp/server/evals/rule-service.ts@@ -4,7 +4,7 @@ import {toApiReadMappings,toPublicEvaluatorType,toStoredMappingList,-} from "@/src/features/evals/server/unstable-public-api/adapters";+} from "@/src/features/public-api/server/evaluation/evaluationAdapters";import { RuleService } from "@/src/features/evals/v2/server/rules/ruleService";import { prisma } from "@langfuse/shared/src/db";import type { z } from "zod";diff --git a/web/src/features/public-api/server/evaluation/evaluationAdapters.ts b/web/src/features/public-api/server/evaluation/evaluationAdapters.tsindex 013d43a721..94dd218180 100644--- a/web/src/features/public-api/server/evaluation/evaluationAdapters.ts+++ b/web/src/features/public-api/server/evaluation/evaluationAdapters.ts@@ -64,7 +64,15 @@ const INTERNAL_MAPPING_COLUMN_TO_PUBLIC_SOURCE: Record<experiment_item_metadata: "experiment_item_metadata",};-function toStoredMappingList(mappings: PromptVariableMappingInputType[]) {+export function toPublicEvaluatorType(type: EvalTemplateType) {+ return type === EvalTemplateType.CODE+ ? PUBLIC_EVALUATOR_TYPE_CODE+ : PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE;+}++export function toStoredMappingList(+ mappings: PromptVariableMappingInputType[],+) {return observationVariableMappingList.parse(mappings.map((mapping) => ({templateVariable: mapping.variable,@@ -75,7 +83,9 @@ function toStoredMappingList(mappings: PromptVariableMappingInputType[]) {);}-function toApiReadMappings(mappings: unknown): PromptVariableMappingReadType[] {+export function toApiReadMappings(+ mappings: unknown,+): PromptVariableMappingReadType[] {const parsed = observationVariableMappingList.safeParse(mappings);if (!parsed.success) {logger.error("Failed to parse public evaluation rule mappings", {diff --git a/web/src/features/public-api/server/index.ts b/web/src/features/public-api/server/index.tsindex bfd55769fb..170d5443a8 100644--- a/web/src/features/public-api/server/index.ts+++ b/web/src/features/public-api/server/index.ts@@ -162,39 +162,3 @@ export {PostDashboardPlacementBody,PostUnstableDashboardBody,} from "../types/unstable-dashboards";-export type {- PatchUnstableEvaluationRuleBodyType,- PostUnstableEvaluationRuleBodyType,-} from "../types/unstable-evaluation-rules";-export type { PostUnstableEvaluatorBodyParsedType } from "../types/unstable-evaluators";-export type {- LegacyPromptVariableMappingType,- PromptVariableMappingInputType,- PromptVariableMappingReadType,- PublicCodeEvaluatorSourceCodeLanguageType,- PublicEvaluationRuleEvaluatorReferenceType,- PublicEvaluationRuleEvaluatorType,- PublicEvaluationRuleFilterType,- PublicEvaluationRuleReadFilterType,- PublicEvaluationRuleLegacyTargetType,- PublicEvaluationRuleReadTargetType,- PublicEvaluationRuleStatusType,- PublicEvaluationRuleTargetType,- PublicEvaluatorModelConfigType,- PublicEvaluatorOutputDefinitionType,- PublicEvaluatorTypeType,-} from "../types/unstable-public-evals-contract";-export {- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS,- ExperimentPromptVariableMappingSource,- LegacyPromptVariableMapping,- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,- ObservationPromptVariableMappingInput,- ObservationPromptVariableMappingSource,- PromptVariableMappingRead,- PUBLIC_EVALUATOR_TYPE_CODE,- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- PublicEvaluationRuleFilter,- PublicEvaluationRuleReadFilter,- PublicEvaluatorType,-} from "../types/unstable-public-evals-contract";diff --git a/web/src/features/public-api/types/evaluation/publicEvalsContract.ts b/web/src/features/public-api/types/evaluation/publicEvalsContract.tsindex 7cdd3b68c1..37c78421a3 100644--- a/web/src/features/public-api/types/evaluation/publicEvalsContract.ts+++ b/web/src/features/public-api/types/evaluation/publicEvalsContract.ts@@ -20,7 +20,7 @@ const PUBLIC_EVALUATOR_TYPES = ["llm_as_judge", "code"] as const;export const [PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE, PUBLIC_EVALUATOR_TYPE_CODE] =PUBLIC_EVALUATOR_TYPES;-const PublicEvaluatorType = z.enum(PUBLIC_EVALUATOR_TYPES);+export const PublicEvaluatorType = z.enum(PUBLIC_EVALUATOR_TYPES);export const PublicCodeEvaluatorSourceCodeLanguage = z.enum(["PYTHON","TYPESCRIPT",@@ -154,7 +154,7 @@ function createMappingSchema<});}-const ObservationPromptVariableMappingInput = createMappingSchema(+export const ObservationPromptVariableMappingInput = createMappingSchema(ObservationPromptVariableMappingSource,);diff --git a/web/src/features/public-api/types/unstable-evaluation-rules.ts b/web/src/features/public-api/types/unstable-evaluation-rules.tsdeleted file mode 100644index 56e18cb31c..0000000000--- a/web/src/features/public-api/types/unstable-evaluation-rules.ts+++ /dev/null@@ -1,254 +0,0 @@-import { JobTimeScopeZod, singleFilter } from "@langfuse/shared";-import { z } from "zod";-import {- ExperimentEvaluationRuleFilter,- ExperimentPromptVariableMappingInput,- LegacyPromptVariableMapping,- ObservationEvaluationRuleFilter,- ObservationPromptVariableMappingInput,- PUBLIC_EVALUATOR_TYPE_CODE,- PublicEvaluationRuleFilter,- PublicEvaluationRuleReadFilter,- PublicEvaluationRuleEvaluator,- PublicEvaluationRuleEvaluatorReference,- PublicEvaluationRuleEvaluatorReferencePatch,- PromptVariableMappingRead,- PublicEvaluationRuleLegacyTarget,- PublicEvaluationRuleStatus,- PublicEvaluationRuleTarget,- UnstablePublicApiPaginationQuery,- UnstablePublicApiPaginationResponse,-} from "@/src/features/public-api/types/unstable-public-evals-contract";--const APIEvaluationRuleBase = {- id: z.string(),- name: z.string(),- // Null when the rule has no evaluator assignments. Prefer `evaluators`.- evaluator: PublicEvaluationRuleEvaluator.nullable(),- enabled: z.boolean(),- status: PublicEvaluationRuleStatus,- pausedReason: z.string().nullable(),- pausedMessage: z.string().nullable(),- sampling: z.number().gt(0).lte(1),- createdAt: z.coerce.date(),- updatedAt: z.coerce.date(),-};--const APIEvaluationRule = z- .object({- ...APIEvaluationRuleBase,- evaluators: z.array(- z.object({- evaluator: PublicEvaluationRuleEvaluator,- mapping: z.array(PromptVariableMappingRead).nullable(),- }),- ),- target: PublicEvaluationRuleTarget,- filter: z.array(PublicEvaluationRuleFilter),- mapping: z.array(PromptVariableMappingRead),- })- .strict();--const APIReadableV2EvaluationRule = APIEvaluationRule.extend({- filter: z.array(PublicEvaluationRuleReadFilter),-});--const APILegacyEvaluationRule = z- .object({- ...APIEvaluationRuleBase,- evaluators: z.array(- z.object({- evaluator: PublicEvaluationRuleEvaluator,- mapping: z.array(LegacyPromptVariableMapping).nullable(),- }),- ),- target: PublicEvaluationRuleLegacyTarget,- delay: z.number().int().nonnegative(),- timeScope: z.array(JobTimeScopeZod),- filter: z.array(singleFilter),- mapping: z.array(LegacyPromptVariableMapping),- })- .strict();--const APIReadableEvaluationRule = z.union([- APIReadableV2EvaluationRule,- APILegacyEvaluationRule,-]);--export const GetUnstableEvaluationRulesQuery = UnstablePublicApiPaginationQuery;--export const GetUnstableEvaluationRulesResponse = z- .object({- data: z.array(APIReadableEvaluationRule),- meta: UnstablePublicApiPaginationResponse,- })- .strict();--export const GetUnstableEvaluationRuleQuery = z.object({- evaluationRuleId: z.string(),-});--/** @alias */-export const GetUnstableEvaluationRuleResponse = APIReadableEvaluationRule;--const EvaluationRuleCreateBase = {- name: z.string().min(1),- evaluator: PublicEvaluationRuleEvaluatorReference.optional(),- enabled: z.boolean(),- sampling: z.number().gt(0).lte(1).default(1),-};--function evaluatorAssignment<T extends z.ZodType>(mapping: T) {- return z.object({- evaluator: PublicEvaluationRuleEvaluatorReference,- mapping: z.array(mapping).optional(),- });-}--const PostUnstableObservationEvaluationRuleBody = z.object({- ...EvaluationRuleCreateBase,- evaluators: z- .array(evaluatorAssignment(ObservationPromptVariableMappingInput))- .min(1)- .max(100)- .optional(),- target: z.literal("observation"),- filter: z.array(ObservationEvaluationRuleFilter).default([]),- mapping: z.array(ObservationPromptVariableMappingInput).optional(),-});--const PostUnstableExperimentEvaluationRuleBody = z.object({- ...EvaluationRuleCreateBase,- evaluators: z- .array(evaluatorAssignment(ExperimentPromptVariableMappingInput))- .min(1)- .max(100)- .optional(),- target: z.literal("experiment"),- filter: z.array(ExperimentEvaluationRuleFilter).default([]),- mapping: z.array(ExperimentPromptVariableMappingInput).optional(),-});--// `code` evaluators use a fixed runtime mapping managed by Langfuse. LLM-// mappings may be omitted to inherit the evaluator version's default mapping.-export const PostUnstableEvaluationRuleBody = z- .discriminatedUnion("target", [- PostUnstableObservationEvaluationRuleBody,- PostUnstableExperimentEvaluationRuleBody,- ])- .refine((data) => Boolean(data.evaluator) !== Boolean(data.evaluators), {- path: ["evaluators"],- message: "Provide exactly one of evaluator or evaluators.",- })- .refine((data) => !data.evaluators || data.mapping === undefined, {- path: ["mapping"],- message:- "`mapping` is the deprecated single-evaluator alias and cannot be combined with `evaluators`. Set the mapping on the matching entry in `evaluators` instead.",- })- .refine(- (data) =>- (!data.evaluator ||- data.evaluator.type !== PUBLIC_EVALUATOR_TYPE_CODE ||- data.mapping === undefined) &&- (data.evaluators ?? []).every(- (assignment) =>- assignment.evaluator.type !== PUBLIC_EVALUATOR_TYPE_CODE ||- assignment.mapping === undefined,- ),- {- path: ["mapping"],- message:- "Code evaluator mappings are managed by Langfuse and cannot be provided in the request body.",- },- );-export type PostUnstableEvaluationRuleBodyType = z.infer<- typeof PostUnstableEvaluationRuleBody->;--/** @alias */-export const PostUnstableEvaluationRuleResponse = APIEvaluationRule;--/** @alias */-export const PatchUnstableEvaluationRuleQuery = GetUnstableEvaluationRuleQuery;--// Exported for reuse (see EvaluationRuleCreateBase) — the create fields, all-// made optional for PATCH.-const EvaluationRulePatchBase = {- name: z.string().min(1).optional(),- evaluator: PublicEvaluationRuleEvaluatorReferencePatch.optional(),- enabled: z.boolean().optional(),- sampling: z.number().gt(0).lte(1).optional(),-};--// Replacing `evaluators` is a full set replacement: entries not listed are-// detached. Mutually exclusive with the deprecated `evaluator`/`mapping` pair,-// which only ever addressed the first assignment.-function patchEvaluators<T extends z.ZodType>(mapping: T) {- return z.array(evaluatorAssignment(mapping)).min(1).max(100).optional();-}--function assertExclusiveEvaluatorFields<- T extends { evaluator?: unknown; mapping?: unknown; evaluators?: unknown },->(data: T) {- return (- !data.evaluators ||- (data.evaluator === undefined && data.mapping === undefined)- );-}--const EXCLUSIVE_EVALUATOR_FIELDS_MESSAGE =- "`evaluators` replaces the whole assignment set and cannot be combined with the deprecated `evaluator` or `mapping` fields.";--const UntargetedEvaluationRulePatch = z.object({- ...EvaluationRulePatchBase,- evaluators: patchEvaluators(ObservationPromptVariableMappingInput),- target: z.undefined().optional(),- filter: z.undefined().optional(),- mapping: z.undefined().optional(),-});--const ObservationEvaluationRulePatch = z.object({- ...EvaluationRulePatchBase,- evaluators: patchEvaluators(ObservationPromptVariableMappingInput),- target: z.literal("observation"),- filter: z.array(ObservationEvaluationRuleFilter).optional(),- mapping: z.array(ObservationPromptVariableMappingInput).optional(),-});--const ExperimentEvaluationRulePatch = z.object({- ...EvaluationRulePatchBase,- evaluators: patchEvaluators(ExperimentPromptVariableMappingInput),- target: z.literal("experiment"),- filter: z.array(ExperimentEvaluationRuleFilter).optional(),- mapping: z.array(ExperimentPromptVariableMappingInput).optional(),-});--export const PatchUnstableEvaluationRuleBody = z- .union([- ObservationEvaluationRulePatch,- ExperimentEvaluationRulePatch,- UntargetedEvaluationRulePatch,- ])- .refine((data) => Object.keys(data).length > 0, {- message:- "Request body cannot be empty. At least one field must be provided for update.",- })- .refine(assertExclusiveEvaluatorFields, {- path: ["evaluators"],- message: EXCLUSIVE_EVALUATOR_FIELDS_MESSAGE,- });-export type PatchUnstableEvaluationRuleBodyType = z.infer<- typeof PatchUnstableEvaluationRuleBody->;--/** @alias */-export const PatchUnstableEvaluationRuleResponse = APIReadableV2EvaluationRule;--/** @alias */-export const DeleteUnstableEvaluationRuleQuery = GetUnstableEvaluationRuleQuery;--export const DeleteUnstableEvaluationRuleResponse = z- .object({- message: z.literal("Evaluation rule successfully deleted"),- })- .strict();diff --git a/web/src/features/public-api/types/unstable-evaluators.ts b/web/src/features/public-api/types/unstable-evaluators.tsdeleted file mode 100644index 74779a0788..0000000000--- a/web/src/features/public-api/types/unstable-evaluators.ts+++ /dev/null@@ -1,122 +0,0 @@-import { z } from "zod";-import {- PUBLIC_EVALUATOR_TYPE_CODE,- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- ObservationPromptVariableMappingInput,- PromptVariableMappingRead,- PublicCodeEvaluatorDefinitionInput,- PublicEvaluatorModelConfig,- PublicEvaluatorOutputDefinition,- PublicLlmAsJudgeEvaluatorDefinitionInput,- UnstablePublicApiPaginationQuery,- UnstablePublicApiPaginationResponse,-} from "@/src/features/public-api/types/unstable-public-evals-contract";--const APIEvaluatorBase = z- .object({- id: z.string(),- name: z.string(),- version: z.number().int().positive(),- variables: z.array(z.string()),- // An evaluator's default mapping can name experiment-only sources, and a legacy one can be- // incomplete, so reads use the permissive schema. Requests stay strict.- mapping: z.array(PromptVariableMappingRead).nullable(),- evaluationRuleCount: z.number().int().nonnegative(),- createdAt: z.coerce.date(),- updatedAt: z.coerce.date(),- })- .strict();--const APILlmAsJudgeEvaluator = APIEvaluatorBase.extend({- type: z.literal(PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE),- prompt: z.string(),- outputDefinition: PublicEvaluatorOutputDefinition,- modelConfig: PublicEvaluatorModelConfig.nullable(),-}).strict();--const APICodeEvaluator = APIEvaluatorBase.extend({- type: z.literal(PUBLIC_EVALUATOR_TYPE_CODE),- sourceCode: z.string().min(1),- sourceCodeLanguage:- PublicCodeEvaluatorDefinitionInput.shape.sourceCodeLanguage,-}).strict();--const APIEvaluator = z.discriminatedUnion("type", [- APILlmAsJudgeEvaluator,- APICodeEvaluator,-]);--export const GetUnstableEvaluatorsQuery = UnstablePublicApiPaginationQuery;--export const GetUnstableEvaluatorsResponse = z- .object({- data: z.array(APIEvaluator),- meta: UnstablePublicApiPaginationResponse,- })- .strict();--export const GetUnstableEvaluatorQuery = z.object({- evaluatorId: z.string(),-});--/** @alias */-export const GetUnstableEvaluatorResponse = APIEvaluator;--/** @alias */-export const DeleteUnstableEvaluatorQuery = GetUnstableEvaluatorQuery;--export const DeleteUnstableEvaluatorResponse = z- .object({- message: z.literal("Evaluator successfully deleted"),- })- .strict();--// Fields shared by every create body, regardless of evaluator type. Exported so-// non-route consumers (e.g. the MCP tool layer) reuse the same definition.-const EvaluatorCreateBase = {- name: z.string().min(1),-};--const PostUnstableLlmAsJudgeEvaluatorBody = z.object({- ...EvaluatorCreateBase,- type: z.literal(PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE),- ...PublicLlmAsJudgeEvaluatorDefinitionInput.shape,- mapping: z.array(ObservationPromptVariableMappingInput).optional(),- sourceCode: z.never().optional(),- sourceCodeLanguage: z.never().optional(),-});--const PostUnstableCodeEvaluatorBody = z.object({- ...EvaluatorCreateBase,- type: z.literal(PUBLIC_EVALUATOR_TYPE_CODE),- ...PublicCodeEvaluatorDefinitionInput.shape,- prompt: z.never().optional(),- outputDefinition: z.never().optional(),- modelConfig: z.never().optional(),- mapping: z.never().optional(),-});--const PostUnstableTypedEvaluatorBody = z.discriminatedUnion("type", [- PostUnstableLlmAsJudgeEvaluatorBody,- PostUnstableCodeEvaluatorBody,-]);--// `type` may be omitted; it defaults to `llm_as_judge` for backwards-// compatibility. New clients should send `type` explicitly.-export const PostUnstableEvaluatorBody = z.preprocess((value) => {- if (typeof value !== "object" || value === null || "type" in value) {- return value;- }-- return {- ...value,- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,- };-}, PostUnstableTypedEvaluatorBody);--export type PostUnstableEvaluatorBodyParsedType = z.infer<- typeof PostUnstableTypedEvaluatorBody->;--/** @alias */-export const PostUnstableEvaluatorResponse = APIEvaluator;diff --git a/web/src/features/public-api/types/unstable-public-evals-contract.ts b/web/src/features/public-api/types/unstable-public-evals-contract.tsdeleted file mode 100644index a172d6e62e..0000000000--- a/web/src/features/public-api/types/unstable-public-evals-contract.ts+++ /dev/null@@ -1,360 +0,0 @@-import {- arrayOptionsFilter,- categoryOptionsFilter,- nullFilter,- numberFilter,- numberObjectFilter,- observationEvalFilterColumns,- paginationMetaResponseZod,- stringFilter,- stringObjectFilter,- stringOptionsFilter,- timeFilter,- experimentEvalFilterColumns,- booleanFilter,- CODE_EVAL_SOURCE_MAX_BYTES,- langfuseObjects,-} from "@langfuse/shared";-import { z } from "zod";-export { StructuredPublicApiErrorResponse } from "./structuredPublicApiErrorSchema";--const PUBLIC_EVALUATOR_TYPES = ["llm_as_judge", "code"] as const;-export const [PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE, PUBLIC_EVALUATOR_TYPE_CODE] =- PUBLIC_EVALUATOR_TYPES;--export const PublicEvaluatorType = z.enum(PUBLIC_EVALUATOR_TYPES);-const PublicCodeEvaluatorSourceCodeLanguage = z.enum(["PYTHON", "TYPESCRIPT"]);--export const PublicEvaluatorModelConfig = z.object({- provider: z.string().min(1),- model: z.string().min(1),-});--const PublicEvaluatorOutputFieldDefinition = z.object({- description: z.string().trim().min(1),-});--const PublicNumericEvaluatorOutputScoreDefinition =- PublicEvaluatorOutputFieldDefinition.extend({- minValue: z.number().optional(),- maxValue: z.number().optional(),- }).refine(- ({ minValue, maxValue }) =>- minValue === undefined || maxValue === undefined || minValue <= maxValue,- {- message: "Minimum value must be less than or equal to maximum value",- },- );--const PublicNumericEvaluatorOutputDefinition = z.object({- dataType: z.literal("NUMERIC"),- reasoning: PublicEvaluatorOutputFieldDefinition,- score: PublicNumericEvaluatorOutputScoreDefinition,-});--const PublicBooleanEvaluatorOutputDefinition = z.object({- dataType: z.literal("BOOLEAN"),- reasoning: PublicEvaluatorOutputFieldDefinition,- score: PublicEvaluatorOutputFieldDefinition,-});--const PublicCategoricalEvaluatorOutputScoreDefinition = z.object({- description: z.string().trim().min(1),- categories: z.array(z.string().trim().min(1)).min(2),- shouldAllowMultipleMatches: z.boolean(),-});--const PublicCategoricalEvaluatorOutputDefinition = z.object({- dataType: z.literal("CATEGORICAL"),- reasoning: PublicEvaluatorOutputFieldDefinition,- score: PublicCategoricalEvaluatorOutputScoreDefinition,-});--export const PublicEvaluatorOutputDefinition = z.discriminatedUnion(- "dataType",- [- PublicNumericEvaluatorOutputDefinition,- PublicBooleanEvaluatorOutputDefinition,- PublicCategoricalEvaluatorOutputDefinition,- ],-);--export const PublicEvaluationRuleTarget = z.enum(["observation", "experiment"]);-export const PublicEvaluationRuleLegacyTarget = z.enum(["trace", "dataset"]);-const _PublicEvaluationRuleReadTarget = z.union([- PublicEvaluationRuleTarget,- PublicEvaluationRuleLegacyTarget,-]);--export const PublicEvaluationRuleStatus = z.enum([- "active",- "inactive",- "paused",-]);--// No `scope`: every evaluator is project-owned, so the field could only ever-// carry one value. Gallery templates are starter definitions that are copied-// into the project on save, not addressable Langfuse-managed resources.-export const PublicEvaluationRuleEvaluatorReference = z.object({- name: z.string().min(1),- type: PublicEvaluatorType.default(PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE),-});--// PATCH intentionally omits `type`: an evaluation rule's evaluator type cannot-// be changed. The service always inherits the rule's current type, so a code-// rule is never retargeted to an LLM evaluator family, which stores its-// mapping per assignment and would inherit none. To use a different evaluator-// type, create a new rule.-export const PublicEvaluationRuleEvaluatorReferencePatch = z.object({- name: z.string().min(1),-});--export const PublicEvaluationRuleEvaluator =- PublicEvaluationRuleEvaluatorReference.extend({- id: z.string(),- });--export const ObservationPromptVariableMappingSource = z.enum([- "input",- "output",- "metadata",- "tool_calls",-]);--export const ExperimentPromptVariableMappingSource = z.enum([- "input",- "output",- "metadata",- "tool_calls",- "expected_output",- "experiment_item_metadata",-]);--function createMappingSchema<- TSource extends z.ZodType<- | "input"- | "output"- | "metadata"- | "tool_calls"- | "expected_output"- | "experiment_item_metadata"- >,->(sourceSchema: TSource) {- return z.object({- variable: z.string().min(1),- source: sourceSchema,- jsonPath: z.string().min(1).optional(),- });-}--export const ObservationPromptVariableMappingInput = createMappingSchema(- ObservationPromptVariableMappingSource,-);--export const ExperimentPromptVariableMappingInput = createMappingSchema(- ExperimentPromptVariableMappingSource,-);--const _PromptVariableMappingInput = z.union([- ObservationPromptVariableMappingInput,- ExperimentPromptVariableMappingInput,-]);--// This shape preserves incomplete mappings so callers can repair them.-// Mapping inputs remain strict and require a concrete source.-export const PromptVariableMappingRead = z.object({- variable: z.string().min(1),- source: ExperimentPromptVariableMappingSource.nullable(),- jsonPath: z.string().min(1).optional(),-});--export const LegacyPromptVariableMapping = z- .object({- variable: z.string().min(1),- langfuseObject: z.enum(langfuseObjects),- objectName: z.string().nullable(),- source: z.string().min(1),- jsonPath: z.string().min(1).optional(),- })- .refine(- (mapping) =>- mapping.langfuseObject === "trace" ||- mapping.langfuseObject === "dataset_item" ||- mapping.objectName !== null,- {- path: ["objectName"],- message: "objectName is required for observation objects",- },- );--const PublicEvaluationRuleReadFilterBase = z- .object({- type: z.string(),- column: z.string(),- operator: z.string(),- value: z.unknown().optional(),- })- .loose();--export const PublicEvaluationRuleReadFilter = z.union([- PublicEvaluationRuleReadFilterBase.safeExtend({ key: z.string() }),- PublicEvaluationRuleReadFilterBase,-]);--const filterSchemaFactories = {- datetime: (columnId: string) =>- timeFilter.safeExtend({ column: z.literal(columnId) }),- string: (columnId: string) =>- stringFilter.safeExtend({ column: z.literal(columnId) }),- number: (columnId: string) =>- numberFilter.safeExtend({ column: z.literal(columnId) }),- stringOptions: (columnId: string) =>- stringOptionsFilter.safeExtend({ column: z.literal(columnId) }),- categoryOptions: (columnId: string) =>- categoryOptionsFilter.safeExtend({ column: z.literal(columnId) }),- arrayOptions: (columnId: string) =>- arrayOptionsFilter.safeExtend({ column: z.literal(columnId) }),- stringObject: (columnId: string) =>- stringObjectFilter.safeExtend({ column: z.literal(columnId) }),- numberObject: (columnId: string) =>- numberObjectFilter.safeExtend({ column: z.literal(columnId) }),- boolean: (columnId: string) =>- booleanFilter.safeExtend({ column: z.literal(columnId) }),- null: (columnId: string) =>- nullFilter.safeExtend({ column: z.literal(columnId) }),-} as const;--type SupportedFilterFactory = keyof typeof filterSchemaFactories;--function createTargetFilterSchema(- columns: Array<{ id: string; type: SupportedFilterFactory }>,-) {- const schemas = columns.map((column) =>- filterSchemaFactories[column.type](column.id),- );-- if (schemas.length === 1) {- return schemas[0]!;- }-- return z.union(- schemas as [- (typeof schemas)[number],- (typeof schemas)[number],- ...Array<(typeof schemas)[number]>,- ],- );-}--// The experiment-root filter is what the `experiment` target *means*, so it is-// not separately addressable: exposing it on `observation` too would give one-// rule two contradictory public representations.-const EXPERIMENT_ROOT_FILTER_COLUMN = "isExperimentItemRootSpan";--export const OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS =- observationEvalFilterColumns- .filter((column) => column.id !== EXPERIMENT_ROOT_FILTER_COLUMN)- .map((column) => ({- id: column.id,- type: column.type as SupportedFilterFactory,- }));--// An experiment rule is an observation rule scoped to experiment root spans,-// so it accepts every observation filter plus the dataset scope.-export const EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS = [- ...OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,- {- id: "datasetId",- type: experimentEvalFilterColumns[0]!.type as SupportedFilterFactory,- },-];--export const ObservationEvaluationRuleFilter = createTargetFilterSchema(- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,-);--export const ExperimentEvaluationRuleFilter = createTargetFilterSchema(- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS,-);--export const PublicEvaluationRuleFilter = z.union([- ObservationEvaluationRuleFilter,- ExperimentEvaluationRuleFilter,-]);--export type PublicEvaluatorModelConfigType = z.infer<- typeof PublicEvaluatorModelConfig->;-export type PublicEvaluatorOutputDefinitionType = z.infer<- typeof PublicEvaluatorOutputDefinition->;-export type PublicEvaluatorTypeType = z.infer<typeof PublicEvaluatorType>;-export type PublicEvaluationRuleTargetType = z.infer<- typeof PublicEvaluationRuleTarget->;-export type PublicEvaluationRuleLegacyTargetType = z.infer<- typeof PublicEvaluationRuleLegacyTarget->;-export type PublicEvaluationRuleReadTargetType = z.infer<- typeof _PublicEvaluationRuleReadTarget->;-export type PublicEvaluationRuleStatusType = z.infer<- typeof PublicEvaluationRuleStatus->;-export type PublicEvaluationRuleEvaluatorReferenceType = z.infer<- typeof PublicEvaluationRuleEvaluatorReference->;-export type PublicEvaluationRuleEvaluatorType = z.infer<- typeof PublicEvaluationRuleEvaluator->;-export type PromptVariableMappingInputType = z.infer<- typeof _PromptVariableMappingInput->;-export type PromptVariableMappingReadType = z.infer<- typeof PromptVariableMappingRead->;-export type LegacyPromptVariableMappingType = z.infer<- typeof LegacyPromptVariableMapping->;-export type PublicEvaluationRuleFilterType = z.infer<- typeof PublicEvaluationRuleFilter->;-export type PublicEvaluationRuleReadFilterType = z.infer<- typeof PublicEvaluationRuleReadFilter->;-export const UnstablePublicApiPaginationQuery = z.object({- page: z.preprocess(- (x) => (x === "" ? undefined : x),- z.coerce.number().int().gt(0).default(1),- ),- limit: z.preprocess(- (x) => (x === "" ? undefined : x),- z.coerce.number().int().gt(0).lte(100).default(50),- ),-});--export const UnstablePublicApiPaginationResponse = paginationMetaResponseZod;--export const PublicLlmAsJudgeEvaluatorDefinitionInput = z.object({- prompt: z.string().min(1),- outputDefinition: PublicEvaluatorOutputDefinition,- modelConfig: PublicEvaluatorModelConfig.nullable().optional(),-});--export const PublicCodeEvaluatorDefinitionInput = z.object({- sourceCode: z- .string()- .min(1)- .refine(- (sourceCode) =>- Buffer.byteLength(sourceCode, "utf8") <= CODE_EVAL_SOURCE_MAX_BYTES,- {- message: `Source code must be ${CODE_EVAL_SOURCE_MAX_BYTES} bytes or less`,- },- ),- sourceCodeLanguage: PublicCodeEvaluatorSourceCodeLanguage,-});--export type PublicCodeEvaluatorSourceCodeLanguageType = z.infer<- typeof PublicCodeEvaluatorSourceCodeLanguage->;diff --git a/web/src/pages/api/public/unstable/evaluation-rules/[evaluationRuleId].ts b/web/src/pages/api/public/unstable/evaluation-rules/[evaluationRuleId].tsdeleted file mode 100644index a0c9571d32..0000000000--- a/web/src/pages/api/public/unstable/evaluation-rules/[evaluationRuleId].ts+++ /dev/null@@ -1,64 +0,0 @@-import {- deletePublicEvaluationRule,- getPublicEvaluationRule,- updatePublicEvaluationRule,-} from "@/src/features/evals/server/unstable-public-api";-import {- createUnstablePublicApiRoute,- withUnstablePublicApiMiddlewares,-} from "@/src/features/public-api/server/unstable-public-api-route";-import {- DeleteUnstableEvaluationRuleQuery,- DeleteUnstableEvaluationRuleResponse,- GetUnstableEvaluationRuleQuery,- GetUnstableEvaluationRuleResponse,- PatchUnstableEvaluationRuleBody,- PatchUnstableEvaluationRuleQuery,- PatchUnstableEvaluationRuleResponse,-} from "@/src/features/public-api/types/unstable-evaluation-rules";--export default withUnstablePublicApiMiddlewares({- GET: createUnstablePublicApiRoute({- name: "Get Unstable Evaluation Rule",- action: "evaluationRule:read",- querySchema: GetUnstableEvaluationRuleQuery,- responseSchema: GetUnstableEvaluationRuleResponse,- fn: async ({ query, auth }) =>- getPublicEvaluationRule({- projectId: auth.scope.projectId,- evaluationRuleId: query.evaluationRuleId,- }),- }),- PATCH: createUnstablePublicApiRoute({- name: "Update Unstable Evaluation Rule",- action: "evaluationRule:CUD",- querySchema: PatchUnstableEvaluationRuleQuery,- bodySchema: PatchUnstableEvaluationRuleBody,- responseSchema: PatchUnstableEvaluationRuleResponse,- fn: async ({ query, body, auth }) =>- updatePublicEvaluationRule({- orgId: auth.scope.orgId,- projectId: auth.scope.projectId,- evaluationRuleId: query.evaluationRuleId,- input: body,- auditScope: auth.scope,- }),- }),- DELETE: createUnstablePublicApiRoute({- name: "Delete Unstable Evaluation Rule",- action: "evaluationRule:CUD",- querySchema: DeleteUnstableEvaluationRuleQuery,- responseSchema: DeleteUnstableEvaluationRuleResponse,- fn: async ({ query, auth }) => {- await deletePublicEvaluationRule({- projectId: auth.scope.projectId,- evaluationRuleId: query.evaluationRuleId,- auditScope: auth.scope,- });-- return {- message: "Evaluation rule successfully deleted" as const,- };- },- }),-});diff --git a/web/src/pages/api/public/unstable/evaluation-rules/index.ts b/web/src/pages/api/public/unstable/evaluation-rules/index.tsdeleted file mode 100644index fde221ab97..0000000000--- a/web/src/pages/api/public/unstable/evaluation-rules/index.ts+++ /dev/null@@ -1,42 +0,0 @@-import {- createPublicEvaluationRule,- listPublicEvaluationRules,-} from "@/src/features/evals/server/unstable-public-api";-import {- createUnstablePublicApiRoute,- withUnstablePublicApiMiddlewares,-} from "@/src/features/public-api/server/unstable-public-api-route";-import {- GetUnstableEvaluationRulesQuery,- GetUnstableEvaluationRulesResponse,- PostUnstableEvaluationRuleBody,- PostUnstableEvaluationRuleResponse,-} from "@/src/features/public-api/types/unstable-evaluation-rules";--export default withUnstablePublicApiMiddlewares({- GET: createUnstablePublicApiRoute({- name: "List Unstable Evaluation Rules",- action: "evaluationRule:read",- querySchema: GetUnstableEvaluationRulesQuery,- responseSchema: GetUnstableEvaluationRulesResponse,- fn: async ({ query, auth }) =>- listPublicEvaluationRules({- projectId: auth.scope.projectId,- page: query.page,- limit: query.limit,- }),- }),- POST: createUnstablePublicApiRoute({- name: "Create Unstable Evaluation Rule",- action: "evaluationRule:CUD",- bodySchema: PostUnstableEvaluationRuleBody,- responseSchema: PostUnstableEvaluationRuleResponse,- fn: async ({ body, auth }) =>- createPublicEvaluationRule({- orgId: auth.scope.orgId,- projectId: auth.scope.projectId,- input: body,- auditScope: auth.scope,- }),- }),-});diff --git a/web/src/pages/api/public/unstable/evaluators/[evaluatorId].ts b/web/src/pages/api/public/unstable/evaluators/[evaluatorId].tsdeleted file mode 100644index a3ca669476..0000000000--- a/web/src/pages/api/public/unstable/evaluators/[evaluatorId].ts+++ /dev/null@@ -1,45 +0,0 @@-import {- deletePublicEvaluator,- getPublicEvaluator,-} from "@/src/features/evals/server/unstable-public-api";-import {- createUnstablePublicApiRoute,- withUnstablePublicApiMiddlewares,-} from "@/src/features/public-api/server/unstable-public-api-route";-import {- DeleteUnstableEvaluatorQuery,- DeleteUnstableEvaluatorResponse,- GetUnstableEvaluatorQuery,- GetUnstableEvaluatorResponse,-} from "@/src/features/public-api/types/unstable-evaluators";--export default withUnstablePublicApiMiddlewares({- GET: createUnstablePublicApiRoute({- name: "Get Unstable Evaluator",- action: "evaluator:read",- querySchema: GetUnstableEvaluatorQuery,- responseSchema: GetUnstableEvaluatorResponse,- fn: async ({ query, auth }) =>- getPublicEvaluator({- projectId: auth.scope.projectId,- evaluatorId: query.evaluatorId,- }),- }),- DELETE: createUnstablePublicApiRoute({- name: "Delete Unstable Evaluator",- action: "evaluator:CUD",- querySchema: DeleteUnstableEvaluatorQuery,- responseSchema: DeleteUnstableEvaluatorResponse,- fn: async ({ query, auth }) => {- await deletePublicEvaluator({- projectId: auth.scope.projectId,- evaluatorId: query.evaluatorId,- auditScope: auth.scope,- });-- return {- message: "Evaluator successfully deleted" as const,- };- },- }),-});diff --git a/web/src/pages/api/public/unstable/evaluators/index.ts b/web/src/pages/api/public/unstable/evaluators/index.tsdeleted file mode 100644index 711e4718c9..0000000000--- a/web/src/pages/api/public/unstable/evaluators/index.ts+++ /dev/null@@ -1,41 +0,0 @@-import {- createPublicEvaluator,- listPublicEvaluators,-} from "@/src/features/evals/server/unstable-public-api";-import {- createUnstablePublicApiRoute,- withUnstablePublicApiMiddlewares,-} from "@/src/features/public-api/server/unstable-public-api-route";-import {- GetUnstableEvaluatorsQuery,- GetUnstableEvaluatorsResponse,- PostUnstableEvaluatorBody,- PostUnstableEvaluatorResponse,-} from "@/src/features/public-api/types/unstable-evaluators";--export default withUnstablePublicApiMiddlewares({- GET: createUnstablePublicApiRoute({- name: "List Unstable Evaluators",- action: "evaluator:read",- querySchema: GetUnstableEvaluatorsQuery,- responseSchema: GetUnstableEvaluatorsResponse,- fn: async ({ query, auth }) =>- listPublicEvaluators({- projectId: auth.scope.projectId,- page: query.page,- limit: query.limit,- }),- }),- POST: createUnstablePublicApiRoute({- name: "Create Unstable Evaluator",- action: "evaluator:CUD",- bodySchema: PostUnstableEvaluatorBody,- responseSchema: PostUnstableEvaluatorResponse,- fn: async ({ body, auth }) =>- createPublicEvaluator({- projectId: auth.scope.projectId,- input: body,- auditScope: auth.scope,- }),- }),-});