Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(api)!: remove unstable evaluation endpoints (#17056)

Tobias Wochinger619dbe32b940
diff --git a/fern/apis/server/definition/dataset-run-items.yml b/fern/apis/server/definition/dataset-run-items.yml
index 4cd54717a7..79c8fe02db 100644
--- a/fern/apis/server/definition/dataset-run-items.yml
+++ b/fern/apis/server/definition/dataset-run-items.yml
@@ -54,7 +54,7 @@ types:
type: optional<datetime>
docs: |
ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g., "2026-01-21T14:35:42Z").
- Specifies the dataset version to use for this experiment run.
+ Specifies the dataset version to use for this experiment run.
If provided, the experiment will use dataset items as they existed at or before this timestamp.
If not provided, uses the latest version of dataset items.
createdAt:
diff --git a/fern/apis/server/definition/unstable/commons.yml b/fern/apis/server/definition/unstable/commons.yml
deleted file mode 100644
index 48e0c0a72a..0000000000
--- a/fern/apis/server/definition/unstable/commons.yml
+++ /dev/null
@@ -1,616 +0,0 @@
-# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
-types:
- EvaluatorType:
- docs: |
- The evaluator type.
-
- The unstable public API supports LLM-as-a-judge and code evaluators.
- enum:
- - llm_as_judge
- - code
-
- CodeEvaluatorSourceCodeLanguage:
- docs: Code evaluator runtime language.
- enum:
- - PYTHON
- - TYPESCRIPT
-
- EvaluationRuleTarget:
- docs: |
- The ingestion object type that should trigger evaluation runs.
-
- Choose the target first, because it changes both the valid filter columns and the valid variable-mapping sources:
- - `observation` evaluates live-ingested observations such as generations, spans, and events.
- It supports mapping from `input`, `output`, `metadata`, and `tool_calls`.
- - `experiment` evaluates live experiment executions and can additionally map `expected_output` and `experiment_item_metadata`.
- It currently supports filtering by `datasetId`.
- Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter.
- enum:
- - observation
- - experiment
-
- EvaluationRuleStatus:
- docs: |
- Effective runtime status of the evaluation rule.
-
- - `active`: enabled and currently runnable.
- - `inactive`: disabled by configuration.
- - `paused`: enabled, but Langfuse has blocked execution until the underlying issue is resolved.
- enum:
- - active
- - inactive
- - paused
-
- PromptVariableMappingSource:
- docs: |
- Source field used to populate a prompt variable.
-
- Use these values when mapping evaluator prompt variables to live data.
-
- Target-specific rules:
- - `target=observation` supports `input`, `output`, `metadata`, and `tool_calls`
- - `target=experiment` supports `input`, `output`, `metadata`, `tool_calls`, `expected_output`, and `experiment_item_metadata`
-
- Source semantics:
- - `input`: the observation or experiment input payload
- - `output`: the observation or experiment output payload
- - `metadata`: the metadata object for the target. Combine with `jsonPath` when you need one nested field instead of the whole object.
- - `tool_calls`: the tool calls recorded on the observation, as an array of `{id, name, arguments, type, index}` objects in the order the model emitted them. Combine with `jsonPath` (for example `$[*].name`) to select parts of each call.
- - `expected_output`: the experiment item's expected output. Only valid for `target=experiment`.
- - `experiment_item_metadata`: the experiment item's metadata object. Only valid for `target=experiment`.
- enum:
- - input
- - output
- - metadata
- - tool_calls
- - expected_output
- - experiment_item_metadata
-
- EvaluatorModelConfig:
- docs: |
- Optional explicit model configuration for an evaluator.
-
- If omitted, Langfuse uses the project's default evaluation model.
- If provided, the model must be available to the project when the evaluator or evaluation rule is enabled.
-
- To discover valid configured `provider` values for a project, call `GET /api/public/llm-connections` and read the `provider` field from the returned connections.
- Use a `provider` value that matches one of the connections already configured in the same project.
-
- Recovery guidance:
- - If evaluator creation returns `422` with `code=evaluator_preflight_failed`, either provide a valid explicit `modelConfig` here or configure the project's default evaluation model, then retry the same request.
- properties:
- provider:
- type: string
- docs: |
- Provider identifier to use for this evaluator, for example `openai` or `anthropic`.
-
- To discover valid values for the current project, call `GET /api/public/llm-connections` and use one of the returned `provider` values.
- model:
- type: string
- docs: Model identifier exposed by the provider, for example `gpt-4.1-mini`.
- examples:
- - value:
- provider: openai
- model: gpt-4.1-mini
-
- EvaluatorOutputDataType:
- docs: |
- Structured score type returned by an evaluator.
-
- This controls the type of score value Langfuse stores for evaluation results:
- - `NUMERIC`: a numeric score such as `0.82`
- - `BOOLEAN`: a boolean score such as `true`
- - `CATEGORICAL`: one or more category labels from a fixed list
- enum:
- - NUMERIC
- - BOOLEAN
- - CATEGORICAL
-
- EvaluatorOutputFieldDefinition:
- properties:
- description:
- type: string
- docs: Human-readable instructions for what the evaluator should return in this field.
-
- EvaluatorOutputDefinition:
- docs: |
- Structured output definition to send when creating an evaluator.
-
- Agent guidance:
- - `dataType` is required.
- - Do not send `version`; that is an internal storage detail and is not part of the public request contract.
- - For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and `score.description`.
- - For `NUMERIC`, `score.minValue` and `score.maxValue` optionally define inclusive bounds. If both are set, `minValue` must not exceed `maxValue`.
- - For `CATEGORICAL`, also provide `score.categories` and `score.shouldAllowMultipleMatches`.
- discriminant: dataType
- union:
- NUMERIC:
- type: PublicNumericEvaluatorOutputDefinition
- BOOLEAN:
- type: PublicBooleanEvaluatorOutputDefinition
- CATEGORICAL:
- type: PublicCategoricalEvaluatorOutputDefinition
- examples:
- - name: Numeric
- value:
- dataType: NUMERIC
- reasoning:
- description: Explain why the answer is correct or incorrect.
- score:
- description: Return a score between 0 and 1.
- minValue: 0
- maxValue: 1
- - name: Boolean
- value:
- dataType: BOOLEAN
- reasoning:
- description: Explain why the output satisfies the requirement.
- score:
- description: Return true if the output satisfies the requirement, otherwise false.
- - name: Categorical
- value:
- dataType: CATEGORICAL
- reasoning:
- description: Explain which category best fits the output.
- score:
- description: Choose the best category.
- categories:
- - correct
- - partially_correct
- - incorrect
- shouldAllowMultipleMatches: false
-
- PublicNumericEvaluatorOutputScoreDefinition:
- properties:
- description:
- type: string
- minValue:
- type: optional<double>
- docs: Inclusive minimum value for numeric evaluator scores.
- maxValue:
- type: optional<double>
- docs: Inclusive maximum value for numeric evaluator scores.
-
- PublicNumericEvaluatorOutputDefinition:
- properties:
- dataType:
- type: EvaluatorOutputDataType
- docs: Always `NUMERIC`.
- reasoning:
- type: EvaluatorOutputFieldDefinition
- score:
- type: PublicNumericEvaluatorOutputScoreDefinition
-
- PublicBooleanEvaluatorOutputDefinition:
- properties:
- dataType:
- type: EvaluatorOutputDataType
- docs: Always `BOOLEAN`.
- reasoning:
- type: EvaluatorOutputFieldDefinition
- score:
- type: EvaluatorOutputFieldDefinition
-
- PublicCategoricalEvaluatorOutputScoreDefinition:
- properties:
- description:
- type: string
- categories:
- type: list<string>
- shouldAllowMultipleMatches:
- type: boolean
-
- PublicCategoricalEvaluatorOutputDefinition:
- properties:
- dataType:
- type: EvaluatorOutputDataType
- docs: Always `CATEGORICAL`.
- reasoning:
- type: EvaluatorOutputFieldDefinition
- score:
- type: PublicCategoricalEvaluatorOutputScoreDefinition
-
- PublicEvaluatorOutputDefinition:
- docs: |
- Evaluator output definition returned by the public API.
-
- This response always includes `dataType` and never includes an internal output-definition `version`.
- Legacy stored evaluator definitions are normalized into this shape before they are returned.
-
- Use this response shape when deciding how to interpret future evaluation scores:
- - `NUMERIC`: expect numeric score values
- - `BOOLEAN`: expect `true` / `false`
- - `CATEGORICAL`: expect one or more values from `score.categories`
- discriminant: dataType
- union:
- NUMERIC:
- type: PublicNumericEvaluatorOutputDefinition
- BOOLEAN:
- type: PublicBooleanEvaluatorOutputDefinition
- CATEGORICAL:
- type: PublicCategoricalEvaluatorOutputDefinition
- examples:
- - name: PublicNumeric
- value:
- dataType: NUMERIC
- reasoning:
- description: Explain why the answer is correct or incorrect.
- score:
- description: Return a score between 0 and 1.
- minValue: 0
- maxValue: 1
- - name: PublicCategorical
- value:
- dataType: CATEGORICAL
- reasoning:
- description: Explain which label best fits the output.
- score:
- description: Choose the best label.
- categories:
- - correct
- - partially_correct
- - incorrect
- shouldAllowMultipleMatches: false
-
- EvaluationRuleStringFilterOperator:
- enum:
- - name: Equals
- value: "="
- - name: Contains
- value: contains
- - name: DoesNotContain
- value: does not contain
- - name: StartsWith
- value: starts with
- - name: EndsWith
- value: ends with
-
- EvaluationRuleNumberFilterOperator:
- enum:
- - name: Equals
- value: "="
- - name: GreaterThan
- value: ">"
- - name: LessThan
- value: "<"
- - name: GreaterThanOrEqual
- value: ">="
- - name: LessThanOrEqual
- value: "<="
-
- EvaluationRuleOptionsFilterOperator:
- enum:
- - name: AnyOf
- value: any of
- - name: NoneOf
- value: none of
-
- EvaluationRuleArrayOptionsFilterOperator:
- enum:
- - name: AnyOf
- value: any of
- - name: NoneOf
- value: none of
- - name: AllOf
- value: all of
-
- EvaluationRuleBooleanFilterOperator:
- enum:
- - name: Equals
- value: "="
- - name: NotEquals
- value: "<>"
-
- EvaluationRuleNullFilterOperator:
- enum:
- - name: IsNull
- value: is null
- - name: IsNotNull
- value: is not null
-
- DateTimeEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on.
- operator:
- type: EvaluationRuleNumberFilterOperator
- docs: Comparison operator for datetime values.
- value:
- type: datetime
- docs: Datetime value to compare against.
-
- StringEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on.
- operator:
- type: EvaluationRuleStringFilterOperator
- value:
- type: string
-
- NumberEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on.
- operator:
- type: EvaluationRuleNumberFilterOperator
- value:
- type: double
-
- StringOptionsEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on.
- operator:
- type: EvaluationRuleOptionsFilterOperator
- value:
- type: list<string>
- docs: One or more allowed string values.
-
- ArrayOptionsEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on.
- operator:
- type: EvaluationRuleArrayOptionsFilterOperator
- value:
- type: list<string>
- docs: One or more array elements to match.
-
- StringObjectEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Object-valued column to filter on. In the unstable public API this is currently `metadata`.
- key:
- type: string
- docs: Top-level key inside the object-valued column to filter on.
- operator:
- type: EvaluationRuleStringFilterOperator
- value:
- type: string
-
- NumberObjectEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Object-valued column to filter on.
- key:
- type: string
- docs: Key inside the object-valued column to filter on.
- operator:
- type: EvaluationRuleNumberFilterOperator
- value:
- type: double
-
- CategoryOptionsEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Object-valued column to filter on.
- key:
- type: string
- docs: Key inside the object-valued column to filter on.
- operator:
- type: EvaluationRuleOptionsFilterOperator
- value:
- type: list<string>
-
- BooleanEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on.
- operator:
- type: EvaluationRuleBooleanFilterOperator
- value:
- type: boolean
-
- NullEvaluationRuleFilter:
- properties:
- column:
- type: string
- docs: Column to filter on. In the unstable public API this is currently `parentObservationId`.
- operator:
- type: EvaluationRuleNullFilterOperator
- value:
- type: optional<string>
- docs: Ignored placeholder value. Clients may omit it or send an empty string.
-
- PromptVariableMappingInput:
- docs: |
- Connects one prompt variable to the available evaluation data.
-
- Manual mappings are used for `llm_as_judge` evaluators. `code` evaluators use a fixed runtime mapping managed by Langfuse.
-
- How to build a valid mapping list:
- 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.
- 2. Read the evaluator `variables` array.
- 3. Add exactly one mapping object for each variable in that array.
- 4. Use the variable name exactly as returned, without braces such as `{{` or `}}`.
- 5. Choose a `source` that is valid for the selected `target`.
-
- `jsonPath` is optional. Use it only when the selected source is a JSON object and you want to extract one nested field before inserting it into the evaluator prompt.
-
- Recovery guidance:
- - `invalid_variable_mapping`: the variable name is unknown for this evaluator, or the selected `source` is not valid for the chosen `target`
- - `missing_variable_mapping`: one or more LLM-as-judge prompt variables are not mapped yet
- - `duplicate_variable_mapping`: the same prompt variable appears more than once
- - `invalid_json_path`: the JSONPath expression is malformed. Remove it or correct it.
- properties:
- variable:
- type: string
- docs: |
- Prompt variable name without braces.
-
- Example: for the prompt `Judge {{input}} against {{output}}`, use `input` and `output`.
- source:
- type: PromptVariableMappingSource
- docs: |
- Source field that should populate the prompt variable.
-
- Quick reference:
- - `target=observation`: `input`, `output`, `metadata`, `tool_calls`
- - `target=experiment`: `input`, `output`, `metadata`, `tool_calls`, `expected_output`, `experiment_item_metadata`
- jsonPath:
- type: optional<string>
- docs: |
- Optional JSONPath selector applied to the selected source before it is passed to the evaluator prompt.
-
- Requirements:
- - Must start with `$`
- - Must be a syntactically valid JSONPath expression
- - Most useful with `source=metadata`
- examples:
- - name: BasicObservationMapping
- value:
- variable: input
- source: input
- - name: MetadataProjectionMapping
- value:
- variable: customer_tier
- source: metadata
- jsonPath: "$.customer.tier"
- - name: ExperimentExpectedOutputMapping
- value:
- variable: expected_output
- source: expected_output
-
- PromptVariableMappingRead:
- docs: |
- Connects one prompt variable to data from a target.
-
- `source` is `null` when the mapping is incomplete.
- properties:
- variable:
- type: string
- docs: Prompt variable name without braces.
- source:
- type: nullable<PromptVariableMappingSource>
- docs: Source field populating the variable, or `null` when the mapping is incomplete.
- jsonPath:
- type: optional<string>
- docs: Optional JSONPath selector applied to the selected source.
- examples:
- - name: ConfiguredReadMapping
- value:
- variable: input
- source: input
- - name: IncompleteReadMapping
- value:
- variable: input
- source: null
-
- EvaluationRuleFilter:
- docs: |
- One filter condition used to decide whether a live-ingested target should be evaluated.
-
- An evaluation rule can include zero or more filter objects. All filters must be satisfied for the target to run.
-
- How to build a valid filter object:
- - Pick the `target` first, because it changes the supported columns.
- - Pick the filter `type`. That determines which fields are required.
- - Use `key` only for object filters such as `metadata`.
- - Use the correct `value` shape for the chosen filter `type`.
-
- Operator quick reference by filter `type`:
- - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends with`
- - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`
- - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`
- - `stringOptions`: `any of`, `none of`
- - `arrayOptions`: `any of`, `none of`, `all of`
- - `stringObject`: same operators as `string`
- - `boolean`: `"="`, `"<>"`
- - `null`: `is null`, `is not null`
-
- Supported columns by target:
- - `target=observation`
- - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`
- - `name`: `stringOptions`, operators `any of` / `none of`
- - `environment`: `stringOptions`, operators `any of` / `none of`
- - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`
- - `version`: `string`
- - `traceName`: `stringOptions`, operators `any of` / `none of`
- - `userId`: `string`
- - `sessionId`: `string`
- - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`
- - `metadata`: `stringObject` with `key`
- - `isRootObservation`: `boolean`, operators `=` / `<>`; true when the observation has no parent or is explicitly marked as an application root
- - `parentObservationId`: `null`, operators `is null` / `is not null`
- - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`
- - `toolCalls`: `number`
- - `target=experiment`
- - `datasetId`: `stringOptions`, operators `any of` / `none of`
- Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
-
- Recovery guidance:
- - `invalid_filter_value` with `details.column` but no `invalidValues`: the selected `column` is not supported for the chosen `target`
- - `invalid_filter_value` with `details.invalidValues`: the selected values are not allowed for that column. Replace them with one of `details.allowedValues` when provided.
- - `invalid_filter_value` for `column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response.
- discriminant: type
- union:
- "datetime":
- type: DateTimeEvaluationRuleFilter
- "string":
- type: StringEvaluationRuleFilter
- "number":
- type: NumberEvaluationRuleFilter
- "stringOptions":
- type: StringOptionsEvaluationRuleFilter
- "categoryOptions":
- type: CategoryOptionsEvaluationRuleFilter
- "arrayOptions":
- type: ArrayOptionsEvaluationRuleFilter
- "stringObject":
- type: StringObjectEvaluationRuleFilter
- "numberObject":
- type: NumberObjectEvaluationRuleFilter
- "boolean":
- type: BooleanEvaluationRuleFilter
- "null":
- type: NullEvaluationRuleFilter
- examples:
- - name: ObservationTypeFilter
- value:
- type: stringOptions
- column: type
- operator: any of
- value:
- - GENERATION
- - name: ObservationMetadataFilter
- value:
- type: stringObject
- column: metadata
- key: customerTier
- operator: "="
- value: enterprise
- - name: ObservationRootOnlyFilter
- value:
- type: "null"
- column: parentObservationId
- operator: is null
- - name: ObservationLogicalRootFilter
- value:
- type: boolean
- column: isRootObservation
- operator: "="
- value: true
- - name: ObservationTagsFilter
- value:
- type: arrayOptions
- column: tags
- operator: any of
- value:
- - production
- - name: ExperimentDatasetFilter
- value:
- type: stringOptions
- column: datasetId
- operator: any of
- value:
- - "550e8400-e29b-41d4-a716-446655440000"
diff --git a/fern/apis/server/definition/unstable/errors.yml b/fern/apis/server/definition/unstable/errors.yml
index 180da7af55..74af0edfa0 100644
--- a/fern/apis/server/definition/unstable/errors.yml
+++ b/fern/apis/server/definition/unstable/errors.yml
@@ -1,232 +1,68 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
types:
PublicApiErrorCode:
- docs: |
- Machine-readable error code returned by the unstable evaluators API.
-
- SDKs, CLIs, and agents should branch on `code` rather than parsing the human-readable `message`.
- The HTTP status still indicates the broad error class, while `code` gives the specific failure reason.
+ docs: Machine-readable error code returned by unstable public API endpoints.
enum:
- authentication_failed
- access_denied
- invalid_request
- invalid_query
- invalid_body
- - invalid_filter_value
- - invalid_json_path
- - invalid_variable_mapping
- - missing_variable_mapping
- - duplicate_variable_mapping
- resource_not_found
- - name_conflict
- - evaluator_preflight_failed
- conflict
- - unprocessable_content
- rate_limited
- method_not_allowed
- internal_error
PublicApiValidationIssue:
- docs: |
- One validation issue returned for malformed request bodies or query parameters.
-
- This mirrors the most important parts of a Zod issue: a machine-readable `code`,
- a human-readable `message`, and a structured `path`.
+ docs: A validation issue returned for malformed request bodies or query parameters.
properties:
- code:
- type: string
- docs: Machine-readable validation issue code emitted by the server validator.
- message:
- type: string
- docs: Human-readable explanation of the validation failure.
- path:
- type: list<unknown>
- docs: Path to the invalid field, for example `["mapping", 0, "jsonPath"]`.
+ code: string
+ message: string
+ path: list<unknown>
PublicApiErrorDetails:
- docs: |
- Optional structured context attached to an unstable-evals error.
-
- The populated fields depend on the error `code`:
- - request parsing failures populate `issues`
- - filter validation failures populate `field`, `column`, `invalidValues`, and `allowedValues`
- - variable mapping failures populate `field`, `variable`, or `variables`
- - JSONPath validation failures populate `field`, `variable`, and `value`
- - evaluator preflight failures populate `evaluatorName`, `provider`, and `model`
- - rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and `resetAt`
+ docs: Optional structured context attached to an error.
properties:
- issues:
- type: optional<list<PublicApiValidationIssue>>
- docs: Validation issues for malformed request bodies or query parameters.
- field:
- type: optional<string>
- docs: Path-like reference to the failing field, for example `mapping[1].jsonPath`.
- column:
- type: optional<string>
- docs: Filter column that failed validation.
- invalidValues:
- type: optional<list<string>>
- docs: Unsupported values supplied by the caller.
- allowedValues:
- type: optional<list<string>>
- docs: Allowed values for the failing filter column.
- variable:
- type: optional<string>
- docs: Evaluator variable involved in the failure.
- variables:
- type: optional<list<string>>
- docs: Multiple evaluator variables involved in the failure, for example missing mappings.
- value:
- type: optional<string>
- docs: Raw invalid value supplied by the caller.
- evaluatorName:
- type: optional<string>
- docs: Evaluator name used during preflight validation.
- provider:
- type: optional<nullable<string>>
- docs: Provider resolved during evaluator preflight, if any.
- model:
- type: optional<nullable<string>>
- docs: Model resolved during evaluator preflight, if any.
- retryAfterSeconds:
- type: optional<integer>
- docs: Suggested retry delay for rate-limited requests.
- limit:
- type: optional<integer>
- docs: Numeric limit associated with the failure, for example the active evaluation-rule cap or the current rate-limit window.
- remaining:
- type: optional<integer>
- docs: Remaining requests in the current rate-limit window.
- resetAt:
- type: optional<string>
- docs: ISO-8601 timestamp when the current rate-limit window resets.
+ issues: optional<list<PublicApiValidationIssue>>
+ field: optional<string>
+ column: optional<string>
+ invalidValues: optional<list<string>>
+ allowedValues: optional<list<string>>
+ value: optional<string>
+ retryAfterSeconds: optional<integer>
+ limit: optional<integer>
+ remaining: optional<integer>
+ resetAt: optional<string>
PublicApiError:
- docs: |
- Standard error envelope for the unstable evaluators API.
-
- Response handling guidance:
- - Use the HTTP status code for the broad class of failure.
- - Use `code` for precise branching in SDKs, CLIs, or agents.
- - Inspect `details` for field-level validation context such as invalid filter values, malformed JSONPath expressions, or missing variable mappings.
- - Retry only after fixing the specific issue described by `code` and `details`.
+ docs: Standard error envelope for unstable public API endpoints.
properties:
message:
type: string
docs: Human-readable description of the failure.
code:
type: PublicApiErrorCode
- docs: Stable machine-readable error code.
+ docs: Machine-readable error code.
details:
type: optional<PublicApiErrorDetails>
- docs: Optional structured error context. Inspect the populated fields based on `code`.
- examples:
- - name: InvalidFilterValue
- value:
- message: 'Filter column "type" contains unsupported value(s): INVALID'
- code: invalid_filter_value
- details:
- field: filter[0].value
- column: type
- invalidValues:
- - INVALID
- allowedValues:
- - GENERATION
- - SPAN
- - EVENT
- - name: InvalidExperimentDatasetFilterValue
- value:
- message: 'Filter column "datasetId" contains dataset id(s) that do not exist in this project: dataset-prod'
- code: invalid_filter_value
- details:
- field: filter[0].value
- column: datasetId
- invalidValues:
- - dataset-prod
- - name: EvaluatorPreflightFailed
- value:
- message: 'No valid LLM model found for evaluator "answer-correctness". No default model or custom model configured for project project_123'
- code: evaluator_preflight_failed
- details:
- evaluatorName: answer-correctness
- provider: null
- model: null
- - name: MissingVariableMapping
- value:
- message: "Missing mappings for evaluator variable(s): output"
- code: missing_variable_mapping
- details:
- field: mapping
- variables:
- - output
- - name: InvalidJsonPath
- value:
- message: 'Mapping for variable "customer_tier" has an invalid jsonPath "$[". JSONPath expressions must use balanced quotes, brackets, and parentheses.'
- code: invalid_json_path
- details:
- field: mapping[0].jsonPath
- variable: customer_tier
- value: "$["
- - name: NameConflict
- value:
- message: 'An evaluation rule named "answer-quality-live" already exists in this project. Use PATCH /api/public/unstable/evaluation-rules/erule_123 to update it instead of creating a duplicate.'
- code: name_conflict
- details:
- field: name
- - name: ActiveEvaluationRuleLimit
- value:
- message: This project already has the maximum number of active evaluation rules (500). Disable an existing active evaluation rule before enabling another one.
- code: conflict
- details:
- limit: 500
- - name: RateLimited
- value:
- message: Rate limit exceeded
- code: rate_limited
- details:
- retryAfterSeconds: 60
- limit: 1000
- remaining: 0
- resetAt: "2026-03-30T10:00:00.000Z"
+ docs: Optional structured error context.
errors:
BadRequestError:
- docs: |
- Request parsing or validation failed.
-
- Typical unstable-eval examples:
- - malformed request body or query parameters
- - unsupported filter value
- - invalid JSONPath selector
- - missing or duplicate variable mappings
-
- Recovery guidance:
- - read `details.issues` for malformed bodies or queries
- - read `details.column`, `details.invalidValues`, and `details.allowedValues` for filter problems
- - for `details.column=datasetId`, call `GET /api/public/v2/datasets` and retry with dataset `id` values from that response
- - read `details.variable` or `details.variables` for mapping problems
+ docs: Request parsing or validation failed.
status-code: 400
type: PublicApiError
UnauthorizedError:
- docs: |
- Authentication failed.
-
- Typical unstable-eval examples:
- - the API key or secret key is invalid
- - the request uses the wrong host or stale credentials
+ docs: Authentication failed.
status-code: 401
type: PublicApiError
AccessDeniedError:
- docs: |
- Authenticated, but the caller is not allowed to access the requested resource.
-
- Typical unstable-eval examples:
- - using an organization-scoped key against project-scoped evaluator endpoints
- - using a valid key that lacks the required access level for this resource
+ docs: The caller is not allowed to access the requested resource.
status-code: 403
type: PublicApiError
NotFoundError:
- docs: The requested evaluator or evaluation rule does not exist within the authorized project.
+ docs: The requested resource does not exist within the authorized project.
status-code: 404
type: PublicApiError
MethodNotAllowedError:
@@ -234,31 +70,9 @@ errors:
status-code: 405
type: PublicApiError
ConflictError:
- docs: |
- The request conflicts with existing state.
-
- Typical unstable-eval examples:
- - evaluator version changed during concurrent creation
- - an evaluation rule name already exists in the project, so the caller should PATCH the existing resource instead
- - enabling another evaluation rule would exceed the project limit of 500 active evaluation rules
- - the underlying state changed between read and write operations, so the client should retry
-
- Recovery guidance:
- - for `name_conflict`, PATCH the existing evaluation rule instead of creating another one
- - for the active-limit conflict, disable another active evaluation rule before enabling a new one
+ docs: The request conflicts with existing state.
status-code: 409
type: PublicApiError
- UnprocessableContentError:
- docs: |
- The request is syntactically valid, but Langfuse cannot accept it in its current form.
-
- The most important unstable-eval case is `code=evaluator_preflight_failed`, which means the evaluator cannot currently run with the resolved model configuration.
-
- Recovery guidance:
- - configure the project's default evaluation model, or send a valid explicit evaluator `modelConfig`
- - once the model configuration is valid, retry the same request
- status-code: 422
- type: PublicApiError
TooManyRequestsError:
docs: The project is rate limited. Retry after the interval described in the response headers and error details.
status-code: 429
diff --git a/fern/apis/server/definition/unstable/evaluation-rules.yml b/fern/apis/server/definition/unstable/evaluation-rules.yml
deleted file mode 100644
index a347e1e506..0000000000
--- a/fern/apis/server/definition/unstable/evaluation-rules.yml
+++ /dev/null
@@ -1,822 +0,0 @@
-# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
-imports:
- commons: ./commons.yml
- errors: ./errors.yml
- evals: ../evaluation-commons.yml
- pagination: ../utils/pagination.yml
-
-service:
- auth: true
- base-path: /api/public/unstable
- endpoints:
- create:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Create an evaluation rule.
-
- An evaluation rule defines **what** incoming data should be evaluated and **how prompt variables should be populated** from that data.
-
- Use this resource after choosing an evaluator from the evaluator endpoints.
-
- Key rules:
- - `name` must be unique within the project for public evaluation rules
- - `target` must be `observation` or `experiment`
- - provide either the compatibility `evaluator` field or the new `evaluators` array, never both
- - each evaluator `name` + `type` must identify an existing evaluator family returned by the evaluator endpoints
- - Langfuse resolves that family to its latest version before saving the evaluation rule
- - for `target=experiment`, use dataset `id` values from `GET /api/public/v2/datasets` when filtering by `datasetId`
- - an omitted LLM-as-judge assignment mapping inherits the evaluator version's default mapping
- - the effective mapping must map every evaluator prompt variable exactly once
- - for `code` evaluators, Langfuse uses the fixed code runtime mapping; omit `mapping` in create and update requests
- - for user-provided `llm_as_judge` mappings, `expected_output` and `experiment_item_metadata` are only valid for `target=experiment`
- - if `enabled=true`, Langfuse validates that the referenced evaluator can currently run
- - at most 500 evaluation rules can be effectively active in one project at the same time (enforced identically by the API, the MCP tools, and the app)
-
- If an evaluation rule with the same `name` already exists in the project, the API returns `409`.
- In that case, update the existing resource with `PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of creating a second one.
-
- If enabling this resource would exceed the 500-active limit, the API also returns `409`.
- In that case, disable or pause another active evaluation rule before enabling a new one.
-
- Current scope:
- - evaluation rules are live-ingestion rules only
- - they do not trigger historical backfills
-
- Recovery guidance:
- - `400 invalid_filter_value`: fix the filter `column` or `value` using `details.column`, `details.invalidValues`, and `details.allowedValues`
- - `400 invalid_filter_value` with `details.column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response
- - `400 missing_variable_mapping`: for `llm_as_judge` evaluators, fetch the evaluator again and make sure every variable in `variables` appears exactly once in `mapping`
- - `400 duplicate_variable_mapping`: remove repeated mappings for the same variable
- - `400 invalid_variable_mapping`: for `llm_as_judge`, switch to a valid `source` for the selected `target`, or fix the variable name
- - `400 invalid_json_path`: remove or correct the `jsonPath`
- - `422 evaluator_preflight_failed`: the selected evaluator cannot run with the resolved model configuration. Fix the evaluator/default model setup, then retry the create request.
- method: POST
- path: /evaluation-rules
- request: CreateEvaluationRuleRequest
- response: EvaluationRule
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.NotFoundError
- - errors.ConflictError
- - errors.MethodNotAllowedError
- - errors.UnprocessableContentError
- - errors.TooManyRequestsError
- - errors.InternalServerError
- examples:
- - name: CreateObservationEvaluationRule
- docs: Deploy an evaluator to score live generations only.
- request:
- name: answer-correctness-live
- evaluator:
- name: answer-correctness
- type: llm_as_judge
- target: observation
- enabled: true
- sampling: 1
- filter:
- - type: stringOptions
- column: type
- operator: any of
- value:
- - GENERATION
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- response:
- body:
- id: erule_123
- name: answer-correctness-live
- evaluator:
- id: evaltmpl_123
- name: answer-correctness
- type: llm_as_judge
- evaluators:
- - evaluator:
- id: evaltmpl_123
- name: answer-correctness
- type: llm_as_judge
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- target: observation
- enabled: true
- status: active
- pausedReason: null
- pausedMessage: null
- sampling: 1
- filter:
- - type: stringOptions
- column: type
- operator: any of
- value:
- - GENERATION
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- createdAt: "2026-03-30T09:20:00.000Z"
- updatedAt: "2026-03-30T09:20:00.000Z"
- - name: CreateCodeObservationEvaluationRule
- docs: Deploy a code evaluator to score live generation outputs.
- request:
- name: toxicity-code-live
- evaluator:
- name: toxicity-detector
- type: code
- target: observation
- enabled: true
- sampling: 1
- filter:
- - type: stringOptions
- column: type
- operator: any of
- value:
- - GENERATION
- response:
- body:
- id: erule_234
- name: toxicity-code-live
- evaluator:
- id: evaltmpl_234
- name: toxicity-detector
- type: code
- evaluators:
- - evaluator:
- id: evaltmpl_234
- name: toxicity-detector
- type: code
- mapping: null
- target: observation
- enabled: true
- status: active
- pausedReason: null
- pausedMessage: null
- sampling: 1
- filter:
- - type: stringOptions
- column: type
- operator: any of
- value:
- - GENERATION
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- - variable: metadata
- source: metadata
- - variable: experimentItemExpectedOutput
- source: expected_output
- - variable: experimentItemMetadata
- source: experiment_item_metadata
- createdAt: "2026-03-30T09:25:00.000Z"
- updatedAt: "2026-03-30T09:25:00.000Z"
- - name: CreateExperimentEvaluationRule
- docs: Deploy an evaluator to compare experiment outputs against expected outputs. Discover valid dataset IDs with `GET /api/public/v2/datasets` first.
- request:
- name: experiment-expected-output-match
- evaluator:
- name: expected-output-match
- type: llm_as_judge
- target: experiment
- enabled: true
- sampling: 0.5
- filter:
- - type: stringOptions
- column: datasetId
- operator: any of
- value:
- - "550e8400-e29b-41d4-a716-446655440000"
- mapping:
- - variable: output
- source: output
- - variable: expected_output
- source: expected_output
- response:
- body:
- id: erule_456
- name: experiment-expected-output-match
- evaluator:
- id: evaltmpl_456
- name: expected-output-match
- type: llm_as_judge
- evaluators:
- - evaluator:
- id: evaltmpl_456
- name: expected-output-match
- type: llm_as_judge
- mapping:
- - variable: output
- source: output
- - variable: expected_output
- source: expected_output
- target: experiment
- enabled: true
- status: active
- pausedReason: null
- pausedMessage: null
- sampling: 0.5
- filter:
- - type: stringOptions
- column: datasetId
- operator: any of
- value:
- - "550e8400-e29b-41d4-a716-446655440000"
- mapping:
- - variable: output
- source: output
- - variable: expected_output
- source: expected_output
- createdAt: "2026-03-30T09:30:00.000Z"
- updatedAt: "2026-03-30T09:30:00.000Z"
-
- list:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- List evaluation rules in the authenticated project.
-
- This includes legacy `trace` and `dataset` rules so they can be inspected and migrated to v4 rules. Legacy rules are read-only through this API; create, update, and delete continue to support only `observation` and `experiment` rules.
- method: GET
- path: /evaluation-rules
- request:
- name: ListEvaluationRulesRequest
- query-parameters:
- page:
- type: optional<integer>
- docs: 1-based page number. Defaults to `1`.
- limit:
- type: optional<integer>
- docs: Maximum number of items per page. Defaults to `50`.
- response: EvaluationRules
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.MethodNotAllowedError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
- get:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Get one evaluation rule by its identifier.
-
- Use this endpoint to inspect the current evaluator, target, mapping, filters, execution timing, and effective runtime status. Legacy `trace` and `dataset` rules are returned for migration and are read-only through this API.
- method: GET
- path: /evaluation-rules/{evaluationRuleId}
- path-parameters:
- evaluationRuleId:
- type: string
- docs: Evaluation rule identifier returned by the evaluation rule endpoints.
- response: ReadableEvaluationRule
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.NotFoundError
- - errors.MethodNotAllowedError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
- update:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Update an evaluation rule.
-
- Typical uses:
- - enable or disable live execution
- - switch to another evaluator
- - adjust sampling
- - change filters
- - update LLM-as-judge variable mappings
-
- Important behavior:
- - provide only the fields you want to change
- - if you provide `evaluator`, Langfuse resolves that evaluator family to its latest version before saving
- - changing `target`, `filter`, or an LLM-as-judge `mapping` must still produce a valid target-specific configuration
- - if you change `target` for an LLM-as-judge rule, also send a compatible `filter` and `mapping` in the same request unless the existing ones are still valid for the new target
- - for `code` evaluator rules, omit `mapping`; Langfuse stores the fixed code runtime mapping automatically
- - if the resulting config is enabled, Langfuse re-validates that the selected evaluator can run
- - if the update would move a non-active evaluation rule into the active state and the project already has 500 active evaluation rules, the API returns `409`
-
- Recovery guidance:
- - if an LLM-as-judge update fails with `missing_variable_mapping` or `invalid_variable_mapping` after changing `evaluator` or `target`, resend the request with a complete new `mapping`
- - if the update fails with `invalid_filter_value` after changing `target`, resend the request with a target-compatible `filter`
- method: PATCH
- path: /evaluation-rules/{evaluationRuleId}
- path-parameters:
- evaluationRuleId:
- type: string
- docs: Evaluation rule identifier.
- request: UpdateEvaluationRuleRequest
- response: ReadableV2EvaluationRule
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.NotFoundError
- - errors.MethodNotAllowedError
- - errors.UnprocessableContentError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
- delete:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Delete an evaluation rule.
-
- This removes the live-ingestion rule only. It does not delete the referenced evaluator.
- method: DELETE
- path: /evaluation-rules/{evaluationRuleId}
- path-parameters:
- evaluationRuleId:
- type: string
- docs: Evaluation rule identifier.
- response: DeleteEvaluationRuleResponse
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.NotFoundError
- - errors.MethodNotAllowedError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
-types:
- EvaluationRuleBase:
- docs: |
- Live evaluation rule for incoming data.
-
- An evaluation rule answers:
- - which evaluator should be used
- - which target objects should trigger scoring
- - how often scoring should run
- - which target fields should populate each prompt variable
- - whether the deployment is active, inactive, or paused
-
- Important status semantics:
- - `enabled` is the desired on/off setting from the client
- - `status` is the effective runtime state after Langfuse applies validation and blocking rules
- - `enabled=true` with `status=paused` means the rule should run, but Langfuse has paused it until the underlying problem is fixed
- properties:
- id:
- type: string
- docs: Stable evaluation rule identifier.
- name:
- type: string
- docs: Human-readable deployment name. This is independent from the evaluator name.
- evaluator:
- type: nullable<EvaluationRuleEvaluator>
- docs: |
- Deprecated compatibility alias for `evaluators[0].evaluator`, or `null`
- when the rule currently has no evaluator assignments.
-
- `id` identifies the evaluator family. The rule automatically uses the
- latest available version of that evaluator.
- enabled:
- type: boolean
- docs: Desired enabled state configured by the client.
- status:
- type: commons.EvaluationRuleStatus
- docs: Effective runtime status after Langfuse applies validation and blocking rules.
- pausedReason:
- type: nullable<string>
- docs: Machine-readable reason when `status=paused`, otherwise `null`.
- pausedMessage:
- type: nullable<string>
- docs: Human-readable explanation when `status=paused`, otherwise `null`.
- sampling:
- type: double
- docs: |
- Fraction of matching target objects that should be evaluated.
-
- Must be greater than `0` and less than or equal to `1`.
- - `1` means evaluate every matching target.
- - `0.25` means evaluate approximately 25% of matching targets.
- createdAt:
- type: datetime
- docs: Timestamp when the evaluation rule was created.
- updatedAt:
- type: datetime
- docs: Timestamp when the evaluation rule was last updated.
-
- EvaluationRule:
- extends: EvaluationRuleBase
- properties:
- evaluators:
- type: list<EvaluationRuleEvaluatorAssignment>
- docs: Evaluators attached to this rule in deterministic assignment order. A `null` mapping inherits the evaluator version's default mapping.
- target:
- type: commons.EvaluationRuleTarget
- docs: Target object type that should trigger scoring.
- filter:
- type: list<commons.EvaluationRuleFilter>
- docs: List of filter conditions used to decide whether a target should be evaluated.
- mapping:
- type: list<commons.PromptVariableMappingRead>
- docs: Deprecated compatibility alias containing the effective mapping for `evaluators[0]`.
-
- ReadableV2EvaluationRule:
- extends: EvaluationRuleBase
- properties:
- evaluators:
- type: list<EvaluationRuleEvaluatorAssignment>
- docs: Evaluators attached to this rule in deterministic assignment order. A `null` mapping inherits the evaluator version's default mapping.
- target:
- type: commons.EvaluationRuleTarget
- docs: Target object type that should trigger scoring.
- filter:
- type: list<evals.EvaluationRuleReadFilter>
- docs: List of stored filter conditions returned verbatim. Historical filters remain readable even when the current write contract no longer accepts their shape.
- mapping:
- type: list<commons.PromptVariableMappingRead>
- docs: Deprecated compatibility alias containing the effective mapping for `evaluators[0]`.
- examples:
- - value:
- id: erule_123
- name: answer-correctness-live
- evaluator:
- id: evaltmpl_123
- name: answer-correctness
- type: llm_as_judge
- evaluators:
- - evaluator:
- id: evaltmpl_123
- name: answer-correctness
- type: llm_as_judge
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- target: observation
- enabled: true
- status: active
- pausedReason: null
- pausedMessage: null
- sampling: 1
- filter:
- - type: stringOptions
- column: type
- operator: any of
- value:
- - GENERATION
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- createdAt: "2026-03-30T09:20:00.000Z"
- updatedAt: "2026-03-30T09:20:00.000Z"
-
- LegacyEvaluationRule:
- docs: |
- **Deprecated:** Legacy trace- or dataset-level evaluation rule returned by list and get for migration.
-
- This resource is read-only through the unstable public API. Its mapping preserves the trace, dataset item, or named observation selected for each prompt variable.
- extends: EvaluationRuleBase
- properties:
- evaluators:
- type: list<LegacyEvaluationRuleEvaluatorAssignment>
- docs: Evaluators attached to this rule in deterministic assignment order.
- target:
- type: LegacyEvaluationRuleTarget
- delay:
- type: integer
- docs: Delay in milliseconds before the legacy evaluation job runs.
- timeScope:
- type: list<EvaluationRuleTimeScope>
- docs: Whether the legacy rule evaluates newly ingested data, existing data, or both.
- filter:
- type: list<commons.EvaluationRuleFilter>
- docs: Stored filters used by the legacy trace or dataset rule.
- mapping:
- type: list<LegacyPromptVariableMapping>
- docs: Stored variable mappings, including the trace, dataset item, or named observation selected for each variable.
-
- ReadableEvaluationRule:
- docs: Evaluation rule returned by list and get, including read-only legacy trace and dataset rules.
- discriminated: false
- union:
- - ReadableV2EvaluationRule
- - LegacyEvaluationRule
-
- EvaluationRules:
- docs: Paginated list of evaluation rules.
- properties:
- data:
- type: list<ReadableEvaluationRule>
- docs: Evaluation rules in the current page.
- meta:
- type: pagination.MetaResponse
- docs: Standard pagination metadata.
-
- CreateEvaluationRuleRequest:
- docs: |
- Request body for creating an evaluation rule.
-
- Checklist for agents and SDK clients:
- - reference an existing evaluator family by `evaluator.name` and `evaluator.type`
- - choose `target=observation` or `target=experiment`
- - if `target=experiment` and you want a dataset filter, call `GET /api/public/v2/datasets` first and use dataset `id` values in `filter[].value`
- - for `llm_as_judge`, fetch or inspect the evaluator first and provide a complete mapping for every prompt variable
- - for `code`, do not send variables or mappings; Langfuse stores the fixed code runtime mapping automatically
- - optionally narrow execution with `filter`
- - set `enabled=true` only when you want live execution immediately
- discriminated: false
- union:
- - CreateLlmAsJudgeEvaluationRuleRequest
- - CreateCodeEvaluationRuleRequest
- - CreateEvaluationRuleWithEvaluatorsRequest
-
- CreateEvaluationRuleWithEvaluatorsRequest:
- properties:
- name:
- type: string
- docs: Human-readable deployment name.
- evaluators:
- type: list<CreateEvaluationRuleEvaluatorAssignment>
- docs: |
- One or more evaluator assignments. Providing the deprecated top-level `evaluator` or `mapping` fields alongside this is rejected with `400`.
-
- Multiple assignments are supported on writable targets.
- target:
- type: commons.EvaluationRuleTarget
- enabled:
- type: boolean
- sampling:
- type: optional<double>
- filter:
- type: optional<list<commons.EvaluationRuleFilter>>
-
- CreateLlmAsJudgeEvaluationRuleRequest:
- properties:
- name:
- type: string
- docs: Human-readable deployment name.
- evaluator:
- type: LlmAsJudgeEvaluationRuleEvaluatorReference
- docs: |
- LLM-as-judge evaluator family to use.
-
- Use `name` and `type` from the evaluator endpoints. If `type` is omitted, Langfuse defaults it to `llm_as_judge` for backwards compatibility.
- Langfuse resolves that family to its latest version before saving the rule.
- target:
- type: commons.EvaluationRuleTarget
- docs: Target object type to evaluate.
- enabled:
- type: boolean
- docs: Whether the deployment should be active immediately after creation.
- sampling:
- type: optional<double>
- docs: Optional sampling fraction. Defaults to `1`.
- filter:
- type: optional<list<commons.EvaluationRuleFilter>>
- docs: |
- Optional filter list.
-
- Omit or pass an empty list to evaluate all matching targets for the selected `target`.
- Each filter object must use a column that is valid for that `target`.
- For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
- mapping:
- type: list<commons.PromptVariableMappingInput>
- docs: |
- LLM-as-judge variable mappings.
-
- Every prompt variable must appear exactly once.
- Build this list from the evaluator's `variables` array.
-
- CreateCodeEvaluationRuleRequest:
- properties:
- name:
- type: string
- docs: Human-readable deployment name.
- evaluator:
- type: CodeEvaluationRuleEvaluatorReference
- docs: |
- Code evaluator family to use.
-
- Use `name` and `type` from the evaluator endpoints.
- Langfuse resolves that family to its latest version before saving the rule.
- target:
- type: commons.EvaluationRuleTarget
- docs: Target object type to evaluate.
- enabled:
- type: boolean
- docs: Whether the deployment should be active immediately after creation.
- sampling:
- type: optional<double>
- docs: Optional sampling fraction. Defaults to `1`.
- filter:
- type: optional<list<commons.EvaluationRuleFilter>>
- docs: |
- Optional filter list.
-
- Omit or pass an empty list to evaluate all matching targets for the selected `target`.
- Each filter object must use a column that is valid for that `target`.
- For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
-
- UpdateEvaluationRuleRequest:
- docs: |
- Partial update body for an evaluation rule.
-
- Provide only the fields you want to change.
- An empty body is rejected.
-
- Practical guidance:
- - If you only want to rename the rule or change sampling, send just those fields.
- - To add, remove, or remap evaluators, send `evaluators`. It replaces the whole assignment set, so include every evaluator the rule should keep.
- - `evaluators` cannot be combined with the deprecated `evaluator`/`mapping` pair, which only ever addressed the first assignment.
- - If you change to an LLM-as-judge `evaluator`, send a fresh `mapping` unless you are certain the existing mapping still matches the prompt variables.
- - If you change `target` for an LLM-as-judge rule, usually send both `filter` and `mapping` in the same request.
- - For code evaluator rules, omit `mapping`; Langfuse stores the fixed code runtime mapping automatically.
- - If you change an experiment `datasetId` filter, call `GET /api/public/v2/datasets` and use dataset `id` values from that response.
- properties:
- name:
- type: optional<string>
- docs: Updated deployment name.
- evaluators:
- type: optional<list<CreateEvaluationRuleEvaluatorAssignment>>
- docs: |
- Full replacement of the rule's evaluator assignments: entries that are
- not listed are detached.
-
- Mutually exclusive with the deprecated `evaluator` and `mapping` fields.
- evaluator:
- type: optional<EvaluationRuleEvaluatorReference>
- docs: |
- Deprecated single-evaluator alias: updates the first assignment only. Prefer `evaluators`.
-
- Langfuse resolves the provided evaluator family to its latest version before saving the rule.
- A rule's evaluator type cannot be changed: provide `name` for an evaluator family of the rule's current type. To use a different evaluator type, create a new rule.
- target:
- type: optional<commons.EvaluationRuleTarget>
- docs: Updated target object type.
- enabled:
- type: optional<boolean>
- docs: Updated desired enabled state.
- sampling:
- type: optional<double>
- docs: Updated sampling fraction.
- filter:
- type: optional<list<commons.EvaluationRuleFilter>>
- docs: |
- Updated filter list.
-
- For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
- mapping:
- type: optional<list<commons.PromptVariableMappingInput>>
- docs: |
- Updated LLM-as-judge variable mappings.
-
- Do not send this field for code evaluator rules. Langfuse stores the fixed code runtime mapping automatically and returns it in the response.
-
- DeleteEvaluationRuleResponse:
- docs: Confirmation response returned after successful deletion.
- properties:
- message:
- type: string
- docs: Always `Evaluation rule successfully deleted`.
-
- EvaluationRuleEvaluatorReference:
- docs: |
- Evaluator family reference used when updating an evaluation rule.
-
- `name` identifies the evaluator family in the authenticated project context.
- A rule's evaluator type cannot be changed, so this reference does not accept a `type`; the family must match the rule's current evaluator type.
- properties:
- name:
- type: string
- docs: Evaluator family name.
-
- LlmAsJudgeEvaluationRuleEvaluatorReference:
- docs: LLM-as-judge evaluator family reference used when creating an evaluation rule.
- properties:
- name:
- type: string
- docs: Evaluator family name.
- type:
- type: optional<LlmAsJudgeEvaluatorType>
- docs: Evaluator type. Defaults to `llm_as_judge` when omitted.
-
- CodeEvaluationRuleEvaluatorReference:
- docs: Code evaluator family reference used when creating an evaluation rule.
- properties:
- name:
- type: string
- docs: Evaluator family name.
- type:
- type: literal<"code">
- docs: Must be `code`.
-
- LlmAsJudgeEvaluatorType:
- enum:
- - llm_as_judge
-
- EvaluationRuleEvaluator:
- docs: |
- Resolved evaluator currently used by the evaluation rule.
-
- `id` identifies the evaluator family. Evaluation runs automatically use
- the latest available evaluator version.
- properties:
- id:
- type: string
- docs: Identifier of the exact evaluator version currently used by the rule.
- name:
- type: string
- docs: Evaluator family name.
- type:
- type: commons.EvaluatorType
- docs: Evaluator type.
-
- EvaluationRuleEvaluatorAssignment:
- properties:
- evaluator:
- type: EvaluationRuleEvaluator
- mapping:
- type: nullable<list<commons.PromptVariableMappingRead>>
- docs: Rule-specific override, or `null` to inherit the evaluator version's default mapping.
-
- LegacyEvaluationRuleEvaluatorAssignment:
- docs: "**Deprecated:** Evaluator assignment on a legacy trace or dataset rule."
- properties:
- evaluator:
- type: EvaluationRuleEvaluator
- mapping:
- type: nullable<list<LegacyPromptVariableMapping>>
-
- EvaluationRuleTimeScope:
- enum:
- - NEW
- - EXISTING
-
- LegacyEvaluationRuleTarget:
- docs: "**Deprecated:** Legacy trace or dataset evaluation target."
- enum:
- - trace
- - dataset
-
- LegacyPromptVariableMapping:
- docs: "**Deprecated:** Connects one prompt variable to data from a trace, dataset item, or named observation in a legacy rule."
- properties:
- variable:
- type: string
- langfuseObject:
- type: LegacyEvaluationObject
- objectName:
- type: nullable<string>
- docs: Observation name to match, or `null` when `langfuseObject` is `trace` or `dataset_item`.
- source:
- type: string
- jsonPath:
- type: optional<string>
-
- LegacyEvaluationObject:
- docs: "**Deprecated:** Legacy Langfuse object used by a trace or dataset evaluation rule."
- enum:
- - trace
- - span
- - generation
- - event
- - agent
- - tool
- - chain
- - retriever
- - evaluator
- - embedding
- - guardrail
- - dataset_item
-
- CreateEvaluationRuleEvaluatorAssignment:
- properties:
- evaluator:
- type: CreateEvaluationRuleEvaluatorReference
- mapping:
- type: optional<list<commons.PromptVariableMappingInput>>
- docs: Optional rule-specific override. Omit to inherit the evaluator version's default mapping. Omit for code evaluators.
-
- CreateEvaluationRuleEvaluatorReference:
- properties:
- name:
- type: string
- type:
- type: optional<commons.EvaluatorType>
- docs: Defaults to `llm_as_judge`.
diff --git a/fern/apis/server/definition/unstable/evaluators.yml b/fern/apis/server/definition/unstable/evaluators.yml
deleted file mode 100644
index 86f64bee17..0000000000
--- a/fern/apis/server/definition/unstable/evaluators.yml
+++ /dev/null
@@ -1,490 +0,0 @@
-# yaml-language-server: $schema=https://raw.githubusercontent.com/fern-api/fern/main/fern.schema.json
-imports:
- commons: ./commons.yml
- errors: ./errors.yml
- pagination: ../utils/pagination.yml
-
-service:
- auth: true
- base-path: /api/public/unstable
- endpoints:
- create:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Create an evaluator in the authenticated project.
-
- Use evaluators to define **how** Langfuse should score data.
- LLM-as-a-judge evaluators define a prompt, expected structured output, and optional model configuration.
- Code evaluators define source code and a runtime language.
-
- Naming behavior:
- - If this is a new evaluator name in your project, Langfuse creates version `1`.
- - If the name already exists in your project, Langfuse creates the next version and returns it.
- - The evaluator `id` remains stable across versions.
- - Existing evaluation rules automatically use the latest evaluator version; no rule update is required.
-
- Recommended workflow:
- 1. Create the evaluator.
- 2. Read the returned `variables` array.
- 3. Read the returned `outputDefinition.dataType` so the client knows whether future scores will be numeric, boolean, or categorical.
- 4. Create one or more evaluation rules that reference the returned evaluator family using `name` and `type`.
-
- Code evaluator validation:
- - At creation, Langfuse only validates the request shape
- - The `sourceCode` itself is not executed here. It is first run (preflight-tested against a sample observation) when you link the evaluator to an evaluation rule, so runtime errors in the code surface at evaluation-rule creation, not at evaluator creation.
-
- Recovery guidance:
- - `422` with `code=evaluator_preflight_failed`: the evaluator cannot run with the resolved model configuration. Add a valid explicit `modelConfig`, or configure the project's default evaluation model, then retry the same request.
- - `400` with `code=invalid_body`: the request shape is malformed. Use the structured `details.issues` array to fix the specific fields and retry.
- - `400` with `code=invalid_body` on `outputDefinition`: for `type=llm_as_judge`, send `dataType`, `reasoning.description`, and `score.description`. Do not send `version`; it is not part of the public request shape.
- - If `type` is omitted, Langfuse treats the request as `type=llm_as_judge` for backwards compatibility. New clients should send `type` explicitly.
-
- Unstable API note:
- - This surface may evolve while the underlying evaluation data model is being redesigned.
- method: POST
- path: /evaluators
- request: CreateEvaluatorRequest
- response: Evaluator
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.MethodNotAllowedError
- - errors.ConflictError
- - errors.UnprocessableContentError
- - errors.TooManyRequestsError
- - errors.InternalServerError
- examples:
- - name: CreateEvaluatorVersion
- docs: Create a new version of an evaluator named `answer-correctness`.
- request:
- name: answer-correctness
- type: llm_as_judge
- prompt: |
- You are grading an answer.
-
- Input:
- {{input}}
-
- Output:
- {{output}}
-
- Return a score between 0 and 1.
- outputDefinition:
- dataType: NUMERIC
- reasoning:
- description: Explain why the score was assigned.
- score:
- description: Correctness score between 0 and 1.
- modelConfig:
- provider: openai
- model: gpt-4.1-mini
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- response:
- body:
- id: evaltmpl_123
- name: answer-correctness
- version: 2
- type: llm_as_judge
- prompt: |
- You are grading an answer.
-
- Input:
- {{input}}
-
- Output:
- {{output}}
-
- Return a score between 0 and 1.
- variables:
- - input
- - output
- mapping:
- - variable: input
- source: input
- - variable: output
- source: output
- outputDefinition:
- dataType: NUMERIC
- reasoning:
- description: Explain why the score was assigned.
- score:
- description: Correctness score between 0 and 1.
- modelConfig:
- provider: openai
- model: gpt-4.1-mini
- evaluationRuleCount: 0
- createdAt: "2026-03-30T09:00:00.000Z"
- updatedAt: "2026-03-30T09:00:00.000Z"
- - name: CreateTypeScriptCodeEvaluator
- docs: |
- Create a TypeScript code evaluator named `exact-match`.
-
- Send only the `evaluate` function; Langfuse injects the `EvaluationContext`,
- `EvaluationResult`, and `Score` type definitions at runtime.
- Code evaluators always expose the fixed runtime payload variables and do not take a `mapping`.
- request:
- name: exact-match
- type: code
- sourceCode: |
- function evaluate(ctx: EvaluationContext): EvaluationResult {
- const input = ctx.observation.input;
- const matchesOutput =
- input !== undefined && ctx.observation.output === input;
-
- return {
- scores: [
- {
- name: "Exact match",
- value: matchesOutput,
- dataType: "BOOLEAN",
- comment: matchesOutput
- ? "Output exactly matches the input."
- : "Output does not match the input.",
- },
- ],
- };
- }
- sourceCodeLanguage: TYPESCRIPT
- response:
- body:
- id: evaltmpl_ts_123
- name: exact-match
- version: 1
- type: code
- variables:
- - input
- - output
- - metadata
- - toolCalls
- - experimentItemExpectedOutput
- - experimentItemMetadata
- mapping: null
- sourceCode: |
- function evaluate(ctx: EvaluationContext): EvaluationResult {
- const input = ctx.observation.input;
- const matchesOutput =
- input !== undefined && ctx.observation.output === input;
-
- return {
- scores: [
- {
- name: "Exact match",
- value: matchesOutput,
- dataType: "BOOLEAN",
- comment: matchesOutput
- ? "Output exactly matches the input."
- : "Output does not match the input.",
- },
- ],
- };
- }
- sourceCodeLanguage: TYPESCRIPT
- evaluationRuleCount: 0
- createdAt: "2026-03-30T09:10:00.000Z"
- updatedAt: "2026-03-30T09:10:00.000Z"
- - name: CreatePythonCodeEvaluator
- docs: |
- Create a Python code evaluator named `exact-match`.
-
- Send only the `evaluate` function; Langfuse injects the `EvaluationContext`,
- `EvaluationResult`, and `Score` definitions at runtime.
- Code evaluators always expose the fixed runtime payload variables and do not take a `mapping`.
- request:
- name: exact-match
- type: code
- sourceCode: |
- def evaluate(ctx: EvaluationContext) -> EvaluationResult:
- """Evaluates one observation and returns one or more Langfuse scores."""
- input = ctx.observation.input
- matches_output = input is not None and ctx.observation.output == input
-
- return EvaluationResult(
- scores=[
- Score(
- name="Exact match",
- value=matches_output,
- data_type="BOOLEAN",
- comment=(
- "Output exactly matches the input."
- if matches_output
- else "Output does not match the input."
- ),
- )
- ]
- )
- sourceCodeLanguage: PYTHON
- response:
- body:
- id: evaltmpl_py_123
- name: exact-match
- version: 1
- type: code
- variables:
- - input
- - output
- - metadata
- - toolCalls
- - experimentItemExpectedOutput
- - experimentItemMetadata
- mapping: null
- sourceCode: |
- def evaluate(ctx: EvaluationContext) -> EvaluationResult:
- """Evaluates one observation and returns one or more Langfuse scores."""
- input = ctx.observation.input
- matches_output = input is not None and ctx.observation.output == input
-
- return EvaluationResult(
- scores=[
- Score(
- name="Exact match",
- value=matches_output,
- data_type="BOOLEAN",
- comment=(
- "Output exactly matches the input."
- if matches_output
- else "Output does not match the input."
- ),
- )
- ]
- )
- sourceCodeLanguage: PYTHON
- evaluationRuleCount: 0
- createdAt: "2026-03-30T09:15:00.000Z"
- updatedAt: "2026-03-30T09:15:00.000Z"
-
- list:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- List the evaluators available to the authenticated project.
-
- Important behavior:
- - This endpoint returns the latest version of each available evaluator.
- - Every evaluator is owned by the authenticated project.
- method: GET
- path: /evaluators
- request:
- name: ListEvaluatorsRequest
- query-parameters:
- page:
- type: optional<integer>
- docs: 1-based page number. Defaults to `1`.
- limit:
- type: optional<integer>
- docs: Maximum number of items per page. Defaults to `50`.
- response: Evaluators
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.MethodNotAllowedError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
- get:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Get one evaluator by `id`.
-
- This endpoint always returns the evaluator's latest version. Use it when you want the current prompt, output definition, model configuration, and derived variables for the evaluator you plan to use in an evaluation rule.
- method: GET
- path: /evaluators/{evaluatorId}
- path-parameters:
- evaluatorId:
- type: string
- docs: Evaluator identifier returned by the evaluator endpoints.
- response: Evaluator
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.NotFoundError
- - errors.MethodNotAllowedError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
- delete:
- availability:
- status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
- docs: |
- Delete an evaluator.
-
- Important behavior:
- - This deletes the evaluator including all of its stored versions.
- - Evaluation rule assignments referencing the evaluator are also deleted.
- - Scores already produced by the evaluator are not deleted.
- method: DELETE
- path: /evaluators/{evaluatorId}
- path-parameters:
- evaluatorId:
- type: string
- docs: Evaluator identifier returned by the evaluator endpoints.
- response: DeleteEvaluatorResponse
- errors:
- - errors.BadRequestError
- - errors.UnauthorizedError
- - errors.AccessDeniedError
- - errors.NotFoundError
- - errors.MethodNotAllowedError
- - errors.TooManyRequestsError
- - errors.InternalServerError
-
-types:
- DeleteEvaluatorResponse:
- docs: Confirmation response returned after successful deletion.
- properties:
- message:
- type: string
- docs: Always `Evaluator successfully deleted`.
-
- Evaluator:
- docs: |
- One evaluator that can be used for scoring.
-
- An evaluator describes **how** to score data.
-
- It does not define **which** live objects are evaluated. That is the job of `evaluation-rules`.
-
- For agent clients, the most important fields are:
- - `type`: determines which evaluator fields are present
- - `variables`: for LLM evaluators, use these exact names when building the evaluation-rule `mapping` array. LLM evaluators require every variable to be mapped. Code evaluators always expose the fixed runtime payload fields and Langfuse maps them automatically.
-
- Versioning behavior:
- - `GET /evaluators` returns the latest version of each available evaluator.
- - `GET /evaluators/{id}` returns the latest version.
- - `id` identifies the evaluator family and remains stable when a new version is created.
- - Evaluation rules always run against the latest version for the selected evaluator name within the same source (`project` or `managed`).
- discriminant: type
- union:
- llm_as_judge:
- type: LlmAsJudgeEvaluator
- code:
- type: CodeEvaluator
-
- EvaluatorBase:
- properties:
- id:
- type: string
- docs: Stable identifier of this evaluator across all versions.
- name:
- type: string
- docs: Evaluator name.
- version:
- type: integer
- docs: Version number of this evaluator.
- variables:
- type: list<string>
- docs: |
- Variables that can be mapped when creating an evaluation rule.
-
- LLM evaluators require every variable to be mapped exactly once. Code evaluators always expose the fixed runtime payload fields and Langfuse maps them automatically.
- mapping:
- type: nullable<list<commons.PromptVariableMappingRead>>
- docs: |
- Default variable mapping for this evaluator version, or `null` when no default is configured.
-
- An entry's `source` is `null` when that variable was never fully configured, and sources
- are not restricted by rule `target` here, because the default is stored on the evaluator
- rather than on any one rule.
- evaluationRuleCount:
- type: integer
- docs: Number of evaluation rules in the project that currently use this evaluator.
- createdAt:
- type: datetime
- docs: Timestamp when this evaluator was created.
- updatedAt:
- type: datetime
- docs: Timestamp when this evaluator was last updated.
-
- LlmAsJudgeEvaluator:
- extends: EvaluatorBase
- properties:
- prompt:
- type: string
- docs: Prompt template used during evaluation.
- outputDefinition:
- type: commons.PublicEvaluatorOutputDefinition
- docs: |
- Structured output schema returned by this evaluator.
-
- Responses always include `dataType` and omit the internal output-definition `version`.
- Use `dataType` to decide how future scores should be interpreted.
- modelConfig:
- type: nullable<commons.EvaluatorModelConfig>
- docs: Explicit model configuration, or `null` when the project default evaluation model is used.
-
- CodeEvaluator:
- extends: EvaluatorBase
- properties:
- sourceCode:
- type: string
- docs: Source code executed for each matched observation.
- sourceCodeLanguage:
- type: commons.CodeEvaluatorSourceCodeLanguage
- docs: Runtime language for `sourceCode`.
-
- Evaluators:
- properties:
- data:
- type: list<Evaluator>
- meta:
- type: pagination.MetaResponse
-
- CreateEvaluatorRequest:
- docs: |
- Request body for creating an evaluator.
-
- If the same `name` already exists in your project, Langfuse creates the next version and returns it.
- Existing evaluation rules automatically use the latest evaluator version.
- If `type` is omitted, Langfuse defaults it to `llm_as_judge` for backwards compatibility.
- discriminant: type
- union:
- llm_as_judge:
- type: CreateLlmAsJudgeEvaluatorRequest
- code:
- type: CreateCodeEvaluatorRequest
-
- CreateLlmAsJudgeEvaluatorRequest:
- properties:
- name:
- type: string
- docs: Evaluator name within the authenticated project.
- prompt:
- type: string
- docs: Prompt template used by the evaluator.
- outputDefinition:
- type: commons.EvaluatorOutputDefinition
- docs: |
- Structured output schema the evaluator must return.
-
- Always send `dataType`.
- Do not send `version`; it is an internal storage detail and not part of the public request contract.
- modelConfig:
- type: optional<nullable<commons.EvaluatorModelConfig>>
- docs: Optional explicit model configuration. Omit or set to `null` to use the project default evaluation model.
- mapping:
- type: optional<list<commons.PromptVariableMappingInput>>
- docs: Optional default variable mapping inherited by rule assignments that do not provide an override.
-
- CreateCodeEvaluatorRequest:
- properties:
- name:
- type: string
- docs: Evaluator name within the authenticated project.
- sourceCode:
- type: string
- docs: Code executed for each matched observation.
- sourceCodeLanguage:
- type: commons.CodeEvaluatorSourceCodeLanguage
- docs: Runtime language for `sourceCode`.
diff --git a/knip.jsonc b/knip.jsonc
index 93e2568e41..1cb5d48f1d 100644
--- a/knip.jsonc
+++ b/knip.jsonc
@@ -7,8 +7,6 @@
"workspaces": {
"packages/shared": {
"ignoreFiles": [
- // Standalone renderer invoked by ClickHouse migration shell scripts and Docker builds.
- "clickhouse/scripts/prepare-migrations.mjs",
// Standalone helper invoked manually as documented in the adjacent framework-traces README.
"scripts/seeder/utils/framework-traces/merge-observations.ts",
],
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index d1b1951f8c..e9e5dcc725 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -760,9 +760,6 @@ importers:
json-schema-faker:
specifier: 0.6.1
version: 0.6.1
- jsonpath-plus:
- specifier: 10.3.0
- version: 10.3.0
langfuse:
specifier: 3.38.4
version: 3.38.4
@@ -19318,17 +19315,6 @@ snapshots:
transitivePeerDependencies:
- supports-color
- dependencies:
- debug: 3.2.7([email protected])
- optionalDependencies:
- eslint-import-resolver-node: 0.3.10([email protected])
- transitivePeerDependencies:
- - supports-color
-
dependencies:
debug: 3.2.7([email protected])
@@ -19384,35 +19370,6 @@ snapshots:
- eslint-import-resolver-webpack
- supports-color
- dependencies:
- '@rtsao/scc': 1.1.0
- array-includes: 3.1.9
- array.prototype.findlastindex: 1.2.6
- array.prototype.flat: 1.3.3
- array.prototype.flatmap: 1.3.3
- debug: 3.2.7([email protected])
- doctrine: 2.1.0
- eslint-import-resolver-node: 0.3.10([email protected])
- hasown: 2.0.4
- is-core-module: 2.16.2
- is-glob: 4.0.3
- minimatch: 3.1.5
- object.fromentries: 2.0.8
- object.groupby: 1.0.3
- object.values: 1.2.1
- semver: 6.3.1
- string.prototype.trimend: 1.0.9
- tsconfig-paths: 3.15.0
- optionalDependencies:
- transitivePeerDependencies:
- - eslint-import-resolver-typescript
- - eslint-import-resolver-webpack
- - supports-color
-
dependencies:
aria-query: 5.3.2
diff --git a/web/.structure-baseline.json b/web/.structure-baseline.json
index adb40eeb2d..ae4a050343 100644
--- a/web/.structure-baseline.json
+++ b/web/.structure-baseline.json
@@ -969,17 +969,6 @@
"src/features/evals/server/evalTemplateDeletion.ts -> src/features/audit-logs/auditLog.ts",
"src/features/evals/server/router.ts -> src/features/audit-logs/auditLog.ts",
"src/features/evals/server/router.ts -> src/features/rbac/utils/checkProjectAccess.ts",
- "src/features/evals/server/unstable-public-api/adapters.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
- "src/features/evals/server/unstable-public-api/evaluation-rule-service.ts -> src/features/audit-logs/auditLog.ts",
- "src/features/evals/server/unstable-public-api/evaluation-rule-service.ts -> src/features/public-api/server/unstable-public-api-error-contract.ts",
- "src/features/evals/server/unstable-public-api/evaluation-rule-service.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",
- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/audit-logs/auditLog.ts",
- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/public-api/server/unstable-public-api-error-contract.ts",
- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/public-api/types/unstable-evaluators.ts",
- "src/features/evals/server/unstable-public-api/evaluator-service.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
- "src/features/evals/server/unstable-public-api/types.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
- "src/features/evals/server/unstable-public-api/validation.ts -> src/features/public-api/server/unstable-public-api-error-contract.ts",
- "src/features/evals/server/unstable-public-api/validation.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
"src/features/events/components/CategoryPresetChips.tsx -> src/features/posthog-analytics/usePostHogClientCapture.ts",
"src/features/events/components/EventsTable.tsx -> src/features/batch-actions/components/AddObservationsToDatasetDialog/index.tsx",
"src/features/events/components/EventsTable.tsx -> src/features/batch-actions/components/RunEvaluationDialog/index.tsx",
@@ -1198,28 +1187,6 @@
"src/features/mcp/features/datasets/tools/upsertDataset.ts -> src/features/public-api/types/datasets.ts",
"src/features/mcp/features/datasets/tools/upsertDatasetItem.ts -> src/features/datasets/server/publicDatasetService.ts",
"src/features/mcp/features/datasets/tools/upsertDatasetItem.ts -> src/features/public-api/types/datasets.ts",
- "src/features/mcp/features/evals/schema.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",
- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
- "src/features/mcp/features/evals/tools/deleteEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/deleteEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",
- "src/features/mcp/features/evals/tools/deleteEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/deleteEvaluator.ts -> src/features/public-api/types/unstable-evaluators.ts",
- "src/features/mcp/features/evals/tools/getEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/getEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",
- "src/features/mcp/features/evals/tools/getEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/getEvaluator.ts -> src/features/public-api/types/unstable-evaluators.ts",
- "src/features/mcp/features/evals/tools/listEvaluationRules.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/listEvaluationRules.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",
- "src/features/mcp/features/evals/tools/listEvaluators.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/listEvaluators.ts -> src/features/public-api/types/unstable-evaluators.ts",
- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/public-api/types/unstable-evaluation-rules.ts",
- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/public-api/types/unstable-evaluators.ts",
- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/public-api/types/unstable-public-evals-contract.ts",
"src/features/mcp/features/experiments/schema.ts -> src/features/public-api/types/experiments.ts",
"src/features/mcp/features/experiments/tools/listExperimentItems.ts -> src/features/experiments/server/public/service.ts",
"src/features/mcp/features/experiments/tools/listExperimentItems.ts -> src/features/public-api/types/experiments.ts",
@@ -1830,15 +1797,6 @@
"src/features/mcp/features/datasets/tools/listDatasets.ts -> src/features/datasets/server/publicDatasetService.ts",
"src/features/mcp/features/datasets/tools/upsertDataset.ts -> src/features/datasets/server/publicDatasetService.ts",
"src/features/mcp/features/datasets/tools/upsertDatasetItem.ts -> src/features/datasets/server/publicDatasetService.ts",
- "src/features/mcp/features/evals/tools/createEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/deleteEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/deleteEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/getEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/getEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/listEvaluationRules.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/listEvaluators.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/updateEvaluationRule.ts -> src/features/evals/server/unstable-public-api/index.ts",
- "src/features/mcp/features/evals/tools/upsertEvaluator.ts -> src/features/evals/server/unstable-public-api/index.ts",
"src/features/mcp/features/experiments/tools/listExperimentItems.ts -> src/features/experiments/server/public/service.ts",
"src/features/mcp/features/experiments/tools/listExperiments.ts -> src/features/experiments/server/public/service.ts",
"src/features/mcp/features/feedback/tools/submitFeedback.ts -> src/features/feedback/server/FeedbackService.ts",
diff --git a/web/AGENTS.md b/web/AGENTS.md
index 7e914bbc19..2b3a15e144 100644
--- a/web/AGENTS.md
+++ b/web/AGENTS.md
@@ -19,7 +19,7 @@
- tRPC router registry: `src/server/api/root.ts`
- tRPC routers: `src/server/api/routers/*`, `src/features/*/server/*`
- Public REST API routes: `src/pages/api/public/*`
-- Unstable public eval APIs: `src/pages/api/public/unstable/{evaluators,evaluation-rules}/*`
+- Public eval APIs: `src/pages/api/public/v2/{evaluators,evaluation-rules}/*`
- Feature modules: `src/features/*`
- Reusable UI components: `src/components/*`
- Tests:
diff --git a/web/package.json b/web/package.json
index 043ecfc929..41f055d906 100644
--- a/web/package.json
+++ b/web/package.json
@@ -128,7 +128,6 @@
"https-proxy-agent": "^7.0.6",
"ioredis": "^5.10.1",
"json-schema-faker": "0.6.1",
- "jsonpath-plus": "10.3.0",
"langfuse": "3.38.4",
"lodash": "^4.18.1",
"lucide-react": "^0.552.0",
diff --git a/web/public/generated/api/openapi.yml b/web/public/generated/api/openapi.yml
index ad7c162133..819726ea60 100644
--- a/web/public/generated/api/openapi.yml
+++ b/web/public/generated/api/openapi.yml
@@ -8571,2893 +8571,36 @@ paths:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security: *ref_0
- /api/public/unstable/evaluation-rules:
- post:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
- are unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Create an evaluation rule.
-
-
- An evaluation rule defines **what** incoming data should be evaluated
- and **how prompt variables should be populated** from that data.
-
-
- Use this resource after choosing an evaluator from the evaluator
- endpoints.
-
-
- Key rules:
-
- - `name` must be unique within the project for public evaluation rules
-
- - `target` must be `observation` or `experiment`
-
- - provide either the compatibility `evaluator` field or the new
- `evaluators` array, never both
-
- - each evaluator `name` + `type` must identify an existing evaluator
- family returned by the evaluator endpoints
-
- - Langfuse resolves that family to its latest version before saving the
- evaluation rule
-
- - for `target=experiment`, use dataset `id` values from `GET
- /api/public/v2/datasets` when filtering by `datasetId`
-
- - an omitted LLM-as-judge assignment mapping inherits the evaluator
- version's default mapping
-
- - the effective mapping must map every evaluator prompt variable exactly
- once
-
- - for `code` evaluators, Langfuse uses the fixed code runtime mapping;
- omit `mapping` in create and update requests
-
- - for user-provided `llm_as_judge` mappings, `expected_output` and
- `experiment_item_metadata` are only valid for `target=experiment`
-
- - if `enabled=true`, Langfuse validates that the referenced evaluator
- can currently run
-
- - at most 500 evaluation rules can be effectively active in one project
- at the same time (enforced identically by the API, the MCP tools, and
- the app)
-
-
- If an evaluation rule with the same `name` already exists in the
- project, the API returns `409`.
-
- In that case, update the existing resource with `PATCH
- /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of
- creating a second one.
-
-
- If enabling this resource would exceed the 500-active limit, the API
- also returns `409`.
-
- In that case, disable or pause another active evaluation rule before
- enabling a new one.
-
-
- Current scope:
-
- - evaluation rules are live-ingestion rules only
-
- - they do not trigger historical backfills
-
-
- Recovery guidance:
-
- - `400 invalid_filter_value`: fix the filter `column` or `value` using
- `details.column`, `details.invalidValues`, and `details.allowedValues`
-
- - `400 invalid_filter_value` with `details.column=datasetId`: call `GET
- /api/public/v2/datasets`, then retry with dataset `id` values from that
- response
-
- - `400 missing_variable_mapping`: for `llm_as_judge` evaluators, fetch
- the evaluator again and make sure every variable in `variables` appears
- exactly once in `mapping`
-
- - `400 duplicate_variable_mapping`: remove repeated mappings for the
- same variable
-
- - `400 invalid_variable_mapping`: for `llm_as_judge`, switch to a valid
- `source` for the selected `target`, or fix the variable name
-
- - `400 invalid_json_path`: remove or correct the `jsonPath`
-
- - `422 evaluator_preflight_failed`: the selected evaluator cannot run
- with the resolved model configuration. Fix the evaluator/default model
- setup, then retry the create request.
- operationId: unstable_evaluationRules_create
- tags:
- - UnstableEvaluationRules
- parameters: []
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableEvaluationRule'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '409':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '422':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- requestBody:
- required: true
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableCreateEvaluationRuleRequest'
- deprecated: true
- get:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
- are unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- List evaluation rules in the authenticated project.
-
-
- This includes legacy `trace` and `dataset` rules so they can be
- inspected and migrated to v4 rules. Legacy rules are read-only through
- this API; create, update, and delete continue to support only
- `observation` and `experiment` rules.
- operationId: unstable_evaluationRules_list
- tags:
- - UnstableEvaluationRules
- parameters:
- - name: page
- in: query
- description: 1-based page number. Defaults to `1`.
- required: false
- schema:
- type: integer
- nullable: true
- - name: limit
- in: query
- description: Maximum number of items per page. Defaults to `50`.
- required: false
- schema:
- type: integer
- nullable: true
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableEvaluationRules'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- deprecated: true
- /api/public/unstable/evaluation-rules/{evaluationRuleId}:
- get:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
- are unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Get one evaluation rule by its identifier.
-
-
- Use this endpoint to inspect the current evaluator, target, mapping,
- filters, execution timing, and effective runtime status. Legacy `trace`
- and `dataset` rules are returned for migration and are read-only through
- this API.
- operationId: unstable_evaluationRules_get
- tags:
- - UnstableEvaluationRules
- parameters:
- - name: evaluationRuleId
- in: path
- description: >-
- Evaluation rule identifier returned by the evaluation rule endpoints.
- required: true
- schema:
- type: string
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableReadableEvaluationRule'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- deprecated: true
- patch:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
- are unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Update an evaluation rule.
-
-
- Typical uses:
-
- - enable or disable live execution
-
- - switch to another evaluator
-
- - adjust sampling
-
- - change filters
-
- - update LLM-as-judge variable mappings
-
-
- Important behavior:
-
- - provide only the fields you want to change
-
- - if you provide `evaluator`, Langfuse resolves that evaluator family to
- its latest version before saving
-
- - changing `target`, `filter`, or an LLM-as-judge `mapping` must still
- produce a valid target-specific configuration
-
- - if you change `target` for an LLM-as-judge rule, also send a
- compatible `filter` and `mapping` in the same request unless the
- existing ones are still valid for the new target
-
- - for `code` evaluator rules, omit `mapping`; Langfuse stores the fixed
- code runtime mapping automatically
-
- - if the resulting config is enabled, Langfuse re-validates that the
- selected evaluator can run
-
- - if the update would move a non-active evaluation rule into the active
- state and the project already has 500 active evaluation rules, the API
- returns `409`
-
-
- Recovery guidance:
-
- - if an LLM-as-judge update fails with `missing_variable_mapping` or
- `invalid_variable_mapping` after changing `evaluator` or `target`,
- resend the request with a complete new `mapping`
-
- - if the update fails with `invalid_filter_value` after changing
- `target`, resend the request with a target-compatible `filter`
- operationId: unstable_evaluationRules_update
- tags:
- - UnstableEvaluationRules
- parameters:
- - name: evaluationRuleId
- in: path
- description: Evaluation rule identifier.
- required: true
- schema:
- type: string
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableReadableV2EvaluationRule'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '422':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- requestBody:
- required: true
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableUpdateEvaluationRuleRequest'
- deprecated: true
- delete:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
- are unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Delete an evaluation rule.
-
-
- This removes the live-ingestion rule only. It does not delete the
- referenced evaluator.
- operationId: unstable_evaluationRules_delete
- tags:
- - UnstableEvaluationRules
- parameters:
- - name: evaluationRuleId
- in: path
- description: Evaluation rule identifier.
- required: true
- schema:
- type: string
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableDeleteEvaluationRuleResponse'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- deprecated: true
- /api/public/unstable/evaluators:
- post:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluators` API instead. Self-hosted deployments are
- unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Create an evaluator in the authenticated project.
-
-
- Use evaluators to define **how** Langfuse should score data.
-
- LLM-as-a-judge evaluators define a prompt, expected structured output,
- and optional model configuration.
-
- Code evaluators define source code and a runtime language.
-
-
- Naming behavior:
-
- - If this is a new evaluator name in your project, Langfuse creates
- version `1`.
-
- - If the name already exists in your project, Langfuse creates the next
- version and returns it.
-
- - The evaluator `id` remains stable across versions.
-
- - Existing evaluation rules automatically use the latest evaluator
- version; no rule update is required.
-
-
- Recommended workflow:
-
- 1. Create the evaluator.
-
- 2. Read the returned `variables` array.
-
- 3. Read the returned `outputDefinition.dataType` so the client knows
- whether future scores will be numeric, boolean, or categorical.
-
- 4. Create one or more evaluation rules that reference the returned
- evaluator family using `name` and `type`.
-
-
- Code evaluator validation:
-
- - At creation, Langfuse only validates the request shape
-
- - The `sourceCode` itself is not executed here. It is first run
- (preflight-tested against a sample observation) when you link the
- evaluator to an evaluation rule, so runtime errors in the code surface
- at evaluation-rule creation, not at evaluator creation.
-
-
- Recovery guidance:
-
- - `422` with `code=evaluator_preflight_failed`: the evaluator cannot run
- with the resolved model configuration. Add a valid explicit
- `modelConfig`, or configure the project's default evaluation model, then
- retry the same request.
-
- - `400` with `code=invalid_body`: the request shape is malformed. Use
- the structured `details.issues` array to fix the specific fields and
- retry.
-
- - `400` with `code=invalid_body` on `outputDefinition`: for
- `type=llm_as_judge`, send `dataType`, `reasoning.description`, and
- `score.description`. Do not send `version`; it is not part of the public
- request shape.
-
- - If `type` is omitted, Langfuse treats the request as
- `type=llm_as_judge` for backwards compatibility. New clients should send
- `type` explicitly.
-
-
- Unstable API note:
-
- - This surface may evolve while the underlying evaluation data model is
- being redesigned.
- operationId: unstable_evaluators_create
- tags:
- - UnstableEvaluators
- parameters: []
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableEvaluator'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '409':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '422':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- requestBody:
- required: true
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableCreateEvaluatorRequest'
- deprecated: true
- get:
- description: |-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
- List the evaluators available to the authenticated project.
-
- Important behavior:
- - This endpoint returns the latest version of each available evaluator.
- - Every evaluator is owned by the authenticated project.
- operationId: unstable_evaluators_list
- tags:
- - UnstableEvaluators
- parameters:
- - name: page
- in: query
- description: 1-based page number. Defaults to `1`.
- required: false
- schema:
- type: integer
- nullable: true
- - name: limit
- in: query
- description: Maximum number of items per page. Defaults to `50`.
- required: false
- schema:
- type: integer
- nullable: true
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableEvaluators'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- deprecated: true
- /api/public/unstable/evaluators/{evaluatorId}:
- get:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluators` API instead. Self-hosted deployments are
- unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Get one evaluator by `id`.
-
-
- This endpoint always returns the evaluator's latest version. Use it when
- you want the current prompt, output definition, model configuration, and
- derived variables for the evaluator you plan to use in an evaluation
- rule.
- operationId: unstable_evaluators_get
- tags:
- - UnstableEvaluators
- parameters:
- - name: evaluatorId
- in: path
- description: Evaluator identifier returned by the evaluator endpoints.
- required: true
- schema:
- type: string
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableEvaluator'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- deprecated: true
- delete:
- description: >-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on September 4, 2026. Use the stable
- `/api/public/v2/evaluators` API instead. Self-hosted deployments are
- unaffected by this date; the endpoint becomes unavailable when they
- upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
- guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
-
-
- Delete an evaluator.
-
-
- Important behavior:
-
- - This deletes the evaluator including all of its stored versions.
-
- - Evaluation rule assignments referencing the evaluator are also
- deleted.
-
- - Scores already produced by the evaluator are not deleted.
- operationId: unstable_evaluators_delete
- tags:
- - UnstableEvaluators
- parameters:
- - name: evaluatorId
- in: path
- description: Evaluator identifier returned by the evaluator endpoints.
- required: true
- schema:
- type: string
- responses:
- '200':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstableDeleteEvaluatorResponse'
- '400':
- description: ''
- content:
- application/json:
- schema: {}
- '401':
- description: ''
- content:
- application/json:
- schema: {}
- '403':
- description: ''
- content:
- application/json:
- schema: {}
- '404':
- description: ''
- content:
- application/json:
- schema: {}
- '405':
- description: ''
- content:
- application/json:
- schema: {}
- '429':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- '500':
- description: ''
- content:
- application/json:
- schema:
- $ref: '#/components/schemas/unstablePublicApiError'
- security: *ref_0
- deprecated: true
-components:
- schemas:
- AnnotationQueueStatus:
- title: AnnotationQueueStatus
- type: string
- enum:
- - PENDING
- - COMPLETED
- AnnotationQueueObjectType:
- title: AnnotationQueueObjectType
- type: string
- enum:
- - TRACE
- - OBSERVATION
- - SESSION
- AnnotationQueue:
- title: AnnotationQueue
- type: object
- properties:
- id:
- type: string
- name:
- type: string
- description:
- type: string
- nullable: true
- scoreConfigIds:
- type: array
- items:
- type: string
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- required:
- - id
- - name
- - description
- - scoreConfigIds
- - createdAt
- - updatedAt
- AnnotationQueueItem:
- title: AnnotationQueueItem
- type: object
- properties:
- id:
- type: string
- queueId:
- type: string
- objectId:
- type: string
- objectType:
- $ref: '#/components/schemas/AnnotationQueueObjectType'
- status:
- $ref: '#/components/schemas/AnnotationQueueStatus'
- completedAt:
- type: string
- format: date-time
- nullable: true
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- required:
- - id
- - queueId
- - objectId
- - objectType
- - status
- - createdAt
- - updatedAt
- PaginatedAnnotationQueues:
- title: PaginatedAnnotationQueues
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/AnnotationQueue'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- required:
- - data
- - meta
- PaginatedAnnotationQueueItems:
- title: PaginatedAnnotationQueueItems
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/AnnotationQueueItem'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- required:
- - data
- - meta
- CreateAnnotationQueueRequest:
- title: CreateAnnotationQueueRequest
- type: object
- properties:
- name:
- type: string
- description:
- type: string
- nullable: true
- scoreConfigIds:
- type: array
- items:
- type: string
- required:
- - name
- - scoreConfigIds
- CreateAnnotationQueueItemRequest:
- title: CreateAnnotationQueueItemRequest
- type: object
- properties:
- objectId:
- type: string
- objectType:
- $ref: '#/components/schemas/AnnotationQueueObjectType'
- status:
- $ref: '#/components/schemas/AnnotationQueueStatus'
- nullable: true
- description: Defaults to PENDING for new queue items
- required:
- - objectId
- - objectType
- UpdateAnnotationQueueItemRequest:
- title: UpdateAnnotationQueueItemRequest
- type: object
- properties:
- status:
- $ref: '#/components/schemas/AnnotationQueueStatus'
- nullable: true
- DeleteAnnotationQueueItemResponse:
- title: DeleteAnnotationQueueItemResponse
- type: object
- properties:
- success:
- type: boolean
- message:
- type: string
- required:
- - success
- - message
- AnnotationQueueAssignmentRequest:
- title: AnnotationQueueAssignmentRequest
- type: object
- properties:
- userId:
- type: string
- required:
- - userId
- DeleteAnnotationQueueAssignmentResponse:
- title: DeleteAnnotationQueueAssignmentResponse
- type: object
- properties:
- success:
- type: boolean
- required:
- - success
- CreateAnnotationQueueAssignmentResponse:
- title: CreateAnnotationQueueAssignmentResponse
- type: object
- properties:
- userId:
- type: string
- queueId:
- type: string
- projectId:
- type: string
- required:
- - userId
- - queueId
- - projectId
- BlobStorageIntegrationType:
- title: BlobStorageIntegrationType
- type: string
- enum:
- - S3
- - S3_COMPATIBLE
- - AZURE_BLOB_STORAGE
- BlobStorageIntegrationFileType:
- title: BlobStorageIntegrationFileType
- type: string
- enum:
- - JSON
- - CSV
- - JSONL
- - PARQUET
- description: >-
- File format for exported data. `PARQUET` is a columnar binary format
- encoded and compressed by the storage engine; gzip compression does not
- apply to it. Note that the model-price columns (`input_price`,
- `output_price`, `total_price`) are not included in Parquet observation
- exports.
- BlobStorageIntegrationFileTypeResponse:
- title: BlobStorageIntegrationFileTypeResponse
- type: string
- enum:
- - JSON
- - CSV
- - JSONL
- - PARQUET
- description: File type reported for an existing integration.
- BlobStorageExportMode:
- title: BlobStorageExportMode
- type: string
- enum:
- - FULL_HISTORY
- - FROM_TODAY
- - FROM_CUSTOM_DATE
- BlobStorageExportFrequency:
- title: BlobStorageExportFrequency
- type: string
- enum:
- - every_20_minutes
- - hourly
- - daily
- - weekly
- BlobStorageExportSource:
- title: BlobStorageExportSource
- type: string
- enum:
- - LEGACY_TRACES_OBSERVATIONS
- - OBSERVATIONS_V2
- - LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS
- description: >-
- What data the integration exports.
-
- - `LEGACY_TRACES_OBSERVATIONS`: traces, observations, and scores tables.
- Observation columns are controlled by `exportFieldGroups`; field groups
- without a counterpart in this data model (e.g. `trace_context`) are
- omitted.
-
- - `OBSERVATIONS_V2`: same data model as the
- `/api/public/v2/observations` endpoint, plus scores. Columns are
- controlled by `exportFieldGroups`.
-
- - `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`: both sets. Observation
- columns of both portions are controlled by `exportFieldGroups`.
-
-
- **Note:** which sources a deployment accepts depends on how far it has
- moved to the v4 data model. `OBSERVATIONS_V2` and the
- enriched-observations portion of
- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the enriched observations
- table, so they require a deployment that already populates it.
- `LEGACY_TRACES_OBSERVATIONS` and the legacy portion of
- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the legacy traces and
- observations tables, so they require a deployment that still populates
- those. A deployment part-way through the migration populates both and
- accepts every source. Selecting a source the deployment cannot serve is
- rejected with `400`, rather than exporting an empty result. See
- https://langfuse.com/docs/v4.
- BlobStorageExportFieldGroup:
- title: BlobStorageExportFieldGroup
- type: string
- enum:
- - core
- - basic
- - time
- - io
- - metadata
- - model
- - usage
- - prompt
- - metrics
- - tools
- - trace_context
- description: >-
- Field group selecting which observation columns are included in the
- export. Applies to all export sources; groups without a counterpart in
- the legacy data model (e.g. `trace_context`) are omitted from the legacy
- observations export.
- CreateBlobStorageIntegrationRequest:
- title: CreateBlobStorageIntegrationRequest
- type: object
- properties:
- projectId:
- type: string
- description: ID of the project in which to configure the blob storage integration
- type:
- $ref: '#/components/schemas/BlobStorageIntegrationType'
- bucketName:
- type: string
- description: >-
- Name of the storage bucket. For AZURE_BLOB_STORAGE, must be a valid
- Azure container name (3-63 chars, lowercase letters, numbers, and
- hyphens only, must start and end with a letter or number, no
- consecutive hyphens).
- endpoint:
- type: string
- nullable: true
- description: Custom endpoint URL (required for S3_COMPATIBLE type)
- region:
- type: string
- description: >-
- Storage region used by S3-compatible clients (AWS, GCS, Cloudflare
- R2, MinIO, Azure location IDs such as eastus, OCI). Leading and
- trailing whitespace is removed. The remaining value must be 1-63
- letters, numbers, or hyphens, and cannot start or end with a hyphen.
- Examples: us-east-1, europe-west1, eastus, auto.
- accessKeyId:
- type: string
- nullable: true
- description: Access key ID for authentication
- secretAccessKey:
- type: string
- nullable: true
- description: Secret access key for authentication (will be encrypted when stored)
- prefix:
- type: string
- nullable: true
- description: >-
- Path prefix for exported files (must end with forward slash if
- provided)
- exportFrequency:
- $ref: '#/components/schemas/BlobStorageExportFrequency'
- enabled:
- type: boolean
- description: Whether the integration is active
- forcePathStyle:
- type: boolean
- description: Use path-style URLs for S3 requests
- fileType:
- $ref: '#/components/schemas/BlobStorageIntegrationFileType'
- exportMode:
- $ref: '#/components/schemas/BlobStorageExportMode'
- exportStartDate:
- type: string
- format: date-time
- nullable: true
- description: >-
- Custom start date for exports (required when exportMode is
- FROM_CUSTOM_DATE). Must not be in the future (27 h tolerance for
- timezone differences).
- compressed:
- type: boolean
- nullable: true
- description: >-
- Enable gzip compression for exported files (.csv.gz, .json.gz,
- .jsonl.gz). Defaults to true.
- exportSource:
- $ref: '#/components/schemas/BlobStorageExportSource'
- nullable: true
- description: >-
- Data to export. When omitted on update, the existing value is
- preserved. When omitted on create, the default is `OBSERVATIONS_V2`
- on Langfuse Cloud, and on self-hosted deployments
- `LEGACY_TRACES_OBSERVATIONS` — or `OBSERVATIONS_V2` where the
- deployment no longer populates the legacy tables. The default is
- never a source the deployment cannot serve. Required when
- `exportFieldGroups` is provided.
-
-
- **Cloud-only project deprecation gate (effective 2026-05-20):** For
- projects created on or after 2026-05-20 on Langfuse Cloud,
- `LEGACY_TRACES_OBSERVATIONS` and
- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP
- 400. Use `OBSERVATIONS_V2` for all new integrations. Self-hosted
- deployments are unaffected.
-
-
- **Cloud-only integration deprecation gate (effective 2026-06-22):**
- On Langfuse Cloud, legacy export sources are only accepted for blob
- storage integrations created before 2026-06-22, regardless of
- project age. Requests that would create a new integration with
- `LEGACY_TRACES_OBSERVATIONS` or
- `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP
- 400. Use `OBSERVATIONS_V2` instead. Self-hosted deployments are
- unaffected.
- exportFieldGroups:
- type: array
- items:
- $ref: '#/components/schemas/BlobStorageExportFieldGroup'
- nullable: true
- description: >-
- Field groups to include in each exported observation row. Applies to
- all export sources; must include `core` if provided. When omitted on
- create, the column default (all groups) applies. When omitted on
- update, the existing value is preserved.
-
-
- `exportFieldGroups` requires `exportSource` to be provided in the
- same request.
- required:
- - projectId
- - type
- - bucketName
- - region
- - exportFrequency
- - enabled
- - forcePathStyle
- - fileType
- - exportMode
- BlobStorageIntegrationResponse:
- title: BlobStorageIntegrationResponse
- type: object
- properties:
- id:
- type: string
- projectId:
- type: string
- type:
- $ref: '#/components/schemas/BlobStorageIntegrationType'
- bucketName:
- type: string
- endpoint:
- type: string
- nullable: true
- region:
- type: string
- accessKeyId:
- type: string
- nullable: true
- prefix:
- type: string
- exportFrequency:
- $ref: '#/components/schemas/BlobStorageExportFrequency'
- enabled:
- type: boolean
- forcePathStyle:
- type: boolean
- fileType:
- $ref: '#/components/schemas/BlobStorageIntegrationFileTypeResponse'
- exportMode:
- $ref: '#/components/schemas/BlobStorageExportMode'
- exportStartDate:
- type: string
- format: date-time
- nullable: true
- compressed:
- type: boolean
- exportSource:
- $ref: '#/components/schemas/BlobStorageExportSource'
- exportFieldGroups:
- type: array
- items:
- $ref: '#/components/schemas/BlobStorageExportFieldGroup'
- nullable: true
- description: >-
- Field groups included in each exported observation row. An empty
- list is treated as all groups during export.
- nextSyncAt:
- type: string
- format: date-time
- nullable: true
- lastSyncAt:
- type: string
- format: date-time
- nullable: true
- lastError:
- type: string
- nullable: true
- lastErrorAt:
- type: string
- format: date-time
- nullable: true
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- required:
- - id
- - projectId
- - type
- - bucketName
- - endpoint
- - region
- - accessKeyId
- - prefix
- - exportFrequency
- - enabled
- - forcePathStyle
- - fileType
- - exportMode
- - exportStartDate
- - compressed
- - exportSource
- - exportFieldGroups
- - nextSyncAt
- - lastSyncAt
- - lastError
- - lastErrorAt
- - createdAt
- - updatedAt
- BlobStorageIntegrationsResponse:
- title: BlobStorageIntegrationsResponse
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/BlobStorageIntegrationResponse'
- required:
- - data
- BlobStorageSyncStatus:
- title: BlobStorageSyncStatus
- type: string
- enum:
- - idle
- - running
- - queued
- - up_to_date
- - disabled
- - error
- description: >-
- Sync status of the blob storage integration:
-
- - `disabled` — integration is not enabled
-
- - `error` — last export failed (see `lastError` for details)
-
- - `running` — an export job is currently being processed
-
- - `queued` — next export is overdue (`nextSyncAt` is in the past) and
- waiting to be picked up by the worker
-
- - `idle` — enabled but has never exported yet and no export is queued
-
- - `up_to_date` — all available data has been exported; next export is
- scheduled for the future
-
-
- **ETL usage**: poll this endpoint and check for `up_to_date` status.
- Compare `lastSyncAt` against your
-
- ETL bookmark to determine if new data is available. Note that exports
- run with a 20-minute lag buffer,
-
- so `lastSyncAt` will always be at least 20 minutes behind real-time.
- BlobStorageIntegrationStatusResponse:
- title: BlobStorageIntegrationStatusResponse
- type: object
- properties:
- id:
- type: string
- projectId:
- type: string
- syncStatus:
- $ref: '#/components/schemas/BlobStorageSyncStatus'
- enabled:
- type: boolean
- lastSyncAt:
- type: string
- format: date-time
- nullable: true
- description: >-
- End of the last successfully exported time window. Compare against
- your ETL bookmark to determine if new data is available. Null if the
- integration has never synced.
- nextSyncAt:
- type: string
- format: date-time
- nullable: true
- description: When the next export is scheduled. Null if no sync has occurred yet.
- lastError:
- type: string
- nullable: true
- description: >-
- Raw error message from the storage provider (S3/Azure/GCS) if the
- last export failed. Cleared on successful export.
- lastErrorAt:
- type: string
- format: date-time
- nullable: true
- description: When the last error occurred. Cleared on successful export.
- required:
- - id
- - projectId
- - syncStatus
- - enabled
- - lastSyncAt
- - nextSyncAt
- - lastError
- - lastErrorAt
- BlobStorageIntegrationDeletionResponse:
- title: BlobStorageIntegrationDeletionResponse
- type: object
- properties:
- message:
- type: string
- required:
- - message
- CreateCommentRequest:
- title: CreateCommentRequest
- type: object
- properties:
- projectId:
- type: string
- description: The id of the project to attach the comment to.
- objectType:
- type: string
- description: >-
- The type of the object to attach the comment to (trace, observation,
- session, prompt).
- objectId:
- type: string
- description: >-
- The id of the object to attach the comment to. If this does not
- reference a valid existing object, an error will be thrown.
- content:
- type: string
- description: >-
- The content of the comment. May include markdown. Currently limited
- to 5000 characters.
- authorUserId:
- type: string
- nullable: true
- description: >-
- The id of the user who created the comment. Must be a member of the
- organization that owns the project, otherwise an error will be
- thrown.
- required:
- - projectId
- - objectType
- - objectId
- - content
- CreateCommentResponse:
- title: CreateCommentResponse
- type: object
- properties:
- id:
- type: string
- description: The id of the created object in Langfuse
- required:
- - id
- GetCommentsResponse:
- title: GetCommentsResponse
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Comment'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- required:
- - data
- - meta
- Deprecation:
- title: Deprecation
- type: object
- description: >-
- Migration signal returned by deprecated endpoints. Optional fields are
- omitted when they have no value.
- properties:
- message:
- type: string
- description: >-
- Human- and agent-readable summary of the deprecation and its
- replacement.
- replacement:
- type: string
- nullable: true
- description: >-
- The replacement endpoint, e.g. "GET /api/public/v2/observations".
- Omitted when the endpoint is being removed without a direct
- replacement.
- docsUrl:
- type: string
- nullable: true
- description: Link to the migration documentation (markdown), when available.
- sunsetAt:
- type: string
- nullable: true
- description: >-
- ISO date after which the endpoint may stop working, when a removal
- date is committed.
- required:
- - message
- Trace:
- title: Trace
- type: object
- properties:
- id:
- type: string
- description: The unique identifier of a trace
- timestamp:
- type: string
- format: date-time
- description: The timestamp when the trace was created
- name:
- type: string
- nullable: true
- description: The name of the trace
- input:
- nullable: true
- description: The input data of the trace. Can be any JSON.
- output:
- nullable: true
- description: The output data of the trace. Can be any JSON.
- sessionId:
- type: string
- nullable: true
- description: The session identifier associated with the trace
- release:
- type: string
- nullable: true
- description: The release version of the application when the trace was created
- version:
- type: string
- nullable: true
- description: The version of the trace
- userId:
- type: string
- nullable: true
- description: The user identifier associated with the trace
- metadata:
- nullable: true
- description: The metadata associated with the trace. Can be any JSON.
- tags:
- type: array
- items:
- type: string
- description: The tags associated with the trace.
- public:
- type: boolean
- description: Public traces are accessible via url without login
- environment:
- type: string
- description: >-
- The environment from which this trace originated. Can be any
- lowercase alphanumeric string with hyphens and underscores that does
- not start with 'langfuse'.
- required:
- - id
- - timestamp
- - name
- - sessionId
- - release
- - version
- - userId
- - tags
- - public
- - environment
- TraceWithDetails:
- title: TraceWithDetails
- type: object
- properties:
- htmlPath:
- type: string
- description: Path of trace in Langfuse UI
- latency:
- type: number
- format: double
- nullable: true
- description: Latency of trace in seconds
- totalCost:
- type: number
- format: double
- nullable: true
- description: Cost of trace in USD
- observations:
- type: array
- items:
- type: string
- nullable: true
- description: List of observation ids
- scores:
- type: array
- items:
- type: string
- nullable: true
- description: List of score ids
- required:
- - htmlPath
- allOf:
- - $ref: '#/components/schemas/Trace'
- TraceWithFullDetails:
- title: TraceWithFullDetails
- type: object
- properties:
- htmlPath:
- type: string
- description: Path of trace in Langfuse UI
- latency:
- type: number
- format: double
- nullable: true
- description: Latency of trace in seconds
- totalCost:
- type: number
- format: double
- nullable: true
- description: Cost of trace in USD
- observations:
- type: array
- items:
- $ref: '#/components/schemas/ObservationsView'
- description: List of observations
- scores:
- type: array
- items:
- $ref: '#/components/schemas/ScoreV1'
- description: List of scores
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
- nullable: true
- required:
- - htmlPath
- - observations
- - scores
- allOf:
- - $ref: '#/components/schemas/Trace'
- Session:
- title: Session
- type: object
- properties:
- id:
- type: string
- createdAt:
- type: string
- format: date-time
- projectId:
- type: string
- environment:
- type: string
- description: The environment from which this session originated.
- required:
- - id
- - createdAt
- - projectId
- - environment
- SessionWithTraces:
- title: SessionWithTraces
- type: object
- properties:
- traces:
- type: array
- items:
- $ref: '#/components/schemas/Trace'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
- nullable: true
- required:
- - traces
- allOf:
- - $ref: '#/components/schemas/Session'
- Observation:
- title: Observation
- type: object
- properties:
- id:
- type: string
- description: The unique identifier of the observation
- traceId:
- type: string
- nullable: true
- description: The trace ID associated with the observation
- type:
- type: string
- description: The type of the observation
- name:
- type: string
- nullable: true
- description: The name of the observation
- startTime:
- type: string
- format: date-time
- description: The start time of the observation
- endTime:
- type: string
- format: date-time
- nullable: true
- description: The end time of the observation.
- completionStartTime:
- type: string
- format: date-time
- nullable: true
- description: The completion start time of the observation
- model:
- type: string
- nullable: true
- description: The model used for the observation
- modelParameters:
- description: The parameters of the model used for the observation
- input:
- description: The input data of the observation
- version:
- type: string
- nullable: true
- description: The version of the observation
- metadata:
- description: Additional metadata of the observation
- output:
- description: The output data of the observation
- usage:
- $ref: '#/components/schemas/Usage'
- description: >-
- (Deprecated. Use usageDetails and costDetails instead.) The usage
- data of the observation
- level:
- $ref: '#/components/schemas/ObservationLevel'
- description: The level of the observation
- statusMessage:
- type: string
- nullable: true
- description: The status message of the observation
- parentObservationId:
- type: string
- nullable: true
- description: The parent observation ID
- promptId:
- type: string
- nullable: true
- description: The prompt ID associated with the observation
- usageDetails:
- type: object
- additionalProperties:
- type: integer
- description: >-
- The usage details of the observation. Key is the name of the usage
- metric, value is the number of units consumed. The total key is the
- sum of all (non-total) usage metrics or the total value ingested.
- costDetails:
- type: object
- additionalProperties:
- type: number
- format: double
- description: >-
- The cost details of the observation. Key is the name of the cost
- metric, value is the cost in USD. The total key is the sum of all
- (non-total) cost metrics or the total value ingested.
- environment:
- type: string
- description: >-
- The environment from which this observation originated. Can be any
- lowercase alphanumeric string with hyphens and underscores that does
- not start with 'langfuse'.
- required:
- - id
- - traceId
- - type
- - name
- - startTime
- - endTime
- - completionStartTime
- - model
- - modelParameters
- - input
- - version
- - metadata
- - output
- - usage
- - level
- - statusMessage
- - parentObservationId
- - promptId
- - usageDetails
- - costDetails
- - environment
- ObservationsView:
- title: ObservationsView
- type: object
- properties:
- promptName:
- type: string
- nullable: true
- description: The name of the prompt associated with the observation
- promptVersion:
- type: integer
- nullable: true
- description: The version of the prompt associated with the observation
- modelId:
- type: string
- nullable: true
- description: The unique identifier of the model
- inputPrice:
- type: number
- format: double
- nullable: true
- description: The price of the input in USD
- outputPrice:
- type: number
- format: double
- nullable: true
- description: The price of the output in USD.
- totalPrice:
- type: number
- format: double
- nullable: true
- description: The total price in USD.
- calculatedInputCost:
- type: number
- format: double
- nullable: true
- description: >-
- (Deprecated. Use usageDetails and costDetails instead.) The
- calculated cost of the input in USD
- calculatedOutputCost:
- type: number
- format: double
- nullable: true
- description: >-
- (Deprecated. Use usageDetails and costDetails instead.) The
- calculated cost of the output in USD
- calculatedTotalCost:
- type: number
- format: double
- nullable: true
- description: >-
- (Deprecated. Use usageDetails and costDetails instead.) The
- calculated total cost in USD
- latency:
- type: number
- format: double
- nullable: true
- description: The latency in seconds.
- timeToFirstToken:
- type: number
- format: double
- nullable: true
- description: The time to the first token in seconds
- required:
- - promptName
- - promptVersion
- - modelId
- - inputPrice
- - outputPrice
- - totalPrice
- - calculatedInputCost
- - calculatedOutputCost
- - calculatedTotalCost
- - latency
- - timeToFirstToken
- allOf:
- - $ref: '#/components/schemas/Observation'
- ObservationsViewSingle:
- title: ObservationsViewSingle
- type: object
- properties:
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
- nullable: true
- allOf:
- - $ref: '#/components/schemas/ObservationsView'
- ObservationV2:
- title: ObservationV2
- type: object
- description: >-
- An observation from the v2 API with field-group-based selection.
-
- Core fields are always present. Other fields are included only when
- their field group is requested.
- properties:
- id:
- type: string
- description: The unique identifier of the observation
- traceId:
- type: string
- nullable: true
- description: The trace ID associated with the observation
- startTime:
- type: string
- format: date-time
- description: The start time of the observation
- endTime:
- type: string
- format: date-time
- nullable: true
- description: The end time of the observation
- projectId:
- type: string
- description: The project ID this observation belongs to
- parentObservationId:
- type: string
- nullable: true
- description: >-
- The physical parent observation ID, if present.
-
- Observations marked as app roots by the SDK may retain a non-null
- parent ID.
- type:
- type: string
- description: The type of the observation (e.g. GENERATION, SPAN, EVENT)
- isRootObservation:
- type: boolean
- nullable: true
- description: >-
- Whether this observation is a logical root.
-
- This is true for observations without a physical parent and
- observations marked as app roots by the SDK.
- name:
- type: string
- nullable: true
- description: The name of the observation
- level:
- $ref: '#/components/schemas/ObservationLevel'
- nullable: true
- description: The level of the observation
- statusMessage:
- type: string
- nullable: true
- description: The status message of the observation
- version:
- type: string
- nullable: true
- description: The version of the observation
- environment:
- type: string
- nullable: true
- description: The environment from which this observation originated
- bookmarked:
- type: boolean
- nullable: true
- description: Whether the observation is bookmarked
- public:
- type: boolean
- nullable: true
- description: Whether the observation is public
- userId:
- type: string
- nullable: true
- description: The user ID associated with the observation
- sessionId:
- type: string
- nullable: true
- description: The session ID associated with the observation
- completionStartTime:
- type: string
- format: date-time
- nullable: true
- description: The completion start time of the observation
- createdAt:
- type: string
- format: date-time
- nullable: true
- description: The creation timestamp of the observation
- updatedAt:
- type: string
- format: date-time
- nullable: true
- description: The last update timestamp of the observation
- input:
- nullable: true
- description: The input data of the observation
- output:
- nullable: true
- description: The output data of the observation
- metadata:
- nullable: true
- description: Additional metadata of the observation
- model:
- type: string
- nullable: true
- description: The model name as provided by the user
- internalModelId:
- type: string
- nullable: true
- description: The internal model ID matched by Langfuse
- modelParameters:
- nullable: true
- description: The parameters of the model used for the observation
- usageDetails:
- type: object
- additionalProperties:
- type: integer
- nullable: true
- description: >-
- The usage details of the observation. Key is the usage metric name,
- value is the number of units consumed.
- costDetails:
- type: object
- additionalProperties:
- type: number
- format: double
- nullable: true
- description: >-
- The cost details of the observation. Key is the cost metric name,
- value is the cost in USD.
- totalCost:
- type: number
- format: double
- nullable: true
- description: The total cost of the observation in USD
- usagePricingTierName:
- type: string
- nullable: true
- description: >-
- The name of the pricing tier applied to this observation's usage
- costs
- promptId:
- type: string
- nullable: true
- description: The prompt ID associated with the observation
- promptName:
- type: string
- nullable: true
- description: The prompt name associated with the observation
- promptVersion:
- type: integer
- nullable: true
- description: The prompt version associated with the observation
- latency:
- type: number
- format: double
- nullable: true
- description: The latency in seconds
- timeToFirstToken:
- type: number
- format: double
- nullable: true
- description: The time to first token in seconds
- modelId:
- type: string
- nullable: true
- description: >-
- The matched model ID. Null when the `model` field group is not
- requested.
- inputPrice:
- type: string
- nullable: true
- description: >-
- The input token price (USD per unit) from the matched model,
- serialized as a decimal string (e.g. "0.0001"). Null when the
- `model` field group is not requested.
- outputPrice:
- type: string
- nullable: true
- description: >-
- The output token price (USD per unit) from the matched model,
- serialized as a decimal string (e.g. "0.0001"). Null when the
- `model` field group is not requested.
- totalPrice:
- type: string
- nullable: true
- description: >-
- The total token price (USD per unit) from the matched model,
- serialized as a decimal string (e.g. "0.0001"). Null when the
- `model` field group is not requested.
- traceName:
- type: string
- nullable: true
- description: The name of the parent trace
- tags:
- type: array
- items:
- type: string
- nullable: true
- description: Tags from the parent trace (denormalized onto the observation)
- release:
- type: string
- nullable: true
- description: The release version of the parent trace
- required:
- - id
- - traceId
- - startTime
- - endTime
- - projectId
- - parentObservationId
- - type
- - modelId
- - inputPrice
- - outputPrice
- - totalPrice
- Usage:
- title: Usage
- type: object
- description: >-
- (Deprecated. Use usageDetails and costDetails instead.) Standard
- interface for usage and cost
- properties:
- input:
- type: integer
- description: Number of input units (e.g. tokens)
- output:
- type: integer
- description: Number of output units (e.g. tokens)
- total:
- type: integer
- description: Defaults to input+output if not set
- unit:
- type: string
- nullable: true
- description: Unit of measurement
- inputCost:
- type: number
- format: double
- nullable: true
- description: USD input cost
- outputCost:
- type: number
- format: double
- nullable: true
- description: USD output cost
- totalCost:
- type: number
- format: double
- nullable: true
- description: USD total cost, defaults to input+output
- required:
- - input
- - output
- - total
- - unit
- ScoreConfig:
- title: ScoreConfig
- type: object
- description: Configuration for a score
- properties:
- id:
- type: string
- name:
- type: string
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- projectId:
- type: string
- dataType:
- $ref: '#/components/schemas/ScoreConfigDataType'
- isArchived:
- type: boolean
- description: Whether the score config is archived. Defaults to false
- minValue:
- type: number
- format: double
- nullable: true
- description: >-
- Sets minimum value for numerical scores. If not set, the minimum
- value defaults to -∞
- maxValue:
- type: number
- format: double
- nullable: true
- description: >-
- Sets maximum value for numerical scores. If not set, the maximum
- value defaults to +∞
- categories:
- type: array
- items:
- $ref: '#/components/schemas/ConfigCategory'
- nullable: true
- description: Configures custom categories for categorical scores
- description:
- type: string
- nullable: true
- description: Description of the score config
- required:
- - id
- - name
- - createdAt
- - updatedAt
- - projectId
- - dataType
- - isArchived
- ConfigCategory:
- title: ConfigCategory
- type: object
- properties:
- value:
- type: number
- format: double
- label:
- type: string
- required:
- - value
- - label
- BaseScoreV1:
- title: BaseScoreV1
- type: object
- properties:
- id:
- type: string
- traceId:
- type: string
- name:
- type: string
- source:
- $ref: '#/components/schemas/ScoreSource'
- observationId:
- type: string
- nullable: true
- description: The observation ID associated with the score
- timestamp:
- type: string
- format: date-time
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- authorUserId:
- type: string
- nullable: true
- description: The user ID of the author
- comment:
- type: string
- nullable: true
- description: Comment on the score
- metadata:
- description: Metadata associated with the score
- configId:
- type: string
- nullable: true
- description: >-
- Reference a score config on a score. When set, config and score name
- must be equal and value must comply to optionally defined numerical
- range
- queueId:
- type: string
- nullable: true
- description: >-
- The annotation queue referenced by the score. Indicates if score was
- initially created while processing annotation queue.
- environment:
- type: string
- description: >-
- The environment from which this score originated. Can be any
- lowercase alphanumeric string with hyphens and underscores that does
- not start with 'langfuse'.
- required:
- - id
- - traceId
- - name
- - source
- - timestamp
- - createdAt
- - updatedAt
- - authorUserId
- - comment
- - metadata
- - configId
- - queueId
- - environment
- NumericScoreV1:
- title: NumericScoreV1
- type: object
- properties:
- value:
- type: number
- format: double
- description: The numeric value of the score
- required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScoreV1'
- BooleanScoreV1:
- title: BooleanScoreV1
- type: object
- properties:
- value:
- type: number
- format: double
- description: >-
- The numeric value of the score. Equals 1 for "True" and 0 for "False"
- stringValue:
- type: string
- description: >-
- The string representation of the score value. Is inferred from the
- numeric value and equals "True" or "False"
- required:
- - value
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScoreV1'
- CategoricalScoreV1:
- title: CategoricalScoreV1
- type: object
- properties:
- value:
- type: number
- format: double
- description: >-
- Represents the numeric category mapping of the stringValue. If no
- config is linked, defaults to 0.
- stringValue:
- type: string
- description: >-
- The string representation of the score value. If no config is
- linked, can be any string. Otherwise, must map to a config category
- required:
- - value
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScoreV1'
- TextScoreV1:
- title: TextScoreV1
- type: object
- properties:
- stringValue:
- type: string
- description: The text content of the score (1-500 characters)
- required:
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScoreV1'
- ScoreV1:
- title: ScoreV1
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: '#/components/schemas/NumericScoreV1'
- required:
- - dataType
- title: NumericScoreV1
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: '#/components/schemas/CategoricalScoreV1'
- required:
- - dataType
- title: CategoricalScoreV1
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: '#/components/schemas/BooleanScoreV1'
- required:
- - dataType
- title: BooleanScoreV1
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - TEXT
- - $ref: '#/components/schemas/TextScoreV1'
- required:
- - dataType
- title: TextScoreV1
- BaseScore:
- title: BaseScore
- type: object
- properties:
- id:
- type: string
- traceId:
- type: string
- nullable: true
- description: The trace ID associated with the score
- sessionId:
- type: string
- nullable: true
- description: The session ID associated with the score
- observationId:
- type: string
- nullable: true
- description: The observation ID associated with the score
- datasetRunId:
- type: string
- nullable: true
- description: The dataset run ID associated with the score
- name:
- type: string
- source:
- $ref: '#/components/schemas/ScoreSource'
- timestamp:
- type: string
- format: date-time
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- authorUserId:
- type: string
- nullable: true
- description: The user ID of the author
- comment:
- type: string
- nullable: true
- description: Comment on the score
- metadata:
- description: Metadata associated with the score
- configId:
- type: string
- nullable: true
- description: >-
- Reference a score config on a score. When set, config and score name
- must be equal and value must comply to optionally defined numerical
- range
- queueId:
- type: string
- nullable: true
- description: >-
- The annotation queue referenced by the score. Indicates if score was
- initially created while processing annotation queue.
- environment:
- type: string
- description: >-
- The environment from which this score originated. Can be any
- lowercase alphanumeric string with hyphens and underscores that does
- not start with 'langfuse'.
- required:
- - id
- - name
- - source
- - timestamp
- - createdAt
- - updatedAt
- - authorUserId
- - comment
- - metadata
- - configId
- - queueId
- - environment
- NumericScore:
- title: NumericScore
- type: object
- properties:
- value:
- type: number
- format: double
- description: The numeric value of the score
- required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScore'
- BooleanScore:
- title: BooleanScore
- type: object
- properties:
- value:
- type: number
- format: double
- description: >-
- The numeric value of the score. Equals 1 for "True" and 0 for "False"
- stringValue:
- type: string
- description: >-
- The string representation of the score value. Is inferred from the
- numeric value and equals "True" or "False"
- required:
- - value
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScore'
- CategoricalScore:
- title: CategoricalScore
- type: object
- properties:
- value:
- type: number
- format: double
- description: >-
- Represents the numeric category mapping of the stringValue. If no
- config is linked, defaults to 0.
- stringValue:
- type: string
- description: >-
- The string representation of the score value. If no config is
- linked, can be any string. Otherwise, must map to a config category
- required:
- - value
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScore'
- CorrectionScore:
- title: CorrectionScore
- type: object
- properties:
- value:
- type: number
- format: double
- description: The numeric value of the score. Always 0 for correction scores.
- stringValue:
- type: string
- description: The string representation of the correction content
- required:
- - value
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScore'
- TextScore:
- title: TextScore
- type: object
- properties:
- stringValue:
- type: string
- description: The text content of the score (1-500 characters)
- required:
- - stringValue
- allOf:
- - $ref: '#/components/schemas/BaseScore'
- Score:
- title: Score
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: '#/components/schemas/NumericScore'
- required:
- - dataType
- title: NumericScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: '#/components/schemas/CategoricalScore'
- required:
- - dataType
- title: CategoricalScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: '#/components/schemas/BooleanScore'
- required:
- - dataType
- title: BooleanScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CORRECTION
- - $ref: '#/components/schemas/CorrectionScore'
- required:
- - dataType
- title: CorrectionScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - TEXT
- - $ref: '#/components/schemas/TextScore'
- required:
- - dataType
- title: TextScore
- properties:
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
- nullable: true
- CreateScoreValue:
- title: CreateScoreValue
- oneOf:
- - type: number
- format: double
- title: DoubleNumber
- - type: string
- title: String
- description: >-
- The value of the score. Must be passed as string for categorical and
- text scores, and numeric for boolean and numeric scores
- Comment:
- title: Comment
- type: object
- properties:
- id:
- type: string
- projectId:
- type: string
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- objectType:
- $ref: '#/components/schemas/CommentObjectType'
- objectId:
- type: string
- content:
- type: string
- authorUserId:
- type: string
- nullable: true
- description: The user ID of the comment author
- required:
- - id
- - projectId
- - createdAt
- - updatedAt
- - objectType
- - objectId
- - content
- Dataset:
- title: Dataset
- type: object
- properties:
- id:
- type: string
- name:
- type: string
- description:
- type: string
- nullable: true
- description: Description of the dataset
- metadata:
- description: Metadata associated with the dataset
- inputSchema:
- nullable: true
- description: JSON Schema for validating dataset item inputs
- expectedOutputSchema:
- nullable: true
- description: JSON Schema for validating dataset item expected outputs
- projectId:
- type: string
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- required:
- - id
- - name
- - description
- - metadata
- - inputSchema
- - expectedOutputSchema
- - projectId
- - createdAt
- - updatedAt
- DatasetItem:
- title: DatasetItem
- type: object
- properties:
- id:
- type: string
- status:
- $ref: '#/components/schemas/DatasetStatus'
- input:
- description: Input data for the dataset item
- expectedOutput:
- description: Expected output for the dataset item
- metadata:
- description: Metadata associated with the dataset item
- sourceTraceId:
- type: string
- nullable: true
- description: The trace ID that sourced this dataset item
- sourceObservationId:
- type: string
- nullable: true
- description: The observation ID that sourced this dataset item
- datasetId:
- type: string
- datasetName:
- type: string
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- mediaReferences:
- type: array
- items:
- $ref: '#/components/schemas/DatasetItemMediaReference'
- description: >-
- Resolved Langfuse media references found in input, expectedOutput,
- and metadata.
- required:
- - id
- - status
- - input
- - expectedOutput
- - metadata
- - sourceTraceId
- - sourceObservationId
- - datasetId
- - datasetName
- - createdAt
- - updatedAt
- - mediaReferences
- DatasetItemMediaReference:
- title: DatasetItemMediaReference
- type: object
- properties:
- field:
- $ref: '#/components/schemas/DatasetItemMediaReferenceField'
- description: The dataset item field containing the reference
- referenceString:
- type: string
- description: >-
- The Langfuse media reference string, e.g.
- `@@@langfuseMedia:type=image/png|id=...|source=bytes@@@`
- jsonPath:
- type: string
- description: >-
- JSONPath of the string holding the reference within the field, e.g.
- `$['image']`
- media:
- $ref: '#/components/schemas/DatasetItemMediaReferenceMedia'
- description: The resolved media record.
- required:
- - field
- - referenceString
- - jsonPath
- - media
- DatasetItemMediaReferenceField:
- title: DatasetItemMediaReferenceField
+components:
+ schemas:
+ AnnotationQueueStatus:
+ title: AnnotationQueueStatus
type: string
enum:
- - input
- - expectedOutput
- - metadata
- DatasetItemMediaReferenceMedia:
- title: DatasetItemMediaReferenceMedia
- type: object
- properties:
- mediaId:
- type: string
- description: The unique langfuse identifier of the media record
- contentType:
- type: string
- description: The MIME type of the media record
- contentLength:
- type: integer
- description: The size of the media record in bytes
- url:
- type: string
- description: The signed download URL of the media record
- urlExpiry:
- type: string
- description: The expiry date and time of the download URL
- required:
- - mediaId
- - contentType
- - contentLength
- - url
- - urlExpiry
- DatasetRunItem:
- title: DatasetRunItem
+ - PENDING
+ - COMPLETED
+ AnnotationQueueObjectType:
+ title: AnnotationQueueObjectType
+ type: string
+ enum:
+ - TRACE
+ - OBSERVATION
+ - SESSION
+ AnnotationQueue:
+ title: AnnotationQueue
type: object
properties:
id:
type: string
- datasetRunId:
- type: string
- datasetRunName:
- type: string
- datasetItemId:
- type: string
- traceId:
+ name:
type: string
- observationId:
+ description:
type: string
nullable: true
- description: The observation ID associated with this run item
+ scoreConfigIds:
+ type: array
+ items:
+ type: string
createdAt:
type: string
format: date-time
@@ -11466,6608 +8609,6618 @@ components:
format: date-time
required:
- id
- - datasetRunId
- - datasetRunName
- - datasetItemId
- - traceId
- - observationId
+ - name
+ - description
+ - scoreConfigIds
- createdAt
- updatedAt
- DatasetRun:
- title: DatasetRun
+ AnnotationQueueItem:
+ title: AnnotationQueueItem
type: object
properties:
id:
type: string
- description: Unique identifier of the dataset run
- name:
- type: string
- description: Name of the dataset run
- description:
+ queueId:
type: string
- nullable: true
- description: Description of the run
- metadata:
- description: Metadata of the dataset run
- datasetId:
+ objectId:
type: string
- description: Id of the associated dataset
- datasetName:
+ objectType:
+ $ref: '#/components/schemas/AnnotationQueueObjectType'
+ status:
+ $ref: '#/components/schemas/AnnotationQueueStatus'
+ completedAt:
type: string
- description: Name of the associated dataset
+ format: date-time
+ nullable: true
createdAt:
type: string
format: date-time
- description: The date and time when the dataset run was created
updatedAt:
type: string
format: date-time
- description: The date and time when the dataset run was last updated
required:
- id
- - name
- - description
- - metadata
- - datasetId
- - datasetName
+ - queueId
+ - objectId
+ - objectType
+ - status
- createdAt
- updatedAt
- DatasetRunWithItems:
- title: DatasetRunWithItems
+ PaginatedAnnotationQueues:
+ title: PaginatedAnnotationQueues
type: object
properties:
- datasetRunItems:
+ data:
type: array
items:
- $ref: '#/components/schemas/DatasetRunItem'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
- nullable: true
+ $ref: '#/components/schemas/AnnotationQueue'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
required:
- - datasetRunItems
- allOf:
- - $ref: '#/components/schemas/DatasetRun'
- Model:
- title: Model
+ - data
+ - meta
+ PaginatedAnnotationQueueItems:
+ title: PaginatedAnnotationQueueItems
type: object
- description: >-
- Model definition used for transforming usage into USD cost and/or
- tokenization.
-
-
- Models can have either simple flat pricing or tiered pricing:
-
- - Flat pricing: Single price per usage type (legacy, but still
- supported)
-
- - Tiered pricing: Multiple pricing tiers with conditional matching based
- on usage patterns or observation attributes
-
-
- The pricing tiers approach is recommended for models with usage-based
- pricing variations.
-
- When using tiered pricing, the flat price fields (inputPrice,
- outputPrice, prices) are populated
-
- from the default tier for backward compatibility.
properties:
- id:
- type: string
- modelName:
- type: string
- description: >-
- Name of the model definition. If multiple with the same name exist,
- they are applied in the following order: (1) custom over built-in,
- (2) newest according to startTime where
- model.startTime<observation.startTime
- matchPattern:
- type: string
- description: >-
- Regex pattern which matches this model definition to
- generation.model. Useful in case of fine-tuned models. If you want
- to exact match, use `(?i)^modelname$`
- startDate:
- type: string
- format: date-time
- nullable: true
- description: Apply only to generations which are newer than this ISO date.
- unit:
- $ref: '#/components/schemas/ModelUsageUnit'
- nullable: true
- description: Unit used by this model.
- inputPrice:
- type: number
- format: double
- nullable: true
- description: Deprecated. See 'prices' instead. Price (USD) per input unit
- outputPrice:
- type: number
- format: double
- nullable: true
- description: Deprecated. See 'prices' instead. Price (USD) per output unit
- totalPrice:
- type: number
- format: double
- nullable: true
- description: >-
- Deprecated. See 'prices' instead. Price (USD) per total unit. Cannot
- be set if input or output price is set.
- tokenizerId:
- type: string
- nullable: true
- description: >-
- Optional. Tokenizer to be applied to observations which match to
- this model. See docs for more details.
- tokenizerConfig:
- description: >-
- Optional. Configuration for the selected tokenizer. Needs to be
- JSON. See docs for more details.
- isLangfuseManaged:
- type: boolean
- createdAt:
- type: string
- format: date-time
- description: Timestamp when the model was created
- prices:
- type: object
- additionalProperties:
- $ref: '#/components/schemas/ModelPrice'
- description: >-
- Deprecated. Use 'pricingTiers' instead for models with usage-based
- pricing variations.
-
-
- This field shows prices by usage type from the default pricing tier.
- Maintained for backward compatibility.
-
- If the model uses tiered pricing, this field will be populated from
- the default tier's prices.
- pricingTiers:
+ data:
type: array
items:
- $ref: '#/components/schemas/PricingTier'
- description: >-
- Array of pricing tiers with conditional pricing based on usage
- thresholds or observation attributes.
-
-
- Pricing tiers enable accurate cost tracking for models that charge
- different rates based on usage patterns
-
- or request attributes (e.g., high-volume usage, large context
- windows, cached tokens, or service tiers).
-
-
- Each model must have exactly one default tier (isDefault=true,
- priority=0) that serves as a fallback.
-
- Additional conditional tiers can be defined with specific matching
- criteria.
-
-
- If this array is empty, the model uses legacy flat pricing from the
- inputPrice/outputPrice/totalPrice fields.
- required:
- - id
- - modelName
- - matchPattern
- - startDate
- - inputPrice
- - outputPrice
- - totalPrice
- - tokenizerId
- - tokenizerConfig
- - isLangfuseManaged
- - createdAt
- - prices
- - pricingTiers
- ModelPrice:
- title: ModelPrice
- type: object
- properties:
- price:
- type: number
- format: double
- required:
- - price
- PricingTierCondition:
- title: PricingTierCondition
- oneOf:
- - $ref: '#/components/schemas/PricingTierUsageCondition'
- - $ref: '#/components/schemas/PricingTierAttributeCondition'
- description: >-
- Condition for matching a pricing tier against usage details or
- observation attributes.
-
-
- Usage-detail conditions treat usageDetailPattern as a regex, sum all
- matching usage values, and compare the sum to the numeric value.
- Model-parameter and metadata conditions match an exact top-level key
- against one or more string values.
- PricingTierUsageCondition:
- title: PricingTierUsageCondition
- type: object
- description: Condition that sums usage details whose keys match a regex.
- properties:
- usageDetailPattern:
- type: string
- description: Regex pattern matched against usage detail keys.
- operator:
- $ref: '#/components/schemas/PricingTierOperator'
- value:
- type: number
- format: double
- description: Numeric threshold for the summed matching usage values.
- caseSensitive:
- type: boolean
- description: Whether the usage-detail regex is case-sensitive.
+ $ref: '#/components/schemas/AnnotationQueueItem'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
required:
- - usageDetailPattern
- - operator
- - value
- - caseSensitive
- PricingTierAttributeCondition:
- title: PricingTierAttributeCondition
+ - data
+ - meta
+ CreateAnnotationQueueRequest:
+ title: CreateAnnotationQueueRequest
type: object
- description: >-
- Condition that matches any configured value for a top-level observation
- attribute.
properties:
- source:
- $ref: '#/components/schemas/PricingTierAttributeSource'
- description: Observation attribute object evaluated by this condition.
- key:
+ name:
type: string
- description: Exact top-level attribute key.
- operator:
+ description:
type: string
- const: in
- description: Membership operator.
- values:
+ nullable: true
+ scoreConfigIds:
type: array
items:
type: string
- description: Accepted string attribute values. At least one value is required.
required:
- - source
- - key
- - operator
- - values
- PricingTierAttributeSource:
- title: PricingTierAttributeSource
- type: string
- enum:
- - model_parameters
- - metadata
- PricingTierConditionInput:
- title: PricingTierConditionInput
- oneOf:
- - $ref: '#/components/schemas/PricingTierUsageConditionInput'
- - $ref: '#/components/schemas/PricingTierAttributeCondition'
- description: Condition input for matching a pricing tier.
- PricingTierUsageConditionInput:
- title: PricingTierUsageConditionInput
+ - name
+ - scoreConfigIds
+ CreateAnnotationQueueItemRequest:
+ title: CreateAnnotationQueueItemRequest
type: object
- description: Input condition that sums usage details whose keys match a regex.
properties:
- usageDetailPattern:
+ objectId:
type: string
- description: Regex pattern matched against usage detail keys.
- operator:
- $ref: '#/components/schemas/PricingTierOperator'
- value:
- type: number
- format: double
- description: Numeric threshold for the summed matching usage values.
- caseSensitive:
- type: boolean
+ objectType:
+ $ref: '#/components/schemas/AnnotationQueueObjectType'
+ status:
+ $ref: '#/components/schemas/AnnotationQueueStatus'
nullable: true
- description: Whether the usage-detail regex is case-sensitive. Defaults to false.
+ description: Defaults to PENDING for new queue items
required:
- - usageDetailPattern
- - operator
- - value
- PricingTier:
- title: PricingTier
+ - objectId
+ - objectType
+ UpdateAnnotationQueueItemRequest:
+ title: UpdateAnnotationQueueItemRequest
+ type: object
+ properties:
+ status:
+ $ref: '#/components/schemas/AnnotationQueueStatus'
+ nullable: true
+ DeleteAnnotationQueueItemResponse:
+ title: DeleteAnnotationQueueItemResponse
+ type: object
+ properties:
+ success:
+ type: boolean
+ message:
+ type: string
+ required:
+ - success
+ - message
+ AnnotationQueueAssignmentRequest:
+ title: AnnotationQueueAssignmentRequest
+ type: object
+ properties:
+ userId:
+ type: string
+ required:
+ - userId
+ DeleteAnnotationQueueAssignmentResponse:
+ title: DeleteAnnotationQueueAssignmentResponse
+ type: object
+ properties:
+ success:
+ type: boolean
+ required:
+ - success
+ CreateAnnotationQueueAssignmentResponse:
+ title: CreateAnnotationQueueAssignmentResponse
type: object
+ properties:
+ userId:
+ type: string
+ queueId:
+ type: string
+ projectId:
+ type: string
+ required:
+ - userId
+ - queueId
+ - projectId
+ BlobStorageIntegrationType:
+ title: BlobStorageIntegrationType
+ type: string
+ enum:
+ - S3
+ - S3_COMPATIBLE
+ - AZURE_BLOB_STORAGE
+ BlobStorageIntegrationFileType:
+ title: BlobStorageIntegrationFileType
+ type: string
+ enum:
+ - JSON
+ - CSV
+ - JSONL
+ - PARQUET
description: >-
- Pricing tier definition with conditional pricing based on usage
- thresholds or observation attributes.
-
-
- Pricing tiers enable accurate cost tracking for LLM providers that
- charge different rates based on usage patterns or request attributes.
-
- For example, some providers charge higher rates when context size
- exceeds certain thresholds or when an accelerated service tier is
- selected.
-
-
- How tier matching works:
-
- 1. Tiers are evaluated in ascending priority order (priority 1 before
- priority 2, etc.)
-
- 2. The first tier where ALL conditions match is selected
-
- 3. If no conditional tiers match, the default tier is used as a fallback
-
- 4. The default tier has priority 0 and no conditions
-
-
- Why priorities matter:
+ File format for exported data. `PARQUET` is a columnar binary format
+ encoded and compressed by the storage engine; gzip compression does not
+ apply to it. Note that the model-price columns (`input_price`,
+ `output_price`, `total_price`) are not included in Parquet observation
+ exports.
+ BlobStorageIntegrationFileTypeResponse:
+ title: BlobStorageIntegrationFileTypeResponse
+ type: string
+ enum:
+ - JSON
+ - CSV
+ - JSONL
+ - PARQUET
+ description: File type reported for an existing integration.
+ BlobStorageExportMode:
+ title: BlobStorageExportMode
+ type: string
+ enum:
+ - FULL_HISTORY
+ - FROM_TODAY
+ - FROM_CUSTOM_DATE
+ BlobStorageExportFrequency:
+ title: BlobStorageExportFrequency
+ type: string
+ enum:
+ - every_20_minutes
+ - hourly
+ - daily
+ - weekly
+ BlobStorageExportSource:
+ title: BlobStorageExportSource
+ type: string
+ enum:
+ - LEGACY_TRACES_OBSERVATIONS
+ - OBSERVATIONS_V2
+ - LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS
+ description: >-
+ What data the integration exports.
- - Lower priority numbers are evaluated first, allowing you to define
- specific cases before general ones
+ - `LEGACY_TRACES_OBSERVATIONS`: traces, observations, and scores tables.
+ Observation columns are controlled by `exportFieldGroups`; field groups
+ without a counterpart in this data model (e.g. `trace_context`) are
+ omitted.
- - Example: Priority 1 for "high usage" (>200K tokens), Priority 2 for
- "medium usage" (>100K tokens), Priority 0 for default
+ - `OBSERVATIONS_V2`: same data model as the
+ `/api/public/v2/observations` endpoint, plus scores. Columns are
+ controlled by `exportFieldGroups`.
- - Without proper ordering, a less specific condition might match before
- a more specific one
+ - `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS`: both sets. Observation
+ columns of both portions are controlled by `exportFieldGroups`.
- Every model must have exactly one default tier to ensure cost
- calculation always succeeds.
+ **Note:** which sources a deployment accepts depends on how far it has
+ moved to the v4 data model. `OBSERVATIONS_V2` and the
+ enriched-observations portion of
+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the enriched observations
+ table, so they require a deployment that already populates it.
+ `LEGACY_TRACES_OBSERVATIONS` and the legacy portion of
+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` read the legacy traces and
+ observations tables, so they require a deployment that still populates
+ those. A deployment part-way through the migration populates both and
+ accepts every source. Selecting a source the deployment cannot serve is
+ rejected with `400`, rather than exporting an empty result. See
+ https://langfuse.com/docs/v4.
+ BlobStorageExportFieldGroup:
+ title: BlobStorageExportFieldGroup
+ type: string
+ enum:
+ - core
+ - basic
+ - time
+ - io
+ - metadata
+ - model
+ - usage
+ - prompt
+ - metrics
+ - tools
+ - trace_context
+ description: >-
+ Field group selecting which observation columns are included in the
+ export. Applies to all export sources; groups without a counterpart in
+ the legacy data model (e.g. `trace_context`) are omitted from the legacy
+ observations export.
+ CreateBlobStorageIntegrationRequest:
+ title: CreateBlobStorageIntegrationRequest
+ type: object
properties:
- id:
+ projectId:
+ type: string
+ description: ID of the project in which to configure the blob storage integration
+ type:
+ $ref: '#/components/schemas/BlobStorageIntegrationType'
+ bucketName:
type: string
- description: Unique identifier for the pricing tier
- name:
+ description: >-
+ Name of the storage bucket. For AZURE_BLOB_STORAGE, must be a valid
+ Azure container name (3-63 chars, lowercase letters, numbers, and
+ hyphens only, must start and end with a letter or number, no
+ consecutive hyphens).
+ endpoint:
+ type: string
+ nullable: true
+ description: Custom endpoint URL (required for S3_COMPATIBLE type)
+ region:
type: string
description: >-
- Name of the pricing tier for display and identification purposes.
-
-
- Examples: "Standard", "High Volume Tier", "Large Context", "Extended
- Context Tier"
- isDefault:
+ Storage region used by S3-compatible clients (AWS, GCS, Cloudflare
+ R2, MinIO, Azure location IDs such as eastus, OCI). Leading and
+ trailing whitespace is removed. The remaining value must be 1-63
+ letters, numbers, or hyphens, and cannot start or end with a hyphen.
+ Examples: us-east-1, europe-west1, eastus, auto.
+ accessKeyId:
+ type: string
+ nullable: true
+ description: Access key ID for authentication
+ secretAccessKey:
+ type: string
+ nullable: true
+ description: Secret access key for authentication (will be encrypted when stored)
+ prefix:
+ type: string
+ nullable: true
+ description: >-
+ Path prefix for exported files (must end with forward slash if
+ provided)
+ exportFrequency:
+ $ref: '#/components/schemas/BlobStorageExportFrequency'
+ enabled:
+ type: boolean
+ description: Whether the integration is active
+ forcePathStyle:
type: boolean
+ description: Use path-style URLs for S3 requests
+ fileType:
+ $ref: '#/components/schemas/BlobStorageIntegrationFileType'
+ exportMode:
+ $ref: '#/components/schemas/BlobStorageExportMode'
+ exportStartDate:
+ type: string
+ format: date-time
+ nullable: true
description: >-
- Whether this is the default tier. Every model must have exactly one
- default tier with priority 0 and no conditions.
-
-
- The default tier serves as a fallback when no conditional tiers
- match, ensuring cost calculation always succeeds.
-
- It typically represents the base pricing for standard usage
- patterns.
- priority:
- type: integer
+ Custom start date for exports (required when exportMode is
+ FROM_CUSTOM_DATE). Must not be in the future (27 h tolerance for
+ timezone differences).
+ compressed:
+ type: boolean
+ nullable: true
description: >-
- Priority for tier matching evaluation. Lower numbers = higher
- priority (evaluated first).
-
-
- The default tier must always have priority 0. Conditional tiers
- should have priority 1, 2, 3, etc.
-
-
- Example ordering:
-
- - Priority 0: Default tier (no conditions, always matches as
- fallback)
+ Enable gzip compression for exported files (.csv.gz, .json.gz,
+ .jsonl.gz). Defaults to true.
+ exportSource:
+ $ref: '#/components/schemas/BlobStorageExportSource'
+ nullable: true
+ description: >-
+ Data to export. When omitted on update, the existing value is
+ preserved. When omitted on create, the default is `OBSERVATIONS_V2`
+ on Langfuse Cloud, and on self-hosted deployments
+ `LEGACY_TRACES_OBSERVATIONS` — or `OBSERVATIONS_V2` where the
+ deployment no longer populates the legacy tables. The default is
+ never a source the deployment cannot serve. Required when
+ `exportFieldGroups` is provided.
- - Priority 1: High usage tier (e.g., >200K tokens)
- - Priority 2: Medium usage tier (e.g., >100K tokens)
+ **Cloud-only project deprecation gate (effective 2026-05-20):** For
+ projects created on or after 2026-05-20 on Langfuse Cloud,
+ `LEGACY_TRACES_OBSERVATIONS` and
+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP
+ 400. Use `OBSERVATIONS_V2` for all new integrations. Self-hosted
+ deployments are unaffected.
- This ensures more specific conditions are checked before general
- ones.
- conditions:
+ **Cloud-only integration deprecation gate (effective 2026-06-22):**
+ On Langfuse Cloud, legacy export sources are only accepted for blob
+ storage integrations created before 2026-06-22, regardless of
+ project age. Requests that would create a new integration with
+ `LEGACY_TRACES_OBSERVATIONS` or
+ `LEGACY_TRACES_AND_ENRICHED_OBSERVATIONS` are rejected with HTTP
+ 400. Use `OBSERVATIONS_V2` instead. Self-hosted deployments are
+ unaffected.
+ exportFieldGroups:
type: array
items:
- $ref: '#/components/schemas/PricingTierCondition'
+ $ref: '#/components/schemas/BlobStorageExportFieldGroup'
+ nullable: true
description: >-
- Array of conditions that must ALL be met for this tier to match (AND
- logic).
-
-
- The default tier must have an empty conditions array. Conditional
- tiers should have one or more conditions
-
- that define when this tier's pricing applies.
+ Field groups to include in each exported observation row. Applies to
+ all export sources; must include `core` if provided. When omitted on
+ create, the column default (all groups) applies. When omitted on
+ update, the existing value is preserved.
- Multiple conditions enable complex matching scenarios (e.g., "high
- input tokens AND low output tokens").
- prices:
- type: object
- additionalProperties:
- type: number
- format: double
+ `exportFieldGroups` requires `exportSource` to be provided in the
+ same request.
+ required:
+ - projectId
+ - type
+ - bucketName
+ - region
+ - exportFrequency
+ - enabled
+ - forcePathStyle
+ - fileType
+ - exportMode
+ BlobStorageIntegrationResponse:
+ title: BlobStorageIntegrationResponse
+ type: object
+ properties:
+ id:
+ type: string
+ projectId:
+ type: string
+ type:
+ $ref: '#/components/schemas/BlobStorageIntegrationType'
+ bucketName:
+ type: string
+ endpoint:
+ type: string
+ nullable: true
+ region:
+ type: string
+ accessKeyId:
+ type: string
+ nullable: true
+ prefix:
+ type: string
+ exportFrequency:
+ $ref: '#/components/schemas/BlobStorageExportFrequency'
+ enabled:
+ type: boolean
+ forcePathStyle:
+ type: boolean
+ fileType:
+ $ref: '#/components/schemas/BlobStorageIntegrationFileTypeResponse'
+ exportMode:
+ $ref: '#/components/schemas/BlobStorageExportMode'
+ exportStartDate:
+ type: string
+ format: date-time
+ nullable: true
+ compressed:
+ type: boolean
+ exportSource:
+ $ref: '#/components/schemas/BlobStorageExportSource'
+ exportFieldGroups:
+ type: array
+ items:
+ $ref: '#/components/schemas/BlobStorageExportFieldGroup'
+ nullable: true
description: >-
- Prices (USD) by usage type for this tier.
-
-
- Common usage types: "input", "output", "total", "request", "image"
-
- Prices are specified in USD per unit (e.g., per token, per request,
- per second).
-
-
- Example: {"input": 0.000003, "output": 0.000015} means $3 per
- million input tokens and $15 per million output tokens.
+ Field groups included in each exported observation row. An empty
+ list is treated as all groups during export.
+ nextSyncAt:
+ type: string
+ format: date-time
+ nullable: true
+ lastSyncAt:
+ type: string
+ format: date-time
+ nullable: true
+ lastError:
+ type: string
+ nullable: true
+ lastErrorAt:
+ type: string
+ format: date-time
+ nullable: true
+ createdAt:
+ type: string
+ format: date-time
+ updatedAt:
+ type: string
+ format: date-time
required:
- id
- - name
- - isDefault
- - priority
- - conditions
- - prices
- PricingTierInput:
- title: PricingTierInput
+ - projectId
+ - type
+ - bucketName
+ - endpoint
+ - region
+ - accessKeyId
+ - prefix
+ - exportFrequency
+ - enabled
+ - forcePathStyle
+ - fileType
+ - exportMode
+ - exportStartDate
+ - compressed
+ - exportSource
+ - exportFieldGroups
+ - nextSyncAt
+ - lastSyncAt
+ - lastError
+ - lastErrorAt
+ - createdAt
+ - updatedAt
+ BlobStorageIntegrationsResponse:
+ title: BlobStorageIntegrationsResponse
type: object
+ properties:
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/BlobStorageIntegrationResponse'
+ required:
+ - data
+ BlobStorageSyncStatus:
+ title: BlobStorageSyncStatus
+ type: string
+ enum:
+ - idle
+ - running
+ - queued
+ - up_to_date
+ - disabled
+ - error
description: >-
- Input schema for creating a pricing tier. The tier ID will be
- automatically generated server-side.
+ Sync status of the blob storage integration:
+ - `disabled` — integration is not enabled
- When creating a model with pricing tiers:
+ - `error` — last export failed (see `lastError` for details)
- - Exactly one tier must have isDefault=true (the fallback tier)
+ - `running` — an export job is currently being processed
- - The default tier must have priority=0 and conditions=[]
+ - `queued` — next export is overdue (`nextSyncAt` is in the past) and
+ waiting to be picked up by the worker
- - All tier names and priorities must be unique within the model
+ - `idle` — enabled but has never exported yet and no export is queued
- - Each tier must define at least one price
+ - `up_to_date` — all available data has been exported; next export is
+ scheduled for the future
- See PricingTier for detailed information about how tiers work and why
- they're useful.
- properties:
- name:
- type: string
- description: >-
- Name of the pricing tier for display and identification purposes.
+ **ETL usage**: poll this endpoint and check for `up_to_date` status.
+ Compare `lastSyncAt` against your
+ ETL bookmark to determine if new data is available. Note that exports
+ run with a 20-minute lag buffer,
- Must be unique within the model. Common patterns: "Standard", "High
- Volume Tier", "Extended Context"
- isDefault:
+ so `lastSyncAt` will always be at least 20 minutes behind real-time.
+ BlobStorageIntegrationStatusResponse:
+ title: BlobStorageIntegrationStatusResponse
+ type: object
+ properties:
+ id:
+ type: string
+ projectId:
+ type: string
+ syncStatus:
+ $ref: '#/components/schemas/BlobStorageSyncStatus'
+ enabled:
type: boolean
+ lastSyncAt:
+ type: string
+ format: date-time
nullable: true
description: >-
- Whether this is the default tier. Exactly one tier per model must be
- marked as default.
-
-
- Requirements for default tier:
-
- - Must have isDefault=true
-
- - Must have priority=0
-
- - Must have empty conditions array (conditions=[])
-
-
- The default tier acts as a fallback when no conditional tiers match.
- Defaults to false when omitted.
- priority:
- type: integer
+ End of the last successfully exported time window. Compare against
+ your ETL bookmark to determine if new data is available. Null if the
+ integration has never synced.
+ nextSyncAt:
+ type: string
+ format: date-time
+ nullable: true
+ description: When the next export is scheduled. Null if no sync has occurred yet.
+ lastError:
+ type: string
+ nullable: true
description: >-
- Priority for tier matching evaluation. Lower numbers = higher
- priority (evaluated first).
-
-
- Must be unique within the model. The default tier must have
- priority=0.
-
- Conditional tiers should use priority 1, 2, 3, etc. based on their
- specificity.
- conditions:
+ Raw error message from the storage provider (S3/Azure/GCS) if the
+ last export failed. Cleared on successful export.
+ lastErrorAt:
+ type: string
+ format: date-time
+ nullable: true
+ description: When the last error occurred. Cleared on successful export.
+ required:
+ - id
+ - projectId
+ - syncStatus
+ - enabled
+ - lastSyncAt
+ - nextSyncAt
+ - lastError
+ - lastErrorAt
+ BlobStorageIntegrationDeletionResponse:
+ title: BlobStorageIntegrationDeletionResponse
+ type: object
+ properties:
+ message:
+ type: string
+ required:
+ - message
+ CreateCommentRequest:
+ title: CreateCommentRequest
+ type: object
+ properties:
+ projectId:
+ type: string
+ description: The id of the project to attach the comment to.
+ objectType:
+ type: string
+ description: >-
+ The type of the object to attach the comment to (trace, observation,
+ session, prompt).
+ objectId:
+ type: string
+ description: >-
+ The id of the object to attach the comment to. If this does not
+ reference a valid existing object, an error will be thrown.
+ content:
+ type: string
+ description: >-
+ The content of the comment. May include markdown. Currently limited
+ to 5000 characters.
+ authorUserId:
+ type: string
+ nullable: true
+ description: >-
+ The id of the user who created the comment. Must be a member of the
+ organization that owns the project, otherwise an error will be
+ thrown.
+ required:
+ - projectId
+ - objectType
+ - objectId
+ - content
+ CreateCommentResponse:
+ title: CreateCommentResponse
+ type: object
+ properties:
+ id:
+ type: string
+ description: The id of the created object in Langfuse
+ required:
+ - id
+ GetCommentsResponse:
+ title: GetCommentsResponse
+ type: object
+ properties:
+ data:
type: array
items:
- $ref: '#/components/schemas/PricingTierConditionInput'
+ $ref: '#/components/schemas/Comment'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ required:
+ - data
+ - meta
+ Deprecation:
+ title: Deprecation
+ type: object
+ description: >-
+ Migration signal returned by deprecated endpoints. Optional fields are
+ omitted when they have no value.
+ properties:
+ message:
+ type: string
description: >-
- Array of conditions that must ALL be met for this tier to match (AND
- logic).
-
-
- The default tier must have an empty array (conditions=[]).
-
- Conditional tiers should define one or more conditions that specify
- when this tier's pricing applies.
-
-
- Conditions can compare summed matching usage details to a numeric
- threshold, or exactly match a top-level model parameter or metadata
- value.
- prices:
- type: object
- additionalProperties:
- type: number
- format: double
+ Human- and agent-readable summary of the deprecation and its
+ replacement.
+ replacement:
+ type: string
+ nullable: true
+ description: >-
+ The replacement endpoint, e.g. "GET /api/public/v2/observations".
+ Omitted when the endpoint is being removed without a direct
+ replacement.
+ docsUrl:
+ type: string
+ nullable: true
+ description: Link to the migration documentation (markdown), when available.
+ sunsetAt:
+ type: string
+ nullable: true
+ description: >-
+ ISO date after which the endpoint may stop working, when a removal
+ date is committed.
+ required:
+ - message
+ Trace:
+ title: Trace
+ type: object
+ properties:
+ id:
+ type: string
+ description: The unique identifier of a trace
+ timestamp:
+ type: string
+ format: date-time
+ description: The timestamp when the trace was created
+ name:
+ type: string
+ nullable: true
+ description: The name of the trace
+ input:
+ nullable: true
+ description: The input data of the trace. Can be any JSON.
+ output:
+ nullable: true
+ description: The output data of the trace. Can be any JSON.
+ sessionId:
+ type: string
+ nullable: true
+ description: The session identifier associated with the trace
+ release:
+ type: string
+ nullable: true
+ description: The release version of the application when the trace was created
+ version:
+ type: string
+ nullable: true
+ description: The version of the trace
+ userId:
+ type: string
+ nullable: true
+ description: The user identifier associated with the trace
+ metadata:
+ nullable: true
+ description: The metadata associated with the trace. Can be any JSON.
+ tags:
+ type: array
+ items:
+ type: string
+ description: The tags associated with the trace.
+ public:
+ type: boolean
+ description: Public traces are accessible via url without login
+ environment:
+ type: string
description: >-
- Prices (USD) by usage type for this tier. At least one price must be
- defined.
-
-
- Common usage types: "input", "output", "total", "request", "image"
-
- Prices are in USD per unit (e.g., per token).
-
-
- Example: {"input": 0.000003, "output": 0.000015} represents $3 per
- million input tokens and $15 per million output tokens.
+ The environment from which this trace originated. Can be any
+ lowercase alphanumeric string with hyphens and underscores that does
+ not start with 'langfuse'.
required:
+ - id
+ - timestamp
- name
- - priority
- - conditions
- - prices
- PricingTierOperator:
- title: PricingTierOperator
- type: string
- enum:
- - gt
- - gte
- - lt
- - lte
- - eq
- - neq
- description: Comparison operators for pricing tier conditions
- ModelUsageUnit:
- title: ModelUsageUnit
- type: string
- enum:
- - CHARACTERS
- - TOKENS
- - MILLISECONDS
- - SECONDS
- - IMAGES
- - REQUESTS
- description: Unit of usage in Langfuse
- ObservationLevel:
- title: ObservationLevel
- type: string
- enum:
- - DEBUG
- - DEFAULT
- - WARNING
- - ERROR
- MapValue:
- title: MapValue
- oneOf:
- - type: string
- nullable: true
- title: String
- - type: integer
+ - sessionId
+ - release
+ - version
+ - userId
+ - tags
+ - public
+ - environment
+ TraceWithDetails:
+ title: TraceWithDetails
+ type: object
+ properties:
+ htmlPath:
+ type: string
+ description: Path of trace in Langfuse UI
+ latency:
+ type: number
+ format: double
nullable: true
- title: Integer
- - type: number
- format: float
+ description: Latency of trace in seconds
+ totalCost:
+ type: number
+ format: double
nullable: true
- title: FloatNumber
- - type: boolean
+ description: Cost of trace in USD
+ observations:
+ type: array
+ items:
+ type: string
nullable: true
- title: Boolean
- - type: array
+ description: List of observation ids
+ scores:
+ type: array
items:
type: string
nullable: true
- title: Array
- CommentObjectType:
- title: CommentObjectType
- type: string
- enum:
- - TRACE
- - OBSERVATION
- - SESSION
- - PROMPT
- DatasetStatus:
- title: DatasetStatus
- type: string
- enum:
- - ACTIVE
- - ARCHIVED
- ScoreSource:
- title: ScoreSource
- type: string
- enum:
- - ANNOTATION
- - API
- - EVAL
- ScoreConfigDataType:
- title: ScoreConfigDataType
- type: string
- enum:
- - NUMERIC
- - BOOLEAN
- - CATEGORICAL
- - TEXT
- ScoreDataType:
- title: ScoreDataType
- type: string
- enum:
- - NUMERIC
- - BOOLEAN
- - CATEGORICAL
- - CORRECTION
- - TEXT
- DeleteDatasetItemResponse:
- title: DeleteDatasetItemResponse
+ description: List of score ids
+ required:
+ - htmlPath
+ allOf:
+ - $ref: '#/components/schemas/Trace'
+ TraceWithFullDetails:
+ title: TraceWithFullDetails
type: object
properties:
- message:
+ htmlPath:
+ type: string
+ description: Path of trace in Langfuse UI
+ latency:
+ type: number
+ format: double
+ nullable: true
+ description: Latency of trace in seconds
+ totalCost:
+ type: number
+ format: double
+ nullable: true
+ description: Cost of trace in USD
+ observations:
+ type: array
+ items:
+ $ref: '#/components/schemas/ObservationsView'
+ description: List of observations
+ scores:
+ type: array
+ items:
+ $ref: '#/components/schemas/ScoreV1'
+ description: List of scores
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
+ required:
+ - htmlPath
+ - observations
+ - scores
+ allOf:
+ - $ref: '#/components/schemas/Trace'
+ Session:
+ title: Session
+ type: object
+ properties:
+ id:
+ type: string
+ createdAt:
+ type: string
+ format: date-time
+ projectId:
+ type: string
+ environment:
+ type: string
+ description: The environment from which this session originated.
+ required:
+ - id
+ - createdAt
+ - projectId
+ - environment
+ SessionWithTraces:
+ title: SessionWithTraces
+ type: object
+ properties:
+ traces:
+ type: array
+ items:
+ $ref: '#/components/schemas/Trace'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
+ required:
+ - traces
+ allOf:
+ - $ref: '#/components/schemas/Session'
+ Observation:
+ title: Observation
+ type: object
+ properties:
+ id:
+ type: string
+ description: The unique identifier of the observation
+ traceId:
+ type: string
+ nullable: true
+ description: The trace ID associated with the observation
+ type:
+ type: string
+ description: The type of the observation
+ name:
+ type: string
+ nullable: true
+ description: The name of the observation
+ startTime:
+ type: string
+ format: date-time
+ description: The start time of the observation
+ endTime:
+ type: string
+ format: date-time
+ nullable: true
+ description: The end time of the observation.
+ completionStartTime:
type: string
- description: Success message after deletion
- required:
- - message
- CreateDatasetItemRequest:
- title: CreateDatasetItemRequest
- type: object
- properties:
- datasetName:
+ format: date-time
+ nullable: true
+ description: The completion start time of the observation
+ model:
type: string
- input:
nullable: true
- expectedOutput:
+ description: The model used for the observation
+ modelParameters:
+ description: The parameters of the model used for the observation
+ input:
+ description: The input data of the observation
+ version:
+ type: string
nullable: true
+ description: The version of the observation
metadata:
+ description: Additional metadata of the observation
+ output:
+ description: The output data of the observation
+ usage:
+ $ref: '#/components/schemas/Usage'
+ description: >-
+ (Deprecated. Use usageDetails and costDetails instead.) The usage
+ data of the observation
+ level:
+ $ref: '#/components/schemas/ObservationLevel'
+ description: The level of the observation
+ statusMessage:
+ type: string
nullable: true
- sourceTraceId:
+ description: The status message of the observation
+ parentObservationId:
type: string
nullable: true
- sourceObservationId:
+ description: The parent observation ID
+ promptId:
type: string
nullable: true
- id:
+ description: The prompt ID associated with the observation
+ usageDetails:
+ type: object
+ additionalProperties:
+ type: integer
+ description: >-
+ The usage details of the observation. Key is the name of the usage
+ metric, value is the number of units consumed. The total key is the
+ sum of all (non-total) usage metrics or the total value ingested.
+ costDetails:
+ type: object
+ additionalProperties:
+ type: number
+ format: double
+ description: >-
+ The cost details of the observation. Key is the name of the cost
+ metric, value is the cost in USD. The total key is the sum of all
+ (non-total) cost metrics or the total value ingested.
+ environment:
+ type: string
+ description: >-
+ The environment from which this observation originated. Can be any
+ lowercase alphanumeric string with hyphens and underscores that does
+ not start with 'langfuse'.
+ required:
+ - id
+ - traceId
+ - type
+ - name
+ - startTime
+ - endTime
+ - completionStartTime
+ - model
+ - modelParameters
+ - input
+ - version
+ - metadata
+ - output
+ - usage
+ - level
+ - statusMessage
+ - parentObservationId
+ - promptId
+ - usageDetails
+ - costDetails
+ - environment
+ ObservationsView:
+ title: ObservationsView
+ type: object
+ properties:
+ promptName:
+ type: string
+ nullable: true
+ description: The name of the prompt associated with the observation
+ promptVersion:
+ type: integer
+ nullable: true
+ description: The version of the prompt associated with the observation
+ modelId:
type: string
nullable: true
+ description: The unique identifier of the model
+ inputPrice:
+ type: number
+ format: double
+ nullable: true
+ description: The price of the input in USD
+ outputPrice:
+ type: number
+ format: double
+ nullable: true
+ description: The price of the output in USD.
+ totalPrice:
+ type: number
+ format: double
+ nullable: true
+ description: The total price in USD.
+ calculatedInputCost:
+ type: number
+ format: double
+ nullable: true
description: >-
- Dataset items are upserted on their id. Id needs to be unique
- (project-level), cannot be reused across datasets, and must be at
- most 255 characters.
- status:
- $ref: '#/components/schemas/DatasetStatus'
+ (Deprecated. Use usageDetails and costDetails instead.) The
+ calculated cost of the input in USD
+ calculatedOutputCost:
+ type: number
+ format: double
nullable: true
- description: Defaults to ACTIVE for newly created items
+ description: >-
+ (Deprecated. Use usageDetails and costDetails instead.) The
+ calculated cost of the output in USD
+ calculatedTotalCost:
+ type: number
+ format: double
+ nullable: true
+ description: >-
+ (Deprecated. Use usageDetails and costDetails instead.) The
+ calculated total cost in USD
+ latency:
+ type: number
+ format: double
+ nullable: true
+ description: The latency in seconds.
+ timeToFirstToken:
+ type: number
+ format: double
+ nullable: true
+ description: The time to the first token in seconds
required:
- - datasetName
- PaginatedDatasetItems:
- title: PaginatedDatasetItems
+ - promptName
+ - promptVersion
+ - modelId
+ - inputPrice
+ - outputPrice
+ - totalPrice
+ - calculatedInputCost
+ - calculatedOutputCost
+ - calculatedTotalCost
+ - latency
+ - timeToFirstToken
+ allOf:
+ - $ref: '#/components/schemas/Observation'
+ ObservationsViewSingle:
+ title: ObservationsViewSingle
type: object
properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/DatasetItem'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- required:
- - data
- - meta
- CreateDatasetRunItemRequest:
- title: CreateDatasetRunItemRequest
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/ObservationsView'
+ ObservationV2:
+ title: ObservationV2
type: object
+ description: >-
+ An observation from the v2 API with field-group-based selection.
+
+ Core fields are always present. Other fields are included only when
+ their field group is requested.
properties:
- runName:
+ id:
type: string
- runDescription:
+ description: The unique identifier of the observation
+ traceId:
+ type: string
+ nullable: true
+ description: The trace ID associated with the observation
+ startTime:
+ type: string
+ format: date-time
+ description: The start time of the observation
+ endTime:
+ type: string
+ format: date-time
+ nullable: true
+ description: The end time of the observation
+ projectId:
+ type: string
+ description: The project ID this observation belongs to
+ parentObservationId:
+ type: string
+ nullable: true
+ description: >-
+ The physical parent observation ID, if present.
+
+ Observations marked as app roots by the SDK may retain a non-null
+ parent ID.
+ type:
+ type: string
+ description: The type of the observation (e.g. GENERATION, SPAN, EVENT)
+ isRootObservation:
+ type: boolean
+ nullable: true
+ description: >-
+ Whether this observation is a logical root.
+
+ This is true for observations without a physical parent and
+ observations marked as app roots by the SDK.
+ name:
+ type: string
+ nullable: true
+ description: The name of the observation
+ level:
+ $ref: '#/components/schemas/ObservationLevel'
+ nullable: true
+ description: The level of the observation
+ statusMessage:
+ type: string
+ nullable: true
+ description: The status message of the observation
+ version:
+ type: string
+ nullable: true
+ description: The version of the observation
+ environment:
type: string
nullable: true
- description: Description of the run. If run exists, description will be updated.
- metadata:
+ description: The environment from which this observation originated
+ bookmarked:
+ type: boolean
+ nullable: true
+ description: Whether the observation is bookmarked
+ public:
+ type: boolean
nullable: true
- description: Metadata of the dataset run, updates run if run already exists
- datasetItemId:
- type: string
- observationId:
+ description: Whether the observation is public
+ userId:
type: string
nullable: true
- traceId:
+ description: The user ID associated with the observation
+ sessionId:
type: string
nullable: true
- description: >-
- traceId should always be provided. For compatibility with older SDK
- versions it can also be inferred from the provided observationId.
- datasetVersion:
+ description: The session ID associated with the observation
+ completionStartTime:
type: string
format: date-time
nullable: true
- description: >-
- ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g.,
- "2026-01-21T14:35:42Z").
-
- Specifies the dataset version to use for this experiment run.
-
- If provided, the experiment will use dataset items as they existed
- at or before this timestamp.
-
- If not provided, uses the latest version of dataset items.
+ description: The completion start time of the observation
createdAt:
type: string
format: date-time
nullable: true
- description: >-
- Optional timestamp to set the createdAt field of the dataset run
- item. If not provided or null, defaults to current timestamp.
- required:
- - runName
- - datasetItemId
- PaginatedDatasetRunItems:
- title: PaginatedDatasetRunItems
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/DatasetRunItem'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
- nullable: true
- required:
- - data
- - meta
- PaginatedDatasets:
- title: PaginatedDatasets
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Dataset'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- required:
- - data
- - meta
- CreateDatasetRequest:
- title: CreateDatasetRequest
- type: object
- properties:
- name:
- type: string
- description:
+ description: The creation timestamp of the observation
+ updatedAt:
type: string
+ format: date-time
nullable: true
- metadata:
+ description: The last update timestamp of the observation
+ input:
nullable: true
- inputSchema:
+ description: The input data of the observation
+ output:
nullable: true
- description: >-
- JSON Schema for validating dataset item inputs. When set, all new
- and existing dataset items will be validated against this schema.
- expectedOutputSchema:
+ description: The output data of the observation
+ metadata:
nullable: true
- description: >-
- JSON Schema for validating dataset item expected outputs. When set,
- all new and existing dataset items will be validated against this
- schema.
- required:
- - name
- PaginatedDatasetRuns:
- title: PaginatedDatasetRuns
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/DatasetRun'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
+ description: Additional metadata of the observation
+ model:
+ type: string
nullable: true
- required:
- - data
- - meta
- DeleteDatasetRunResponse:
- title: DeleteDatasetRunResponse
- type: object
- properties:
- message:
+ description: The model name as provided by the user
+ internalModelId:
type: string
- required:
- - message
- EvaluatorType:
- title: EvaluatorType
- type: string
- enum:
- - llm_as_judge
- - code
- description: |-
- The evaluator type.
-
- The public API supports LLM-as-a-judge and code evaluators.
- CodeEvaluatorSourceCodeLanguage:
- title: CodeEvaluatorSourceCodeLanguage
- type: string
- enum:
- - PYTHON
- - TYPESCRIPT
- description: Code evaluator runtime language.
- PromptVariableMappingSource:
- title: PromptVariableMappingSource
- type: string
- enum:
- - input
- - output
- - metadata
- - tool_calls
- - expected_output
- - experiment_item_metadata
- description: >-
- Source field used to populate a prompt variable.
-
-
- Use these values when mapping evaluator prompt variables to live data.
-
-
- Source semantics:
-
- - `input`: the observation input payload
-
- - `output`: the observation output payload
-
- - `metadata`: the observation metadata object. Combine with `jsonPath`
- when you need one nested field instead of the whole object.
-
- - `tool_calls`: the tool calls recorded on the observation, as an array
- of `{id, name, arguments, type, index}` objects in the order the model
- emitted them. Combine with `jsonPath` (for example `$[*].name`) to
- select parts of each call.
-
- - `expected_output`: the experiment item's expected output when the
- observation belongs to an experiment.
-
- - `experiment_item_metadata`: the experiment item's metadata when the
- observation belongs to an experiment.
- EvaluatorOutputScoreType:
- title: EvaluatorOutputScoreType
- type: string
- enum:
- - NUMERIC
- - BOOLEAN
- - CATEGORICAL
- description: >-
- Structured score type returned by an evaluator.
-
-
- This controls the type of score value Langfuse stores for evaluation
- results:
-
- - `NUMERIC`: a numeric score such as `0.82`
-
- - `BOOLEAN`: a boolean score such as `true`
-
- - `CATEGORICAL`: one or more category labels from a fixed list
- EvaluatorOutputDefinition:
- title: EvaluatorOutputDefinition
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: '#/components/schemas/PublicEvaluatorNumericScore'
- required:
- - dataType
- title: PublicEvaluatorNumericScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'
- required:
- - dataType
- title: PublicEvaluatorBooleanScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'
- required:
- - dataType
- title: PublicEvaluatorCategoricalScore
- description: >-
- Flat structured output definition used when creating or updating an
- evaluator.
-
-
- - `dataType` is required.
-
- - `scoreReasoningInstructions` and `scoreValueInstructions` are optional
- instructions.
-
- - `minValue` and `maxValue` apply only to `NUMERIC` outputs. If both are
- set, `minValue` must not exceed `maxValue`.
-
- - `categories` and `shouldAllowMultipleMatches` apply only to
- `CATEGORICAL` outputs.
-
- - Do not send `version`; that is an internal storage detail.
- EvaluatorOutputDefinitionBase:
- title: EvaluatorOutputDefinitionBase
- type: object
- properties:
- scoreReasoningInstructions:
+ nullable: true
+ description: The internal model ID matched by Langfuse
+ modelParameters:
+ nullable: true
+ description: The parameters of the model used for the observation
+ usageDetails:
+ type: object
+ additionalProperties:
+ type: integer
+ nullable: true
+ description: >-
+ The usage details of the observation. Key is the usage metric name,
+ value is the number of units consumed.
+ costDetails:
+ type: object
+ additionalProperties:
+ type: number
+ format: double
+ nullable: true
+ description: >-
+ The cost details of the observation. Key is the cost metric name,
+ value is the cost in USD.
+ totalCost:
+ type: number
+ format: double
+ nullable: true
+ description: The total cost of the observation in USD
+ usagePricingTierName:
type: string
nullable: true
description: >-
- Optional instructions for deriving the reasoning returned with the
- score.
- scoreValueInstructions:
+ The name of the pricing tier applied to this observation's usage
+ costs
+ promptId:
type: string
nullable: true
- description: Optional instructions for deriving the score value.
- PublicEvaluatorNumericScore:
- title: PublicEvaluatorNumericScore
- type: object
- properties:
- dataType:
+ description: The prompt ID associated with the observation
+ promptName:
type: string
- const: NUMERIC
- description: Numeric score output.
- minValue:
+ nullable: true
+ description: The prompt name associated with the observation
+ promptVersion:
+ type: integer
+ nullable: true
+ description: The prompt version associated with the observation
+ latency:
type: number
format: double
nullable: true
- description: Optional inclusive minimum value.
- maxValue:
+ description: The latency in seconds
+ timeToFirstToken:
type: number
format: double
nullable: true
- description: Optional inclusive maximum value.
- required:
- - dataType
- allOf:
- - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'
- PublicEvaluatorBooleanScore:
- title: PublicEvaluatorBooleanScore
- type: object
- properties:
- dataType:
- type: string
- const: BOOLEAN
- description: Boolean score output.
- required:
- - dataType
- allOf:
- - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'
- PublicEvaluatorCategoricalScore:
- title: PublicEvaluatorCategoricalScore
- type: object
- properties:
- dataType:
- type: string
- const: CATEGORICAL
- description: Categorical score output.
- categories:
- type: array
- items:
- type: string
- description: Allowed category values. At least two unique values are required.
- shouldAllowMultipleMatches:
- type: boolean
- description: Whether the evaluator may return more than one category.
- required:
- - dataType
- - categories
- - shouldAllowMultipleMatches
- allOf:
- - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'
- PublicEvaluatorOutputDefinition:
- title: PublicEvaluatorOutputDefinition
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: '#/components/schemas/PublicEvaluatorNumericScore'
- required:
- - dataType
- title: PublicEvaluatorNumericScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'
- required:
- - dataType
- title: PublicEvaluatorBooleanScore
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'
- required:
- - dataType
- title: PublicEvaluatorCategoricalScore
- description: >-
- Flat evaluator output definition returned by the public API.
-
-
- This response always includes `dataType` and never includes an internal
- output-definition `version`.
-
- Optional empty descriptions from legacy definitions are omitted.
- EvaluationRuleStringFilterOperator:
- title: EvaluationRuleStringFilterOperator
- type: string
- enum:
- - '='
- - contains
- - does not contain
- - starts with
- - ends with
- EvaluationRuleNumberFilterOperator:
- title: EvaluationRuleNumberFilterOperator
- type: string
- enum:
- - '='
- - '>'
- - <
- - '>='
- - <=
- EvaluationRuleOptionsFilterOperator:
- title: EvaluationRuleOptionsFilterOperator
- type: string
- enum:
- - any of
- - none of
- EvaluationRuleArrayOptionsFilterOperator:
- title: EvaluationRuleArrayOptionsFilterOperator
- type: string
- enum:
- - any of
- - none of
- - all of
- EvaluationRuleBooleanFilterOperator:
- title: EvaluationRuleBooleanFilterOperator
- type: string
- enum:
- - '='
- - <>
- EvaluationRuleNullFilterOperator:
- title: EvaluationRuleNullFilterOperator
- type: string
- enum:
- - is null
- - is not null
- DateTimeEvaluationRuleFilter:
- title: DateTimeEvaluationRuleFilter
- type: object
- properties:
- column:
- type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'
- description: Comparison operator for datetime values.
- value:
+ description: The time to first token in seconds
+ modelId:
type: string
- format: date-time
- description: Datetime value to compare against.
- required:
- - column
- - operator
- - value
- StringEvaluationRuleFilter:
- title: StringEvaluationRuleFilter
- type: object
- properties:
- column:
+ nullable: true
+ description: >-
+ The matched model ID. Null when the `model` field group is not
+ requested.
+ inputPrice:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'
- value:
+ nullable: true
+ description: >-
+ The input token price (USD per unit) from the matched model,
+ serialized as a decimal string (e.g. "0.0001"). Null when the
+ `model` field group is not requested.
+ outputPrice:
type: string
- required:
- - column
- - operator
- - value
- NumberEvaluationRuleFilter:
- title: NumberEvaluationRuleFilter
- type: object
- properties:
- column:
+ nullable: true
+ description: >-
+ The output token price (USD per unit) from the matched model,
+ serialized as a decimal string (e.g. "0.0001"). Null when the
+ `model` field group is not requested.
+ totalPrice:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'
- value:
- type: number
- format: double
- required:
- - column
- - operator
- - value
- StringOptionsEvaluationRuleFilter:
- title: StringOptionsEvaluationRuleFilter
- type: object
- properties:
- column:
+ nullable: true
+ description: >-
+ The total token price (USD per unit) from the matched model,
+ serialized as a decimal string (e.g. "0.0001"). Null when the
+ `model` field group is not requested.
+ traceName:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'
- value:
+ nullable: true
+ description: The name of the parent trace
+ tags:
type: array
items:
type: string
- description: One or more allowed string values.
+ nullable: true
+ description: Tags from the parent trace (denormalized onto the observation)
+ release:
+ type: string
+ nullable: true
+ description: The release version of the parent trace
required:
- - column
- - operator
- - value
- ArrayOptionsEvaluationRuleFilter:
- title: ArrayOptionsEvaluationRuleFilter
+ - id
+ - traceId
+ - startTime
+ - endTime
+ - projectId
+ - parentObservationId
+ - type
+ - modelId
+ - inputPrice
+ - outputPrice
+ - totalPrice
+ Usage:
+ title: Usage
type: object
+ description: >-
+ (Deprecated. Use usageDetails and costDetails instead.) Standard
+ interface for usage and cost
properties:
- column:
+ input:
+ type: integer
+ description: Number of input units (e.g. tokens)
+ output:
+ type: integer
+ description: Number of output units (e.g. tokens)
+ total:
+ type: integer
+ description: Defaults to input+output if not set
+ unit:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleArrayOptionsFilterOperator'
- value:
- type: array
- items:
- type: string
- description: One or more array elements to match.
+ nullable: true
+ description: Unit of measurement
+ inputCost:
+ type: number
+ format: double
+ nullable: true
+ description: USD input cost
+ outputCost:
+ type: number
+ format: double
+ nullable: true
+ description: USD output cost
+ totalCost:
+ type: number
+ format: double
+ nullable: true
+ description: USD total cost, defaults to input+output
required:
- - column
- - operator
- - value
- StringObjectEvaluationRuleFilter:
- title: StringObjectEvaluationRuleFilter
+ - input
+ - output
+ - total
+ - unit
+ ScoreConfig:
+ title: ScoreConfig
type: object
+ description: Configuration for a score
properties:
- column:
+ id:
type: string
- description: >-
- Object-valued column to filter on. Currently only `metadata` is
- supported.
- key:
+ name:
type: string
- description: Top-level key inside the object-valued column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'
- value:
+ createdAt:
type: string
- required:
- - column
- - key
- - operator
- - value
- NumberObjectEvaluationRuleFilter:
- title: NumberObjectEvaluationRuleFilter
- type: object
- properties:
- column:
+ format: date-time
+ updatedAt:
type: string
- description: Object-valued column to filter on.
- key:
+ format: date-time
+ projectId:
type: string
- description: Key inside the object-valued column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'
- value:
+ dataType:
+ $ref: '#/components/schemas/ScoreConfigDataType'
+ isArchived:
+ type: boolean
+ description: Whether the score config is archived. Defaults to false
+ minValue:
type: number
format: double
- required:
- - column
- - key
- - operator
- - value
- CategoryOptionsEvaluationRuleFilter:
- title: CategoryOptionsEvaluationRuleFilter
- type: object
- properties:
- column:
- type: string
- description: Object-valued column to filter on.
- key:
- type: string
- description: Key inside the object-valued column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'
- value:
+ nullable: true
+ description: >-
+ Sets minimum value for numerical scores. If not set, the minimum
+ value defaults to -∞
+ maxValue:
+ type: number
+ format: double
+ nullable: true
+ description: >-
+ Sets maximum value for numerical scores. If not set, the maximum
+ value defaults to +∞
+ categories:
type: array
items:
- type: string
- required:
- - column
- - key
- - operator
- - value
- BooleanEvaluationRuleFilter:
- title: BooleanEvaluationRuleFilter
- type: object
- properties:
- column:
+ $ref: '#/components/schemas/ConfigCategory'
+ nullable: true
+ description: Configures custom categories for categorical scores
+ description:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/EvaluationRuleBooleanFilterOperator'
- value:
- type: boolean
+ nullable: true
+ description: Description of the score config
required:
- - column
- - operator
- - value
- NullEvaluationRuleFilter:
- title: NullEvaluationRuleFilter
+ - id
+ - name
+ - createdAt
+ - updatedAt
+ - projectId
+ - dataType
+ - isArchived
+ ConfigCategory:
+ title: ConfigCategory
type: object
properties:
- column:
- type: string
- description: Column to filter on, for example `parentObservationId`.
- operator:
- $ref: '#/components/schemas/EvaluationRuleNullFilterOperator'
value:
+ type: number
+ format: double
+ label:
type: string
- const: ''
- description: >-
- Required empty-string placeholder used by the runtime filter
- contract.
- required:
- - column
- - operator
- - value
- PromptVariableMappingInput:
- title: PromptVariableMappingInput
- type: object
- description: >-
- Connects one prompt variable to data from an observation or experiment.
-
-
- Manual mappings are used for `llm_as_judge` evaluators. `code`
- evaluators use a fixed runtime mapping managed by Langfuse.
-
-
- How to build a valid mapping list:
-
- 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.
-
- 2. Read the evaluator `variables` array.
-
- 3. Add exactly one mapping object for each variable in that array.
-
- 4. Use the variable name exactly as returned, without braces such as
- `{{` or `}}`.
-
- 5. Choose the source field that should populate the variable.
-
-
- `jsonPath` is optional. Use it only when the selected source is a JSON
- object and you want to extract one nested field before inserting it into
- the evaluator prompt.
-
-
- Invalid, missing, or duplicate mappings return a validation error.
- Malformed JSONPath expressions are also rejected.
- properties:
- variable:
- type: string
- description: >-
- Prompt variable name without braces.
-
-
- Example: for the prompt `Judge {{input}} against {{output}}`, use
- `input` and `output`.
- source:
- $ref: '#/components/schemas/PromptVariableMappingSource'
- description: >-
- Source field that should populate the prompt variable.
-
-
- Available sources are `input`, `output`, `metadata`, `tool_calls`,
- `expected_output`, and `experiment_item_metadata`.
- jsonPath:
- type: string
- nullable: true
- description: >-
- Optional JSONPath selector applied to the selected source before it
- is passed to the evaluator prompt.
-
-
- Requirements:
-
- - Must start with `$`
-
- - Must be a syntactically valid JSONPath expression
-
- - Most useful with `source=metadata`
required:
- - variable
- - source
- PromptVariableMappingRead:
- title: PromptVariableMappingRead
+ - value
+ - label
+ BaseScoreV1:
+ title: BaseScoreV1
type: object
- description: |-
- Connects one prompt variable to source data.
-
- `source` is `null` when the mapping is incomplete.
properties:
- variable:
+ id:
+ type: string
+ traceId:
+ type: string
+ name:
type: string
- description: Prompt variable name without braces.
source:
+ $ref: '#/components/schemas/ScoreSource'
+ observationId:
type: string
nullable: true
- description: >-
- Stored source field populating the variable, or `null` when the
- mapping is incomplete.
- jsonPath:
+ description: The observation ID associated with the score
+ timestamp:
type: string
- nullable: true
- description: Optional JSONPath selector applied to the selected source.
- required:
- - variable
- - source
- PromptVariableMapping:
- title: PromptVariableMapping
- oneOf:
- - $ref: '#/components/schemas/PromptVariableMappingRead'
- - $ref: '#/components/schemas/LegacyPromptVariableMapping'
- description: Connects one prompt variable to source data.
- LegacyPromptVariableMapping:
- title: LegacyPromptVariableMapping
- type: object
- description: >-
- **Deprecated:** Connects one prompt variable to data from a legacy trace
- or dataset evaluation rule.
-
-
- `langfuseObject` selects the object kind. `objectName` separately
- selects a named observation and is `null` for trace and dataset-item
- mappings.
- properties:
- mappingType:
+ format: date-time
+ createdAt:
type: string
- const: legacy
- description: Explicitly marks this as a legacy mapping.
- variable:
+ format: date-time
+ updatedAt:
type: string
- description: Prompt variable name without braces.
- langfuseObject:
- $ref: '#/components/schemas/LegacyEvaluationObject'
- description: Legacy object kind selected as the mapping source.
- objectName:
+ format: date-time
+ authorUserId:
type: string
nullable: true
- description: >-
- Observation name to match, or `null` when `langfuseObject` is
- `trace` or `dataset_item`.
- source:
+ description: The user ID of the author
+ comment:
type: string
- description: Field selected from the legacy object.
- jsonPath:
+ nullable: true
+ description: Comment on the score
+ metadata:
+ description: Metadata associated with the score
+ configId:
type: string
nullable: true
- description: Optional JSONPath selector applied to the selected field.
+ description: >-
+ Reference a score config on a score. When set, config and score name
+ must be equal and value must comply to optionally defined numerical
+ range
+ queueId:
+ type: string
+ nullable: true
+ description: >-
+ The annotation queue referenced by the score. Indicates if score was
+ initially created while processing annotation queue.
+ environment:
+ type: string
+ description: >-
+ The environment from which this score originated. Can be any
+ lowercase alphanumeric string with hyphens and underscores that does
+ not start with 'langfuse'.
required:
- - mappingType
- - variable
- - langfuseObject
- - objectName
+ - id
+ - traceId
+ - name
- source
- LegacyEvaluationObject:
- title: LegacyEvaluationObject
- type: string
- enum:
- - trace
- - span
- - generation
- - event
- - agent
- - tool
- - chain
- - retriever
- - evaluator
- - embedding
- - guardrail
- - dataset_item
- description: >-
- **Deprecated:** Legacy Langfuse object kind used by trace and dataset
- evaluation rules.
- EvaluationRuleFilter:
- title: EvaluationRuleFilter
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - datetime
- - $ref: '#/components/schemas/DateTimeEvaluationRuleFilter'
- required:
- - type
- title: DateTimeEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - string
- - $ref: '#/components/schemas/StringEvaluationRuleFilter'
- required:
- - type
- title: StringEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - number
- - $ref: '#/components/schemas/NumberEvaluationRuleFilter'
- required:
- - type
- title: NumberEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - stringOptions
- - $ref: '#/components/schemas/StringOptionsEvaluationRuleFilter'
- required:
- - type
- title: StringOptionsEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - categoryOptions
- - $ref: '#/components/schemas/CategoryOptionsEvaluationRuleFilter'
- required:
- - type
- title: CategoryOptionsEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - arrayOptions
- - $ref: '#/components/schemas/ArrayOptionsEvaluationRuleFilter'
- required:
- - type
- title: ArrayOptionsEvaluationRuleFilter
+ - timestamp
+ - createdAt
+ - updatedAt
+ - authorUserId
+ - comment
+ - metadata
+ - configId
+ - queueId
+ - environment
+ NumericScoreV1:
+ title: NumericScoreV1
+ type: object
+ properties:
+ value:
+ type: number
+ format: double
+ description: The numeric value of the score
+ required:
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV1'
+ BooleanScoreV1:
+ title: BooleanScoreV1
+ type: object
+ properties:
+ value:
+ type: number
+ format: double
+ description: >-
+ The numeric value of the score. Equals 1 for "True" and 0 for "False"
+ stringValue:
+ type: string
+ description: >-
+ The string representation of the score value. Is inferred from the
+ numeric value and equals "True" or "False"
+ required:
+ - value
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV1'
+ CategoricalScoreV1:
+ title: CategoricalScoreV1
+ type: object
+ properties:
+ value:
+ type: number
+ format: double
+ description: >-
+ Represents the numeric category mapping of the stringValue. If no
+ config is linked, defaults to 0.
+ stringValue:
+ type: string
+ description: >-
+ The string representation of the score value. If no config is
+ linked, can be any string. Otherwise, must map to a config category
+ required:
+ - value
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV1'
+ TextScoreV1:
+ title: TextScoreV1
+ type: object
+ properties:
+ stringValue:
+ type: string
+ description: The text content of the score (1-500 characters)
+ required:
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV1'
+ ScoreV1:
+ title: ScoreV1
+ oneOf:
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - stringObject
- - $ref: '#/components/schemas/StringObjectEvaluationRuleFilter'
+ - NUMERIC
+ - $ref: '#/components/schemas/NumericScoreV1'
required:
- - type
- title: StringObjectEvaluationRuleFilter
+ - dataType
+ title: NumericScoreV1
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - numberObject
- - $ref: '#/components/schemas/NumberObjectEvaluationRuleFilter'
+ - CATEGORICAL
+ - $ref: '#/components/schemas/CategoricalScoreV1'
required:
- - type
- title: NumberObjectEvaluationRuleFilter
+ - dataType
+ title: CategoricalScoreV1
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - boolean
- - $ref: '#/components/schemas/BooleanEvaluationRuleFilter'
+ - BOOLEAN
+ - $ref: '#/components/schemas/BooleanScoreV1'
required:
- - type
- title: BooleanEvaluationRuleFilter
+ - dataType
+ title: BooleanScoreV1
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - 'null'
- - $ref: '#/components/schemas/NullEvaluationRuleFilter'
+ - TEXT
+ - $ref: '#/components/schemas/TextScoreV1'
required:
- - type
- title: NullEvaluationRuleFilter
- description: >-
- One filter condition used to decide whether a live-ingested observation
- should be evaluated.
-
-
- All filters must be satisfied for the rule to run. Experiment scope is
- expressed with filters rather than a separate target field:
-
- - `isExperimentItemRootSpan = true` limits execution to experiment item
- roots.
-
- - `datasetId` limits execution to experiments for the selected datasets.
- Use dataset IDs from `GET /api/public/v2/datasets`.
-
-
- Pick the filter `type` first, as it determines the required fields and
- value shape. Use `key` only for object filters such as `metadata`.
-
-
- Operator quick reference by filter `type`:
-
- - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends
- with`
-
- - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`
-
- - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`
-
- - `stringOptions`: `any of`, `none of`
-
- - `arrayOptions`: `any of`, `none of`, `all of`
-
- - `stringObject`: same operators as `string`
-
- - `boolean`: `"="`, `"<>"`
-
- - `null`: `is null`, `is not null`
-
-
- Supported columns. Each column accepts exactly one filter `type`;
- sending a column with a different `type` is rejected with `400`.
-
- - `type`: `stringOptions`, values `SPAN`, `EVENT`, `GENERATION`,
- `AGENT`, `TOOL`, `CHAIN`, `RETRIEVER`, `EVALUATOR`, `EMBEDDING`,
- `GUARDRAIL`
-
- - `name`: `stringOptions`
-
- - `environment`: `stringOptions`
-
- - `level`: `stringOptions`, values `DEBUG`, `DEFAULT`, `WARNING`,
- `ERROR`
-
- - `version`: `string`
-
- - `traceName`: `stringOptions`
-
- - `userId`: `string`
-
- - `sessionId`: `string`
-
- - `tags`: `arrayOptions`
-
- - `metadata`: `stringObject`, requires `key`
-
- - `isRootObservation`: `boolean`; true when the observation has no
- parent or is explicitly marked as an application root
-
- - `parentObservationId`: `null`
-
- - `experimentId`: `stringOptions`
-
- - `isExperimentItemRootSpan`: `boolean`
-
- - `calledToolNames`: `arrayOptions`
-
- - `toolCalls`: `number`, the number of tool calls on the observation
-
- - `datasetId`: `stringOptions`. Use dataset `id` values from `GET
- /api/public/v2/datasets`, not dataset names.
- EvaluationRuleReadFilterBase:
- title: EvaluationRuleReadFilterBase
+ - dataType
+ title: TextScoreV1
+ BaseScore:
+ title: BaseScore
type: object
- description: Stored filter returned verbatim by evaluation-rule read endpoints.
properties:
- type:
- type: string
- description: >-
- Stored filter type. This is not broken down into separate public
- response types.
- column:
+ id:
type: string
- operator:
+ traceId:
type: string
- value:
nullable: true
- description: Stored filter value. Its shape depends on the filter type.
- required:
- - type
- - column
- - operator
- EvaluationRuleReadFilterWithKey:
- title: EvaluationRuleReadFilterWithKey
- type: object
- properties:
- key:
+ description: The trace ID associated with the score
+ sessionId:
type: string
- description: Stored object or legacy filter key.
- required:
- - key
- allOf:
- - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'
- EvaluationRuleReadFilter:
- title: EvaluationRuleReadFilter
- oneOf:
- - $ref: '#/components/schemas/EvaluationRuleReadFilterWithKey'
- - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'
- description: >-
- Filter returned verbatim by evaluation-rule read endpoints.
-
-
- Filters with a stored `key` use the keyed shape. All other filters use
- the base shape. Response filters are intentionally not broken down by
- their internal `type` value.
- PublicApiErrorCode:
- title: PublicApiErrorCode
- type: string
- enum:
- - authentication_failed
- - access_denied
- - invalid_request
- - invalid_query
- - invalid_body
- - resource_not_found
- - conflict
- - rate_limited
- - method_not_allowed
- - internal_error
- description: >-
- Stable machine-readable error code returned by the evaluators and
- evaluation-rules API.
- PublicApiValidationIssue:
- title: PublicApiValidationIssue
- type: object
- description: One request validation issue.
- properties:
- code:
+ nullable: true
+ description: The session ID associated with the score
+ observationId:
type: string
- description: Validator issue code.
- message:
+ nullable: true
+ description: The observation ID associated with the score
+ datasetRunId:
type: string
- description: Human-readable explanation.
- path:
- type: array
- items: {}
- description: Path to the invalid request field.
- required:
- - code
- - message
- - path
- PublicApiErrorDetails:
- title: PublicApiErrorDetails
- type: object
- description: Optional structured context for validation and rate-limit errors.
- properties:
- issues:
- type: array
- items:
- $ref: '#/components/schemas/PublicApiValidationIssue'
nullable: true
- description: Validation issues for an invalid request body or query.
- retryAfterSeconds:
- type: integer
+ description: The dataset run ID associated with the score
+ name:
+ type: string
+ source:
+ $ref: '#/components/schemas/ScoreSource'
+ timestamp:
+ type: string
+ format: date-time
+ createdAt:
+ type: string
+ format: date-time
+ updatedAt:
+ type: string
+ format: date-time
+ authorUserId:
+ type: string
nullable: true
- description: Number of seconds to wait before retrying a rate-limited request.
- limit:
- type: integer
+ description: The user ID of the author
+ comment:
+ type: string
nullable: true
- description: Rate-limit request allowance.
- remaining:
- type: integer
+ description: Comment on the score
+ metadata:
+ description: Metadata associated with the score
+ configId:
+ type: string
nullable: true
- description: Remaining requests in the current rate-limit window.
- resetAt:
+ description: >-
+ Reference a score config on a score. When set, config and score name
+ must be equal and value must comply to optionally defined numerical
+ range
+ queueId:
type: string
- format: date-time
nullable: true
- description: Time when the current rate-limit window resets.
- PublicApiError:
- title: PublicApiError
+ description: >-
+ The annotation queue referenced by the score. Indicates if score was
+ initially created while processing annotation queue.
+ environment:
+ type: string
+ description: >-
+ The environment from which this score originated. Can be any
+ lowercase alphanumeric string with hyphens and underscores that does
+ not start with 'langfuse'.
+ required:
+ - id
+ - name
+ - source
+ - timestamp
+ - createdAt
+ - updatedAt
+ - authorUserId
+ - comment
+ - metadata
+ - configId
+ - queueId
+ - environment
+ NumericScore:
+ title: NumericScore
type: object
- description: >-
- Standard error response for the stable evaluators and evaluation-rules
- API.
-
-
- Use the HTTP status for the broad failure class and `code` for
- programmatic handling. `details` is included when field-level validation
- or retry information is available.
properties:
- message:
- type: string
- description: Human-readable description of the failure.
- code:
- $ref: '#/components/schemas/PublicApiErrorCode'
- description: Stable machine-readable error code.
- details:
- $ref: '#/components/schemas/PublicApiErrorDetails'
- nullable: true
- description: Optional structured validation or rate-limit context.
+ value:
+ type: number
+ format: double
+ description: The numeric value of the score
required:
- - message
- - code
- EvaluationRuleEvaluatorAssignmentInput:
- title: EvaluationRuleEvaluatorAssignmentInput
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScore'
+ BooleanScore:
+ title: BooleanScore
type: object
properties:
- evaluatorId:
- type: string
+ value:
+ type: number
+ format: double
description: >-
- Stable evaluator identifier. The rule automatically uses that
- evaluator's latest version.
- variableMapping:
- type: array
- items:
- $ref: '#/components/schemas/PromptVariableMappingInput'
- nullable: true
+ The numeric value of the score. Equals 1 for "True" and 0 for "False"
+ stringValue:
+ type: string
description: >-
- Rule-specific prompt-variable mapping.
-
-
- Set to `null` or omit to inherit the evaluator's latest default
- mapping. Code evaluators use the fixed runtime mapping and should
- use `null`.
+ The string representation of the score value. Is inferred from the
+ numeric value and equals "True" or "False"
required:
- - evaluatorId
- EvaluatorAssignment:
- title: EvaluatorAssignment
+ - value
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScore'
+ CategoricalScore:
+ title: CategoricalScore
type: object
properties:
- evaluatorId:
+ value:
+ type: number
+ format: double
+ description: >-
+ Represents the numeric category mapping of the stringValue. If no
+ config is linked, defaults to 0.
+ stringValue:
type: string
- description: Stable identifier of the evaluator associated with this rule.
- variableMapping:
- type: array
- items:
- $ref: '#/components/schemas/PromptVariableMapping'
- nullable: true
description: >-
- Stored rule-specific override, or `null` when the evaluator's latest
- default mapping is inherited. Legacy mappings are returned as an
- explicit `legacy` variant.
+ The string representation of the score value. If no config is
+ linked, can be any string. Otherwise, must map to a config category
required:
- - evaluatorId
- - variableMapping
- EvaluationRule:
- title: EvaluationRule
+ - value
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScore'
+ CorrectionScore:
+ title: CorrectionScore
type: object
- description: >-
- Live evaluation rule for incoming observations.
-
-
- A rule determines which evaluators should be used, which observations
- should trigger scoring, how often scoring should run, and which
- observation fields should populate prompt variables.
properties:
- id:
+ value:
+ type: number
+ format: double
+ description: The numeric value of the score. Always 0 for correction scores.
+ stringValue:
type: string
- description: Stable evaluation-rule identifier.
- name:
+ description: The string representation of the correction content
+ required:
+ - value
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScore'
+ TextScore:
+ title: TextScore
+ type: object
+ properties:
+ stringValue:
type: string
- description: >-
- Human-readable rule name. This is independent from evaluator names
- and does not need to be unique.
- createdBy:
- $ref: '#/components/schemas/Creator'
+ description: The text content of the score (1-500 characters)
+ required:
+ - stringValue
+ allOf:
+ - $ref: '#/components/schemas/BaseScore'
+ Score:
+ title: Score
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - NUMERIC
+ - $ref: '#/components/schemas/NumericScore'
+ required:
+ - dataType
+ title: NumericScore
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CATEGORICAL
+ - $ref: '#/components/schemas/CategoricalScore'
+ required:
+ - dataType
+ title: CategoricalScore
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - BOOLEAN
+ - $ref: '#/components/schemas/BooleanScore'
+ required:
+ - dataType
+ title: BooleanScore
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CORRECTION
+ - $ref: '#/components/schemas/CorrectionScore'
+ required:
+ - dataType
+ title: CorrectionScore
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - TEXT
+ - $ref: '#/components/schemas/TextScore'
+ required:
+ - dataType
+ title: TextScore
+ properties:
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
nullable: true
- description: User who created this rule, or `null` when no user can be resolved.
- enabled:
- type: boolean
- description: Whether live execution is enabled for this rule.
- sampling:
- type: number
+ CreateScoreValue:
+ title: CreateScoreValue
+ oneOf:
+ - type: number
format: double
- description: |-
- Fraction of matching observations that should be evaluated.
-
- Must be between `0` and `1`.
- - `1` evaluates every matching observation.
- - `0.25` evaluates approximately 25% of matching observations.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleReadFilter'
- description: >-
- List of stored filter conditions returned verbatim. Filters with a
- `key` use the keyed response shape; all others use the base shape.
- These response shapes are not broken down by internal filter type.
- An empty list matches every incoming object.
- evaluatorAssignments:
- type: array
- items:
- $ref: '#/components/schemas/EvaluatorAssignment'
- description: Evaluators attached to this rule in deterministic assignment order.
+ title: DoubleNumber
+ - type: string
+ title: String
+ description: >-
+ The value of the score. Must be passed as string for categorical and
+ text scores, and numeric for boolean and numeric scores
+ Comment:
+ title: Comment
+ type: object
+ properties:
+ id:
+ type: string
+ projectId:
+ type: string
createdAt:
type: string
format: date-time
- description: Timestamp when the evaluation rule was created.
updatedAt:
type: string
format: date-time
- description: Timestamp when the evaluation rule was last updated.
+ objectType:
+ $ref: '#/components/schemas/CommentObjectType'
+ objectId:
+ type: string
+ content:
+ type: string
+ authorUserId:
+ type: string
+ nullable: true
+ description: The user ID of the comment author
required:
- id
- - name
- - createdBy
- - enabled
- - sampling
- - filter
- - evaluatorAssignments
+ - projectId
- createdAt
- updatedAt
- CreateEvaluationRuleRequest:
- title: CreateEvaluationRuleRequest
+ - objectType
+ - objectId
+ - content
+ Dataset:
+ title: Dataset
type: object
properties:
- name:
+ id:
type: string
- description: >-
- Human-readable rule name. Names are not identifiers and do not need
- to be unique.
- enabled:
- type: boolean
- description: >-
- Whether live execution should start immediately. Enabled rules
- require at least one evaluator assignment.
- sampling:
- type: number
- format: double
- nullable: true
- description: >-
- Fraction of matching observations to evaluate. Omit this field to
- use the default of `1`, which evaluates every match. `null` is not
- accepted.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleFilter'
- nullable: true
- description: >-
- Conditions used to select observations. Defaults to an empty list,
- which matches every incoming observation.
- evaluatorAssignments:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'
- description: >-
- Evaluators to attach to this rule. Disabled rules may use an empty
- list as a draft.
- required:
- - name
- - enabled
- - evaluatorAssignments
- UpdateEvaluationRuleRequest:
- title: UpdateEvaluationRuleRequest
- type: object
- description: At least one field must be provided.
- properties:
name:
type: string
+ description:
+ type: string
nullable: true
- description: New human-readable rule name.
- enabled:
- type: boolean
- nullable: true
- description: New desired live-execution state.
- sampling:
- type: number
- format: double
- nullable: true
- description: >-
- New fraction of matching observations to evaluate. Omit to keep the
- current value.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleFilter'
+ description: Description of the dataset
+ metadata:
+ description: Metadata associated with the dataset
+ inputSchema:
nullable: true
- description: >-
- Complete replacement filter list. An empty list matches every
- incoming observation.
- evaluatorAssignments:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'
+ description: JSON Schema for validating dataset item inputs
+ expectedOutputSchema:
nullable: true
- description: >-
- Complete replacement assignment list. An empty list disables the
- rule.
- EvaluationRulesPage:
- title: EvaluationRulesPage
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRule'
- description: Evaluation rules for this page.
- meta:
- $ref: '#/components/schemas/CursorMeta'
- description: Cursor pagination metadata.
- required:
- - data
- - meta
- DeletedEvaluationRule:
- title: DeletedEvaluationRule
- type: object
- description: Confirmation returned after successful evaluation-rule deletion.
- properties:
- id:
+ description: JSON Schema for validating dataset item expected outputs
+ projectId:
type: string
- description: Identifier of the deleted evaluation rule.
- required:
- - id
- Creator:
- title: Creator
- type: object
- description: User who created the resource, or `null` when no user can be resolved.
- properties:
- id:
+ createdAt:
type: string
- description: User identifier.
- name:
+ format: date-time
+ updatedAt:
type: string
- nullable: true
- description: User name, or `null` when unavailable.
+ format: date-time
required:
- id
- name
- EvaluatorModelConfig:
- title: EvaluatorModelConfig
- type: object
- description: >-
- Explicit model configuration for an evaluator.
-
-
- If the complete `modelConfig` is `null`, Langfuse uses the project's
- default evaluation model.
-
- If provided, the model must be available to the project when the
- evaluator or evaluation rule is enabled.
- properties:
- provider:
- type: string
- description: >-
- Provider identifier, for example `openai` or `anthropic`.
-
-
- To discover valid values for the current project, call `GET
- /api/public/llm-connections` and use one of the returned `provider`
- values.
- model:
- type: string
- description: >-
- Model identifier exposed by the provider, for example `gpt-4.1-mini`.
- required:
- - provider
- - model
- EvaluatorChatMessageRole:
- title: EvaluatorChatMessageRole
- type: string
- enum:
- - system
- - user
- - assistant
- description: Role of an evaluator prompt message.
- EvaluatorChatMessage:
- title: EvaluatorChatMessage
- type: object
- description: One chat message in an evaluator prompt.
- properties:
- role:
- $ref: '#/components/schemas/EvaluatorChatMessageRole'
- content:
- type: string
- description: Message content. Evaluator variables use `{{variable}}` syntax.
- required:
- - role
- - content
- EvaluatorChatPrompt:
- title: EvaluatorChatPrompt
- type: array
- items:
- $ref: '#/components/schemas/EvaluatorChatMessage'
- description: >-
- An ordered list of chat messages. A system message is only allowed as
- the first message.
- EvaluatorChatPromptInput:
- title: EvaluatorChatPromptInput
- oneOf:
- - type: string
- title: String
- - $ref: '#/components/schemas/EvaluatorChatPrompt'
- description: A user prompt string shortcut, or an ordered list of chat messages.
- EvaluatorVersionBase:
- title: EvaluatorVersionBase
+ - description
+ - metadata
+ - inputSchema
+ - expectedOutputSchema
+ - projectId
+ - createdAt
+ - updatedAt
+ DatasetItem:
+ title: DatasetItem
type: object
properties:
id:
type: string
- description: Stable identifier of this evaluator version.
- version:
- type: integer
- description: Monotonically increasing evaluator version number.
+ status:
+ $ref: '#/components/schemas/DatasetStatus'
+ input:
+ description: Input data for the dataset item
+ expectedOutput:
+ description: Expected output for the dataset item
+ metadata:
+ description: Metadata associated with the dataset item
+ sourceTraceId:
+ type: string
+ nullable: true
+ description: The trace ID that sourced this dataset item
+ sourceObservationId:
+ type: string
+ nullable: true
+ description: The observation ID that sourced this dataset item
+ datasetId:
+ type: string
+ datasetName:
+ type: string
createdAt:
type: string
format: date-time
- description: Timestamp when this evaluator version was created.
- createdBy:
- $ref: '#/components/schemas/Creator'
- nullable: true
+ updatedAt:
+ type: string
+ format: date-time
+ mediaReferences:
+ type: array
+ items:
+ $ref: '#/components/schemas/DatasetItemMediaReference'
description: >-
- User who created this version, or `null` when no user can be
- resolved.
+ Resolved Langfuse media references found in input, expectedOutput,
+ and metadata.
required:
- id
- - version
+ - status
+ - input
+ - expectedOutput
+ - metadata
+ - sourceTraceId
+ - sourceObservationId
+ - datasetId
+ - datasetName
- createdAt
- - createdBy
- LlmAsJudgeEvaluatorVersion:
- title: LlmAsJudgeEvaluatorVersion
+ - updatedAt
+ - mediaReferences
+ DatasetItemMediaReference:
+ title: DatasetItemMediaReference
type: object
properties:
- type:
+ field:
+ $ref: '#/components/schemas/DatasetItemMediaReferenceField'
+ description: The dataset item field containing the reference
+ referenceString:
type: string
- const: llm_as_judge
- description: Evaluator type.
- prompt:
- $ref: '#/components/schemas/EvaluatorChatPrompt'
- description: Ordered chat messages used during evaluation.
- variables:
- type: array
- items:
- type: string
- description: >-
- Variables extracted from the prompt and available for
- evaluation-rule mappings.
-
-
- Every variable must be mapped exactly once when a rule provides an
- explicit mapping.
- variableMapping:
- type: array
- items:
- $ref: '#/components/schemas/PromptVariableMappingRead'
- nullable: true
- description: >-
- Default variable mapping for this evaluator version, or `null` when
- no default is configured.
- modelConfig:
- $ref: '#/components/schemas/EvaluatorModelConfig'
- nullable: true
description: >-
- Explicit model configuration, or `null` when the project's default
- evaluation model is used.
- outputDefinition:
- $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'
+ The Langfuse media reference string, e.g.
+ `@@@langfuseMedia:type=image/png|id=...|source=bytes@@@`
+ jsonPath:
+ type: string
description: >-
- Structured output schema returned by this evaluator.
-
-
- Responses include `dataType` and omit the internal output-definition
- persistence version.
+ JSONPath of the string holding the reference within the field, e.g.
+ `$['image']`
+ media:
+ $ref: '#/components/schemas/DatasetItemMediaReferenceMedia'
+ description: The resolved media record.
required:
- - type
- - prompt
- - variables
- - variableMapping
- - modelConfig
- - outputDefinition
- allOf:
- - $ref: '#/components/schemas/EvaluatorVersionBase'
- CodeEvaluatorVersion:
- title: CodeEvaluatorVersion
+ - field
+ - referenceString
+ - jsonPath
+ - media
+ DatasetItemMediaReferenceField:
+ title: DatasetItemMediaReferenceField
+ type: string
+ enum:
+ - input
+ - expectedOutput
+ - metadata
+ DatasetItemMediaReferenceMedia:
+ title: DatasetItemMediaReferenceMedia
type: object
properties:
- type:
+ mediaId:
type: string
- const: code
- description: Evaluator type.
- sourceCode:
+ description: The unique langfuse identifier of the media record
+ contentType:
type: string
- description: Source code executed for each matched observation.
- sourceCodeLanguage:
- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
- description: Runtime language used to execute the source code.
+ description: The MIME type of the media record
+ contentLength:
+ type: integer
+ description: The size of the media record in bytes
+ url:
+ type: string
+ description: The signed download URL of the media record
+ urlExpiry:
+ type: string
+ description: The expiry date and time of the download URL
required:
- - type
- - sourceCode
- - sourceCodeLanguage
- allOf:
- - $ref: '#/components/schemas/EvaluatorVersionBase'
- EvaluatorVersion:
- title: EvaluatorVersion
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - llm_as_judge
- - $ref: '#/components/schemas/LlmAsJudgeEvaluatorVersion'
- required:
- - type
- title: LlmAsJudgeEvaluatorVersion
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - code
- - $ref: '#/components/schemas/CodeEvaluatorVersion'
- required:
- - type
- title: CodeEvaluatorVersion
- EvaluationRuleAssignment:
- title: EvaluationRuleAssignment
+ - mediaId
+ - contentType
+ - contentLength
+ - url
+ - urlExpiry
+ DatasetRunItem:
+ title: DatasetRunItem
type: object
- description: Evaluation-rule assignment associated with this evaluator.
properties:
- evaluationRuleId:
+ id:
+ type: string
+ datasetRunId:
+ type: string
+ datasetRunName:
+ type: string
+ datasetItemId:
+ type: string
+ traceId:
+ type: string
+ observationId:
type: string
- description: Stable identifier of the assigned evaluation rule.
- variableMappingOverride:
- type: array
- items:
- $ref: '#/components/schemas/PromptVariableMapping'
nullable: true
- description: >-
- Rule-specific variable mapping override. Omitted when the
- evaluator's latest default mapping is inherited. Legacy mappings use
- the explicit `legacy` variant.
+ description: The observation ID associated with this run item
+ createdAt:
+ type: string
+ format: date-time
+ updatedAt:
+ type: string
+ format: date-time
required:
- - evaluationRuleId
- EvaluatorBase:
- title: EvaluatorBase
+ - id
+ - datasetRunId
+ - datasetRunName
+ - datasetItemId
+ - traceId
+ - observationId
+ - createdAt
+ - updatedAt
+ DatasetRun:
+ title: DatasetRun
type: object
properties:
id:
type: string
- description: Stable identifier of this evaluator across all versions.
+ description: Unique identifier of the dataset run
name:
type: string
- description: >-
- Human-readable evaluator name. Names are not identifiers and do not
- need to be unique.
+ description: Name of the dataset run
description:
type: string
nullable: true
- description: Optional human-readable evaluator description.
- createdBy:
- $ref: '#/components/schemas/Creator'
- nullable: true
- description: >-
- User who created this evaluator, or `null` when no user can be
- resolved.
- status:
- $ref: '#/components/schemas/EvaluatorStatus'
- description: >-
- Effective evaluator status after Langfuse validates its runtime
- configuration.
- pausedAt:
- type: string
- format: date-time
- nullable: true
- description: Timestamp when the evaluator was paused, otherwise `null`.
- pausedReason:
+ description: Description of the run
+ metadata:
+ description: Metadata of the dataset run
+ datasetId:
type: string
- nullable: true
- description: Machine-readable reason when `status=paused`, otherwise `null`.
- pausedMessage:
+ description: Id of the associated dataset
+ datasetName:
type: string
- nullable: true
- description: Human-readable explanation when `status=paused`, otherwise `null`.
- evaluationRuleAssignments:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleAssignment'
- description: >-
- All modern and legacy evaluation-rule assignments in
- newest-assignment-first order. Rule-specific mappings are exposed as
- `variableMappingOverride`; inherited defaults are omitted.
+ description: Name of the associated dataset
createdAt:
type: string
format: date-time
- description: Timestamp when the evaluator was created.
+ description: The date and time when the dataset run was created
updatedAt:
type: string
format: date-time
- description: Timestamp when the evaluator was last updated.
- versionId:
- type: string
- description: Stable identifier of the latest evaluator version.
- version:
- type: integer
- description: Monotonically increasing latest evaluator version number.
- versionCreatedAt:
- type: string
- format: date-time
- description: Timestamp when the latest evaluator version was created.
- versionCreatedBy:
- $ref: '#/components/schemas/Creator'
- nullable: true
- description: >-
- User who created the latest evaluator version, or `null` when no
- user can be resolved.
+ description: The date and time when the dataset run was last updated
required:
- id
- name
- description
- - createdBy
- - status
- - pausedAt
- - pausedReason
- - pausedMessage
- - evaluationRuleAssignments
+ - metadata
+ - datasetId
+ - datasetName
- createdAt
- updatedAt
- - versionId
- - version
- - versionCreatedAt
- - versionCreatedBy
- LlmAsJudgeEvaluator:
- title: LlmAsJudgeEvaluator
+ DatasetRunWithItems:
+ title: DatasetRunWithItems
type: object
properties:
- type:
- type: string
- const: llm_as_judge
- description: Evaluator type.
- prompt:
- $ref: '#/components/schemas/EvaluatorChatPrompt'
- description: Ordered chat messages used by the latest evaluator version.
- variables:
- type: array
- items:
- type: string
- description: >-
- Variables extracted from the latest prompt and available for
- evaluation-rule mappings.
- variableMapping:
+ datasetRunItems:
type: array
items:
- $ref: '#/components/schemas/PromptVariableMappingRead'
- nullable: true
- description: >-
- Default variable mapping for the latest version, or `null` when no
- default is configured.
- modelConfig:
- $ref: '#/components/schemas/EvaluatorModelConfig'
+ $ref: '#/components/schemas/DatasetRunItem'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
nullable: true
- description: >-
- Explicit model configuration for the latest version, or `null` when
- the project's default evaluation model is used.
- outputDefinition:
- $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'
- description: Structured output schema returned by the latest evaluator version.
required:
- - type
- - prompt
- - variables
- - variableMapping
- - modelConfig
- - outputDefinition
+ - datasetRunItems
allOf:
- - $ref: '#/components/schemas/EvaluatorBase'
- CodeEvaluator:
- title: CodeEvaluator
+ - $ref: '#/components/schemas/DatasetRun'
+ Model:
+ title: Model
type: object
+ description: >-
+ Model definition used for transforming usage into USD cost and/or
+ tokenization.
+
+
+ Models can have either simple flat pricing or tiered pricing:
+
+ - Flat pricing: Single price per usage type (legacy, but still
+ supported)
+
+ - Tiered pricing: Multiple pricing tiers with conditional matching based
+ on usage patterns or observation attributes
+
+
+ The pricing tiers approach is recommended for models with usage-based
+ pricing variations.
+
+ When using tiered pricing, the flat price fields (inputPrice,
+ outputPrice, prices) are populated
+
+ from the default tier for backward compatibility.
properties:
- type:
+ id:
type: string
- const: code
- description: Evaluator type.
- sourceCode:
+ modelName:
type: string
- description: Source code executed by the latest evaluator version.
- sourceCodeLanguage:
- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
- description: Runtime language used to execute the latest source code.
- required:
- - type
- - sourceCode
- - sourceCodeLanguage
- allOf:
- - $ref: '#/components/schemas/EvaluatorBase'
- Evaluator:
- title: Evaluator
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - llm_as_judge
- - $ref: '#/components/schemas/LlmAsJudgeEvaluator'
- required:
- - type
- title: LlmAsJudgeEvaluator
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - code
- - $ref: '#/components/schemas/CodeEvaluator'
- required:
- - type
- title: CodeEvaluator
- description: >-
- One evaluator that can be used for scoring.
+ description: >-
+ Name of the model definition. If multiple with the same name exist,
+ they are applied in the following order: (1) custom over built-in,
+ (2) newest according to startTime where
+ model.startTime<observation.startTime
+ matchPattern:
+ type: string
+ description: >-
+ Regex pattern which matches this model definition to
+ generation.model. Useful in case of fine-tuned models. If you want
+ to exact match, use `(?i)^modelname$`
+ startDate:
+ type: string
+ format: date-time
+ nullable: true
+ description: Apply only to generations which are newer than this ISO date.
+ unit:
+ $ref: '#/components/schemas/ModelUsageUnit'
+ nullable: true
+ description: Unit used by this model.
+ inputPrice:
+ type: number
+ format: double
+ nullable: true
+ description: Deprecated. See 'prices' instead. Price (USD) per input unit
+ outputPrice:
+ type: number
+ format: double
+ nullable: true
+ description: Deprecated. See 'prices' instead. Price (USD) per output unit
+ totalPrice:
+ type: number
+ format: double
+ nullable: true
+ description: >-
+ Deprecated. See 'prices' instead. Price (USD) per total unit. Cannot
+ be set if input or output price is set.
+ tokenizerId:
+ type: string
+ nullable: true
+ description: >-
+ Optional. Tokenizer to be applied to observations which match to
+ this model. See docs for more details.
+ tokenizerConfig:
+ description: >-
+ Optional. Configuration for the selected tokenizer. Needs to be
+ JSON. See docs for more details.
+ isLangfuseManaged:
+ type: boolean
+ createdAt:
+ type: string
+ format: date-time
+ description: Timestamp when the model was created
+ prices:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/ModelPrice'
+ description: >-
+ Deprecated. Use 'pricingTiers' instead for models with usage-based
+ pricing variations.
- An evaluator describes **how** to score data. Associated evaluation
- rules describe **which** live objects should be evaluated.
- EvaluatorStatus:
- title: EvaluatorStatus
- type: string
- enum:
- - active
- - paused
- description: >-
- Effective evaluator runtime status.
+ This field shows prices by usage type from the default pricing tier.
+ Maintained for backward compatibility.
+
+ If the model uses tiered pricing, this field will be populated from
+ the default tier's prices.
+ pricingTiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/PricingTier'
+ description: >-
+ Array of pricing tiers with conditional pricing based on usage
+ thresholds or observation attributes.
+
+
+ Pricing tiers enable accurate cost tracking for models that charge
+ different rates based on usage patterns
+
+ or request attributes (e.g., high-volume usage, large context
+ windows, cached tokens, or service tiers).
+
+
+ Each model must have exactly one default tier (isDefault=true,
+ priority=0) that serves as a fallback.
+ Additional conditional tiers can be defined with specific matching
+ criteria.
- - `active`: the evaluator can run.
- - `paused`: Langfuse paused execution until the underlying issue is
- resolved.
- CreateLlmAsJudgeEvaluatorRequest:
- title: CreateLlmAsJudgeEvaluatorRequest
+ If this array is empty, the model uses legacy flat pricing from the
+ inputPrice/outputPrice/totalPrice fields.
+ required:
+ - id
+ - modelName
+ - matchPattern
+ - startDate
+ - inputPrice
+ - outputPrice
+ - totalPrice
+ - tokenizerId
+ - tokenizerConfig
+ - isLangfuseManaged
+ - createdAt
+ - prices
+ - pricingTiers
+ ModelPrice:
+ title: ModelPrice
type: object
properties:
- name:
- type: string
- description: >-
- Human-readable evaluator name. Names are not identifiers and do not
- need to be unique.
- description:
- type: string
- nullable: true
- description: Optional human-readable evaluator description.
- type:
- type: string
- const: llm_as_judge
- description: Evaluator type.
- prompt:
- $ref: '#/components/schemas/EvaluatorChatPromptInput'
- description: User prompt string shortcut or an ordered list of chat messages.
- modelConfig:
- $ref: '#/components/schemas/EvaluatorModelConfig'
- nullable: true
- description: >-
- Explicit model configuration. Set to `null` or omit to use the
- project's default evaluation model.
- variableMapping:
- type: array
- items:
- $ref: '#/components/schemas/PromptVariableMappingInput'
- nullable: true
- description: >-
- Default prompt-variable mapping, or `null` when no default is
- configured.
- outputDefinition:
- $ref: '#/components/schemas/EvaluatorOutputDefinition'
- description: >-
- Structured output schema returned by this evaluator. Do not include
- an internal persistence version.
+ price:
+ type: number
+ format: double
required:
- - name
- - type
- - prompt
- - outputDefinition
- CreateCodeEvaluatorRequest:
- title: CreateCodeEvaluatorRequest
+ - price
+ PricingTierCondition:
+ title: PricingTierCondition
+ oneOf:
+ - $ref: '#/components/schemas/PricingTierUsageCondition'
+ - $ref: '#/components/schemas/PricingTierAttributeCondition'
+ description: >-
+ Condition for matching a pricing tier against usage details or
+ observation attributes.
+
+
+ Usage-detail conditions treat usageDetailPattern as a regex, sum all
+ matching usage values, and compare the sum to the numeric value.
+ Model-parameter and metadata conditions match an exact top-level key
+ against one or more string values.
+ PricingTierUsageCondition:
+ title: PricingTierUsageCondition
type: object
+ description: Condition that sums usage details whose keys match a regex.
properties:
- name:
- type: string
- description: >-
- Human-readable evaluator name. Names are not identifiers and do not
- need to be unique.
- description:
+ usageDetailPattern:
type: string
- nullable: true
- description: Optional human-readable evaluator description.
- type:
+ description: Regex pattern matched against usage detail keys.
+ operator:
+ $ref: '#/components/schemas/PricingTierOperator'
+ value:
+ type: number
+ format: double
+ description: Numeric threshold for the summed matching usage values.
+ caseSensitive:
+ type: boolean
+ description: Whether the usage-detail regex is case-sensitive.
+ required:
+ - usageDetailPattern
+ - operator
+ - value
+ - caseSensitive
+ PricingTierAttributeCondition:
+ title: PricingTierAttributeCondition
+ type: object
+ description: >-
+ Condition that matches any configured value for a top-level observation
+ attribute.
+ properties:
+ source:
+ $ref: '#/components/schemas/PricingTierAttributeSource'
+ description: Observation attribute object evaluated by this condition.
+ key:
type: string
- const: code
- description: Evaluator type.
- sourceCode:
+ description: Exact top-level attribute key.
+ operator:
type: string
- description: Source code executed for each matched observation.
- sourceCodeLanguage:
- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
- description: Runtime language used to execute the source code.
+ const: in
+ description: Membership operator.
+ values:
+ type: array
+ items:
+ type: string
+ description: Accepted string attribute values. At least one value is required.
required:
- - name
- - type
- - sourceCode
- - sourceCodeLanguage
- CreateEvaluatorRequest:
- title: CreateEvaluatorRequest
+ - source
+ - key
+ - operator
+ - values
+ PricingTierAttributeSource:
+ title: PricingTierAttributeSource
+ type: string
+ enum:
+ - model_parameters
+ - metadata
+ PricingTierConditionInput:
+ title: PricingTierConditionInput
oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - llm_as_judge
- - $ref: '#/components/schemas/CreateLlmAsJudgeEvaluatorRequest'
- required:
- - type
- title: CreateLlmAsJudgeEvaluatorRequest
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - code
- - $ref: '#/components/schemas/CreateCodeEvaluatorRequest'
- required:
- - type
- title: CreateCodeEvaluatorRequest
- description: >-
- Initial evaluator definition with metadata and definition fields at the
- same level. The returned evaluator starts at version `1`.
- UpdateEvaluatorMetadataRequest:
- title: UpdateEvaluatorMetadataRequest
+ - $ref: '#/components/schemas/PricingTierUsageConditionInput'
+ - $ref: '#/components/schemas/PricingTierAttributeCondition'
+ description: Condition input for matching a pricing tier.
+ PricingTierUsageConditionInput:
+ title: PricingTierUsageConditionInput
type: object
- description: Metadata-only update. At least one field must be provided.
+ description: Input condition that sums usage details whose keys match a regex.
properties:
- name:
- type: string
- nullable: true
- description: New human-readable evaluator name.
- description:
+ usageDetailPattern:
type: string
+ description: Regex pattern matched against usage detail keys.
+ operator:
+ $ref: '#/components/schemas/PricingTierOperator'
+ value:
+ type: number
+ format: double
+ description: Numeric threshold for the summed matching usage values.
+ caseSensitive:
+ type: boolean
nullable: true
- description: New description. Set to `null` to clear it.
- UpdateLlmAsJudgeEvaluatorRequest:
- title: UpdateLlmAsJudgeEvaluatorRequest
+ description: Whether the usage-detail regex is case-sensitive. Defaults to false.
+ required:
+ - usageDetailPattern
+ - operator
+ - value
+ PricingTier:
+ title: PricingTier
type: object
+ description: >-
+ Pricing tier definition with conditional pricing based on usage
+ thresholds or observation attributes.
+
+
+ Pricing tiers enable accurate cost tracking for LLM providers that
+ charge different rates based on usage patterns or request attributes.
+
+ For example, some providers charge higher rates when context size
+ exceeds certain thresholds or when an accelerated service tier is
+ selected.
+
+
+ How tier matching works:
+
+ 1. Tiers are evaluated in ascending priority order (priority 1 before
+ priority 2, etc.)
+
+ 2. The first tier where ALL conditions match is selected
+
+ 3. If no conditional tiers match, the default tier is used as a fallback
+
+ 4. The default tier has priority 0 and no conditions
+
+
+ Why priorities matter:
+
+ - Lower priority numbers are evaluated first, allowing you to define
+ specific cases before general ones
+
+ - Example: Priority 1 for "high usage" (>200K tokens), Priority 2 for
+ "medium usage" (>100K tokens), Priority 0 for default
+
+ - Without proper ordering, a less specific condition might match before
+ a more specific one
+
+
+ Every model must have exactly one default tier to ensure cost
+ calculation always succeeds.
properties:
- name:
- type: string
- nullable: true
- description: New human-readable evaluator name.
- description:
+ id:
type: string
- nullable: true
- description: New description. Set to `null` to clear it.
- type:
+ description: Unique identifier for the pricing tier
+ name:
type: string
- const: llm_as_judge
- description: Evaluator type. The type of an existing evaluator cannot change.
- prompt:
- $ref: '#/components/schemas/EvaluatorChatPromptInput'
description: >-
- Complete replacement user prompt string or an ordered list of chat
- messages.
- modelConfig:
- $ref: '#/components/schemas/EvaluatorModelConfig'
- nullable: true
+ Name of the pricing tier for display and identification purposes.
+
+
+ Examples: "Standard", "High Volume Tier", "Large Context", "Extended
+ Context Tier"
+ isDefault:
+ type: boolean
description: >-
- Explicit model configuration. Set to `null` or omit to use the
- project's default evaluation model.
- variableMapping:
+ Whether this is the default tier. Every model must have exactly one
+ default tier with priority 0 and no conditions.
+
+
+ The default tier serves as a fallback when no conditional tiers
+ match, ensuring cost calculation always succeeds.
+
+ It typically represents the base pricing for standard usage
+ patterns.
+ priority:
+ type: integer
+ description: >-
+ Priority for tier matching evaluation. Lower numbers = higher
+ priority (evaluated first).
+
+
+ The default tier must always have priority 0. Conditional tiers
+ should have priority 1, 2, 3, etc.
+
+
+ Example ordering:
+
+ - Priority 0: Default tier (no conditions, always matches as
+ fallback)
+
+ - Priority 1: High usage tier (e.g., >200K tokens)
+
+ - Priority 2: Medium usage tier (e.g., >100K tokens)
+
+
+ This ensures more specific conditions are checked before general
+ ones.
+ conditions:
type: array
items:
- $ref: '#/components/schemas/PromptVariableMappingInput'
- nullable: true
+ $ref: '#/components/schemas/PricingTierCondition'
description: >-
- Complete replacement default variable mapping, or `null` when no
- default is configured.
- outputDefinition:
- $ref: '#/components/schemas/EvaluatorOutputDefinition'
+ Array of conditions that must ALL be met for this tier to match (AND
+ logic).
+
+
+ The default tier must have an empty conditions array. Conditional
+ tiers should have one or more conditions
+
+ that define when this tier's pricing applies.
+
+
+ Multiple conditions enable complex matching scenarios (e.g., "high
+ input tokens AND low output tokens").
+ prices:
+ type: object
+ additionalProperties:
+ type: number
+ format: double
description: >-
- Complete replacement output schema. Do not include an internal
- persistence version.
+ Prices (USD) by usage type for this tier.
+
+
+ Common usage types: "input", "output", "total", "request", "image"
+
+ Prices are specified in USD per unit (e.g., per token, per request,
+ per second).
+
+
+ Example: {"input": 0.000003, "output": 0.000015} means $3 per
+ million input tokens and $15 per million output tokens.
required:
- - type
- - prompt
- - outputDefinition
- UpdateCodeEvaluatorRequest:
- title: UpdateCodeEvaluatorRequest
+ - id
+ - name
+ - isDefault
+ - priority
+ - conditions
+ - prices
+ PricingTierInput:
+ title: PricingTierInput
type: object
+ description: >-
+ Input schema for creating a pricing tier. The tier ID will be
+ automatically generated server-side.
+
+
+ When creating a model with pricing tiers:
+
+ - Exactly one tier must have isDefault=true (the fallback tier)
+
+ - The default tier must have priority=0 and conditions=[]
+
+ - All tier names and priorities must be unique within the model
+
+ - Each tier must define at least one price
+
+
+ See PricingTier for detailed information about how tiers work and why
+ they're useful.
properties:
name:
type: string
+ description: >-
+ Name of the pricing tier for display and identification purposes.
+
+
+ Must be unique within the model. Common patterns: "Standard", "High
+ Volume Tier", "Extended Context"
+ isDefault:
+ type: boolean
nullable: true
- description: New human-readable evaluator name.
- description:
- type: string
- nullable: true
- description: New description. Set to `null` to clear it.
- type:
- type: string
- const: code
- description: Evaluator type. The type of an existing evaluator cannot change.
- sourceCode:
- type: string
- description: Complete replacement source code.
- sourceCodeLanguage:
- $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
- description: Runtime language used to execute the source code.
+ description: >-
+ Whether this is the default tier. Exactly one tier per model must be
+ marked as default.
+
+
+ Requirements for default tier:
+
+ - Must have isDefault=true
+
+ - Must have priority=0
+
+ - Must have empty conditions array (conditions=[])
+
+
+ The default tier acts as a fallback when no conditional tiers match.
+ Defaults to false when omitted.
+ priority:
+ type: integer
+ description: >-
+ Priority for tier matching evaluation. Lower numbers = higher
+ priority (evaluated first).
+
+
+ Must be unique within the model. The default tier must have
+ priority=0.
+
+ Conditional tiers should use priority 1, 2, 3, etc. based on their
+ specificity.
+ conditions:
+ type: array
+ items:
+ $ref: '#/components/schemas/PricingTierConditionInput'
+ description: >-
+ Array of conditions that must ALL be met for this tier to match (AND
+ logic).
+
+
+ The default tier must have an empty array (conditions=[]).
+
+ Conditional tiers should define one or more conditions that specify
+ when this tier's pricing applies.
+
+
+ Conditions can compare summed matching usage details to a numeric
+ threshold, or exactly match a top-level model parameter or metadata
+ value.
+ prices:
+ type: object
+ additionalProperties:
+ type: number
+ format: double
+ description: >-
+ Prices (USD) by usage type for this tier. At least one price must be
+ defined.
+
+
+ Common usage types: "input", "output", "total", "request", "image"
+
+ Prices are in USD per unit (e.g., per token).
+
+
+ Example: {"input": 0.000003, "output": 0.000015} represents $3 per
+ million input tokens and $15 per million output tokens.
required:
- - type
- - sourceCode
- - sourceCodeLanguage
- UpdateEvaluatorRequest:
- title: UpdateEvaluatorRequest
+ - name
+ - priority
+ - conditions
+ - prices
+ PricingTierOperator:
+ title: PricingTierOperator
+ type: string
+ enum:
+ - gt
+ - gte
+ - lt
+ - lte
+ - eq
+ - neq
+ description: Comparison operators for pricing tier conditions
+ ModelUsageUnit:
+ title: ModelUsageUnit
+ type: string
+ enum:
+ - CHARACTERS
+ - TOKENS
+ - MILLISECONDS
+ - SECONDS
+ - IMAGES
+ - REQUESTS
+ description: Unit of usage in Langfuse
+ ObservationLevel:
+ title: ObservationLevel
+ type: string
+ enum:
+ - DEBUG
+ - DEFAULT
+ - WARNING
+ - ERROR
+ MapValue:
+ title: MapValue
oneOf:
- - $ref: '#/components/schemas/UpdateEvaluatorMetadataRequest'
- - $ref: '#/components/schemas/UpdateLlmAsJudgeEvaluatorRequest'
- - $ref: '#/components/schemas/UpdateCodeEvaluatorRequest'
- description: >-
- Metadata-only update or complete flattened definition replacement. At
- least one field must be provided.
- EvaluatorsPage:
- title: EvaluatorsPage
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Evaluator'
- description: Evaluators for this page.
- meta:
- $ref: '#/components/schemas/CursorMeta'
- description: Cursor pagination metadata.
- required:
- - data
- - meta
- EvaluatorVersionsPage:
- title: EvaluatorVersionsPage
- type: object
- properties:
- data:
- type: array
+ - type: string
+ nullable: true
+ title: String
+ - type: integer
+ nullable: true
+ title: Integer
+ - type: number
+ format: float
+ nullable: true
+ title: FloatNumber
+ - type: boolean
+ nullable: true
+ title: Boolean
+ - type: array
items:
- $ref: '#/components/schemas/EvaluatorVersion'
- description: Evaluator versions for this page in newest-first order.
- meta:
- $ref: '#/components/schemas/CursorMeta'
- description: Cursor pagination metadata.
- required:
- - data
- - meta
- CursorMeta:
- title: CursorMeta
- type: object
- properties:
- cursor:
- type: string
+ type: string
nullable: true
- description: Opaque cursor for the next page. Omitted when there is no next page.
- DeletedEvaluator:
- title: DeletedEvaluator
+ title: Array
+ CommentObjectType:
+ title: CommentObjectType
+ type: string
+ enum:
+ - TRACE
+ - OBSERVATION
+ - SESSION
+ - PROMPT
+ DatasetStatus:
+ title: DatasetStatus
+ type: string
+ enum:
+ - ACTIVE
+ - ARCHIVED
+ ScoreSource:
+ title: ScoreSource
+ type: string
+ enum:
+ - ANNOTATION
+ - API
+ - EVAL
+ ScoreConfigDataType:
+ title: ScoreConfigDataType
+ type: string
+ enum:
+ - NUMERIC
+ - BOOLEAN
+ - CATEGORICAL
+ - TEXT
+ ScoreDataType:
+ title: ScoreDataType
+ type: string
+ enum:
+ - NUMERIC
+ - BOOLEAN
+ - CATEGORICAL
+ - CORRECTION
+ - TEXT
+ DeleteDatasetItemResponse:
+ title: DeleteDatasetItemResponse
type: object
- description: Confirmation returned after successful evaluator deletion.
properties:
- id:
+ message:
type: string
- description: Identifier of the deleted evaluator.
- required:
- - id
- ExperimentsResponse:
- title: ExperimentsResponse
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Experiment'
- meta:
- $ref: '#/components/schemas/ExperimentsResponseMeta'
+ description: Success message after deletion
required:
- - data
- - meta
- ExperimentsResponseMeta:
- title: ExperimentsResponseMeta
+ - message
+ CreateDatasetItemRequest:
+ title: CreateDatasetItemRequest
type: object
properties:
- cursor:
+ datasetName:
type: string
+ input:
nullable: true
- description: >-
- Versioned base64url cursor for retrieving the next page. Absent when
- there are no more results.
- Experiment:
- title: Experiment
- type: object
- properties:
- id:
- type: string
- name:
- type: string
- description:
- type: string
+ expectedOutput:
nullable: true
- startTime:
- type: string
- format: date-time
- description: |-
- Start of the experiment, i.e. the earliest event within the
- requested time range. Clipped to `fromStartTime` when the
- experiment started before the requested range.
- endTime:
- type: string
- format: date-time
- description: |-
- End of the experiment, i.e. the latest event end within the
- requested time range.
- itemCount:
- type: integer
- description: Number of experiment items within the requested time range.
- datasetId:
+ metadata:
+ nullable: true
+ sourceTraceId:
type: string
nullable: true
- description: Null when the experiment is not associated with a dataset.
- metadata:
- type: object
- additionalProperties: true
+ sourceObservationId:
+ type: string
nullable: true
- description: Included only when `fields=metadata` is requested.
- scores:
- type: array
- items:
- $ref: '#/components/schemas/ScoreV3'
+ id:
+ type: string
nullable: true
description: >-
- Included only when `fields=scores` is requested. Contains scores
- directly attached to the experiment.
+ Dataset items are upserted on their id. Id needs to be unique
+ (project-level), cannot be reused across datasets, and must be at
+ most 255 characters.
+ status:
+ $ref: '#/components/schemas/DatasetStatus'
+ nullable: true
+ description: Defaults to ACTIVE for newly created items
required:
- - id
- - name
- - description
- - startTime
- - endTime
- - itemCount
- - datasetId
- ExperimentItemsResponse:
- title: ExperimentItemsResponse
+ - datasetName
+ PaginatedDatasetItems:
+ title: PaginatedDatasetItems
type: object
properties:
data:
type: array
items:
- $ref: '#/components/schemas/ExperimentItem'
+ $ref: '#/components/schemas/DatasetItem'
meta:
- $ref: '#/components/schemas/ExperimentsResponseMeta'
+ $ref: '#/components/schemas/utilsMetaResponse'
required:
- data
- meta
- ExperimentItem:
- title: ExperimentItem
+ CreateDatasetRunItemRequest:
+ title: CreateDatasetRunItemRequest
type: object
properties:
- id:
- type: string
- traceId:
- type: string
- startTime:
- type: string
- format: date-time
- endTime:
- type: string
- format: date-time
- nullable: true
- level:
- $ref: '#/components/schemas/ObservationLevel'
- environment:
- type: string
- experimentId:
- type: string
- experimentName:
- type: string
- experimentItemId:
- type: string
- experimentDatasetId:
+ runName:
type: string
- nullable: true
- description: Included when `fields=dataset` is requested.
- experimentItemVersion:
+ runDescription:
type: string
- format: date-time
- nullable: true
- description: Included when `fields=dataset` is requested.
- input:
- nullable: true
- description: Included when `fields=io` is requested.
- output:
- nullable: true
- description: Included when `fields=io` is requested.
- expectedOutput:
nullable: true
- description: Included when `fields=io` is requested.
+ description: Description of the run. If run exists, description will be updated.
metadata:
- type: object
- additionalProperties: true
nullable: true
- description: Included when `fields=metadata` is requested.
- experimentItemMetadata:
- type: object
- additionalProperties: true
+ description: Metadata of the dataset run, updates run if run already exists
+ datasetItemId:
+ type: string
+ observationId:
+ type: string
nullable: true
- description: Included when `fields=itemMetadata` is requested.
- experimentMetadata:
- type: object
- additionalProperties: true
+ traceId:
+ type: string
nullable: true
- description: Included when `fields=experimentMetadata` is requested.
- experimentDescription:
+ description: >-
+ traceId should always be provided. For compatibility with older SDK
+ versions it can also be inferred from the provided observationId.
+ datasetVersion:
type: string
+ format: date-time
nullable: true
- description: Included when `fields=experimentMetadata` is requested.
- scores:
+ description: >-
+ ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g.,
+ "2026-01-21T14:35:42Z").
+
+ Specifies the dataset version to use for this experiment run.
+
+ If provided, the experiment will use dataset items as they existed
+ at or before this timestamp.
+
+ If not provided, uses the latest version of dataset items.
+ createdAt:
+ type: string
+ format: date-time
+ nullable: true
+ description: >-
+ Optional timestamp to set the createdAt field of the dataset run
+ item. If not provided or null, defaults to current timestamp.
+ required:
+ - runName
+ - datasetItemId
+ PaginatedDatasetRunItems:
+ title: PaginatedDatasetRunItems
+ type: object
+ properties:
+ data:
type: array
items:
- $ref: '#/components/schemas/ScoreV3'
+ $ref: '#/components/schemas/DatasetRunItem'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
nullable: true
- description: >-
- Included only when `fields=scores` is requested. Contains item and
- trace scores only; experiment-level scores are returned by the
- experiments endpoint.
required:
- - id
- - traceId
- - startTime
- - endTime
- - level
- - environment
- - experimentId
- - experimentName
- - experimentItemId
- FeedbackTargetType:
- title: FeedbackTargetType
- type: string
- enum:
- - skill
- - mcp-tool
- - cli
- - docs
- - public-api
- - other
- SubmitFeedbackRequest:
- title: SubmitFeedbackRequest
+ - data
+ - meta
+ PaginatedDatasets:
+ title: PaginatedDatasets
type: object
properties:
- targetType:
- $ref: '#/components/schemas/FeedbackTargetType'
- description: Category of the thing the feedback is about.
- target:
- type: string
- description: >-
- The specific instance within targetType: the skill name, MCP tool
- name, CLI command, API endpoint path, or docs page path (e.g.
- 'queryMetrics', '/docs/mcp'). An identifier, not a sentence. Must be
- between 1 and 200 characters.
- feedback:
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/Dataset'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ required:
+ - data
+ - meta
+ CreateDatasetRequest:
+ title: CreateDatasetRequest
+ type: object
+ properties:
+ name:
type: string
- description: >-
- Concise feedback text approved by the user. Must be between 1 and
- 3000 characters.
- goal:
+ description:
type: string
nullable: true
+ metadata:
+ nullable: true
+ inputSchema:
+ nullable: true
description: >-
- Optional user-approved goal or use case they were trying to achieve.
- Must be between 1 and 1500 characters when provided. Do not include
- secrets, customer data, trace payloads, or broad unrelated context.
- referenceUrl:
- type: string
+ JSON Schema for validating dataset item inputs. When set, all new
+ and existing dataset items will be validated against this schema.
+ expectedOutputSchema:
nullable: true
description: >-
- Optional HTTP(S) reference URL. Langfuse stores it as text for
- triage and does not fetch it.
+ JSON Schema for validating dataset item expected outputs. When set,
+ all new and existing dataset items will be validated against this
+ schema.
required:
- - targetType
- - target
- - feedback
- SubmitFeedbackResponse:
- title: SubmitFeedbackResponse
+ - name
+ PaginatedDatasetRuns:
+ title: PaginatedDatasetRuns
type: object
properties:
- id:
- type: string
- description: Correlation ID for the submitted feedback.
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/DatasetRun'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
required:
- - id
- HealthResponse:
- title: HealthResponse
+ - data
+ - meta
+ DeleteDatasetRunResponse:
+ title: DeleteDatasetRunResponse
type: object
properties:
- version:
- type: string
- description: Langfuse server version
- status:
+ message:
type: string
required:
- - version
- - status
- IngestionEvent:
- title: IngestionEvent
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - trace-create
- - $ref: '#/components/schemas/TraceEvent'
- required:
- - type
- title: TraceEvent
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - score-create
- - $ref: '#/components/schemas/ScoreEvent'
- required:
- - type
- title: ScoreEvent
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - span-create
- - $ref: '#/components/schemas/CreateSpanEvent'
- required:
- - type
- title: CreateSpanEvent
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - span-update
- - $ref: '#/components/schemas/UpdateSpanEvent'
- required:
- - type
- title: UpdateSpanEvent
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - generation-create
- - $ref: '#/components/schemas/CreateGenerationEvent'
- required:
- - type
- title: CreateGenerationEvent
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - generation-update
- - $ref: '#/components/schemas/UpdateGenerationEvent'
- required:
- - type
- title: UpdateGenerationEvent
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - event-create
- - $ref: '#/components/schemas/CreateEventEvent'
- required:
- - type
- title: CreateEventEvent
+ - message
+ EvaluatorType:
+ title: EvaluatorType
+ type: string
+ enum:
+ - llm_as_judge
+ - code
+ description: |-
+ The evaluator type.
+
+ The public API supports LLM-as-a-judge and code evaluators.
+ CodeEvaluatorSourceCodeLanguage:
+ title: CodeEvaluatorSourceCodeLanguage
+ type: string
+ enum:
+ - PYTHON
+ - TYPESCRIPT
+ description: Code evaluator runtime language.
+ PromptVariableMappingSource:
+ title: PromptVariableMappingSource
+ type: string
+ enum:
+ - input
+ - output
+ - metadata
+ - tool_calls
+ - expected_output
+ - experiment_item_metadata
+ description: >-
+ Source field used to populate a prompt variable.
+
+
+ Use these values when mapping evaluator prompt variables to live data.
+
+
+ Source semantics:
+
+ - `input`: the observation input payload
+
+ - `output`: the observation output payload
+
+ - `metadata`: the observation metadata object. Combine with `jsonPath`
+ when you need one nested field instead of the whole object.
+
+ - `tool_calls`: the tool calls recorded on the observation, as an array
+ of `{id, name, arguments, type, index}` objects in the order the model
+ emitted them. Combine with `jsonPath` (for example `$[*].name`) to
+ select parts of each call.
+
+ - `expected_output`: the experiment item's expected output when the
+ observation belongs to an experiment.
+
+ - `experiment_item_metadata`: the experiment item's metadata when the
+ observation belongs to an experiment.
+ EvaluatorOutputScoreType:
+ title: EvaluatorOutputScoreType
+ type: string
+ enum:
+ - NUMERIC
+ - BOOLEAN
+ - CATEGORICAL
+ description: >-
+ Structured score type returned by an evaluator.
+
+
+ This controls the type of score value Langfuse stores for evaluation
+ results:
+
+ - `NUMERIC`: a numeric score such as `0.82`
+
+ - `BOOLEAN`: a boolean score such as `true`
+
+ - `CATEGORICAL`: one or more category labels from a fixed list
+ EvaluatorOutputDefinition:
+ title: EvaluatorOutputDefinition
+ oneOf:
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - sdk-log
- - $ref: '#/components/schemas/SDKLogEvent'
+ - NUMERIC
+ - $ref: '#/components/schemas/PublicEvaluatorNumericScore'
required:
- - type
- title: SDKLogEvent
+ - dataType
+ title: PublicEvaluatorNumericScore
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - observation-create
- - $ref: '#/components/schemas/CreateObservationEvent'
+ - BOOLEAN
+ - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'
required:
- - type
- title: CreateObservationEvent
+ - dataType
+ title: PublicEvaluatorBooleanScore
- type: object
allOf:
- type: object
properties:
- type:
+ dataType:
type: string
enum:
- - observation-update
- - $ref: '#/components/schemas/UpdateObservationEvent'
+ - CATEGORICAL
+ - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'
required:
- - type
- title: UpdateObservationEvent
- ObservationType:
- title: ObservationType
- type: string
- enum:
- - SPAN
- - GENERATION
- - EVENT
- - AGENT
- - TOOL
- - CHAIN
- - RETRIEVER
- - EVALUATOR
- - EMBEDDING
- - GUARDRAIL
- IngestionUsage:
- title: IngestionUsage
- oneOf:
- - $ref: '#/components/schemas/Usage'
- - $ref: '#/components/schemas/OpenAIUsage'
- OpenAIUsage:
- title: OpenAIUsage
- type: object
- description: Usage interface of OpenAI for improved compatibility.
- properties:
- promptTokens:
- type: integer
- nullable: true
- completionTokens:
- type: integer
- nullable: true
- totalTokens:
- type: integer
- nullable: true
- OptionalObservationBody:
- title: OptionalObservationBody
- type: object
- properties:
- traceId:
- type: string
- nullable: true
- name:
- type: string
- nullable: true
- startTime:
- type: string
- format: date-time
- nullable: true
- metadata:
- nullable: true
- input:
- nullable: true
- output:
- nullable: true
- level:
- $ref: '#/components/schemas/ObservationLevel'
- nullable: true
- statusMessage:
- type: string
- nullable: true
- parentObservationId:
- type: string
- nullable: true
- version:
- type: string
- nullable: true
- environment:
- type: string
- nullable: true
- CreateEventBody:
- title: CreateEventBody
- type: object
- properties:
- id:
- type: string
- nullable: true
- allOf:
- - $ref: '#/components/schemas/OptionalObservationBody'
- UpdateEventBody:
- title: UpdateEventBody
- type: object
- properties:
- id:
- type: string
- required:
- - id
- allOf:
- - $ref: '#/components/schemas/OptionalObservationBody'
- CreateSpanBody:
- title: CreateSpanBody
- type: object
- properties:
- endTime:
- type: string
- format: date-time
- nullable: true
- allOf:
- - $ref: '#/components/schemas/CreateEventBody'
- UpdateSpanBody:
- title: UpdateSpanBody
- type: object
- properties:
- endTime:
- type: string
- format: date-time
- nullable: true
- allOf:
- - $ref: '#/components/schemas/UpdateEventBody'
- CreateGenerationBody:
- title: CreateGenerationBody
- type: object
- properties:
- completionStartTime:
- type: string
- format: date-time
- nullable: true
- model:
- type: string
- nullable: true
- modelParameters:
- type: object
- additionalProperties:
- $ref: '#/components/schemas/MapValue'
- nullable: true
- usage:
- $ref: '#/components/schemas/IngestionUsage'
- nullable: true
- usageDetails:
- $ref: '#/components/schemas/UsageDetails'
- nullable: true
- costDetails:
- type: object
- additionalProperties:
- type: number
- format: double
- nullable: true
- promptName:
- type: string
- nullable: true
- promptVersion:
- type: integer
- nullable: true
- allOf:
- - $ref: '#/components/schemas/CreateSpanBody'
- UpdateGenerationBody:
- title: UpdateGenerationBody
+ - dataType
+ title: PublicEvaluatorCategoricalScore
+ description: >-
+ Flat structured output definition used when creating or updating an
+ evaluator.
+
+
+ - `dataType` is required.
+
+ - `scoreReasoningInstructions` and `scoreValueInstructions` are optional
+ instructions.
+
+ - `minValue` and `maxValue` apply only to `NUMERIC` outputs. If both are
+ set, `minValue` must not exceed `maxValue`.
+
+ - `categories` and `shouldAllowMultipleMatches` apply only to
+ `CATEGORICAL` outputs.
+
+ - Do not send `version`; that is an internal storage detail.
+ EvaluatorOutputDefinitionBase:
+ title: EvaluatorOutputDefinitionBase
type: object
properties:
- completionStartTime:
+ scoreReasoningInstructions:
type: string
- format: date-time
nullable: true
- model:
+ description: >-
+ Optional instructions for deriving the reasoning returned with the
+ score.
+ scoreValueInstructions:
type: string
nullable: true
- modelParameters:
- type: object
- additionalProperties:
- $ref: '#/components/schemas/MapValue'
- nullable: true
- usage:
- $ref: '#/components/schemas/IngestionUsage'
- nullable: true
- promptName:
+ description: Optional instructions for deriving the score value.
+ PublicEvaluatorNumericScore:
+ title: PublicEvaluatorNumericScore
+ type: object
+ properties:
+ dataType:
type: string
+ const: NUMERIC
+ description: Numeric score output.
+ minValue:
+ type: number
+ format: double
nullable: true
- usageDetails:
- $ref: '#/components/schemas/UsageDetails'
- nullable: true
- costDetails:
- type: object
- additionalProperties:
- type: number
- format: double
- nullable: true
- promptVersion:
- type: integer
+ description: Optional inclusive minimum value.
+ maxValue:
+ type: number
+ format: double
nullable: true
+ description: Optional inclusive maximum value.
+ required:
+ - dataType
allOf:
- - $ref: '#/components/schemas/UpdateSpanBody'
- ObservationBody:
- title: ObservationBody
+ - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'
+ PublicEvaluatorBooleanScore:
+ title: PublicEvaluatorBooleanScore
type: object
properties:
- id:
- type: string
- nullable: true
- traceId:
- type: string
- nullable: true
- type:
- $ref: '#/components/schemas/ObservationType'
- name:
+ dataType:
type: string
- nullable: true
- startTime:
+ const: BOOLEAN
+ description: Boolean score output.
+ required:
+ - dataType
+ allOf:
+ - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'
+ PublicEvaluatorCategoricalScore:
+ title: PublicEvaluatorCategoricalScore
+ type: object
+ properties:
+ dataType:
type: string
- format: date-time
- nullable: true
- endTime:
+ const: CATEGORICAL
+ description: Categorical score output.
+ categories:
+ type: array
+ items:
+ type: string
+ description: Allowed category values. At least two unique values are required.
+ shouldAllowMultipleMatches:
+ type: boolean
+ description: Whether the evaluator may return more than one category.
+ required:
+ - dataType
+ - categories
+ - shouldAllowMultipleMatches
+ allOf:
+ - $ref: '#/components/schemas/EvaluatorOutputDefinitionBase'
+ PublicEvaluatorOutputDefinition:
+ title: PublicEvaluatorOutputDefinition
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - NUMERIC
+ - $ref: '#/components/schemas/PublicEvaluatorNumericScore'
+ required:
+ - dataType
+ title: PublicEvaluatorNumericScore
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - BOOLEAN
+ - $ref: '#/components/schemas/PublicEvaluatorBooleanScore'
+ required:
+ - dataType
+ title: PublicEvaluatorBooleanScore
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CATEGORICAL
+ - $ref: '#/components/schemas/PublicEvaluatorCategoricalScore'
+ required:
+ - dataType
+ title: PublicEvaluatorCategoricalScore
+ description: >-
+ Flat evaluator output definition returned by the public API.
+
+
+ This response always includes `dataType` and never includes an internal
+ output-definition `version`.
+
+ Optional empty descriptions from legacy definitions are omitted.
+ EvaluationRuleStringFilterOperator:
+ title: EvaluationRuleStringFilterOperator
+ type: string
+ enum:
+ - '='
+ - contains
+ - does not contain
+ - starts with
+ - ends with
+ EvaluationRuleNumberFilterOperator:
+ title: EvaluationRuleNumberFilterOperator
+ type: string
+ enum:
+ - '='
+ - '>'
+ - <
+ - '>='
+ - <=
+ EvaluationRuleOptionsFilterOperator:
+ title: EvaluationRuleOptionsFilterOperator
+ type: string
+ enum:
+ - any of
+ - none of
+ EvaluationRuleArrayOptionsFilterOperator:
+ title: EvaluationRuleArrayOptionsFilterOperator
+ type: string
+ enum:
+ - any of
+ - none of
+ - all of
+ EvaluationRuleBooleanFilterOperator:
+ title: EvaluationRuleBooleanFilterOperator
+ type: string
+ enum:
+ - '='
+ - <>
+ EvaluationRuleNullFilterOperator:
+ title: EvaluationRuleNullFilterOperator
+ type: string
+ enum:
+ - is null
+ - is not null
+ DateTimeEvaluationRuleFilter:
+ title: DateTimeEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- format: date-time
- nullable: true
- completionStartTime:
+ description: Column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'
+ description: Comparison operator for datetime values.
+ value:
type: string
format: date-time
- nullable: true
- model:
+ description: Datetime value to compare against.
+ required:
+ - column
+ - operator
+ - value
+ StringEvaluationRuleFilter:
+ title: StringEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
- modelParameters:
- type: object
- additionalProperties:
- $ref: '#/components/schemas/MapValue'
- nullable: true
- input:
- nullable: true
- version:
+ description: Column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'
+ value:
type: string
- nullable: true
- metadata:
- nullable: true
- output:
- nullable: true
- usage:
- $ref: '#/components/schemas/Usage'
- nullable: true
- level:
- $ref: '#/components/schemas/ObservationLevel'
- nullable: true
- statusMessage:
+ required:
+ - column
+ - operator
+ - value
+ NumberEvaluationRuleFilter:
+ title: NumberEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
- parentObservationId:
+ description: Column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'
+ value:
+ type: number
+ format: double
+ required:
+ - column
+ - operator
+ - value
+ StringOptionsEvaluationRuleFilter:
+ title: StringOptionsEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
- environment:
+ description: Column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'
+ value:
+ type: array
+ items:
+ type: string
+ description: One or more allowed string values.
+ required:
+ - column
+ - operator
+ - value
+ ArrayOptionsEvaluationRuleFilter:
+ title: ArrayOptionsEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
+ description: Column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleArrayOptionsFilterOperator'
+ value:
+ type: array
+ items:
+ type: string
+ description: One or more array elements to match.
required:
- - type
- TraceBody:
- title: TraceBody
+ - column
+ - operator
+ - value
+ StringObjectEvaluationRuleFilter:
+ title: StringObjectEvaluationRuleFilter
type: object
properties:
- id:
+ column:
type: string
- nullable: true
- timestamp:
+ description: >-
+ Object-valued column to filter on. Currently only `metadata` is
+ supported.
+ key:
type: string
- format: date-time
- nullable: true
- name:
+ description: Top-level key inside the object-valued column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleStringFilterOperator'
+ value:
type: string
- nullable: true
- userId:
+ required:
+ - column
+ - key
+ - operator
+ - value
+ NumberObjectEvaluationRuleFilter:
+ title: NumberObjectEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
- input:
- nullable: true
- output:
- nullable: true
- sessionId:
+ description: Object-valued column to filter on.
+ key:
type: string
- nullable: true
- release:
+ description: Key inside the object-valued column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleNumberFilterOperator'
+ value:
+ type: number
+ format: double
+ required:
+ - column
+ - key
+ - operator
+ - value
+ CategoryOptionsEvaluationRuleFilter:
+ title: CategoryOptionsEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
- version:
+ description: Object-valued column to filter on.
+ key:
type: string
- nullable: true
- metadata:
- nullable: true
- tags:
+ description: Key inside the object-valued column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleOptionsFilterOperator'
+ value:
type: array
items:
type: string
- nullable: true
- environment:
+ required:
+ - column
+ - key
+ - operator
+ - value
+ BooleanEvaluationRuleFilter:
+ title: BooleanEvaluationRuleFilter
+ type: object
+ properties:
+ column:
type: string
- nullable: true
- public:
+ description: Column to filter on.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleBooleanFilterOperator'
+ value:
type: boolean
- nullable: true
- description: Make trace publicly accessible via url
- SDKLogBody:
- title: SDKLogBody
+ required:
+ - column
+ - operator
+ - value
+ NullEvaluationRuleFilter:
+ title: NullEvaluationRuleFilter
type: object
properties:
- log: {}
+ column:
+ type: string
+ description: Column to filter on, for example `parentObservationId`.
+ operator:
+ $ref: '#/components/schemas/EvaluationRuleNullFilterOperator'
+ value:
+ type: string
+ const: ''
+ description: >-
+ Required empty-string placeholder used by the runtime filter
+ contract.
required:
- - log
- ScoreBody:
- title: ScoreBody
+ - column
+ - operator
+ - value
+ PromptVariableMappingInput:
+ title: PromptVariableMappingInput
type: object
+ description: >-
+ Connects one prompt variable to data from an observation or experiment.
+
+
+ Manual mappings are used for `llm_as_judge` evaluators. `code`
+ evaluators use a fixed runtime mapping managed by Langfuse.
+
+
+ How to build a valid mapping list:
+
+ 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.
+
+ 2. Read the evaluator `variables` array.
+
+ 3. Add exactly one mapping object for each variable in that array.
+
+ 4. Use the variable name exactly as returned, without braces such as
+ `{{` or `}}`.
+
+ 5. Choose the source field that should populate the variable.
+
+
+ `jsonPath` is optional. Use it only when the selected source is a JSON
+ object and you want to extract one nested field before inserting it into
+ the evaluator prompt.
+
+
+ Invalid, missing, or duplicate mappings return a validation error.
+ Malformed JSONPath expressions are also rejected.
properties:
- id:
+ variable:
type: string
- nullable: true
- traceId:
+ description: >-
+ Prompt variable name without braces.
+
+
+ Example: for the prompt `Judge {{input}} against {{output}}`, use
+ `input` and `output`.
+ source:
+ $ref: '#/components/schemas/PromptVariableMappingSource'
+ description: >-
+ Source field that should populate the prompt variable.
+
+
+ Available sources are `input`, `output`, `metadata`, `tool_calls`,
+ `expected_output`, and `experiment_item_metadata`.
+ jsonPath:
type: string
nullable: true
- sessionId:
+ description: >-
+ Optional JSONPath selector applied to the selected source before it
+ is passed to the evaluator prompt.
+
+
+ Requirements:
+
+ - Must start with `$`
+
+ - Must be a syntactically valid JSONPath expression
+
+ - Most useful with `source=metadata`
+ required:
+ - variable
+ - source
+ PromptVariableMappingRead:
+ title: PromptVariableMappingRead
+ type: object
+ description: |-
+ Connects one prompt variable to source data.
+
+ `source` is `null` when the mapping is incomplete.
+ properties:
+ variable:
type: string
- nullable: true
- observationId:
+ description: Prompt variable name without braces.
+ source:
type: string
nullable: true
- datasetRunId:
+ description: >-
+ Stored source field populating the variable, or `null` when the
+ mapping is incomplete.
+ jsonPath:
type: string
nullable: true
- name:
+ description: Optional JSONPath selector applied to the selected source.
+ required:
+ - variable
+ - source
+ PromptVariableMapping:
+ title: PromptVariableMapping
+ oneOf:
+ - $ref: '#/components/schemas/PromptVariableMappingRead'
+ - $ref: '#/components/schemas/LegacyPromptVariableMapping'
+ description: Connects one prompt variable to source data.
+ LegacyPromptVariableMapping:
+ title: LegacyPromptVariableMapping
+ type: object
+ description: >-
+ **Deprecated:** Connects one prompt variable to data from a legacy trace
+ or dataset evaluation rule.
+
+
+ `langfuseObject` selects the object kind. `objectName` separately
+ selects a named observation and is `null` for trace and dataset-item
+ mappings.
+ properties:
+ mappingType:
type: string
- description: >-
- The name of the score. Always overrides "output" for correction
- scores.
- environment:
+ const: legacy
+ description: Explicitly marks this as a legacy mapping.
+ variable:
type: string
- nullable: true
- queueId:
+ description: Prompt variable name without braces.
+ langfuseObject:
+ $ref: '#/components/schemas/LegacyEvaluationObject'
+ description: Legacy object kind selected as the mapping source.
+ objectName:
type: string
nullable: true
description: >-
- The annotation queue referenced by the score. Indicates if score was
- initially created while processing annotation queue.
- value:
- $ref: '#/components/schemas/CreateScoreValue'
- description: >-
- The value of the score. Must be passed as string for categorical and
- text scores, and numeric for boolean and numeric scores. Boolean
- score values must equal either 1 or 0 (true or false). Text score
- values must be between 1 and 500 characters.
- comment:
+ Observation name to match, or `null` when `langfuseObject` is
+ `trace` or `dataset_item`.
+ source:
type: string
- nullable: true
- metadata:
- nullable: true
- dataType:
- $ref: '#/components/schemas/ScoreDataType'
- nullable: true
- description: >-
- When set, must match the score value's type. If not set, will be
- inferred from the score value or config
- configId:
+ description: Field selected from the legacy object.
+ jsonPath:
type: string
nullable: true
- description: >-
- Reference a score config on a score. When set, the score name must
- equal the config name and scores must comply with the config's range
- and data type. For categorical scores, the value must map to a
- config category. Numeric scores might be constrained by the score
- config's max and min values
+ description: Optional JSONPath selector applied to the selected field.
required:
- - name
- - value
- BaseEvent:
- title: BaseEvent
+ - mappingType
+ - variable
+ - langfuseObject
+ - objectName
+ - source
+ LegacyEvaluationObject:
+ title: LegacyEvaluationObject
+ type: string
+ enum:
+ - trace
+ - span
+ - generation
+ - event
+ - agent
+ - tool
+ - chain
+ - retriever
+ - evaluator
+ - embedding
+ - guardrail
+ - dataset_item
+ description: >-
+ **Deprecated:** Legacy Langfuse object kind used by trace and dataset
+ evaluation rules.
+ EvaluationRuleFilter:
+ title: EvaluationRuleFilter
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - datetime
+ - $ref: '#/components/schemas/DateTimeEvaluationRuleFilter'
+ required:
+ - type
+ title: DateTimeEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - string
+ - $ref: '#/components/schemas/StringEvaluationRuleFilter'
+ required:
+ - type
+ title: StringEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - number
+ - $ref: '#/components/schemas/NumberEvaluationRuleFilter'
+ required:
+ - type
+ title: NumberEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - stringOptions
+ - $ref: '#/components/schemas/StringOptionsEvaluationRuleFilter'
+ required:
+ - type
+ title: StringOptionsEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - categoryOptions
+ - $ref: '#/components/schemas/CategoryOptionsEvaluationRuleFilter'
+ required:
+ - type
+ title: CategoryOptionsEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - arrayOptions
+ - $ref: '#/components/schemas/ArrayOptionsEvaluationRuleFilter'
+ required:
+ - type
+ title: ArrayOptionsEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - stringObject
+ - $ref: '#/components/schemas/StringObjectEvaluationRuleFilter'
+ required:
+ - type
+ title: StringObjectEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - numberObject
+ - $ref: '#/components/schemas/NumberObjectEvaluationRuleFilter'
+ required:
+ - type
+ title: NumberObjectEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - boolean
+ - $ref: '#/components/schemas/BooleanEvaluationRuleFilter'
+ required:
+ - type
+ title: BooleanEvaluationRuleFilter
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - 'null'
+ - $ref: '#/components/schemas/NullEvaluationRuleFilter'
+ required:
+ - type
+ title: NullEvaluationRuleFilter
+ description: >-
+ One filter condition used to decide whether a live-ingested observation
+ should be evaluated.
+
+
+ All filters must be satisfied for the rule to run. Experiment scope is
+ expressed with filters rather than a separate target field:
+
+ - `isExperimentItemRootSpan = true` limits execution to experiment item
+ roots.
+
+ - `datasetId` limits execution to experiments for the selected datasets.
+ Use dataset IDs from `GET /api/public/v2/datasets`.
+
+
+ Pick the filter `type` first, as it determines the required fields and
+ value shape. Use `key` only for object filters such as `metadata`.
+
+
+ Operator quick reference by filter `type`:
+
+ - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends
+ with`
+
+ - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`
+
+ - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`
+
+ - `stringOptions`: `any of`, `none of`
+
+ - `arrayOptions`: `any of`, `none of`, `all of`
+
+ - `stringObject`: same operators as `string`
+
+ - `boolean`: `"="`, `"<>"`
+
+ - `null`: `is null`, `is not null`
+
+
+ Supported columns. Each column accepts exactly one filter `type`;
+ sending a column with a different `type` is rejected with `400`.
+
+ - `type`: `stringOptions`, values `SPAN`, `EVENT`, `GENERATION`,
+ `AGENT`, `TOOL`, `CHAIN`, `RETRIEVER`, `EVALUATOR`, `EMBEDDING`,
+ `GUARDRAIL`
+
+ - `name`: `stringOptions`
+
+ - `environment`: `stringOptions`
+
+ - `level`: `stringOptions`, values `DEBUG`, `DEFAULT`, `WARNING`,
+ `ERROR`
+
+ - `version`: `string`
+
+ - `traceName`: `stringOptions`
+
+ - `userId`: `string`
+
+ - `sessionId`: `string`
+
+ - `tags`: `arrayOptions`
+
+ - `metadata`: `stringObject`, requires `key`
+
+ - `isRootObservation`: `boolean`; true when the observation has no
+ parent or is explicitly marked as an application root
+
+ - `parentObservationId`: `null`
+
+ - `experimentId`: `stringOptions`
+
+ - `isExperimentItemRootSpan`: `boolean`
+
+ - `calledToolNames`: `arrayOptions`
+
+ - `toolCalls`: `number`, the number of tool calls on the observation
+
+ - `datasetId`: `stringOptions`. Use dataset `id` values from `GET
+ /api/public/v2/datasets`, not dataset names.
+ EvaluationRuleReadFilterBase:
+ title: EvaluationRuleReadFilterBase
type: object
+ description: Stored filter returned verbatim by evaluation-rule read endpoints.
properties:
- id:
- type: string
- description: UUID v4 that identifies the event
- timestamp:
+ type:
type: string
description: >-
- Datetime (ISO 8601) of event creation in client. Should be as close
- to actual event creation in client as possible, this timestamp will
- be used for ordering of events in future release. Resolution:
- milliseconds (required), microseconds (optimal).
- metadata:
+ Stored filter type. This is not broken down into separate public
+ response types.
+ column:
+ type: string
+ operator:
+ type: string
+ value:
nullable: true
- description: Optional. Metadata field used by the Langfuse SDKs for debugging.
- required:
- - id
- - timestamp
- TraceEvent:
- title: TraceEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/TraceBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- CreateObservationEvent:
- title: CreateObservationEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/ObservationBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- UpdateObservationEvent:
- title: UpdateObservationEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/ObservationBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- ScoreEvent:
- title: ScoreEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/ScoreBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- SDKLogEvent:
- title: SDKLogEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/SDKLogBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- CreateGenerationEvent:
- title: CreateGenerationEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/CreateGenerationBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- UpdateGenerationEvent:
- title: UpdateGenerationEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/UpdateGenerationBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- CreateSpanEvent:
- title: CreateSpanEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/CreateSpanBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- UpdateSpanEvent:
- title: UpdateSpanEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/UpdateSpanBody'
- required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- CreateEventEvent:
- title: CreateEventEvent
- type: object
- properties:
- body:
- $ref: '#/components/schemas/CreateEventBody'
+ description: Stored filter value. Its shape depends on the filter type.
required:
- - body
- allOf:
- - $ref: '#/components/schemas/BaseEvent'
- IngestionSuccess:
- title: IngestionSuccess
+ - type
+ - column
+ - operator
+ EvaluationRuleReadFilterWithKey:
+ title: EvaluationRuleReadFilterWithKey
type: object
properties:
- id:
+ key:
type: string
- status:
- type: integer
+ description: Stored object or legacy filter key.
required:
- - id
- - status
- IngestionError:
- title: IngestionError
+ - key
+ allOf:
+ - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'
+ EvaluationRuleReadFilter:
+ title: EvaluationRuleReadFilter
+ oneOf:
+ - $ref: '#/components/schemas/EvaluationRuleReadFilterWithKey'
+ - $ref: '#/components/schemas/EvaluationRuleReadFilterBase'
+ description: >-
+ Filter returned verbatim by evaluation-rule read endpoints.
+
+
+ Filters with a stored `key` use the keyed shape. All other filters use
+ the base shape. Response filters are intentionally not broken down by
+ their internal `type` value.
+ PublicApiErrorCode:
+ title: PublicApiErrorCode
+ type: string
+ enum:
+ - authentication_failed
+ - access_denied
+ - invalid_request
+ - invalid_query
+ - invalid_body
+ - resource_not_found
+ - conflict
+ - rate_limited
+ - method_not_allowed
+ - internal_error
+ description: >-
+ Stable machine-readable error code returned by the evaluators and
+ evaluation-rules API.
+ PublicApiValidationIssue:
+ title: PublicApiValidationIssue
type: object
+ description: One request validation issue.
properties:
- id:
+ code:
type: string
- status:
- type: integer
+ description: Validator issue code.
message:
type: string
- nullable: true
- error:
- nullable: true
- required:
- - id
- - status
- IngestionResponse:
- title: IngestionResponse
- type: object
- properties:
- successes:
- type: array
- items:
- $ref: '#/components/schemas/IngestionSuccess'
- errors:
+ description: Human-readable explanation.
+ path:
type: array
- items:
- $ref: '#/components/schemas/IngestionError'
- required:
- - successes
- - errors
- OpenAICompletionUsageSchema:
- title: OpenAICompletionUsageSchema
- type: object
- description: OpenAI Usage schema from (Chat-)Completion APIs
- properties:
- prompt_tokens:
- type: integer
- completion_tokens:
- type: integer
- total_tokens:
- type: integer
- prompt_tokens_details:
- type: object
- additionalProperties:
- type: integer
- nullable: true
- nullable: true
- completion_tokens_details:
- type: object
- additionalProperties:
- type: integer
- nullable: true
- nullable: true
- required:
- - prompt_tokens
- - completion_tokens
- - total_tokens
- OpenAIResponseUsageSchema:
- title: OpenAIResponseUsageSchema
+ items: {}
+ description: Path to the invalid request field.
+ required:
+ - code
+ - message
+ - path
+ PublicApiErrorDetails:
+ title: PublicApiErrorDetails
type: object
- description: OpenAI Usage schema from Response API
+ description: Optional structured context for validation and rate-limit errors.
properties:
- input_tokens:
+ issues:
+ type: array
+ items:
+ $ref: '#/components/schemas/PublicApiValidationIssue'
+ nullable: true
+ description: Validation issues for an invalid request body or query.
+ retryAfterSeconds:
type: integer
- output_tokens:
+ nullable: true
+ description: Number of seconds to wait before retrying a rate-limited request.
+ limit:
type: integer
- total_tokens:
+ nullable: true
+ description: Rate-limit request allowance.
+ remaining:
type: integer
- input_tokens_details:
- type: object
- additionalProperties:
- type: integer
- nullable: true
nullable: true
- output_tokens_details:
- type: object
- additionalProperties:
- type: integer
- nullable: true
+ description: Remaining requests in the current rate-limit window.
+ resetAt:
+ type: string
+ format: date-time
nullable: true
- required:
- - input_tokens
- - output_tokens
- - total_tokens
- UsageDetails:
- title: UsageDetails
- oneOf:
- - type: object
- additionalProperties:
- type: integer
- title: Object
- - $ref: '#/components/schemas/OpenAICompletionUsageSchema'
- - $ref: '#/components/schemas/OpenAIResponseUsageSchema'
- legacyMetricsResponse:
- title: legacyMetricsResponse
+ description: Time when the current rate-limit window resets.
+ PublicApiError:
+ title: PublicApiError
type: object
- properties:
- data:
- type: array
- items:
- type: object
- additionalProperties: true
- description: >-
- The metrics data. Each item in the list contains the metric values
- and dimensions requested in the query.
+ description: >-
+ Standard error response for the stable evaluators and evaluation-rules
+ API.
- Format varies based on the query parameters.
- Histograms will return an array with [lower, upper, height] tuples.
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
+ Use the HTTP status for the broad failure class and `code` for
+ programmatic handling. `details` is included when field-level validation
+ or retry information is available.
+ properties:
+ message:
+ type: string
+ description: Human-readable description of the failure.
+ code:
+ $ref: '#/components/schemas/PublicApiErrorCode'
+ description: Stable machine-readable error code.
+ details:
+ $ref: '#/components/schemas/PublicApiErrorDetails'
nullable: true
+ description: Optional structured validation or rate-limit context.
required:
- - data
- legacyObservations:
- title: legacyObservations
+ - message
+ - code
+ EvaluationRuleEvaluatorAssignmentInput:
+ title: EvaluationRuleEvaluatorAssignmentInput
type: object
properties:
- data:
+ evaluatorId:
+ type: string
+ description: >-
+ Stable evaluator identifier. The rule automatically uses that
+ evaluator's latest version.
+ variableMapping:
type: array
items:
- $ref: '#/components/schemas/Observation'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
+ $ref: '#/components/schemas/PromptVariableMappingInput'
+ nullable: true
+ description: >-
+ Rule-specific prompt-variable mapping.
+
+
+ Set to `null` or omit to inherit the evaluator's latest default
+ mapping. Code evaluators use the fixed runtime mapping and should
+ use `null`.
required:
- - data
- - meta
- legacyObservationsViews:
- title: legacyObservationsViews
+ - evaluatorId
+ EvaluatorAssignment:
+ title: EvaluatorAssignment
type: object
properties:
- data:
+ evaluatorId:
+ type: string
+ description: Stable identifier of the evaluator associated with this rule.
+ variableMapping:
type: array
items:
- $ref: '#/components/schemas/ObservationsView'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
+ $ref: '#/components/schemas/PromptVariableMapping'
nullable: true
+ description: >-
+ Stored rule-specific override, or `null` when the evaluator's latest
+ default mapping is inherited. Legacy mappings are returned as an
+ explicit `legacy` variant.
required:
- - data
- - meta
- LlmConnection:
- title: LlmConnection
+ - evaluatorId
+ - variableMapping
+ EvaluationRule:
+ title: EvaluationRule
type: object
- description: LLM API connection configuration (secrets excluded)
+ description: >-
+ Live evaluation rule for incoming observations.
+
+
+ A rule determines which evaluators should be used, which observations
+ should trigger scoring, how often scoring should run, and which
+ observation fields should populate prompt variables.
properties:
id:
type: string
- provider:
+ description: Stable evaluation-rule identifier.
+ name:
type: string
description: >-
- Provider name (e.g., 'openai', 'my-gateway'). Must be unique in
- project, used for upserting.
- adapter:
+ Human-readable rule name. This is independent from evaluator names
+ and does not need to be unique.
+ createdBy:
+ $ref: '#/components/schemas/Creator'
+ nullable: true
+ description: User who created this rule, or `null` when no user can be resolved.
+ enabled:
+ type: boolean
+ description: Whether live execution is enabled for this rule.
+ sampling:
+ type: number
+ format: double
+ description: |-
+ Fraction of matching observations that should be evaluated.
+
+ Must be between `0` and `1`.
+ - `1` evaluates every matching observation.
+ - `0.25` evaluates approximately 25% of matching observations.
+ filter:
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluationRuleReadFilter'
+ description: >-
+ List of stored filter conditions returned verbatim. Filters with a
+ `key` use the keyed response shape; all others use the base shape.
+ These response shapes are not broken down by internal filter type.
+ An empty list matches every incoming object.
+ evaluatorAssignments:
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluatorAssignment'
+ description: Evaluators attached to this rule in deterministic assignment order.
+ createdAt:
type: string
- description: The adapter used to interface with the LLM
- displaySecretKey:
+ format: date-time
+ description: Timestamp when the evaluation rule was created.
+ updatedAt:
type: string
- description: Masked version of the secret key for display purposes
- baseURL:
+ format: date-time
+ description: Timestamp when the evaluation rule was last updated.
+ required:
+ - id
+ - name
+ - createdBy
+ - enabled
+ - sampling
+ - filter
+ - evaluatorAssignments
+ - createdAt
+ - updatedAt
+ CreateEvaluationRuleRequest:
+ title: CreateEvaluationRuleRequest
+ type: object
+ properties:
+ name:
+ type: string
+ description: >-
+ Human-readable rule name. Names are not identifiers and do not need
+ to be unique.
+ enabled:
+ type: boolean
+ description: >-
+ Whether live execution should start immediately. Enabled rules
+ require at least one evaluator assignment.
+ sampling:
+ type: number
+ format: double
+ nullable: true
+ description: >-
+ Fraction of matching observations to evaluate. Omit this field to
+ use the default of `1`, which evaluates every match. `null` is not
+ accepted.
+ filter:
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluationRuleFilter'
+ nullable: true
+ description: >-
+ Conditions used to select observations. Defaults to an empty list,
+ which matches every incoming observation.
+ evaluatorAssignments:
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'
+ description: >-
+ Evaluators to attach to this rule. Disabled rules may use an empty
+ list as a draft.
+ required:
+ - name
+ - enabled
+ - evaluatorAssignments
+ UpdateEvaluationRuleRequest:
+ title: UpdateEvaluationRuleRequest
+ type: object
+ description: At least one field must be provided.
+ properties:
+ name:
type: string
nullable: true
- description: Custom base URL for the LLM API
- customModels:
- type: array
- items:
- type: string
- description: List of custom model names available for this connection
- withDefaultModels:
+ description: New human-readable rule name.
+ enabled:
type: boolean
- description: Whether to include default models for this adapter
- extraHeaderKeys:
+ nullable: true
+ description: New desired live-execution state.
+ sampling:
+ type: number
+ format: double
+ nullable: true
+ description: >-
+ New fraction of matching observations to evaluate. Omit to keep the
+ current value.
+ filter:
type: array
items:
- type: string
+ $ref: '#/components/schemas/EvaluationRuleFilter'
+ nullable: true
description: >-
- Keys of extra headers sent with requests (values excluded for
- security)
- config:
- type: object
- additionalProperties: true
+ Complete replacement filter list. An empty list matches every
+ incoming observation.
+ evaluatorAssignments:
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluationRuleEvaluatorAssignmentInput'
nullable: true
description: >-
- Adapter-specific configuration. Required for Bedrock
- (`{"region":"us-east-1"}`), optional for OpenAI
- (`{"useResponsesApi":true}`), optional for VertexAI
- (`{"location":"us-central1"}`), not used by other adapters.
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- required:
- - id
- - provider
- - adapter
- - displaySecretKey
- - customModels
- - withDefaultModels
- - extraHeaderKeys
- - createdAt
- - updatedAt
- PaginatedLlmConnections:
- title: PaginatedLlmConnections
+ Complete replacement assignment list. An empty list disables the
+ rule.
+ EvaluationRulesPage:
+ title: EvaluationRulesPage
type: object
properties:
data:
type: array
items:
- $ref: '#/components/schemas/LlmConnection'
+ $ref: '#/components/schemas/EvaluationRule'
+ description: Evaluation rules for this page.
meta:
- $ref: '#/components/schemas/utilsMetaResponse'
+ $ref: '#/components/schemas/CursorMeta'
+ description: Cursor pagination metadata.
required:
- data
- meta
- UpsertLlmConnectionRequest:
- title: UpsertLlmConnectionRequest
+ DeletedEvaluationRule:
+ title: DeletedEvaluationRule
type: object
- description: Request to create or update an LLM connection (upsert)
+ description: Confirmation returned after successful evaluation-rule deletion.
properties:
- provider:
+ id:
type: string
- description: >-
- Provider name (e.g., 'openai', 'my-gateway'). Must be unique in
- project, used for upserting.
- adapter:
- $ref: '#/components/schemas/LlmAdapter'
- description: The adapter used to interface with the LLM
- secretKey:
+ description: Identifier of the deleted evaluation rule.
+ required:
+ - id
+ Creator:
+ title: Creator
+ type: object
+ description: User who created the resource, or `null` when no user can be resolved.
+ properties:
+ id:
type: string
- description: Secret key for the LLM API.
- baseURL:
+ description: User identifier.
+ name:
type: string
nullable: true
- description: Custom base URL for the LLM API
- customModels:
- type: array
- items:
- type: string
- nullable: true
- description: List of custom model names
- withDefaultModels:
- type: boolean
- nullable: true
- description: Whether to include default models. Default is true.
- extraHeaders:
- type: object
- additionalProperties:
- type: string
- nullable: true
- description: Extra headers to send with requests
- config:
- type: object
- additionalProperties: true
- nullable: true
- description: >-
- Adapter-specific configuration. Validation rules: - **Bedrock**:
- Required. Must be `{"region": "<aws-region>"}` (e.g.,
- `{"region":"us-east-1"}`) - **OpenAI**: Optional. If provided, must
- be `{"useResponsesApi": <boolean>}` to control whether Langfuse
- routes calls through OpenAI's Responses API. - **VertexAI**:
- Optional. If provided, must be `{"location": "<gcp-location>"}`
- (e.g., `{"location":"us-central1"}`) - **Other adapters**: Not
- supported. Omit this field or set to null.
+ description: User name, or `null` when unavailable.
required:
- - provider
- - adapter
- - secretKey
- DeleteLlmConnectionResponse:
- title: DeleteLlmConnectionResponse
+ - id
+ - name
+ EvaluatorModelConfig:
+ title: EvaluatorModelConfig
type: object
+ description: >-
+ Explicit model configuration for an evaluator.
+
+
+ If the complete `modelConfig` is `null`, Langfuse uses the project's
+ default evaluation model.
+
+ If provided, the model must be available to the project when the
+ evaluator or evaluation rule is enabled.
properties:
- message:
+ provider:
+ type: string
+ description: >-
+ Provider identifier, for example `openai` or `anthropic`.
+
+
+ To discover valid values for the current project, call `GET
+ /api/public/llm-connections` and use one of the returned `provider`
+ values.
+ model:
type: string
+ description: >-
+ Model identifier exposed by the provider, for example `gpt-4.1-mini`.
required:
- - message
- LlmAdapter:
- title: LlmAdapter
+ - provider
+ - model
+ EvaluatorChatMessageRole:
+ title: EvaluatorChatMessageRole
type: string
enum:
- - anthropic
- - openai
- - azure
- - bedrock
- - google-vertex-ai
- - google-ai-studio
- GetMediaResponse:
- title: GetMediaResponse
+ - system
+ - user
+ - assistant
+ description: Role of an evaluator prompt message.
+ EvaluatorChatMessage:
+ title: EvaluatorChatMessage
type: object
+ description: One chat message in an evaluator prompt.
properties:
- mediaId:
- type: string
- description: The unique langfuse identifier of a media record
- contentType:
- type: string
- description: The MIME type of the media record
- contentLength:
- type: integer
- description: The size of the media record in bytes
- uploadedAt:
- type: string
- format: date-time
- description: The date and time when the media record was uploaded
- url:
- type: string
- description: The download URL of the media record
- urlExpiry:
+ role:
+ $ref: '#/components/schemas/EvaluatorChatMessageRole'
+ content:
type: string
- description: The expiry date and time of the media record download URL
+ description: Message content. Evaluator variables use `{{variable}}` syntax.
required:
- - mediaId
- - contentType
- - contentLength
- - uploadedAt
- - url
- - urlExpiry
- PatchMediaBody:
- title: PatchMediaBody
+ - role
+ - content
+ EvaluatorChatPrompt:
+ title: EvaluatorChatPrompt
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluatorChatMessage'
+ description: >-
+ An ordered list of chat messages. A system message is only allowed as
+ the first message.
+ EvaluatorChatPromptInput:
+ title: EvaluatorChatPromptInput
+ oneOf:
+ - type: string
+ title: String
+ - $ref: '#/components/schemas/EvaluatorChatPrompt'
+ description: A user prompt string shortcut, or an ordered list of chat messages.
+ EvaluatorVersionBase:
+ title: EvaluatorVersionBase
type: object
properties:
- uploadedAt:
- type: string
- format: date-time
- description: The date and time when the media record was uploaded
- uploadHttpStatus:
- type: integer
- description: The HTTP status code of the upload
- uploadHttpError:
+ id:
type: string
- nullable: true
- description: The HTTP error message of the upload
- uploadTimeMs:
+ description: Stable identifier of this evaluator version.
+ version:
type: integer
- nullable: true
- description: The time in milliseconds it took to upload the media record
- required:
- - uploadedAt
- - uploadHttpStatus
- GetMediaUploadUrlRequest:
- title: GetMediaUploadUrlRequest
- type: object
- description: >-
- Request a presigned media upload URL. Provide exactly one context: a
- trace (traceId, optionally observationId) or a dataset item (datasetId +
- datasetItemId). field is required and must match the chosen context.
- properties:
- traceId:
+ description: Monotonically increasing evaluator version number.
+ createdAt:
type: string
+ format: date-time
+ description: Timestamp when this evaluator version was created.
+ createdBy:
+ $ref: '#/components/schemas/Creator'
nullable: true
description: >-
- The trace the media is associated with. Null for dataset item media
- uploads.
- observationId:
+ User who created this version, or `null` when no user can be
+ resolved.
+ required:
+ - id
+ - version
+ - createdAt
+ - createdBy
+ LlmAsJudgeEvaluatorVersion:
+ title: LlmAsJudgeEvaluatorVersion
+ type: object
+ properties:
+ type:
type: string
- nullable: true
+ const: llm_as_judge
+ description: Evaluator type.
+ prompt:
+ $ref: '#/components/schemas/EvaluatorChatPrompt'
+ description: Ordered chat messages used during evaluation.
+ variables:
+ type: array
+ items:
+ type: string
description: >-
- The observation ID associated with the media record. If the media
- record is associated directly with a trace, this will be null.
- datasetId:
- type: string
+ Variables extracted from the prompt and available for
+ evaluation-rule mappings.
+
+
+ Every variable must be mapped exactly once when a rule provides an
+ explicit mapping.
+ variableMapping:
+ type: array
+ items:
+ $ref: '#/components/schemas/PromptVariableMappingRead'
nullable: true
description: >-
- The dataset the media belongs to. Null for trace/observation media
- uploads.
- datasetItemId:
- type: string
+ Default variable mapping for this evaluator version, or `null` when
+ no default is configured.
+ modelConfig:
+ $ref: '#/components/schemas/EvaluatorModelConfig'
nullable: true
description: >-
- The dataset item the media is associated with (need not exist yet).
- Null for trace/observation media uploads.
- contentType:
- $ref: '#/components/schemas/MediaContentType'
- contentLength:
- type: integer
- description: The size of the media record in bytes
- sha256Hash:
- type: string
- description: The SHA-256 hash of the media record
- field:
- type: string
+ Explicit model configuration, or `null` when the project's default
+ evaluation model is used.
+ outputDefinition:
+ $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'
description: >-
- The item field the media is in: `input`/`output`/`metadata` (trace)
- or `input`/`expectedOutput`/`metadata` (dataset item).
+ Structured output schema returned by this evaluator.
+
+
+ Responses include `dataType` and omit the internal output-definition
+ persistence version.
required:
- - contentType
- - contentLength
- - sha256Hash
- - field
- GetMediaUploadUrlResponse:
- title: GetMediaUploadUrlResponse
+ - type
+ - prompt
+ - variables
+ - variableMapping
+ - modelConfig
+ - outputDefinition
+ allOf:
+ - $ref: '#/components/schemas/EvaluatorVersionBase'
+ CodeEvaluatorVersion:
+ title: CodeEvaluatorVersion
type: object
properties:
- uploadUrl:
+ type:
type: string
- nullable: true
- description: >-
- The presigned upload URL. If the asset is already uploaded, this
- will be null
- mediaId:
+ const: code
+ description: Evaluator type.
+ sourceCode:
type: string
- description: The unique langfuse identifier of a media record
+ description: Source code executed for each matched observation.
+ sourceCodeLanguage:
+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
+ description: Runtime language used to execute the source code.
required:
- - mediaId
- MediaContentType:
- title: MediaContentType
- type: string
- enum:
- - image/png
- - image/jpeg
- - image/jpg
- - image/webp
- - image/gif
- - image/svg+xml
- - image/tiff
- - image/bmp
- - image/avif
- - image/heic
- - audio/mpeg
- - audio/mp3
- - audio/wav
- - audio/ogg
- - audio/oga
- - audio/aac
- - audio/mp4
- - audio/flac
- - audio/opus
- - audio/webm
- - video/mp4
- - video/webm
- - video/ogg
- - video/mpeg
- - video/quicktime
- - video/x-msvideo
- - video/x-matroska
- - text/plain
- - text/html
- - text/css
- - text/csv
- - text/markdown
- - text/x-python
- - application/javascript
- - text/x-typescript
- - application/x-yaml
- - application/pdf
- - application/msword
- - application/vnd.ms-excel
- - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
- - application/zip
- - application/json
- - application/xml
- - application/octet-stream
- - >-
- application/vnd.openxmlformats-officedocument.wordprocessingml.document
- - >-
- application/vnd.openxmlformats-officedocument.presentationml.presentation
- - application/rtf
- - application/x-ndjson
- - application/vnd.apache.parquet
- - application/gzip
- - application/x-tar
- - application/x-7z-compressed
- description: The MIME type of the media record
- MetricsV2Response:
- title: MetricsV2Response
+ - type
+ - sourceCode
+ - sourceCodeLanguage
+ allOf:
+ - $ref: '#/components/schemas/EvaluatorVersionBase'
+ EvaluatorVersion:
+ title: EvaluatorVersion
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - llm_as_judge
+ - $ref: '#/components/schemas/LlmAsJudgeEvaluatorVersion'
+ required:
+ - type
+ title: LlmAsJudgeEvaluatorVersion
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - code
+ - $ref: '#/components/schemas/CodeEvaluatorVersion'
+ required:
+ - type
+ title: CodeEvaluatorVersion
+ EvaluationRuleAssignment:
+ title: EvaluationRuleAssignment
type: object
+ description: Evaluation-rule assignment associated with this evaluator.
properties:
- data:
+ evaluationRuleId:
+ type: string
+ description: Stable identifier of the assigned evaluation rule.
+ variableMappingOverride:
type: array
items:
- type: object
- additionalProperties: true
+ $ref: '#/components/schemas/PromptVariableMapping'
+ nullable: true
description: >-
- The metrics data. Each item in the list contains the metric values
- and dimensions requested in the query.
-
- Format varies based on the query parameters.
-
- Histograms will return an array with [lower, upper, height] tuples.
- required:
- - data
- ModelTokenizerId:
- title: ModelTokenizerId
- type: string
- enum:
- - openai
- - claude
- description: Tokenizer supported by Langfuse for model usage inference.
- PaginatedModels:
- title: PaginatedModels
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Model'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
+ Rule-specific variable mapping override. Omitted when the
+ evaluator's latest default mapping is inherited. Legacy mappings use
+ the explicit `legacy` variant.
required:
- - data
- - meta
- CreateModelRequest:
- title: CreateModelRequest
+ - evaluationRuleId
+ EvaluatorBase:
+ title: EvaluatorBase
type: object
properties:
- modelName:
+ id:
+ type: string
+ description: Stable identifier of this evaluator across all versions.
+ name:
+ type: string
+ description: >-
+ Human-readable evaluator name. Names are not identifiers and do not
+ need to be unique.
+ description:
type: string
+ nullable: true
+ description: Optional human-readable evaluator description.
+ createdBy:
+ $ref: '#/components/schemas/Creator'
+ nullable: true
description: >-
- Name of the model definition. If multiple with the same name exist,
- they are applied in the following order: (1) custom over built-in,
- (2) newest according to startTime where
- model.startTime<observation.startTime
- matchPattern:
- type: string
+ User who created this evaluator, or `null` when no user can be
+ resolved.
+ status:
+ $ref: '#/components/schemas/EvaluatorStatus'
description: >-
- Regex pattern which matches this model definition to
- generation.model. Useful in case of fine-tuned models. If you want
- to exact match, use `(?i)^modelname$`
- startDate:
+ Effective evaluator status after Langfuse validates its runtime
+ configuration.
+ pausedAt:
type: string
format: date-time
nullable: true
- description: Apply only to generations which are newer than this ISO date.
- unit:
- $ref: '#/components/schemas/ModelUsageUnit'
- description: Unit used by this model.
- inputPrice:
- type: number
- format: double
+ description: Timestamp when the evaluator was paused, otherwise `null`.
+ pausedReason:
+ type: string
nullable: true
- description: >-
- Deprecated. Use 'pricingTiers' instead. Price (USD) per input unit.
- Creates a default tier if pricingTiers not provided.
- outputPrice:
- type: number
- format: double
+ description: Machine-readable reason when `status=paused`, otherwise `null`.
+ pausedMessage:
+ type: string
nullable: true
+ description: Human-readable explanation when `status=paused`, otherwise `null`.
+ evaluationRuleAssignments:
+ type: array
+ items:
+ $ref: '#/components/schemas/EvaluationRuleAssignment'
description: >-
- Deprecated. Use 'pricingTiers' instead. Price (USD) per output unit.
- Creates a default tier if pricingTiers not provided.
- totalPrice:
- type: number
- format: double
+ All modern and legacy evaluation-rule assignments in
+ newest-assignment-first order. Rule-specific mappings are exposed as
+ `variableMappingOverride`; inherited defaults are omitted.
+ createdAt:
+ type: string
+ format: date-time
+ description: Timestamp when the evaluator was created.
+ updatedAt:
+ type: string
+ format: date-time
+ description: Timestamp when the evaluator was last updated.
+ versionId:
+ type: string
+ description: Stable identifier of the latest evaluator version.
+ version:
+ type: integer
+ description: Monotonically increasing latest evaluator version number.
+ versionCreatedAt:
+ type: string
+ format: date-time
+ description: Timestamp when the latest evaluator version was created.
+ versionCreatedBy:
+ $ref: '#/components/schemas/Creator'
nullable: true
description: >-
- Deprecated. Use 'pricingTiers' instead. Price (USD) per total units.
- Cannot be set if input or output price is set. Creates a default
- tier if pricingTiers not provided.
- pricingTiers:
+ User who created the latest evaluator version, or `null` when no
+ user can be resolved.
+ required:
+ - id
+ - name
+ - description
+ - createdBy
+ - status
+ - pausedAt
+ - pausedReason
+ - pausedMessage
+ - evaluationRuleAssignments
+ - createdAt
+ - updatedAt
+ - versionId
+ - version
+ - versionCreatedAt
+ - versionCreatedBy
+ LlmAsJudgeEvaluator:
+ title: LlmAsJudgeEvaluator
+ type: object
+ properties:
+ type:
+ type: string
+ const: llm_as_judge
+ description: Evaluator type.
+ prompt:
+ $ref: '#/components/schemas/EvaluatorChatPrompt'
+ description: Ordered chat messages used by the latest evaluator version.
+ variables:
type: array
items:
- $ref: '#/components/schemas/PricingTierInput'
- nullable: true
+ type: string
description: >-
- Optional. Array of pricing tiers for this model.
-
-
- Use pricing tiers for all models - both those with threshold-based
- pricing variations and those with simple flat pricing:
-
-
- - For models with standard flat pricing: Create a single default
- tier with your prices
- (e.g., one tier with isDefault=true, priority=0, conditions=[], and your standard prices)
-
- - For models with threshold-based pricing: Create a default tier
- plus additional conditional tiers
- (e.g., default tier for standard usage + high-volume tier for usage above certain thresholds)
-
- Requirements:
-
- - Cannot be provided with flat prices
- (inputPrice/outputPrice/totalPrice) - use one approach or the other
-
- - Must include exactly one default tier with isDefault=true,
- priority=0, and conditions=[]
-
- - All tier names and priorities must be unique within the model
-
- - Each tier must define at least one price
-
-
- If omitted, you must provide flat prices instead
- (inputPrice/outputPrice/totalPrice),
-
- which will automatically create a single default tier named
- "Standard".
- tokenizerId:
- $ref: '#/components/schemas/ModelTokenizerId'
+ Variables extracted from the latest prompt and available for
+ evaluation-rule mappings.
+ variableMapping:
+ type: array
+ items:
+ $ref: '#/components/schemas/PromptVariableMappingRead'
nullable: true
description: >-
- Optional. Tokenizer to be applied to observations which match to
- this model. See docs for more details.
- tokenizerConfig:
+ Default variable mapping for the latest version, or `null` when no
+ default is configured.
+ modelConfig:
+ $ref: '#/components/schemas/EvaluatorModelConfig'
nullable: true
description: >-
- Optional. Configuration for the selected tokenizer. Needs to be
- JSON. See docs for more details.
+ Explicit model configuration for the latest version, or `null` when
+ the project's default evaluation model is used.
+ outputDefinition:
+ $ref: '#/components/schemas/PublicEvaluatorOutputDefinition'
+ description: Structured output schema returned by the latest evaluator version.
required:
- - modelName
- - matchPattern
- - unit
- ObservationsV2Response:
- title: ObservationsV2Response
+ - type
+ - prompt
+ - variables
+ - variableMapping
+ - modelConfig
+ - outputDefinition
+ allOf:
+ - $ref: '#/components/schemas/EvaluatorBase'
+ CodeEvaluator:
+ title: CodeEvaluator
type: object
+ properties:
+ type:
+ type: string
+ const: code
+ description: Evaluator type.
+ sourceCode:
+ type: string
+ description: Source code executed by the latest evaluator version.
+ sourceCodeLanguage:
+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
+ description: Runtime language used to execute the latest source code.
+ required:
+ - type
+ - sourceCode
+ - sourceCodeLanguage
+ allOf:
+ - $ref: '#/components/schemas/EvaluatorBase'
+ Evaluator:
+ title: Evaluator
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - llm_as_judge
+ - $ref: '#/components/schemas/LlmAsJudgeEvaluator'
+ required:
+ - type
+ title: LlmAsJudgeEvaluator
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - code
+ - $ref: '#/components/schemas/CodeEvaluator'
+ required:
+ - type
+ title: CodeEvaluator
description: >-
- Response containing observations with field-group-based filtering and
- cursor-based pagination.
-
+ One evaluator that can be used for scoring.
- The `data` array contains observation objects with only the requested
- field groups included.
- Use the `cursor` in `meta` to retrieve the next page of results.
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/ObservationV2'
- description: >-
- Array of observation objects. Fields included depend on the `fields`
- parameter in the request.
- meta:
- $ref: '#/components/schemas/ObservationsV2Meta'
- required:
- - data
- - meta
- ObservationsV2Meta:
- title: ObservationsV2Meta
+ An evaluator describes **how** to score data. Associated evaluation
+ rules describe **which** live objects should be evaluated.
+ EvaluatorStatus:
+ title: EvaluatorStatus
+ type: string
+ enum:
+ - active
+ - paused
+ description: >-
+ Effective evaluator runtime status.
+
+
+ - `active`: the evaluator can run.
+
+ - `paused`: Langfuse paused execution until the underlying issue is
+ resolved.
+ CreateLlmAsJudgeEvaluatorRequest:
+ title: CreateLlmAsJudgeEvaluatorRequest
type: object
- description: Metadata for cursor-based pagination
properties:
- cursor:
+ name:
type: string
- nullable: true
description: >-
- Base64-encoded cursor to use for retrieving the next page. If not
- present, there are no more results.
- OtelResourceSpan:
- title: OtelResourceSpan
- type: object
- description: >-
- Represents a collection of spans from a single resource as per OTLP
- specification
- properties:
- resource:
- $ref: '#/components/schemas/OtelResource'
+ Human-readable evaluator name. Names are not identifiers and do not
+ need to be unique.
+ description:
+ type: string
nullable: true
- description: Resource information
- scopeSpans:
+ description: Optional human-readable evaluator description.
+ type:
+ type: string
+ const: llm_as_judge
+ description: Evaluator type.
+ prompt:
+ $ref: '#/components/schemas/EvaluatorChatPromptInput'
+ description: User prompt string shortcut or an ordered list of chat messages.
+ modelConfig:
+ $ref: '#/components/schemas/EvaluatorModelConfig'
+ nullable: true
+ description: >-
+ Explicit model configuration. Set to `null` or omit to use the
+ project's default evaluation model.
+ variableMapping:
type: array
items:
- $ref: '#/components/schemas/OtelScopeSpan'
+ $ref: '#/components/schemas/PromptVariableMappingInput'
nullable: true
- description: Array of scope spans
- OtelResource:
- title: OtelResource
+ description: >-
+ Default prompt-variable mapping, or `null` when no default is
+ configured.
+ outputDefinition:
+ $ref: '#/components/schemas/EvaluatorOutputDefinition'
+ description: >-
+ Structured output schema returned by this evaluator. Do not include
+ an internal persistence version.
+ required:
+ - name
+ - type
+ - prompt
+ - outputDefinition
+ CreateCodeEvaluatorRequest:
+ title: CreateCodeEvaluatorRequest
type: object
- description: Resource attributes identifying the source of telemetry
properties:
- attributes:
- type: array
- items:
- $ref: '#/components/schemas/OtelAttribute'
+ name:
+ type: string
+ description: >-
+ Human-readable evaluator name. Names are not identifiers and do not
+ need to be unique.
+ description:
+ type: string
nullable: true
- description: Resource attributes like service.name, service.version, etc.
- OtelScopeSpan:
- title: OtelScopeSpan
+ description: Optional human-readable evaluator description.
+ type:
+ type: string
+ const: code
+ description: Evaluator type.
+ sourceCode:
+ type: string
+ description: Source code executed for each matched observation.
+ sourceCodeLanguage:
+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
+ description: Runtime language used to execute the source code.
+ required:
+ - name
+ - type
+ - sourceCode
+ - sourceCodeLanguage
+ CreateEvaluatorRequest:
+ title: CreateEvaluatorRequest
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - llm_as_judge
+ - $ref: '#/components/schemas/CreateLlmAsJudgeEvaluatorRequest'
+ required:
+ - type
+ title: CreateLlmAsJudgeEvaluatorRequest
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - code
+ - $ref: '#/components/schemas/CreateCodeEvaluatorRequest'
+ required:
+ - type
+ title: CreateCodeEvaluatorRequest
+ description: >-
+ Initial evaluator definition with metadata and definition fields at the
+ same level. The returned evaluator starts at version `1`.
+ UpdateEvaluatorMetadataRequest:
+ title: UpdateEvaluatorMetadataRequest
type: object
- description: Collection of spans from a single instrumentation scope
+ description: Metadata-only update. At least one field must be provided.
properties:
- scope:
- $ref: '#/components/schemas/OtelScope'
+ name:
+ type: string
nullable: true
- description: Instrumentation scope information
- spans:
- type: array
- items:
- $ref: '#/components/schemas/OtelSpan'
+ description: New human-readable evaluator name.
+ description:
+ type: string
nullable: true
- description: Array of spans
- OtelScope:
- title: OtelScope
+ description: New description. Set to `null` to clear it.
+ UpdateLlmAsJudgeEvaluatorRequest:
+ title: UpdateLlmAsJudgeEvaluatorRequest
type: object
- description: Instrumentation scope information
properties:
name:
type: string
nullable: true
- description: Instrumentation scope name
- version:
+ description: New human-readable evaluator name.
+ description:
type: string
nullable: true
- description: Instrumentation scope version
- attributes:
+ description: New description. Set to `null` to clear it.
+ type:
+ type: string
+ const: llm_as_judge
+ description: Evaluator type. The type of an existing evaluator cannot change.
+ prompt:
+ $ref: '#/components/schemas/EvaluatorChatPromptInput'
+ description: >-
+ Complete replacement user prompt string or an ordered list of chat
+ messages.
+ modelConfig:
+ $ref: '#/components/schemas/EvaluatorModelConfig'
+ nullable: true
+ description: >-
+ Explicit model configuration. Set to `null` or omit to use the
+ project's default evaluation model.
+ variableMapping:
type: array
items:
- $ref: '#/components/schemas/OtelAttribute'
+ $ref: '#/components/schemas/PromptVariableMappingInput'
nullable: true
- description: Additional scope attributes
- OtelSpan:
- title: OtelSpan
+ description: >-
+ Complete replacement default variable mapping, or `null` when no
+ default is configured.
+ outputDefinition:
+ $ref: '#/components/schemas/EvaluatorOutputDefinition'
+ description: >-
+ Complete replacement output schema. Do not include an internal
+ persistence version.
+ required:
+ - type
+ - prompt
+ - outputDefinition
+ UpdateCodeEvaluatorRequest:
+ title: UpdateCodeEvaluatorRequest
type: object
- description: Individual span representing a unit of work or operation
properties:
- traceId:
- nullable: true
- description: Trace ID (16 bytes, hex-encoded string in JSON or Buffer in binary)
- spanId:
- nullable: true
- description: Span ID (8 bytes, hex-encoded string in JSON or Buffer in binary)
- parentSpanId:
- nullable: true
- description: Parent span ID if this is a child span
name:
type: string
nullable: true
- description: Span name describing the operation
- kind:
- type: integer
- nullable: true
- description: Span kind (1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER)
- startTimeUnixNano:
- nullable: true
- description: Start time in nanoseconds since Unix epoch
- endTimeUnixNano:
+ description: New human-readable evaluator name.
+ description:
+ type: string
nullable: true
- description: End time in nanoseconds since Unix epoch
- attributes:
+ description: New description. Set to `null` to clear it.
+ type:
+ type: string
+ const: code
+ description: Evaluator type. The type of an existing evaluator cannot change.
+ sourceCode:
+ type: string
+ description: Complete replacement source code.
+ sourceCodeLanguage:
+ $ref: '#/components/schemas/CodeEvaluatorSourceCodeLanguage'
+ description: Runtime language used to execute the source code.
+ required:
+ - type
+ - sourceCode
+ - sourceCodeLanguage
+ UpdateEvaluatorRequest:
+ title: UpdateEvaluatorRequest
+ oneOf:
+ - $ref: '#/components/schemas/UpdateEvaluatorMetadataRequest'
+ - $ref: '#/components/schemas/UpdateLlmAsJudgeEvaluatorRequest'
+ - $ref: '#/components/schemas/UpdateCodeEvaluatorRequest'
+ description: >-
+ Metadata-only update or complete flattened definition replacement. At
+ least one field must be provided.
+ EvaluatorsPage:
+ title: EvaluatorsPage
+ type: object
+ properties:
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/Evaluator'
+ description: Evaluators for this page.
+ meta:
+ $ref: '#/components/schemas/CursorMeta'
+ description: Cursor pagination metadata.
+ required:
+ - data
+ - meta
+ EvaluatorVersionsPage:
+ title: EvaluatorVersionsPage
+ type: object
+ properties:
+ data:
type: array
items:
- $ref: '#/components/schemas/OtelAttribute'
- nullable: true
- description: >-
- Span attributes including Langfuse-specific attributes
- (langfuse.observation.*)
- status:
- nullable: true
- description: Span status object
- OtelAttribute:
- title: OtelAttribute
+ $ref: '#/components/schemas/EvaluatorVersion'
+ description: Evaluator versions for this page in newest-first order.
+ meta:
+ $ref: '#/components/schemas/CursorMeta'
+ description: Cursor pagination metadata.
+ required:
+ - data
+ - meta
+ CursorMeta:
+ title: CursorMeta
type: object
- description: Key-value attribute pair for resources, scopes, or spans
properties:
- key:
+ cursor:
type: string
nullable: true
- description: Attribute key (e.g., "service.name", "langfuse.observation.type")
- value:
- $ref: '#/components/schemas/OtelAttributeValue'
- nullable: true
- description: Attribute value
- OtelAttributeValue:
- title: OtelAttributeValue
+ description: Opaque cursor for the next page. Omitted when there is no next page.
+ DeletedEvaluator:
+ title: DeletedEvaluator
type: object
- description: Attribute value wrapper supporting different value types
+ description: Confirmation returned after successful evaluator deletion.
properties:
- stringValue:
+ id:
type: string
- nullable: true
- description: String value
- intValue:
- type: integer
- nullable: true
- description: Integer value
- doubleValue:
- type: number
- format: double
- nullable: true
- description: Double value
- boolValue:
- type: boolean
- nullable: true
- description: Boolean value
- OtelTraceResponse:
- title: OtelTraceResponse
- type: object
- description: Response from trace export request. Empty object indicates success.
- properties: {}
- MembershipRole:
- title: MembershipRole
- type: string
- enum:
- - OWNER
- - ADMIN
- - MEMBER
- - VIEWER
- MembershipRequest:
- title: MembershipRequest
+ description: Identifier of the deleted evaluator.
+ required:
+ - id
+ ExperimentsResponse:
+ title: ExperimentsResponse
type: object
properties:
- userId:
- type: string
- role:
- $ref: '#/components/schemas/MembershipRole'
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/Experiment'
+ meta:
+ $ref: '#/components/schemas/ExperimentsResponseMeta'
required:
- - userId
- - role
- DeleteMembershipRequest:
- title: DeleteMembershipRequest
+ - data
+ - meta
+ ExperimentsResponseMeta:
+ title: ExperimentsResponseMeta
type: object
properties:
- userId:
+ cursor:
type: string
- required:
- - userId
- MembershipResponse:
- title: MembershipResponse
+ nullable: true
+ description: >-
+ Versioned base64url cursor for retrieving the next page. Absent when
+ there are no more results.
+ Experiment:
+ title: Experiment
type: object
properties:
- userId:
- type: string
- role:
- $ref: '#/components/schemas/MembershipRole'
- email:
+ id:
type: string
name:
type: string
- required:
- - userId
- - role
- - email
- - name
- MembershipDeletionResponse:
- title: MembershipDeletionResponse
- type: object
- properties:
- message:
+ description:
type: string
- userId:
+ nullable: true
+ startTime:
+ type: string
+ format: date-time
+ description: |-
+ Start of the experiment, i.e. the earliest event within the
+ requested time range. Clipped to `fromStartTime` when the
+ experiment started before the requested range.
+ endTime:
+ type: string
+ format: date-time
+ description: |-
+ End of the experiment, i.e. the latest event end within the
+ requested time range.
+ itemCount:
+ type: integer
+ description: Number of experiment items within the requested time range.
+ datasetId:
type: string
+ nullable: true
+ description: Null when the experiment is not associated with a dataset.
+ metadata:
+ type: object
+ additionalProperties: true
+ nullable: true
+ description: Included only when `fields=metadata` is requested.
+ scores:
+ type: array
+ items:
+ $ref: '#/components/schemas/ScoreV3'
+ nullable: true
+ description: >-
+ Included only when `fields=scores` is requested. Contains scores
+ directly attached to the experiment.
required:
- - message
- - userId
- MembershipsResponse:
- title: MembershipsResponse
+ - id
+ - name
+ - description
+ - startTime
+ - endTime
+ - itemCount
+ - datasetId
+ ExperimentItemsResponse:
+ title: ExperimentItemsResponse
type: object
properties:
- memberships:
+ data:
type: array
items:
- $ref: '#/components/schemas/MembershipResponse'
+ $ref: '#/components/schemas/ExperimentItem'
+ meta:
+ $ref: '#/components/schemas/ExperimentsResponseMeta'
required:
- - memberships
- OrganizationProject:
- title: OrganizationProject
+ - data
+ - meta
+ ExperimentItem:
+ title: ExperimentItem
type: object
properties:
id:
type: string
- name:
+ traceId:
+ type: string
+ startTime:
+ type: string
+ format: date-time
+ endTime:
+ type: string
+ format: date-time
+ nullable: true
+ level:
+ $ref: '#/components/schemas/ObservationLevel'
+ environment:
+ type: string
+ experimentId:
+ type: string
+ experimentName:
+ type: string
+ experimentItemId:
+ type: string
+ experimentDatasetId:
+ type: string
+ nullable: true
+ description: Included when `fields=dataset` is requested.
+ experimentItemVersion:
type: string
+ format: date-time
+ nullable: true
+ description: Included when `fields=dataset` is requested.
+ input:
+ nullable: true
+ description: Included when `fields=io` is requested.
+ output:
+ nullable: true
+ description: Included when `fields=io` is requested.
+ expectedOutput:
+ nullable: true
+ description: Included when `fields=io` is requested.
metadata:
type: object
additionalProperties: true
nullable: true
- createdAt:
- type: string
- format: date-time
- updatedAt:
+ description: Included when `fields=metadata` is requested.
+ experimentItemMetadata:
+ type: object
+ additionalProperties: true
+ nullable: true
+ description: Included when `fields=itemMetadata` is requested.
+ experimentMetadata:
+ type: object
+ additionalProperties: true
+ nullable: true
+ description: Included when `fields=experimentMetadata` is requested.
+ experimentDescription:
type: string
- format: date-time
- required:
- - id
- - name
- - createdAt
- - updatedAt
- OrganizationProjectsResponse:
- title: OrganizationProjectsResponse
- type: object
- properties:
- projects:
+ nullable: true
+ description: Included when `fields=experimentMetadata` is requested.
+ scores:
type: array
items:
- $ref: '#/components/schemas/OrganizationProject'
+ $ref: '#/components/schemas/ScoreV3'
+ nullable: true
+ description: >-
+ Included only when `fields=scores` is requested. Contains item and
+ trace scores only; experiment-level scores are returned by the
+ experiments endpoint.
required:
- - projects
- OrganizationApiKey:
- title: OrganizationApiKey
+ - id
+ - traceId
+ - startTime
+ - endTime
+ - level
+ - environment
+ - experimentId
+ - experimentName
+ - experimentItemId
+ FeedbackTargetType:
+ title: FeedbackTargetType
+ type: string
+ enum:
+ - skill
+ - mcp-tool
+ - cli
+ - docs
+ - public-api
+ - other
+ SubmitFeedbackRequest:
+ title: SubmitFeedbackRequest
type: object
properties:
- id:
- type: string
- createdAt:
+ targetType:
+ $ref: '#/components/schemas/FeedbackTargetType'
+ description: Category of the thing the feedback is about.
+ target:
type: string
- format: date-time
- expiresAt:
+ description: >-
+ The specific instance within targetType: the skill name, MCP tool
+ name, CLI command, API endpoint path, or docs page path (e.g.
+ 'queryMetrics', '/docs/mcp'). An identifier, not a sentence. Must be
+ between 1 and 200 characters.
+ feedback:
type: string
- format: date-time
- nullable: true
- lastUsedAt:
+ description: >-
+ Concise feedback text approved by the user. Must be between 1 and
+ 3000 characters.
+ goal:
type: string
- format: date-time
nullable: true
- note:
+ description: >-
+ Optional user-approved goal or use case they were trying to achieve.
+ Must be between 1 and 1500 characters when provided. Do not include
+ secrets, customer data, trace payloads, or broad unrelated context.
+ referenceUrl:
type: string
nullable: true
- publicKey:
- type: string
- displaySecretKey:
+ description: >-
+ Optional HTTP(S) reference URL. Langfuse stores it as text for
+ triage and does not fetch it.
+ required:
+ - targetType
+ - target
+ - feedback
+ SubmitFeedbackResponse:
+ title: SubmitFeedbackResponse
+ type: object
+ properties:
+ id:
type: string
+ description: Correlation ID for the submitted feedback.
required:
- id
- - createdAt
- - publicKey
- - displaySecretKey
- OrganizationApiKeysResponse:
- title: OrganizationApiKeysResponse
+ HealthResponse:
+ title: HealthResponse
type: object
properties:
- apiKeys:
- type: array
- items:
- $ref: '#/components/schemas/OrganizationApiKey'
+ version:
+ type: string
+ description: Langfuse server version
+ status:
+ type: string
required:
- - apiKeys
- Projects:
- title: Projects
+ - version
+ - status
+ IngestionEvent:
+ title: IngestionEvent
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - trace-create
+ - $ref: '#/components/schemas/TraceEvent'
+ required:
+ - type
+ title: TraceEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - score-create
+ - $ref: '#/components/schemas/ScoreEvent'
+ required:
+ - type
+ title: ScoreEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - span-create
+ - $ref: '#/components/schemas/CreateSpanEvent'
+ required:
+ - type
+ title: CreateSpanEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - span-update
+ - $ref: '#/components/schemas/UpdateSpanEvent'
+ required:
+ - type
+ title: UpdateSpanEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - generation-create
+ - $ref: '#/components/schemas/CreateGenerationEvent'
+ required:
+ - type
+ title: CreateGenerationEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - generation-update
+ - $ref: '#/components/schemas/UpdateGenerationEvent'
+ required:
+ - type
+ title: UpdateGenerationEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - event-create
+ - $ref: '#/components/schemas/CreateEventEvent'
+ required:
+ - type
+ title: CreateEventEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - sdk-log
+ - $ref: '#/components/schemas/SDKLogEvent'
+ required:
+ - type
+ title: SDKLogEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - observation-create
+ - $ref: '#/components/schemas/CreateObservationEvent'
+ required:
+ - type
+ title: CreateObservationEvent
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - observation-update
+ - $ref: '#/components/schemas/UpdateObservationEvent'
+ required:
+ - type
+ title: UpdateObservationEvent
+ ObservationType:
+ title: ObservationType
+ type: string
+ enum:
+ - SPAN
+ - GENERATION
+ - EVENT
+ - AGENT
+ - TOOL
+ - CHAIN
+ - RETRIEVER
+ - EVALUATOR
+ - EMBEDDING
+ - GUARDRAIL
+ IngestionUsage:
+ title: IngestionUsage
+ oneOf:
+ - $ref: '#/components/schemas/Usage'
+ - $ref: '#/components/schemas/OpenAIUsage'
+ OpenAIUsage:
+ title: OpenAIUsage
type: object
+ description: Usage interface of OpenAI for improved compatibility.
properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Project'
- required:
- - data
- Organization:
- title: Organization
+ promptTokens:
+ type: integer
+ nullable: true
+ completionTokens:
+ type: integer
+ nullable: true
+ totalTokens:
+ type: integer
+ nullable: true
+ OptionalObservationBody:
+ title: OptionalObservationBody
type: object
properties:
- id:
+ traceId:
type: string
- description: The unique identifier of the organization
+ nullable: true
name:
type: string
- description: The name of the organization
- required:
- - id
- - name
- Project:
- title: Project
- type: object
- properties:
- id:
- type: string
- name:
+ nullable: true
+ startTime:
type: string
- organization:
- $ref: '#/components/schemas/Organization'
- description: The organization this project belongs to
+ format: date-time
+ nullable: true
metadata:
- type: object
- additionalProperties: true
- description: Metadata for the project
- retentionDays:
- type: integer
nullable: true
- description: >-
- Number of days to retain data. Null or 0 means no retention. Omitted
- if no retention is configured.
- required:
- - id
- - name
- - organization
- - metadata
- ProjectDeletionResponse:
- title: ProjectDeletionResponse
+ input:
+ nullable: true
+ output:
+ nullable: true
+ level:
+ $ref: '#/components/schemas/ObservationLevel'
+ nullable: true
+ statusMessage:
+ type: string
+ nullable: true
+ parentObservationId:
+ type: string
+ nullable: true
+ version:
+ type: string
+ nullable: true
+ environment:
+ type: string
+ nullable: true
+ CreateEventBody:
+ title: CreateEventBody
type: object
properties:
- success:
- type: boolean
- message:
+ id:
type: string
- required:
- - success
- - message
- ApiKeyList:
- title: ApiKeyList
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/OptionalObservationBody'
+ UpdateEventBody:
+ title: UpdateEventBody
type: object
- description: List of API keys for a project
properties:
- apiKeys:
- type: array
- items:
- $ref: '#/components/schemas/ApiKeySummary'
+ id:
+ type: string
required:
- - apiKeys
- ApiKeySummary:
- title: ApiKeySummary
+ - id
+ allOf:
+ - $ref: '#/components/schemas/OptionalObservationBody'
+ CreateSpanBody:
+ title: CreateSpanBody
type: object
- description: Summary of an API key
properties:
- id:
- type: string
- createdAt:
+ endTime:
type: string
format: date-time
- expiresAt:
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/CreateEventBody'
+ UpdateSpanBody:
+ title: UpdateSpanBody
+ type: object
+ properties:
+ endTime:
type: string
format: date-time
nullable: true
- lastUsedAt:
+ allOf:
+ - $ref: '#/components/schemas/UpdateEventBody'
+ CreateGenerationBody:
+ title: CreateGenerationBody
+ type: object
+ properties:
+ completionStartTime:
type: string
format: date-time
nullable: true
- note:
+ model:
type: string
nullable: true
- publicKey:
- type: string
- displaySecretKey:
+ modelParameters:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/MapValue'
+ nullable: true
+ usage:
+ $ref: '#/components/schemas/IngestionUsage'
+ nullable: true
+ usageDetails:
+ $ref: '#/components/schemas/UsageDetails'
+ nullable: true
+ costDetails:
+ type: object
+ additionalProperties:
+ type: number
+ format: double
+ nullable: true
+ promptName:
type: string
- required:
- - id
- - createdAt
- - publicKey
- - displaySecretKey
- ApiKeyResponse:
- title: ApiKeyResponse
+ nullable: true
+ promptVersion:
+ type: integer
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/CreateSpanBody'
+ UpdateGenerationBody:
+ title: UpdateGenerationBody
type: object
- description: Response for API key creation
properties:
- id:
- type: string
- createdAt:
+ completionStartTime:
type: string
format: date-time
- publicKey:
- type: string
- secretKey:
- type: string
- displaySecretKey:
- type: string
- note:
- type: string
nullable: true
- required:
- - id
- - createdAt
- - publicKey
- - secretKey
- - displaySecretKey
- ApiKeyDeletionResponse:
- title: ApiKeyDeletionResponse
- type: object
- description: Response for API key deletion
- properties:
- success:
- type: boolean
- required:
- - success
- PromptMetaListResponse:
- title: PromptMetaListResponse
- type: object
- properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/PromptMeta'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- required:
- - data
- - meta
- PromptMeta:
- title: PromptMeta
- type: object
- properties:
- name:
+ model:
type: string
- type:
- $ref: '#/components/schemas/PromptType'
- description: Indicates whether the prompt is a text or chat prompt.
- versions:
- type: array
- items:
- type: integer
- labels:
- type: array
- items:
- type: string
- tags:
- type: array
- items:
- type: string
- lastUpdatedAt:
+ nullable: true
+ modelParameters:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/MapValue'
+ nullable: true
+ usage:
+ $ref: '#/components/schemas/IngestionUsage'
+ nullable: true
+ promptName:
type: string
- format: date-time
- lastConfig:
- description: >-
- Config object of the most recent prompt version that matches the
- filters (if any are provided)
- required:
- - name
- - type
- - versions
- - labels
- - tags
- - lastUpdatedAt
- - lastConfig
- CreatePromptRequest:
- title: CreatePromptRequest
- oneOf:
- - $ref: '#/components/schemas/CreateChatPromptRequest'
- - $ref: '#/components/schemas/CreateTextPromptRequest'
- CreateChatPromptRequest:
- title: CreateChatPromptRequest
+ nullable: true
+ usageDetails:
+ $ref: '#/components/schemas/UsageDetails'
+ nullable: true
+ costDetails:
+ type: object
+ additionalProperties:
+ type: number
+ format: double
+ nullable: true
+ promptVersion:
+ type: integer
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/UpdateSpanBody'
+ ObservationBody:
+ title: ObservationBody
type: object
properties:
- name:
+ id:
+ type: string
+ nullable: true
+ traceId:
type: string
- prompt:
- type: array
- items:
- $ref: '#/components/schemas/ChatMessageWithPlaceholders'
- config:
nullable: true
type:
- $ref: '#/components/schemas/CreateChatPromptType'
- labels:
- type: array
- items:
- type: string
+ $ref: '#/components/schemas/ObservationType'
+ name:
+ type: string
nullable: true
- description: List of deployment labels of this prompt version.
- tags:
- type: array
- items:
- type: string
+ startTime:
+ type: string
+ format: date-time
nullable: true
- description: List of tags to apply to all versions of this prompt.
- commitMessage:
+ endTime:
+ type: string
+ format: date-time
+ nullable: true
+ completionStartTime:
+ type: string
+ format: date-time
+ nullable: true
+ model:
+ type: string
+ nullable: true
+ modelParameters:
+ type: object
+ additionalProperties:
+ $ref: '#/components/schemas/MapValue'
+ nullable: true
+ input:
+ nullable: true
+ version:
+ type: string
+ nullable: true
+ metadata:
+ nullable: true
+ output:
+ nullable: true
+ usage:
+ $ref: '#/components/schemas/Usage'
+ nullable: true
+ level:
+ $ref: '#/components/schemas/ObservationLevel'
+ nullable: true
+ statusMessage:
+ type: string
+ nullable: true
+ parentObservationId:
+ type: string
+ nullable: true
+ environment:
type: string
nullable: true
- description: Commit message for this prompt version.
required:
- - name
- - prompt
- type
- CreateTextPromptRequest:
- title: CreateTextPromptRequest
+ TraceBody:
+ title: TraceBody
type: object
properties:
+ id:
+ type: string
+ nullable: true
+ timestamp:
+ type: string
+ format: date-time
+ nullable: true
name:
type: string
- prompt:
+ nullable: true
+ userId:
type: string
- config:
nullable: true
- type:
- $ref: '#/components/schemas/CreateTextPromptType'
+ input:
nullable: true
- labels:
- type: array
- items:
- type: string
+ output:
+ nullable: true
+ sessionId:
+ type: string
+ nullable: true
+ release:
+ type: string
+ nullable: true
+ version:
+ type: string
+ nullable: true
+ metadata:
nullable: true
- description: List of deployment labels of this prompt version.
tags:
type: array
items:
type: string
nullable: true
- description: List of tags to apply to all versions of this prompt.
- commitMessage:
+ environment:
type: string
nullable: true
- description: Commit message for this prompt version.
+ public:
+ type: boolean
+ nullable: true
+ description: Make trace publicly accessible via url
+ SDKLogBody:
+ title: SDKLogBody
+ type: object
+ properties:
+ log: {}
required:
- - name
- - prompt
- Prompt:
- title: Prompt
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - chat
- - $ref: '#/components/schemas/ChatPrompt'
- required:
- - type
- title: ChatPrompt
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - text
- - $ref: '#/components/schemas/TextPrompt'
- required:
- - type
- title: TextPrompt
- PromptType:
- title: PromptType
- type: string
- enum:
- - chat
- - text
- BasePrompt:
- title: BasePrompt
+ - log
+ ScoreBody:
+ title: ScoreBody
type: object
properties:
+ id:
+ type: string
+ nullable: true
+ traceId:
+ type: string
+ nullable: true
+ sessionId:
+ type: string
+ nullable: true
+ observationId:
+ type: string
+ nullable: true
+ datasetRunId:
+ type: string
+ nullable: true
name:
type: string
- version:
- type: integer
- config: {}
- labels:
- type: array
- items:
- type: string
- description: List of deployment labels of this prompt version.
- tags:
- type: array
- items:
- type: string
description: >-
- List of tags. Used to filter via UI and API. The same across
- versions of a prompt.
- commitMessage:
+ The name of the score. Always overrides "output" for correction
+ scores.
+ environment:
type: string
nullable: true
- description: Commit message for this prompt version.
- resolutionGraph:
- type: object
- additionalProperties: true
+ queueId:
+ type: string
nullable: true
description: >-
- The dependency resolution graph for the current prompt. Null if the
- prompt has no dependencies or if `resolve=false` was used.
- required:
- - name
- - version
- - config
- - labels
- - tags
- ChatMessageWithPlaceholders:
- title: ChatMessageWithPlaceholders
- oneOf:
- - $ref: '#/components/schemas/ChatMessage'
- - $ref: '#/components/schemas/PlaceholderMessage'
- ChatMessage:
- title: ChatMessage
- type: object
- properties:
- role:
+ The annotation queue referenced by the score. Indicates if score was
+ initially created while processing annotation queue.
+ value:
+ $ref: '#/components/schemas/CreateScoreValue'
+ description: >-
+ The value of the score. Must be passed as string for categorical and
+ text scores, and numeric for boolean and numeric scores. Boolean
+ score values must equal either 1 or 0 (true or false). Text score
+ values must be between 1 and 500 characters.
+ comment:
type: string
- content:
+ nullable: true
+ metadata:
+ nullable: true
+ dataType:
+ $ref: '#/components/schemas/ScoreDataType'
+ nullable: true
+ description: >-
+ When set, must match the score value's type. If not set, will be
+ inferred from the score value or config
+ configId:
type: string
- type:
- $ref: '#/components/schemas/ChatMessageType'
nullable: true
+ description: >-
+ Reference a score config on a score. When set, the score name must
+ equal the config name and scores must comply with the config's range
+ and data type. For categorical scores, the value must map to a
+ config category. Numeric scores might be constrained by the score
+ config's max and min values
required:
- - role
- - content
- ChatMessageType:
- title: ChatMessageType
- type: string
- enum:
- - chatmessage
- PlaceholderMessage:
- title: PlaceholderMessage
+ - name
+ - value
+ BaseEvent:
+ title: BaseEvent
type: object
properties:
- name:
+ id:
type: string
- type:
- $ref: '#/components/schemas/PlaceholderMessageType'
+ description: UUID v4 that identifies the event
+ timestamp:
+ type: string
+ description: >-
+ Datetime (ISO 8601) of event creation in client. Should be as close
+ to actual event creation in client as possible, this timestamp will
+ be used for ordering of events in future release. Resolution:
+ milliseconds (required), microseconds (optimal).
+ metadata:
nullable: true
+ description: Optional. Metadata field used by the Langfuse SDKs for debugging.
required:
- - name
- PlaceholderMessageType:
- title: PlaceholderMessageType
- type: string
- enum:
- - placeholder
- TextPrompt:
- title: TextPrompt
+ - id
+ - timestamp
+ TraceEvent:
+ title: TraceEvent
type: object
properties:
- prompt:
- type: string
+ body:
+ $ref: '#/components/schemas/TraceBody'
required:
- - prompt
+ - body
allOf:
- - $ref: '#/components/schemas/BasePrompt'
- ChatPrompt:
- title: ChatPrompt
+ - $ref: '#/components/schemas/BaseEvent'
+ CreateObservationEvent:
+ title: CreateObservationEvent
type: object
properties:
- prompt:
- type: array
- items:
- $ref: '#/components/schemas/ChatMessageWithPlaceholders'
+ body:
+ $ref: '#/components/schemas/ObservationBody'
required:
- - prompt
+ - body
allOf:
- - $ref: '#/components/schemas/BasePrompt'
- CreateChatPromptType:
- title: CreateChatPromptType
- type: string
- enum:
- - chat
- CreateTextPromptType:
- title: CreateTextPromptType
- type: string
- enum:
- - text
- ServiceProviderConfig:
- title: ServiceProviderConfig
+ - $ref: '#/components/schemas/BaseEvent'
+ UpdateObservationEvent:
+ title: UpdateObservationEvent
type: object
properties:
- schemas:
- type: array
- items:
- type: string
- documentationUri:
- type: string
- patch:
- $ref: '#/components/schemas/ScimFeatureSupport'
- bulk:
- $ref: '#/components/schemas/BulkConfig'
- filter:
- $ref: '#/components/schemas/FilterConfig'
- changePassword:
- $ref: '#/components/schemas/ScimFeatureSupport'
- sort:
- $ref: '#/components/schemas/ScimFeatureSupport'
- etag:
- $ref: '#/components/schemas/ScimFeatureSupport'
- authenticationSchemes:
- type: array
- items:
- $ref: '#/components/schemas/AuthenticationScheme'
- meta:
- $ref: '#/components/schemas/ResourceMeta'
+ body:
+ $ref: '#/components/schemas/ObservationBody'
required:
- - schemas
- - documentationUri
- - patch
- - bulk
- - filter
- - changePassword
- - sort
- - etag
- - authenticationSchemes
- - meta
- ScimFeatureSupport:
- title: ScimFeatureSupport
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ ScoreEvent:
+ title: ScoreEvent
type: object
properties:
- supported:
- type: boolean
+ body:
+ $ref: '#/components/schemas/ScoreBody'
required:
- - supported
- BulkConfig:
- title: BulkConfig
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ SDKLogEvent:
+ title: SDKLogEvent
type: object
properties:
- supported:
- type: boolean
- maxOperations:
- type: integer
- maxPayloadSize:
- type: integer
+ body:
+ $ref: '#/components/schemas/SDKLogBody'
required:
- - supported
- - maxOperations
- - maxPayloadSize
- FilterConfig:
- title: FilterConfig
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ CreateGenerationEvent:
+ title: CreateGenerationEvent
type: object
properties:
- supported:
- type: boolean
- maxResults:
- type: integer
+ body:
+ $ref: '#/components/schemas/CreateGenerationBody'
required:
- - supported
- - maxResults
- ResourceMeta:
- title: ResourceMeta
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ UpdateGenerationEvent:
+ title: UpdateGenerationEvent
type: object
properties:
- resourceType:
- type: string
- location:
- type: string
+ body:
+ $ref: '#/components/schemas/UpdateGenerationBody'
required:
- - resourceType
- - location
- AuthenticationScheme:
- title: AuthenticationScheme
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ CreateSpanEvent:
+ title: CreateSpanEvent
type: object
properties:
- name:
- type: string
- description:
- type: string
- specUri:
- type: string
- type:
- type: string
- primary:
- type: boolean
+ body:
+ $ref: '#/components/schemas/CreateSpanBody'
required:
- - name
- - description
- - specUri
- - type
- - primary
- ResourceTypesResponse:
- title: ResourceTypesResponse
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ UpdateSpanEvent:
+ title: UpdateSpanEvent
type: object
properties:
- schemas:
- type: array
- items:
- type: string
- totalResults:
- type: integer
- Resources:
- type: array
- items:
- $ref: '#/components/schemas/ResourceType'
+ body:
+ $ref: '#/components/schemas/UpdateSpanBody'
required:
- - schemas
- - totalResults
- - Resources
- ResourceType:
- title: ResourceType
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ CreateEventEvent:
+ title: CreateEventEvent
+ type: object
+ properties:
+ body:
+ $ref: '#/components/schemas/CreateEventBody'
+ required:
+ - body
+ allOf:
+ - $ref: '#/components/schemas/BaseEvent'
+ IngestionSuccess:
+ title: IngestionSuccess
type: object
properties:
- schemas:
- type: array
- items:
- type: string
- nullable: true
id:
type: string
- name:
- type: string
- endpoint:
- type: string
- description:
- type: string
- schema:
- type: string
- schemaExtensions:
- type: array
- items:
- $ref: '#/components/schemas/SchemaExtension'
- meta:
- $ref: '#/components/schemas/ResourceMeta'
+ status:
+ type: integer
required:
- id
- - name
- - endpoint
- - description
- - schema
- - schemaExtensions
- - meta
- SchemaExtension:
- title: SchemaExtension
+ - status
+ IngestionError:
+ title: IngestionError
type: object
properties:
- schema:
+ id:
type: string
- required:
- type: boolean
+ status:
+ type: integer
+ message:
+ type: string
+ nullable: true
+ error:
+ nullable: true
required:
- - schema
- - required
- SchemasResponse:
- title: SchemasResponse
+ - id
+ - status
+ IngestionResponse:
+ title: IngestionResponse
type: object
properties:
- schemas:
+ successes:
type: array
items:
- type: string
- totalResults:
- type: integer
- Resources:
+ $ref: '#/components/schemas/IngestionSuccess'
+ errors:
type: array
items:
- $ref: '#/components/schemas/SchemaResource'
+ $ref: '#/components/schemas/IngestionError'
required:
- - schemas
- - totalResults
- - Resources
- SchemaResource:
- title: SchemaResource
+ - successes
+ - errors
+ OpenAICompletionUsageSchema:
+ title: OpenAICompletionUsageSchema
type: object
+ description: OpenAI Usage schema from (Chat-)Completion APIs
properties:
- id:
- type: string
- name:
- type: string
- description:
- type: string
- attributes:
- type: array
- items: {}
- meta:
- $ref: '#/components/schemas/ResourceMeta'
+ prompt_tokens:
+ type: integer
+ completion_tokens:
+ type: integer
+ total_tokens:
+ type: integer
+ prompt_tokens_details:
+ type: object
+ additionalProperties:
+ type: integer
+ nullable: true
+ nullable: true
+ completion_tokens_details:
+ type: object
+ additionalProperties:
+ type: integer
+ nullable: true
+ nullable: true
required:
- - id
- - name
- - description
- - attributes
- - meta
- ScimUsersListResponse:
- title: ScimUsersListResponse
+ - prompt_tokens
+ - completion_tokens
+ - total_tokens
+ OpenAIResponseUsageSchema:
+ title: OpenAIResponseUsageSchema
type: object
+ description: OpenAI Usage schema from Response API
properties:
- schemas:
- type: array
- items:
- type: string
- totalResults:
+ input_tokens:
type: integer
- startIndex:
+ output_tokens:
type: integer
- itemsPerPage:
+ total_tokens:
type: integer
- Resources:
+ input_tokens_details:
+ type: object
+ additionalProperties:
+ type: integer
+ nullable: true
+ nullable: true
+ output_tokens_details:
+ type: object
+ additionalProperties:
+ type: integer
+ nullable: true
+ nullable: true
+ required:
+ - input_tokens
+ - output_tokens
+ - total_tokens
+ UsageDetails:
+ title: UsageDetails
+ oneOf:
+ - type: object
+ additionalProperties:
+ type: integer
+ title: Object
+ - $ref: '#/components/schemas/OpenAICompletionUsageSchema'
+ - $ref: '#/components/schemas/OpenAIResponseUsageSchema'
+ legacyMetricsResponse:
+ title: legacyMetricsResponse
+ type: object
+ properties:
+ data:
type: array
items:
- $ref: '#/components/schemas/ScimUser'
+ type: object
+ additionalProperties: true
+ description: >-
+ The metrics data. Each item in the list contains the metric values
+ and dimensions requested in the query.
+
+ Format varies based on the query parameters.
+
+ Histograms will return an array with [lower, upper, height] tuples.
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
required:
- - schemas
- - totalResults
- - startIndex
- - itemsPerPage
- - Resources
- ScimUser:
- title: ScimUser
+ - data
+ legacyObservations:
+ title: legacyObservations
type: object
properties:
- schemas:
+ data:
type: array
items:
- type: string
- id:
- type: string
- userName:
- type: string
- name:
- $ref: '#/components/schemas/ScimName'
- emails:
+ $ref: '#/components/schemas/Observation'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ required:
+ - data
+ - meta
+ legacyObservationsViews:
+ title: legacyObservationsViews
+ type: object
+ properties:
+ data:
type: array
items:
- $ref: '#/components/schemas/ScimEmail'
+ $ref: '#/components/schemas/ObservationsView'
meta:
- $ref: '#/components/schemas/UserMeta'
+ $ref: '#/components/schemas/utilsMetaResponse'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
required:
- - schemas
- - id
- - userName
- - name
- - emails
+ - data
- meta
- UserMeta:
- title: UserMeta
+ LlmConnection:
+ title: LlmConnection
type: object
+ description: LLM API connection configuration (secrets excluded)
properties:
- resourceType:
+ id:
type: string
- created:
+ provider:
type: string
- nullable: true
- lastModified:
+ description: >-
+ Provider name (e.g., 'openai', 'my-gateway'). Must be unique in
+ project, used for upserting.
+ adapter:
type: string
- nullable: true
- required:
- - resourceType
- ScimName:
- title: ScimName
- type: object
- properties:
- formatted:
+ description: The adapter used to interface with the LLM
+ displaySecretKey:
+ type: string
+ description: Masked version of the secret key for display purposes
+ baseURL:
type: string
nullable: true
- ScimEmail:
- title: ScimEmail
- type: object
- properties:
- primary:
+ description: Custom base URL for the LLM API
+ customModels:
+ type: array
+ items:
+ type: string
+ description: List of custom model names available for this connection
+ withDefaultModels:
type: boolean
- value:
+ description: Whether to include default models for this adapter
+ extraHeaderKeys:
+ type: array
+ items:
+ type: string
+ description: >-
+ Keys of extra headers sent with requests (values excluded for
+ security)
+ config:
+ type: object
+ additionalProperties: true
+ nullable: true
+ description: >-
+ Adapter-specific configuration. Required for Bedrock
+ (`{"region":"us-east-1"}`), optional for OpenAI
+ (`{"useResponsesApi":true}`), optional for VertexAI
+ (`{"location":"us-central1"}`), not used by other adapters.
+ createdAt:
type: string
- type:
+ format: date-time
+ updatedAt:
type: string
+ format: date-time
required:
- - primary
- - value
- - type
- EmptyResponse:
- title: EmptyResponse
- type: object
- description: Empty response for 204 No Content responses
- properties: {}
- ScoreConfigs:
- title: ScoreConfigs
+ - id
+ - provider
+ - adapter
+ - displaySecretKey
+ - customModels
+ - withDefaultModels
+ - extraHeaderKeys
+ - createdAt
+ - updatedAt
+ PaginatedLlmConnections:
+ title: PaginatedLlmConnections
type: object
properties:
data:
type: array
items:
- $ref: '#/components/schemas/ScoreConfig'
+ $ref: '#/components/schemas/LlmConnection'
meta:
$ref: '#/components/schemas/utilsMetaResponse'
required:
- data
- meta
- CreateScoreConfigRequest:
- title: CreateScoreConfigRequest
+ UpsertLlmConnectionRequest:
+ title: UpsertLlmConnectionRequest
type: object
+ description: Request to create or update an LLM connection (upsert)
properties:
- name:
+ provider:
type: string
description: >-
- Name of the score config. Max 35 characters. Only letters, numbers,
- underscores, spaces, periods, parentheses, and hyphens are allowed.
- dataType:
- $ref: '#/components/schemas/ScoreConfigDataType'
- categories:
- type: array
- items:
- $ref: '#/components/schemas/ConfigCategory'
- nullable: true
- description: >-
- Configure custom categories for categorical scores. Pass a list of
- objects with `label` and `value` properties. Categories are
- autogenerated for boolean configs and cannot be passed
- minValue:
- type: number
- format: double
- nullable: true
- description: >-
- Configure a minimum value for numerical scores. If not set, the
- minimum value defaults to -∞
- maxValue:
- type: number
- format: double
- nullable: true
- description: >-
- Configure a maximum value for numerical scores. If not set, the
- maximum value defaults to +∞
- description:
+ Provider name (e.g., 'openai', 'my-gateway'). Must be unique in
+ project, used for upserting.
+ adapter:
+ $ref: '#/components/schemas/LlmAdapter'
+ description: The adapter used to interface with the LLM
+ secretKey:
type: string
- nullable: true
- description: >-
- Description is shown across the Langfuse UI and can be used to e.g.
- explain the config categories in detail, why a numeric range was
- set, or provide additional context on config name or usage
- required:
- - name
- - dataType
- UpdateScoreConfigRequest:
- title: UpdateScoreConfigRequest
- type: object
- properties:
- isArchived:
- type: boolean
- nullable: true
- description: The status of the score config showing if it is archived or not
- name:
+ description: Secret key for the LLM API.
+ baseURL:
type: string
nullable: true
- description: >-
- Name of the score config. Max 35 characters. Only letters, numbers,
- underscores, spaces, periods, parentheses, and hyphens are allowed.
- categories:
+ description: Custom base URL for the LLM API
+ customModels:
type: array
items:
- $ref: '#/components/schemas/ConfigCategory'
+ type: string
nullable: true
- description: >-
- Configure custom categories for categorical scores. Pass a list of
- objects with `label` and `value` properties. Categories are
- autogenerated for boolean configs and cannot be passed
- minValue:
- type: number
- format: double
+ description: List of custom model names
+ withDefaultModels:
+ type: boolean
nullable: true
- description: >-
- Configure a minimum value for numerical scores. If not set, the
- minimum value defaults to -∞
- maxValue:
- type: number
- format: double
+ description: Whether to include default models. Default is true.
+ extraHeaders:
+ type: object
+ additionalProperties:
+ type: string
nullable: true
- description: >-
- Configure a maximum value for numerical scores. If not set, the
- maximum value defaults to +∞
- description:
- type: string
+ description: Extra headers to send with requests
+ config:
+ type: object
+ additionalProperties: true
nullable: true
description: >-
- Description is shown across the Langfuse UI and can be used to e.g.
- explain the config categories in detail, why a numeric range was
- set, or provide additional context on config name or usage
- ScoreSubjectTraceV3:
- title: ScoreSubjectTraceV3
- type: object
- properties:
- id:
- type: string
- description: The trace ID.
- required:
- - id
- ScoreSubjectObservationV3:
- title: ScoreSubjectObservationV3
- type: object
- properties:
- id:
- type: string
- description: The observation ID.
- traceId:
- type: string
- nullable: true
- description: The parent trace ID, if available.
- required:
- - id
- ScoreSubjectSessionV3:
- title: ScoreSubjectSessionV3
- type: object
- properties:
- id:
- type: string
- description: The session ID.
+ Adapter-specific configuration. Validation rules: - **Bedrock**:
+ Required. Must be `{"region": "<aws-region>"}` (e.g.,
+ `{"region":"us-east-1"}`) - **OpenAI**: Optional. If provided, must
+ be `{"useResponsesApi": <boolean>}` to control whether Langfuse
+ routes calls through OpenAI's Responses API. - **VertexAI**:
+ Optional. If provided, must be `{"location": "<gcp-location>"}`
+ (e.g., `{"location":"us-central1"}`) - **Other adapters**: Not
+ supported. Omit this field or set to null.
required:
- - id
- ScoreSubjectExperimentV3:
- title: ScoreSubjectExperimentV3
+ - provider
+ - adapter
+ - secretKey
+ DeleteLlmConnectionResponse:
+ title: DeleteLlmConnectionResponse
type: object
properties:
- id:
+ message:
type: string
- description: The dataset run ID (experiment ID).
required:
- - id
- ScoreSubjectV3:
- title: ScoreSubjectV3
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- kind:
- type: string
- enum:
- - trace
- - $ref: '#/components/schemas/ScoreSubjectTraceV3'
- required:
- - kind
- title: ScoreSubjectTraceV3
- - type: object
- allOf:
- - type: object
- properties:
- kind:
- type: string
- enum:
- - observation
- - $ref: '#/components/schemas/ScoreSubjectObservationV3'
- required:
- - kind
- title: ScoreSubjectObservationV3
- - type: object
- allOf:
- - type: object
- properties:
- kind:
- type: string
- enum:
- - session
- - $ref: '#/components/schemas/ScoreSubjectSessionV3'
- required:
- - kind
- title: ScoreSubjectSessionV3
- - type: object
- allOf:
- - type: object
- properties:
- kind:
- type: string
- enum:
- - experiment
- - $ref: '#/components/schemas/ScoreSubjectExperimentV3'
- required:
- - kind
- title: ScoreSubjectExperimentV3
- description: >-
- A reference to the entity this score is attached to. Discriminated by
- "kind" — one of trace, observation, session, or experiment.
- BaseScoreV3:
- title: BaseScoreV3
+ - message
+ LlmAdapter:
+ title: LlmAdapter
+ type: string
+ enum:
+ - anthropic
+ - openai
+ - azure
+ - bedrock
+ - google-vertex-ai
+ - google-ai-studio
+ GetMediaResponse:
+ title: GetMediaResponse
type: object
properties:
- id:
- type: string
- projectId:
+ mediaId:
type: string
- name:
+ description: The unique langfuse identifier of a media record
+ contentType:
type: string
- source:
- $ref: '#/components/schemas/ScoreSource'
- timestamp:
+ description: The MIME type of the media record
+ contentLength:
+ type: integer
+ description: The size of the media record in bytes
+ uploadedAt:
type: string
format: date-time
- environment:
+ description: The date and time when the media record was uploaded
+ url:
type: string
- description: The environment from which this score originated.
- createdAt:
+ description: The download URL of the media record
+ urlExpiry:
type: string
- format: date-time
- updatedAt:
+ description: The expiry date and time of the media record download URL
+ required:
+ - mediaId
+ - contentType
+ - contentLength
+ - uploadedAt
+ - url
+ - urlExpiry
+ PatchMediaBody:
+ title: PatchMediaBody
+ type: object
+ properties:
+ uploadedAt:
type: string
format: date-time
- comment:
+ description: The date and time when the media record was uploaded
+ uploadHttpStatus:
+ type: integer
+ description: The HTTP status code of the upload
+ uploadHttpError:
type: string
nullable: true
- description: >-
- Optional comment attached to the score. Present when "details" is
- included in the fields parameter.
- configId:
- type: string
+ description: The HTTP error message of the upload
+ uploadTimeMs:
+ type: integer
nullable: true
- description: >-
- The score config ID, if this score was created from a config.
- Present when "details" is included in the fields parameter.
- metadata:
- type: object
- additionalProperties: true
+ description: The time in milliseconds it took to upload the media record
+ required:
+ - uploadedAt
+ - uploadHttpStatus
+ GetMediaUploadUrlRequest:
+ title: GetMediaUploadUrlRequest
+ type: object
+ description: >-
+ Request a presigned media upload URL. Provide exactly one context: a
+ trace (traceId, optionally observationId) or a dataset item (datasetId +
+ datasetItemId). field is required and must match the chosen context.
+ properties:
+ traceId:
+ type: string
nullable: true
description: >-
- Arbitrary metadata attached to the score. Present when "details" is
- included in the fields parameter.
- authorUserId:
+ The trace the media is associated with. Null for dataset item media
+ uploads.
+ observationId:
type: string
nullable: true
description: >-
- The user who created this score, if available. Present when
- "annotation" is included in the fields parameter.
- queueId:
+ The observation ID associated with the media record. If the media
+ record is associated directly with a trace, this will be null.
+ datasetId:
type: string
nullable: true
description: >-
- The annotation queue this score belongs to, if any. Present when
- "annotation" is included in the fields parameter.
- subject:
- $ref: '#/components/schemas/ScoreSubjectV3'
+ The dataset the media belongs to. Null for trace/observation media
+ uploads.
+ datasetItemId:
+ type: string
nullable: true
description: >-
- The entity this score is attached to (trace, observation, session,
- or experiment). Present when "subject" is included in the fields
- parameter.
- required:
- - id
- - projectId
- - name
- - source
- - timestamp
- - environment
- - createdAt
- - updatedAt
- NumericScoreV3:
- title: NumericScoreV3
- type: object
- properties:
- value:
- type: number
- format: double
- description: The numeric value of the score.
- required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScoreV3'
- BooleanScoreV3:
- title: BooleanScoreV3
- type: object
- properties:
- value:
- type: boolean
- description: The boolean value of the score.
- required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScoreV3'
- CategoricalScoreV3:
- title: CategoricalScoreV3
- type: object
- properties:
- value:
+ The dataset item the media is associated with (need not exist yet).
+ Null for trace/observation media uploads.
+ contentType:
+ $ref: '#/components/schemas/MediaContentType'
+ contentLength:
+ type: integer
+ description: The size of the media record in bytes
+ sha256Hash:
type: string
- description: The string category value of the score.
- required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScoreV3'
- TextScoreV3:
- title: TextScoreV3
- type: object
- properties:
- value:
+ description: The SHA-256 hash of the media record
+ field:
type: string
- description: The text content of the score.
+ description: >-
+ The item field the media is in: `input`/`output`/`metadata` (trace)
+ or `input`/`expectedOutput`/`metadata` (dataset item).
required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScoreV3'
- CorrectionScoreV3:
- title: CorrectionScoreV3
+ - contentType
+ - contentLength
+ - sha256Hash
+ - field
+ GetMediaUploadUrlResponse:
+ title: GetMediaUploadUrlResponse
type: object
properties:
- value:
+ uploadUrl:
type: string
- description: The correction content of the score. Empty string if not set.
+ nullable: true
+ description: >-
+ The presigned upload URL. If the asset is already uploaded, this
+ will be null
+ mediaId:
+ type: string
+ description: The unique langfuse identifier of a media record
required:
- - value
- allOf:
- - $ref: '#/components/schemas/BaseScoreV3'
- ScoreV3:
- title: ScoreV3
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: '#/components/schemas/NumericScoreV3'
- required:
- - dataType
- title: NumericScoreV3
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: '#/components/schemas/BooleanScoreV3'
- required:
- - dataType
- title: BooleanScoreV3
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: '#/components/schemas/CategoricalScoreV3'
- required:
- - dataType
- title: CategoricalScoreV3
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - TEXT
- - $ref: '#/components/schemas/TextScoreV3'
- required:
- - dataType
- title: TextScoreV3
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CORRECTION
- - $ref: '#/components/schemas/CorrectionScoreV3'
- required:
- - dataType
- title: CorrectionScoreV3
- GetScoresV3Meta:
- title: GetScoresV3Meta
+ - mediaId
+ MediaContentType:
+ title: MediaContentType
+ type: string
+ enum:
+ - image/png
+ - image/jpeg
+ - image/jpg
+ - image/webp
+ - image/gif
+ - image/svg+xml
+ - image/tiff
+ - image/bmp
+ - image/avif
+ - image/heic
+ - audio/mpeg
+ - audio/mp3
+ - audio/wav
+ - audio/ogg
+ - audio/oga
+ - audio/aac
+ - audio/mp4
+ - audio/flac
+ - audio/opus
+ - audio/webm
+ - video/mp4
+ - video/webm
+ - video/ogg
+ - video/mpeg
+ - video/quicktime
+ - video/x-msvideo
+ - video/x-matroska
+ - text/plain
+ - text/html
+ - text/css
+ - text/csv
+ - text/markdown
+ - text/x-python
+ - application/javascript
+ - text/x-typescript
+ - application/x-yaml
+ - application/pdf
+ - application/msword
+ - application/vnd.ms-excel
+ - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
+ - application/zip
+ - application/json
+ - application/xml
+ - application/octet-stream
+ - >-
+ application/vnd.openxmlformats-officedocument.wordprocessingml.document
+ - >-
+ application/vnd.openxmlformats-officedocument.presentationml.presentation
+ - application/rtf
+ - application/x-ndjson
+ - application/vnd.apache.parquet
+ - application/gzip
+ - application/x-tar
+ - application/x-7z-compressed
+ description: The MIME type of the media record
+ MetricsV2Response:
+ title: MetricsV2Response
type: object
properties:
- limit:
- type: integer
- cursor:
- type: string
- nullable: true
+ data:
+ type: array
+ items:
+ type: object
+ additionalProperties: true
description: >-
- URL-safe base64 (base64url) cursor for the next page. Absent when
- there are no more results.
+ The metrics data. Each item in the list contains the metric values
+ and dimensions requested in the query.
+
+ Format varies based on the query parameters.
+
+ Histograms will return an array with [lower, upper, height] tuples.
required:
- - limit
- GetScoresV3Response:
- title: GetScoresV3Response
+ - data
+ ModelTokenizerId:
+ title: ModelTokenizerId
+ type: string
+ enum:
+ - openai
+ - claude
+ description: Tokenizer supported by Langfuse for model usage inference.
+ PaginatedModels:
+ title: PaginatedModels
type: object
properties:
data:
type: array
items:
- $ref: '#/components/schemas/ScoreV3'
+ $ref: '#/components/schemas/Model'
meta:
- $ref: '#/components/schemas/GetScoresV3Meta'
+ $ref: '#/components/schemas/utilsMetaResponse'
required:
- data
- meta
- CreateScoreRequest:
- title: CreateScoreRequest
+ CreateModelRequest:
+ title: CreateModelRequest
type: object
properties:
- id:
- type: string
- nullable: true
- traceId:
+ modelName:
type: string
- nullable: true
- sessionId:
+ description: >-
+ Name of the model definition. If multiple with the same name exist,
+ they are applied in the following order: (1) custom over built-in,
+ (2) newest according to startTime where
+ model.startTime<observation.startTime
+ matchPattern:
type: string
- nullable: true
- observationId:
+ description: >-
+ Regex pattern which matches this model definition to
+ generation.model. Useful in case of fine-tuned models. If you want
+ to exact match, use `(?i)^modelname$`
+ startDate:
type: string
+ format: date-time
nullable: true
- datasetRunId:
- type: string
+ description: Apply only to generations which are newer than this ISO date.
+ unit:
+ $ref: '#/components/schemas/ModelUsageUnit'
+ description: Unit used by this model.
+ inputPrice:
+ type: number
+ format: double
nullable: true
- name:
- type: string
- value:
- $ref: '#/components/schemas/CreateScoreValue'
description: >-
- The value of the score. Must be passed as string for categorical and
- text scores, and numeric for boolean and numeric scores. Boolean
- score values must equal either 1 or 0 (true or false). Text score
- values must be between 1 and 500 characters.
- comment:
- type: string
+ Deprecated. Use 'pricingTiers' instead. Price (USD) per input unit.
+ Creates a default tier if pricingTiers not provided.
+ outputPrice:
+ type: number
+ format: double
nullable: true
- metadata:
- type: object
- additionalProperties: true
+ description: >-
+ Deprecated. Use 'pricingTiers' instead. Price (USD) per output unit.
+ Creates a default tier if pricingTiers not provided.
+ totalPrice:
+ type: number
+ format: double
nullable: true
- environment:
- type: string
+ description: >-
+ Deprecated. Use 'pricingTiers' instead. Price (USD) per total units.
+ Cannot be set if input or output price is set. Creates a default
+ tier if pricingTiers not provided.
+ pricingTiers:
+ type: array
+ items:
+ $ref: '#/components/schemas/PricingTierInput'
nullable: true
description: >-
- The environment of the score. Can be any lowercase alphanumeric
- string with hyphens and underscores that does not start with
- 'langfuse'.
- queueId:
- type: string
+ Optional. Array of pricing tiers for this model.
+
+
+ Use pricing tiers for all models - both those with threshold-based
+ pricing variations and those with simple flat pricing:
+
+
+ - For models with standard flat pricing: Create a single default
+ tier with your prices
+ (e.g., one tier with isDefault=true, priority=0, conditions=[], and your standard prices)
+
+ - For models with threshold-based pricing: Create a default tier
+ plus additional conditional tiers
+ (e.g., default tier for standard usage + high-volume tier for usage above certain thresholds)
+
+ Requirements:
+
+ - Cannot be provided with flat prices
+ (inputPrice/outputPrice/totalPrice) - use one approach or the other
+
+ - Must include exactly one default tier with isDefault=true,
+ priority=0, and conditions=[]
+
+ - All tier names and priorities must be unique within the model
+
+ - Each tier must define at least one price
+
+
+ If omitted, you must provide flat prices instead
+ (inputPrice/outputPrice/totalPrice),
+
+ which will automatically create a single default tier named
+ "Standard".
+ tokenizerId:
+ $ref: '#/components/schemas/ModelTokenizerId'
nullable: true
description: >-
- The annotation queue referenced by the score. Indicates if score was
- initially created while processing annotation queue.
- dataType:
- $ref: '#/components/schemas/ScoreDataType'
+ Optional. Tokenizer to be applied to observations which match to
+ this model. See docs for more details.
+ tokenizerConfig:
nullable: true
description: >-
- The data type of the score. When passing a configId this field is
- inferred. Otherwise, this field must be passed or will default to
- numeric.
- configId:
+ Optional. Configuration for the selected tokenizer. Needs to be
+ JSON. See docs for more details.
+ required:
+ - modelName
+ - matchPattern
+ - unit
+ ObservationsV2Response:
+ title: ObservationsV2Response
+ type: object
+ description: >-
+ Response containing observations with field-group-based filtering and
+ cursor-based pagination.
+
+
+ The `data` array contains observation objects with only the requested
+ field groups included.
+
+ Use the `cursor` in `meta` to retrieve the next page of results.
+ properties:
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/ObservationV2'
+ description: >-
+ Array of observation objects. Fields included depend on the `fields`
+ parameter in the request.
+ meta:
+ $ref: '#/components/schemas/ObservationsV2Meta'
+ required:
+ - data
+ - meta
+ ObservationsV2Meta:
+ title: ObservationsV2Meta
+ type: object
+ description: Metadata for cursor-based pagination
+ properties:
+ cursor:
type: string
nullable: true
description: >-
- Reference a score config on a score. The unique langfuse identifier
- of a score config. When passing this field, the dataType and
- stringValue fields are automatically populated.
- source:
- $ref: '#/components/schemas/CreateScoreSource'
+ Base64-encoded cursor to use for retrieving the next page. If not
+ present, there are no more results.
+ OtelResourceSpan:
+ title: OtelResourceSpan
+ type: object
+ description: >-
+ Represents a collection of spans from a single resource as per OTLP
+ specification
+ properties:
+ resource:
+ $ref: '#/components/schemas/OtelResource'
nullable: true
- description: >-
- The source of the score. Defaults to API. Set to ANNOTATION to
- prefill scores (e.g. from an LLM) for a human reviewer to verify in
- an annotation queue. When source is ANNOTATION, a configId is
- required unless dataType is CORRECTION. EVAL is reserved for
- internal evaluator outputs and is not accepted on this endpoint.
- required:
- - name
- - value
- CreateScoreSource:
- title: CreateScoreSource
- type: string
- enum:
- - API
- - ANNOTATION
- description: |-
- Source values accepted when creating a score via the public REST API.
- EVAL is reserved for internal evaluator outputs and is intentionally not
- exposed here — use commons.ScoreSource when reading scores.
- CreateScoreResponse:
- title: CreateScoreResponse
+ description: Resource information
+ scopeSpans:
+ type: array
+ items:
+ $ref: '#/components/schemas/OtelScopeSpan'
+ nullable: true
+ description: Array of scope spans
+ OtelResource:
+ title: OtelResource
type: object
+ description: Resource attributes identifying the source of telemetry
properties:
- id:
- type: string
- description: The id of the created object in Langfuse
- required:
- - id
- GetScoresResponseTraceData:
- title: GetScoresResponseTraceData
+ attributes:
+ type: array
+ items:
+ $ref: '#/components/schemas/OtelAttribute'
+ nullable: true
+ description: Resource attributes like service.name, service.version, etc.
+ OtelScopeSpan:
+ title: OtelScopeSpan
type: object
+ description: Collection of spans from a single instrumentation scope
properties:
- userId:
- type: string
+ scope:
+ $ref: '#/components/schemas/OtelScope'
nullable: true
- description: The user ID associated with the trace referenced by score
- tags:
+ description: Instrumentation scope information
+ spans:
type: array
items:
- type: string
+ $ref: '#/components/schemas/OtelSpan'
nullable: true
- description: A list of tags associated with the trace referenced by score
- environment:
+ description: Array of spans
+ OtelScope:
+ title: OtelScope
+ type: object
+ description: Instrumentation scope information
+ properties:
+ name:
type: string
nullable: true
- description: The environment of the trace referenced by score
- sessionId:
+ description: Instrumentation scope name
+ version:
type: string
nullable: true
- description: The session ID associated with the trace referenced by score
- GetScoresResponseDataNumeric:
- title: GetScoresResponseDataNumeric
- type: object
- properties:
- trace:
- $ref: '#/components/schemas/GetScoresResponseTraceData'
+ description: Instrumentation scope version
+ attributes:
+ type: array
+ items:
+ $ref: '#/components/schemas/OtelAttribute'
nullable: true
- allOf:
- - $ref: '#/components/schemas/NumericScore'
- GetScoresResponseDataCategorical:
- title: GetScoresResponseDataCategorical
+ description: Additional scope attributes
+ OtelSpan:
+ title: OtelSpan
type: object
+ description: Individual span representing a unit of work or operation
properties:
- trace:
- $ref: '#/components/schemas/GetScoresResponseTraceData'
+ traceId:
nullable: true
- allOf:
- - $ref: '#/components/schemas/CategoricalScore'
- GetScoresResponseDataBoolean:
- title: GetScoresResponseDataBoolean
- type: object
- properties:
- trace:
- $ref: '#/components/schemas/GetScoresResponseTraceData'
+ description: Trace ID (16 bytes, hex-encoded string in JSON or Buffer in binary)
+ spanId:
nullable: true
- allOf:
- - $ref: '#/components/schemas/BooleanScore'
- GetScoresResponseDataCorrection:
- title: GetScoresResponseDataCorrection
- type: object
- properties:
- trace:
- $ref: '#/components/schemas/GetScoresResponseTraceData'
+ description: Span ID (8 bytes, hex-encoded string in JSON or Buffer in binary)
+ parentSpanId:
nullable: true
- allOf:
- - $ref: '#/components/schemas/CorrectionScore'
- GetScoresResponseDataText:
- title: GetScoresResponseDataText
- type: object
- properties:
- trace:
- $ref: '#/components/schemas/GetScoresResponseTraceData'
+ description: Parent span ID if this is a child span
+ name:
+ type: string
nullable: true
- allOf:
- - $ref: '#/components/schemas/TextScore'
- GetScoresResponseData:
- title: GetScoresResponseData
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: '#/components/schemas/GetScoresResponseDataNumeric'
- required:
- - dataType
- title: GetScoresResponseDataNumeric
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: '#/components/schemas/GetScoresResponseDataCategorical'
- required:
- - dataType
- title: GetScoresResponseDataCategorical
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: '#/components/schemas/GetScoresResponseDataBoolean'
- required:
- - dataType
- title: GetScoresResponseDataBoolean
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CORRECTION
- - $ref: '#/components/schemas/GetScoresResponseDataCorrection'
- required:
- - dataType
- title: GetScoresResponseDataCorrection
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - TEXT
- - $ref: '#/components/schemas/GetScoresResponseDataText'
- required:
- - dataType
- title: GetScoresResponseDataText
- GetScoresResponse:
- title: GetScoresResponse
- type: object
- properties:
- data:
+ description: Span name describing the operation
+ kind:
+ type: integer
+ nullable: true
+ description: Span kind (1=INTERNAL, 2=SERVER, 3=CLIENT, 4=PRODUCER, 5=CONSUMER)
+ startTimeUnixNano:
+ nullable: true
+ description: Start time in nanoseconds since Unix epoch
+ endTimeUnixNano:
+ nullable: true
+ description: End time in nanoseconds since Unix epoch
+ attributes:
type: array
items:
- $ref: '#/components/schemas/GetScoresResponseData'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
+ $ref: '#/components/schemas/OtelAttribute'
nullable: true
- required:
- - data
- - meta
- PaginatedSessions:
- title: PaginatedSessions
+ description: >-
+ Span attributes including Langfuse-specific attributes
+ (langfuse.observation.*)
+ status:
+ nullable: true
+ description: Span status object
+ OtelAttribute:
+ title: OtelAttribute
type: object
+ description: Key-value attribute pair for resources, scopes, or spans
properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/Session'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
+ key:
+ type: string
nullable: true
- required:
- - data
- - meta
- Traces:
- title: Traces
+ description: Attribute key (e.g., "service.name", "langfuse.observation.type")
+ value:
+ $ref: '#/components/schemas/OtelAttributeValue'
+ nullable: true
+ description: Attribute value
+ OtelAttributeValue:
+ title: OtelAttributeValue
type: object
+ description: Attribute value wrapper supporting different value types
properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/TraceWithDetails'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
- _deprecation:
- $ref: '#/components/schemas/Deprecation'
+ stringValue:
+ type: string
nullable: true
- required:
- - data
- - meta
- DeleteTraceResponse:
- title: DeleteTraceResponse
+ description: String value
+ intValue:
+ type: integer
+ nullable: true
+ description: Integer value
+ doubleValue:
+ type: number
+ format: double
+ nullable: true
+ description: Double value
+ boolValue:
+ type: boolean
+ nullable: true
+ description: Boolean value
+ OtelTraceResponse:
+ title: OtelTraceResponse
+ type: object
+ description: Response from trace export request. Empty object indicates success.
+ properties: {}
+ MembershipRole:
+ title: MembershipRole
+ type: string
+ enum:
+ - OWNER
+ - ADMIN
+ - MEMBER
+ - VIEWER
+ MembershipRequest:
+ title: MembershipRequest
type: object
properties:
- message:
+ userId:
type: string
+ role:
+ $ref: '#/components/schemas/MembershipRole'
required:
- - message
- Sort:
- title: Sort
+ - userId
+ - role
+ DeleteMembershipRequest:
+ title: DeleteMembershipRequest
type: object
properties:
- id:
+ userId:
type: string
required:
- - id
- unstableEvaluatorType:
- title: unstableEvaluatorType
- type: string
- enum:
- - llm_as_judge
- - code
- description: |-
- The evaluator type.
-
- The unstable public API supports LLM-as-a-judge and code evaluators.
- unstableCodeEvaluatorSourceCodeLanguage:
- title: unstableCodeEvaluatorSourceCodeLanguage
- type: string
- enum:
- - PYTHON
- - TYPESCRIPT
- description: Code evaluator runtime language.
- unstableEvaluationRuleTarget:
- title: unstableEvaluationRuleTarget
- type: string
- enum:
- - observation
- - experiment
- description: >-
- The ingestion object type that should trigger evaluation runs.
-
-
- Choose the target first, because it changes both the valid filter
- columns and the valid variable-mapping sources:
-
- - `observation` evaluates live-ingested observations such as
- generations, spans, and events.
- It supports mapping from `input`, `output`, `metadata`, and `tool_calls`.
- - `experiment` evaluates live experiment executions and can additionally
- map `expected_output` and `experiment_item_metadata`.
- It currently supports filtering by `datasetId`.
- Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter.
- unstableEvaluationRuleStatus:
- title: unstableEvaluationRuleStatus
- type: string
- enum:
- - active
- - inactive
- - paused
- description: >-
- Effective runtime status of the evaluation rule.
-
-
- - `active`: enabled and currently runnable.
-
- - `inactive`: disabled by configuration.
-
- - `paused`: enabled, but Langfuse has blocked execution until the
- underlying issue is resolved.
- unstablePromptVariableMappingSource:
- title: unstablePromptVariableMappingSource
- type: string
- enum:
- - input
- - output
- - metadata
- - tool_calls
- - expected_output
- - experiment_item_metadata
- description: >-
- Source field used to populate a prompt variable.
-
-
- Use these values when mapping evaluator prompt variables to live data.
-
-
- Target-specific rules:
-
- - `target=observation` supports `input`, `output`, `metadata`, and
- `tool_calls`
-
- - `target=experiment` supports `input`, `output`, `metadata`,
- `tool_calls`, `expected_output`, and `experiment_item_metadata`
-
-
- Source semantics:
-
- - `input`: the observation or experiment input payload
-
- - `output`: the observation or experiment output payload
-
- - `metadata`: the metadata object for the target. Combine with
- `jsonPath` when you need one nested field instead of the whole object.
-
- - `tool_calls`: the tool calls recorded on the observation, as an array
- of `{id, name, arguments, type, index}` objects in the order the model
- emitted them. Combine with `jsonPath` (for example `$[*].name`) to
- select parts of each call.
-
- - `expected_output`: the experiment item's expected output. Only valid
- for `target=experiment`.
-
- - `experiment_item_metadata`: the experiment item's metadata object.
- Only valid for `target=experiment`.
- unstableEvaluatorModelConfig:
- title: unstableEvaluatorModelConfig
+ - userId
+ MembershipResponse:
+ title: MembershipResponse
type: object
- description: >-
- Optional explicit model configuration for an evaluator.
-
-
- If omitted, Langfuse uses the project's default evaluation model.
-
- If provided, the model must be available to the project when the
- evaluator or evaluation rule is enabled.
-
-
- To discover valid configured `provider` values for a project, call `GET
- /api/public/llm-connections` and read the `provider` field from the
- returned connections.
-
- Use a `provider` value that matches one of the connections already
- configured in the same project.
-
-
- Recovery guidance:
-
- - If evaluator creation returns `422` with
- `code=evaluator_preflight_failed`, either provide a valid explicit
- `modelConfig` here or configure the project's default evaluation model,
- then retry the same request.
properties:
- provider:
+ userId:
type: string
- description: >-
- Provider identifier to use for this evaluator, for example `openai`
- or `anthropic`.
-
-
- To discover valid values for the current project, call `GET
- /api/public/llm-connections` and use one of the returned `provider`
- values.
- model:
+ role:
+ $ref: '#/components/schemas/MembershipRole'
+ email:
+ type: string
+ name:
type: string
- description: >-
- Model identifier exposed by the provider, for example `gpt-4.1-mini`.
required:
- - provider
- - model
- unstableEvaluatorOutputDataType:
- title: unstableEvaluatorOutputDataType
- type: string
- enum:
- - NUMERIC
- - BOOLEAN
- - CATEGORICAL
- description: >-
- Structured score type returned by an evaluator.
-
-
- This controls the type of score value Langfuse stores for evaluation
- results:
-
- - `NUMERIC`: a numeric score such as `0.82`
-
- - `BOOLEAN`: a boolean score such as `true`
-
- - `CATEGORICAL`: one or more category labels from a fixed list
- unstableEvaluatorOutputFieldDefinition:
- title: unstableEvaluatorOutputFieldDefinition
+ - userId
+ - role
+ - email
+ - name
+ MembershipDeletionResponse:
+ title: MembershipDeletionResponse
type: object
properties:
- description:
+ message:
+ type: string
+ userId:
type: string
- description: >-
- Human-readable instructions for what the evaluator should return in
- this field.
required:
- - description
- unstableEvaluatorOutputDefinition:
- title: unstableEvaluatorOutputDefinition
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: >-
- #/components/schemas/unstablePublicNumericEvaluatorOutputDefinition
- required:
- - dataType
- title: UnstablePublicNumericEvaluatorOutputDefinition
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: >-
- #/components/schemas/unstablePublicBooleanEvaluatorOutputDefinition
- required:
- - dataType
- title: UnstablePublicBooleanEvaluatorOutputDefinition
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: >-
- #/components/schemas/unstablePublicCategoricalEvaluatorOutputDefinition
- required:
- - dataType
- title: UnstablePublicCategoricalEvaluatorOutputDefinition
- description: >-
- Structured output definition to send when creating an evaluator.
-
-
- Agent guidance:
-
- - `dataType` is required.
-
- - Do not send `version`; that is an internal storage detail and is not
- part of the public request contract.
-
- - For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and
- `score.description`.
-
- - For `NUMERIC`, `score.minValue` and `score.maxValue` optionally define
- inclusive bounds. If both are set, `minValue` must not exceed
- `maxValue`.
-
- - For `CATEGORICAL`, also provide `score.categories` and
- `score.shouldAllowMultipleMatches`.
- unstablePublicNumericEvaluatorOutputScoreDefinition:
- title: unstablePublicNumericEvaluatorOutputScoreDefinition
+ - message
+ - userId
+ MembershipsResponse:
+ title: MembershipsResponse
type: object
properties:
- description:
+ memberships:
+ type: array
+ items:
+ $ref: '#/components/schemas/MembershipResponse'
+ required:
+ - memberships
+ OrganizationProject:
+ title: OrganizationProject
+ type: object
+ properties:
+ id:
type: string
- minValue:
- type: number
- format: double
- nullable: true
- description: Inclusive minimum value for numeric evaluator scores.
- maxValue:
- type: number
- format: double
+ name:
+ type: string
+ metadata:
+ type: object
+ additionalProperties: true
nullable: true
- description: Inclusive maximum value for numeric evaluator scores.
+ createdAt:
+ type: string
+ format: date-time
+ updatedAt:
+ type: string
+ format: date-time
required:
- - description
- unstablePublicNumericEvaluatorOutputDefinition:
- title: unstablePublicNumericEvaluatorOutputDefinition
+ - id
+ - name
+ - createdAt
+ - updatedAt
+ OrganizationProjectsResponse:
+ title: OrganizationProjectsResponse
type: object
properties:
- dataType:
- $ref: '#/components/schemas/unstableEvaluatorOutputDataType'
- description: Always `NUMERIC`.
- reasoning:
- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
- score:
- $ref: >-
- #/components/schemas/unstablePublicNumericEvaluatorOutputScoreDefinition
+ projects:
+ type: array
+ items:
+ $ref: '#/components/schemas/OrganizationProject'
required:
- - dataType
- - reasoning
- - score
- unstablePublicBooleanEvaluatorOutputDefinition:
- title: unstablePublicBooleanEvaluatorOutputDefinition
+ - projects
+ OrganizationApiKey:
+ title: OrganizationApiKey
type: object
properties:
- dataType:
- $ref: '#/components/schemas/unstableEvaluatorOutputDataType'
- description: Always `BOOLEAN`.
- reasoning:
- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
- score:
- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
+ id:
+ type: string
+ createdAt:
+ type: string
+ format: date-time
+ expiresAt:
+ type: string
+ format: date-time
+ nullable: true
+ lastUsedAt:
+ type: string
+ format: date-time
+ nullable: true
+ note:
+ type: string
+ nullable: true
+ publicKey:
+ type: string
+ displaySecretKey:
+ type: string
required:
- - dataType
- - reasoning
- - score
- unstablePublicCategoricalEvaluatorOutputScoreDefinition:
- title: unstablePublicCategoricalEvaluatorOutputScoreDefinition
+ - id
+ - createdAt
+ - publicKey
+ - displaySecretKey
+ OrganizationApiKeysResponse:
+ title: OrganizationApiKeysResponse
type: object
properties:
- description:
- type: string
- categories:
+ apiKeys:
type: array
items:
- type: string
- shouldAllowMultipleMatches:
- type: boolean
+ $ref: '#/components/schemas/OrganizationApiKey'
required:
- - description
- - categories
- - shouldAllowMultipleMatches
- unstablePublicCategoricalEvaluatorOutputDefinition:
- title: unstablePublicCategoricalEvaluatorOutputDefinition
+ - apiKeys
+ Projects:
+ title: Projects
type: object
properties:
- dataType:
- $ref: '#/components/schemas/unstableEvaluatorOutputDataType'
- description: Always `CATEGORICAL`.
- reasoning:
- $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
- score:
- $ref: >-
- #/components/schemas/unstablePublicCategoricalEvaluatorOutputScoreDefinition
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/Project'
required:
- - dataType
- - reasoning
- - score
- unstablePublicEvaluatorOutputDefinition:
- title: unstablePublicEvaluatorOutputDefinition
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - NUMERIC
- - $ref: >-
- #/components/schemas/unstablePublicNumericEvaluatorOutputDefinition
- required:
- - dataType
- title: UnstablePublicNumericEvaluatorOutputDefinition
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - BOOLEAN
- - $ref: >-
- #/components/schemas/unstablePublicBooleanEvaluatorOutputDefinition
- required:
- - dataType
- title: UnstablePublicBooleanEvaluatorOutputDefinition
- - type: object
- allOf:
- - type: object
- properties:
- dataType:
- type: string
- enum:
- - CATEGORICAL
- - $ref: >-
- #/components/schemas/unstablePublicCategoricalEvaluatorOutputDefinition
- required:
- - dataType
- title: UnstablePublicCategoricalEvaluatorOutputDefinition
- description: >-
- Evaluator output definition returned by the public API.
-
-
- This response always includes `dataType` and never includes an internal
- output-definition `version`.
-
- Legacy stored evaluator definitions are normalized into this shape
- before they are returned.
-
-
- Use this response shape when deciding how to interpret future evaluation
- scores:
-
- - `NUMERIC`: expect numeric score values
-
- - `BOOLEAN`: expect `true` / `false`
-
- - `CATEGORICAL`: expect one or more values from `score.categories`
- unstableEvaluationRuleStringFilterOperator:
- title: unstableEvaluationRuleStringFilterOperator
- type: string
- enum:
- - '='
- - contains
- - does not contain
- - starts with
- - ends with
- unstableEvaluationRuleNumberFilterOperator:
- title: unstableEvaluationRuleNumberFilterOperator
- type: string
- enum:
- - '='
- - '>'
- - <
- - '>='
- - <=
- unstableEvaluationRuleOptionsFilterOperator:
- title: unstableEvaluationRuleOptionsFilterOperator
- type: string
- enum:
- - any of
- - none of
- unstableEvaluationRuleArrayOptionsFilterOperator:
- title: unstableEvaluationRuleArrayOptionsFilterOperator
- type: string
- enum:
- - any of
- - none of
- - all of
- unstableEvaluationRuleBooleanFilterOperator:
- title: unstableEvaluationRuleBooleanFilterOperator
- type: string
- enum:
- - '='
- - <>
- unstableEvaluationRuleNullFilterOperator:
- title: unstableEvaluationRuleNullFilterOperator
- type: string
- enum:
- - is null
- - is not null
- unstableDateTimeEvaluationRuleFilter:
- title: unstableDateTimeEvaluationRuleFilter
+ - data
+ Organization:
+ title: Organization
type: object
properties:
- column:
+ id:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'
- description: Comparison operator for datetime values.
- value:
+ description: The unique identifier of the organization
+ name:
type: string
- format: date-time
- description: Datetime value to compare against.
+ description: The name of the organization
required:
- - column
- - operator
- - value
- unstableStringEvaluationRuleFilter:
- title: unstableStringEvaluationRuleFilter
+ - id
+ - name
+ Project:
+ title: Project
type: object
properties:
- column:
+ id:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleStringFilterOperator'
- value:
+ name:
type: string
+ organization:
+ $ref: '#/components/schemas/Organization'
+ description: The organization this project belongs to
+ metadata:
+ type: object
+ additionalProperties: true
+ description: Metadata for the project
+ retentionDays:
+ type: integer
+ nullable: true
+ description: >-
+ Number of days to retain data. Null or 0 means no retention. Omitted
+ if no retention is configured.
required:
- - column
- - operator
- - value
- unstableNumberEvaluationRuleFilter:
- title: unstableNumberEvaluationRuleFilter
+ - id
+ - name
+ - organization
+ - metadata
+ ProjectDeletionResponse:
+ title: ProjectDeletionResponse
type: object
properties:
- column:
+ success:
+ type: boolean
+ message:
type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'
- value:
- type: number
- format: double
required:
- - column
- - operator
- - value
- unstableStringOptionsEvaluationRuleFilter:
- title: unstableStringOptionsEvaluationRuleFilter
+ - success
+ - message
+ ApiKeyList:
+ title: ApiKeyList
type: object
+ description: List of API keys for a project
properties:
- column:
- type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleOptionsFilterOperator'
- value:
+ apiKeys:
type: array
items:
- type: string
- description: One or more allowed string values.
+ $ref: '#/components/schemas/ApiKeySummary'
required:
- - column
- - operator
- - value
- unstableArrayOptionsEvaluationRuleFilter:
- title: unstableArrayOptionsEvaluationRuleFilter
+ - apiKeys
+ ApiKeySummary:
+ title: ApiKeySummary
type: object
+ description: Summary of an API key
properties:
- column:
+ id:
type: string
- description: Column to filter on.
- operator:
- $ref: >-
- #/components/schemas/unstableEvaluationRuleArrayOptionsFilterOperator
- value:
- type: array
- items:
- type: string
- description: One or more array elements to match.
- required:
- - column
- - operator
- - value
- unstableStringObjectEvaluationRuleFilter:
- title: unstableStringObjectEvaluationRuleFilter
- type: object
- properties:
- column:
+ createdAt:
type: string
- description: >-
- Object-valued column to filter on. In the unstable public API this
- is currently `metadata`.
- key:
+ format: date-time
+ expiresAt:
type: string
- description: Top-level key inside the object-valued column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleStringFilterOperator'
- value:
+ format: date-time
+ nullable: true
+ lastUsedAt:
type: string
- required:
- - column
- - key
- - operator
- - value
- unstableNumberObjectEvaluationRuleFilter:
- title: unstableNumberObjectEvaluationRuleFilter
- type: object
- properties:
- column:
+ format: date-time
+ nullable: true
+ note:
type: string
- description: Object-valued column to filter on.
- key:
+ nullable: true
+ publicKey:
+ type: string
+ displaySecretKey:
type: string
- description: Key inside the object-valued column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'
- value:
- type: number
- format: double
required:
- - column
- - key
- - operator
- - value
- unstableCategoryOptionsEvaluationRuleFilter:
- title: unstableCategoryOptionsEvaluationRuleFilter
+ - id
+ - createdAt
+ - publicKey
+ - displaySecretKey
+ ApiKeyResponse:
+ title: ApiKeyResponse
type: object
+ description: Response for API key creation
properties:
- column:
+ id:
type: string
- description: Object-valued column to filter on.
- key:
+ createdAt:
type: string
- description: Key inside the object-valued column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleOptionsFilterOperator'
- value:
- type: array
- items:
- type: string
+ format: date-time
+ publicKey:
+ type: string
+ secretKey:
+ type: string
+ displaySecretKey:
+ type: string
+ note:
+ type: string
+ nullable: true
required:
- - column
- - key
- - operator
- - value
- unstableBooleanEvaluationRuleFilter:
- title: unstableBooleanEvaluationRuleFilter
+ - id
+ - createdAt
+ - publicKey
+ - secretKey
+ - displaySecretKey
+ ApiKeyDeletionResponse:
+ title: ApiKeyDeletionResponse
type: object
+ description: Response for API key deletion
properties:
- column:
- type: string
- description: Column to filter on.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleBooleanFilterOperator'
- value:
+ success:
type: boolean
required:
- - column
- - operator
- - value
- unstableNullEvaluationRuleFilter:
- title: unstableNullEvaluationRuleFilter
+ - success
+ PromptMetaListResponse:
+ title: PromptMetaListResponse
type: object
properties:
- column:
- type: string
- description: >-
- Column to filter on. In the unstable public API this is currently
- `parentObservationId`.
- operator:
- $ref: '#/components/schemas/unstableEvaluationRuleNullFilterOperator'
- value:
- type: string
- nullable: true
- description: >-
- Ignored placeholder value. Clients may omit it or send an empty
- string.
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/PromptMeta'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
required:
- - column
- - operator
- unstablePromptVariableMappingInput:
- title: unstablePromptVariableMappingInput
+ - data
+ - meta
+ PromptMeta:
+ title: PromptMeta
type: object
- description: >-
- Connects one prompt variable to the available evaluation data.
-
-
- Manual mappings are used for `llm_as_judge` evaluators. `code`
- evaluators use a fixed runtime mapping managed by Langfuse.
-
-
- How to build a valid mapping list:
-
- 1. Create the evaluator or fetch it with `GET /evaluators/{id}`.
-
- 2. Read the evaluator `variables` array.
-
- 3. Add exactly one mapping object for each variable in that array.
-
- 4. Use the variable name exactly as returned, without braces such as
- `{{` or `}}`.
-
- 5. Choose a `source` that is valid for the selected `target`.
-
-
- `jsonPath` is optional. Use it only when the selected source is a JSON
- object and you want to extract one nested field before inserting it into
- the evaluator prompt.
-
-
- Recovery guidance:
-
- - `invalid_variable_mapping`: the variable name is unknown for this
- evaluator, or the selected `source` is not valid for the chosen `target`
-
- - `missing_variable_mapping`: one or more LLM-as-judge prompt variables
- are not mapped yet
-
- - `duplicate_variable_mapping`: the same prompt variable appears more
- than once
-
- - `invalid_json_path`: the JSONPath expression is malformed. Remove it
- or correct it.
properties:
- variable:
+ name:
type: string
- description: >-
- Prompt variable name without braces.
-
-
- Example: for the prompt `Judge {{input}} against {{output}}`, use
- `input` and `output`.
- source:
- $ref: '#/components/schemas/unstablePromptVariableMappingSource'
- description: >-
- Source field that should populate the prompt variable.
-
-
- Quick reference:
-
- - `target=observation`: `input`, `output`, `metadata`, `tool_calls`
-
- - `target=experiment`: `input`, `output`, `metadata`, `tool_calls`,
- `expected_output`, `experiment_item_metadata`
- jsonPath:
+ type:
+ $ref: '#/components/schemas/PromptType'
+ description: Indicates whether the prompt is a text or chat prompt.
+ versions:
+ type: array
+ items:
+ type: integer
+ labels:
+ type: array
+ items:
+ type: string
+ tags:
+ type: array
+ items:
+ type: string
+ lastUpdatedAt:
type: string
- nullable: true
+ format: date-time
+ lastConfig:
description: >-
- Optional JSONPath selector applied to the selected source before it
- is passed to the evaluator prompt.
-
-
- Requirements:
-
- - Must start with `$`
-
- - Must be a syntactically valid JSONPath expression
-
- - Most useful with `source=metadata`
+ Config object of the most recent prompt version that matches the
+ filters (if any are provided)
required:
- - variable
- - source
- unstablePromptVariableMappingRead:
- title: unstablePromptVariableMappingRead
+ - name
+ - type
+ - versions
+ - labels
+ - tags
+ - lastUpdatedAt
+ - lastConfig
+ CreatePromptRequest:
+ title: CreatePromptRequest
+ oneOf:
+ - $ref: '#/components/schemas/CreateChatPromptRequest'
+ - $ref: '#/components/schemas/CreateTextPromptRequest'
+ CreateChatPromptRequest:
+ title: CreateChatPromptRequest
type: object
- description: |-
- Connects one prompt variable to data from a target.
-
- `source` is `null` when the mapping is incomplete.
properties:
- variable:
+ name:
type: string
- description: Prompt variable name without braces.
- source:
- $ref: '#/components/schemas/unstablePromptVariableMappingSource'
+ prompt:
+ type: array
+ items:
+ $ref: '#/components/schemas/ChatMessageWithPlaceholders'
+ config:
nullable: true
- description: >-
- Source field populating the variable, or `null` when the mapping is
- incomplete.
- jsonPath:
- type: string
+ type:
+ $ref: '#/components/schemas/CreateChatPromptType'
+ labels:
+ type: array
+ items:
+ type: string
nullable: true
- description: Optional JSONPath selector applied to the selected source.
- required:
- - variable
- - source
- unstableEvaluationRuleFilter:
- title: unstableEvaluationRuleFilter
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - datetime
- - $ref: '#/components/schemas/unstableDateTimeEvaluationRuleFilter'
- required:
- - type
- title: UnstableDateTimeEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - string
- - $ref: '#/components/schemas/unstableStringEvaluationRuleFilter'
- required:
- - type
- title: UnstableStringEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - number
- - $ref: '#/components/schemas/unstableNumberEvaluationRuleFilter'
- required:
- - type
- title: UnstableNumberEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - stringOptions
- - $ref: '#/components/schemas/unstableStringOptionsEvaluationRuleFilter'
- required:
- - type
- title: UnstableStringOptionsEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - categoryOptions
- - $ref: '#/components/schemas/unstableCategoryOptionsEvaluationRuleFilter'
- required:
- - type
- title: UnstableCategoryOptionsEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - arrayOptions
- - $ref: '#/components/schemas/unstableArrayOptionsEvaluationRuleFilter'
- required:
- - type
- title: UnstableArrayOptionsEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - stringObject
- - $ref: '#/components/schemas/unstableStringObjectEvaluationRuleFilter'
- required:
- - type
- title: UnstableStringObjectEvaluationRuleFilter
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - numberObject
- - $ref: '#/components/schemas/unstableNumberObjectEvaluationRuleFilter'
- required:
- - type
- title: UnstableNumberObjectEvaluationRuleFilter
+ description: List of deployment labels of this prompt version.
+ tags:
+ type: array
+ items:
+ type: string
+ nullable: true
+ description: List of tags to apply to all versions of this prompt.
+ commitMessage:
+ type: string
+ nullable: true
+ description: Commit message for this prompt version.
+ required:
+ - name
+ - prompt
+ - type
+ CreateTextPromptRequest:
+ title: CreateTextPromptRequest
+ type: object
+ properties:
+ name:
+ type: string
+ prompt:
+ type: string
+ config:
+ nullable: true
+ type:
+ $ref: '#/components/schemas/CreateTextPromptType'
+ nullable: true
+ labels:
+ type: array
+ items:
+ type: string
+ nullable: true
+ description: List of deployment labels of this prompt version.
+ tags:
+ type: array
+ items:
+ type: string
+ nullable: true
+ description: List of tags to apply to all versions of this prompt.
+ commitMessage:
+ type: string
+ nullable: true
+ description: Commit message for this prompt version.
+ required:
+ - name
+ - prompt
+ Prompt:
+ title: Prompt
+ oneOf:
- type: object
allOf:
- type: object
@@ -18075,11 +15228,11 @@ components:
type:
type: string
enum:
- - boolean
- - $ref: '#/components/schemas/unstableBooleanEvaluationRuleFilter'
+ - chat
+ - $ref: '#/components/schemas/ChatPrompt'
required:
- type
- title: UnstableBooleanEvaluationRuleFilter
+ title: ChatPrompt
- type: object
allOf:
- type: object
@@ -18087,1641 +15240,1528 @@ components:
type:
type: string
enum:
- - 'null'
- - $ref: '#/components/schemas/unstableNullEvaluationRuleFilter'
+ - text
+ - $ref: '#/components/schemas/TextPrompt'
required:
- type
- title: UnstableNullEvaluationRuleFilter
- description: >-
- One filter condition used to decide whether a live-ingested target
- should be evaluated.
-
-
- An evaluation rule can include zero or more filter objects. All filters
- must be satisfied for the target to run.
-
-
- How to build a valid filter object:
-
- - Pick the `target` first, because it changes the supported columns.
-
- - Pick the filter `type`. That determines which fields are required.
-
- - Use `key` only for object filters such as `metadata`.
-
- - Use the correct `value` shape for the chosen filter `type`.
-
-
- Operator quick reference by filter `type`:
-
- - `string`: `"="`, `contains`, `does not contain`, `starts with`, `ends
- with`
-
- - `number`: `"="`, `">"`, `"<"`, `">="`, `"<="`
-
- - `datetime`: `"="`, `">"`, `"<"`, `">="`, `"<="`
-
- - `stringOptions`: `any of`, `none of`
-
- - `arrayOptions`: `any of`, `none of`, `all of`
-
- - `stringObject`: same operators as `string`
-
- - `boolean`: `"="`, `"<>"`
-
- - `null`: `is null`, `is not null`
-
-
- Supported columns by target:
-
- - `target=observation`
- - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`
- - `name`: `stringOptions`, operators `any of` / `none of`
- - `environment`: `stringOptions`, operators `any of` / `none of`
- - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`
- - `version`: `string`
- - `traceName`: `stringOptions`, operators `any of` / `none of`
- - `userId`: `string`
- - `sessionId`: `string`
- - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`
- - `metadata`: `stringObject` with `key`
- - `isRootObservation`: `boolean`, operators `=` / `<>`; true when the observation has no parent or is explicitly marked as an application root
- - `parentObservationId`: `null`, operators `is null` / `is not null`
- - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`
- - `toolCalls`: `number`
- - `target=experiment`
- - `datasetId`: `stringOptions`, operators `any of` / `none of`
- Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.
-
- Recovery guidance:
-
- - `invalid_filter_value` with `details.column` but no `invalidValues`:
- the selected `column` is not supported for the chosen `target`
-
- - `invalid_filter_value` with `details.invalidValues`: the selected
- values are not allowed for that column. Replace them with one of
- `details.allowedValues` when provided.
-
- - `invalid_filter_value` for `column=datasetId`: call `GET
- /api/public/v2/datasets`, then retry with dataset `id` values from that
- response.
- unstableDashboardWidgetView:
- title: unstableDashboardWidgetView
+ title: TextPrompt
+ PromptType:
+ title: PromptType
type: string
enum:
- - observations
- - scores-numeric
- - scores-boolean
- - scores-categorical
- unstableDashboardWidgetViewWithLegacy:
- title: unstableDashboardWidgetViewWithLegacy
+ - chat
+ - text
+ BasePrompt:
+ title: BasePrompt
+ type: object
+ properties:
+ name:
+ type: string
+ version:
+ type: integer
+ config: {}
+ labels:
+ type: array
+ items:
+ type: string
+ description: List of deployment labels of this prompt version.
+ tags:
+ type: array
+ items:
+ type: string
+ description: >-
+ List of tags. Used to filter via UI and API. The same across
+ versions of a prompt.
+ commitMessage:
+ type: string
+ nullable: true
+ description: Commit message for this prompt version.
+ resolutionGraph:
+ type: object
+ additionalProperties: true
+ nullable: true
+ description: >-
+ The dependency resolution graph for the current prompt. Null if the
+ prompt has no dependencies or if `resolve=false` was used.
+ required:
+ - name
+ - version
+ - config
+ - labels
+ - tags
+ ChatMessageWithPlaceholders:
+ title: ChatMessageWithPlaceholders
+ oneOf:
+ - $ref: '#/components/schemas/ChatMessage'
+ - $ref: '#/components/schemas/PlaceholderMessage'
+ ChatMessage:
+ title: ChatMessage
+ type: object
+ properties:
+ role:
+ type: string
+ content:
+ type: string
+ type:
+ $ref: '#/components/schemas/ChatMessageType'
+ nullable: true
+ required:
+ - role
+ - content
+ ChatMessageType:
+ title: ChatMessageType
type: string
enum:
- - observations
- - scores-numeric
- - scores-boolean
- - scores-categorical
- - traces
- description: |-
- Widget data view. Responses may include the legacy `traces` value for
- widgets created before this API existed.
- unstableDashboardWidgetChartType:
- title: unstableDashboardWidgetChartType
+ - chatmessage
+ PlaceholderMessage:
+ title: PlaceholderMessage
+ type: object
+ properties:
+ name:
+ type: string
+ type:
+ $ref: '#/components/schemas/PlaceholderMessageType'
+ nullable: true
+ required:
+ - name
+ PlaceholderMessageType:
+ title: PlaceholderMessageType
type: string
enum:
- - LINE_TIME_SERIES
- - AREA_TIME_SERIES
- - BAR_TIME_SERIES
- - HORIZONTAL_BAR
- - VERTICAL_BAR
- - PIE
- - NUMBER
- - HISTOGRAM
- - PIVOT_TABLE
- unstableDashboardWidgetMetricAggregation:
- title: unstableDashboardWidgetMetricAggregation
+ - placeholder
+ TextPrompt:
+ title: TextPrompt
+ type: object
+ properties:
+ prompt:
+ type: string
+ required:
+ - prompt
+ allOf:
+ - $ref: '#/components/schemas/BasePrompt'
+ ChatPrompt:
+ title: ChatPrompt
+ type: object
+ properties:
+ prompt:
+ type: array
+ items:
+ $ref: '#/components/schemas/ChatMessageWithPlaceholders'
+ required:
+ - prompt
+ allOf:
+ - $ref: '#/components/schemas/BasePrompt'
+ CreateChatPromptType:
+ title: CreateChatPromptType
type: string
enum:
- - sum
- - avg
- - count
- - max
- - min
- - p50
- - p75
- - p90
- - p95
- - p99
- - histogram
- - uniq
- unstableDashboardWidgetDimension:
- title: unstableDashboardWidgetDimension
+ - chat
+ CreateTextPromptType:
+ title: CreateTextPromptType
+ type: string
+ enum:
+ - text
+ ServiceProviderConfig:
+ title: ServiceProviderConfig
+ type: object
+ properties:
+ schemas:
+ type: array
+ items:
+ type: string
+ documentationUri:
+ type: string
+ patch:
+ $ref: '#/components/schemas/ScimFeatureSupport'
+ bulk:
+ $ref: '#/components/schemas/BulkConfig'
+ filter:
+ $ref: '#/components/schemas/FilterConfig'
+ changePassword:
+ $ref: '#/components/schemas/ScimFeatureSupport'
+ sort:
+ $ref: '#/components/schemas/ScimFeatureSupport'
+ etag:
+ $ref: '#/components/schemas/ScimFeatureSupport'
+ authenticationSchemes:
+ type: array
+ items:
+ $ref: '#/components/schemas/AuthenticationScheme'
+ meta:
+ $ref: '#/components/schemas/ResourceMeta'
+ required:
+ - schemas
+ - documentationUri
+ - patch
+ - bulk
+ - filter
+ - changePassword
+ - sort
+ - etag
+ - authenticationSchemes
+ - meta
+ ScimFeatureSupport:
+ title: ScimFeatureSupport
type: object
properties:
- field:
+ supported:
+ type: boolean
+ required:
+ - supported
+ BulkConfig:
+ title: BulkConfig
+ type: object
+ properties:
+ supported:
+ type: boolean
+ maxOperations:
+ type: integer
+ maxPayloadSize:
+ type: integer
+ required:
+ - supported
+ - maxOperations
+ - maxPayloadSize
+ FilterConfig:
+ title: FilterConfig
+ type: object
+ properties:
+ supported:
+ type: boolean
+ maxResults:
+ type: integer
+ required:
+ - supported
+ - maxResults
+ ResourceMeta:
+ title: ResourceMeta
+ type: object
+ properties:
+ resourceType:
+ type: string
+ location:
type: string
required:
- - field
- unstableDashboardWidgetMetric:
- title: unstableDashboardWidgetMetric
+ - resourceType
+ - location
+ AuthenticationScheme:
+ title: AuthenticationScheme
type: object
properties:
- measure:
+ name:
type: string
- agg:
- $ref: '#/components/schemas/unstableDashboardWidgetMetricAggregation'
+ description:
+ type: string
+ specUri:
+ type: string
+ type:
+ type: string
+ primary:
+ type: boolean
required:
- - measure
- - agg
- unstableDashboardWidgetFilter:
- title: unstableDashboardWidgetFilter
+ - name
+ - description
+ - specUri
+ - type
+ - primary
+ ResourceTypesResponse:
+ title: ResourceTypesResponse
+ type: object
+ properties:
+ schemas:
+ type: array
+ items:
+ type: string
+ totalResults:
+ type: integer
+ Resources:
+ type: array
+ items:
+ $ref: '#/components/schemas/ResourceType'
+ required:
+ - schemas
+ - totalResults
+ - Resources
+ ResourceType:
+ title: ResourceType
+ type: object
+ properties:
+ schemas:
+ type: array
+ items:
+ type: string
+ nullable: true
+ id:
+ type: string
+ name:
+ type: string
+ endpoint:
+ type: string
+ description:
+ type: string
+ schema:
+ type: string
+ schemaExtensions:
+ type: array
+ items:
+ $ref: '#/components/schemas/SchemaExtension'
+ meta:
+ $ref: '#/components/schemas/ResourceMeta'
+ required:
+ - id
+ - name
+ - endpoint
+ - description
+ - schema
+ - schemaExtensions
+ - meta
+ SchemaExtension:
+ title: SchemaExtension
+ type: object
+ properties:
+ schema:
+ type: string
+ required:
+ type: boolean
+ required:
+ - schema
+ - required
+ SchemasResponse:
+ title: SchemasResponse
+ type: object
+ properties:
+ schemas:
+ type: array
+ items:
+ type: string
+ totalResults:
+ type: integer
+ Resources:
+ type: array
+ items:
+ $ref: '#/components/schemas/SchemaResource'
+ required:
+ - schemas
+ - totalResults
+ - Resources
+ SchemaResource:
+ title: SchemaResource
+ type: object
+ properties:
+ id:
+ type: string
+ name:
+ type: string
+ description:
+ type: string
+ attributes:
+ type: array
+ items: {}
+ meta:
+ $ref: '#/components/schemas/ResourceMeta'
+ required:
+ - id
+ - name
+ - description
+ - attributes
+ - meta
+ ScimUsersListResponse:
+ title: ScimUsersListResponse
+ type: object
+ properties:
+ schemas:
+ type: array
+ items:
+ type: string
+ totalResults:
+ type: integer
+ startIndex:
+ type: integer
+ itemsPerPage:
+ type: integer
+ Resources:
+ type: array
+ items:
+ $ref: '#/components/schemas/ScimUser'
+ required:
+ - schemas
+ - totalResults
+ - startIndex
+ - itemsPerPage
+ - Resources
+ ScimUser:
+ title: ScimUser
+ type: object
+ properties:
+ schemas:
+ type: array
+ items:
+ type: string
+ id:
+ type: string
+ userName:
+ type: string
+ name:
+ $ref: '#/components/schemas/ScimName'
+ emails:
+ type: array
+ items:
+ $ref: '#/components/schemas/ScimEmail'
+ meta:
+ $ref: '#/components/schemas/UserMeta'
+ required:
+ - schemas
+ - id
+ - userName
+ - name
+ - emails
+ - meta
+ UserMeta:
+ title: UserMeta
type: object
- description: >-
- A filter in Langfuse filter-state shape. The `value` shape and the
-
- allowed operators depend on `type`:
-
-
- | `type` | `value` | operators |
-
- |---|---|---|
-
- | `string` | string | `=`, `contains`, `does not contain`, `starts
- with`, `ends with` |
-
- | `number` | number | `=`, `>`, `<`, `>=`, `<=` |
-
- | `datetime` | ISO datetime string | `>`, `<`, `>=`, `<=` |
-
- | `boolean` | boolean | `=`, `<>` |
-
- | `null` | `""` | `is null`, `is not null` |
-
- | `stringOptions` | list of strings | `any of`, `none of` |
-
- | `arrayOptions` | list of strings | `any of`, `none of`, `all of` |
-
- | `categoryOptions` | list of strings (requires `key`) | `any of`, `none
- of` |
-
- | `stringObject` | string (requires `key`, e.g. a metadata key) | same
- as `string` |
-
- | `numberObject` | number (requires `key`, e.g. a score name) | same as
- `number` |
-
- | `booleanObject` | boolean (requires `key`) | `=`, `<>` |
properties:
- column:
- type: string
- operator:
+ resourceType:
type: string
- type:
+ created:
type: string
- value:
nullable: true
- key:
+ lastModified:
type: string
nullable: true
required:
- - column
- - operator
- - type
- unstableDashboardWidgetChartConfig:
- title: unstableDashboardWidgetChartConfig
+ - resourceType
+ ScimName:
+ title: ScimName
type: object
- description: |-
- Chart-specific widget configuration.
-
- `type` must match the top-level `chartType`.
- `row_limit` applies to total-value charts and pivot tables.
- `bins` applies to histograms.
- `defaultSort` applies to pivot tables.
properties:
- type:
- $ref: '#/components/schemas/unstableDashboardWidgetChartType'
- row_limit:
- type: integer
+ formatted:
+ type: string
nullable: true
- show_value_labels:
+ ScimEmail:
+ title: ScimEmail
+ type: object
+ properties:
+ primary:
type: boolean
- nullable: true
- bins:
- type: integer
- nullable: true
- defaultSort:
- $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'
- nullable: true
+ value:
+ type: string
+ type:
+ type: string
required:
+ - primary
+ - value
- type
- unstableDashboardWidgetDefaultSort:
- title: unstableDashboardWidgetDefaultSort
+ EmptyResponse:
+ title: EmptyResponse
+ type: object
+ description: Empty response for 204 No Content responses
+ properties: {}
+ ScoreConfigs:
+ title: ScoreConfigs
type: object
properties:
- column:
- type: string
- order:
- $ref: '#/components/schemas/unstableDashboardWidgetSortOrder'
+ data:
+ type: array
+ items:
+ $ref: '#/components/schemas/ScoreConfig'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
required:
- - column
- - order
- unstableDashboardWidgetSortOrder:
- title: unstableDashboardWidgetSortOrder
- type: string
- enum:
- - ASC
- - DESC
- unstableDashboardWidgetChartConfigInput:
- title: unstableDashboardWidgetChartConfigInput
+ - data
+ - meta
+ CreateScoreConfigRequest:
+ title: CreateScoreConfigRequest
type: object
- description: |-
- Input-side chart config. `type` is optional and defaults to the
- widget's `chartType`; when given it must match.
properties:
- type:
- $ref: '#/components/schemas/unstableDashboardWidgetChartType'
- nullable: true
- row_limit:
- type: integer
- nullable: true
- show_value_labels:
- type: boolean
+ name:
+ type: string
+ description: >-
+ Name of the score config. Max 35 characters. Only letters, numbers,
+ underscores, spaces, periods, parentheses, and hyphens are allowed.
+ dataType:
+ $ref: '#/components/schemas/ScoreConfigDataType'
+ categories:
+ type: array
+ items:
+ $ref: '#/components/schemas/ConfigCategory'
nullable: true
- bins:
- type: integer
+ description: >-
+ Configure custom categories for categorical scores. Pass a list of
+ objects with `label` and `value` properties. Categories are
+ autogenerated for boolean configs and cannot be passed
+ minValue:
+ type: number
+ format: double
nullable: true
- defaultSort:
- $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'
+ description: >-
+ Configure a minimum value for numerical scores. If not set, the
+ minimum value defaults to -∞
+ maxValue:
+ type: number
+ format: double
nullable: true
- unstableCreateDashboardWidgetRequest:
- title: unstableCreateDashboardWidgetRequest
- type: object
- properties:
- name:
- type: string
+ description: >-
+ Configure a maximum value for numerical scores. If not set, the
+ maximum value defaults to +∞
description:
type: string
nullable: true
- description: Defaults to an empty string.
- view:
- $ref: '#/components/schemas/unstableDashboardWidgetView'
- dimensions:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetDimension'
- metrics:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetMetric'
- filters:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetFilter'
- chartType:
- $ref: '#/components/schemas/unstableDashboardWidgetChartType'
- chartConfig:
- $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'
- nullable: true
- description: Defaults to the plain config for `chartType`.
+ description: >-
+ Description is shown across the Langfuse UI and can be used to e.g.
+ explain the config categories in detail, why a numeric range was
+ set, or provide additional context on config name or usage
required:
- name
- - view
- - dimensions
- - metrics
- - filters
- - chartType
- unstableUpdateDashboardWidgetRequest:
- title: unstableUpdateDashboardWidgetRequest
+ - dataType
+ UpdateScoreConfigRequest:
+ title: UpdateScoreConfigRequest
type: object
properties:
- name:
- type: string
+ isArchived:
+ type: boolean
nullable: true
- description:
+ description: The status of the score config showing if it is archived or not
+ name:
type: string
nullable: true
- view:
- $ref: '#/components/schemas/unstableDashboardWidgetView'
- nullable: true
- dimensions:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetDimension'
- nullable: true
- metrics:
+ description: >-
+ Name of the score config. Max 35 characters. Only letters, numbers,
+ underscores, spaces, periods, parentheses, and hyphens are allowed.
+ categories:
type: array
items:
- $ref: '#/components/schemas/unstableDashboardWidgetMetric'
+ $ref: '#/components/schemas/ConfigCategory'
nullable: true
- filters:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetFilter'
+ description: >-
+ Configure custom categories for categorical scores. Pass a list of
+ objects with `label` and `value` properties. Categories are
+ autogenerated for boolean configs and cannot be passed
+ minValue:
+ type: number
+ format: double
nullable: true
- chartType:
- $ref: '#/components/schemas/unstableDashboardWidgetChartType'
+ description: >-
+ Configure a minimum value for numerical scores. If not set, the
+ minimum value defaults to -∞
+ maxValue:
+ type: number
+ format: double
nullable: true
- chartConfig:
- $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'
+ description: >-
+ Configure a maximum value for numerical scores. If not set, the
+ maximum value defaults to +∞
+ description:
+ type: string
nullable: true
- unstableDashboardWidgetList:
- title: unstableDashboardWidgetList
+ description: >-
+ Description is shown across the Langfuse UI and can be used to e.g.
+ explain the config categories in detail, why a numeric range was
+ set, or provide additional context on config name or usage
+ ScoreSubjectTraceV3:
+ title: ScoreSubjectTraceV3
type: object
properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidget'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
+ id:
+ type: string
+ description: The trace ID.
+ required:
+ - id
+ ScoreSubjectObservationV3:
+ title: ScoreSubjectObservationV3
+ type: object
+ properties:
+ id:
+ type: string
+ description: The observation ID.
+ traceId:
+ type: string
+ nullable: true
+ description: The parent trace ID, if available.
required:
- - data
- - meta
- unstableDeleteDashboardWidgetResponse:
- title: unstableDeleteDashboardWidgetResponse
+ - id
+ ScoreSubjectSessionV3:
+ title: ScoreSubjectSessionV3
type: object
properties:
- message:
+ id:
type: string
+ description: The session ID.
required:
- - message
- unstableDashboardWidget:
- title: unstableDashboardWidget
+ - id
+ ScoreSubjectExperimentV3:
+ title: ScoreSubjectExperimentV3
type: object
properties:
id:
type: string
- createdAt:
- type: string
- format: date-time
- updatedAt:
- type: string
- format: date-time
- name:
- type: string
- description:
- type: string
- view:
- $ref: '#/components/schemas/unstableDashboardWidgetViewWithLegacy'
- dimensions:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetDimension'
- metrics:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetMetric'
- filters:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetFilter'
- chartType:
- $ref: '#/components/schemas/unstableDashboardWidgetChartType'
- chartConfig:
- $ref: '#/components/schemas/unstableDashboardWidgetChartConfig'
+ description: The dataset run ID (experiment ID).
required:
- id
- - createdAt
- - updatedAt
- - name
- - description
- - view
- - dimensions
- - metrics
- - filters
- - chartType
- - chartConfig
- unstableDashboardPlacement:
- title: unstableDashboardPlacement
+ ScoreSubjectV3:
+ title: ScoreSubjectV3
oneOf:
- type: object
allOf:
- type: object
properties:
- type:
+ kind:
type: string
enum:
- - widget
- - $ref: '#/components/schemas/unstableWidgetPlacement'
+ - trace
+ - $ref: '#/components/schemas/ScoreSubjectTraceV3'
required:
- - type
- title: UnstableWidgetPlacement
+ - kind
+ title: ScoreSubjectTraceV3
- type: object
allOf:
- type: object
properties:
- type:
+ kind:
type: string
enum:
- - preset
- - $ref: '#/components/schemas/unstablePresetPlacement'
+ - observation
+ - $ref: '#/components/schemas/ScoreSubjectObservationV3'
required:
- - type
- title: UnstablePresetPlacement
- description: |-
- A tile on the dashboard's 12-column grid. `x`/`y` are the tile's
- top-left cell (0-based; `y` grows downward), `width`/`height` its size
- in cells. The UI default tile is 6x6 (half width). Overlapping tiles
- are not rejected; prefer appending below existing tiles (or omit the
- position on create to let the server do it).
- unstableWidgetPlacement:
- title: unstableWidgetPlacement
- type: object
- properties:
- id:
- type: string
- widgetId:
- type: string
- x:
- type: integer
- 'y':
- type: integer
- width:
- type: integer
- height:
- type: integer
- required:
- - id
- - widgetId
- - x
- - 'y'
- - width
- - height
- unstablePresetPlacement:
- title: unstablePresetPlacement
- type: object
- properties:
- id:
- type: string
- presetId:
- type: string
- x:
- type: integer
- 'y':
- type: integer
- width:
- type: integer
- height:
- type: integer
- required:
- - id
- - presetId
- - x
- - 'y'
- - width
- - height
- unstableCreateDashboardPlacementRequest:
- title: unstableCreateDashboardPlacementRequest
- oneOf:
+ - kind
+ title: ScoreSubjectObservationV3
- type: object
allOf:
- type: object
properties:
- type:
+ kind:
type: string
enum:
- - widget
- - $ref: '#/components/schemas/unstableCreateWidgetPlacement'
+ - session
+ - $ref: '#/components/schemas/ScoreSubjectSessionV3'
required:
- - type
- title: UnstableCreateWidgetPlacement
+ - kind
+ title: ScoreSubjectSessionV3
- type: object
allOf:
- type: object
properties:
- type:
+ kind:
type: string
enum:
- - preset
- - $ref: '#/components/schemas/unstableCreatePresetPlacement'
+ - experiment
+ - $ref: '#/components/schemas/ScoreSubjectExperimentV3'
required:
- - type
- title: UnstableCreatePresetPlacement
- unstableCreateWidgetPlacement:
- title: unstableCreateWidgetPlacement
- type: object
- properties:
- id:
- type: string
- nullable: true
- description: Server-generated when omitted.
- widgetId:
- type: string
- x:
- type: integer
- nullable: true
- description: Grid column (12-column grid). Defaults to `0`.
- 'y':
- type: integer
- nullable: true
- description: Grid row. Defaults to the first row below all existing tiles.
- width:
- type: integer
- nullable: true
- description: Width in grid columns. Defaults to `6`.
- height:
- type: integer
- nullable: true
- description: Height in grid rows. Defaults to `6`.
- required:
- - widgetId
- unstableCreatePresetPlacement:
- title: unstableCreatePresetPlacement
+ - kind
+ title: ScoreSubjectExperimentV3
+ description: >-
+ A reference to the entity this score is attached to. Discriminated by
+ "kind" — one of trace, observation, session, or experiment.
+ BaseScoreV3:
+ title: BaseScoreV3
type: object
properties:
id:
type: string
- nullable: true
- description: Server-generated when omitted.
- presetId:
- type: string
- x:
- type: integer
- nullable: true
- description: Grid column (12-column grid). Defaults to `0`.
- 'y':
- type: integer
- nullable: true
- description: Grid row. Defaults to the first row below all existing tiles.
- width:
- type: integer
- nullable: true
- description: Width in grid columns. Defaults to `6`.
- height:
- type: integer
- nullable: true
- description: Height in grid rows. Defaults to `6`.
- required:
- - presetId
- unstableUpdateDashboardPlacementRequest:
- title: unstableUpdateDashboardPlacementRequest
- type: object
- properties:
- x:
- type: integer
- nullable: true
- description: Grid column (12-column grid).
- 'y':
- type: integer
- nullable: true
- description: Grid row.
- width:
- type: integer
- nullable: true
- description: Width in grid columns.
- height:
- type: integer
- nullable: true
- description: Height in grid rows.
- unstableDeleteDashboardPlacementResponse:
- title: unstableDeleteDashboardPlacementResponse
- type: object
- properties:
- message:
+ projectId:
type: string
- required:
- - message
- unstableDashboardDefinition:
- title: unstableDashboardDefinition
- type: object
- properties:
- widgets:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardPlacement'
- required:
- - widgets
- unstableDashboard:
- title: unstableDashboard
- type: object
- properties:
- id:
+ name:
+ type: string
+ source:
+ $ref: '#/components/schemas/ScoreSource'
+ timestamp:
+ type: string
+ format: date-time
+ environment:
type: string
+ description: The environment from which this score originated.
createdAt:
type: string
format: date-time
updatedAt:
type: string
format: date-time
- name:
+ comment:
type: string
- description:
+ nullable: true
+ description: >-
+ Optional comment attached to the score. Present when "details" is
+ included in the fields parameter.
+ configId:
type: string
- definition:
- $ref: '#/components/schemas/unstableDashboardDefinition'
- filters:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetFilter'
- description: Dashboard-level filters applied to all widgets on the dashboard.
+ nullable: true
+ description: >-
+ The score config ID, if this score was created from a config.
+ Present when "details" is included in the fields parameter.
+ metadata:
+ type: object
+ additionalProperties: true
+ nullable: true
+ description: >-
+ Arbitrary metadata attached to the score. Present when "details" is
+ included in the fields parameter.
+ authorUserId:
+ type: string
+ nullable: true
+ description: >-
+ The user who created this score, if available. Present when
+ "annotation" is included in the fields parameter.
+ queueId:
+ type: string
+ nullable: true
+ description: >-
+ The annotation queue this score belongs to, if any. Present when
+ "annotation" is included in the fields parameter.
+ subject:
+ $ref: '#/components/schemas/ScoreSubjectV3'
+ nullable: true
+ description: >-
+ The entity this score is attached to (trace, observation, session,
+ or experiment). Present when "subject" is included in the fields
+ parameter.
required:
- id
+ - projectId
+ - name
+ - source
+ - timestamp
+ - environment
- createdAt
- updatedAt
- - name
- - description
- - definition
- - filters
- unstableDashboardList:
- title: unstableDashboardList
+ NumericScoreV3:
+ title: NumericScoreV3
type: object
properties:
- data:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboard'
- meta:
- $ref: '#/components/schemas/utilsMetaResponse'
+ value:
+ type: number
+ format: double
+ description: The numeric value of the score.
required:
- - data
- - meta
- unstableCreateDashboardRequest:
- title: unstableCreateDashboardRequest
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV3'
+ BooleanScoreV3:
+ title: BooleanScoreV3
type: object
properties:
- name:
- type: string
- description:
- type: string
- nullable: true
- definition:
- $ref: '#/components/schemas/unstableDashboardDefinition'
- nullable: true
- filters:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetFilter'
- nullable: true
+ value:
+ type: boolean
+ description: The boolean value of the score.
required:
- - name
- unstableUpdateDashboardRequest:
- title: unstableUpdateDashboardRequest
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV3'
+ CategoricalScoreV3:
+ title: CategoricalScoreV3
type: object
properties:
- name:
+ value:
type: string
- nullable: true
- description:
+ description: The string category value of the score.
+ required:
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV3'
+ TextScoreV3:
+ title: TextScoreV3
+ type: object
+ properties:
+ value:
type: string
- nullable: true
- definition:
- $ref: '#/components/schemas/unstableDashboardDefinition'
- nullable: true
- filters:
- type: array
- items:
- $ref: '#/components/schemas/unstableDashboardWidgetFilter'
- nullable: true
- unstableDeleteDashboardResponse:
- title: unstableDeleteDashboardResponse
+ description: The text content of the score.
+ required:
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV3'
+ CorrectionScoreV3:
+ title: CorrectionScoreV3
type: object
properties:
- message:
+ value:
type: string
+ description: The correction content of the score. Empty string if not set.
required:
- - message
- unstablePublicApiErrorCode:
- title: unstablePublicApiErrorCode
- type: string
- enum:
- - authentication_failed
- - access_denied
- - invalid_request
- - invalid_query
- - invalid_body
- - invalid_filter_value
- - invalid_json_path
- - invalid_variable_mapping
- - missing_variable_mapping
- - duplicate_variable_mapping
- - resource_not_found
- - name_conflict
- - evaluator_preflight_failed
- - conflict
- - unprocessable_content
- - rate_limited
- - method_not_allowed
- - internal_error
- description: >-
- Machine-readable error code returned by the unstable evaluators API.
-
-
- SDKs, CLIs, and agents should branch on `code` rather than parsing the
- human-readable `message`.
-
- The HTTP status still indicates the broad error class, while `code`
- gives the specific failure reason.
- unstablePublicApiValidationIssue:
- title: unstablePublicApiValidationIssue
+ - value
+ allOf:
+ - $ref: '#/components/schemas/BaseScoreV3'
+ ScoreV3:
+ title: ScoreV3
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - NUMERIC
+ - $ref: '#/components/schemas/NumericScoreV3'
+ required:
+ - dataType
+ title: NumericScoreV3
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - BOOLEAN
+ - $ref: '#/components/schemas/BooleanScoreV3'
+ required:
+ - dataType
+ title: BooleanScoreV3
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CATEGORICAL
+ - $ref: '#/components/schemas/CategoricalScoreV3'
+ required:
+ - dataType
+ title: CategoricalScoreV3
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - TEXT
+ - $ref: '#/components/schemas/TextScoreV3'
+ required:
+ - dataType
+ title: TextScoreV3
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CORRECTION
+ - $ref: '#/components/schemas/CorrectionScoreV3'
+ required:
+ - dataType
+ title: CorrectionScoreV3
+ GetScoresV3Meta:
+ title: GetScoresV3Meta
type: object
- description: >-
- One validation issue returned for malformed request bodies or query
- parameters.
-
-
- This mirrors the most important parts of a Zod issue: a machine-readable
- `code`,
-
- a human-readable `message`, and a structured `path`.
properties:
- code:
+ limit:
+ type: integer
+ cursor:
type: string
+ nullable: true
description: >-
- Machine-readable validation issue code emitted by the server
- validator.
- message:
- type: string
- description: Human-readable explanation of the validation failure.
- path:
- type: array
- items: {}
- description: Path to the invalid field, for example `["mapping", 0, "jsonPath"]`.
+ URL-safe base64 (base64url) cursor for the next page. Absent when
+ there are no more results.
required:
- - code
- - message
- - path
- unstablePublicApiErrorDetails:
- title: unstablePublicApiErrorDetails
+ - limit
+ GetScoresV3Response:
+ title: GetScoresV3Response
type: object
- description: >-
- Optional structured context attached to an unstable-evals error.
-
-
- The populated fields depend on the error `code`:
-
- - request parsing failures populate `issues`
-
- - filter validation failures populate `field`, `column`,
- `invalidValues`, and `allowedValues`
-
- - variable mapping failures populate `field`, `variable`, or `variables`
-
- - JSONPath validation failures populate `field`, `variable`, and `value`
-
- - evaluator preflight failures populate `evaluatorName`, `provider`, and
- `model`
-
- - rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and
- `resetAt`
properties:
- issues:
+ data:
type: array
items:
- $ref: '#/components/schemas/unstablePublicApiValidationIssue'
- nullable: true
- description: Validation issues for malformed request bodies or query parameters.
- field:
+ $ref: '#/components/schemas/ScoreV3'
+ meta:
+ $ref: '#/components/schemas/GetScoresV3Meta'
+ required:
+ - data
+ - meta
+ CreateScoreRequest:
+ title: CreateScoreRequest
+ type: object
+ properties:
+ id:
type: string
nullable: true
- description: >-
- Path-like reference to the failing field, for example
- `mapping[1].jsonPath`.
- column:
+ traceId:
type: string
nullable: true
- description: Filter column that failed validation.
- invalidValues:
- type: array
- items:
- type: string
- nullable: true
- description: Unsupported values supplied by the caller.
- allowedValues:
- type: array
- items:
- type: string
+ sessionId:
+ type: string
nullable: true
- description: Allowed values for the failing filter column.
- variable:
+ observationId:
type: string
nullable: true
- description: Evaluator variable involved in the failure.
- variables:
- type: array
- items:
- type: string
+ datasetRunId:
+ type: string
nullable: true
- description: >-
- Multiple evaluator variables involved in the failure, for example
- missing mappings.
+ name:
+ type: string
value:
+ $ref: '#/components/schemas/CreateScoreValue'
+ description: >-
+ The value of the score. Must be passed as string for categorical and
+ text scores, and numeric for boolean and numeric scores. Boolean
+ score values must equal either 1 or 0 (true or false). Text score
+ values must be between 1 and 500 characters.
+ comment:
type: string
nullable: true
- description: Raw invalid value supplied by the caller.
- evaluatorName:
- type: string
+ metadata:
+ type: object
+ additionalProperties: true
nullable: true
- description: Evaluator name used during preflight validation.
- provider:
+ environment:
type: string
nullable: true
- description: Provider resolved during evaluator preflight, if any.
- model:
+ description: >-
+ The environment of the score. Can be any lowercase alphanumeric
+ string with hyphens and underscores that does not start with
+ 'langfuse'.
+ queueId:
type: string
nullable: true
- description: Model resolved during evaluator preflight, if any.
- retryAfterSeconds:
- type: integer
- nullable: true
- description: Suggested retry delay for rate-limited requests.
- limit:
- type: integer
- nullable: true
description: >-
- Numeric limit associated with the failure, for example the active
- evaluation-rule cap or the current rate-limit window.
- remaining:
- type: integer
+ The annotation queue referenced by the score. Indicates if score was
+ initially created while processing annotation queue.
+ dataType:
+ $ref: '#/components/schemas/ScoreDataType'
nullable: true
- description: Remaining requests in the current rate-limit window.
- resetAt:
+ description: >-
+ The data type of the score. When passing a configId this field is
+ inferred. Otherwise, this field must be passed or will default to
+ numeric.
+ configId:
type: string
nullable: true
- description: ISO-8601 timestamp when the current rate-limit window resets.
- unstablePublicApiError:
- title: unstablePublicApiError
- type: object
- description: >-
- Standard error envelope for the unstable evaluators API.
-
-
- Response handling guidance:
-
- - Use the HTTP status code for the broad class of failure.
-
- - Use `code` for precise branching in SDKs, CLIs, or agents.
-
- - Inspect `details` for field-level validation context such as invalid
- filter values, malformed JSONPath expressions, or missing variable
- mappings.
-
- - Retry only after fixing the specific issue described by `code` and
- `details`.
- properties:
- message:
- type: string
- description: Human-readable description of the failure.
- code:
- $ref: '#/components/schemas/unstablePublicApiErrorCode'
- description: Stable machine-readable error code.
- details:
- $ref: '#/components/schemas/unstablePublicApiErrorDetails'
+ description: >-
+ Reference a score config on a score. The unique langfuse identifier
+ of a score config. When passing this field, the dataType and
+ stringValue fields are automatically populated.
+ source:
+ $ref: '#/components/schemas/CreateScoreSource'
nullable: true
description: >-
- Optional structured error context. Inspect the populated fields
- based on `code`.
+ The source of the score. Defaults to API. Set to ANNOTATION to
+ prefill scores (e.g. from an LLM) for a human reviewer to verify in
+ an annotation queue. When source is ANNOTATION, a configId is
+ required unless dataType is CORRECTION. EVAL is reserved for
+ internal evaluator outputs and is not accepted on this endpoint.
required:
- - message
- - code
- unstableEvaluationRuleBase:
- title: unstableEvaluationRuleBase
+ - name
+ - value
+ CreateScoreSource:
+ title: CreateScoreSource
+ type: string
+ enum:
+ - API
+ - ANNOTATION
+ description: |-
+ Source values accepted when creating a score via the public REST API.
+ EVAL is reserved for internal evaluator outputs and is intentionally not
+ exposed here — use commons.ScoreSource when reading scores.
+ CreateScoreResponse:
+ title: CreateScoreResponse
type: object
- description: >-
- Live evaluation rule for incoming data.
-
-
- An evaluation rule answers:
-
- - which evaluator should be used
-
- - which target objects should trigger scoring
-
- - how often scoring should run
-
- - which target fields should populate each prompt variable
-
- - whether the deployment is active, inactive, or paused
-
-
- Important status semantics:
-
- - `enabled` is the desired on/off setting from the client
-
- - `status` is the effective runtime state after Langfuse applies
- validation and blocking rules
-
- - `enabled=true` with `status=paused` means the rule should run, but
- Langfuse has paused it until the underlying problem is fixed
properties:
id:
type: string
- description: Stable evaluation rule identifier.
- name:
+ description: The id of the created object in Langfuse
+ required:
+ - id
+ GetScoresResponseTraceData:
+ title: GetScoresResponseTraceData
+ type: object
+ properties:
+ userId:
type: string
- description: >-
- Human-readable deployment name. This is independent from the
- evaluator name.
- evaluator:
- $ref: '#/components/schemas/unstableEvaluationRuleEvaluator'
nullable: true
- description: >-
- Deprecated compatibility alias for `evaluators[0].evaluator`, or
- `null`
-
- when the rule currently has no evaluator assignments.
-
-
- `id` identifies the evaluator family. The rule automatically uses
- the
-
- latest available version of that evaluator.
- enabled:
- type: boolean
- description: Desired enabled state configured by the client.
- status:
- $ref: '#/components/schemas/unstableEvaluationRuleStatus'
- description: >-
- Effective runtime status after Langfuse applies validation and
- blocking rules.
- pausedReason:
- type: string
+ description: The user ID associated with the trace referenced by score
+ tags:
+ type: array
+ items:
+ type: string
nullable: true
- description: Machine-readable reason when `status=paused`, otherwise `null`.
- pausedMessage:
+ description: A list of tags associated with the trace referenced by score
+ environment:
type: string
nullable: true
- description: Human-readable explanation when `status=paused`, otherwise `null`.
- sampling:
- type: number
- format: double
- description: |-
- Fraction of matching target objects that should be evaluated.
-
- Must be greater than `0` and less than or equal to `1`.
- - `1` means evaluate every matching target.
- - `0.25` means evaluate approximately 25% of matching targets.
- createdAt:
- type: string
- format: date-time
- description: Timestamp when the evaluation rule was created.
- updatedAt:
+ description: The environment of the trace referenced by score
+ sessionId:
type: string
- format: date-time
- description: Timestamp when the evaluation rule was last updated.
- required:
- - id
- - name
- - evaluator
- - enabled
- - status
- - pausedReason
- - pausedMessage
- - sampling
- - createdAt
- - updatedAt
- unstableEvaluationRule:
- title: unstableEvaluationRule
+ nullable: true
+ description: The session ID associated with the trace referenced by score
+ GetScoresResponseDataNumeric:
+ title: GetScoresResponseDataNumeric
type: object
properties:
- evaluators:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleEvaluatorAssignment'
- description: >-
- Evaluators attached to this rule in deterministic assignment order.
- A `null` mapping inherits the evaluator version's default mapping.
- target:
- $ref: '#/components/schemas/unstableEvaluationRuleTarget'
- description: Target object type that should trigger scoring.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleFilter'
- description: >-
- List of filter conditions used to decide whether a target should be
- evaluated.
- mapping:
- type: array
- items:
- $ref: '#/components/schemas/unstablePromptVariableMappingRead'
- description: >-
- Deprecated compatibility alias containing the effective mapping for
- `evaluators[0]`.
- required:
- - evaluators
- - target
- - filter
- - mapping
+ trace:
+ $ref: '#/components/schemas/GetScoresResponseTraceData'
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/NumericScore'
+ GetScoresResponseDataCategorical:
+ title: GetScoresResponseDataCategorical
+ type: object
+ properties:
+ trace:
+ $ref: '#/components/schemas/GetScoresResponseTraceData'
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/CategoricalScore'
+ GetScoresResponseDataBoolean:
+ title: GetScoresResponseDataBoolean
+ type: object
+ properties:
+ trace:
+ $ref: '#/components/schemas/GetScoresResponseTraceData'
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/BooleanScore'
+ GetScoresResponseDataCorrection:
+ title: GetScoresResponseDataCorrection
+ type: object
+ properties:
+ trace:
+ $ref: '#/components/schemas/GetScoresResponseTraceData'
+ nullable: true
+ allOf:
+ - $ref: '#/components/schemas/CorrectionScore'
+ GetScoresResponseDataText:
+ title: GetScoresResponseDataText
+ type: object
+ properties:
+ trace:
+ $ref: '#/components/schemas/GetScoresResponseTraceData'
+ nullable: true
allOf:
- - $ref: '#/components/schemas/unstableEvaluationRuleBase'
- unstableReadableV2EvaluationRule:
- title: unstableReadableV2EvaluationRule
+ - $ref: '#/components/schemas/TextScore'
+ GetScoresResponseData:
+ title: GetScoresResponseData
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - NUMERIC
+ - $ref: '#/components/schemas/GetScoresResponseDataNumeric'
+ required:
+ - dataType
+ title: GetScoresResponseDataNumeric
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CATEGORICAL
+ - $ref: '#/components/schemas/GetScoresResponseDataCategorical'
+ required:
+ - dataType
+ title: GetScoresResponseDataCategorical
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - BOOLEAN
+ - $ref: '#/components/schemas/GetScoresResponseDataBoolean'
+ required:
+ - dataType
+ title: GetScoresResponseDataBoolean
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - CORRECTION
+ - $ref: '#/components/schemas/GetScoresResponseDataCorrection'
+ required:
+ - dataType
+ title: GetScoresResponseDataCorrection
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ dataType:
+ type: string
+ enum:
+ - TEXT
+ - $ref: '#/components/schemas/GetScoresResponseDataText'
+ required:
+ - dataType
+ title: GetScoresResponseDataText
+ GetScoresResponse:
+ title: GetScoresResponse
type: object
properties:
- evaluators:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleEvaluatorAssignment'
- description: >-
- Evaluators attached to this rule in deterministic assignment order.
- A `null` mapping inherits the evaluator version's default mapping.
- target:
- $ref: '#/components/schemas/unstableEvaluationRuleTarget'
- description: Target object type that should trigger scoring.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/EvaluationRuleReadFilter'
- description: >-
- List of stored filter conditions returned verbatim. Historical
- filters remain readable even when the current write contract no
- longer accepts their shape.
- mapping:
+ data:
type: array
items:
- $ref: '#/components/schemas/unstablePromptVariableMappingRead'
- description: >-
- Deprecated compatibility alias containing the effective mapping for
- `evaluators[0]`.
+ $ref: '#/components/schemas/GetScoresResponseData'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
required:
- - evaluators
- - target
- - filter
- - mapping
- allOf:
- - $ref: '#/components/schemas/unstableEvaluationRuleBase'
- unstableLegacyEvaluationRule:
- title: unstableLegacyEvaluationRule
+ - data
+ - meta
+ PaginatedSessions:
+ title: PaginatedSessions
type: object
- description: >-
- **Deprecated:** Legacy trace- or dataset-level evaluation rule returned
- by list and get for migration.
-
-
- This resource is read-only through the unstable public API. Its mapping
- preserves the trace, dataset item, or named observation selected for
- each prompt variable.
properties:
- evaluators:
- type: array
- items:
- $ref: >-
- #/components/schemas/unstableLegacyEvaluationRuleEvaluatorAssignment
- description: Evaluators attached to this rule in deterministic assignment order.
- target:
- $ref: '#/components/schemas/unstableLegacyEvaluationRuleTarget'
- delay:
- type: integer
- description: Delay in milliseconds before the legacy evaluation job runs.
- timeScope:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleTimeScope'
- description: >-
- Whether the legacy rule evaluates newly ingested data, existing
- data, or both.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleFilter'
- description: Stored filters used by the legacy trace or dataset rule.
- mapping:
+ data:
type: array
items:
- $ref: '#/components/schemas/unstableLegacyPromptVariableMapping'
- description: >-
- Stored variable mappings, including the trace, dataset item, or
- named observation selected for each variable.
+ $ref: '#/components/schemas/Session'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
required:
- - evaluators
- - target
- - delay
- - timeScope
- - filter
- - mapping
- allOf:
- - $ref: '#/components/schemas/unstableEvaluationRuleBase'
- unstableReadableEvaluationRule:
- title: unstableReadableEvaluationRule
- oneOf:
- - $ref: '#/components/schemas/unstableReadableV2EvaluationRule'
- - $ref: '#/components/schemas/unstableLegacyEvaluationRule'
- description: >-
- Evaluation rule returned by list and get, including read-only legacy
- trace and dataset rules.
- unstableEvaluationRules:
- title: unstableEvaluationRules
+ - data
+ - meta
+ Traces:
+ title: Traces
type: object
- description: Paginated list of evaluation rules.
properties:
data:
type: array
items:
- $ref: '#/components/schemas/unstableReadableEvaluationRule'
- description: Evaluation rules in the current page.
+ $ref: '#/components/schemas/TraceWithDetails'
meta:
$ref: '#/components/schemas/utilsMetaResponse'
- description: Standard pagination metadata.
+ _deprecation:
+ $ref: '#/components/schemas/Deprecation'
+ nullable: true
required:
- data
- meta
- unstableCreateEvaluationRuleRequest:
- title: unstableCreateEvaluationRuleRequest
- oneOf:
- - $ref: '#/components/schemas/unstableCreateLlmAsJudgeEvaluationRuleRequest'
- - $ref: '#/components/schemas/unstableCreateCodeEvaluationRuleRequest'
- - $ref: >-
- #/components/schemas/unstableCreateEvaluationRuleWithEvaluatorsRequest
- description: >-
- Request body for creating an evaluation rule.
-
-
- Checklist for agents and SDK clients:
-
- - reference an existing evaluator family by `evaluator.name` and
- `evaluator.type`
-
- - choose `target=observation` or `target=experiment`
-
- - if `target=experiment` and you want a dataset filter, call `GET
- /api/public/v2/datasets` first and use dataset `id` values in
- `filter[].value`
-
- - for `llm_as_judge`, fetch or inspect the evaluator first and provide a
- complete mapping for every prompt variable
-
- - for `code`, do not send variables or mappings; Langfuse stores the
- fixed code runtime mapping automatically
-
- - optionally narrow execution with `filter`
-
- - set `enabled=true` only when you want live execution immediately
- unstableCreateEvaluationRuleWithEvaluatorsRequest:
- title: unstableCreateEvaluationRuleWithEvaluatorsRequest
+ DeleteTraceResponse:
+ title: DeleteTraceResponse
type: object
properties:
- name:
+ message:
type: string
- description: Human-readable deployment name.
- evaluators:
- type: array
- items:
- $ref: >-
- #/components/schemas/unstableCreateEvaluationRuleEvaluatorAssignment
- description: >-
- One or more evaluator assignments. Providing the deprecated
- top-level `evaluator` or `mapping` fields alongside this is rejected
- with `400`.
-
-
- Multiple assignments are supported on writable targets.
- target:
- $ref: '#/components/schemas/unstableEvaluationRuleTarget'
- enabled:
- type: boolean
- sampling:
- type: number
- format: double
- nullable: true
- filter:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleFilter'
- nullable: true
required:
- - name
- - evaluators
- - target
- - enabled
- unstableCreateLlmAsJudgeEvaluationRuleRequest:
- title: unstableCreateLlmAsJudgeEvaluationRuleRequest
+ - message
+ Sort:
+ title: Sort
type: object
properties:
- name:
+ id:
type: string
- description: Human-readable deployment name.
- evaluator:
- $ref: >-
- #/components/schemas/unstableLlmAsJudgeEvaluationRuleEvaluatorReference
- description: >-
- LLM-as-judge evaluator family to use.
-
-
- Use `name` and `type` from the evaluator endpoints. If `type` is
- omitted, Langfuse defaults it to `llm_as_judge` for backwards
- compatibility.
-
- Langfuse resolves that family to its latest version before saving
- the rule.
- target:
- $ref: '#/components/schemas/unstableEvaluationRuleTarget'
- description: Target object type to evaluate.
- enabled:
- type: boolean
- description: Whether the deployment should be active immediately after creation.
- sampling:
- type: number
- format: double
- nullable: true
- description: Optional sampling fraction. Defaults to `1`.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleFilter'
- nullable: true
- description: >-
- Optional filter list.
-
-
- Omit or pass an empty list to evaluate all matching targets for the
- selected `target`.
-
- Each filter object must use a column that is valid for that
- `target`.
-
- For `target=experiment`, `column=datasetId` expects dataset `id`
- values from `GET /api/public/v2/datasets`, not dataset names.
- mapping:
- type: array
- items:
- $ref: '#/components/schemas/unstablePromptVariableMappingInput'
- description: |-
- LLM-as-judge variable mappings.
-
- Every prompt variable must appear exactly once.
- Build this list from the evaluator's `variables` array.
required:
- - name
- - evaluator
- - target
- - enabled
- - mapping
- unstableCreateCodeEvaluationRuleRequest:
- title: unstableCreateCodeEvaluationRuleRequest
+ - id
+ unstableDashboardWidgetView:
+ title: unstableDashboardWidgetView
+ type: string
+ enum:
+ - observations
+ - scores-numeric
+ - scores-boolean
+ - scores-categorical
+ unstableDashboardWidgetViewWithLegacy:
+ title: unstableDashboardWidgetViewWithLegacy
+ type: string
+ enum:
+ - observations
+ - scores-numeric
+ - scores-boolean
+ - scores-categorical
+ - traces
+ description: |-
+ Widget data view. Responses may include the legacy `traces` value for
+ widgets created before this API existed.
+ unstableDashboardWidgetChartType:
+ title: unstableDashboardWidgetChartType
+ type: string
+ enum:
+ - LINE_TIME_SERIES
+ - AREA_TIME_SERIES
+ - BAR_TIME_SERIES
+ - HORIZONTAL_BAR
+ - VERTICAL_BAR
+ - PIE
+ - NUMBER
+ - HISTOGRAM
+ - PIVOT_TABLE
+ unstableDashboardWidgetMetricAggregation:
+ title: unstableDashboardWidgetMetricAggregation
+ type: string
+ enum:
+ - sum
+ - avg
+ - count
+ - max
+ - min
+ - p50
+ - p75
+ - p90
+ - p95
+ - p99
+ - histogram
+ - uniq
+ unstableDashboardWidgetDimension:
+ title: unstableDashboardWidgetDimension
+ type: object
+ properties:
+ field:
+ type: string
+ required:
+ - field
+ unstableDashboardWidgetMetric:
+ title: unstableDashboardWidgetMetric
type: object
properties:
- name:
+ measure:
type: string
- description: Human-readable deployment name.
- evaluator:
- $ref: '#/components/schemas/unstableCodeEvaluationRuleEvaluatorReference'
- description: >-
- Code evaluator family to use.
-
-
- Use `name` and `type` from the evaluator endpoints.
-
- Langfuse resolves that family to its latest version before saving
- the rule.
- target:
- $ref: '#/components/schemas/unstableEvaluationRuleTarget'
- description: Target object type to evaluate.
- enabled:
- type: boolean
- description: Whether the deployment should be active immediately after creation.
- sampling:
- type: number
- format: double
- nullable: true
- description: Optional sampling fraction. Defaults to `1`.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleFilter'
- nullable: true
- description: >-
- Optional filter list.
-
-
- Omit or pass an empty list to evaluate all matching targets for the
- selected `target`.
-
- Each filter object must use a column that is valid for that
- `target`.
-
- For `target=experiment`, `column=datasetId` expects dataset `id`
- values from `GET /api/public/v2/datasets`, not dataset names.
+ agg:
+ $ref: '#/components/schemas/unstableDashboardWidgetMetricAggregation'
required:
- - name
- - evaluator
- - target
- - enabled
- unstableUpdateEvaluationRuleRequest:
- title: unstableUpdateEvaluationRuleRequest
+ - measure
+ - agg
+ unstableDashboardWidgetFilter:
+ title: unstableDashboardWidgetFilter
type: object
description: >-
- Partial update body for an evaluation rule.
+ A filter in Langfuse filter-state shape. The `value` shape and the
+ allowed operators depend on `type`:
- Provide only the fields you want to change.
- An empty body is rejected.
+ | `type` | `value` | operators |
+ |---|---|---|
- Practical guidance:
+ | `string` | string | `=`, `contains`, `does not contain`, `starts
+ with`, `ends with` |
- - If you only want to rename the rule or change sampling, send just
- those fields.
+ | `number` | number | `=`, `>`, `<`, `>=`, `<=` |
- - To add, remove, or remap evaluators, send `evaluators`. It replaces
- the whole assignment set, so include every evaluator the rule should
- keep.
+ | `datetime` | ISO datetime string | `>`, `<`, `>=`, `<=` |
- - `evaluators` cannot be combined with the deprecated
- `evaluator`/`mapping` pair, which only ever addressed the first
- assignment.
+ | `boolean` | boolean | `=`, `<>` |
- - If you change to an LLM-as-judge `evaluator`, send a fresh `mapping`
- unless you are certain the existing mapping still matches the prompt
- variables.
+ | `null` | `""` | `is null`, `is not null` |
- - If you change `target` for an LLM-as-judge rule, usually send both
- `filter` and `mapping` in the same request.
+ | `stringOptions` | list of strings | `any of`, `none of` |
- - For code evaluator rules, omit `mapping`; Langfuse stores the fixed
- code runtime mapping automatically.
+ | `arrayOptions` | list of strings | `any of`, `none of`, `all of` |
- - If you change an experiment `datasetId` filter, call `GET
- /api/public/v2/datasets` and use dataset `id` values from that response.
- properties:
- name:
- type: string
- nullable: true
- description: Updated deployment name.
- evaluators:
- type: array
- items:
- $ref: >-
- #/components/schemas/unstableCreateEvaluationRuleEvaluatorAssignment
- nullable: true
- description: >-
- Full replacement of the rule's evaluator assignments: entries that
- are
+ | `categoryOptions` | list of strings (requires `key`) | `any of`, `none
+ of` |
- not listed are detached.
+ | `stringObject` | string (requires `key`, e.g. a metadata key) | same
+ as `string` |
+ | `numberObject` | number (requires `key`, e.g. a score name) | same as
+ `number` |
- Mutually exclusive with the deprecated `evaluator` and `mapping`
- fields.
- evaluator:
- $ref: '#/components/schemas/unstableEvaluationRuleEvaluatorReference'
+ | `booleanObject` | boolean (requires `key`) | `=`, `<>` |
+ properties:
+ column:
+ type: string
+ operator:
+ type: string
+ type:
+ type: string
+ value:
nullable: true
- description: >-
- Deprecated single-evaluator alias: updates the first assignment
- only. Prefer `evaluators`.
-
-
- Langfuse resolves the provided evaluator family to its latest
- version before saving the rule.
+ key:
+ type: string
+ nullable: true
+ required:
+ - column
+ - operator
+ - type
+ unstableDashboardWidgetChartConfig:
+ title: unstableDashboardWidgetChartConfig
+ type: object
+ description: |-
+ Chart-specific widget configuration.
- A rule's evaluator type cannot be changed: provide `name` for an
- evaluator family of the rule's current type. To use a different
- evaluator type, create a new rule.
- target:
- $ref: '#/components/schemas/unstableEvaluationRuleTarget'
+ `type` must match the top-level `chartType`.
+ `row_limit` applies to total-value charts and pivot tables.
+ `bins` applies to histograms.
+ `defaultSort` applies to pivot tables.
+ properties:
+ type:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'
+ row_limit:
+ type: integer
nullable: true
- description: Updated target object type.
- enabled:
+ show_value_labels:
type: boolean
nullable: true
- description: Updated desired enabled state.
- sampling:
- type: number
- format: double
- nullable: true
- description: Updated sampling fraction.
- filter:
- type: array
- items:
- $ref: '#/components/schemas/unstableEvaluationRuleFilter'
+ bins:
+ type: integer
nullable: true
- description: >-
- Updated filter list.
-
-
- For `target=experiment`, `column=datasetId` expects dataset `id`
- values from `GET /api/public/v2/datasets`, not dataset names.
- mapping:
- type: array
- items:
- $ref: '#/components/schemas/unstablePromptVariableMappingInput'
+ defaultSort:
+ $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'
nullable: true
- description: >-
- Updated LLM-as-judge variable mappings.
-
-
- Do not send this field for code evaluator rules. Langfuse stores the
- fixed code runtime mapping automatically and returns it in the
- response.
- unstableDeleteEvaluationRuleResponse:
- title: unstableDeleteEvaluationRuleResponse
- type: object
- description: Confirmation response returned after successful deletion.
- properties:
- message:
- type: string
- description: Always `Evaluation rule successfully deleted`.
required:
- - message
- unstableEvaluationRuleEvaluatorReference:
- title: unstableEvaluationRuleEvaluatorReference
+ - type
+ unstableDashboardWidgetDefaultSort:
+ title: unstableDashboardWidgetDefaultSort
type: object
- description: >-
- Evaluator family reference used when updating an evaluation rule.
-
-
- `name` identifies the evaluator family in the authenticated project
- context.
-
- A rule's evaluator type cannot be changed, so this reference does not
- accept a `type`; the family must match the rule's current evaluator
- type.
properties:
- name:
+ column:
type: string
- description: Evaluator family name.
+ order:
+ $ref: '#/components/schemas/unstableDashboardWidgetSortOrder'
required:
- - name
- unstableLlmAsJudgeEvaluationRuleEvaluatorReference:
- title: unstableLlmAsJudgeEvaluationRuleEvaluatorReference
+ - column
+ - order
+ unstableDashboardWidgetSortOrder:
+ title: unstableDashboardWidgetSortOrder
+ type: string
+ enum:
+ - ASC
+ - DESC
+ unstableDashboardWidgetChartConfigInput:
+ title: unstableDashboardWidgetChartConfigInput
type: object
- description: >-
- LLM-as-judge evaluator family reference used when creating an evaluation
- rule.
+ description: |-
+ Input-side chart config. `type` is optional and defaults to the
+ widget's `chartType`; when given it must match.
properties:
- name:
- type: string
- description: Evaluator family name.
type:
- $ref: '#/components/schemas/unstableLlmAsJudgeEvaluatorType'
+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'
nullable: true
- description: Evaluator type. Defaults to `llm_as_judge` when omitted.
- required:
- - name
- unstableCodeEvaluationRuleEvaluatorReference:
- title: unstableCodeEvaluationRuleEvaluatorReference
+ row_limit:
+ type: integer
+ nullable: true
+ show_value_labels:
+ type: boolean
+ nullable: true
+ bins:
+ type: integer
+ nullable: true
+ defaultSort:
+ $ref: '#/components/schemas/unstableDashboardWidgetDefaultSort'
+ nullable: true
+ unstableCreateDashboardWidgetRequest:
+ title: unstableCreateDashboardWidgetRequest
type: object
- description: Code evaluator family reference used when creating an evaluation rule.
properties:
name:
type: string
- description: Evaluator family name.
- type:
+ description:
type: string
- const: code
- description: Must be `code`.
+ nullable: true
+ description: Defaults to an empty string.
+ view:
+ $ref: '#/components/schemas/unstableDashboardWidgetView'
+ dimensions:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetDimension'
+ metrics:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetMetric'
+ filters:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'
+ chartType:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'
+ chartConfig:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'
+ nullable: true
+ description: Defaults to the plain config for `chartType`.
required:
- name
- - type
- unstableLlmAsJudgeEvaluatorType:
- title: unstableLlmAsJudgeEvaluatorType
- type: string
- enum:
- - llm_as_judge
- unstableEvaluationRuleEvaluator:
- title: unstableEvaluationRuleEvaluator
+ - view
+ - dimensions
+ - metrics
+ - filters
+ - chartType
+ unstableUpdateDashboardWidgetRequest:
+ title: unstableUpdateDashboardWidgetRequest
type: object
- description: |-
- Resolved evaluator currently used by the evaluation rule.
-
- `id` identifies the evaluator family. Evaluation runs automatically use
- the latest available evaluator version.
properties:
- id:
- type: string
- description: >-
- Identifier of the exact evaluator version currently used by the rule.
name:
type: string
- description: Evaluator family name.
- type:
- $ref: '#/components/schemas/unstableEvaluatorType'
- description: Evaluator type.
- required:
- - id
- - name
- - type
- unstableEvaluationRuleEvaluatorAssignment:
- title: unstableEvaluationRuleEvaluatorAssignment
- type: object
- properties:
- evaluator:
- $ref: '#/components/schemas/unstableEvaluationRuleEvaluator'
- mapping:
+ nullable: true
+ description:
+ type: string
+ nullable: true
+ view:
+ $ref: '#/components/schemas/unstableDashboardWidgetView'
+ nullable: true
+ dimensions:
type: array
items:
- $ref: '#/components/schemas/unstablePromptVariableMappingRead'
+ $ref: '#/components/schemas/unstableDashboardWidgetDimension'
nullable: true
- description: >-
- Rule-specific override, or `null` to inherit the evaluator version's
- default mapping.
- required:
- - evaluator
- - mapping
- unstableLegacyEvaluationRuleEvaluatorAssignment:
- title: unstableLegacyEvaluationRuleEvaluatorAssignment
+ metrics:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetMetric'
+ nullable: true
+ filters:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'
+ nullable: true
+ chartType:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'
+ nullable: true
+ chartConfig:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartConfigInput'
+ nullable: true
+ unstableDashboardWidgetList:
+ title: unstableDashboardWidgetList
type: object
- description: '**Deprecated:** Evaluator assignment on a legacy trace or dataset rule.'
properties:
- evaluator:
- $ref: '#/components/schemas/unstableEvaluationRuleEvaluator'
- mapping:
+ data:
type: array
items:
- $ref: '#/components/schemas/unstableLegacyPromptVariableMapping'
- nullable: true
+ $ref: '#/components/schemas/unstableDashboardWidget'
+ meta:
+ $ref: '#/components/schemas/utilsMetaResponse'
+ required:
+ - data
+ - meta
+ unstableDeleteDashboardWidgetResponse:
+ title: unstableDeleteDashboardWidgetResponse
+ type: object
+ properties:
+ message:
+ type: string
required:
- - evaluator
- - mapping
- unstableEvaluationRuleTimeScope:
- title: unstableEvaluationRuleTimeScope
- type: string
- enum:
- - NEW
- - EXISTING
- unstableLegacyEvaluationRuleTarget:
- title: unstableLegacyEvaluationRuleTarget
- type: string
- enum:
- - trace
- - dataset
- description: '**Deprecated:** Legacy trace or dataset evaluation target.'
- unstableLegacyPromptVariableMapping:
- title: unstableLegacyPromptVariableMapping
+ - message
+ unstableDashboardWidget:
+ title: unstableDashboardWidget
type: object
- description: >-
- **Deprecated:** Connects one prompt variable to data from a trace,
- dataset item, or named observation in a legacy rule.
properties:
- variable:
+ id:
type: string
- langfuseObject:
- $ref: '#/components/schemas/unstableLegacyEvaluationObject'
- objectName:
+ createdAt:
type: string
- nullable: true
- description: >-
- Observation name to match, or `null` when `langfuseObject` is
- `trace` or `dataset_item`.
- source:
+ format: date-time
+ updatedAt:
type: string
- jsonPath:
+ format: date-time
+ name:
type: string
- nullable: true
- required:
- - variable
- - langfuseObject
- - objectName
- - source
- unstableLegacyEvaluationObject:
- title: unstableLegacyEvaluationObject
- type: string
- enum:
- - trace
- - span
- - generation
- - event
- - agent
- - tool
- - chain
- - retriever
- - evaluator
- - embedding
- - guardrail
- - dataset_item
- description: >-
- **Deprecated:** Legacy Langfuse object used by a trace or dataset
- evaluation rule.
- unstableCreateEvaluationRuleEvaluatorAssignment:
- title: unstableCreateEvaluationRuleEvaluatorAssignment
- type: object
- properties:
- evaluator:
- $ref: '#/components/schemas/unstableCreateEvaluationRuleEvaluatorReference'
- mapping:
+ description:
+ type: string
+ view:
+ $ref: '#/components/schemas/unstableDashboardWidgetViewWithLegacy'
+ dimensions:
type: array
items:
- $ref: '#/components/schemas/unstablePromptVariableMappingInput'
- nullable: true
- description: >-
- Optional rule-specific override. Omit to inherit the evaluator
- version's default mapping. Omit for code evaluators.
+ $ref: '#/components/schemas/unstableDashboardWidgetDimension'
+ metrics:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetMetric'
+ filters:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'
+ chartType:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartType'
+ chartConfig:
+ $ref: '#/components/schemas/unstableDashboardWidgetChartConfig'
required:
- - evaluator
- unstableCreateEvaluationRuleEvaluatorReference:
- title: unstableCreateEvaluationRuleEvaluatorReference
+ - id
+ - createdAt
+ - updatedAt
+ - name
+ - description
+ - view
+ - dimensions
+ - metrics
+ - filters
+ - chartType
+ - chartConfig
+ unstableDashboardPlacement:
+ title: unstableDashboardPlacement
+ oneOf:
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - widget
+ - $ref: '#/components/schemas/unstableWidgetPlacement'
+ required:
+ - type
+ title: UnstableWidgetPlacement
+ - type: object
+ allOf:
+ - type: object
+ properties:
+ type:
+ type: string
+ enum:
+ - preset
+ - $ref: '#/components/schemas/unstablePresetPlacement'
+ required:
+ - type
+ title: UnstablePresetPlacement
+ description: |-
+ A tile on the dashboard's 12-column grid. `x`/`y` are the tile's
+ top-left cell (0-based; `y` grows downward), `width`/`height` its size
+ in cells. The UI default tile is 6x6 (half width). Overlapping tiles
+ are not rejected; prefer appending below existing tiles (or omit the
+ position on create to let the server do it).
+ unstableWidgetPlacement:
+ title: unstableWidgetPlacement
type: object
properties:
- name:
+ id:
type: string
- type:
- $ref: '#/components/schemas/unstableEvaluatorType'
- nullable: true
- description: Defaults to `llm_as_judge`.
+ widgetId:
+ type: string
+ x:
+ type: integer
+ 'y':
+ type: integer
+ width:
+ type: integer
+ height:
+ type: integer
required:
- - name
- unstableDeleteEvaluatorResponse:
- title: unstableDeleteEvaluatorResponse
+ - id
+ - widgetId
+ - x
+ - 'y'
+ - width
+ - height
+ unstablePresetPlacement:
+ title: unstablePresetPlacement
type: object
- description: Confirmation response returned after successful deletion.
properties:
- message:
+ id:
type: string
- description: Always `Evaluator successfully deleted`.
+ presetId:
+ type: string
+ x:
+ type: integer
+ 'y':
+ type: integer
+ width:
+ type: integer
+ height:
+ type: integer
required:
- - message
- unstableEvaluator:
- title: unstableEvaluator
+ - id
+ - presetId
+ - x
+ - 'y'
+ - width
+ - height
+ unstableCreateDashboardPlacementRequest:
+ title: unstableCreateDashboardPlacementRequest
oneOf:
- type: object
allOf:
@@ -19730,11 +16770,11 @@ components:
type:
type: string
enum:
- - llm_as_judge
- - $ref: '#/components/schemas/unstableLlmAsJudgeEvaluator'
+ - widget
+ - $ref: '#/components/schemas/unstableCreateWidgetPlacement'
required:
- type
- title: UnstableLlmAsJudgeEvaluator
+ title: UnstableCreateWidgetPlacement
- type: object
allOf:
- type: object
@@ -19742,258 +16782,286 @@ components:
type:
type: string
enum:
- - code
- - $ref: '#/components/schemas/unstableCodeEvaluator'
+ - preset
+ - $ref: '#/components/schemas/unstableCreatePresetPlacement'
required:
- type
- title: UnstableCodeEvaluator
- description: >-
- One evaluator that can be used for scoring.
-
-
- An evaluator describes **how** to score data.
-
-
- It does not define **which** live objects are evaluated. That is the job
- of `evaluation-rules`.
-
-
- For agent clients, the most important fields are:
-
- - `type`: determines which evaluator fields are present
-
- - `variables`: for LLM evaluators, use these exact names when building
- the evaluation-rule `mapping` array. LLM evaluators require every
- variable to be mapped. Code evaluators always expose the fixed runtime
- payload fields and Langfuse maps them automatically.
-
-
- Versioning behavior:
-
- - `GET /evaluators` returns the latest version of each available
- evaluator.
-
- - `GET /evaluators/{id}` returns the latest version.
-
- - `id` identifies the evaluator family and remains stable when a new
- version is created.
-
- - Evaluation rules always run against the latest version for the
- selected evaluator name within the same source (`project` or `managed`).
- unstableEvaluatorBase:
- title: unstableEvaluatorBase
+ title: UnstableCreatePresetPlacement
+ unstableCreateWidgetPlacement:
+ title: unstableCreateWidgetPlacement
+ type: object
+ properties:
+ id:
+ type: string
+ nullable: true
+ description: Server-generated when omitted.
+ widgetId:
+ type: string
+ x:
+ type: integer
+ nullable: true
+ description: Grid column (12-column grid). Defaults to `0`.
+ 'y':
+ type: integer
+ nullable: true
+ description: Grid row. Defaults to the first row below all existing tiles.
+ width:
+ type: integer
+ nullable: true
+ description: Width in grid columns. Defaults to `6`.
+ height:
+ type: integer
+ nullable: true
+ description: Height in grid rows. Defaults to `6`.
+ required:
+ - widgetId
+ unstableCreatePresetPlacement:
+ title: unstableCreatePresetPlacement
+ type: object
+ properties:
+ id:
+ type: string
+ nullable: true
+ description: Server-generated when omitted.
+ presetId:
+ type: string
+ x:
+ type: integer
+ nullable: true
+ description: Grid column (12-column grid). Defaults to `0`.
+ 'y':
+ type: integer
+ nullable: true
+ description: Grid row. Defaults to the first row below all existing tiles.
+ width:
+ type: integer
+ nullable: true
+ description: Width in grid columns. Defaults to `6`.
+ height:
+ type: integer
+ nullable: true
+ description: Height in grid rows. Defaults to `6`.
+ required:
+ - presetId
+ unstableUpdateDashboardPlacementRequest:
+ title: unstableUpdateDashboardPlacementRequest
+ type: object
+ properties:
+ x:
+ type: integer
+ nullable: true
+ description: Grid column (12-column grid).
+ 'y':
+ type: integer
+ nullable: true
+ description: Grid row.
+ width:
+ type: integer
+ nullable: true
+ description: Width in grid columns.
+ height:
+ type: integer
+ nullable: true
+ description: Height in grid rows.
+ unstableDeleteDashboardPlacementResponse:
+ title: unstableDeleteDashboardPlacementResponse
type: object
properties:
- id:
- type: string
- description: Stable identifier of this evaluator across all versions.
- name:
+ message:
type: string
- description: Evaluator name.
- version:
- type: integer
- description: Version number of this evaluator.
- variables:
- type: array
- items:
- type: string
- description: >-
- Variables that can be mapped when creating an evaluation rule.
-
-
- LLM evaluators require every variable to be mapped exactly once.
- Code evaluators always expose the fixed runtime payload fields and
- Langfuse maps them automatically.
- mapping:
+ required:
+ - message
+ unstableDashboardDefinition:
+ title: unstableDashboardDefinition
+ type: object
+ properties:
+ widgets:
type: array
items:
- $ref: '#/components/schemas/unstablePromptVariableMappingRead'
- nullable: true
- description: >-
- Default variable mapping for this evaluator version, or `null` when
- no default is configured.
-
-
- An entry's `source` is `null` when that variable was never fully
- configured, and sources
-
- are not restricted by rule `target` here, because the default is
- stored on the evaluator
-
- rather than on any one rule.
- evaluationRuleCount:
- type: integer
- description: >-
- Number of evaluation rules in the project that currently use this
- evaluator.
+ $ref: '#/components/schemas/unstableDashboardPlacement'
+ required:
+ - widgets
+ unstableDashboard:
+ title: unstableDashboard
+ type: object
+ properties:
+ id:
+ type: string
createdAt:
type: string
format: date-time
- description: Timestamp when this evaluator was created.
updatedAt:
type: string
format: date-time
- description: Timestamp when this evaluator was last updated.
+ name:
+ type: string
+ description:
+ type: string
+ definition:
+ $ref: '#/components/schemas/unstableDashboardDefinition'
+ filters:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'
+ description: Dashboard-level filters applied to all widgets on the dashboard.
required:
- id
- - name
- - version
- - variables
- - mapping
- - evaluationRuleCount
- createdAt
- updatedAt
- unstableLlmAsJudgeEvaluator:
- title: unstableLlmAsJudgeEvaluator
- type: object
- properties:
- prompt:
- type: string
- description: Prompt template used during evaluation.
- outputDefinition:
- $ref: '#/components/schemas/unstablePublicEvaluatorOutputDefinition'
- description: >-
- Structured output schema returned by this evaluator.
-
-
- Responses always include `dataType` and omit the internal
- output-definition `version`.
-
- Use `dataType` to decide how future scores should be interpreted.
- modelConfig:
- $ref: '#/components/schemas/unstableEvaluatorModelConfig'
- nullable: true
- description: >-
- Explicit model configuration, or `null` when the project default
- evaluation model is used.
- required:
- - prompt
- - outputDefinition
- - modelConfig
- allOf:
- - $ref: '#/components/schemas/unstableEvaluatorBase'
- unstableCodeEvaluator:
- title: unstableCodeEvaluator
- type: object
- properties:
- sourceCode:
- type: string
- description: Source code executed for each matched observation.
- sourceCodeLanguage:
- $ref: '#/components/schemas/unstableCodeEvaluatorSourceCodeLanguage'
- description: Runtime language for `sourceCode`.
- required:
- - sourceCode
- - sourceCodeLanguage
- allOf:
- - $ref: '#/components/schemas/unstableEvaluatorBase'
- unstableEvaluators:
- title: unstableEvaluators
+ - name
+ - description
+ - definition
+ - filters
+ unstableDashboardList:
+ title: unstableDashboardList
type: object
properties:
data:
type: array
items:
- $ref: '#/components/schemas/unstableEvaluator'
+ $ref: '#/components/schemas/unstableDashboard'
meta:
$ref: '#/components/schemas/utilsMetaResponse'
required:
- data
- meta
- unstableCreateEvaluatorRequest:
- title: unstableCreateEvaluatorRequest
- oneOf:
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - llm_as_judge
- - $ref: '#/components/schemas/unstableCreateLlmAsJudgeEvaluatorRequest'
- required:
- - type
- title: UnstableCreateLlmAsJudgeEvaluatorRequest
- - type: object
- allOf:
- - type: object
- properties:
- type:
- type: string
- enum:
- - code
- - $ref: '#/components/schemas/unstableCreateCodeEvaluatorRequest'
- required:
- - type
- title: UnstableCreateCodeEvaluatorRequest
- description: >-
- Request body for creating an evaluator.
-
-
- If the same `name` already exists in your project, Langfuse creates the
- next version and returns it.
-
- Existing evaluation rules automatically use the latest evaluator
- version.
-
- If `type` is omitted, Langfuse defaults it to `llm_as_judge` for
- backwards compatibility.
- unstableCreateLlmAsJudgeEvaluatorRequest:
- title: unstableCreateLlmAsJudgeEvaluatorRequest
+ unstableCreateDashboardRequest:
+ title: unstableCreateDashboardRequest
type: object
properties:
name:
type: string
- description: Evaluator name within the authenticated project.
- prompt:
+ description:
type: string
- description: Prompt template used by the evaluator.
- outputDefinition:
- $ref: '#/components/schemas/unstableEvaluatorOutputDefinition'
- description: >-
- Structured output schema the evaluator must return.
-
-
- Always send `dataType`.
-
- Do not send `version`; it is an internal storage detail and not part
- of the public request contract.
- modelConfig:
- $ref: '#/components/schemas/unstableEvaluatorModelConfig'
nullable: true
- description: >-
- Optional explicit model configuration. Omit or set to `null` to use
- the project default evaluation model.
- mapping:
+ definition:
+ $ref: '#/components/schemas/unstableDashboardDefinition'
+ nullable: true
+ filters:
type: array
items:
- $ref: '#/components/schemas/unstablePromptVariableMappingInput'
+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'
nullable: true
- description: >-
- Optional default variable mapping inherited by rule assignments that
- do not provide an override.
required:
- name
- - prompt
- - outputDefinition
- unstableCreateCodeEvaluatorRequest:
- title: unstableCreateCodeEvaluatorRequest
+ unstableUpdateDashboardRequest:
+ title: unstableUpdateDashboardRequest
type: object
properties:
name:
type: string
- description: Evaluator name within the authenticated project.
- sourceCode:
+ nullable: true
+ description:
+ type: string
+ nullable: true
+ definition:
+ $ref: '#/components/schemas/unstableDashboardDefinition'
+ nullable: true
+ filters:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstableDashboardWidgetFilter'
+ nullable: true
+ unstableDeleteDashboardResponse:
+ title: unstableDeleteDashboardResponse
+ type: object
+ properties:
+ message:
type: string
- description: Code executed for each matched observation.
- sourceCodeLanguage:
- $ref: '#/components/schemas/unstableCodeEvaluatorSourceCodeLanguage'
- description: Runtime language for `sourceCode`.
required:
- - name
- - sourceCode
- - sourceCodeLanguage
+ - message
+ unstablePublicApiErrorCode:
+ title: unstablePublicApiErrorCode
+ type: string
+ enum:
+ - authentication_failed
+ - access_denied
+ - invalid_request
+ - invalid_query
+ - invalid_body
+ - resource_not_found
+ - conflict
+ - rate_limited
+ - method_not_allowed
+ - internal_error
+ description: Machine-readable error code returned by unstable public API endpoints.
+ unstablePublicApiValidationIssue:
+ title: unstablePublicApiValidationIssue
+ type: object
+ description: >-
+ A validation issue returned for malformed request bodies or query
+ parameters.
+ properties:
+ code:
+ type: string
+ message:
+ type: string
+ path:
+ type: array
+ items: {}
+ required:
+ - code
+ - message
+ - path
+ unstablePublicApiErrorDetails:
+ title: unstablePublicApiErrorDetails
+ type: object
+ description: Optional structured context attached to an error.
+ properties:
+ issues:
+ type: array
+ items:
+ $ref: '#/components/schemas/unstablePublicApiValidationIssue'
+ nullable: true
+ field:
+ type: string
+ nullable: true
+ column:
+ type: string
+ nullable: true
+ invalidValues:
+ type: array
+ items:
+ type: string
+ nullable: true
+ allowedValues:
+ type: array
+ items:
+ type: string
+ nullable: true
+ value:
+ type: string
+ nullable: true
+ retryAfterSeconds:
+ type: integer
+ nullable: true
+ limit:
+ type: integer
+ nullable: true
+ remaining:
+ type: integer
+ nullable: true
+ resetAt:
+ type: string
+ nullable: true
+ unstablePublicApiError:
+ title: unstablePublicApiError
+ type: object
+ description: Standard error envelope for unstable public API endpoints.
+ properties:
+ message:
+ type: string
+ description: Human-readable description of the failure.
+ code:
+ $ref: '#/components/schemas/unstablePublicApiErrorCode'
+ description: Machine-readable error code.
+ details:
+ $ref: '#/components/schemas/unstablePublicApiErrorDetails'
+ nullable: true
+ description: Optional structured error context.
+ required:
+ - message
+ - code
utilsMetaResponse:
title: utilsMetaResponse
type: object
diff --git a/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap b/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap
index 2f0c41a22b..21f53d21e8 100644
--- a/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap
+++ b/web/src/__tests__/server/__snapshots__/project-route-auth-parity.servertest.ts.snap
@@ -86,20 +86,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1
"DELETE unstable/dashboards/[dashboardId]/placements/[placementId] | org/bearer": 401,
"DELETE unstable/dashboards/[dashboardId]/placements/[placementId] | project/basic": 404,
"DELETE unstable/dashboards/[dashboardId]/placements/[placementId] | project/bearer": 403,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | admin/admin": 401,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | agent/basic": 401,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | org/basic": 403,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | org/bearer": 401,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | project/basic": 404,
- "DELETE unstable/evaluation-rules/[evaluationRuleId] | project/bearer": 403,
- "DELETE unstable/evaluators/[evaluatorId] | admin/admin": 401,
- "DELETE unstable/evaluators/[evaluatorId] | agent/basic": 401,
- "DELETE unstable/evaluators/[evaluatorId] | agent/bearer": 401,
- "DELETE unstable/evaluators/[evaluatorId] | org/basic": 403,
- "DELETE unstable/evaluators/[evaluatorId] | org/bearer": 401,
- "DELETE unstable/evaluators/[evaluatorId] | project/basic": 404,
- "DELETE unstable/evaluators/[evaluatorId] | project/bearer": 403,
"DELETE v2/evaluation-rules/[evaluationRuleId] | admin/admin": 401,
"DELETE v2/evaluation-rules/[evaluationRuleId] | agent/basic": 401,
"DELETE v2/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,
@@ -359,34 +345,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1
"GET unstable/dashboards/index | org/bearer": 401,
"GET unstable/dashboards/index | project/basic": 200,
"GET unstable/dashboards/index | project/bearer": 403,
- "GET unstable/evaluation-rules/[evaluationRuleId] | admin/admin": 401,
- "GET unstable/evaluation-rules/[evaluationRuleId] | agent/basic": 401,
- "GET unstable/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,
- "GET unstable/evaluation-rules/[evaluationRuleId] | org/basic": 403,
- "GET unstable/evaluation-rules/[evaluationRuleId] | org/bearer": 401,
- "GET unstable/evaluation-rules/[evaluationRuleId] | project/basic": 404,
- "GET unstable/evaluation-rules/[evaluationRuleId] | project/bearer": 403,
- "GET unstable/evaluation-rules/index | admin/admin": 401,
- "GET unstable/evaluation-rules/index | agent/basic": 401,
- "GET unstable/evaluation-rules/index | agent/bearer": 401,
- "GET unstable/evaluation-rules/index | org/basic": 403,
- "GET unstable/evaluation-rules/index | org/bearer": 401,
- "GET unstable/evaluation-rules/index | project/basic": 200,
- "GET unstable/evaluation-rules/index | project/bearer": 403,
- "GET unstable/evaluators/[evaluatorId] | admin/admin": 401,
- "GET unstable/evaluators/[evaluatorId] | agent/basic": 401,
- "GET unstable/evaluators/[evaluatorId] | agent/bearer": 401,
- "GET unstable/evaluators/[evaluatorId] | org/basic": 403,
- "GET unstable/evaluators/[evaluatorId] | org/bearer": 401,
- "GET unstable/evaluators/[evaluatorId] | project/basic": 404,
- "GET unstable/evaluators/[evaluatorId] | project/bearer": 403,
- "GET unstable/evaluators/index | admin/admin": 401,
- "GET unstable/evaluators/index | agent/basic": 401,
- "GET unstable/evaluators/index | agent/bearer": 401,
- "GET unstable/evaluators/index | org/basic": 403,
- "GET unstable/evaluators/index | org/bearer": 401,
- "GET unstable/evaluators/index | project/basic": 200,
- "GET unstable/evaluators/index | project/bearer": 403,
"GET v2/datasets/[datasetName]/index | admin/admin": 401,
"GET v2/datasets/[datasetName]/index | agent/basic": 401,
"GET v2/datasets/[datasetName]/index | agent/bearer": 401,
@@ -527,13 +485,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1
"PATCH unstable/dashboards/[dashboardId]/placements/[placementId] | org/bearer": 401,
"PATCH unstable/dashboards/[dashboardId]/placements/[placementId] | project/basic": 400,
"PATCH unstable/dashboards/[dashboardId]/placements/[placementId] | project/bearer": 403,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | admin/admin": 401,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | agent/basic": 401,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | org/basic": 403,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | org/bearer": 401,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | project/basic": 400,
- "PATCH unstable/evaluation-rules/[evaluationRuleId] | project/bearer": 403,
"PATCH v2/evaluation-rules/[evaluationRuleId] | admin/admin": 401,
"PATCH v2/evaluation-rules/[evaluationRuleId] | agent/basic": 401,
"PATCH v2/evaluation-rules/[evaluationRuleId] | agent/bearer": 401,
@@ -681,20 +632,6 @@ exports[`project-route auth parity > legacy matches the main-captured baseline 1
"POST unstable/dashboards/index | org/bearer": 401,
"POST unstable/dashboards/index | project/basic": 400,
"POST unstable/dashboards/index | project/bearer": 403,
- "POST unstable/evaluation-rules/index | admin/admin": 401,
- "POST unstable/evaluation-rules/index | agent/basic": 401,
- "POST unstable/evaluation-rules/index | agent/bearer": 401,
- "POST unstable/evaluation-rules/index | org/basic": 403,
- "POST unstable/evaluation-rules/index | org/bearer": 401,
- "POST unstable/evaluation-rules/index | project/basic": 400,
- "POST unstable/evaluation-rules/index | project/bearer": 403,
- "POST unstable/evaluators/index | admin/admin": 401,
- "POST unstable/evaluators/index | agent/basic": 401,
- "POST unstable/evaluators/index | agent/bearer": 401,
- "POST unstable/evaluators/index | org/basic": 403,
- "POST unstable/evaluators/index | org/bearer": 401,
- "POST unstable/evaluators/index | project/basic": 400,
- "POST unstable/evaluators/index | project/bearer": 403,
"POST v2/datasets/index | admin/admin": 401,
"POST v2/datasets/index | agent/basic": 401,
"POST v2/datasets/index | agent/bearer": 401,
diff --git a/web/src/__tests__/server/project-route-auth-parity.servertest.ts b/web/src/__tests__/server/project-route-auth-parity.servertest.ts
index b92cc1dc03..b3280c6ffe 100644
--- a/web/src/__tests__/server/project-route-auth-parity.servertest.ts
+++ b/web/src/__tests__/server/project-route-auth-parity.servertest.ts
@@ -97,13 +97,6 @@ const projectRoutes: Route[] = [
route: "unstable/dashboard-widgets/[widgetId]",
methods: ["GET", "PATCH", "DELETE"],
},
- { route: "unstable/evaluators/index", methods: ["GET", "POST"] },
- { route: "unstable/evaluators/[evaluatorId]", methods: ["GET", "DELETE"] },
- { route: "unstable/evaluation-rules/index", methods: ["GET", "POST"] },
- {
- route: "unstable/evaluation-rules/[evaluationRuleId]",
- methods: ["GET", "PATCH", "DELETE"],
- },
{ route: "v2/evaluators/index", methods: ["GET", "POST"] },
{
route: "v2/evaluators/[evaluatorId]",
diff --git a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts b/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts
index 991af5cb04..e352d01c48 100644
--- a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts
+++ b/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts
@@ -35,18 +35,6 @@ const openApiPath = path.resolve(
);
const MESSAGE = "Use `GET /api/public/v2/observations?from=<from>` instead.";
-const UNSTABLE_EVALS_SUNSET_HUMAN = "September 4, 2026";
-const UNSTABLE_EVAL_OPERATIONS = new Set([
- "POST /api/public/unstable/evaluators",
- "GET /api/public/unstable/evaluators",
- "GET /api/public/unstable/evaluators/{evaluatorId}",
- "DELETE /api/public/unstable/evaluators/{evaluatorId}",
- "POST /api/public/unstable/evaluation-rules",
- "GET /api/public/unstable/evaluation-rules",
- "GET /api/public/unstable/evaluation-rules/{evaluationRuleId}",
- "PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}",
- "DELETE /api/public/unstable/evaluation-rules/{evaluationRuleId}",
-]);
/** Spec fixture in the shape `fern export` produces, aliased `security` included. */
const SPEC = `openapi: 3.0.1
@@ -139,11 +127,10 @@ describe("OpenAPI deprecations", () => {
definitionDirectory,
)) {
const operation = `${method.toUpperCase()} ${endpointPath}`;
- const sunsetDate = UNSTABLE_EVAL_OPERATIONS.has(operation)
- ? UNSTABLE_EVALS_SUNSET_HUMAN
- : V3_SUNSET_HUMAN;
- expect(message, operation).toContain(`will be removed on ${sunsetDate}.`);
+ expect(message, operation).toContain(
+ `will be removed on ${V3_SUNSET_HUMAN}.`,
+ );
}
});
diff --git a/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts b/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts
index 6e5538249e..6d92c99f44 100644
--- a/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts
+++ b/web/src/__tests__/server/unit/python-sdk-deprecations.servertest.ts
@@ -118,15 +118,6 @@ describe("Python SDK deprecations", () => {
"sessions:get",
"trace:get",
"trace:list",
- "unstable/evaluation_rules:create",
- "unstable/evaluation_rules:list",
- "unstable/evaluation_rules:get",
- "unstable/evaluation_rules:update",
- "unstable/evaluation_rules:delete",
- "unstable/evaluators:create",
- "unstable/evaluators:list",
- "unstable/evaluators:get",
- "unstable/evaluators:delete",
]);
});
diff --git a/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts b/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts
index 43c9afd798..9ee7ac7ca1 100644
--- a/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts
+++ b/web/src/__tests__/server/unit/typescript-sdk-deprecations.servertest.ts
@@ -146,15 +146,6 @@ describe("TypeScript SDK deprecations", () => {
"api/resources/sessions/client/Client.ts:get",
"api/resources/trace/client/Client.ts:get",
"api/resources/trace/client/Client.ts:list",
- "api/resources/unstable/resources/evaluationRules/client/Client.ts:create",
- "api/resources/unstable/resources/evaluationRules/client/Client.ts:list",
- "api/resources/unstable/resources/evaluationRules/client/Client.ts:get",
- "api/resources/unstable/resources/evaluationRules/client/Client.ts:update",
- "api/resources/unstable/resources/evaluationRules/client/Client.ts:delete",
- "api/resources/unstable/resources/evaluators/client/Client.ts:create",
- "api/resources/unstable/resources/evaluators/client/Client.ts:list",
- "api/resources/unstable/resources/evaluators/client/Client.ts:get",
- "api/resources/unstable/resources/evaluators/client/Client.ts:delete",
]);
expect(
deprecations
diff --git a/web/src/__tests__/server/unit/unstable-evals-contract.servertest.ts b/web/src/__tests__/server/unit/unstable-evals-contract.servertest.ts
deleted file mode 100644
index 63da9f8d13..0000000000
--- a/web/src/__tests__/server/unit/unstable-evals-contract.servertest.ts
+++ /dev/null
@@ -1,949 +0,0 @@
-import {
- createNumericEvalOutputDefinition,
- EvalTargetObject,
- JobConfigState,
- CODE_EVAL_TEMPLATE_VARIABLES,
-} from "@langfuse/shared";
-import { CODE_EVAL_SOURCE_MAX_BYTES } from "@langfuse/shared/src/server";
-import { EvalTemplateType, Prisma } from "@langfuse/shared/src/db";
-import {
- toApiEvaluator,
- toApiV2EvaluationRule,
- toEvaluationRuleInput,
-} from "@/src/features/evals/server/unstable-public-api/adapters";
-import { StructuredPublicApiError } from "@/src/features/public-api";
-import type {
- StoredPublicEvaluatorTemplate,
- StoredPublicV2EvaluationRule,
-} from "@/src/features/evals/server/unstable-public-api/types";
-import {
- GetUnstableEvaluationRulesQuery,
- PatchUnstableEvaluationRuleBody,
- PostUnstableEvaluationRuleBody,
-} from "@/src/features/public-api/types/unstable-evaluation-rules";
-import {
- GetUnstableEvaluatorsQuery,
- PostUnstableEvaluatorBody,
-} from "@/src/features/public-api/types/unstable-evaluators";
-import {
- PUBLIC_EVALUATOR_TYPE_CODE,
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
-} from "@/src/features/public-api/types/unstable-public-evals-contract";
-
-const numericOutputDefinition = createNumericEvalOutputDefinition({
- reasoningDescription: "Why the score was assigned",
- scoreDescription: "A score between 0 and 1",
- minValue: 0,
- maxValue: 1,
-});
-
-const expectStructuredPublicApiError = (
- fn: () => unknown,
- params: {
- code: StructuredPublicApiError["code"];
- message?: string;
- details?: Record<string, unknown>;
- },
-) => {
- try {
- fn();
- throw new Error("Expected function to throw StructuredPublicApiError");
- } catch (error) {
- expect(error).toBeInstanceOf(StructuredPublicApiError);
-
- const unstableError = error as StructuredPublicApiError;
- expect(unstableError.code).toBe(params.code);
-
- if (params.message) {
- expect(unstableError.message).toContain(params.message);
- }
-
- if (params.details) {
- expect(unstableError.details).toMatchObject(params.details);
- }
- }
-};
-
-describe("unstable public eval contracts", () => {
- it("ignores unknown evaluator creation fields for forward compatibility", () => {
- const parsed = PostUnstableEvaluatorBody.parse({
- name: "Answer correctness",
- description: "future field",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- modelConfig: {
- provider: "openai",
- model: "gpt-4.1-mini",
- futureSetting: "ignored",
- },
- });
-
- expect(parsed).toEqual({
- name: "Answer correctness",
- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: {
- dataType: "NUMERIC",
- reasoning: {
- description: "Why the score was assigned",
- },
- score: {
- description: "A score between 0 and 1",
- minValue: 0,
- maxValue: 1,
- },
- },
- modelConfig: {
- provider: "openai",
- model: "gpt-4.1-mini",
- },
- });
- });
-
- it("accepts evaluator default mappings", () => {
- const parsed = PostUnstableEvaluatorBody.parse({
- name: "Answer correctness",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- mapping: [
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ],
- });
-
- expect(parsed.mapping).toEqual([
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ]);
- });
-
- it("requires outputDefinition.dataType for evaluator creation", () => {
- const parsed = PostUnstableEvaluatorBody.safeParse({
- name: "Answer correctness",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: {
- reasoning: "Explain why the answer is correct or incorrect.",
- score: "Return a score between 0 and 1.",
- },
- });
-
- expect(parsed.success).toBe(false);
- expect(parsed.error?.issues).toEqual(
- expect.arrayContaining([
- expect.objectContaining({
- path: ["outputDefinition", "dataType"],
- }),
- ]),
- );
- });
-
- it("accepts explicitly typed code evaluator creation bodies", () => {
- const parsed = PostUnstableEvaluatorBody.parse({
- name: "Exact match",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- sourceCode:
- 'function evaluate() { return { scores: [{ name: "match", value: true, dataType: "BOOLEAN" }] }; }',
- sourceCodeLanguage: "TYPESCRIPT",
- });
-
- expect(parsed).toEqual({
- name: "Exact match",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- sourceCode:
- 'function evaluate() { return { scores: [{ name: "match", value: true, dataType: "BOOLEAN" }] }; }',
- sourceCodeLanguage: "TYPESCRIPT",
- });
- });
-
- it("rejects LLM-only fields on code evaluator creation bodies", () => {
- const parsed = PostUnstableEvaluatorBody.safeParse({
- name: "Exact match",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- prompt: "Judge {{input}}",
- sourceCode:
- 'function evaluate() { return { scores: [{ name: "match", value: true, dataType: "BOOLEAN" }] }; }',
- sourceCodeLanguage: "TYPESCRIPT",
- });
-
- expect(parsed.success).toBe(false);
- expect(parsed.error?.issues).toEqual(
- expect.arrayContaining([
- expect.objectContaining({
- path: ["prompt"],
- }),
- ]),
- );
- });
-
- it("rejects code evaluator creation bodies when source code is too large", () => {
- const parsed = PostUnstableEvaluatorBody.safeParse({
- name: "Oversized code",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- sourceCode: "a".repeat(CODE_EVAL_SOURCE_MAX_BYTES + 1),
- sourceCodeLanguage: "TYPESCRIPT",
- });
-
- expect(parsed.success).toBe(false);
- expect(parsed.error?.issues).toEqual(
- expect.arrayContaining([
- expect.objectContaining({
- path: ["sourceCode"],
- message: `Source code must be ${CODE_EVAL_SOURCE_MAX_BYTES} bytes or less`,
- }),
- ]),
- );
- });
-
- it("rejects observation evaluation rules that use expected_output mappings", () => {
- const parsed = PostUnstableEvaluationRuleBody.safeParse({
- name: "answer_quality",
- evaluator: {
- name: "Answer correctness",
- },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [
- { variable: "output", source: "output" },
- { variable: "expected_output", source: "expected_output" },
- ],
- });
-
- expect(parsed.success).toBe(false);
- });
-
- it("accepts both root filters on code rule bodies without mappings", () => {
- const rootFilters = [
- {
- type: "boolean",
- column: "isRootObservation",
- operator: "=",
- value: true,
- },
- {
- type: "null",
- column: "parentObservationId",
- operator: "is null",
- value: "",
- },
- ] as const;
- const parsed = PostUnstableEvaluationRuleBody.parse({
- name: "toxicity-code-live",
- evaluator: {
- name: "Toxicity detector",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: rootFilters,
- });
-
- expect(parsed.filter).toEqual(rootFilters);
- expect("mapping" in parsed).toBe(false);
- expect("variableMapping" in parsed).toBe(false);
- });
-
- it("keeps trace rules read-only", () => {
- const parsed = PostUnstableEvaluationRuleBody.safeParse({
- name: "trace-quality",
- evaluator: {
- name: "Quality",
- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- },
- target: "trace",
- enabled: true,
- filter: [],
- mapping: [
- {
- variable: "input",
- langfuseObject: "trace",
- objectName: null,
- source: "input",
- },
- {
- variable: "output",
- langfuseObject: "generation",
- objectName: "answer-generation",
- source: "output",
- },
- ],
- });
-
- expect(parsed.success).toBe(false);
- });
-
- it("rejects code evaluation rule create bodies with mappings", () => {
- const parsed = PostUnstableEvaluationRuleBody.safeParse({
- name: "toxicity-code-live",
- evaluator: {
- name: "Toxicity detector",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "output", source: "output" }],
- });
-
- expect(parsed.success).toBe(false);
- });
-
- it("accepts LLM evaluation rule create bodies without mappings to inherit evaluator defaults", () => {
- const parsed = PostUnstableEvaluationRuleBody.safeParse({
- name: "answer-correctness-live",
- evaluator: {
- name: "Answer correctness",
- },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- });
-
- expect(parsed.success).toBe(true);
- });
-
- it("accepts multiple evaluator assignments", () => {
- const parsed = PostUnstableEvaluationRuleBody.parse({
- name: "answer-quality-live",
- evaluators: [
- {
- evaluator: { name: "Correctness" },
- },
- {
- evaluator: { name: "Tone" },
- mapping: [{ variable: "output", source: "output" }],
- },
- ],
- target: "observation",
- enabled: true,
- filter: [],
- });
-
- expect(parsed.evaluators).toHaveLength(2);
- });
-
- it("rejects create bodies containing both compatibility and array evaluator fields", () => {
- const parsed = PostUnstableEvaluationRuleBody.safeParse({
- name: "answer-quality-live",
- evaluator: { name: "Correctness" },
- evaluators: [{ evaluator: { name: "Tone" } }],
- target: "observation",
- enabled: true,
- filter: [],
- });
-
- expect(parsed.success).toBe(false);
- });
-
- it("rejects create bodies that pair the deprecated mapping alias with evaluators", () => {
- const parsed = PostUnstableEvaluationRuleBody.safeParse({
- name: "answer-quality-live",
- evaluators: [{ evaluator: { name: "Tone" } }],
- mapping: [{ variable: "output", source: "output" }],
- target: "observation",
- enabled: true,
- filter: [],
- });
-
- expect(parsed.success).toBe(false);
- });
-
- it("accepts evaluators on patch bodies as a full assignment replacement", () => {
- const parsed = PatchUnstableEvaluationRuleBody.parse({
- evaluators: [
- { evaluator: { name: "Correctness" } },
- {
- evaluator: { name: "Tone" },
- mapping: [{ variable: "output", source: "output" }],
- },
- ],
- });
-
- expect(parsed).toMatchObject({ evaluators: expect.any(Array) });
- });
-
- it("rejects patch bodies pairing evaluators with the deprecated aliases", () => {
- for (const conflicting of [
- { evaluator: { name: "Correctness" } },
- {
- target: "observation" as const,
- mapping: [{ variable: "output", source: "output" as const }],
- },
- ]) {
- expect(
- PatchUnstableEvaluationRuleBody.safeParse({
- evaluators: [{ evaluator: { name: "Tone" } }],
- ...conflicting,
- }).success,
- ).toBe(false);
- }
- });
-
- it("rejects experiment filter updates unless target is provided", () => {
- const parsed = PatchUnstableEvaluationRuleBody.safeParse({
- filter: [
- {
- type: "stringOptions",
- column: "datasetId",
- operator: "any of",
- value: ["dataset-1"],
- },
- ],
- });
-
- expect(parsed.success).toBe(false);
- expect(parsed.error?.issues.length).toBeGreaterThan(0);
- });
-
- it("rejects non-positive unstable pagination limits", () => {
- expect(
- GetUnstableEvaluatorsQuery.safeParse({
- page: 1,
- limit: 0,
- }).success,
- ).toBe(false);
- expect(
- GetUnstableEvaluationRulesQuery.safeParse({
- page: 1,
- limit: -1,
- }).success,
- ).toBe(false);
- });
-
- it("rejects non-integer unstable pagination values", () => {
- expect(
- GetUnstableEvaluatorsQuery.safeParse({
- page: 1.5,
- limit: 10,
- }).success,
- ).toBe(false);
- expect(
- GetUnstableEvaluationRulesQuery.safeParse({
- page: 1,
- limit: 1.5,
- }).success,
- ).toBe(false);
- });
-
- it("ignores unknown evaluation rule update fields for forward compatibility", () => {
- const parsed = PatchUnstableEvaluationRuleBody.parse({
- name: "answer_quality",
- newFieldFromFutureVersion: true,
- });
-
- expect(parsed).toEqual({
- name: "answer_quality",
- });
- });
-
- it("preserves target-specific patch fields instead of dropping them into the untargeted schema", () => {
- const parsed = PatchUnstableEvaluationRuleBody.parse({
- name: "experiment-expected-output-match",
- target: "experiment",
- filter: [
- {
- type: "stringOptions",
- column: "datasetId",
- operator: "any of",
- value: ["dataset-prod"],
- },
- ],
- mapping: [
- { variable: "output", source: "output" },
- { variable: "expected_output", source: "expected_output" },
- ],
- });
-
- expect(parsed).toEqual({
- name: "experiment-expected-output-match",
- target: "experiment",
- filter: [
- {
- type: "stringOptions",
- column: "datasetId",
- operator: "any of",
- value: ["dataset-prod"],
- },
- ],
- mapping: [
- { variable: "output", source: "output" },
- { variable: "expected_output", source: "expected_output" },
- ],
- });
- });
-
- it("rejects invalid target-specific patch payloads instead of silently parsing a name-only subset", () => {
- const parsed = PatchUnstableEvaluationRuleBody.safeParse({
- name: "experiment-expected-output-match",
- target: "experiment",
- filter: [
- // The experiment-root filter is what `target: "experiment"` means, so
- // it is not separately addressable on either target.
- {
- type: "boolean",
- column: "isExperimentItemRootSpan",
- operator: "=",
- value: true,
- },
- ],
- });
-
- expect(parsed.success).toBe(false);
- expect(parsed.error?.issues.length).toBeGreaterThan(0);
- });
-
- it("accepts observation filters on the experiment target", () => {
- // An experiment rule is an observation rule scoped to experiment root
- // spans, so every observation filter stays available alongside datasetId.
- const parsed = PatchUnstableEvaluationRuleBody.safeParse({
- target: "experiment",
- filter: [
- {
- type: "stringOptions",
- column: "type",
- operator: "any of",
- value: ["GENERATION"],
- },
- {
- type: "stringOptions",
- column: "datasetId",
- operator: "any of",
- value: ["dataset-prod"],
- },
- ],
- });
-
- expect(parsed.success).toBe(true);
- });
-
- it("strips evaluator type from patch bodies so a rule's evaluator type cannot be changed", () => {
- const parsed = PatchUnstableEvaluationRuleBody.parse({
- evaluator: {
- name: "toxicity-detector",
- type: "llm_as_judge",
- },
- });
-
- // `type` is dropped: the service inherits the rule's current evaluator type,
- // so a code rule cannot be retargeted to an LLM evaluator family.
- expect(parsed).toEqual({
- evaluator: { name: "toxicity-detector" },
- });
- expect(
- (parsed as { evaluator: { type?: string } }).evaluator.type,
- ).toBeUndefined();
- });
-});
-
-describe("unstable public eval adapters", () => {
- it("returns an empty effective mapping for zero-variable evaluators", () => {
- const createdAt = new Date("2026-08-14T00:00:00.000Z");
- const rule = {
- id: "rule_zero_variables",
- createdAt,
- updatedAt: createdAt,
- projectId: "project_123",
- createdByUserId: null,
- name: "Always pass",
- status: JobConfigState.ACTIVE,
- targetObject: EvalTargetObject.EVENT,
- filter: [],
- sampling: new Prisma.Decimal(1),
- delay: 0,
- timeScope: ["NEW"],
- assignments: [
- {
- id: "assignment_zero_variables",
- createdAt,
- updatedAt: createdAt,
- projectId: "project_123",
- evaluationRuleId: "rule_zero_variables",
- evaluatorId: "evaluator_zero_variables",
- variableMapping: null,
- evaluator: {
- id: "evaluator_zero_variables",
- createdAt,
- updatedAt: createdAt,
- projectId: "project_123",
- name: "Always pass",
- type: EvalTemplateType.LLM_AS_JUDGE,
- description: null,
- createdByUserId: null,
- blockedAt: null,
- blockReason: null,
- blockMessage: null,
- versions: [
- {
- id: "version_zero_variables",
- createdAt,
- evaluatorId: "evaluator_zero_variables",
- version: 1,
- createdByUserId: null,
- prompt: "Always return a score of 1",
- promptMessages: null,
- partner: null,
- model: null,
- provider: null,
- modelParams: null,
- vars: [],
- variableMapping: null,
- outputDefinition: numericOutputDefinition,
- sourceCode: null,
- sourceCodeLanguage: null,
- },
- ],
- },
- },
- ],
- } satisfies StoredPublicV2EvaluationRule;
-
- expect(toApiV2EvaluationRule(rule)).toMatchObject({
- mapping: [],
- evaluators: [{ mapping: null }],
- });
- });
-
- it("translates evaluation rule writes into job configuration inputs", () => {
- const writeModel = toEvaluationRuleInput({
- input: {
- name: "expected_output_match",
- target: "experiment",
- enabled: false,
- sampling: 0.25,
- filter: [
- {
- type: "stringOptions",
- column: "datasetId",
- operator: "any of",
- value: ["dataset-1"],
- },
- ],
- mapping: [
- { variable: "output", source: "output" },
- { variable: "expected_output", source: "expected_output" },
- ],
- },
- evaluatorVariables: ["output", "expected_output"],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- });
-
- expect(writeModel).toMatchObject({
- scoreName: "expected_output_match",
- targetObject: EvalTargetObject.EXPERIMENT,
- sampling: 0.25,
- status: JobConfigState.INACTIVE,
- filter: [
- {
- type: "stringOptions",
- column: "experimentDatasetId",
- operator: "any of",
- value: ["dataset-1"],
- },
- ],
- variableMapping: [
- {
- templateVariable: "output",
- selectedColumnId: "output",
- jsonSelector: null,
- },
- {
- templateVariable: "expected_output",
- selectedColumnId: "experimentItemExpectedOutput",
- jsonSelector: null,
- },
- ],
- });
- });
-
- it("rejects invalid static filter option values", () => {
- expectStructuredPublicApiError(
- () =>
- toEvaluationRuleInput({
- input: {
- name: "answer_quality",
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [
- {
- type: "stringOptions",
- column: "type",
- operator: "any of",
- value: ["NOT_A_REAL_TYPE"],
- },
- ],
- mapping: [{ variable: "input", source: "input" }],
- },
- evaluatorVariables: ["input"],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- }),
- {
- code: "invalid_filter_value",
- details: {
- field: "filter[0].value",
- column: "type",
- invalidValues: ["NOT_A_REAL_TYPE"],
- },
- },
- );
- });
-
- it("rejects malformed jsonPath selectors", () => {
- expectStructuredPublicApiError(
- () =>
- toEvaluationRuleInput({
- input: {
- name: "metadata_projection",
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [
- {
- variable: "input",
- source: "metadata",
- jsonPath: "$[",
- },
- ],
- },
- evaluatorVariables: ["input"],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- }),
- {
- code: "invalid_json_path",
- details: {
- field: "mapping[0].jsonPath",
- variable: "input",
- value: "$[",
- },
- },
- );
- });
-
- it("rejects mapping sources that are incompatible with the selected target", () => {
- expectStructuredPublicApiError(
- () =>
- toEvaluationRuleInput({
- input: {
- name: "answer_quality",
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [
- { variable: "expected_output", source: "expected_output" },
- ],
- },
- evaluatorVariables: ["expected_output"],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- }),
- {
- code: "invalid_variable_mapping",
- details: {
- field: "mapping[0].source",
- variable: "expected_output",
- },
- },
- );
- });
-
- it("accepts tool_calls mappings for both targets", () => {
- for (const target of ["observation", "experiment"] as const) {
- const writeModel = toEvaluationRuleInput({
- input: {
- name: "tool_call_check",
- target,
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [
- { variable: "calls", source: "tool_calls", jsonPath: "$[*].name" },
- ],
- },
- evaluatorVariables: ["calls"],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- });
-
- expect(writeModel.variableMapping).toEqual([
- {
- templateVariable: "calls",
- selectedColumnId: "toolCalls",
- jsonSelector: "$[*].name",
- },
- ]);
- }
- });
-
- it("rejects missing evaluator variable mappings", () => {
- expectStructuredPublicApiError(
- () =>
- toEvaluationRuleInput({
- input: {
- name: "answer_quality",
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "input", source: "input" }],
- },
- evaluatorVariables: ["input", "output"],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- }),
- {
- code: "missing_variable_mapping",
- details: {
- field: "mapping",
- variables: ["output"],
- },
- },
- );
- });
-
- it("inherits canonical evaluator mappings for code evaluation rule writes", () => {
- const writeModel = toEvaluationRuleInput({
- input: {
- name: "toxicity-code-live",
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- },
- evaluatorVariables: [],
- evaluatorType: PUBLIC_EVALUATOR_TYPE_CODE,
- });
-
- expect(writeModel.variableMapping).toBeNull();
- });
-
- it("maps evaluator records to exact template versions", () => {
- const template: StoredPublicEvaluatorTemplate = {
- id: "tmpl_latest",
- projectId: "project_123",
- name: "answer-correctness",
- version: 7,
- type: EvalTemplateType.LLM_AS_JUDGE,
- prompt: "Judge {{input}} against {{output}}",
- partner: "ragas",
- provider: "openai",
- model: "gpt-4.1-mini",
- modelParams: { temperature: 0 },
- vars: ["input", "output"],
- variableMapping: null,
- outputDefinition: numericOutputDefinition,
- sourceCode: null,
- sourceCodeLanguage: null,
- createdAt: new Date("2026-03-30T08:00:00.000Z"),
- updatedAt: new Date("2026-03-30T09:00:00.000Z"),
- };
-
- expect(
- toApiEvaluator({
- template,
- evaluationRuleCount: 2,
- }),
- ).toMatchObject({
- id: "tmpl_latest",
- name: "answer-correctness",
- version: 7,
- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- outputDefinition: {
- dataType: "NUMERIC",
- reasoning: {
- description: "Why the score was assigned",
- },
- score: {
- description: "A score between 0 and 1",
- minValue: 0,
- maxValue: 1,
- },
- },
- modelConfig: {
- provider: "openai",
- model: "gpt-4.1-mini",
- },
- variables: ["input", "output"],
- evaluationRuleCount: 2,
- });
- });
-
- it("maps code evaluator records with canonical runtime variables", () => {
- const template: StoredPublicEvaluatorTemplate = {
- id: "tmpl_code",
- projectId: "project_123",
- name: "toxicity-detector",
- version: 1,
- type: EvalTemplateType.CODE,
- prompt: null,
- partner: null,
- provider: null,
- model: null,
- modelParams: null,
- vars: ["stale"],
- variableMapping: null,
- outputDefinition: null,
- sourceCode:
- 'function evaluate() { return { scores: [{ name: "toxicity", value: false, dataType: "BOOLEAN" }] }; }',
- sourceCodeLanguage: "TYPESCRIPT",
- createdAt: new Date("2026-03-30T08:00:00.000Z"),
- updatedAt: new Date("2026-03-30T09:00:00.000Z"),
- };
-
- expect(
- toApiEvaluator({
- template,
- evaluationRuleCount: 0,
- }),
- ).toMatchObject({
- id: "tmpl_code",
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- variables: [...CODE_EVAL_TEMPLATE_VARIABLES],
- sourceCodeLanguage: "TYPESCRIPT",
- });
- });
-
- it("normalizes legacy evaluator output definitions into the public response shape", () => {
- const template: StoredPublicEvaluatorTemplate = {
- id: "tmpl_legacy",
- projectId: "project_123",
- name: "answer-correctness",
- version: 1,
- type: EvalTemplateType.LLM_AS_JUDGE,
- prompt: "Judge {{input}} against {{output}}",
- partner: null,
- provider: null,
- model: null,
- modelParams: null,
- vars: ["input", "output"],
- variableMapping: null,
- outputDefinition: {
- reasoning: "Explain why the answer is correct or incorrect.",
- score: "Return a score between 0 and 1.",
- },
- sourceCode: null,
- sourceCodeLanguage: null,
- createdAt: new Date("2026-03-30T08:00:00.000Z"),
- updatedAt: new Date("2026-03-30T09:00:00.000Z"),
- };
-
- expect(
- toApiEvaluator({
- template,
- evaluationRuleCount: 0,
- }),
- ).toMatchObject({
- outputDefinition: {
- dataType: "NUMERIC",
- reasoning: {
- description: "Explain why the answer is correct or incorrect.",
- },
- score: {
- description: "Return a score between 0 and 1.",
- },
- },
- });
- });
-});
diff --git a/web/src/__tests__/server/unit/unstable-evals-queries.servertest.ts b/web/src/__tests__/server/unit/unstable-evals-queries.servertest.ts
deleted file mode 100644
index 1fde058e51..0000000000
--- a/web/src/__tests__/server/unit/unstable-evals-queries.servertest.ts
+++ /dev/null
@@ -1,130 +0,0 @@
-import type * as PrismaClientModule from "@prisma/client";
-import type { Mock } from "vitest";
-
-vi.mock("@langfuse/shared/src/db", async () => {
- const { EvalTemplateType, JobConfigState } =
- await vi.importActual<typeof PrismaClientModule>("@prisma/client");
- return {
- EvalTemplateType,
- JobConfigState,
- prisma: {
- evaluationRule: {
- count: vi.fn(),
- findFirst: vi.fn(),
- findMany: vi.fn(),
- },
- evaluator: { findFirst: vi.fn() },
- },
- };
-});
-
-import { prisma } from "@langfuse/shared/src/db";
-import { countActiveEvaluationRules } from "@/src/features/evals/v2/server/rules/ruleErrors";
-import {
- findPublicV2EvaluatorById,
- findPublicV2EvaluatorInFamily,
- findPublicV2EvaluationRule,
- listPublicEvaluationRulePage,
-} from "@/src/features/evals/server/unstable-public-api/queries";
-
-describe("unstable public eval queries", () => {
- beforeEach(() => vi.clearAllMocks());
-
- it("lists rules exclusively from evaluation_rules", async () => {
- (prisma.evaluationRule.findMany as Mock).mockResolvedValue([
- { id: "rule" },
- ]);
- (prisma.evaluationRule.count as Mock).mockResolvedValue(1);
-
- await expect(
- listPublicEvaluationRulePage({
- projectId: "project",
- page: 1,
- limit: 20,
- }),
- ).resolves.toEqual({ records: [{ id: "rule" }], totalItems: 1 });
-
- expect(prisma.evaluationRule.findMany).toHaveBeenCalledWith(
- expect.objectContaining({
- where: expect.objectContaining({
- projectId: "project",
- targetObject: {
- in: ["event", "experiment", "trace", "dataset"],
- },
- }),
- }),
- );
- });
-
- it("counts active rules exclusively from evaluation_rules", async () => {
- (prisma.evaluationRule.count as Mock).mockResolvedValue(4);
-
- await expect(
- countActiveEvaluationRules({ prisma, projectId: "project" }),
- ).resolves.toBe(4);
- expect(prisma.evaluationRule.count).toHaveBeenCalledWith({
- where: expect.objectContaining({
- targetObject: { in: ["event", "experiment"] },
- }),
- });
- });
-
- it("loads rules and assignments within the project", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue({ id: "rule" });
-
- await expect(
- findPublicV2EvaluationRule({
- projectId: "project",
- evaluationRuleId: "rule",
- }),
- ).resolves.toEqual({ id: "rule" });
- expect(prisma.evaluationRule.findFirst).toHaveBeenCalledWith(
- expect.objectContaining({ where: { id: "rule", projectId: "project" } }),
- );
- });
-
- it("resolves project evaluators from the evaluators table", async () => {
- (prisma.evaluator.findFirst as Mock).mockResolvedValue({
- id: "evaluator",
- versions: [{ version: 2 }],
- });
-
- await expect(
- findPublicV2EvaluatorInFamily({
- projectId: "project",
- evaluator: {
- name: "Quality",
- type: "llm_as_judge",
- },
- }),
- ).resolves.toMatchObject({ id: "evaluator" });
- await expect(
- findPublicV2EvaluatorById({
- projectId: "project",
- evaluatorId: "evaluator",
- }),
- ).resolves.toMatchObject({ id: "evaluator" });
- });
-
- it("resolves a family within the project by name and type", async () => {
- (prisma.evaluator.findFirst as Mock).mockResolvedValue(null);
-
- await expect(
- findPublicV2EvaluatorInFamily({
- projectId: "project",
- evaluator: {
- name: "Missing",
- type: "llm_as_judge",
- },
- }),
- ).resolves.toBeNull();
- expect(prisma.evaluator.findFirst).toHaveBeenCalledWith(
- expect.objectContaining({
- where: expect.objectContaining({
- projectId: "project",
- name: "Missing",
- }),
- }),
- );
- });
-});
diff --git a/web/src/__tests__/server/unit/unstable-evals-service.servertest.ts b/web/src/__tests__/server/unit/unstable-evals-service.servertest.ts
deleted file mode 100644
index f1918047a1..0000000000
--- a/web/src/__tests__/server/unit/unstable-evals-service.servertest.ts
+++ /dev/null
@@ -1,584 +0,0 @@
-import type * as PrismaClientModule from "@prisma/client";
-import type { Mock } from "vitest";
-
-const { evaluator, codeEvaluator } = vi.hoisted(() => ({
- codeEvaluator: {
- id: "code-evaluator",
- projectId: "project",
- name: "Toxicity",
- type: "CODE" as const,
- description: null,
- createdByUserId: null,
- blockedAt: null,
- blockReason: null,
- blockMessage: null,
- createdAt: new Date("2026-01-01"),
- updatedAt: new Date("2026-01-01"),
- versions: [
- {
- id: "code-version",
- evaluatorId: "code-evaluator",
- version: 1,
- createdAt: new Date("2026-01-01"),
- createdByUserId: null,
- prompt: null,
- partner: null,
- provider: null,
- model: null,
- modelParams: null,
- vars: [],
- variableMapping: null,
- outputDefinition: null,
- sourceCode: "def evaluate(ctx): return []",
- sourceCodeLanguage: "PYTHON" as const,
- },
- ],
- },
- evaluator: {
- id: "evaluator",
- projectId: "project",
- name: "Quality",
- type: "LLM_AS_JUDGE" as const,
- description: null,
- createdByUserId: null,
- blockedAt: null,
- blockReason: null,
- blockMessage: null,
- createdAt: new Date("2026-01-01"),
- updatedAt: new Date("2026-01-01"),
- versions: [
- {
- id: "version",
- evaluatorId: "evaluator",
- version: 1,
- createdAt: new Date("2026-01-01"),
- createdByUserId: null,
- prompt: "Judge {{output}}",
- partner: null,
- provider: "openai",
- model: "gpt-4o-mini",
- modelParams: {},
- vars: ["output"],
- variableMapping: [
- {
- templateVariable: "output",
- selectedColumnId: "output",
- jsonSelector: null,
- },
- ],
- outputDefinition: { score: { type: "NUMERIC", min: 0, max: 1 } },
- sourceCode: null,
- sourceCodeLanguage: null,
- },
- ],
- },
-}));
-
-vi.mock("@langfuse/shared/src/db", async () => {
- const actual =
- await vi.importActual<typeof PrismaClientModule>("@prisma/client");
- const prisma = {
- evaluator: { count: vi.fn(), findMany: vi.fn() },
- evaluationRule: {
- count: vi.fn(),
- findFirst: vi.fn(),
- create: vi.fn(),
- delete: vi.fn(),
- deleteMany: vi.fn(),
- update: vi.fn(),
- },
- jobExecution: { deleteMany: vi.fn() },
- $transaction: vi.fn(),
- evaluationRuleEvaluatorAssignment: { update: vi.fn(), create: vi.fn() },
- };
- prisma.$transaction.mockImplementation(
- (callback: (client: typeof prisma) => unknown) => callback(prisma),
- );
- return {
- EvalTemplateType: actual.EvalTemplateType,
- JobConfigState: actual.JobConfigState,
- Prisma: { DbNull: { dbNull: true } },
- prisma,
- };
-});
-
-vi.mock("@/src/features/evals/server/unstable-public-api/queries", () => ({
- findPublicV2EvaluatorByIdOrThrow: vi.fn().mockResolvedValue(evaluator),
- findPublicV2EvaluatorInFamilyOrThrow: vi.fn().mockResolvedValue(evaluator),
- findPublicV2EvaluationRule: vi.fn(),
- listPublicEvaluationRulePage: vi.fn(),
-}));
-
-vi.mock(
- "@/src/features/evals/server/unstable-public-api/validation",
- async () => ({
- ...(await vi.importActual(
- "@/src/features/evals/server/unstable-public-api/validation",
- )),
- assertEvaluationRuleFilterValuesExistForProject: vi.fn(),
- assertEvaluatorDefinitionCanRunForPublicApi: vi.fn(),
- }),
-);
-
-vi.mock("@/src/features/evals/server/codeEvalJobConfigValidation", () => ({
- CodeEvalJobConfigError: class CodeEvalJobConfigError extends Error {
- constructor(
- message: string,
- public readonly code = "preflight_failed",
- ) {
- super(message);
- }
- },
- assertCodeEvalRuleCanRun: vi.fn(),
-}));
-
-vi.mock("@/src/features/evals/server/isCodeEvalEnabled", () => ({
- isCodeEvalEnabled: vi.fn().mockReturnValue(true),
- isCodeEvalSourceCodeLanguageSupported: vi.fn().mockReturnValue(true),
-}));
-
-vi.mock("@langfuse/shared/src/server", async () => ({
- ...(await vi.importActual("@langfuse/shared/src/server")),
- invalidateProjectEvalConfigCaches: vi.fn(),
-}));
-
-import { prisma } from "@langfuse/shared/src/db";
-import {
- CodeEvalJobConfigError,
- assertCodeEvalRuleCanRun,
-} from "@/src/features/evals/server/codeEvalJobConfigValidation";
-import {
- createPublicEvaluationRule,
- deletePublicEvaluationRule,
- listPublicEvaluationRules,
- updatePublicEvaluationRule,
-} from "@/src/features/evals/server/unstable-public-api/evaluation-rule-service";
-import {
- findPublicV2EvaluationRule,
- findPublicV2EvaluatorInFamilyOrThrow,
- listPublicEvaluationRulePage,
-} from "@/src/features/evals/server/unstable-public-api/queries";
-import { MAX_ACTIVE_EVALUATION_RULES } from "@/src/features/evals/v2/server/rules/ruleErrors";
-
-function buildCreatedRule(data: {
- projectId: string;
- name: string;
- status: string;
- targetObject: string;
- filter: unknown;
- sampling: number;
-}) {
- return {
- id: "rule",
- createdAt: new Date("2026-01-01"),
- updatedAt: new Date("2026-01-01"),
- projectId: data.projectId,
- createdByUserId: null,
- name: data.name,
- status: data.status,
- targetObject: data.targetObject,
- filter: data.filter,
- sampling: {
- toNumber: () => data.sampling,
- valueOf: () => data.sampling,
- },
- delay: 0,
- timeScope: ["NEW"],
- assignments: [
- {
- id: "assignment",
- createdAt: new Date("2026-01-01"),
- updatedAt: new Date("2026-01-01"),
- projectId: "project",
- evaluationRuleId: "rule",
- evaluatorId: evaluator.id,
- variableMapping: null,
- evaluator,
- },
- ],
- };
-}
-
-function buildMigratedRuleWithLegacyFilter() {
- const migratedRule = buildCreatedRule({
- projectId: "project",
- name: "Migrated rule",
- status: "ACTIVE",
- targetObject: "event",
- filter: [
- {
- type: "string",
- column: "environment",
- operator: "does not contain",
- value: "langfuse-",
- },
- {
- type: "null",
- column: "experimentId",
- operator: "is null",
- value: "",
- },
- ],
- sampling: 1,
- });
- return migratedRule;
-}
-
-describe("unstable public evaluation-rule service", () => {
- beforeEach(() => {
- vi.clearAllMocks();
- (prisma.evaluator.count as Mock).mockResolvedValue(1);
- (prisma.evaluator.findMany as Mock).mockResolvedValue([evaluator]);
- (prisma.evaluationRule.count as Mock).mockResolvedValue(0);
- (findPublicV2EvaluatorInFamilyOrThrow as Mock).mockResolvedValue(evaluator);
- (findPublicV2EvaluationRule as Mock).mockImplementation(
- () => (prisma.evaluationRule.create as Mock).mock.results.at(-1)?.value,
- );
- });
-
- it("creates rules and assignments only in the new tables", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);
- (prisma.evaluationRule.create as Mock).mockImplementation(({ data }) =>
- buildCreatedRule(data),
- );
-
- await expect(
- createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "Quality rule",
- evaluator: {
- name: "Quality",
- type: "llm_as_judge",
- },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "output", source: "output" }],
- },
- }),
- ).resolves.toMatchObject({ id: "rule", target: "observation" });
- expect(prisma.evaluationRule.create).toHaveBeenCalledOnce();
- });
-
- it("resolves the evaluator family by name and type within the project", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);
- (prisma.evaluationRule.create as Mock).mockImplementation(({ data }) =>
- buildCreatedRule(data),
- );
-
- await createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "Quality rule",
- evaluator: { name: "Quality", type: "llm_as_judge" },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "output", source: "output" }],
- },
- });
-
- expect(findPublicV2EvaluatorInFamilyOrThrow).toHaveBeenCalledWith({
- projectId: "project",
- evaluator: { name: "Quality", type: "llm_as_judge" },
- });
- });
-
- it("rejects creating more active evaluation rules than the shared cap", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);
- (prisma.evaluationRule.count as Mock).mockResolvedValue(
- MAX_ACTIVE_EVALUATION_RULES,
- );
-
- await expect(
- createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "One too many",
- evaluator: {
- name: "Quality",
- type: "llm_as_judge",
- },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "output", source: "output" }],
- },
- }),
- ).rejects.toMatchObject({
- httpCode: 409,
- code: "conflict",
- details: { limit: MAX_ACTIVE_EVALUATION_RULES },
- });
- expect(prisma.evaluationRule.create).not.toHaveBeenCalled();
- });
-
- it("does not check the active limit for disabled evaluation rules", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);
- (prisma.evaluationRule.create as Mock).mockImplementation(({ data }) =>
- buildCreatedRule(data),
- );
-
- await createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "Disabled rule",
- evaluator: { name: "Quality", type: "llm_as_judge" },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "output", source: "output" }],
- },
- });
-
- expect(prisma.evaluationRule.count).not.toHaveBeenCalled();
- });
-
- it("returns a name conflict when a rule with that name already exists", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue({
- id: "existing-rule",
- });
-
- await expect(
- createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "Quality rule",
- evaluator: {
- name: "Quality",
- type: "llm_as_judge",
- },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [],
- mapping: [{ variable: "output", source: "output" }],
- },
- }),
- ).rejects.toMatchObject({
- httpCode: 409,
- code: "name_conflict",
- details: { field: "name" },
- });
- expect(prisma.evaluationRule.create).not.toHaveBeenCalled();
- });
-
- it("translates code-eval preflight failures into structured public API errors", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);
- (findPublicV2EvaluatorInFamilyOrThrow as Mock).mockResolvedValue(
- codeEvaluator,
- );
- (assertCodeEvalRuleCanRun as Mock).mockRejectedValue(
- new CodeEvalJobConfigError("Sandbox run failed", "preflight_failed"),
- );
-
- await expect(
- createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "Code rule",
- evaluator: { name: "Toxicity", type: "code" },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- },
- }),
- ).rejects.toMatchObject({
- httpCode: 422,
- code: "evaluator_preflight_failed",
- message: "Sandbox run failed",
- details: { evaluatorName: "Toxicity" },
- });
- expect(prisma.evaluationRule.create).not.toHaveBeenCalled();
- });
-
- it("translates code-eval invalid-target failures into a 400", async () => {
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(null);
- (findPublicV2EvaluatorInFamilyOrThrow as Mock).mockResolvedValue(
- codeEvaluator,
- );
- (assertCodeEvalRuleCanRun as Mock).mockRejectedValue(
- new CodeEvalJobConfigError("Unsupported target", "invalid_target"),
- );
-
- await expect(
- createPublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- input: {
- name: "Code rule",
- evaluator: { name: "Toxicity", type: "code" },
- target: "observation",
- enabled: true,
- sampling: 1,
- filter: [],
- },
- }),
- ).rejects.toMatchObject({ httpCode: 400, code: "invalid_request" });
- });
-
- it("lists migrated rules with filters that do not match their current target", async () => {
- const migratedRule = buildMigratedRuleWithLegacyFilter();
- (listPublicEvaluationRulePage as Mock).mockResolvedValue({
- records: [migratedRule],
- totalItems: 1,
- });
-
- await expect(
- listPublicEvaluationRules({ projectId: "project", page: 1, limit: 50 }),
- ).resolves.toMatchObject({
- data: [
- {
- id: "rule",
- filter: [
- {
- type: "string",
- column: "environment",
- operator: "does not contain",
- value: "langfuse-",
- },
- {
- type: "null",
- column: "experimentId",
- operator: "is null",
- value: "",
- },
- ],
- },
- ],
- meta: { totalItems: 1 },
- });
- });
-
- it("preserves migrated filters on patches that do not replace them", async () => {
- const migratedRule = buildMigratedRuleWithLegacyFilter();
- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.update as Mock).mockImplementation(({ data }) => {
- migratedRule.name = data.name ?? migratedRule.name;
- return migratedRule;
- });
-
- await expect(
- updatePublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- evaluationRuleId: "rule",
- input: { name: "Renamed rule", target: "observation" },
- }),
- ).resolves.toMatchObject({
- name: "Renamed rule",
- filter: [{ column: "environment" }, { column: "experimentId" }],
- });
- expect(prisma.evaluationRule.update).toHaveBeenCalledOnce();
- });
-
- it("repairs migrated rules with a V2-compatible replacement filter", async () => {
- const migratedRule = buildMigratedRuleWithLegacyFilter();
- const replacementFilter = [
- {
- type: "boolean" as const,
- column: "isRootObservation",
- operator: "=" as const,
- value: true,
- },
- ];
- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.update as Mock).mockImplementation(({ data }) => {
- migratedRule.filter = data.filter;
- migratedRule.status = data.status;
- return migratedRule;
- });
-
- await expect(
- updatePublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- evaluationRuleId: "rule",
- input: {
- target: "observation",
- filter: replacementFilter,
- enabled: false,
- },
- }),
- ).resolves.toMatchObject({ filter: replacementFilter, enabled: false });
- expect(prisma.evaluationRule.update).toHaveBeenCalledOnce();
- });
-
- it("requires a replacement filter when changing the target of a migrated rule", async () => {
- const migratedRule = buildMigratedRuleWithLegacyFilter();
- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);
-
- await expect(
- updatePublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- evaluationRuleId: "rule",
- input: { target: "experiment" },
- }),
- ).rejects.toMatchObject({
- httpCode: 400,
- code: "invalid_body",
- details: { field: "filter" },
- });
- expect(prisma.evaluationRule.update).not.toHaveBeenCalled();
- });
-
- it.each([
- {
- operation: "disabled",
- mutate: () =>
- updatePublicEvaluationRule({
- orgId: "org",
- projectId: "project",
- evaluationRuleId: "rule",
- input: { enabled: false },
- }),
- },
- {
- operation: "deleted",
- mutate: () =>
- deletePublicEvaluationRule({
- projectId: "project",
- evaluationRuleId: "rule",
- }),
- },
- ])(
- "allows a migrated rule with an unsupported stored filter to be $operation",
- async ({ mutate, operation }) => {
- const migratedRule = buildMigratedRuleWithLegacyFilter();
- (findPublicV2EvaluationRule as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.findFirst as Mock).mockResolvedValue(migratedRule);
- (prisma.evaluationRule.update as Mock).mockImplementation(({ data }) => {
- migratedRule.status = data.status ?? migratedRule.status;
- return migratedRule;
- });
- (prisma.evaluationRule.deleteMany as Mock).mockResolvedValue({
- count: 1,
- });
- (prisma.jobExecution.deleteMany as Mock).mockResolvedValue({ count: 0 });
-
- const result = await mutate();
- if (operation === "disabled") {
- expect(result).toMatchObject({ enabled: false, status: "inactive" });
- expect(prisma.evaluationRule.update).toHaveBeenCalledOnce();
- } else {
- expect(prisma.evaluationRule.deleteMany).toHaveBeenCalledOnce();
- }
- },
- );
-});
diff --git a/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts b/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts
index fa3f361771..35cbf747e0 100644
--- a/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts
+++ b/web/src/__tests__/server/unstable-dashboard-widgets-api.servertest.ts
@@ -8,7 +8,7 @@ import {
PatchUnstableDashboardWidgetResponse,
PostUnstableDashboardWidgetResponse,
} from "@/src/features/public-api/types/unstable-dashboard-widgets";
-import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/unstable-public-evals-contract";
+import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/structuredPublicApiErrorSchema";
import { DashboardWidgetViews, prisma } from "@langfuse/shared/src/db";
import {
createOrgProjectAndApiKey,
diff --git a/web/src/__tests__/server/unstable-dashboards-api.servertest.ts b/web/src/__tests__/server/unstable-dashboards-api.servertest.ts
index e9459c8694..a749d0b61b 100644
--- a/web/src/__tests__/server/unstable-dashboards-api.servertest.ts
+++ b/web/src/__tests__/server/unstable-dashboards-api.servertest.ts
@@ -19,7 +19,7 @@ import {
PostDashboardPlacementResponse,
PostUnstableDashboardResponse,
} from "@/src/features/public-api/types/unstable-dashboards";
-import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/unstable-public-evals-contract";
+import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/structuredPublicApiErrorSchema";
import { createOrgProjectAndApiKey } from "@langfuse/shared/src/server";
import { prisma } from "@langfuse/shared/src/db";
import { nanoid } from "nanoid";
diff --git a/web/src/__tests__/server/unstable-evals-api.servertest.ts b/web/src/__tests__/server/unstable-evals-api.servertest.ts
deleted file mode 100644
index 1b1c2d720a..0000000000
--- a/web/src/__tests__/server/unstable-evals-api.servertest.ts
+++ /dev/null
@@ -1,1123 +0,0 @@
-import {
- makeAPICall,
- makeZodVerifiedAPICall,
-} from "@/src/__tests__/test-utils";
-import {
- DeleteUnstableEvaluationRuleResponse,
- GetUnstableEvaluationRuleResponse,
- GetUnstableEvaluationRulesResponse,
- PostUnstableEvaluationRuleResponse,
- PatchUnstableEvaluationRuleResponse,
-} from "@/src/features/public-api/types/unstable-evaluation-rules";
-import {
- DeleteUnstableEvaluatorResponse,
- GetUnstableEvaluatorResponse,
- GetUnstableEvaluatorsResponse,
- PostUnstableEvaluatorResponse,
-} from "@/src/features/public-api/types/unstable-evaluators";
-import {
- createNumericEvalOutputDefinition,
- LLMAdapter,
-} from "@langfuse/shared";
-import { prisma } from "@langfuse/shared/src/db";
-import {
- createAndAddApiKeysToDb,
- createBasicAuthHeader,
- createOrgProjectAndApiKey,
-} from "@langfuse/shared/src/server";
-import { ApiKeyScope } from "@prisma/client";
-import { StructuredPublicApiErrorResponse } from "@/src/features/public-api/types/unstable-public-evals-contract";
-import type { z } from "zod";
-import { encrypt } from "@langfuse/shared/encryption";
-
-const __orgIds: string[] = [];
-const __managedTemplateIds: string[] = [];
-
-const numericOutputDefinition = createNumericEvalOutputDefinition({
- reasoningDescription: "Why the score was assigned",
- scoreDescription: "A score between 0 and 1",
-});
-
-const provisionDefaultEvalModel = async (projectId: string) => {
- const provider = `openai-${projectId}`;
-
- const llmApiKey = await prisma.llmApiKeys.create({
- data: {
- projectId,
- provider,
- adapter: LLMAdapter.OpenAI,
- secretKey: encrypt("sk-test"),
- displaySecretKey: "...test",
- baseURL: "https://api.openai.com/v1",
- customModels: [],
- withDefaultModels: true,
- extraHeaders: null,
- extraHeaderKeys: [],
- },
- });
-
- await prisma.defaultLlmModel.create({
- data: {
- projectId,
- llmApiKeyId: llmApiKey.id,
- provider,
- adapter: LLMAdapter.OpenAI,
- model: "gpt-4.1-mini",
- },
- });
-};
-
-const expectUnstableError = (
- response: Awaited<ReturnType<typeof makeAPICall>>,
- params: {
- status: number;
- code: z.infer<typeof StructuredPublicApiErrorResponse>["code"];
- },
-) => {
- expect(response.status).toBe(params.status);
- const body = StructuredPublicApiErrorResponse.parse(response.body);
- expect(body.code).toBe(params.code);
- return body;
-};
-
-const createManagedEvaluator = async (params: {
- name: string;
- version: number;
- prompt?: string;
-}) => {
- const template = await prisma.evalTemplate.create({
- data: {
- projectId: null,
- name: params.name,
- version: params.version,
- prompt: params.prompt ?? "Judge {{input}} against {{output}}",
- partner: "ragas",
- vars: ["input", "output"],
- outputDefinition: numericOutputDefinition,
- },
- });
-
- __managedTemplateIds.push(template.id);
- return template;
-};
-
-describe("/api/public/unstable evaluators API", () => {
- let auth: string;
- let projectId: string;
-
- beforeEach(async () => {
- const result = await createOrgProjectAndApiKey();
- auth = result.auth;
- projectId = result.projectId;
- __orgIds.push(result.orgId);
- await provisionDefaultEvalModel(result.projectId);
- });
-
- afterAll(async () => {
- await prisma.jobConfiguration.deleteMany({
- where: {
- evalTemplateId: {
- in: __managedTemplateIds,
- },
- },
- });
- await prisma.evalTemplate.deleteMany({
- where: {
- id: {
- in: __managedTemplateIds,
- },
- },
- });
- await prisma.organization.deleteMany({
- where: {
- id: {
- in: __orgIds,
- },
- },
- });
- });
-
- it("keeps one evaluator id across versions and returns only the latest version", async () => {
- const v1 = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Answer correctness",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- const v2 = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Answer correctness",
- prompt: "Judge {{input}} versus {{output}}",
- outputDefinition: numericOutputDefinition,
- mapping: [
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ],
- },
- auth,
- );
-
- expect(v1.body).toMatchObject({
- id: expect.any(String),
- name: "Answer correctness",
- version: 1,
- });
- expect(v2.body).toMatchObject({
- id: expect.any(String),
- name: "Answer correctness",
- version: 2,
- });
- expect(v2.body.id).toBe(v1.body.id);
-
- const fetched = await makeZodVerifiedAPICall(
- GetUnstableEvaluatorResponse,
- "GET",
- `/api/public/unstable/evaluators/${v1.body.id}`,
- undefined,
- auth,
- );
- expect(fetched.body).toMatchObject({
- id: v1.body.id,
- version: 2,
- mapping: [
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ],
- });
-
- const listed = await makeZodVerifiedAPICall(
- GetUnstableEvaluatorsResponse,
- "GET",
- "/api/public/unstable/evaluators?page=1&limit=50",
- undefined,
- auth,
- );
-
- const projectFamilyEntries = listed.body.data.filter(
- (evaluator) => evaluator.name === "Answer correctness",
- );
-
- expect(projectFamilyEntries).toEqual([
- expect.objectContaining({
- id: v2.body.id,
- name: "Answer correctness",
- version: 2,
- }),
- ]);
- expect(listed.body.data).toContainEqual(
- expect.objectContaining({ id: v1.body.id, version: 2 }),
- );
- });
-
- it("deletes an evaluator including all of its versions by evaluator id", async () => {
- const v1 = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Deletable correctness",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- const v2 = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Deletable correctness",
- prompt: "Judge {{input}} versus {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- expect(v2.body.id).toBe(v1.body.id);
- const deleted = await makeZodVerifiedAPICall(
- DeleteUnstableEvaluatorResponse,
- "DELETE",
- `/api/public/unstable/evaluators/${v1.body.id}`,
- undefined,
- auth,
- );
- expect(deleted.body.message).toBe("Evaluator successfully deleted");
-
- const fetchLatest = await makeAPICall(
- "GET",
- `/api/public/unstable/evaluators/${v2.body.id}`,
- undefined,
- auth,
- );
- expectUnstableError(fetchLatest, {
- status: 404,
- code: "resource_not_found",
- });
- });
-
- it("returns 404 when deleting an unknown evaluator", async () => {
- const response = await makeAPICall(
- "DELETE",
- "/api/public/unstable/evaluators/unknown-evaluator-id",
- undefined,
- auth,
- );
- expectUnstableError(response, {
- status: 404,
- code: "resource_not_found",
- });
- });
-
- it("does not resolve evaluators from legacy managed templates", async () => {
- await createManagedEvaluator({
- name: "Answer relevance",
- version: 3,
- });
-
- const response = await makeAPICall(
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "answer_relevance_managed",
- evaluator: {
- name: "Answer relevance",
- },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [],
- mapping: [
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ],
- },
- auth,
- );
- expectUnstableError(response, {
- status: 404,
- code: "resource_not_found",
- });
- });
-
- it("rejects unsupported filters without creating a rule", async () => {
- const response = await makeAPICall(
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "unsupported_filter_rule",
- evaluator: { name: "unused", type: "llm_as_judge" },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [
- {
- column: "totalCost",
- type: "number",
- operator: ">",
- value: 0.01,
- },
- ],
- },
- auth,
- );
-
- expectUnstableError(response, { status: 400, code: "invalid_body" });
- await expect(
- prisma.evaluationRule.count({ where: { projectId } }),
- ).resolves.toBe(0);
- });
-
- it("uses public project evaluators in observation rules without changing the API contract", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Public project evaluator",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- const created = await makeZodVerifiedAPICall(
- PostUnstableEvaluationRuleResponse,
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "public_project_rule",
- evaluator: {
- name: evaluator.body.name,
- type: "llm_as_judge",
- },
- target: "observation",
- enabled: false,
- sampling: 0.5,
- filter: [],
- mapping: [
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ],
- },
- auth,
- );
- expect(created.body).toMatchObject({
- name: "public_project_rule",
- evaluator: { id: evaluator.body.id },
- target: "observation",
- enabled: false,
- sampling: 0.5,
- });
-
- await expect(
- prisma.evaluationRule.findUniqueOrThrow({
- where: { id: created.body.id },
- include: { assignments: true },
- }),
- ).resolves.toMatchObject({
- assignments: [{ evaluatorId: evaluator.body.id }],
- });
-
- const fetched = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRuleResponse,
- "GET",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- undefined,
- auth,
- );
- expect(fetched.body).toEqual(created.body);
-
- const listed = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRulesResponse,
- "GET",
- "/api/public/unstable/evaluation-rules?page=1&limit=50",
- undefined,
- auth,
- );
- expect(listed.body.data).toContainEqual(created.body);
-
- const updated = await makeZodVerifiedAPICall(
- PatchUnstableEvaluationRuleResponse,
- "PATCH",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- { name: "public_project_rule_updated", enabled: false },
- auth,
- );
- expect(updated.body).toMatchObject({
- id: created.body.id,
- name: "public_project_rule_updated",
- enabled: false,
- status: "inactive",
- evaluator: { id: evaluator.body.id },
- });
-
- const deleted = await makeZodVerifiedAPICall(
- DeleteUnstableEvaluationRuleResponse,
- "DELETE",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- undefined,
- auth,
- );
- expect(deleted.body.message).toBe("Evaluation rule successfully deleted");
- await expect(
- prisma.evaluationRule.findUnique({ where: { id: created.body.id } }),
- ).resolves.toBeNull();
- });
-
- it("returns incomplete inherited legacy mappings from create and read endpoints", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Legacy mapping evaluator",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
- await prisma.evaluatorVersion.updateMany({
- where: { evaluatorId: evaluator.body.id },
- data: {
- variableMapping: [
- {
- templateVariable: "input",
- langfuseObject: "trace",
- objectName: null,
- selectedColumnId: "input",
- jsonSelector: null,
- },
- {
- templateVariable: "output",
- langfuseObject: "trace",
- objectName: null,
- selectedColumnId: "output",
- jsonSelector: null,
- },
- ],
- },
- });
-
- const created = await makeZodVerifiedAPICall(
- PostUnstableEvaluationRuleResponse,
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "incomplete_inherited_mapping_rule",
- evaluator: { name: evaluator.body.name },
- target: "observation",
- enabled: false,
- sampling: 1,
- filter: [],
- },
- auth,
- );
- const incompleteMapping = [
- { variable: "input", source: null },
- { variable: "output", source: null },
- ];
- expect(created.body).toMatchObject({
- mapping: incompleteMapping,
- evaluators: [{ mapping: incompleteMapping }],
- });
-
- const fetched = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRuleResponse,
- "GET",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- undefined,
- auth,
- );
- expect(fetched.body).toEqual(created.body);
-
- const listed = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRulesResponse,
- "GET",
- "/api/public/unstable/evaluation-rules?page=1&limit=50",
- undefined,
- auth,
- );
- expect(listed.body.data).toContainEqual(created.body);
- });
-
- it("creates and returns multiple evaluator assignments with compatibility aliases", async () => {
- const createEvaluator = (name: string) =>
- makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name,
- prompt: "Judge {{input}}",
- outputDefinition: numericOutputDefinition,
- mapping: [{ variable: "input", source: "input" }],
- },
- auth,
- );
- const [correctness, tone] = await Promise.all([
- createEvaluator("Multi-rule correctness"),
- createEvaluator("Multi-rule tone"),
- ]);
-
- const created = await makeZodVerifiedAPICall(
- PostUnstableEvaluationRuleResponse,
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "multi_evaluator_rule",
- evaluators: [
- {
- evaluator: {
- name: correctness.body.name,
- type: "llm_as_judge",
- },
- },
- {
- evaluator: {
- name: tone.body.name,
- type: "llm_as_judge",
- },
- mapping: [{ variable: "input", source: "output" }],
- },
- ],
- target: "observation",
- enabled: false,
- filter: [],
- },
- auth,
- );
-
- expect(created.body).toMatchObject({
- evaluator: { id: correctness.body.id },
- mapping: [{ variable: "input", source: "input" }],
- evaluators: [
- { evaluator: { id: correctness.body.id }, mapping: null },
- {
- evaluator: { id: tone.body.id },
- mapping: [{ variable: "input", source: "output" }],
- },
- ],
- });
-
- const updated = await makeZodVerifiedAPICall(
- PatchUnstableEvaluationRuleResponse,
- "PATCH",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- { name: "multi_evaluator_rule_updated" },
- auth,
- );
- expect(updated.body.evaluators).toHaveLength(2);
- await expect(
- prisma.evaluationRuleEvaluatorAssignment.count({
- where: { evaluationRuleId: created.body.id },
- }),
- ).resolves.toBe(2);
- });
-
- it("replaces the whole assignment set through evaluators on patch", async () => {
- const createEvaluator = (name: string) =>
- makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name,
- prompt: "Judge {{input}}",
- outputDefinition: numericOutputDefinition,
- mapping: [{ variable: "input", source: "input" }],
- },
- auth,
- );
- const [first, second, third] = await Promise.all([
- createEvaluator("Patch set first"),
- createEvaluator("Patch set second"),
- createEvaluator("Patch set third"),
- ]);
-
- const created = await makeZodVerifiedAPICall(
- PostUnstableEvaluationRuleResponse,
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "patch_evaluator_set_rule",
- evaluators: [
- { evaluator: { name: first.body.name, type: "llm_as_judge" } },
- { evaluator: { name: second.body.name, type: "llm_as_judge" } },
- ],
- target: "observation",
- enabled: false,
- filter: [],
- },
- auth,
- );
-
- // Drops `second`, keeps `first`, attaches `third` with an override.
- const updated = await makeZodVerifiedAPICall(
- PatchUnstableEvaluationRuleResponse,
- "PATCH",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- {
- evaluators: [
- { evaluator: { name: first.body.name, type: "llm_as_judge" } },
- {
- evaluator: { name: third.body.name, type: "llm_as_judge" },
- mapping: [{ variable: "input", source: "output" }],
- },
- ],
- },
- auth,
- );
-
- expect(updated.body.evaluators).toEqual([
- expect.objectContaining({
- evaluator: expect.objectContaining({ id: first.body.id }),
- mapping: null,
- }),
- expect.objectContaining({
- evaluator: expect.objectContaining({ id: third.body.id }),
- mapping: [{ variable: "input", source: "output" }],
- }),
- ]);
- await expect(
- prisma.evaluationRuleEvaluatorAssignment.count({
- where: { evaluationRuleId: created.body.id },
- }),
- ).resolves.toBe(2);
- });
-
- it("keeps a rule readable and deletable after its last evaluator is detached", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Detached rule evaluator",
- prompt: "Judge {{input}}",
- outputDefinition: numericOutputDefinition,
- mapping: [{ variable: "input", source: "input" }],
- },
- auth,
- );
- const created = await makeZodVerifiedAPICall(
- PostUnstableEvaluationRuleResponse,
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "detached_assignments_rule",
- evaluator: { name: evaluator.body.name, type: "llm_as_judge" },
- target: "observation",
- enabled: false,
- filter: [],
- },
- auth,
- );
-
- // Mirrors detaching the last evaluator from the app or MCP tools.
- await prisma.evaluationRuleEvaluatorAssignment.deleteMany({
- where: { evaluationRuleId: created.body.id },
- });
-
- const fetched = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRuleResponse,
- "GET",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- undefined,
- auth,
- );
- expect(fetched.body).toMatchObject({
- id: created.body.id,
- evaluator: null,
- evaluators: [],
- mapping: [],
- });
-
- const listed = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRulesResponse,
- "GET",
- "/api/public/unstable/evaluation-rules?page=1&limit=50",
- undefined,
- auth,
- );
- expect(listed.body.data.map((rule) => rule.id)).toContain(created.body.id);
-
- const deleted = await makeAPICall(
- "DELETE",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- undefined,
- auth,
- );
- expect(deleted.status).toBe(200);
- });
-
- it("rejects create bodies that combine evaluators with the deprecated mapping alias", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Conflicting alias evaluator",
- prompt: "Judge {{input}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- const response = await makeAPICall(
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "conflicting_alias_rule",
- evaluators: [
- { evaluator: { name: evaluator.body.name, type: "llm_as_judge" } },
- ],
- mapping: [{ variable: "input", source: "input" }],
- target: "observation",
- enabled: false,
- filter: [],
- },
- auth,
- );
- expect(response.status).toBe(400);
- });
-
- it("stores experiment rules and evaluator usage in the new tables", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Project experiment evaluator",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- const created = await makeZodVerifiedAPICall(
- PostUnstableEvaluationRuleResponse,
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "project_experiment_rule",
- evaluator: {
- name: evaluator.body.name,
- type: "llm_as_judge",
- },
- target: "experiment",
- enabled: false,
- filter: [],
- mapping: [
- { variable: "input", source: "input" },
- { variable: "output", source: "output" },
- ],
- },
- auth,
- );
- expect(created.body).toMatchObject({
- evaluator: { id: evaluator.body.id },
- target: "experiment",
- enabled: false,
- });
- await expect(
- prisma.evaluationRule.findUniqueOrThrow({
- where: { id: created.body.id },
- include: { assignments: true },
- }),
- ).resolves.toMatchObject({
- targetObject: "event",
- filter: [
- {
- column: "isExperimentItemRootSpan",
- operator: "=",
- value: true,
- },
- ],
- assignments: [{ evaluatorId: evaluator.body.id }],
- });
- await expect(
- prisma.jobConfiguration.findUnique({ where: { id: created.body.id } }),
- ).resolves.toBeNull();
-
- const observationRule = await makeZodVerifiedAPICall(
- PatchUnstableEvaluationRuleResponse,
- "PATCH",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- { target: "observation" },
- auth,
- );
- expect(observationRule.body).toMatchObject({ target: "observation" });
- await expect(
- prisma.evaluationRule.findUniqueOrThrow({
- where: { id: created.body.id },
- }),
- ).resolves.toMatchObject({ targetObject: "event", filter: [] });
-
- const experimentRule = await makeZodVerifiedAPICall(
- PatchUnstableEvaluationRuleResponse,
- "PATCH",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- { target: "experiment" },
- auth,
- );
- expect(experimentRule.body).toMatchObject({ target: "experiment" });
- await expect(
- prisma.evaluationRule.findUniqueOrThrow({
- where: { id: created.body.id },
- }),
- ).resolves.toMatchObject({
- targetObject: "event",
- filter: [
- {
- column: "isExperimentItemRootSpan",
- operator: "=",
- value: true,
- },
- ],
- });
-
- // An experiment rule is an observation rule scoped to experiment root
- // spans, so observation filters survive the round trip alongside the
- // implicit root filter rather than failing the experiment filter schema.
- const scopedExperimentRule = await makeZodVerifiedAPICall(
- PatchUnstableEvaluationRuleResponse,
- "PATCH",
- `/api/public/unstable/evaluation-rules/${created.body.id}`,
- {
- target: "experiment",
- filter: [
- {
- type: "stringOptions",
- column: "environment",
- operator: "any of",
- value: ["production"],
- },
- ],
- },
- auth,
- );
- expect(scopedExperimentRule.body).toMatchObject({
- target: "experiment",
- filter: [{ column: "environment", value: ["production"] }],
- });
- await expect(
- prisma.evaluationRule.findUniqueOrThrow({
- where: { id: created.body.id },
- }),
- ).resolves.toMatchObject({
- targetObject: "event",
- filter: [
- { column: "environment" },
- { column: "isExperimentItemRootSpan", operator: "=", value: true },
- ],
- });
-
- const fetchedEvaluator = await makeZodVerifiedAPICall(
- GetUnstableEvaluatorResponse,
- "GET",
- `/api/public/unstable/evaluators/${evaluator.body.id}`,
- undefined,
- auth,
- );
- expect(fetchedEvaluator.body.evaluationRuleCount).toBe(1);
-
- await makeZodVerifiedAPICall(
- DeleteUnstableEvaluatorResponse,
- "DELETE",
- `/api/public/unstable/evaluators/${evaluator.body.id}`,
- undefined,
- auth,
- );
- await expect(
- prisma.evaluationRule.findUnique({
- where: { id: created.body.id },
- include: { assignments: true },
- }),
- ).resolves.toMatchObject({ id: created.body.id, assignments: [] });
- });
-
- it("reads native trace and dataset rules but keeps them read-only", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Project trace evaluator",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
- const mapping = [
- {
- variable: "input",
- langfuseObject: "trace",
- objectName: null,
- source: "input",
- },
- {
- variable: "output",
- langfuseObject: "generation",
- objectName: "answer-generation",
- source: "output",
- jsonPath: "$.answer",
- },
- ];
-
- const nativeRule = await prisma.evaluationRule.create({
- data: {
- projectId,
- name: "project_trace_rule",
- targetObject: "trace",
- status: "ACTIVE",
- sampling: 0.5,
- delay: 250,
- timeScope: ["NEW"],
- filter: [],
- assignments: {
- create: {
- projectId,
- evaluatorId: evaluator.body.id,
- variableMapping: mapping.map((item) => ({
- templateVariable: item.variable,
- langfuseObject: item.langfuseObject,
- objectName: item.objectName,
- selectedColumnId: item.source,
- jsonSelector: item.jsonPath ?? null,
- })),
- },
- },
- },
- });
-
- const fetched = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRuleResponse,
- "GET",
- `/api/public/unstable/evaluation-rules/${nativeRule.id}`,
- undefined,
- auth,
- );
- expect(fetched.body).toMatchObject({
- id: nativeRule.id,
- name: "project_trace_rule",
- evaluator: { id: evaluator.body.id },
- target: "trace",
- enabled: true,
- sampling: 0.5,
- delay: 250,
- timeScope: ["NEW"],
- mapping,
- });
-
- const listed = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRulesResponse,
- "GET",
- "/api/public/unstable/evaluation-rules?page=1&limit=50",
- undefined,
- auth,
- );
- expect(listed.body.data).toContainEqual(fetched.body);
-
- const createResponse = await makeAPICall(
- "POST",
- "/api/public/unstable/evaluation-rules",
- {
- name: "new_trace_rule",
- evaluator: {
- name: evaluator.body.name,
- type: "llm_as_judge",
- },
- target: "trace",
- enabled: false,
- filter: [],
- mapping,
- },
- auth,
- );
- expectUnstableError(createResponse, { status: 400, code: "invalid_body" });
-
- const updateResponse = await makeAPICall(
- "PATCH",
- `/api/public/unstable/evaluation-rules/${nativeRule.id}`,
- { name: "project_trace_rule_updated", enabled: false },
- auth,
- );
- expectUnstableError(updateResponse, {
- status: 404,
- code: "resource_not_found",
- });
-
- const deleteResponse = await makeAPICall(
- "DELETE",
- `/api/public/unstable/evaluation-rules/${nativeRule.id}`,
- undefined,
- auth,
- );
- expectUnstableError(deleteResponse, {
- status: 404,
- code: "resource_not_found",
- });
-
- const nativeDatasetRule = await prisma.evaluationRule.create({
- data: {
- projectId,
- name: "project_dataset_rule",
- targetObject: "dataset",
- status: "INACTIVE",
- sampling: 1,
- delay: 0,
- timeScope: ["EXISTING"],
- filter: [],
- assignments: {
- create: {
- projectId,
- evaluatorId: evaluator.body.id,
- variableMapping: mapping.map((item) => ({
- templateVariable: item.variable,
- langfuseObject: "dataset_item",
- objectName: null,
- selectedColumnId: item.source,
- jsonSelector: item.jsonPath ?? null,
- })),
- },
- },
- },
- });
- const fetchedDatasetRule = await makeZodVerifiedAPICall(
- GetUnstableEvaluationRuleResponse,
- "GET",
- `/api/public/unstable/evaluation-rules/${nativeDatasetRule.id}`,
- undefined,
- auth,
- );
- expect(fetchedDatasetRule.body).toMatchObject({
- id: nativeDatasetRule.id,
- target: "dataset",
- enabled: false,
- timeScope: ["EXISTING"],
- mapping: [
- expect.objectContaining({ langfuseObject: "dataset_item" }),
- expect.objectContaining({ langfuseObject: "dataset_item" }),
- ],
- });
- });
-
- it("returns method_not_allowed for evaluator patch", async () => {
- const evaluator = await makeZodVerifiedAPICall(
- PostUnstableEvaluatorResponse,
- "POST",
- "/api/public/unstable/evaluators",
- {
- name: "Correctness",
- prompt: "Judge {{input}} against {{output}}",
- outputDefinition: numericOutputDefinition,
- },
- auth,
- );
-
- const patchRes = await makeAPICall(
- "PATCH",
- `/api/public/unstable/evaluators/${evaluator.body.id}`,
- {
- prompt: "Updated",
- },
- auth,
- );
-
- expectUnstableError(patchRes, {
- status: 405,
- code: "method_not_allowed",
- });
- });
-
- it("still rejects invalid auth with the unstable error envelope", async () => {
- const result = await createOrgProjectAndApiKey();
- __orgIds.push(result.orgId);
- const orgApiKey = await createAndAddApiKeysToDb({
- prisma,
- entityId: result.orgId,
- scope: ApiKeyScope.ORGANIZATION,
- });
-
- const response = await makeAPICall(
- "GET",
- "/api/public/unstable/evaluators?page=1&limit=10",
- undefined,
- createBasicAuthHeader(orgApiKey.publicKey, orgApiKey.secretKey),
- );
-
- expectUnstableError(response, {
- status: 403,
- code: "access_denied",
- });
- });
-});
diff --git a/web/src/__tests__/server/unstable-evaluator-v2-api.servertest.ts b/web/src/__tests__/server/unstable-evaluator-v2-api.servertest.ts
deleted file mode 100644
index ee73cdbec8..0000000000
--- a/web/src/__tests__/server/unstable-evaluator-v2-api.servertest.ts
+++ /dev/null
@@ -1,304 +0,0 @@
-import {
- afterAll,
- afterEach,
- beforeAll,
- describe,
- expect,
- it,
- vi,
-} from "vitest";
-
-vi.mock(
- "@/src/features/evals/v2/server/evaluators/evaluatorValidation",
- () => ({ assertEvaluatorConfigurationValid: vi.fn() }),
-);
-
-import { Prisma, prisma } from "@langfuse/shared/src/db";
-import { createOrgProjectAndApiKey } from "@langfuse/shared/src/server";
-import {
- createPublicEvaluator,
- deletePublicEvaluator,
- getPublicEvaluator,
- listPublicEvaluators,
-} from "@/src/features/evals/server/unstable-public-api/evaluator-service";
-
-const orgIds: string[] = [];
-let projectId = "";
-let otherProjectId = "";
-
-const input = (name: string, prompt: string) => ({
- name,
- type: "llm_as_judge" as const,
- prompt,
- modelConfig: null,
- outputDefinition: {
- dataType: "NUMERIC" as const,
- score: { description: "Quality" },
- reasoning: { description: "Reasoning" },
- },
-});
-
-beforeAll(async () => {
- const [first, second] = await Promise.all([
- createOrgProjectAndApiKey(),
- createOrgProjectAndApiKey(),
- ]);
- projectId = first.project.id;
- otherProjectId = second.project.id;
- orgIds.push(first.org.id, second.org.id);
-});
-
-afterEach(async () => {
- await prisma.evaluationRule.deleteMany({
- where: { projectId: { in: [projectId, otherProjectId] } },
- });
- await prisma.evaluator.deleteMany({
- where: { projectId: { in: [projectId, otherProjectId] } },
- });
-});
-
-afterAll(async () => {
- await prisma.organization.deleteMany({ where: { id: { in: orgIds } } });
-});
-
-describe("unstable evaluator API on stable evaluator storage", () => {
- it("creates and versions an evaluator by name", async () => {
- const first = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}}"),
- });
- const second = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}} strictly"),
- });
- const third = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}} strictly"),
- });
-
- expect(first).toMatchObject({ name: "Public evaluator", version: 1 });
- expect(second).toMatchObject({ name: first.name, version: 2 });
- expect(third).toMatchObject({ name: first.name, version: 3 });
- expect(second.id).toBe(first.id);
- expect(third.id).toBe(first.id);
- });
-
- it("returns the latest evaluator and its rule count", async () => {
- const first = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}}"),
- });
- const latest = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}} strictly"),
- });
- await prisma.evaluationRule.create({
- data: {
- projectId,
- name: "Public evaluator rule",
- status: "INACTIVE",
- targetObject: "EVENT",
- filter: [],
- sampling: 1,
- delay: 0,
- assignments: {
- create: {
- projectId,
- evaluatorId: first.id,
- variableMapping: Prisma.DbNull,
- },
- },
- },
- });
-
- await expect(
- getPublicEvaluator({ projectId, evaluatorId: first.id }),
- ).resolves.toMatchObject({
- id: first.id,
- version: latest.version,
- evaluationRuleCount: 1,
- });
- });
-
- it("ignores stale legacy job configurations when counting evaluator rules", async () => {
- const evaluator = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}}"),
- });
- await prisma.evalTemplate.create({
- data: {
- id: evaluator.id,
- projectId,
- name: evaluator.name,
- version: 1,
- prompt: "Judge {{output}}",
- type: "LLM_AS_JUDGE",
- vars: ["output"],
- outputDefinition: Prisma.DbNull,
- },
- });
- const legacyRule = await prisma.jobConfiguration.create({
- data: {
- projectId,
- jobType: "EVAL",
- status: "INACTIVE",
- evalTemplateId: evaluator.id,
- scoreName: "Stale legacy rule",
- targetObject: "event",
- filter: [],
- variableMapping: [],
- sampling: 1,
- delay: 0,
- },
- });
-
- await expect(
- getPublicEvaluator({ projectId, evaluatorId: evaluator.id }),
- ).resolves.toMatchObject({ evaluationRuleCount: 0 });
-
- await prisma.jobConfiguration.delete({ where: { id: legacyRule.id } });
- await prisma.evalTemplate.delete({ where: { id: evaluator.id } });
- });
-
- it("rejects legacy version IDs and cross-project reads", async () => {
- const evaluator = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}}"),
- });
- const oldVersion = await prisma.evaluatorVersion.findFirstOrThrow({
- where: { evaluatorId: evaluator.id, version: 1 },
- select: { id: true },
- });
- await expect(
- getPublicEvaluator({ projectId, evaluatorId: oldVersion.id }),
- ).rejects.toThrow("Evaluator not found");
- await expect(
- getPublicEvaluator({
- projectId: otherProjectId,
- evaluatorId: evaluator.id,
- }),
- ).rejects.toThrow("Evaluator not found");
- });
-
- it("returns an empty evaluator list", async () => {
- await expect(
- listPublicEvaluators({ projectId, page: 1, limit: 50 }),
- ).resolves.toMatchObject({
- data: [],
- meta: { page: 1, limit: 50, totalItems: 0, totalPages: 0 },
- });
- });
-
- it("lists only the latest evaluators from the requested project", async () => {
- const first = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}}"),
- });
- const latest = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}} strictly"),
- });
- await createPublicEvaluator({
- projectId: otherProjectId,
- input: input("Public evaluator", "Other project {{output}}"),
- });
-
- const projectList = await listPublicEvaluators({
- projectId,
- page: 1,
- limit: 50,
- });
- expect(projectList.data).toEqual([
- expect.objectContaining({ id: first.id, version: latest.version }),
- ]);
- expect(projectList.meta).toMatchObject({ totalItems: 1, totalPages: 1 });
- });
-
- it("paginates evaluator lists", async () => {
- const [older, newer] = await Promise.all([
- createPublicEvaluator({
- projectId,
- input: input("Older evaluator", "Judge {{output}}"),
- }),
- createPublicEvaluator({
- projectId,
- input: input("Newer evaluator", "Judge {{output}}"),
- }),
- ]);
- await Promise.all([
- prisma.evaluator.update({
- where: { id: older.id },
- data: { updatedAt: new Date("2026-01-01T00:00:00.000Z") },
- }),
- prisma.evaluator.update({
- where: { id: newer.id },
- data: { updatedAt: new Date("2026-01-02T00:00:00.000Z") },
- }),
- ]);
-
- await expect(
- listPublicEvaluators({ projectId, page: 1, limit: 1 }),
- ).resolves.toMatchObject({
- data: [{ id: newer.id }],
- meta: { page: 1, limit: 1, totalItems: 2, totalPages: 2 },
- });
- await expect(
- listPublicEvaluators({ projectId, page: 2, limit: 1 }),
- ).resolves.toMatchObject({
- data: [{ id: older.id }],
- meta: { page: 2, limit: 1, totalItems: 2, totalPages: 2 },
- });
- });
-
- it("deletes the evaluator and its rule assignments", async () => {
- const evaluator = await createPublicEvaluator({
- projectId,
- input: input("Public evaluator", "Judge {{output}}"),
- });
- const rule = await prisma.evaluationRule.create({
- data: {
- projectId,
- name: "Public evaluator rule",
- status: "INACTIVE",
- targetObject: "EVENT",
- filter: [],
- sampling: 1,
- delay: 0,
- assignments: {
- create: {
- projectId,
- evaluatorId: evaluator.id,
- variableMapping: Prisma.DbNull,
- },
- },
- },
- });
-
- await deletePublicEvaluator({ projectId, evaluatorId: evaluator.id });
-
- await expect(
- prisma.evaluationRuleEvaluatorAssignment.count({
- where: { evaluationRuleId: rule.id, evaluatorId: evaluator.id },
- }),
- ).resolves.toBe(0);
- await expect(
- prisma.evaluator.findUnique({ where: { id: evaluator.id } }),
- ).resolves.toBeNull();
- });
-
- it("rejects ambiguous legacy name upserts", async () => {
- await prisma.evaluator.createMany({
- data: [
- { projectId, name: "Duplicate", type: "LLM_AS_JUDGE" },
- { projectId, name: "Duplicate", type: "LLM_AS_JUDGE" },
- ],
- });
-
- await expect(
- createPublicEvaluator({
- projectId,
- input: input("Duplicate", "Judge {{output}}"),
- }),
- ).rejects.toMatchObject({ httpCode: 409 });
- });
-});
diff --git a/web/src/features/evals/server/codeEvalJobConfigValidation.ts b/web/src/features/evals/server/codeEvalJobConfigValidation.ts
index c9c84c4262..c40ff7f1dd 100644
--- a/web/src/features/evals/server/codeEvalJobConfigValidation.ts
+++ b/web/src/features/evals/server/codeEvalJobConfigValidation.ts
@@ -9,7 +9,6 @@ import type { PrismaClient } from "@prisma/client";
import { env } from "@/src/env.mjs";
import {
CodeEvalTestRunSetupError,
- runCodeEvalTestForEvaluationRule,
runCodeEvalTestForJobConfig,
} from "@/src/features/evals/server/codeEvalTestRun";
import { assertUnreachable } from "@/src/utils/types";
@@ -51,26 +50,6 @@ export async function assertCodeEvalJobConfigCanRun(params: {
});
}
-export async function assertCodeEvalRuleCanRun(params: {
- prisma: PrismaClient;
- orgId: string;
- projectId: string;
- evaluatorId: string;
- target: EvalTargetObjectType;
- mapping: unknown;
- scoreName: string;
- filter: FilterCondition[] | null;
-}): Promise<void> {
- return assertCodeEvalCanRun({
- ...params,
- run: (mapping) =>
- runCodeEvalTestForEvaluationRule({
- ...params,
- mapping,
- }),
- });
-}
-
async function assertCodeEvalCanRun(params: {
target: EvalTargetObjectType;
mapping: unknown;
diff --git a/web/src/features/evals/server/codeEvalTestRun.ts b/web/src/features/evals/server/codeEvalTestRun.ts
index e094161f09..18fd176d1b 100644
--- a/web/src/features/evals/server/codeEvalTestRun.ts
+++ b/web/src/features/evals/server/codeEvalTestRun.ts
@@ -115,62 +115,6 @@ export async function runCodeEvalTestForJobConfig(params: {
});
}
-export async function runCodeEvalTestForEvaluationRule(params: {
- prisma: PrismaClient;
- orgId: string;
- projectId: string;
- evaluatorId: string;
- target: EvalTargetObject;
- mapping: ObservationVariableMapping[];
- scoreName: string;
- filter: FilterCondition[] | null;
-}): Promise<CodeEvalTestRunResult | null> {
- const observation = await getObservationForEvalByFilter({
- prisma: params.prisma,
- projectId: params.projectId,
- target: params.target,
- filter: params.filter,
- });
-
- if (!observation) return null;
-
- const evaluator = await params.prisma.evaluator.findFirst({
- where: {
- id: params.evaluatorId,
- projectId: params.projectId,
- type: EvalTemplateType.CODE,
- },
- include: {
- versions: {
- where: { sourceCode: { not: null }, sourceCodeLanguage: { not: null } },
- orderBy: { version: "desc" },
- take: 1,
- },
- },
- });
- const version = evaluator?.versions[0];
- if (!evaluator || !version?.sourceCode || !version.sourceCodeLanguage) {
- throw new CodeEvalTestRunSetupError(
- "TEMPLATE_NOT_FOUND",
- "Evaluator not found",
- );
- }
-
- return runCodeEvalTestForObservationWithEvaluator({
- ...params,
- observation,
- evaluator,
- version: {
- ...version,
- sourceCode: version.sourceCode,
- sourceCodeLanguage: version.sourceCodeLanguage,
- },
- evaluatorMetadata: {
- evaluator_version: version.version,
- },
- });
-}
-
async function runCodeEvalTestForObservation(params: {
prisma: PrismaClient;
orgId: string;
diff --git a/web/src/features/evals/server/unstable-public-api/adapters.ts b/web/src/features/evals/server/unstable-public-api/adapters.ts
deleted file mode 100644
index 42f6f516c0..0000000000
--- a/web/src/features/evals/server/unstable-public-api/adapters.ts
+++ /dev/null
@@ -1,690 +0,0 @@
-import {
- createBooleanEvalOutputDefinition,
- createCategoricalEvalOutputDefinition,
- createNumericEvalOutputDefinition,
- EvalTargetObject,
- extractVariables,
- JobTimeScopeZod,
- JobConfigState,
- isExperimentEvaluationRule,
- observationVariableMappingList,
- PersistedEvalOutputDefinitionSchema,
- resolvePersistedEvalOutputDefinition,
- singleFilter,
- stripExperimentRootFilter,
- type FilterState,
- type ObservationVariableMapping,
- type PersistedEvalOutputDefinition,
- variableMappingList,
- InternalServerError,
- CODE_EVAL_TEMPLATE_VARIABLES,
- getCodeEvalVariableMapping,
-} from "@langfuse/shared";
-import { EvalTemplateType } from "@langfuse/shared/src/db";
-import { logger } from "@langfuse/shared/src/server";
-import { z } from "zod";
-import {
- getLegacyEvaluatorPrompt,
- reconcileEvaluatorPromptMessages,
-} from "@/src/features/evals/v2/server/evaluators/evaluatorService";
-import {
- LegacyPromptVariableMapping,
- PUBLIC_EVALUATOR_TYPE_CODE,
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- PublicEvaluationRuleFilter,
- PublicEvaluationRuleReadFilter,
- type PublicEvaluationRuleFilterType,
- type PromptVariableMappingInputType,
- type PromptVariableMappingReadType,
- type PublicEvaluationRuleReadTargetType,
- type PublicEvaluationRuleStatusType,
- type PublicEvaluationRuleTargetType,
- type LegacyPromptVariableMappingType,
- type PublicEvaluatorModelConfigType,
- type PublicEvaluatorOutputDefinitionType,
- type PublicEvaluatorTypeType,
-} from "@/src/features/public-api/server";
-import type {
- ApiEvaluatorRecord,
- ApiEvaluationRuleRecord,
- ApiWritableEvaluationRuleRecord,
- StoredPublicEvaluatorTemplate,
- StoredPublicV2EvaluationRule,
-} from "./types";
-import {
- validateEvaluationRuleFilters,
- validateEvaluatorVariableMappings,
-} from "./validation";
-
-export function toPublicEvaluatorType(type: EvalTemplateType) {
- return type === EvalTemplateType.CODE
- ? PUBLIC_EVALUATOR_TYPE_CODE
- : PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE;
-}
-
-export function toStoredEvaluatorType(
- type: PublicEvaluatorTypeType,
-): EvalTemplateType {
- return type === PUBLIC_EVALUATOR_TYPE_CODE
- ? EvalTemplateType.CODE
- : EvalTemplateType.LLM_AS_JUDGE;
-}
-
-const PUBLIC_TARGET_TO_INTERNAL_TARGET_OBJECT = {
- observation: EvalTargetObject.EVENT,
- experiment: EvalTargetObject.EXPERIMENT,
-} satisfies Record<PublicEvaluationRuleTargetType, EvalTargetObject>;
-
-const INTERNAL_TARGET_OBJECT_TO_PUBLIC_TARGET: Record<
- string,
- PublicEvaluationRuleReadTargetType
-> = {
- [EvalTargetObject.EVENT]: "observation",
- [EvalTargetObject.EXPERIMENT]: "experiment",
- [EvalTargetObject.TRACE]: "trace",
- [EvalTargetObject.DATASET]: "dataset",
-};
-
-const PUBLIC_MAPPING_SOURCE_TO_INTERNAL_COLUMN: Record<
- PromptVariableMappingInputType["source"],
- ObservationVariableMapping["selectedColumnId"]
-> = {
- input: "input",
- output: "output",
- metadata: "metadata",
- tool_calls: "toolCalls",
- expected_output: "experimentItemExpectedOutput",
- experiment_item_metadata: "experimentItemMetadata",
-};
-
-const INTERNAL_MAPPING_COLUMN_TO_PUBLIC_SOURCE: Record<
- string,
- PromptVariableMappingInputType["source"]
-> = {
- input: "input",
- output: "output",
- metadata: "metadata",
- // Only camelCase for tool calls: the column id is new with tool-call
- // support, so unlike expected_output no legacy snake_case rows exist. An
- // accidental "tool_calls" write should surface at the corrupted-mapping
- // error boundary, not be absorbed here.
- toolCalls: "tool_calls",
- expected_output: "expected_output",
- expectedOutput: "expected_output",
- experiment_item_expected_output: "expected_output",
- experimentItemExpectedOutput: "expected_output",
- experimentItemMetadata: "experiment_item_metadata",
- experiment_item_metadata: "experiment_item_metadata",
-};
-
-export function deriveEvaluatorVariables(
- template: Pick<
- StoredPublicEvaluatorTemplate,
- "vars" | "prompt" | "promptMessages"
- >,
-) {
- return template.vars.length > 0
- ? template.vars
- : reconcileEvaluatorPromptMessages(template).flatMap(({ content }) =>
- extractVariables(content),
- );
-}
-
-export function toStoredVariableMappings(params: {
- mappings: PromptVariableMappingInputType[];
- variables: string[];
- target: PublicEvaluationRuleTargetType;
-}) {
- validateEvaluatorVariableMappings({
- mappings: params.mappings,
- variables: params.variables,
- target: params.target,
- });
-
- return toStoredMappingList(params.mappings);
-}
-
-/**
- * Shape-only translation of the public mapping contract to storage. Use this
- * where the evaluator's variables are not known and coverage is validated
- * elsewhere; prefer `toStoredVariableMappings` when they are.
- */
-export function toStoredMappingList(
- mappings: PromptVariableMappingInputType[],
-) {
- return observationVariableMappingList.parse(
- mappings.map((mapping) => ({
- templateVariable: mapping.variable,
- selectedColumnId:
- PUBLIC_MAPPING_SOURCE_TO_INTERNAL_COLUMN[mapping.source],
- jsonSelector: mapping.jsonPath ?? null,
- })),
- );
-}
-
-function parseStoredOutputDefinition(
- template: Pick<StoredPublicEvaluatorTemplate, "id" | "outputDefinition">,
-): PublicEvaluatorOutputDefinitionType {
- const parsed = PersistedEvalOutputDefinitionSchema.safeParse(
- template.outputDefinition,
- );
-
- if (!parsed.success) {
- logger.error(
- "Failed to parse unstable public evaluator output definition",
- {
- issues: parsed.error.issues,
- templateId: "id" in template ? template.id : undefined,
- },
- );
- throw new InternalServerError("Evaluator output definition is corrupted");
- }
-
- const resolvedOutputDefinition = resolvePersistedEvalOutputDefinition(
- parsed.data,
- );
-
- if (resolvedOutputDefinition.dataType === "CATEGORICAL") {
- return {
- dataType: "CATEGORICAL",
- reasoning: {
- description: resolvedOutputDefinition.reasoningDescription,
- },
- score: {
- description: resolvedOutputDefinition.scoreDescription,
- categories: resolvedOutputDefinition.categories,
- shouldAllowMultipleMatches:
- resolvedOutputDefinition.shouldAllowMultipleMatches,
- },
- };
- }
-
- if (resolvedOutputDefinition.dataType === "NUMERIC") {
- const { minValue, maxValue } = resolvedOutputDefinition;
- return {
- dataType: "NUMERIC",
- reasoning: {
- description: resolvedOutputDefinition.reasoningDescription,
- },
- score: {
- description: resolvedOutputDefinition.scoreDescription,
- ...(minValue !== undefined ? { minValue } : {}),
- ...(maxValue !== undefined ? { maxValue } : {}),
- },
- };
- }
-
- return {
- dataType: resolvedOutputDefinition.dataType,
- reasoning: {
- description: resolvedOutputDefinition.reasoningDescription,
- },
- score: {
- description: resolvedOutputDefinition.scoreDescription,
- },
- };
-}
-
-export function toStoredOutputDefinition(
- outputDefinition: PublicEvaluatorOutputDefinitionType,
-): PersistedEvalOutputDefinition {
- switch (outputDefinition.dataType) {
- case "NUMERIC":
- return createNumericEvalOutputDefinition({
- reasoningDescription: outputDefinition.reasoning.description,
- scoreDescription: outputDefinition.score.description,
- minValue: outputDefinition.score.minValue,
- maxValue: outputDefinition.score.maxValue,
- });
- case "BOOLEAN":
- return createBooleanEvalOutputDefinition({
- reasoningDescription: outputDefinition.reasoning.description,
- scoreDescription: outputDefinition.score.description,
- });
- case "CATEGORICAL":
- return createCategoricalEvalOutputDefinition({
- reasoningDescription: outputDefinition.reasoning.description,
- scoreDescription: outputDefinition.score.description,
- categories: outputDefinition.score.categories,
- shouldAllowMultipleMatches:
- outputDefinition.score.shouldAllowMultipleMatches,
- });
- }
-}
-
-function toApiModelConfig(
- template: Pick<
- StoredPublicEvaluatorTemplate,
- "provider" | "model" | "modelParams"
- >,
-): PublicEvaluatorModelConfigType | null {
- if (!template.provider || !template.model) {
- return null;
- }
-
- return {
- provider: template.provider,
- model: template.model,
- };
-}
-
-function assertPublicTarget(
- targetObject: string,
-): PublicEvaluationRuleReadTargetType {
- const publicTarget = INTERNAL_TARGET_OBJECT_TO_PUBLIC_TARGET[targetObject];
-
- if (!publicTarget) {
- throw new InternalServerError("Evaluation rule target is corrupted");
- }
-
- return publicTarget;
-}
-
-function toStoredFilter(
- filter: PublicEvaluationRuleFilterType,
- target: PublicEvaluationRuleTargetType,
-): PublicEvaluationRuleFilterType {
- if (target === "experiment" && filter.column === "datasetId") {
- return {
- ...filter,
- column: "experimentDatasetId",
- };
- }
-
- return filter;
-}
-
-function toApiFilter(filter: z.infer<typeof singleFilter>) {
- if (filter.column === "experimentDatasetId") {
- return {
- ...filter,
- column: "datasetId",
- };
- }
-
- return filter;
-}
-
-export function toApiReadMappings(
- mappings: unknown,
-): PromptVariableMappingReadType[] {
- const parsed = observationVariableMappingList.safeParse(mappings);
-
- if (!parsed.success) {
- logger.error("Failed to parse unstable public evaluation rule mappings", {
- issues: parsed.error.issues,
- });
- throw new InternalServerError("Evaluation rule mapping is corrupted");
- }
-
- return parsed.data.map((mapping) => {
- const source =
- INTERNAL_MAPPING_COLUMN_TO_PUBLIC_SOURCE[mapping.selectedColumnId];
-
- if (!source && mapping.selectedColumnId.trim()) {
- throw new InternalServerError("Evaluation rule mapping is corrupted");
- }
-
- return {
- variable: mapping.templateVariable,
- source: source ?? null,
- ...(mapping.jsonSelector ? { jsonPath: mapping.jsonSelector } : {}),
- };
- });
-}
-
-export function toApiMappings(
- mappings: unknown,
-): PromptVariableMappingInputType[] {
- return toApiReadMappings(mappings).map((mapping) => {
- if (mapping.source === null) {
- throw new InternalServerError("Evaluation rule mapping is corrupted");
- }
-
- return { ...mapping, source: mapping.source };
- });
-}
-
-function toApiLegacyMappings(
- mappings: unknown,
-): LegacyPromptVariableMappingType[] {
- const parsed = variableMappingList.safeParse(mappings);
-
- if (!parsed.success) {
- logger.error("Failed to parse unstable public legacy rule mappings", {
- issues: parsed.error.issues,
- });
- throw new InternalServerError("Evaluation rule mapping is corrupted");
- }
-
- const publicMappings = parsed.data.map((mapping) => ({
- variable: mapping.templateVariable,
- langfuseObject: mapping.langfuseObject,
- objectName: mapping.objectName ?? null,
- source: mapping.selectedColumnId,
- ...(mapping.jsonSelector ? { jsonPath: mapping.jsonSelector } : {}),
- }));
- const parsedPublicMappings = z
- .array(LegacyPromptVariableMapping)
- .safeParse(publicMappings);
- if (!parsedPublicMappings.success) {
- logger.error("Failed to parse unstable public legacy rule mappings", {
- issues: parsedPublicMappings.error.issues,
- });
- throw new InternalServerError("Evaluation rule mapping is corrupted");
- }
- return parsedPublicMappings.data;
-}
-
-function toDefaultLegacyMappings(
- mappings: unknown,
- target: "trace" | "dataset",
-): LegacyPromptVariableMappingType[] {
- return toApiMappings(mappings).map((mapping) => ({
- ...mapping,
- langfuseObject: target === "trace" ? "trace" : "dataset_item",
- objectName: null,
- }));
-}
-
-function getLegacyCodeEvalVariableMapping(target: "trace" | "dataset") {
- return variableMappingList.parse(
- CODE_EVAL_TEMPLATE_VARIABLES.map((variable) => ({
- templateVariable: variable,
- langfuseObject: target === "trace" ? "trace" : "dataset_item",
- objectName: null,
- selectedColumnId: variable,
- jsonSelector: null,
- })),
- );
-}
-
-function toApiFilters(
- filters: unknown,
- target: PublicEvaluationRuleTargetType,
-) {
- const storedFilters = z.array(singleFilter).safeParse(filters);
-
- if (!storedFilters.success) {
- logger.error("Failed to parse unstable public evaluation rule filters", {
- issues: storedFilters.error.issues,
- });
- throw new InternalServerError("Evaluation rule filter is corrupted");
- }
-
- const effectiveFilters =
- target === "experiment"
- ? stripExperimentRootFilter(storedFilters.data)
- : storedFilters.data;
- return z
- .array(PublicEvaluationRuleReadFilter)
- .parse(effectiveFilters.map(toApiFilter));
-}
-
-export function toApiEvaluator(params: {
- template: StoredPublicEvaluatorTemplate;
- evaluationRuleCount: number;
-}): ApiEvaluatorRecord {
- const template = params.template;
- const base = {
- id: template.id,
- name: template.name,
- version: template.version,
- variables: deriveEvaluatorVariables(template),
- mapping:
- template.variableMapping == null
- ? null
- : toApiReadMappings(template.variableMapping),
- evaluationRuleCount: params.evaluationRuleCount,
- createdAt: template.createdAt,
- updatedAt: template.updatedAt,
- } as const;
-
- if (template.type === EvalTemplateType.CODE) {
- if (!template.sourceCode || !template.sourceCodeLanguage) {
- throw new InternalServerError("Code evaluator definition is corrupted");
- }
-
- return {
- ...base,
- type: PUBLIC_EVALUATOR_TYPE_CODE,
- variables: [...CODE_EVAL_TEMPLATE_VARIABLES],
- sourceCode: template.sourceCode,
- sourceCodeLanguage: template.sourceCodeLanguage,
- };
- }
-
- const promptMessages = reconcileEvaluatorPromptMessages(template);
- if (promptMessages.every(({ content }) => content.length === 0)) {
- throw new InternalServerError("Evaluator prompt messages are corrupted");
- }
-
- return {
- ...base,
- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- prompt: getLegacyEvaluatorPrompt(promptMessages),
- outputDefinition: parseStoredOutputDefinition(template),
- modelConfig: toApiModelConfig(template),
- };
-}
-
-// Blocking used to live on the job configuration, so the public contract reports it on the rule.
-// In the v2 model it lives on the evaluator, so read it back off the rule's assignments to keep the
-// contract intact. Every migrated rule has exactly one evaluator, which is also the only shape the
-// pre-v2 API could produce, so those reproduce the old responses one for one. A rule with several
-// evaluators reports paused as soon as one of them is blocked, since it is no longer writing every
-// score it is configured to write.
-function toApiRuleBlockState(rule: StoredPublicV2EvaluationRule) {
- const blocked = rule.assignments
- .map((candidate) => candidate.evaluator)
- .filter(
- (candidate): candidate is typeof candidate & { blockedAt: Date } =>
- candidate.blockedAt !== null,
- )
- // most recently blocked wins, matching how the backfill picked a group's reason
- .sort(
- (left, right) => right.blockedAt.getTime() - left.blockedAt.getTime(),
- );
- const source = blocked[0];
- return {
- isBlocked: source !== undefined,
- pausedReason: source?.blockReason ?? null,
- pausedMessage: source?.blockMessage ?? null,
- };
-}
-
-function toApiEvaluationRuleStatus(
- status: JobConfigState,
- isBlocked: boolean,
-): PublicEvaluationRuleStatusType {
- if (status !== JobConfigState.ACTIVE) {
- return "inactive";
- }
- return isBlocked ? "paused" : "active";
-}
-
-export function toApiV2EvaluationRule(
- rule: StoredPublicV2EvaluationRule,
-): ApiEvaluationRuleRecord {
- // A rule can legitimately have no assignments (e.g. its last evaluator was
- // detached). The compatibility `evaluator`/`mapping` aliases then report
- // null/empty rather than the resource 404ing.
- const assignment = rule.assignments[0] ?? null;
- const evaluator = assignment?.evaluator ?? null;
- const block = toApiRuleBlockState(rule);
-
- const toEvaluator = (candidate: NonNullable<typeof assignment>) => ({
- id: candidate.evaluator.id,
- name: candidate.evaluator.name,
- type: toPublicEvaluatorType(candidate.evaluator.type),
- });
- const storedFilters = z.array(singleFilter).safeParse(rule.filter);
- const target =
- storedFilters.success &&
- isExperimentEvaluationRule({
- targetObject: rule.targetObject as EvalTargetObject,
- filter: storedFilters.data as FilterState,
- })
- ? "experiment"
- : assertPublicTarget(rule.targetObject);
- if (target === "trace" || target === "dataset") {
- const evaluators = rule.assignments.map((candidate) => ({
- evaluator: toEvaluator(candidate),
- mapping:
- candidate.variableMapping == null
- ? null
- : toApiLegacyMappings(candidate.variableMapping),
- }));
- const effectiveFirstMapping =
- assignment === null || evaluator === null
- ? []
- : assignment.variableMapping != null
- ? toApiLegacyMappings(assignment.variableMapping)
- : evaluator.type === EvalTemplateType.CODE
- ? toApiLegacyMappings(getLegacyCodeEvalVariableMapping(target))
- : toDefaultLegacyMappings(
- evaluator.versions[0]?.variableMapping,
- target,
- );
- const timeScope = z.array(JobTimeScopeZod).safeParse(rule.timeScope);
- if (!timeScope.success) {
- throw new InternalServerError("Evaluation rule time scope is corrupted");
- }
-
- return {
- id: rule.id,
- name: rule.name,
- evaluator: assignment === null ? null : toEvaluator(assignment),
- evaluators,
- enabled: rule.status === JobConfigState.ACTIVE,
- status: toApiEvaluationRuleStatus(rule.status, block.isBlocked),
- pausedReason: block.pausedReason,
- pausedMessage: block.pausedMessage,
- sampling: Number(rule.sampling),
- target,
- delay: rule.delay,
- timeScope: timeScope.data,
- filter: z.array(singleFilter).parse(rule.filter),
- mapping: effectiveFirstMapping,
- createdAt: rule.createdAt,
- updatedAt: rule.updatedAt,
- };
- }
-
- const evaluators = rule.assignments.map((candidate) => ({
- evaluator: toEvaluator(candidate),
- mapping:
- candidate.variableMapping == null
- ? null
- : toApiReadMappings(candidate.variableMapping),
- }));
- const effectiveFirstMapping =
- assignment === null || evaluator === null
- ? []
- : evaluator.type === EvalTemplateType.CODE
- ? getCodeEvalVariableMapping()
- : (assignment.variableMapping ??
- evaluator.versions[0]?.variableMapping ??
- []);
-
- return {
- id: rule.id,
- name: rule.name,
- evaluator: assignment === null ? null : toEvaluator(assignment),
- evaluators,
- enabled: rule.status === JobConfigState.ACTIVE,
- status: toApiEvaluationRuleStatus(rule.status, block.isBlocked),
- pausedReason: block.pausedReason,
- pausedMessage: block.pausedMessage,
- sampling: Number(rule.sampling),
- target,
- filter: toApiFilters(rule.filter, target),
- mapping: toApiReadMappings(effectiveFirstMapping),
- createdAt: rule.createdAt,
- updatedAt: rule.updatedAt,
- };
-}
-
-export function toApiWritableV2EvaluationRule(
- rule: StoredPublicV2EvaluationRule,
-): ApiWritableEvaluationRuleRecord {
- const evaluationRule = toApiV2EvaluationRule(rule);
- if (
- evaluationRule.target !== "observation" &&
- evaluationRule.target !== "experiment"
- ) {
- throw new InternalServerError("Evaluation rule target is corrupted");
- }
-
- const filter = z
- .array(PublicEvaluationRuleFilter)
- .safeParse(evaluationRule.filter);
- if (!filter.success) {
- throw new InternalServerError("Evaluation rule filter is corrupted");
- }
-
- return { ...evaluationRule, filter: filter.data };
-}
-
-export function toEvaluationRuleInput(params: {
- input: {
- name: string;
- target: PublicEvaluationRuleTargetType;
- enabled: boolean;
- sampling: number;
- filter: PublicEvaluationRuleFilterType[];
- mapping?: PromptVariableMappingInputType[];
- };
- evaluatorVariables: string[];
- evaluatorType: PublicEvaluatorTypeType;
-}) {
- return {
- ...toEvaluationRuleFields(params.input),
- variableMapping: toStoredAssignmentMapping({
- target: params.input.target,
- mapping: params.input.mapping,
- evaluatorVariables: params.evaluatorVariables,
- evaluatorType: params.evaluatorType,
- }),
- };
-}
-
-/**
- * Rule-level columns. Independent of any evaluator, so a rule with no
- * assignments can still be written.
- */
-export function toEvaluationRuleFields(input: {
- name: string;
- target: PublicEvaluationRuleTargetType;
- enabled: boolean;
- sampling: number;
- filter: PublicEvaluationRuleFilterType[];
-}) {
- const target = input.target;
-
- validateEvaluationRuleFilters({ target, filters: input.filter });
-
- return {
- scoreName: input.name,
- targetObject: PUBLIC_TARGET_TO_INTERNAL_TARGET_OBJECT[target],
- filter: input.filter.map((filter) => toStoredFilter(filter, target)),
- sampling: input.sampling,
- status: input.enabled ? JobConfigState.ACTIVE : JobConfigState.INACTIVE,
- };
-}
-
-/** Stored mapping for one assignment. Code evaluators inherit their fixed evaluator mapping. */
-export function toStoredAssignmentMapping(params: {
- target: PublicEvaluationRuleTargetType;
- mapping?: PromptVariableMappingInputType[];
- evaluatorVariables: string[];
- evaluatorType: PublicEvaluatorTypeType;
-}) {
- return params.evaluatorType === PUBLIC_EVALUATOR_TYPE_CODE
- ? null
- : toStoredVariableMappings({
- mappings: params.mapping ?? [],
- variables: params.evaluatorVariables,
- target: params.target,
- });
-}
diff --git a/web/src/features/evals/server/unstable-public-api/evaluation-rule-service.ts b/web/src/features/evals/server/unstable-public-api/evaluation-rule-service.ts
deleted file mode 100644
index be14fcdbd0..0000000000
--- a/web/src/features/evals/server/unstable-public-api/evaluation-rule-service.ts
+++ /dev/null
@@ -1,882 +0,0 @@
-import { type ApiAccessScope } from "@langfuse/shared/src/server";
-import { EvalTemplateType, prisma } from "@langfuse/shared/src/db";
-import {
- JobConfigState,
- InternalServerError,
- LangfuseNotFoundError,
- EvalTargetObject,
- type FilterCondition,
- type FilterState,
- type ObservationVariableMapping,
-} from "@langfuse/shared";
-import {
- assertCodeEvalRuleCanRun,
- CodeEvalJobConfigError,
-} from "@/src/features/evals/server/codeEvalJobConfigValidation";
-import { auditLog } from "@/src/features/audit-logs/auditLog";
-import { JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE } from "@/src/features/evals/server/audit-log-resource-types";
-import {
- isCodeEvalEnabled,
- isCodeEvalSourceCodeLanguageSupported,
-} from "@/src/features/evals/server/isCodeEvalEnabled";
-import { createStructuredPublicApiError } from "@/src/features/public-api";
-import {
- PublicEvaluationRuleFilter,
- type PatchUnstableEvaluationRuleBodyType,
- type PostUnstableEvaluationRuleBodyType,
- type PromptVariableMappingInputType,
- type PromptVariableMappingReadType,
- type PublicEvaluationRuleTargetType,
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
-} from "@/src/features/public-api/server";
-import {
- deriveEvaluatorVariables,
- toApiMappings,
- toApiV2EvaluationRule,
- toApiWritableV2EvaluationRule,
- toEvaluationRuleFields,
- toEvaluationRuleInput,
- toPublicEvaluatorType,
- toStoredAssignmentMapping,
-} from "./adapters";
-import {
- ActiveEvaluationRuleLimitError,
- assertActiveRuleLimitNotExceeded,
-} from "@/src/features/evals/v2/server/rules/ruleErrors";
-import { RuleService } from "@/src/features/evals/v2/server/rules/ruleService";
-import {
- findPublicV2EvaluatorByIdOrThrow,
- findPublicV2EvaluationRule,
- findPublicV2EvaluatorInFamilyOrThrow,
- listPublicEvaluationRulePage,
-} from "./queries";
-import type {
- EvaluationRuleEvaluatorFamilyReference,
- StoredPublicEvaluatorTemplate,
- StoredPublicV2EvaluationRule,
-} from "./types";
-import {
- assertEvaluationRuleFilterValuesExistForProject,
- assertEvaluatorDefinitionCanRunForPublicApi,
-} from "./validation";
-import { assertUnreachable } from "@/src/utils/types";
-
-const ruleService = new RuleService(prisma, async () => undefined);
-
-function getRequestedEvaluatorAssignments(
- input: PostUnstableEvaluationRuleBodyType,
-) {
- if (input.evaluators) return input.evaluators;
- return [{ evaluator: input.evaluator!, mapping: input.mapping }];
-}
-
-type PublicV2Evaluator = Awaited<
- ReturnType<typeof findPublicV2EvaluatorInFamilyOrThrow>
->;
-
-function toEvaluatorDefinition(
- evaluator: PublicV2Evaluator,
-): StoredPublicEvaluatorTemplate {
- const version = evaluator.versions[0];
- if (!version) {
- throw createStructuredPublicApiError({
- httpCode: 500,
- code: "internal_error",
- message: "Evaluator version is missing",
- });
- }
- return {
- id: evaluator.id,
- projectId: evaluator.projectId,
- name: evaluator.name,
- version: version.version,
- type: evaluator.type,
- prompt: version.prompt,
- promptMessages: version.promptMessages,
- partner: version.partner,
- provider: version.provider,
- model: version.model,
- modelParams: version.modelParams,
- vars: version.vars,
- outputDefinition: version.outputDefinition,
- sourceCode: version.sourceCode,
- sourceCodeLanguage: version.sourceCodeLanguage,
- variableMapping: version.variableMapping,
- createdAt: evaluator.createdAt,
- updatedAt: evaluator.updatedAt,
- };
-}
-
-async function resolveProjectEvaluator(params: {
- projectId: string;
- evaluator: EvaluationRuleEvaluatorFamilyReference;
- evaluatorId?: string;
-}) {
- return params.evaluatorId
- ? findPublicV2EvaluatorByIdOrThrow({
- projectId: params.projectId,
- evaluatorId: params.evaluatorId,
- })
- : findPublicV2EvaluatorInFamilyOrThrow({
- projectId: params.projectId,
- evaluator: params.evaluator,
- });
-}
-
-async function assertEvaluationRuleCanRunForPublicApi(params: {
- orgId: string;
- projectId: string;
- evaluatorId: string;
- template: StoredPublicEvaluatorTemplate;
- target: EvalTargetObject;
- mapping: unknown;
- scoreName: string;
- filter: FilterCondition[] | null;
-}) {
- if (params.template.type !== EvalTemplateType.CODE) {
- await assertEvaluatorDefinitionCanRunForPublicApi({
- projectId: params.projectId,
- template: {
- name: params.template.name,
- provider: params.template.provider,
- model: params.template.model,
- modelParams: params.template.modelParams,
- outputDefinition: params.template.outputDefinition,
- },
- });
- return;
- }
-
- if (!isCodeEvalEnabled()) {
- throw createStructuredPublicApiError({
- httpCode: 403,
- code: "access_denied",
- message: "Code evals are not enabled",
- details: { evaluatorName: params.template.name },
- });
- }
- if (
- !isCodeEvalSourceCodeLanguageSupported(params.template.sourceCodeLanguage)
- ) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_request",
- message:
- "This code evaluator language is not supported by the configured dispatcher.",
- details: { evaluatorName: params.template.name },
- });
- }
-
- try {
- await assertCodeEvalRuleCanRun({
- prisma,
- orgId: params.orgId,
- projectId: params.projectId,
- evaluatorId: params.evaluatorId,
- target: params.target,
- mapping: params.mapping,
- scoreName: params.scoreName,
- filter: params.filter,
- });
- } catch (error) {
- if (!(error instanceof CodeEvalJobConfigError)) throw error;
- const details = { evaluatorName: params.template.name };
- switch (error.code) {
- case "invalid_target":
- case "invalid_request":
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_request",
- message: error.message,
- details,
- });
- case "resource_not_found":
- throw createStructuredPublicApiError({
- httpCode: 404,
- code: "resource_not_found",
- message: error.message,
- details,
- });
- case "preflight_failed":
- throw createStructuredPublicApiError({
- httpCode: 422,
- code: "evaluator_preflight_failed",
- message: error.message,
- details,
- });
- default:
- return assertUnreachable(error.code);
- }
- }
-}
-
-// The cap itself lives in the rule service so tRPC, MCP, and this API cannot
-// drift; only the error envelope is public-API specific.
-async function assertActivePublicApiEvaluationRuleLimitNotExceeded(
- projectId: string,
-) {
- try {
- await assertActiveRuleLimitNotExceeded({
- prisma,
- projectId,
- additionalActiveRules: 1,
- });
- } catch (error) {
- throwPublicRuleServiceError(error);
- }
-}
-
-function throwPublicRuleServiceError(error: unknown): never {
- if (error instanceof ActiveEvaluationRuleLimitError) {
- throw createStructuredPublicApiError({
- httpCode: 409,
- code: "conflict",
- message: error.message,
- details: { limit: error.limit },
- });
- }
- throw error;
-}
-
-async function runRuleServiceMutation<T>(mutation: () => Promise<T>) {
- try {
- return await mutation();
- } catch (error) {
- throwPublicRuleServiceError(error);
- }
-}
-
-async function findPublicV2EvaluationRuleOrThrow(params: {
- projectId: string;
- evaluationRuleId: string;
-}) {
- const rule = await findPublicV2EvaluationRule(params);
- if (!rule) {
- throw new LangfuseNotFoundError(
- "Evaluation rule not found within authorized project",
- );
- }
- // A rule with no assignments is still a real, addressable resource: it must
- // stay readable and deletable rather than 404 while consuming a slot.
- return rule;
-}
-
-async function findPublicWritableV2EvaluationRuleOrThrow(params: {
- projectId: string;
- evaluationRuleId: string;
-}) {
- const rule = await findPublicV2EvaluationRuleOrThrow(params);
- if (
- rule.targetObject !== EvalTargetObject.EVENT &&
- rule.targetObject !== EvalTargetObject.EXPERIMENT
- ) {
- throw new LangfuseNotFoundError(
- "Evaluation rule not found within authorized project",
- );
- }
- return rule;
-}
-
-function toApiReadableV2EvaluationRule(rule: StoredPublicV2EvaluationRule) {
- const evaluationRule = toApiV2EvaluationRule(rule);
- if (
- evaluationRule.target !== "observation" &&
- evaluationRule.target !== "experiment"
- ) {
- throw new InternalServerError("Evaluation rule target is corrupted");
- }
- return evaluationRule;
-}
-
-export async function listPublicEvaluationRules(params: {
- projectId: string;
- page: number;
- limit: number;
-}) {
- const { records, totalItems } = await listPublicEvaluationRulePage(params);
- return {
- data: records.map(toApiV2EvaluationRule),
- meta: {
- page: params.page,
- limit: params.limit,
- totalItems,
- totalPages: Math.ceil(totalItems / params.limit),
- },
- };
-}
-
-export async function getPublicEvaluationRule(params: {
- projectId: string;
- evaluationRuleId: string;
-}) {
- return toApiV2EvaluationRule(await findPublicV2EvaluationRuleOrThrow(params));
-}
-
-export async function createPublicEvaluationRule(params: {
- orgId: string;
- projectId: string;
- input: PostUnstableEvaluationRuleBodyType;
- evaluatorId?: string;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
-}) {
- // Scoped to the targets this API can write. Rule names carry no uniqueness constraint, so this
- // only exists to hand back the id to PATCH instead of a second rule -- and PATCH 404s on the
- // trace/dataset rules the backfill brought in, so including those would be a dead end.
- const existing = await prisma.evaluationRule.findFirst({
- where: {
- projectId: params.projectId,
- name: params.input.name,
- targetObject: {
- in: [EvalTargetObject.EVENT, EvalTargetObject.EXPERIMENT],
- },
- },
- select: { id: true },
- });
- if (existing) {
- throw createStructuredPublicApiError({
- httpCode: 409,
- code: "name_conflict",
- message: `An evaluation rule named "${params.input.name}" already exists in this project. Use PATCH /api/public/unstable/evaluation-rules/${existing.id} to update it instead of creating a duplicate.`,
- details: { field: "name" },
- });
- }
- if (params.input.enabled) {
- await assertActivePublicApiEvaluationRuleLimitNotExceeded(params.projectId);
- }
- await assertEvaluationRuleFilterValuesExistForProject({
- projectId: params.projectId,
- target: params.input.target,
- filters: params.input.filter,
- });
-
- const requestedAssignments = getRequestedEvaluatorAssignments(params.input);
- const evaluators = await Promise.all(
- requestedAssignments.map(({ evaluator }, index) =>
- resolveProjectEvaluator({
- projectId: params.projectId,
- evaluator,
- evaluatorId:
- params.evaluatorId && requestedAssignments.length === 1 && index === 0
- ? params.evaluatorId
- : undefined,
- }),
- ),
- );
- if (new Set(evaluators.map(({ id }) => id)).size !== evaluators.length) {
- throw createStructuredPublicApiError({
- httpCode: 409,
- code: "conflict",
- message: "An evaluator can only be attached once to an evaluation rule.",
- });
- }
-
- const preparedAssignments = requestedAssignments.map((assignment, index) => {
- const evaluator = evaluators[index]!;
- const template = toEvaluatorDefinition(evaluator);
- const effectiveMapping =
- assignment.mapping ??
- (template.variableMapping == null
- ? undefined
- : toApiMappings(template.variableMapping));
- const data = toEvaluationRuleInput({
- input: {
- name: params.input.name,
- target: params.input.target,
- enabled: params.input.enabled,
- sampling: params.input.sampling,
- filter: params.input.filter,
- mapping: effectiveMapping,
- },
- evaluatorVariables: deriveEvaluatorVariables(template),
- evaluatorType: toPublicEvaluatorType(template.type),
- });
- return { assignment, evaluator, template, data };
- });
-
- await Promise.all(
- preparedAssignments.map(({ evaluator, template, data }) =>
- data.status === JobConfigState.ACTIVE
- ? assertEvaluationRuleCanRunForPublicApi({
- orgId: params.orgId,
- projectId: params.projectId,
- evaluatorId: evaluator.id,
- template,
- target: data.targetObject as EvalTargetObject,
- mapping: data.variableMapping,
- scoreName: data.scoreName,
- filter: data.filter,
- })
- : Promise.resolve(),
- ),
- );
-
- const firstData = preparedAssignments[0]!.data;
- const created = await createRuleWithRuleService({
- projectId: params.projectId,
- data: firstData,
- assignments: preparedAssignments,
- });
- const evaluationRule = toApiWritableV2EvaluationRule(created);
- if (params.auditScope) {
- await auditLog({
- action: "create",
- resourceType: JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE,
- resourceId: evaluationRule.id,
- projectId: params.projectId,
- orgId: params.auditScope.orgId,
- apiKeyId: params.auditScope.apiKeyId,
- after: evaluationRule,
- });
- }
- return evaluationRule;
-}
-
-async function createRuleWithRuleService(params: {
- projectId: string;
- data: ReturnType<typeof toEvaluationRuleInput>;
- assignments: Array<{
- evaluator: PublicV2Evaluator;
- assignment: { mapping?: PromptVariableMappingInputType[] };
- data: ReturnType<typeof toEvaluationRuleInput>;
- }>;
-}) {
- const created = await runRuleServiceMutation(() =>
- ruleService.create(
- {
- projectId: params.projectId,
- name: params.data.scoreName,
- targetObject: params.data.targetObject,
- enabled: params.data.status === JobConfigState.ACTIVE,
- sampling: params.data.sampling,
- filter: params.data.filter as FilterState,
- evaluatorAssignments: params.assignments.map(
- ({ assignment, evaluator, data }) => ({
- evaluatorId: evaluator.id,
- variableMapping:
- assignment.mapping === undefined
- ? null
- : (data.variableMapping as ObservationVariableMapping[]),
- }),
- ),
- },
- null,
- ),
- );
- return findPublicV2EvaluationRuleOrThrow({
- projectId: params.projectId,
- evaluationRuleId: created.id,
- });
-}
-
-type PreparedAssignment = {
- evaluatorId: string;
- template: StoredPublicEvaluatorTemplate;
- /** null stores "inherit the evaluator version's default mapping". */
- storedMapping: unknown;
- /** Mapping the rule would actually run with, for preflight validation. */
- effectiveMapping: unknown;
-};
-
-function toWritableMappingsIfComplete(
- mappings: PromptVariableMappingReadType[],
-): PromptVariableMappingInputType[] | undefined {
- const completeMappings: PromptVariableMappingInputType[] = [];
- for (const mapping of mappings) {
- if (mapping.source === null) return undefined;
- completeMappings.push({ ...mapping, source: mapping.source });
- }
- return completeMappings;
-}
-
-/**
- * Resolve one requested assignment (public shape) into its stored form.
- * An omitted mapping inherits the evaluator version's default.
- */
-async function prepareAssignment(params: {
- projectId: string;
- target: PublicEvaluationRuleTargetType;
- evaluator: EvaluationRuleEvaluatorFamilyReference;
- mapping?: PromptVariableMappingInputType[];
- evaluatorId?: string;
-}): Promise<PreparedAssignment> {
- const evaluator = await resolveProjectEvaluator({
- projectId: params.projectId,
- evaluator: params.evaluator,
- evaluatorId: params.evaluatorId,
- });
- const template = toEvaluatorDefinition(evaluator);
- const inherited =
- template.variableMapping == null
- ? undefined
- : toApiMappings(template.variableMapping);
- const storedMapping =
- params.mapping === undefined
- ? null
- : toStoredAssignmentMapping({
- target: params.target,
- mapping: params.mapping,
- evaluatorVariables: deriveEvaluatorVariables(template),
- evaluatorType: toPublicEvaluatorType(template.type),
- });
- const effectiveMapping = toStoredAssignmentMapping({
- target: params.target,
- mapping: params.mapping ?? inherited,
- evaluatorVariables: deriveEvaluatorVariables(template),
- evaluatorType: toPublicEvaluatorType(template.type),
- });
- return {
- evaluatorId: evaluator.id,
- template,
- storedMapping,
- effectiveMapping,
- };
-}
-
-function assertUniqueEvaluators(assignments: Array<{ evaluatorId: string }>) {
- const ids = assignments.map(({ evaluatorId }) => evaluatorId);
- if (new Set(ids).size !== ids.length) {
- throw createStructuredPublicApiError({
- httpCode: 409,
- code: "conflict",
- message: "An evaluator can only be attached once to an evaluation rule.",
- });
- }
-}
-
-/** Preflight every assignment the rule would run with once it is active. */
-async function assertAssignmentsCanRun(params: {
- orgId: string;
- projectId: string;
- assignments: PreparedAssignment[];
- targetObject: EvalTargetObject;
- scoreName: string;
- filter: FilterCondition[] | null;
-}) {
- await Promise.all(
- params.assignments.map((assignment) =>
- assertEvaluationRuleCanRunForPublicApi({
- orgId: params.orgId,
- projectId: params.projectId,
- evaluatorId: assignment.evaluatorId,
- template: assignment.template,
- target: params.targetObject,
- mapping: assignment.effectiveMapping,
- scoreName: params.scoreName,
- filter: params.filter,
- }),
- ),
- );
-}
-
-export async function updatePublicEvaluationRule(params: {
- orgId: string;
- projectId: string;
- evaluationRuleId: string;
- input: PatchUnstableEvaluationRuleBodyType;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
-}) {
- const existing = await findPublicWritableV2EvaluationRuleOrThrow(params);
- const existingPublic = toApiReadableV2EvaluationRule(existing);
- const nextEnabled = params.input.enabled ?? existingPublic.enabled;
- if (nextEnabled && existingPublic.status !== "active") {
- await assertActivePublicApiEvaluationRuleLimitNotExceeded(params.projectId);
- }
- const suppliedTarget =
- "target" in params.input ? params.input.target : undefined;
- const suppliedFilter =
- "filter" in params.input ? params.input.filter : undefined;
- const nextTarget = suppliedTarget ?? existingPublic.target;
- if (suppliedFilter !== undefined) {
- await assertEvaluationRuleFilterValuesExistForProject({
- projectId: params.projectId,
- target: nextTarget,
- filters: suppliedFilter,
- });
- }
- const updatesFilterOrTarget =
- suppliedFilter !== undefined ||
- (suppliedTarget !== undefined && suppliedTarget !== existingPublic.target);
- const parsedExistingFilter = PublicEvaluationRuleFilter.array().safeParse(
- existingPublic.filter,
- );
- if (
- updatesFilterOrTarget &&
- suppliedFilter === undefined &&
- !parsedExistingFilter.success
- ) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_body",
- message:
- "The stored filter is no longer accepted for writes. Provide a target-compatible filter with this update.",
- details: { field: "filter" },
- });
- }
- const ruleFields = toEvaluationRuleFields({
- name: params.input.name ?? existingPublic.name,
- target: nextTarget,
- enabled: nextEnabled,
- sampling: params.input.sampling ?? existingPublic.sampling,
- // Target changes must also validate the effective filter against the new target.
- filter: updatesFilterOrTarget
- ? (suppliedFilter ?? parsedExistingFilter.data ?? [])
- : [],
- });
-
- const firstAssignment = existing.assignments[0];
- const patchesFirstAssignment =
- params.input.evaluator !== undefined ||
- ("mapping" in params.input && params.input.mapping !== undefined);
-
- if (
- "mapping" in params.input &&
- params.input.mapping !== undefined &&
- firstAssignment?.evaluator.type === EvalTemplateType.CODE
- ) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_body",
- message:
- "Code evaluator mappings are managed by Langfuse and cannot be provided in the request body.",
- details: { field: "mapping" },
- });
- }
- if (patchesFirstAssignment && !firstAssignment && !params.input.evaluator) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_body",
- message:
- "This evaluation rule has no evaluator assignments, so there is no mapping to update. Provide `evaluators` to attach one.",
- details: { field: "mapping" },
- });
- }
-
- // Three shapes: replace the whole set, patch the first assignment (deprecated
- // aliases), or leave assignments untouched.
- const replacementAssignments = params.input.evaluators
- ? await Promise.all(
- params.input.evaluators.map((assignment) =>
- prepareAssignment({
- projectId: params.projectId,
- target: nextTarget,
- evaluator: assignment.evaluator,
- mapping: assignment.mapping,
- }),
- ),
- )
- : null;
- if (replacementAssignments) assertUniqueEvaluators(replacementAssignments);
-
- const patchedFirstAssignment =
- replacementAssignments || !patchesFirstAssignment
- ? null
- : await prepareAssignment({
- projectId: params.projectId,
- target: nextTarget,
- evaluator: params.input.evaluator
- ? {
- ...params.input.evaluator,
- // No current assignment to inherit a type from: the reference
- // schema's own default applies.
- type:
- existingPublic.evaluator?.type ??
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- }
- : {
- name: existingPublic.evaluator!.name,
- type: existingPublic.evaluator!.type,
- },
- mapping:
- "mapping" in params.input && params.input.mapping !== undefined
- ? params.input.mapping
- : toWritableMappingsIfComplete(existingPublic.mapping),
- });
- if (patchedFirstAssignment) {
- assertUniqueEvaluators([
- patchedFirstAssignment,
- ...existing.assignments.slice(1),
- ]);
- }
-
- // Assignments the rule ends up running with, whether or not this call rewrites them.
- const effectiveAssignments: PreparedAssignment[] = replacementAssignments ?? [
- ...(patchedFirstAssignment
- ? [patchedFirstAssignment]
- : firstAssignment
- ? [toPreparedExistingAssignment(firstAssignment, nextTarget)]
- : []),
- ...existing.assignments
- .slice(1)
- .map((assignment) =>
- toPreparedExistingAssignment(assignment, nextTarget),
- ),
- ];
-
- if (ruleFields.status === JobConfigState.ACTIVE) {
- await assertAssignmentsCanRun({
- orgId: params.orgId,
- projectId: params.projectId,
- assignments: effectiveAssignments,
- targetObject: ruleFields.targetObject,
- scoreName: ruleFields.scoreName,
- filter: updatesFilterOrTarget
- ? ruleFields.filter
- : (existing.filter as FilterCondition[]),
- });
- }
-
- const updated = await updateRuleWithRuleService({
- projectId: params.projectId,
- evaluationRuleId: params.evaluationRuleId,
- input: params.input,
- fields: ruleFields,
- existing,
- replacementAssignments,
- patchedFirstAssignment,
- effectiveAssignmentCount: effectiveAssignments.length,
- updatesFilterOrTarget,
- });
- const evaluationRule = toApiReadableV2EvaluationRule(updated);
- if (params.auditScope) {
- await auditLog({
- action: "update",
- resourceType: JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE,
- resourceId: evaluationRule.id,
- projectId: params.projectId,
- orgId: params.auditScope.orgId,
- apiKeyId: params.auditScope.apiKeyId,
- before: existingPublic,
- after: evaluationRule,
- });
- }
- return evaluationRule;
-}
-
-function getPatchedFirstStoredMapping(params: {
- input: PatchUnstableEvaluationRuleBodyType;
- firstAssignment:
- | StoredPublicV2EvaluationRule["assignments"][number]
- | undefined;
- patchedFirstAssignment: PreparedAssignment;
-}) {
- return !("mapping" in params.input) || params.input.mapping === undefined
- ? // Evaluator swapped without an explicit mapping: keep the stored
- // override so an inherited mapping stays inherited.
- (params.firstAssignment?.variableMapping ?? null)
- : params.patchedFirstAssignment.storedMapping;
-}
-
-async function updateRuleWithRuleService(params: {
- projectId: string;
- evaluationRuleId: string;
- input: PatchUnstableEvaluationRuleBodyType;
- fields: ReturnType<typeof toEvaluationRuleFields>;
- existing: StoredPublicV2EvaluationRule;
- replacementAssignments: PreparedAssignment[] | null;
- patchedFirstAssignment: PreparedAssignment | null;
- effectiveAssignmentCount: number;
- updatesFilterOrTarget: boolean;
-}) {
- const firstAssignment = params.existing.assignments[0];
- const evaluatorMappings = params.replacementAssignments
- ? params.replacementAssignments.map((assignment) => ({
- evaluatorId: assignment.evaluatorId,
- variableMapping: assignment.storedMapping as
- | ObservationVariableMapping[]
- | null,
- }))
- : params.patchedFirstAssignment
- ? [
- {
- evaluatorId: params.patchedFirstAssignment.evaluatorId,
- variableMapping: getPatchedFirstStoredMapping({
- input: params.input,
- firstAssignment,
- patchedFirstAssignment: params.patchedFirstAssignment,
- }) as ObservationVariableMapping[] | null,
- },
- ...params.existing.assignments.slice(1).map((assignment) => ({
- evaluatorId: assignment.evaluatorId,
- variableMapping: assignment.variableMapping as
- | ObservationVariableMapping[]
- | null,
- })),
- ]
- : undefined;
-
- await runRuleServiceMutation(() =>
- ruleService.update({
- projectId: params.projectId,
- ruleId: params.evaluationRuleId,
- targetObject:
- params.updatesFilterOrTarget && "target" in params.input
- ? params.fields.targetObject
- : undefined,
- name: params.input.name,
- enabled:
- params.input.enabled === undefined
- ? undefined
- : params.effectiveAssignmentCount > 0 &&
- params.fields.status === JobConfigState.ACTIVE,
- sampling: params.input.sampling,
- filter: params.updatesFilterOrTarget
- ? (params.fields.filter as FilterState)
- : undefined,
- evaluatorMappings,
- }),
- );
- return findPublicV2EvaluationRuleOrThrow({
- projectId: params.projectId,
- evaluationRuleId: params.evaluationRuleId,
- });
-}
-
-/**
- * Existing stored assignment in the shape the preflight needs, resolving an
- * inherited (null) override against the evaluator version's default mapping.
- */
-function toPreparedExistingAssignment(
- assignment: StoredPublicV2EvaluationRule["assignments"][number],
- target: PublicEvaluationRuleTargetType,
-): PreparedAssignment {
- const template = toEvaluatorDefinition(assignment.evaluator);
- const inherited = assignment.variableMapping ?? template.variableMapping;
- return {
- evaluatorId: assignment.evaluatorId,
- template,
- storedMapping: assignment.variableMapping,
- effectiveMapping: toStoredAssignmentMapping({
- target,
- mapping:
- template.type === EvalTemplateType.CODE || inherited == null
- ? undefined
- : toApiMappings(inherited),
- evaluatorVariables: deriveEvaluatorVariables(template),
- evaluatorType: toPublicEvaluatorType(template.type),
- }),
- };
-}
-
-export async function deletePublicEvaluationRule(params: {
- projectId: string;
- evaluationRuleId: string;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
-}) {
- const existing = await findPublicWritableV2EvaluationRuleOrThrow(params);
- const existingPublic = toApiReadableV2EvaluationRule(existing);
- await runRuleServiceMutation(() =>
- ruleService.delete(params.projectId, params.evaluationRuleId),
- );
- if (params.auditScope) {
- await auditLog({
- action: "delete",
- resourceType: JOB_CONFIGURATION_AUDIT_LOG_RESOURCE_TYPE,
- resourceId: params.evaluationRuleId,
- projectId: params.projectId,
- orgId: params.auditScope.orgId,
- apiKeyId: params.auditScope.apiKeyId,
- before: existingPublic,
- });
- }
- return existing;
-}
diff --git a/web/src/features/evals/server/unstable-public-api/evaluator-service.ts b/web/src/features/evals/server/unstable-public-api/evaluator-service.ts
deleted file mode 100644
index 53b79e411a..0000000000
--- a/web/src/features/evals/server/unstable-public-api/evaluator-service.ts
+++ /dev/null
@@ -1,296 +0,0 @@
-import {
- EvalTemplateType,
- extractVariables,
- LangfuseConflictError,
-} from "@langfuse/shared";
-import { prisma } from "@langfuse/shared/src/db";
-import {
- invalidateProjectEvalConfigCaches,
- type ApiAccessScope,
-} from "@langfuse/shared/src/server";
-import { auditLog } from "@/src/features/audit-logs/auditLog";
-import { EVAL_TEMPLATE_AUDIT_LOG_RESOURCE_TYPE } from "@/src/features/evals/server/audit-log-resource-types";
-import {
- isCodeEvalEnabled,
- isCodeEvalSourceCodeLanguageSupported,
-} from "@/src/features/evals/server/isCodeEvalEnabled";
-import type { EvaluatorDefinition } from "@/src/features/evals/v2/server/evaluators/evaluatorTypes";
-import {
- EvaluatorService,
- reconcileEvaluatorPromptMessages,
-} from "@/src/features/evals/v2/server/evaluators/evaluatorService";
-import {
- EvaluatorConfigurationError,
- EvaluatorVersionConflictError,
-} from "@/src/features/evals/v2/server/evaluators/evaluatorErrors";
-import { RuleService } from "@/src/features/evals/v2/server/rules/ruleService";
-import { createStructuredPublicApiError } from "@/src/features/public-api";
-import {
- type PostUnstableEvaluatorBodyParsedType,
- PUBLIC_EVALUATOR_TYPE_CODE,
-} from "@/src/features/public-api/server";
-import {
- toApiEvaluator,
- toStoredOutputDefinition,
- toStoredVariableMappings,
-} from "./adapters";
-import type { StoredPublicEvaluatorTemplate } from "./types";
-
-function readServiceForPublicApi() {
- return new EvaluatorService(prisma, async () => undefined);
-}
-
-function serviceForPublicApi(params: {
- projectId: string;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
-}) {
- return new EvaluatorService(prisma, ({ action, evaluatorId }) =>
- auditPublicEvaluator(params, action, evaluatorId),
- );
-}
-
-function ruleServiceForPublicApi() {
- return new RuleService(prisma, async () => undefined);
-}
-
-function auditPublicEvaluator(
- params: {
- projectId: string;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
- },
- action: "create" | "update" | "delete",
- evaluatorId: string,
-) {
- if (!params.auditScope) return Promise.resolve();
- return auditLog({
- action,
- resourceType: EVAL_TEMPLATE_AUDIT_LOG_RESOURCE_TYPE,
- resourceId: evaluatorId,
- projectId: params.projectId,
- orgId: params.auditScope.orgId,
- apiKeyId: params.auditScope.apiKeyId,
- });
-}
-
-function assertCodeEvaluatorDefinitionCanRunForPublicApi(
- input: Extract<
- PostUnstableEvaluatorBodyParsedType,
- { type: typeof PUBLIC_EVALUATOR_TYPE_CODE }
- >,
-) {
- if (!isCodeEvalEnabled()) {
- throw createStructuredPublicApiError({
- httpCode: 403,
- code: "access_denied",
- message: "Code evals are not enabled",
- });
- }
-
- if (!isCodeEvalSourceCodeLanguageSupported(input.sourceCodeLanguage)) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_request",
- message:
- "This code evaluator language is not supported by the configured dispatcher.",
- details: {
- field: "sourceCodeLanguage",
- value: input.sourceCodeLanguage,
- },
- });
- }
-}
-
-function toDefinition(
- input: PostUnstableEvaluatorBodyParsedType,
-): EvaluatorDefinition {
- if (input.type === PUBLIC_EVALUATOR_TYPE_CODE) {
- return {
- type: EvalTemplateType.CODE,
- sourceCode: input.sourceCode,
- sourceCodeLanguage: input.sourceCodeLanguage,
- };
- }
-
- return {
- type: EvalTemplateType.LLM_AS_JUDGE,
- promptMessages: reconcileEvaluatorPromptMessages({ prompt: input.prompt }),
- provider: input.modelConfig?.provider ?? null,
- model: input.modelConfig?.model ?? null,
- modelParams: null,
- vars: extractVariables(input.prompt),
- variableMapping:
- input.mapping === undefined
- ? null
- : toStoredVariableMappings({
- mappings: input.mapping,
- variables: extractVariables(input.prompt),
- target: "observation",
- }),
- outputDefinition: toStoredOutputDefinition(input.outputDefinition),
- };
-}
-
-type StableEvaluator =
- | Awaited<ReturnType<EvaluatorService["get"]>>
- | Awaited<ReturnType<EvaluatorService["list"]>>["evaluators"][number];
-
-function toLatestTemplate(
- evaluator: StableEvaluator,
-): StoredPublicEvaluatorTemplate {
- const version = evaluator.versions[0];
- if (!version) {
- throw createStructuredPublicApiError({
- httpCode: 500,
- code: "internal_error",
- message: "Evaluator version is missing",
- });
- }
-
- return {
- id: evaluator.id,
- projectId: evaluator.projectId,
- name: evaluator.name,
- version: version.version,
- type: evaluator.type,
- prompt: null,
- promptMessages: version.promptMessages,
- partner: version.partner,
- provider: version.provider,
- model: version.model,
- modelParams: version.modelParams,
- vars: version.vars,
- outputDefinition: version.outputDefinition,
- sourceCode: version.sourceCode,
- sourceCodeLanguage: version.sourceCodeLanguage,
- variableMapping: version.variableMapping,
- createdAt: evaluator.createdAt,
- updatedAt: evaluator.updatedAt,
- };
-}
-
-function toPublicEvaluator(
- evaluator: StableEvaluator,
- evaluationRuleCount = 0,
-) {
- return toApiEvaluator({
- template: toLatestTemplate(evaluator),
- evaluationRuleCount,
- });
-}
-
-export async function listPublicEvaluators(params: {
- projectId: string;
- page: number;
- limit: number;
-}) {
- const service = readServiceForPublicApi();
- const ruleService = ruleServiceForPublicApi();
- const { evaluators, totalItems } = await service.list(params);
- const evaluatorIds = evaluators.map((evaluator) => evaluator.id);
- const assignmentCounts = await ruleService.countRulesForEvaluators(
- params.projectId,
- evaluatorIds,
- );
- return {
- data: evaluators.map((evaluator) =>
- toPublicEvaluator(evaluator, assignmentCounts[evaluator.id] ?? 0),
- ),
- meta: {
- page: params.page,
- limit: params.limit,
- totalItems,
- totalPages: Math.ceil(totalItems / params.limit),
- },
- };
-}
-
-export async function getPublicEvaluator(params: {
- projectId: string;
- evaluatorId: string;
-}) {
- const service = readServiceForPublicApi();
- const evaluator = await service.get(params.projectId, params.evaluatorId);
- const ruleCounts = await ruleServiceForPublicApi().countRulesForEvaluators(
- params.projectId,
- [evaluator.id],
- );
- return toPublicEvaluator(evaluator, ruleCounts[evaluator.id] ?? 0);
-}
-
-export async function createPublicEvaluator(params: {
- projectId: string;
- input: PostUnstableEvaluatorBodyParsedType;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
-}) {
- const { input } = params;
- const definition = toDefinition(input);
-
- if (input.type === PUBLIC_EVALUATOR_TYPE_CODE) {
- assertCodeEvaluatorDefinitionCanRunForPublicApi(input);
- }
-
- const service = serviceForPublicApi(params);
- try {
- const result = await service.upsertByName(
- {
- projectId: params.projectId,
- name: input.name,
- description: null,
- definition,
- },
- null,
- );
- const { evaluator } = result;
- const ruleCounts = await ruleServiceForPublicApi().countRulesForEvaluators(
- params.projectId,
- [evaluator.id],
- );
- return toPublicEvaluator(evaluator, ruleCounts[evaluator.id] ?? 0);
- } catch (error) {
- if (error instanceof EvaluatorConfigurationError) {
- throw createStructuredPublicApiError({
- httpCode: 422,
- code: "evaluator_preflight_failed",
- message: error.message,
- details: {
- evaluatorName: input.name,
- provider:
- input.type === PUBLIC_EVALUATOR_TYPE_CODE
- ? null
- : (input.modelConfig?.provider ?? null),
- model:
- input.type === PUBLIC_EVALUATOR_TYPE_CODE
- ? null
- : (input.modelConfig?.model ?? null),
- },
- });
- }
- if (error instanceof EvaluatorVersionConflictError) {
- throw createStructuredPublicApiError({
- httpCode: 409,
- code: "conflict",
- message: error.message,
- });
- }
- if (error instanceof LangfuseConflictError) {
- throw createStructuredPublicApiError({
- httpCode: 409,
- code: "name_conflict",
- message: error.message,
- details: { field: "name" },
- });
- }
- throw error;
- }
-}
-
-export async function deletePublicEvaluator(params: {
- projectId: string;
- evaluatorId: string;
- auditScope?: Pick<ApiAccessScope, "orgId" | "apiKeyId">;
-}) {
- const service = serviceForPublicApi(params);
- await service.get(params.projectId, params.evaluatorId);
- await service.delete(params.projectId, params.evaluatorId);
- await invalidateProjectEvalConfigCaches(params.projectId);
-}
diff --git a/web/src/features/evals/server/unstable-public-api/index.ts b/web/src/features/evals/server/unstable-public-api/index.ts
deleted file mode 100644
index efec33996f..0000000000
--- a/web/src/features/evals/server/unstable-public-api/index.ts
+++ /dev/null
@@ -1,13 +0,0 @@
-export {
- createPublicEvaluationRule,
- deletePublicEvaluationRule,
- getPublicEvaluationRule,
- listPublicEvaluationRules,
- updatePublicEvaluationRule,
-} from "./evaluation-rule-service";
-export {
- createPublicEvaluator,
- deletePublicEvaluator,
- getPublicEvaluator,
- listPublicEvaluators,
-} from "./evaluator-service";
diff --git a/web/src/features/evals/server/unstable-public-api/queries.ts b/web/src/features/evals/server/unstable-public-api/queries.ts
deleted file mode 100644
index 8a291ad97f..0000000000
--- a/web/src/features/evals/server/unstable-public-api/queries.ts
+++ /dev/null
@@ -1,125 +0,0 @@
-import { EvalTargetObject, LangfuseNotFoundError } from "@langfuse/shared";
-import { prisma } from "@langfuse/shared/src/db";
-import type {
- EvaluationRuleEvaluatorFamilyReference,
- PrismaClientLike,
- StoredPublicV2EvaluationRule,
-} from "./types";
-import { toStoredEvaluatorType } from "./adapters";
-
-function getPrismaClient(client?: PrismaClientLike) {
- return client ?? prisma;
-}
-
-const publicV2RuleInclude = (projectId: string) => ({
- assignments: {
- where: { projectId, evaluator: { projectId } },
- orderBy: [{ createdAt: "asc" as const }, { id: "asc" as const }],
- include: {
- evaluator: {
- include: {
- versions: { orderBy: { version: "desc" as const }, take: 1 },
- },
- },
- },
- },
-});
-
-export async function findPublicV2EvaluationRule(params: {
- client?: PrismaClientLike;
- projectId: string;
- evaluationRuleId: string;
-}) {
- const client = getPrismaClient(params.client);
- return client.evaluationRule.findFirst({
- where: { id: params.evaluationRuleId, projectId: params.projectId },
- include: publicV2RuleInclude(params.projectId),
- }) as Promise<StoredPublicV2EvaluationRule | null>;
-}
-
-export async function findPublicV2EvaluatorInFamily(params: {
- client?: PrismaClientLike;
- projectId: string;
- evaluator: EvaluationRuleEvaluatorFamilyReference;
-}) {
- const client = getPrismaClient(params.client);
- const evaluator = await client.evaluator.findFirst({
- where: {
- projectId: params.projectId,
- name: params.evaluator.name,
- type: toStoredEvaluatorType(params.evaluator.type),
- },
- include: { versions: { orderBy: { version: "desc" }, take: 1 } },
- });
- return evaluator?.versions.length ? evaluator : null;
-}
-
-export async function findPublicV2EvaluatorById(params: {
- client?: PrismaClientLike;
- projectId: string;
- evaluatorId: string;
-}) {
- const client = getPrismaClient(params.client);
- const evaluator = await client.evaluator.findFirst({
- where: { id: params.evaluatorId, projectId: params.projectId },
- include: { versions: { orderBy: { version: "desc" }, take: 1 } },
- });
- return evaluator?.versions.length ? evaluator : null;
-}
-
-export async function findPublicV2EvaluatorByIdOrThrow(
- params: Parameters<typeof findPublicV2EvaluatorById>[0],
-) {
- const evaluator = await findPublicV2EvaluatorById(params);
- if (!evaluator) {
- throw new LangfuseNotFoundError(
- "Latest evaluator version not found within authorized project",
- );
- }
- return evaluator;
-}
-
-export async function findPublicV2EvaluatorInFamilyOrThrow(
- params: Parameters<typeof findPublicV2EvaluatorInFamily>[0],
-) {
- const evaluator = await findPublicV2EvaluatorInFamily(params);
- if (!evaluator) {
- throw new LangfuseNotFoundError(
- "Latest evaluator version not found within authorized project",
- );
- }
- return evaluator;
-}
-
-export async function listPublicEvaluationRulePage(params: {
- projectId: string;
- page: number;
- limit: number;
-}) {
- const where = {
- projectId: params.projectId,
- targetObject: {
- in: [
- EvalTargetObject.EVENT,
- EvalTargetObject.EXPERIMENT,
- EvalTargetObject.TRACE,
- EvalTargetObject.DATASET,
- ],
- },
- };
- const [records, totalItems] = await Promise.all([
- prisma.evaluationRule.findMany({
- where,
- include: publicV2RuleInclude(params.projectId),
- orderBy: [{ createdAt: "desc" }, { id: "desc" }],
- take: params.limit,
- skip: (params.page - 1) * params.limit,
- }),
- prisma.evaluationRule.count({ where }),
- ]);
-
- return {
- records: records as StoredPublicV2EvaluationRule[],
- totalItems,
- };
-}
diff --git a/web/src/features/evals/server/unstable-public-api/types.ts b/web/src/features/evals/server/unstable-public-api/types.ts
deleted file mode 100644
index 067c6d34e2..0000000000
--- a/web/src/features/evals/server/unstable-public-api/types.ts
+++ /dev/null
@@ -1,141 +0,0 @@
-import type {
- EvaluationRule,
- EvaluationRuleEvaluatorAssignment,
- Evaluator,
- EvaluatorVersion,
- Prisma as PrismaNamespace,
- prisma,
-} from "@langfuse/shared/src/db";
-import type {
- PublicEvaluationRuleEvaluatorReferenceType,
- PublicEvaluationRuleEvaluatorType,
- PublicEvaluationRuleFilterType,
- PublicEvaluationRuleReadFilterType,
- PromptVariableMappingReadType,
- PublicEvaluationRuleStatusType,
- PublicEvaluationRuleTargetType,
- LegacyPromptVariableMappingType,
- PublicEvaluationRuleLegacyTargetType,
- PublicEvaluatorModelConfigType,
- PublicEvaluatorOutputDefinitionType,
- PublicCodeEvaluatorSourceCodeLanguageType,
- PUBLIC_EVALUATOR_TYPE_CODE,
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
-} from "@/src/features/public-api/server";
-import type {
- CODE_EVAL_TEMPLATE_VARIABLES,
- FilterCondition,
- JobTimeScope,
-} from "@langfuse/shared";
-
-export type PrismaClientLike =
- | typeof prisma
- | PrismaNamespace.TransactionClient;
-
-type ApiEvaluatorRecordBase = {
- id: string;
- name: string;
- version: number;
- variables: string[];
- // Read shape: an evaluator default can be incomplete or name experiment-only sources.
- mapping: PromptVariableMappingReadType[] | null;
- evaluationRuleCount: number;
- createdAt: Date;
- updatedAt: Date;
-};
-
-export type StoredPublicV2EvaluationRule = EvaluationRule & {
- assignments: Array<
- EvaluationRuleEvaluatorAssignment & {
- evaluator: Evaluator & { versions: EvaluatorVersion[] };
- }
- >;
-};
-
-type ApiLlmAsJudgeEvaluatorRecord = ApiEvaluatorRecordBase & {
- type: typeof PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE;
- prompt: string;
- outputDefinition: PublicEvaluatorOutputDefinitionType;
- modelConfig: PublicEvaluatorModelConfigType | null;
-};
-
-type ApiCodeEvaluatorRecord = ApiEvaluatorRecordBase & {
- type: typeof PUBLIC_EVALUATOR_TYPE_CODE;
- variables: Array<(typeof CODE_EVAL_TEMPLATE_VARIABLES)[number]>;
- sourceCode: string;
- sourceCodeLanguage: PublicCodeEvaluatorSourceCodeLanguageType;
-};
-
-export type ApiEvaluatorRecord =
- | ApiLlmAsJudgeEvaluatorRecord
- | ApiCodeEvaluatorRecord;
-
-type ApiEvaluationRuleRecordBase = {
- id: string;
- name: string;
- evaluator: PublicEvaluationRuleEvaluatorType | null;
- enabled: boolean;
- status: PublicEvaluationRuleStatusType;
- pausedReason: string | null;
- pausedMessage: string | null;
- sampling: number;
- createdAt: Date;
- updatedAt: Date;
-};
-
-export type ApiWritableEvaluationRuleRecord = ApiEvaluationRuleRecordBase & {
- evaluators: Array<{
- evaluator: PublicEvaluationRuleEvaluatorType;
- mapping: PromptVariableMappingReadType[] | null;
- }>;
- target: PublicEvaluationRuleTargetType;
- filter: PublicEvaluationRuleFilterType[];
- mapping: PromptVariableMappingReadType[];
-};
-
-type ApiLegacyEvaluationRuleRecord = ApiEvaluationRuleRecordBase & {
- evaluators: Array<{
- evaluator: PublicEvaluationRuleEvaluatorType;
- mapping: LegacyPromptVariableMappingType[] | null;
- }>;
- target: PublicEvaluationRuleLegacyTargetType;
- delay: number;
- timeScope: JobTimeScope[];
- filter: FilterCondition[];
- mapping: LegacyPromptVariableMappingType[];
-};
-
-type ApiReadableV2EvaluationRuleRecord = Omit<
- ApiWritableEvaluationRuleRecord,
- "filter"
-> & {
- filter: PublicEvaluationRuleReadFilterType[];
-};
-
-export type ApiEvaluationRuleRecord =
- | ApiReadableV2EvaluationRuleRecord
- | ApiLegacyEvaluationRuleRecord;
-
-export type EvaluationRuleEvaluatorFamilyReference =
- PublicEvaluationRuleEvaluatorReferenceType;
-
-export type StoredPublicEvaluatorTemplate = Pick<
- Evaluator,
- "id" | "projectId" | "name" | "type" | "createdAt" | "updatedAt"
-> &
- Pick<
- EvaluatorVersion,
- | "version"
- | "prompt"
- | "partner"
- | "provider"
- | "model"
- | "modelParams"
- | "vars"
- | "outputDefinition"
- | "sourceCode"
- | "sourceCodeLanguage"
- | "variableMapping"
- > & {
- promptMessages?: EvaluatorVersion["promptMessages"];
- };
diff --git a/web/src/features/evals/server/unstable-public-api/validation.ts b/web/src/features/evals/server/unstable-public-api/validation.ts
deleted file mode 100644
index 3d10ab35cc..0000000000
--- a/web/src/features/evals/server/unstable-public-api/validation.ts
+++ /dev/null
@@ -1,393 +0,0 @@
-import {
- experimentEvalFilterColumns,
- observationEvalFilterColumns,
-} from "@langfuse/shared";
-import { prisma } from "@langfuse/shared/src/db";
-import { JSONPath } from "jsonpath-plus";
-import { createStructuredPublicApiError } from "@/src/features/public-api";
-import {
- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS,
- ExperimentPromptVariableMappingSource,
- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,
- ObservationPromptVariableMappingSource,
- type PublicEvaluationRuleFilterType,
- type PromptVariableMappingInputType,
- type PublicEvaluationRuleTargetType,
-} from "@/src/features/public-api/server";
-import { getEvaluatorDefinitionPreflightError } from "@/src/features/evals/server/evaluator-preflight";
-
-// Derived from the same column lists as the request schemas, so this runtime
-// check cannot drift from the documented contract.
-const toStaticFilterOptions = (columns: typeof observationEvalFilterColumns) =>
- new Map(
- columns.flatMap((column) => {
- if (!("options" in column) || !Array.isArray(column.options)) {
- return [];
- }
-
- if (column.options.length === 0) {
- return [];
- }
-
- return [
- [
- column.id === "experimentDatasetId" ? "datasetId" : column.id,
- new Set(
- column.options.flatMap((option) =>
- "value" in option ? [String(option.value)] : option.values,
- ),
- ),
- ] as const,
- ];
- }),
- );
-
-const STATIC_FILTER_OPTIONS_BY_TARGET = {
- observation: toStaticFilterOptions(observationEvalFilterColumns),
- experiment: toStaticFilterOptions([
- ...observationEvalFilterColumns,
- ...experimentEvalFilterColumns,
- ]),
-} as const satisfies Record<
- PublicEvaluationRuleTargetType,
- Map<string, Set<string>>
->;
-
-const SUPPORTED_FILTER_COLUMNS_BY_TARGET = {
- observation: new Set(
- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS.map((column) => column.id),
- ),
- experiment: new Set(
- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS.map((column) => column.id),
- ),
-} as const satisfies Record<PublicEvaluationRuleTargetType, Set<string>>;
-
-// Derived from the per-target request schemas so this runtime check (which
-// also guards paths that parse the target-agnostic mapping union) can never
-// drift from the documented contract.
-const SUPPORTED_MAPPING_SOURCES_BY_TARGET = {
- observation: new Set<string>(ObservationPromptVariableMappingSource.options),
- experiment: new Set<string>(ExperimentPromptVariableMappingSource.options),
-} as const satisfies Record<PublicEvaluationRuleTargetType, Set<string>>;
-
-export function validateEvaluationRuleFilters(params: {
- target: PublicEvaluationRuleTargetType;
- filters: PublicEvaluationRuleFilterType[];
-}) {
- const knownOptionValues = STATIC_FILTER_OPTIONS_BY_TARGET[params.target];
- const supportedColumns = SUPPORTED_FILTER_COLUMNS_BY_TARGET[params.target];
-
- for (const [filterIndex, filter] of params.filters.entries()) {
- if (!supportedColumns.has(filter.column)) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_filter_value",
- message: `Filter column "${filter.column}" is not supported for target "${params.target}"`,
- details: {
- field: `filter[${filterIndex}].column`,
- column: filter.column,
- allowedValues: Array.from(supportedColumns),
- },
- });
- }
-
- const allowedValues = knownOptionValues.get(filter.column);
-
- if (
- !allowedValues ||
- !("value" in filter) ||
- !Array.isArray(filter.value)
- ) {
- continue;
- }
-
- const invalidValues = filter.value.filter(
- (value) => !allowedValues.has(value),
- );
-
- if (invalidValues.length > 0) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_filter_value",
- message: `Filter column "${filter.column}" contains unsupported value(s): ${invalidValues.join(", ")}`,
- details: {
- field: `filter[${filterIndex}].value`,
- column: filter.column,
- invalidValues,
- allowedValues: Array.from(allowedValues),
- },
- });
- }
- }
-}
-
-export async function assertEvaluationRuleFilterValuesExistForProject(params: {
- projectId: string;
- target: PublicEvaluationRuleTargetType;
- filters: PublicEvaluationRuleFilterType[];
-}) {
- if (params.target !== "experiment") {
- return;
- }
-
- const datasetFilters = params.filters.flatMap((filter, filterIndex) => {
- if (
- filter.column !== "datasetId" ||
- !("value" in filter) ||
- !Array.isArray(filter.value)
- ) {
- return [];
- }
-
- return [{ filterIndex, values: filter.value }] as const;
- });
-
- if (datasetFilters.length === 0) {
- return;
- }
-
- const requestedDatasetIds = Array.from(
- new Set(datasetFilters.flatMap((filter) => filter.values)),
- );
- const existingDatasets = await prisma.dataset.findMany({
- where: {
- projectId: params.projectId,
- id: {
- in: requestedDatasetIds,
- },
- },
- select: {
- id: true,
- },
- });
- const existingDatasetIds = new Set(
- existingDatasets.map((dataset) => dataset.id),
- );
-
- for (const filter of datasetFilters) {
- const invalidValues = filter.values.filter(
- (value) => !existingDatasetIds.has(value),
- );
-
- if (invalidValues.length > 0) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_filter_value",
- message: `Filter column "datasetId" contains dataset id(s) that do not exist in this project: ${invalidValues.join(", ")}`,
- details: {
- field: `filter[${filter.filterIndex}].value`,
- column: "datasetId",
- invalidValues,
- },
- });
- }
- }
-}
-
-function validateJsonPath(params: {
- jsonPath: string;
- variable: string;
- mappingIndex: number;
-}) {
- const { jsonPath, variable, mappingIndex } = params;
-
- if (!jsonPath.startsWith("$")) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_json_path",
- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". JSONPath expressions must start with "$".`,
- details: {
- field: `mapping[${mappingIndex}].jsonPath`,
- variable,
- value: jsonPath,
- },
- });
- }
-
- let openQuote: "'" | '"' | null = null;
- let isEscaped = false;
- const delimiterStack: Array<"[" | "("> = [];
-
- for (const character of jsonPath) {
- if (openQuote) {
- if (isEscaped) {
- isEscaped = false;
- continue;
- }
-
- if (character === "\\") {
- isEscaped = true;
- continue;
- }
-
- if (character === openQuote) {
- openQuote = null;
- }
-
- continue;
- }
-
- if (character === "'" || character === '"') {
- openQuote = character;
- continue;
- }
-
- if (character === "[" || character === "(") {
- delimiterStack.push(character);
- continue;
- }
-
- if (character === "]" || character === ")") {
- const expectedDelimiter = character === "]" ? "[" : "(";
-
- if (delimiterStack.pop() !== expectedDelimiter) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_json_path",
- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". JSONPath expressions must use balanced brackets and parentheses.`,
- details: {
- field: `mapping[${mappingIndex}].jsonPath`,
- variable,
- value: jsonPath,
- },
- });
- }
- }
- }
-
- if (openQuote || delimiterStack.length > 0) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_json_path",
- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". JSONPath expressions must use balanced quotes, brackets, and parentheses.`,
- details: {
- field: `mapping[${mappingIndex}].jsonPath`,
- variable,
- value: jsonPath,
- },
- });
- }
-
- try {
- JSONPath({
- path: jsonPath,
- json: {},
- eval: false,
- });
- } catch (error) {
- const message = error instanceof Error ? error.message : "Unknown error";
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_json_path",
- message: `Mapping for variable "${variable}" has an invalid jsonPath "${jsonPath}". ${message}`,
- details: {
- field: `mapping[${mappingIndex}].jsonPath`,
- variable,
- value: jsonPath,
- },
- });
- }
-}
-
-export function validateEvaluatorVariableMappings(params: {
- mappings: PromptVariableMappingInputType[];
- variables: string[];
- target: PublicEvaluationRuleTargetType;
-}) {
- const variableSet = new Set(params.variables);
- const mappedVariables = new Set<string>();
- const allowedSources = SUPPORTED_MAPPING_SOURCES_BY_TARGET[params.target];
-
- for (const [mappingIndex, mapping] of params.mappings.entries()) {
- if (!allowedSources.has(mapping.source)) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_variable_mapping",
- message: `Mapping source "${mapping.source}" is not supported for target "${params.target}"`,
- details: {
- field: `mapping[${mappingIndex}].source`,
- variable: mapping.variable,
- allowedValues: Array.from(allowedSources),
- },
- });
- }
-
- if (!variableSet.has(mapping.variable)) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "invalid_variable_mapping",
- message: `Mapping variable "${mapping.variable}" is not present in the evaluator prompt`,
- details: {
- field: `mapping[${mappingIndex}].variable`,
- variable: mapping.variable,
- },
- });
- }
-
- if (mappedVariables.has(mapping.variable)) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "duplicate_variable_mapping",
- message: `Mapping variable "${mapping.variable}" can only be mapped once`,
- details: {
- field: "mapping",
- variable: mapping.variable,
- },
- });
- }
-
- mappedVariables.add(mapping.variable);
- }
-
- const missingVariables = params.variables.filter(
- (variable) => !mappedVariables.has(variable),
- );
-
- if (missingVariables.length > 0) {
- throw createStructuredPublicApiError({
- httpCode: 400,
- code: "missing_variable_mapping",
- message: `Missing mappings for evaluator variables: ${missingVariables.join(", ")}`,
- details: {
- field: "mapping",
- variables: missingVariables,
- },
- });
- }
-
- for (const [mappingIndex, mapping] of params.mappings.entries()) {
- if (mapping.jsonPath) {
- validateJsonPath({
- jsonPath: mapping.jsonPath,
- variable: mapping.variable,
- mappingIndex,
- });
- }
- }
-}
-
-export async function assertEvaluatorDefinitionCanRunForPublicApi(params: {
- projectId: string;
- template: {
- name: string;
- provider?: string | null;
- model?: string | null;
- modelParams?: unknown;
- outputDefinition: unknown;
- };
-}) {
- const error = await getEvaluatorDefinitionPreflightError(params);
-
- if (error) {
- throw createStructuredPublicApiError({
- httpCode: 422,
- code: "evaluator_preflight_failed",
- message: error,
- details: {
- evaluatorName: params.template.name,
- provider: params.template.provider ?? null,
- model: params.template.model ?? null,
- },
- });
- }
-}
diff --git a/web/src/features/mcp/server/evals/rule-schema.ts b/web/src/features/mcp/server/evals/rule-schema.ts
index 4b4ef80005..6a3cd074ce 100644
--- a/web/src/features/mcp/server/evals/rule-schema.ts
+++ b/web/src/features/mcp/server/evals/rule-schema.ts
@@ -12,7 +12,7 @@ import {
ObservationPromptVariableMappingInput,
PromptVariableMappingRead,
PublicEvaluatorType,
-} from "@/src/features/public-api/server";
+} from "@/src/features/public-api/types/evaluation/publicEvalsContract";
import { McpAdvancedFilterBaseSchema } from "../../core/filter-schema";
const RuleFilterBaseSchema = McpAdvancedFilterBaseSchema.describe(
diff --git a/web/src/features/mcp/server/evals/rule-service.ts b/web/src/features/mcp/server/evals/rule-service.ts
index aec4809a7b..ef26147a58 100644
--- a/web/src/features/mcp/server/evals/rule-service.ts
+++ b/web/src/features/mcp/server/evals/rule-service.ts
@@ -4,7 +4,7 @@ import {
toApiReadMappings,
toPublicEvaluatorType,
toStoredMappingList,
-} from "@/src/features/evals/server/unstable-public-api/adapters";
+} from "@/src/features/public-api/server/evaluation/evaluationAdapters";
import { RuleService } from "@/src/features/evals/v2/server/rules/ruleService";
import { prisma } from "@langfuse/shared/src/db";
import type { z } from "zod";
diff --git a/web/src/features/public-api/server/evaluation/evaluationAdapters.ts b/web/src/features/public-api/server/evaluation/evaluationAdapters.ts
index 013d43a721..94dd218180 100644
--- a/web/src/features/public-api/server/evaluation/evaluationAdapters.ts
+++ b/web/src/features/public-api/server/evaluation/evaluationAdapters.ts
@@ -64,7 +64,15 @@ const INTERNAL_MAPPING_COLUMN_TO_PUBLIC_SOURCE: Record<
experiment_item_metadata: "experiment_item_metadata",
};
-function toStoredMappingList(mappings: PromptVariableMappingInputType[]) {
+export function toPublicEvaluatorType(type: EvalTemplateType) {
+ return type === EvalTemplateType.CODE
+ ? PUBLIC_EVALUATOR_TYPE_CODE
+ : PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE;
+}
+
+export function toStoredMappingList(
+ mappings: PromptVariableMappingInputType[],
+) {
return observationVariableMappingList.parse(
mappings.map((mapping) => ({
templateVariable: mapping.variable,
@@ -75,7 +83,9 @@ function toStoredMappingList(mappings: PromptVariableMappingInputType[]) {
);
}
-function toApiReadMappings(mappings: unknown): PromptVariableMappingReadType[] {
+export function toApiReadMappings(
+ mappings: unknown,
+): PromptVariableMappingReadType[] {
const parsed = observationVariableMappingList.safeParse(mappings);
if (!parsed.success) {
logger.error("Failed to parse public evaluation rule mappings", {
diff --git a/web/src/features/public-api/server/index.ts b/web/src/features/public-api/server/index.ts
index bfd55769fb..170d5443a8 100644
--- a/web/src/features/public-api/server/index.ts
+++ b/web/src/features/public-api/server/index.ts
@@ -162,39 +162,3 @@ export {
PostDashboardPlacementBody,
PostUnstableDashboardBody,
} from "../types/unstable-dashboards";
-export type {
- PatchUnstableEvaluationRuleBodyType,
- PostUnstableEvaluationRuleBodyType,
-} from "../types/unstable-evaluation-rules";
-export type { PostUnstableEvaluatorBodyParsedType } from "../types/unstable-evaluators";
-export type {
- LegacyPromptVariableMappingType,
- PromptVariableMappingInputType,
- PromptVariableMappingReadType,
- PublicCodeEvaluatorSourceCodeLanguageType,
- PublicEvaluationRuleEvaluatorReferenceType,
- PublicEvaluationRuleEvaluatorType,
- PublicEvaluationRuleFilterType,
- PublicEvaluationRuleReadFilterType,
- PublicEvaluationRuleLegacyTargetType,
- PublicEvaluationRuleReadTargetType,
- PublicEvaluationRuleStatusType,
- PublicEvaluationRuleTargetType,
- PublicEvaluatorModelConfigType,
- PublicEvaluatorOutputDefinitionType,
- PublicEvaluatorTypeType,
-} from "../types/unstable-public-evals-contract";
-export {
- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS,
- ExperimentPromptVariableMappingSource,
- LegacyPromptVariableMapping,
- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,
- ObservationPromptVariableMappingInput,
- ObservationPromptVariableMappingSource,
- PromptVariableMappingRead,
- PUBLIC_EVALUATOR_TYPE_CODE,
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- PublicEvaluationRuleFilter,
- PublicEvaluationRuleReadFilter,
- PublicEvaluatorType,
-} from "../types/unstable-public-evals-contract";
diff --git a/web/src/features/public-api/types/evaluation/publicEvalsContract.ts b/web/src/features/public-api/types/evaluation/publicEvalsContract.ts
index 7cdd3b68c1..37c78421a3 100644
--- a/web/src/features/public-api/types/evaluation/publicEvalsContract.ts
+++ b/web/src/features/public-api/types/evaluation/publicEvalsContract.ts
@@ -20,7 +20,7 @@ const PUBLIC_EVALUATOR_TYPES = ["llm_as_judge", "code"] as const;
export const [PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE, PUBLIC_EVALUATOR_TYPE_CODE] =
PUBLIC_EVALUATOR_TYPES;
-const PublicEvaluatorType = z.enum(PUBLIC_EVALUATOR_TYPES);
+export const PublicEvaluatorType = z.enum(PUBLIC_EVALUATOR_TYPES);
export const PublicCodeEvaluatorSourceCodeLanguage = z.enum([
"PYTHON",
"TYPESCRIPT",
@@ -154,7 +154,7 @@ function createMappingSchema<
});
}
-const ObservationPromptVariableMappingInput = createMappingSchema(
+export const ObservationPromptVariableMappingInput = createMappingSchema(
ObservationPromptVariableMappingSource,
);
diff --git a/web/src/features/public-api/types/unstable-evaluation-rules.ts b/web/src/features/public-api/types/unstable-evaluation-rules.ts
deleted file mode 100644
index 56e18cb31c..0000000000
--- a/web/src/features/public-api/types/unstable-evaluation-rules.ts
+++ /dev/null
@@ -1,254 +0,0 @@
-import { JobTimeScopeZod, singleFilter } from "@langfuse/shared";
-import { z } from "zod";
-import {
- ExperimentEvaluationRuleFilter,
- ExperimentPromptVariableMappingInput,
- LegacyPromptVariableMapping,
- ObservationEvaluationRuleFilter,
- ObservationPromptVariableMappingInput,
- PUBLIC_EVALUATOR_TYPE_CODE,
- PublicEvaluationRuleFilter,
- PublicEvaluationRuleReadFilter,
- PublicEvaluationRuleEvaluator,
- PublicEvaluationRuleEvaluatorReference,
- PublicEvaluationRuleEvaluatorReferencePatch,
- PromptVariableMappingRead,
- PublicEvaluationRuleLegacyTarget,
- PublicEvaluationRuleStatus,
- PublicEvaluationRuleTarget,
- UnstablePublicApiPaginationQuery,
- UnstablePublicApiPaginationResponse,
-} from "@/src/features/public-api/types/unstable-public-evals-contract";
-
-const APIEvaluationRuleBase = {
- id: z.string(),
- name: z.string(),
- // Null when the rule has no evaluator assignments. Prefer `evaluators`.
- evaluator: PublicEvaluationRuleEvaluator.nullable(),
- enabled: z.boolean(),
- status: PublicEvaluationRuleStatus,
- pausedReason: z.string().nullable(),
- pausedMessage: z.string().nullable(),
- sampling: z.number().gt(0).lte(1),
- createdAt: z.coerce.date(),
- updatedAt: z.coerce.date(),
-};
-
-const APIEvaluationRule = z
- .object({
- ...APIEvaluationRuleBase,
- evaluators: z.array(
- z.object({
- evaluator: PublicEvaluationRuleEvaluator,
- mapping: z.array(PromptVariableMappingRead).nullable(),
- }),
- ),
- target: PublicEvaluationRuleTarget,
- filter: z.array(PublicEvaluationRuleFilter),
- mapping: z.array(PromptVariableMappingRead),
- })
- .strict();
-
-const APIReadableV2EvaluationRule = APIEvaluationRule.extend({
- filter: z.array(PublicEvaluationRuleReadFilter),
-});
-
-const APILegacyEvaluationRule = z
- .object({
- ...APIEvaluationRuleBase,
- evaluators: z.array(
- z.object({
- evaluator: PublicEvaluationRuleEvaluator,
- mapping: z.array(LegacyPromptVariableMapping).nullable(),
- }),
- ),
- target: PublicEvaluationRuleLegacyTarget,
- delay: z.number().int().nonnegative(),
- timeScope: z.array(JobTimeScopeZod),
- filter: z.array(singleFilter),
- mapping: z.array(LegacyPromptVariableMapping),
- })
- .strict();
-
-const APIReadableEvaluationRule = z.union([
- APIReadableV2EvaluationRule,
- APILegacyEvaluationRule,
-]);
-
-export const GetUnstableEvaluationRulesQuery = UnstablePublicApiPaginationQuery;
-
-export const GetUnstableEvaluationRulesResponse = z
- .object({
- data: z.array(APIReadableEvaluationRule),
- meta: UnstablePublicApiPaginationResponse,
- })
- .strict();
-
-export const GetUnstableEvaluationRuleQuery = z.object({
- evaluationRuleId: z.string(),
-});
-
-/** @alias */
-export const GetUnstableEvaluationRuleResponse = APIReadableEvaluationRule;
-
-const EvaluationRuleCreateBase = {
- name: z.string().min(1),
- evaluator: PublicEvaluationRuleEvaluatorReference.optional(),
- enabled: z.boolean(),
- sampling: z.number().gt(0).lte(1).default(1),
-};
-
-function evaluatorAssignment<T extends z.ZodType>(mapping: T) {
- return z.object({
- evaluator: PublicEvaluationRuleEvaluatorReference,
- mapping: z.array(mapping).optional(),
- });
-}
-
-const PostUnstableObservationEvaluationRuleBody = z.object({
- ...EvaluationRuleCreateBase,
- evaluators: z
- .array(evaluatorAssignment(ObservationPromptVariableMappingInput))
- .min(1)
- .max(100)
- .optional(),
- target: z.literal("observation"),
- filter: z.array(ObservationEvaluationRuleFilter).default([]),
- mapping: z.array(ObservationPromptVariableMappingInput).optional(),
-});
-
-const PostUnstableExperimentEvaluationRuleBody = z.object({
- ...EvaluationRuleCreateBase,
- evaluators: z
- .array(evaluatorAssignment(ExperimentPromptVariableMappingInput))
- .min(1)
- .max(100)
- .optional(),
- target: z.literal("experiment"),
- filter: z.array(ExperimentEvaluationRuleFilter).default([]),
- mapping: z.array(ExperimentPromptVariableMappingInput).optional(),
-});
-
-// `code` evaluators use a fixed runtime mapping managed by Langfuse. LLM
-// mappings may be omitted to inherit the evaluator version's default mapping.
-export const PostUnstableEvaluationRuleBody = z
- .discriminatedUnion("target", [
- PostUnstableObservationEvaluationRuleBody,
- PostUnstableExperimentEvaluationRuleBody,
- ])
- .refine((data) => Boolean(data.evaluator) !== Boolean(data.evaluators), {
- path: ["evaluators"],
- message: "Provide exactly one of evaluator or evaluators.",
- })
- .refine((data) => !data.evaluators || data.mapping === undefined, {
- path: ["mapping"],
- message:
- "`mapping` is the deprecated single-evaluator alias and cannot be combined with `evaluators`. Set the mapping on the matching entry in `evaluators` instead.",
- })
- .refine(
- (data) =>
- (!data.evaluator ||
- data.evaluator.type !== PUBLIC_EVALUATOR_TYPE_CODE ||
- data.mapping === undefined) &&
- (data.evaluators ?? []).every(
- (assignment) =>
- assignment.evaluator.type !== PUBLIC_EVALUATOR_TYPE_CODE ||
- assignment.mapping === undefined,
- ),
- {
- path: ["mapping"],
- message:
- "Code evaluator mappings are managed by Langfuse and cannot be provided in the request body.",
- },
- );
-export type PostUnstableEvaluationRuleBodyType = z.infer<
- typeof PostUnstableEvaluationRuleBody
->;
-
-/** @alias */
-export const PostUnstableEvaluationRuleResponse = APIEvaluationRule;
-
-/** @alias */
-export const PatchUnstableEvaluationRuleQuery = GetUnstableEvaluationRuleQuery;
-
-// Exported for reuse (see EvaluationRuleCreateBase) — the create fields, all
-// made optional for PATCH.
-const EvaluationRulePatchBase = {
- name: z.string().min(1).optional(),
- evaluator: PublicEvaluationRuleEvaluatorReferencePatch.optional(),
- enabled: z.boolean().optional(),
- sampling: z.number().gt(0).lte(1).optional(),
-};
-
-// Replacing `evaluators` is a full set replacement: entries not listed are
-// detached. Mutually exclusive with the deprecated `evaluator`/`mapping` pair,
-// which only ever addressed the first assignment.
-function patchEvaluators<T extends z.ZodType>(mapping: T) {
- return z.array(evaluatorAssignment(mapping)).min(1).max(100).optional();
-}
-
-function assertExclusiveEvaluatorFields<
- T extends { evaluator?: unknown; mapping?: unknown; evaluators?: unknown },
->(data: T) {
- return (
- !data.evaluators ||
- (data.evaluator === undefined && data.mapping === undefined)
- );
-}
-
-const EXCLUSIVE_EVALUATOR_FIELDS_MESSAGE =
- "`evaluators` replaces the whole assignment set and cannot be combined with the deprecated `evaluator` or `mapping` fields.";
-
-const UntargetedEvaluationRulePatch = z.object({
- ...EvaluationRulePatchBase,
- evaluators: patchEvaluators(ObservationPromptVariableMappingInput),
- target: z.undefined().optional(),
- filter: z.undefined().optional(),
- mapping: z.undefined().optional(),
-});
-
-const ObservationEvaluationRulePatch = z.object({
- ...EvaluationRulePatchBase,
- evaluators: patchEvaluators(ObservationPromptVariableMappingInput),
- target: z.literal("observation"),
- filter: z.array(ObservationEvaluationRuleFilter).optional(),
- mapping: z.array(ObservationPromptVariableMappingInput).optional(),
-});
-
-const ExperimentEvaluationRulePatch = z.object({
- ...EvaluationRulePatchBase,
- evaluators: patchEvaluators(ExperimentPromptVariableMappingInput),
- target: z.literal("experiment"),
- filter: z.array(ExperimentEvaluationRuleFilter).optional(),
- mapping: z.array(ExperimentPromptVariableMappingInput).optional(),
-});
-
-export const PatchUnstableEvaluationRuleBody = z
- .union([
- ObservationEvaluationRulePatch,
- ExperimentEvaluationRulePatch,
- UntargetedEvaluationRulePatch,
- ])
- .refine((data) => Object.keys(data).length > 0, {
- message:
- "Request body cannot be empty. At least one field must be provided for update.",
- })
- .refine(assertExclusiveEvaluatorFields, {
- path: ["evaluators"],
- message: EXCLUSIVE_EVALUATOR_FIELDS_MESSAGE,
- });
-export type PatchUnstableEvaluationRuleBodyType = z.infer<
- typeof PatchUnstableEvaluationRuleBody
->;
-
-/** @alias */
-export const PatchUnstableEvaluationRuleResponse = APIReadableV2EvaluationRule;
-
-/** @alias */
-export const DeleteUnstableEvaluationRuleQuery = GetUnstableEvaluationRuleQuery;
-
-export const DeleteUnstableEvaluationRuleResponse = z
- .object({
- message: z.literal("Evaluation rule successfully deleted"),
- })
- .strict();
diff --git a/web/src/features/public-api/types/unstable-evaluators.ts b/web/src/features/public-api/types/unstable-evaluators.ts
deleted file mode 100644
index 74779a0788..0000000000
--- a/web/src/features/public-api/types/unstable-evaluators.ts
+++ /dev/null
@@ -1,122 +0,0 @@
-import { z } from "zod";
-import {
- PUBLIC_EVALUATOR_TYPE_CODE,
- PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- ObservationPromptVariableMappingInput,
- PromptVariableMappingRead,
- PublicCodeEvaluatorDefinitionInput,
- PublicEvaluatorModelConfig,
- PublicEvaluatorOutputDefinition,
- PublicLlmAsJudgeEvaluatorDefinitionInput,
- UnstablePublicApiPaginationQuery,
- UnstablePublicApiPaginationResponse,
-} from "@/src/features/public-api/types/unstable-public-evals-contract";
-
-const APIEvaluatorBase = z
- .object({
- id: z.string(),
- name: z.string(),
- version: z.number().int().positive(),
- variables: z.array(z.string()),
- // An evaluator's default mapping can name experiment-only sources, and a legacy one can be
- // incomplete, so reads use the permissive schema. Requests stay strict.
- mapping: z.array(PromptVariableMappingRead).nullable(),
- evaluationRuleCount: z.number().int().nonnegative(),
- createdAt: z.coerce.date(),
- updatedAt: z.coerce.date(),
- })
- .strict();
-
-const APILlmAsJudgeEvaluator = APIEvaluatorBase.extend({
- type: z.literal(PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE),
- prompt: z.string(),
- outputDefinition: PublicEvaluatorOutputDefinition,
- modelConfig: PublicEvaluatorModelConfig.nullable(),
-}).strict();
-
-const APICodeEvaluator = APIEvaluatorBase.extend({
- type: z.literal(PUBLIC_EVALUATOR_TYPE_CODE),
- sourceCode: z.string().min(1),
- sourceCodeLanguage:
- PublicCodeEvaluatorDefinitionInput.shape.sourceCodeLanguage,
-}).strict();
-
-const APIEvaluator = z.discriminatedUnion("type", [
- APILlmAsJudgeEvaluator,
- APICodeEvaluator,
-]);
-
-export const GetUnstableEvaluatorsQuery = UnstablePublicApiPaginationQuery;
-
-export const GetUnstableEvaluatorsResponse = z
- .object({
- data: z.array(APIEvaluator),
- meta: UnstablePublicApiPaginationResponse,
- })
- .strict();
-
-export const GetUnstableEvaluatorQuery = z.object({
- evaluatorId: z.string(),
-});
-
-/** @alias */
-export const GetUnstableEvaluatorResponse = APIEvaluator;
-
-/** @alias */
-export const DeleteUnstableEvaluatorQuery = GetUnstableEvaluatorQuery;
-
-export const DeleteUnstableEvaluatorResponse = z
- .object({
- message: z.literal("Evaluator successfully deleted"),
- })
- .strict();
-
-// Fields shared by every create body, regardless of evaluator type. Exported so
-// non-route consumers (e.g. the MCP tool layer) reuse the same definition.
-const EvaluatorCreateBase = {
- name: z.string().min(1),
-};
-
-const PostUnstableLlmAsJudgeEvaluatorBody = z.object({
- ...EvaluatorCreateBase,
- type: z.literal(PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE),
- ...PublicLlmAsJudgeEvaluatorDefinitionInput.shape,
- mapping: z.array(ObservationPromptVariableMappingInput).optional(),
- sourceCode: z.never().optional(),
- sourceCodeLanguage: z.never().optional(),
-});
-
-const PostUnstableCodeEvaluatorBody = z.object({
- ...EvaluatorCreateBase,
- type: z.literal(PUBLIC_EVALUATOR_TYPE_CODE),
- ...PublicCodeEvaluatorDefinitionInput.shape,
- prompt: z.never().optional(),
- outputDefinition: z.never().optional(),
- modelConfig: z.never().optional(),
- mapping: z.never().optional(),
-});
-
-const PostUnstableTypedEvaluatorBody = z.discriminatedUnion("type", [
- PostUnstableLlmAsJudgeEvaluatorBody,
- PostUnstableCodeEvaluatorBody,
-]);
-
-// `type` may be omitted; it defaults to `llm_as_judge` for backwards
-// compatibility. New clients should send `type` explicitly.
-export const PostUnstableEvaluatorBody = z.preprocess((value) => {
- if (typeof value !== "object" || value === null || "type" in value) {
- return value;
- }
-
- return {
- ...value,
- type: PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE,
- };
-}, PostUnstableTypedEvaluatorBody);
-
-export type PostUnstableEvaluatorBodyParsedType = z.infer<
- typeof PostUnstableTypedEvaluatorBody
->;
-
-/** @alias */
-export const PostUnstableEvaluatorResponse = APIEvaluator;
diff --git a/web/src/features/public-api/types/unstable-public-evals-contract.ts b/web/src/features/public-api/types/unstable-public-evals-contract.ts
deleted file mode 100644
index a172d6e62e..0000000000
--- a/web/src/features/public-api/types/unstable-public-evals-contract.ts
+++ /dev/null
@@ -1,360 +0,0 @@
-import {
- arrayOptionsFilter,
- categoryOptionsFilter,
- nullFilter,
- numberFilter,
- numberObjectFilter,
- observationEvalFilterColumns,
- paginationMetaResponseZod,
- stringFilter,
- stringObjectFilter,
- stringOptionsFilter,
- timeFilter,
- experimentEvalFilterColumns,
- booleanFilter,
- CODE_EVAL_SOURCE_MAX_BYTES,
- langfuseObjects,
-} from "@langfuse/shared";
-import { z } from "zod";
-export { StructuredPublicApiErrorResponse } from "./structuredPublicApiErrorSchema";
-
-const PUBLIC_EVALUATOR_TYPES = ["llm_as_judge", "code"] as const;
-export const [PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE, PUBLIC_EVALUATOR_TYPE_CODE] =
- PUBLIC_EVALUATOR_TYPES;
-
-export const PublicEvaluatorType = z.enum(PUBLIC_EVALUATOR_TYPES);
-const PublicCodeEvaluatorSourceCodeLanguage = z.enum(["PYTHON", "TYPESCRIPT"]);
-
-export const PublicEvaluatorModelConfig = z.object({
- provider: z.string().min(1),
- model: z.string().min(1),
-});
-
-const PublicEvaluatorOutputFieldDefinition = z.object({
- description: z.string().trim().min(1),
-});
-
-const PublicNumericEvaluatorOutputScoreDefinition =
- PublicEvaluatorOutputFieldDefinition.extend({
- minValue: z.number().optional(),
- maxValue: z.number().optional(),
- }).refine(
- ({ minValue, maxValue }) =>
- minValue === undefined || maxValue === undefined || minValue <= maxValue,
- {
- message: "Minimum value must be less than or equal to maximum value",
- },
- );
-
-const PublicNumericEvaluatorOutputDefinition = z.object({
- dataType: z.literal("NUMERIC"),
- reasoning: PublicEvaluatorOutputFieldDefinition,
- score: PublicNumericEvaluatorOutputScoreDefinition,
-});
-
-const PublicBooleanEvaluatorOutputDefinition = z.object({
- dataType: z.literal("BOOLEAN"),
- reasoning: PublicEvaluatorOutputFieldDefinition,
- score: PublicEvaluatorOutputFieldDefinition,
-});
-
-const PublicCategoricalEvaluatorOutputScoreDefinition = z.object({
- description: z.string().trim().min(1),
- categories: z.array(z.string().trim().min(1)).min(2),
- shouldAllowMultipleMatches: z.boolean(),
-});
-
-const PublicCategoricalEvaluatorOutputDefinition = z.object({
- dataType: z.literal("CATEGORICAL"),
- reasoning: PublicEvaluatorOutputFieldDefinition,
- score: PublicCategoricalEvaluatorOutputScoreDefinition,
-});
-
-export const PublicEvaluatorOutputDefinition = z.discriminatedUnion(
- "dataType",
- [
- PublicNumericEvaluatorOutputDefinition,
- PublicBooleanEvaluatorOutputDefinition,
- PublicCategoricalEvaluatorOutputDefinition,
- ],
-);
-
-export const PublicEvaluationRuleTarget = z.enum(["observation", "experiment"]);
-export const PublicEvaluationRuleLegacyTarget = z.enum(["trace", "dataset"]);
-const _PublicEvaluationRuleReadTarget = z.union([
- PublicEvaluationRuleTarget,
- PublicEvaluationRuleLegacyTarget,
-]);
-
-export const PublicEvaluationRuleStatus = z.enum([
- "active",
- "inactive",
- "paused",
-]);
-
-// No `scope`: every evaluator is project-owned, so the field could only ever
-// carry one value. Gallery templates are starter definitions that are copied
-// into the project on save, not addressable Langfuse-managed resources.
-export const PublicEvaluationRuleEvaluatorReference = z.object({
- name: z.string().min(1),
- type: PublicEvaluatorType.default(PUBLIC_EVALUATOR_TYPE_LLM_AS_JUDGE),
-});
-
-// PATCH intentionally omits `type`: an evaluation rule's evaluator type cannot
-// be changed. The service always inherits the rule's current type, so a code
-// rule is never retargeted to an LLM evaluator family, which stores its
-// mapping per assignment and would inherit none. To use a different evaluator
-// type, create a new rule.
-export const PublicEvaluationRuleEvaluatorReferencePatch = z.object({
- name: z.string().min(1),
-});
-
-export const PublicEvaluationRuleEvaluator =
- PublicEvaluationRuleEvaluatorReference.extend({
- id: z.string(),
- });
-
-export const ObservationPromptVariableMappingSource = z.enum([
- "input",
- "output",
- "metadata",
- "tool_calls",
-]);
-
-export const ExperimentPromptVariableMappingSource = z.enum([
- "input",
- "output",
- "metadata",
- "tool_calls",
- "expected_output",
- "experiment_item_metadata",
-]);
-
-function createMappingSchema<
- TSource extends z.ZodType<
- | "input"
- | "output"
- | "metadata"
- | "tool_calls"
- | "expected_output"
- | "experiment_item_metadata"
- >,
->(sourceSchema: TSource) {
- return z.object({
- variable: z.string().min(1),
- source: sourceSchema,
- jsonPath: z.string().min(1).optional(),
- });
-}
-
-export const ObservationPromptVariableMappingInput = createMappingSchema(
- ObservationPromptVariableMappingSource,
-);
-
-export const ExperimentPromptVariableMappingInput = createMappingSchema(
- ExperimentPromptVariableMappingSource,
-);
-
-const _PromptVariableMappingInput = z.union([
- ObservationPromptVariableMappingInput,
- ExperimentPromptVariableMappingInput,
-]);
-
-// This shape preserves incomplete mappings so callers can repair them.
-// Mapping inputs remain strict and require a concrete source.
-export const PromptVariableMappingRead = z.object({
- variable: z.string().min(1),
- source: ExperimentPromptVariableMappingSource.nullable(),
- jsonPath: z.string().min(1).optional(),
-});
-
-export const LegacyPromptVariableMapping = z
- .object({
- variable: z.string().min(1),
- langfuseObject: z.enum(langfuseObjects),
- objectName: z.string().nullable(),
- source: z.string().min(1),
- jsonPath: z.string().min(1).optional(),
- })
- .refine(
- (mapping) =>
- mapping.langfuseObject === "trace" ||
- mapping.langfuseObject === "dataset_item" ||
- mapping.objectName !== null,
- {
- path: ["objectName"],
- message: "objectName is required for observation objects",
- },
- );
-
-const PublicEvaluationRuleReadFilterBase = z
- .object({
- type: z.string(),
- column: z.string(),
- operator: z.string(),
- value: z.unknown().optional(),
- })
- .loose();
-
-export const PublicEvaluationRuleReadFilter = z.union([
- PublicEvaluationRuleReadFilterBase.safeExtend({ key: z.string() }),
- PublicEvaluationRuleReadFilterBase,
-]);
-
-const filterSchemaFactories = {
- datetime: (columnId: string) =>
- timeFilter.safeExtend({ column: z.literal(columnId) }),
- string: (columnId: string) =>
- stringFilter.safeExtend({ column: z.literal(columnId) }),
- number: (columnId: string) =>
- numberFilter.safeExtend({ column: z.literal(columnId) }),
- stringOptions: (columnId: string) =>
- stringOptionsFilter.safeExtend({ column: z.literal(columnId) }),
- categoryOptions: (columnId: string) =>
- categoryOptionsFilter.safeExtend({ column: z.literal(columnId) }),
- arrayOptions: (columnId: string) =>
- arrayOptionsFilter.safeExtend({ column: z.literal(columnId) }),
- stringObject: (columnId: string) =>
- stringObjectFilter.safeExtend({ column: z.literal(columnId) }),
- numberObject: (columnId: string) =>
- numberObjectFilter.safeExtend({ column: z.literal(columnId) }),
- boolean: (columnId: string) =>
- booleanFilter.safeExtend({ column: z.literal(columnId) }),
- null: (columnId: string) =>
- nullFilter.safeExtend({ column: z.literal(columnId) }),
-} as const;
-
-type SupportedFilterFactory = keyof typeof filterSchemaFactories;
-
-function createTargetFilterSchema(
- columns: Array<{ id: string; type: SupportedFilterFactory }>,
-) {
- const schemas = columns.map((column) =>
- filterSchemaFactories[column.type](column.id),
- );
-
- if (schemas.length === 1) {
- return schemas[0]!;
- }
-
- return z.union(
- schemas as [
- (typeof schemas)[number],
- (typeof schemas)[number],
- ...Array<(typeof schemas)[number]>,
- ],
- );
-}
-
-// The experiment-root filter is what the `experiment` target *means*, so it is
-// not separately addressable: exposing it on `observation` too would give one
-// rule two contradictory public representations.
-const EXPERIMENT_ROOT_FILTER_COLUMN = "isExperimentItemRootSpan";
-
-export const OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS =
- observationEvalFilterColumns
- .filter((column) => column.id !== EXPERIMENT_ROOT_FILTER_COLUMN)
- .map((column) => ({
- id: column.id,
- type: column.type as SupportedFilterFactory,
- }));
-
-// An experiment rule is an observation rule scoped to experiment root spans,
-// so it accepts every observation filter plus the dataset scope.
-export const EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS = [
- ...OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,
- {
- id: "datasetId",
- type: experimentEvalFilterColumns[0]!.type as SupportedFilterFactory,
- },
-];
-
-export const ObservationEvaluationRuleFilter = createTargetFilterSchema(
- OBSERVATION_EVALUATION_RULE_FILTER_COLUMNS,
-);
-
-export const ExperimentEvaluationRuleFilter = createTargetFilterSchema(
- EXPERIMENT_EVALUATION_RULE_FILTER_COLUMNS,
-);
-
-export const PublicEvaluationRuleFilter = z.union([
- ObservationEvaluationRuleFilter,
- ExperimentEvaluationRuleFilter,
-]);
-
-export type PublicEvaluatorModelConfigType = z.infer<
- typeof PublicEvaluatorModelConfig
->;
-export type PublicEvaluatorOutputDefinitionType = z.infer<
- typeof PublicEvaluatorOutputDefinition
->;
-export type PublicEvaluatorTypeType = z.infer<typeof PublicEvaluatorType>;
-export type PublicEvaluationRuleTargetType = z.infer<
- typeof PublicEvaluationRuleTarget
->;
-export type PublicEvaluationRuleLegacyTargetType = z.infer<
- typeof PublicEvaluationRuleLegacyTarget
->;
-export type PublicEvaluationRuleReadTargetType = z.infer<
- typeof _PublicEvaluationRuleReadTarget
->;
-export type PublicEvaluationRuleStatusType = z.infer<
- typeof PublicEvaluationRuleStatus
->;
-export type PublicEvaluationRuleEvaluatorReferenceType = z.infer<
- typeof PublicEvaluationRuleEvaluatorReference
->;
-export type PublicEvaluationRuleEvaluatorType = z.infer<
- typeof PublicEvaluationRuleEvaluator
->;
-export type PromptVariableMappingInputType = z.infer<
- typeof _PromptVariableMappingInput
->;
-export type PromptVariableMappingReadType = z.infer<
- typeof PromptVariableMappingRead
->;
-export type LegacyPromptVariableMappingType = z.infer<
- typeof LegacyPromptVariableMapping
->;
-export type PublicEvaluationRuleFilterType = z.infer<
- typeof PublicEvaluationRuleFilter
->;
-export type PublicEvaluationRuleReadFilterType = z.infer<
- typeof PublicEvaluationRuleReadFilter
->;
-export const UnstablePublicApiPaginationQuery = z.object({
- page: z.preprocess(
- (x) => (x === "" ? undefined : x),
- z.coerce.number().int().gt(0).default(1),
- ),
- limit: z.preprocess(
- (x) => (x === "" ? undefined : x),
- z.coerce.number().int().gt(0).lte(100).default(50),
- ),
-});
-
-export const UnstablePublicApiPaginationResponse = paginationMetaResponseZod;
-
-export const PublicLlmAsJudgeEvaluatorDefinitionInput = z.object({
- prompt: z.string().min(1),
- outputDefinition: PublicEvaluatorOutputDefinition,
- modelConfig: PublicEvaluatorModelConfig.nullable().optional(),
-});
-
-export const PublicCodeEvaluatorDefinitionInput = z.object({
- sourceCode: z
- .string()
- .min(1)
- .refine(
- (sourceCode) =>
- Buffer.byteLength(sourceCode, "utf8") <= CODE_EVAL_SOURCE_MAX_BYTES,
- {
- message: `Source code must be ${CODE_EVAL_SOURCE_MAX_BYTES} bytes or less`,
- },
- ),
- sourceCodeLanguage: PublicCodeEvaluatorSourceCodeLanguage,
-});
-
-export type PublicCodeEvaluatorSourceCodeLanguageType = z.infer<
- typeof PublicCodeEvaluatorSourceCodeLanguage
->;
diff --git a/web/src/pages/api/public/unstable/evaluation-rules/[evaluationRuleId].ts b/web/src/pages/api/public/unstable/evaluation-rules/[evaluationRuleId].ts
deleted file mode 100644
index a0c9571d32..0000000000
--- a/web/src/pages/api/public/unstable/evaluation-rules/[evaluationRuleId].ts
+++ /dev/null
@@ -1,64 +0,0 @@
-import {
- deletePublicEvaluationRule,
- getPublicEvaluationRule,
- updatePublicEvaluationRule,
-} from "@/src/features/evals/server/unstable-public-api";
-import {
- createUnstablePublicApiRoute,
- withUnstablePublicApiMiddlewares,
-} from "@/src/features/public-api/server/unstable-public-api-route";
-import {
- DeleteUnstableEvaluationRuleQuery,
- DeleteUnstableEvaluationRuleResponse,
- GetUnstableEvaluationRuleQuery,
- GetUnstableEvaluationRuleResponse,
- PatchUnstableEvaluationRuleBody,
- PatchUnstableEvaluationRuleQuery,
- PatchUnstableEvaluationRuleResponse,
-} from "@/src/features/public-api/types/unstable-evaluation-rules";
-
-export default withUnstablePublicApiMiddlewares({
- GET: createUnstablePublicApiRoute({
- name: "Get Unstable Evaluation Rule",
- action: "evaluationRule:read",
- querySchema: GetUnstableEvaluationRuleQuery,
- responseSchema: GetUnstableEvaluationRuleResponse,
- fn: async ({ query, auth }) =>
- getPublicEvaluationRule({
- projectId: auth.scope.projectId,
- evaluationRuleId: query.evaluationRuleId,
- }),
- }),
- PATCH: createUnstablePublicApiRoute({
- name: "Update Unstable Evaluation Rule",
- action: "evaluationRule:CUD",
- querySchema: PatchUnstableEvaluationRuleQuery,
- bodySchema: PatchUnstableEvaluationRuleBody,
- responseSchema: PatchUnstableEvaluationRuleResponse,
- fn: async ({ query, body, auth }) =>
- updatePublicEvaluationRule({
- orgId: auth.scope.orgId,
- projectId: auth.scope.projectId,
- evaluationRuleId: query.evaluationRuleId,
- input: body,
- auditScope: auth.scope,
- }),
- }),
- DELETE: createUnstablePublicApiRoute({
- name: "Delete Unstable Evaluation Rule",
- action: "evaluationRule:CUD",
- querySchema: DeleteUnstableEvaluationRuleQuery,
- responseSchema: DeleteUnstableEvaluationRuleResponse,
- fn: async ({ query, auth }) => {
- await deletePublicEvaluationRule({
- projectId: auth.scope.projectId,
- evaluationRuleId: query.evaluationRuleId,
- auditScope: auth.scope,
- });
-
- return {
- message: "Evaluation rule successfully deleted" as const,
- };
- },
- }),
-});
diff --git a/web/src/pages/api/public/unstable/evaluation-rules/index.ts b/web/src/pages/api/public/unstable/evaluation-rules/index.ts
deleted file mode 100644
index fde221ab97..0000000000
--- a/web/src/pages/api/public/unstable/evaluation-rules/index.ts
+++ /dev/null
@@ -1,42 +0,0 @@
-import {
- createPublicEvaluationRule,
- listPublicEvaluationRules,
-} from "@/src/features/evals/server/unstable-public-api";
-import {
- createUnstablePublicApiRoute,
- withUnstablePublicApiMiddlewares,
-} from "@/src/features/public-api/server/unstable-public-api-route";
-import {
- GetUnstableEvaluationRulesQuery,
- GetUnstableEvaluationRulesResponse,
- PostUnstableEvaluationRuleBody,
- PostUnstableEvaluationRuleResponse,
-} from "@/src/features/public-api/types/unstable-evaluation-rules";
-
-export default withUnstablePublicApiMiddlewares({
- GET: createUnstablePublicApiRoute({
- name: "List Unstable Evaluation Rules",
- action: "evaluationRule:read",
- querySchema: GetUnstableEvaluationRulesQuery,
- responseSchema: GetUnstableEvaluationRulesResponse,
- fn: async ({ query, auth }) =>
- listPublicEvaluationRules({
- projectId: auth.scope.projectId,
- page: query.page,
- limit: query.limit,
- }),
- }),
- POST: createUnstablePublicApiRoute({
- name: "Create Unstable Evaluation Rule",
- action: "evaluationRule:CUD",
- bodySchema: PostUnstableEvaluationRuleBody,
- responseSchema: PostUnstableEvaluationRuleResponse,
- fn: async ({ body, auth }) =>
- createPublicEvaluationRule({
- orgId: auth.scope.orgId,
- projectId: auth.scope.projectId,
- input: body,
- auditScope: auth.scope,
- }),
- }),
-});
diff --git a/web/src/pages/api/public/unstable/evaluators/[evaluatorId].ts b/web/src/pages/api/public/unstable/evaluators/[evaluatorId].ts
deleted file mode 100644
index a3ca669476..0000000000
--- a/web/src/pages/api/public/unstable/evaluators/[evaluatorId].ts
+++ /dev/null
@@ -1,45 +0,0 @@
-import {
- deletePublicEvaluator,
- getPublicEvaluator,
-} from "@/src/features/evals/server/unstable-public-api";
-import {
- createUnstablePublicApiRoute,
- withUnstablePublicApiMiddlewares,
-} from "@/src/features/public-api/server/unstable-public-api-route";
-import {
- DeleteUnstableEvaluatorQuery,
- DeleteUnstableEvaluatorResponse,
- GetUnstableEvaluatorQuery,
- GetUnstableEvaluatorResponse,
-} from "@/src/features/public-api/types/unstable-evaluators";
-
-export default withUnstablePublicApiMiddlewares({
- GET: createUnstablePublicApiRoute({
- name: "Get Unstable Evaluator",
- action: "evaluator:read",
- querySchema: GetUnstableEvaluatorQuery,
- responseSchema: GetUnstableEvaluatorResponse,
- fn: async ({ query, auth }) =>
- getPublicEvaluator({
- projectId: auth.scope.projectId,
- evaluatorId: query.evaluatorId,
- }),
- }),
- DELETE: createUnstablePublicApiRoute({
- name: "Delete Unstable Evaluator",
- action: "evaluator:CUD",
- querySchema: DeleteUnstableEvaluatorQuery,
- responseSchema: DeleteUnstableEvaluatorResponse,
- fn: async ({ query, auth }) => {
- await deletePublicEvaluator({
- projectId: auth.scope.projectId,
- evaluatorId: query.evaluatorId,
- auditScope: auth.scope,
- });
-
- return {
- message: "Evaluator successfully deleted" as const,
- };
- },
- }),
-});
diff --git a/web/src/pages/api/public/unstable/evaluators/index.ts b/web/src/pages/api/public/unstable/evaluators/index.ts
deleted file mode 100644
index 711e4718c9..0000000000
--- a/web/src/pages/api/public/unstable/evaluators/index.ts
+++ /dev/null
@@ -1,41 +0,0 @@
-import {
- createPublicEvaluator,
- listPublicEvaluators,
-} from "@/src/features/evals/server/unstable-public-api";
-import {
- createUnstablePublicApiRoute,
- withUnstablePublicApiMiddlewares,
-} from "@/src/features/public-api/server/unstable-public-api-route";
-import {
- GetUnstableEvaluatorsQuery,
- GetUnstableEvaluatorsResponse,
- PostUnstableEvaluatorBody,
- PostUnstableEvaluatorResponse,
-} from "@/src/features/public-api/types/unstable-evaluators";
-
-export default withUnstablePublicApiMiddlewares({
- GET: createUnstablePublicApiRoute({
- name: "List Unstable Evaluators",
- action: "evaluator:read",
- querySchema: GetUnstableEvaluatorsQuery,
- responseSchema: GetUnstableEvaluatorsResponse,
- fn: async ({ query, auth }) =>
- listPublicEvaluators({
- projectId: auth.scope.projectId,
- page: query.page,
- limit: query.limit,
- }),
- }),
- POST: createUnstablePublicApiRoute({
- name: "Create Unstable Evaluator",
- action: "evaluator:CUD",
- bodySchema: PostUnstableEvaluatorBody,
- responseSchema: PostUnstableEvaluatorResponse,
- fn: async ({ body, auth }) =>
- createPublicEvaluator({
- projectId: auth.scope.projectId,
- input: body,
- auditScope: auth.scope,
- }),
- }),
-});