:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): configure boolean decision thresholds (#122105)
Bernat Torres693258080878
diff --git a/docs/internal/ai-observability-judge-inputs.md b/docs/internal/ai-observability-judge-inputs.mdindex 80eee6a04d6d..a3da16392aed 100644--- a/docs/internal/ai-observability-judge-inputs.md+++ b/docs/internal/ai-observability-judge-inputs.md@@ -64,6 +64,7 @@ Decision models are available under the existing LLM judge option. They return tThe evaluation editor separates **Judge method** from **Model**. Choose **LLM with explanation** for chat completions or **Decision model** for typed decisions.Each method lists models with the corresponding capability from the project's saved provider connections. API keys stay in AI provider settings.Switching methods keeps a compatible model and its saved connection; an incompatible model is cleared without changing the criteria or output settings.+Leaving decision judging for LLM judging or Hog resets an invalid threshold to its default and preserves valid thresholds.Changing the judge method marks the evaluation as edited, including when the selected model supports both methods.The selection is stored as `evaluation_config.judge_method` (`llm` or `decision`). Evaluations without this field keep automatic routing until a method or model is explicitly selected.Automatic OpenRouter routing selects decisions only for models without text output. Models supporting both methods keep chat routing unless decisions are explicitly selected.@@ -147,7 +148,10 @@ API compatibility does not guarantee equivalent judgments or calibration acrossCompare results on representative inputs when changing models.For boolean evaluations, the prompt becomes a [Noul question](https://docs.typesafe.ai/primitives/noul).-A probability of at least 0.5 produces `true`; the evaluation's existing pass/fail polarity still applies.+A probability at or above `output_config.decision_threshold` produces `true`; the evaluation's existing pass/fail polarity still applies.+The threshold accepts finite numbers from 0 to 1, inclusive, and defaults to 0.5 when omitted or null.+For boolean judges, selecting **Decision model** shows **Decision threshold** as a percentage inside the **Judge model** section, below the model selector.+Changing the threshold affects new runs only; historical verdicts remain unchanged.The raw probability is stored in `$ai_evaluation_probability` when the criteria apply, with available token usage and the configured model ID.Missing or invalid token counts remain unknown and do not discard a valid answer.Ingestion estimates cost from the configured model and token usage when the model appears in the existing pricing catalog.diff --git a/frontend/snapshots.yml b/frontend/snapshots.ymlindex ffceec9a538b..1ec1575d5f2b 100644--- a/frontend/snapshots.yml+++ b/frontend/snapshots.yml@@ -12871,13 +12871,13 @@ snapshots:scenes-app-llm-observability-evaluation-templates--narrow--light:hash: v1.k794b7964.acb588946dd41382c78934be3f8912589d1a15df06ef9a976f617fad21f1ddd2.0LsEsqRoncEFZhEVh5X8gbA_QoGZelV2t1ABDsxGeaoscenes-app-llm-observability-evaluationmodelpicker--decision--dark:- hash: v1.k794b7964.a9c0baac61b11f86986d2939d35b4a97a7e81802c86fd4ccbf07c9ee3369fd15.XiipYzOUcq18KZELmDMUdwA6n0FdToNVsUCXbmaCfns+ hash: v1.k794b7964.0325f8084a7ecb6745b8164ee4c198f67f133e02c8c6e7431b611a560d757bbf.txtMNRNYHw5Xmf1gnMpp7hAyShiaVIDV2m1SYroxnMEscenes-app-llm-observability-evaluationmodelpicker--decision--light:- hash: v1.k794b7964.45efc6f2f9903de48507033653d344909ed13bf1731c89056bad14e69b96e12a.9NGJH7yteZxQASERxRgjjh0STONDRx5CP2OvAl3uBD8+ hash: v1.k794b7964.a3ad1f30e680fd2cdf3add2e75822037390ff220d3c7aef549f1f32cebdcc7c6.-bNzbWVEQXCqou9cnKIz70PTxSFkha0pDknYqaC5tnMscenes-app-llm-observability-evaluationmodelpicker--decision-narrow--dark:- hash: v1.k794b7964.73dd8de58ec7a3f458f01cf723a64e3114ab680dc6efc965ebf9a6d65d6e1e0e.cC-ebDDJe9CAloNQ1UyVCVd7utawTwykH4Wuo-uVSp4+ hash: v1.k794b7964.58dc8d73bf85d6c8bcd7a10b08c1e38efefd840fbe912cbbdff34bc4596041d8.5_RSGKERITVltMJ5ap2pmDAF2IC77kXkHPRfyXJi2oYscenes-app-llm-observability-evaluationmodelpicker--decision-narrow--light:- hash: v1.k794b7964.0c1371aecd01164f6f1070fbfc682a391e11e0f9e36277cefd75db5d479b7235.iY4gG-YkcdP2VT2N3wvgPIvQf-EScIKvrF58syfVw7U+ hash: v1.k794b7964.28a30a7cb478ba7b585a0736e53e74735c643e3b61b0b84059494f6040493c29.haO943nTQB3hFa_f-TgDhACaCqM3_bOKVEdmp0SXTzgscenes-app-llm-observability-evaluationmodelpicker--flag-off--dark:hash: v1.k794b7964.277344d00b8edc7551d8850029611c83e1740ac1ea3adb9e679ceffd2037064b.0gLUQZyT3_xppzOwry0tkGL0HlIjU-jdxq6V8VIAGs4scenes-app-llm-observability-evaluationmodelpicker--flag-off--light:@@ -12887,17 +12887,17 @@ snapshots:scenes-app-llm-observability-evaluationmodelpicker--llm--light:hash: v1.k794b7964.5fbd678972f802f1baf3bb1e1dc4869a6d2b78c8279bc5f03e3b55e149d24c8f.FAWGdhvRzIqww3vfNkkv-FISpmlDCBK9QHYPiQ6sS6cscenes-app-llm-observability-evaluationmodelpicker--native-open-ai--dark:- hash: v1.k794b7964.a9c0baac61b11f86986d2939d35b4a97a7e81802c86fd4ccbf07c9ee3369fd15.bW1bQMgWLkSMejVZ8-8gt9tEchiuvd_AeJf0gGcv3Yc+ hash: v1.k794b7964.0325f8084a7ecb6745b8164ee4c198f67f133e02c8c6e7431b611a560d757bbf.hqOKsQgqwHGk8LsGxX1mnQMpHyhzl00XVFCNrp8uPe0scenes-app-llm-observability-evaluationmodelpicker--native-open-ai--light:- hash: v1.k794b7964.45efc6f2f9903de48507033653d344909ed13bf1731c89056bad14e69b96e12a.VJAyamIoCFCRiqrLboD2BBVE_2bk2JzQNdSGQt9w61c+ hash: v1.k794b7964.a3ad1f30e680fd2cdf3add2e75822037390ff220d3c7aef549f1f32cebdcc7c6.wlwriSaiiF198e3yVP5IaIdmT9kaWmMlGSOehEhg5rUscenes-app-llm-observability-evaluationmodelpicker--native-open-ai-narrow--dark:- hash: v1.k794b7964.73dd8de58ec7a3f458f01cf723a64e3114ab680dc6efc965ebf9a6d65d6e1e0e.73I_zrtRgTcZD5J-9y-5A96QnzewfQbOlS-GzqXqDN8+ hash: v1.k794b7964.58dc8d73bf85d6c8bcd7a10b08c1e38efefd840fbe912cbbdff34bc4596041d8.u5g-WppnRiIjPg1k0jD2eqEX-5Qt1nlyABrap6Toqasscenes-app-llm-observability-evaluationmodelpicker--native-open-ai-narrow--light:- hash: v1.k794b7964.0c1371aecd01164f6f1070fbfc682a391e11e0f9e36277cefd75db5d479b7235.BQM5LKEuJFwtTUd-VRi-E516jncsN5ywMGN8NjsLWq0+ hash: v1.k794b7964.28a30a7cb478ba7b585a0736e53e74735c643e3b61b0b84059494f6040493c29.DaQknPZWc0jSBbq13iEjT3m5vvDAlm5QHELIewdA7PYscenes-app-llm-observability-evaluationmodelpicker--no-decision-models--dark:- hash: v1.k794b7964.ba5206f054f85372119023e24c4a01da4d0777ba31f66807cdc0523de4d5edc0.qr36U7Blkwgs46fNy1QTqCUJXEtr0q0kjeFteZJwJWo+ hash: v1.k794b7964.681693474e46e0164b0a832b7d33c991f1b6c1c944cf5f23c64e3ee90a223096.MrFeO9h-IUD2Yd0l-zbg2KzxcUkskx57GnPNKwKhcv0scenes-app-llm-observability-evaluationmodelpicker--no-decision-models--light:- hash: v1.k794b7964.454b4b8ed00b0edb5cc452a5c4c5621df9c689076553f4c435ceb9897a90aac0.uf4yjayq1yAn_Mc0x4cZFyMuXDsZ629E1m2ARKrTPW8+ hash: v1.k794b7964.39f6167f6c51638beef826219c123100b5ae9207c817bfaa6811f5428beb4da5._LdpuKYQp9L79AcfpQpKH-XHosP2Tymw21tcAFJDWCkscenes-app-llm-observability-evaluationreportconfig--lemon-ui--dark:hash: v1.k794b7964.195aa4a5a39e8f947e17fe67f5c23303b7dd0579c278907761be24a08780be92.IkGQtAI95T60kJKVZcmmFBIWNCd4meuJv-bcNDoohwUscenes-app-llm-observability-evaluationreportconfig--lemon-ui--light:diff --git a/posthog/temporal/ai_observability/evaluation_llm_judge.py b/posthog/temporal/ai_observability/evaluation_llm_judge.pyindex 6879d7e6b7d8..6eff5f45e0aa 100644--- a/posthog/temporal/ai_observability/evaluation_llm_judge.py+++ b/posthog/temporal/ai_observability/evaluation_llm_judge.py@@ -79,6 +79,7 @@ from products.ai_observability.backend.llm.providers.openai_decisions import Opefrom products.ai_observability.backend.llm.providers.openrouter import OPENROUTER_DECISIONS_BASE_URL, decision_model_idsfrom products.ai_observability.backend.llm.types import CompletionResponsefrom products.ai_observability.backend.models.evaluation_configs import (+ BooleanOutputConfig,CategoricalOutputConfig,NumericOutputConfig,NumericScoreOutOfBounds,@@ -784,13 +785,16 @@ def call_llm_judge(if not isinstance(verdict_answer, NoulAnswer):raise StructuredOutputParseError("The endpoint returned an invalid verdict answer.")probability = verdict_answer.probability+ configured_threshold = BooleanOutputConfig.model_validate(output_config).decision_threshold+ # Zero is a valid cutoff.+ threshold = configured_threshold if configured_threshold is not None else 0.5parsed = (BooleanWithNAEvalResult(reasoning="",- outcome="not_applicable" if not applicable else "pass" if probability >= 0.5 else "fail",+ outcome="not_applicable" if not applicable else "pass" if probability >= threshold else "fail",)if allows_na- else BooleanEvalResult(reasoning="", verdict=probability >= 0.5)+ else BooleanEvalResult(reasoning="", verdict=probability >= threshold))response = CompletionResponse(content="",diff --git a/posthog/temporal/ai_observability/test_run_evaluation.py b/posthog/temporal/ai_observability/test_run_evaluation.pyindex bf7da3f00279..debb51292861 100644--- a/posthog/temporal/ai_observability/test_run_evaluation.py+++ b/posthog/temporal/ai_observability/test_run_evaluation.py@@ -411,6 +411,18 @@ def test_legacy_and_explicit_llm_methods_use_chat_for_a_dual_capability_model([{"name": "verdict", "type": "predicate", "probability": 0.5}],{"verdict": True, "probability": 0.5},),+ (+ "boolean",+ {"decision_threshold": 0.8},+ [{"name": "verdict", "type": "predicate", "probability": 0.7}],+ {"verdict": False, "probability": 0.7},+ ),+ (+ "boolean",+ {"decision_threshold": 0.35},+ [{"name": "verdict", "type": "predicate", "probability": 0.35}],+ {"verdict": True, "probability": 0.35},+ ),("categorical",{@@ -625,8 +637,20 @@ def test_openrouter_decision_model_with_disabled_flag_skips_without_disabling(fl],)@pytest.mark.parametrize(- "probability,applicability,allows_na,verdict",- [(0.49, 1.0, False, False), (0.5, 1.0, False, True), (0.9, 0.1, True, None), (0.0, 0.9, True, False)],+ "probability,applicability,allows_na,threshold,verdict",+ [+ (0.49, 1.0, False, None, False),+ (0.5, 1.0, False, None, True),+ (0.9, 0.1, True, None, None),+ (0.0, 0.9, True, None, False),+ (0.7, 1.0, False, 0.8, False),+ (0.35, 1.0, False, 0.35, True),+ (0.0, 1.0, False, 0.0, True),+ (1.0, 1.0, False, 1.0, True),+ (0.7, 0.6, True, 0.8, False),+ (0.8, 0.6, True, 0.8, True),+ (0.9, 0.4, True, 0.2, None),+ ],)@pytest.mark.parametrize("judge_method", [None, "decision"])def test_system_one_judge_emits_boolean_probability_without_reasoning(@@ -634,6 +658,7 @@ def test_system_one_judge_emits_boolean_probability_without_reasoning(probability: float,applicability: float,allows_na: bool,+ threshold: float | None,verdict: bool | None,provider: str,connection_config: dict[str, str],@@ -657,6 +682,7 @@ def test_system_one_judge_emits_boolean_probability_without_reasoning("id": "test-evaluation","name": "Politeness","team_id": 1,+ "output_config": {"decision_threshold": threshold} if threshold is not None else {},"evaluation_config": {"prompt": prompt,**({"judge_method": judge_method} if judge_method else {}),diff --git a/products/ai_observability/backend/api/evaluations.py b/products/ai_observability/backend/api/evaluations.pyindex 78fc82a8b3ed..d8b8742743fa 100644--- a/products/ai_observability/backend/api/evaluations.py+++ b/products/ai_observability/backend/api/evaluations.py@@ -169,6 +169,13 @@ class _EvaluationConfigField(serializers.JSONField):"Set it to true for detector-style evaluations, so a true ""result is counted and labeled as a fail.",},+ "decision_threshold": {+ "type": "number",+ "nullable": True,+ "minimum": 0,+ "maximum": 1,+ "description": "Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.",+ },"min": {"type": "number","nullable": True,@@ -425,6 +432,7 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSehelp_text=("Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and ""{true_is_failure} to declare that a true result means the evaluation found a problem. "+ "For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). ""For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. ""For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."),diff --git a/products/ai_observability/backend/api/test/test_evaluations.py b/products/ai_observability/backend/api/test/test_evaluations.pyindex 31c47cd06933..2464809e4cd5 100644--- a/products/ai_observability/backend/api/test/test_evaluations.py+++ b/products/ai_observability/backend/api/test/test_evaluations.py@@ -575,13 +575,15 @@ class TestEvaluationConfigsApi(APIBaseTest):@parameterized.expand([- ("openai", "gpt-5-mini", None),- ("openai", "gpt-5-mini", "llm"),- ("system_one", "example-judge-v1", None),- ("system_one", "example-judge-v1", "decision"),+ ("openai", "gpt-5-mini", None, None),+ ("openai", "gpt-5-mini", "llm", None),+ ("system_one", "example-judge-v1", None, 0.35),+ ("system_one", "example-judge-v1", "decision", 0.8),])- def test_can_create_evaluation_config(self, provider: str, model: str, judge_method: str | None) -> None:+ def test_can_create_evaluation_config(+ self, provider: str, model: str, judge_method: str | None, threshold: float | None+ ) -> None:config = {"prompt": "Test prompt"}if judge_method:config["judge_method"] = judge_method@@ -607,7 +609,7 @@ class TestEvaluationConfigsApi(APIBaseTest):else _DEFAULT_MODEL_CONFIGURATION,"evaluation_config": config,"output_type": "boolean",- "output_config": {},+ "output_config": {"decision_threshold": threshold} if threshold is not None else {},"conditions": [{"id": "test-condition", "rollout_percentage": 50, "properties": []}],},)@@ -646,7 +648,10 @@ class TestEvaluationConfigsApi(APIBaseTest):self.assertEqual(malformed.status_code, status.HTTP_400_BAD_REQUEST)self.assertEqual(malformed.data["attr"], "evaluation_config")self.assertEqual(evaluation_config.output_type, "boolean")- self.assertEqual(evaluation_config.output_config, {"allows_na": False, "true_is_failure": False})+ expected_output_config: dict[str, bool | float] = {"allows_na": False, "true_is_failure": False}+ if threshold is not None:+ expected_output_config["decision_threshold"] = threshold+ self.assertEqual(evaluation_config.output_config, expected_output_config)self.assertEqual(len(evaluation_config.conditions), 1)self.assertEqual(evaluation_config.conditions[0]["id"], "test-condition")self.assertEqual(evaluation_config.team, self.team)@@ -670,17 +675,27 @@ class TestEvaluationConfigsApi(APIBaseTest):("allows_na",{"allows_na": False},- {"allows_na": False, "true_is_failure": True},+ {"allows_na": False, "true_is_failure": True, "decision_threshold": 0.35},),("true_is_failure",{"true_is_failure": False},- {"allows_na": True, "true_is_failure": False},+ {"allows_na": True, "true_is_failure": False, "decision_threshold": 0.35},+ ),+ (+ "decision_threshold",+ {"decision_threshold": 0.8},+ {"allows_na": True, "true_is_failure": True, "decision_threshold": 0.8},+ ),+ (+ "reset_decision_threshold",+ {"decision_threshold": None},+ {"allows_na": True, "true_is_failure": True},),])def test_partial_update_preserves_other_output_config_fields(- self, _field_name: str, output_config: dict[str, bool], expected: dict[str, bool]+ self, _field_name: str, output_config: dict[str, bool | float | None], expected: dict[str, bool | float]) -> None:evaluation = Evaluation.objects.create(team=self.team,@@ -688,7 +703,7 @@ class TestEvaluationConfigsApi(APIBaseTest):evaluation_type="hog",evaluation_config={"source": "return true"},output_type="boolean",- output_config={"allows_na": True, "true_is_failure": True},+ output_config={"allows_na": True, "true_is_failure": True, "decision_threshold": 0.35},created_by=self.user,)@@ -701,6 +716,22 @@ class TestEvaluationConfigsApi(APIBaseTest):self.assertEqual(response.status_code, status.HTTP_200_OK)self.assertEqual(response.json()["output_config"], expected)+ def test_rejects_invalid_boolean_decision_threshold(self) -> None:+ response = self.client.post(+ f"/api/environments/{self.team.id}/evaluations/",+ {+ "name": "Invalid threshold",+ "evaluation_type": "hog",+ "evaluation_config": {"source": "return true"},+ "output_type": "boolean",+ "output_config": {"decision_threshold": 1.1},+ "conditions": [{"id": "test-condition", "rollout_percentage": 100, "properties": []}],+ },+ )+ self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)+ self.assertEqual(response.data["attr"], "config")+ self.assertIn("decision_threshold", response.data["detail"])+def test_target_defaults_to_generation(self):response = self.client.post(f"/api/environments/{self.team.id}/evaluations/",diff --git a/products/ai_observability/backend/models/evaluation_configs.py b/products/ai_observability/backend/models/evaluation_configs.pyindex 86d374bc89c1..2af1f2c76b11 100644--- a/products/ai_observability/backend/models/evaluation_configs.py+++ b/products/ai_observability/backend/models/evaluation_configs.py@@ -63,6 +63,7 @@ class BooleanOutputConfig(BaseModel):# must be reported as a fail. Defaulting to False keeps stored configs written before this field# reading exactly as they did, with no backfill and no default to re-supply at each read site.true_is_failure: bool = False+ decision_threshold: float | None = Field(default=None, ge=0, le=1, strict=True, allow_inf_nan=False)class NumericPassingRule(BaseModel):diff --git a/products/ai_observability/backend/models/test/test_evaluation_configs.py b/products/ai_observability/backend/models/test/test_evaluation_configs.pyindex 3c8fb5a8cb2a..b6e9d45b7373 100644--- a/products/ai_observability/backend/models/test/test_evaluation_configs.py+++ b/products/ai_observability/backend/models/test/test_evaluation_configs.py@@ -11,6 +11,25 @@ from products.ai_observability.backend.models.evaluation_configs import ()+class TestBooleanOutputConfig:+ @pytest.mark.parametrize("threshold", [None, 0, 0.35, 1])+ def test_decision_threshold(self, threshold: float | None) -> None:+ _, output = validate_evaluation_configs(+ "llm_judge", "boolean", {"prompt": "Is the response correct?"}, {"decision_threshold": threshold}+ )+ expected: dict[str, bool | float] = {"allows_na": False, "true_is_failure": False}+ if threshold is not None:+ expected["decision_threshold"] = threshold+ assert output == expected++ @pytest.mark.parametrize("threshold", [-0.01, 1.01, float("nan"), float("inf"), True, "0.5"])+ def test_invalid_decision_threshold(self, threshold: object) -> None:+ with pytest.raises(ValueError):+ validate_evaluation_configs(+ "llm_judge", "boolean", {"prompt": "Is the response correct?"}, {"decision_threshold": threshold}+ )++class TestNumericOutputConfig:@pytest.mark.parametrize("bounds,score,expected",diff --git a/products/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsx b/products/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsxindex 26f305203c50..dad46be4d9c5 100644--- a/products/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsx+++ b/products/ai_observability/frontend/evaluations/AIObservabilityEvaluation.tsx@@ -47,7 +47,12 @@ import { EvaluationReportsTab } from './components/EvaluationReportsTab'import { EvaluationRunsTable } from './components/EvaluationRunsTable'import { EvaluationTriggers } from './components/EvaluationTriggers'import { NumericEvaluationConfig } from './components/NumericEvaluationConfig'-import { formatNumericEvaluationScore, numericOutputConfigError, categoricalOutputConfigError } from './constants'+import {+ formatNumericEvaluationScore,+ booleanOutputConfigError,+ numericOutputConfigError,+ categoricalOutputConfigError,+} from './constants'import {evaluationOffersSessionTarget,evaluationSupportsReportHistory,@@ -180,7 +185,7 @@ export function AIObservabilityEvaluation(): JSX.Element {? (categoricalOutputConfigError(evaluation.output_config) ?? undefined): evaluation.output_type === 'numeric'? (numericOutputConfigError(evaluation.output_config, numericBoundsRequired) ?? undefined)- : undefined+ : (booleanOutputConfigError(evaluation.output_config) ?? undefined)const focusTriggers = (): void => {setActiveTab('configuration')diff --git a/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.tsxindex a3f7d3bd8769..6b16f8d7d15f 100644--- a/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.tsx+++ b/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.tsx@@ -1,6 +1,6 @@import { useActions, useValues } from 'kea'-import { LemonSegmentedButton } from '@posthog/lemon-ui'+import { LemonInput, LemonSegmentedButton } from '@posthog/lemon-ui'import { FEATURE_FLAGS } from 'lib/constants'import { LemonField } from 'lib/lemon-ui/LemonField'@@ -9,6 +9,7 @@ import { featureFlagLogic } from 'lib/logic/featureFlagLogic'import { ByokModelPickerNotice } from '../../ByokModelPickerNotice'import { getModelPickerFooterLink, ModelPicker } from '../../ModelPicker'import { modelPickerLogic } from '../../modelPickerLogic'+import { booleanOutputConfigError } from '../constants'import { llmEvaluationLogic } from '../llmEvaluationLogic'import type { JudgeMethod } from '../types'@@ -23,7 +24,7 @@ export function EvaluationModelPicker(): JSX.Element {judgeMethod,judgeModelGroups: groups,} = useValues(llmEvaluationLogic)- const { selectModelFromPicker, setJudgeMethod } = useActions(llmEvaluationLogic)+ const { selectModelFromPicker, setJudgeMethod, patchOutputConfig } = useActions(llmEvaluationLogic)const { featureFlags } = useValues(featureFlagLogic)// Evals always run on the team's own provider key, so only BYOK models are offered.@@ -95,7 +96,7 @@ export function EvaluationModelPicker(): JSX.Element {? 'This decision model selects categories without written reasoning. For multiple selections, each category is included when its probability is 50% or higher.': evaluation.output_type === 'numeric'? 'This decision model estimates a score between your minimum and maximum without written reasoning. Define what low and high scores mean in your evaluation prompt. Scores can be fractional.'- : 'This decision model returns a probability without written reasoning. A probability of 50% or higher produces a true result.'}+ : 'This decision model returns a probability without written reasoning. The decision threshold determines whether the result is true.'}</p>)}{modelSelectionRequired && !selectedModel && (@@ -103,6 +104,37 @@ export function EvaluationModelPicker(): JSX.Element {)}</div></LemonField.Pure>+ {evaluation?.evaluation_type === 'llm_judge' &&+ evaluation.output_type === 'boolean' &&+ usesDecisionModel && (+ <LemonField.Pure+ label="Decision threshold"+ htmlFor="llma-evaluation-decision-threshold"+ error={booleanOutputConfigError(evaluation.output_config)}+ >+ <LemonInput+ id="llma-evaluation-decision-threshold"+ type="number"+ value={Number(+ ((evaluation.output_config.decision_threshold ?? 0.5) * 100).toPrecision(15)+ )}+ onChange={(percentage) =>+ patchOutputConfig({+ decision_threshold: percentage == null ? NaN : percentage / 100,+ })+ }+ min={0}+ max={100}+ step="any"+ suffix={<span>%</span>}+ data-attr="llma-evaluation-decision-threshold"+ />+ <p className="text-muted text-sm">+ A probability at or above this threshold produces a true result. Applies to new runs;+ past results stay unchanged.+ </p>+ </LemonField.Pure>+ )}</div></div>)diff --git a/products/ai_observability/frontend/evaluations/constants.ts b/products/ai_observability/frontend/evaluations/constants.tsindex 789b1f064c0e..2544eaeef2f9 100644--- a/products/ai_observability/frontend/evaluations/constants.ts+++ b/products/ai_observability/frontend/evaluations/constants.ts@@ -21,6 +21,18 @@ const EVALUATION_RESULT_FALSE_HOGQL = "properties.$ai_evaluation_result = 'falseexport const EVALUATION_NOT_SKIPPED_HOGQL ="(isNull(properties.$ai_evaluation_skipped) OR properties.$ai_evaluation_skipped != 'true')"+export function booleanOutputConfigError(config: EvaluationOutputConfig): string | null {+ const threshold = config.decision_threshold+ if (threshold != null && (!Number.isFinite(threshold) || threshold < 0 || threshold > 1)) {+ return 'Enter a decision threshold between 0% and 100%.'+ }+ return null+}++export function clearInvalidDecisionThreshold(config: EvaluationOutputConfig): EvaluationOutputConfig {+ return booleanOutputConfigError(config) ? { ...config, decision_threshold: null } : config+}+export function numericOutputConfigError(config: EvaluationOutputConfig, requiresBounds = false): string | null {const { min, max, step } = configconst passing_rule = config.passing_rule && 'threshold' in config.passing_rule ? config.passing_rule : nulldiff --git a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.tsindex 940b5bff05e8..5aaac3253909 100644--- a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts+++ b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts@@ -725,6 +725,72 @@ return result`,await expectLogic(logic).toMatchValues({ formValid: true })})+ it.each([+ [undefined, true],+ [0, true],+ [0.35, true],+ [1, true],+ [-0.01, false],+ [1.01, false],+ [NaN, false],+ [Infinity, false],+ ])('validates a decision threshold of %s as %s', async (threshold, valid) => {+ await expectLogic(logic).toDispatchActions(['loadEvaluationSuccess'])+ logic.actions.loadEvaluationSuccess({+ ...mockEvaluation,+ output_config: {},+ model_configuration: { provider: 'system_one', model: 'custom-model', provider_key_id: 'key-1' },+ })+ logic.actions.patchOutputConfig({ decision_threshold: threshold })+ expect(logic.values.evaluation?.output_config.decision_threshold).toBe(threshold)+ expect(logic.values.formValid).toBe(valid)+ })++ it.each([+ ['llm', NaN, null],+ ['llm', 1.5, null],+ ['hog', NaN, null],+ ['hog', 1.5, null],+ ['llm', 0, 0],+ ['llm', 0.35, 0.35],+ ['hog', 0, 0],+ ['hog', 0.35, 0.35],+ ] as const)(+ 'allows saving after leaving decision judging for %s with threshold %s',+ async (method, threshold, expectedThreshold) => {+ await expectLogic(logic).toDispatchActions(['loadEvaluationSuccess'])+ logic.actions.loadEvaluationSuccess({+ ...mockEvaluation,+ evaluation_config: { prompt: 'Check the response.', judge_method: 'decision' },+ output_config: { allows_na: true, true_is_failure: true },+ model_configuration: {+ provider: 'system_one',+ model: 'custom-model',+ provider_key_id: 'key-1',+ },+ })+ logic.actions.patchOutputConfig({ decision_threshold: threshold })++ if (method === 'llm') {+ await expectLogic(logic, () => logic.actions.setJudgeMethod('llm')).toFinishAllListeners()+ logic.actions.setModelConfiguration({+ provider: 'openai',+ model: 'gpt-5-mini',+ provider_key_id: 'key-1',+ })+ } else {+ logic.actions.setEvaluationType('hog')+ }++ expect(logic.values.evaluation?.output_config).toEqual({+ allows_na: method === 'llm',+ true_is_failure: true,+ decision_threshold: expectedThreshold,+ })+ expect(logic.values.formValid).toBe(true)+ }+ )+it.each([['system_one', false, true],['openrouter', true, true],diff --git a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.tsindex 27889b4d4357..ae6c0fa469b1 100644--- a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts+++ b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts@@ -37,6 +37,8 @@ import {evaluationPassedHogQL,evaluationPassRateHogQL,evaluationIsDetector,+ booleanOutputConfigError,+ clearInvalidDecisionThreshold,numericOutputConfigError,numericScorePasses,categoricalResultPasses,@@ -187,7 +189,7 @@ function toHogEvaluation(evaluation: EvaluationConfig): HogEvaluation {model_configuration: null,output_config: ['numeric', 'categorical'].includes(evaluation.output_type)? evaluation.output_config- : { ...evaluation.output_config, allows_na: false },+ : { ...clearInvalidDecisionThreshold(evaluation.output_config), allows_na: false },}}@@ -938,7 +940,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([return { ...state, output_type: outputType, output_config }},patchOutputConfig: (state, { patch }) => {- if (state?.output_type !== 'numeric' && state?.output_type !== 'categorical') {+ if (!state || state.output_type === 'sentiment') {return state}const output_config = { ...state.output_config, ...patch }@@ -976,7 +978,14 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([: state,setJudgeMethod: (state, { judgeMethod }) =>state && isLLMJudgeEvaluation(state)- ? { ...state, evaluation_config: { ...state.evaluation_config, judge_method: judgeMethod } }+ ? {+ ...state,+ evaluation_config: { ...state.evaluation_config, judge_method: judgeMethod },+ output_config:+ judgeMethod === 'llm'+ ? clearInvalidDecisionThreshold(state.output_config)+ : state.output_config,+ }: state,setEvaluationType: (state, { evaluationType }) => {if (!state) {@@ -1688,6 +1697,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([return false}if (+ (evaluation.output_type === 'boolean' && booleanOutputConfigError(evaluation.output_config)) ||(evaluation.output_type === 'numeric' &&numericOutputConfigError(evaluation.output_config, numericBoundsRequired)) ||(evaluation.output_type === 'categorical' && categoricalOutputConfigError(evaluation.output_config))diff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.tsindex 5265a0cb9272..353e29ca7293 100644--- a/products/ai_observability/frontend/generated/api.schemas.ts+++ b/products/ai_observability/frontend/generated/api.schemas.ts@@ -2123,13 +2123,20 @@ export type EvaluationApiOutputConfigPassingRule =| null/**- * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.+ * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.*/export type EvaluationApiOutputConfig = {/** Whether the evaluation can return N/A for non-applicable generations. */allows_na?: boolean/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullable@@ -2226,7 +2233,7 @@ export interface EvaluationApi {* * `categorical` - Categorical* * `sentiment` - Sentiment */output_type: OutputTypeEnumApi- /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */+ /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */output_config?: EvaluationApiOutputConfig/** Trigger conditions that filter which events are evaluated. OR between condition sets, AND within each. Each set is {id, rollout_percentage, properties[]} — `rollout_percentage` (0-100, defaults to 100) is the sampling field the dispatcher reads. */conditions?: EvaluationConditionApi[]@@ -2456,13 +2463,20 @@ export type PatchedEvaluationApiOutputConfigPassingRule =| null/**- * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.+ * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.*/export type PatchedEvaluationApiOutputConfig = {/** Whether the evaluation can return N/A for non-applicable generations. */allows_na?: boolean/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullable@@ -2570,7 +2584,7 @@ export interface PatchedEvaluationApi {* * `categorical` - Categorical* * `sentiment` - Sentiment */output_type?: OutputTypeEnumApi- /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */+ /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */output_config?: PatchedEvaluationApiOutputConfig/** Trigger conditions that filter which events are evaluated. OR between condition sets, AND within each. Each set is {id, rollout_percentage, properties[]} — `rollout_percentage` (0-100, defaults to 100) is the sampling field the dispatcher reads. */conditions?: EvaluationConditionApi[]@@ -2648,6 +2662,13 @@ export type TestHogRequestApiOutputConfig = {allows_na?: boolean/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullable@@ -3380,6 +3401,13 @@ export type EvaluationReportMetricsApiOutputConfig = {allows_na?: boolean/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullablediff --git a/products/ai_observability/frontend/generated/api.zod.ts b/products/ai_observability/frontend/generated/api.zod.tsindex 4f7d9c32beff..8c722c0ee8dc 100644--- a/products/ai_observability/frontend/generated/api.zod.ts+++ b/products/ai_observability/frontend/generated/api.zod.ts@@ -543,6 +543,9 @@ export const EvaluationRunsCreateBody = /* @__PURE__ */ zod.object({export const evaluationsCreateBodyNameMax = 400export const evaluationsCreateBodyEvaluationConfigThreeSourceDefault = `user_messages`+export const evaluationsCreateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsCreateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsCreateBodyOutputConfigStepExclusiveMin = 0export const evaluationsCreateBodyOutputConfigOptionsItemKeyMax = 128@@ -640,6 +643,14 @@ export const EvaluationsCreateBody = /* @__PURE__ */ zod.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsCreateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsCreateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()@@ -711,7 +722,7 @@ export const EvaluationsCreateBody = /* @__PURE__ */ zod}).optional().describe(- "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."+ "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."),conditions: zod.array(@@ -927,6 +938,9 @@ export const EvaluationsBackfillsEstimateCreateBody = /* @__PURE__ */ zod.objectexport const evaluationsUpdateBodyNameMax = 400export const evaluationsUpdateBodyEvaluationConfigThreeSourceDefault = `user_messages`+export const evaluationsUpdateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsUpdateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsUpdateBodyOutputConfigStepExclusiveMin = 0export const evaluationsUpdateBodyOutputConfigOptionsItemKeyMax = 128@@ -1024,6 +1038,14 @@ export const EvaluationsUpdateBody = /* @__PURE__ */ zod.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsUpdateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsUpdateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()@@ -1095,7 +1117,7 @@ export const EvaluationsUpdateBody = /* @__PURE__ */ zod}).optional().describe(- "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."+ "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."),conditions: zod.array(@@ -1213,6 +1235,9 @@ export const EvaluationsUpdateBody = /* @__PURE__ */ zodexport const evaluationsPartialUpdateBodyNameMax = 400export const evaluationsPartialUpdateBodyEvaluationConfigThreeSourceDefault = `user_messages`+export const evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsPartialUpdateBodyOutputConfigStepExclusiveMin = 0export const evaluationsPartialUpdateBodyOutputConfigOptionsItemKeyMax = 128@@ -1312,6 +1337,14 @@ export const EvaluationsPartialUpdateBody = /* @__PURE__ */ zod.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()@@ -1383,7 +1416,7 @@ export const EvaluationsPartialUpdateBody = /* @__PURE__ */ zod}).optional().describe(- "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."+ "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."),conditions: zod.array(@@ -1502,6 +1535,9 @@ export const EvaluationsPartialUpdateBody = /* @__PURE__ */ zod* Test Hog evaluation code against sample events without saving.*/export const evaluationsTestHogCreateBodyOutputTypeDefault = `boolean`+export const evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsTestHogCreateBodyOutputConfigStepExclusiveMin = 0export const evaluationsTestHogCreateBodyOutputConfigOptionsItemKeyMax = 128@@ -1544,6 +1580,14 @@ export const EvaluationsTestHogCreateBody = /* @__PURE__ */ zod.object({.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.tsindex 1f1d89da1f0d..1ef73b1610f1 100644--- a/products/desktop/packages/api-client/src/generated.ts+++ b/products/desktop/packages/api-client/src/generated.ts@@ -9068,6 +9068,10 @@ export namespace Schemas {* Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.*/true_is_failure: boolean;+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ */+ decision_threshold: number | null;/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.*/@@ -19194,6 +19198,7 @@ export namespace Schemas {output_config: Partial<{allows_na: boolean;true_is_failure: boolean;+ decision_threshold: number | null;min: number | null;max: number | null;step: number | null;diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 2d8f8abd1638..8e8278397658 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -42719,13 +42719,20 @@ export namespace Schemas {} | null;/**- * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.+ * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.*/export type EvaluationOutputConfig = {/** Whether the evaluation can return N/A for non-applicable generations. */allows_na?: boolean;/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean;+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null;/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullable@@ -42984,7 +42991,7 @@ export namespace Schemas {* * `categorical` - Categorical* * `sentiment` - Sentiment */output_type: OutputTypeEnum;- /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */+ /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */output_config?: EvaluationOutputConfig;/** Trigger conditions that filter which events are evaluated. OR between condition sets, AND within each. Each set is {id, rollout_percentage, properties[]} — `rollout_percentage` (0-100, defaults to 100) is the sampling field the dispatcher reads. */conditions?: EvaluationCondition[];@@ -43376,6 +43383,13 @@ export namespace Schemas {allows_na?: boolean;/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean;+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null;/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullable@@ -83029,13 +83043,20 @@ export namespace Schemas {} | null;/**- * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.+ * Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.*/export type PatchedEvaluationOutputConfig = {/** Whether the evaluation can return N/A for non-applicable generations. */allows_na?: boolean;/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean;+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null;/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullable@@ -83143,7 +83164,7 @@ export namespace Schemas {* * `categorical` - Categorical* * `sentiment` - Sentiment */output_type?: OutputTypeEnum;- /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */+ /** Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}. */output_config?: PatchedEvaluationOutputConfig;/** Trigger conditions that filter which events are evaluated. OR between condition sets, AND within each. Each set is {id, rollout_percentage, properties[]} — `rollout_percentage` (0-100, defaults to 100) is the sampling field the dispatcher reads. */conditions?: EvaluationCondition[];@@ -115633,6 +115654,13 @@ export namespace Schemas {allows_na?: boolean;/** Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail. */true_is_failure?: boolean;+ /**+ * Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.+ * @minimum 0+ * @maximum 1+ * @nullable+ */+ decision_threshold?: number | null;/*** Inclusive minimum numeric score. Omit for no lower bound. Required for numeric decision models.* @nullablediff --git a/services/mcp/src/generated/ai_observability/api.ts b/services/mcp/src/generated/ai_observability/api.tsindex 2ece2aece355..2185af4720d5 100644--- a/services/mcp/src/generated/ai_observability/api.ts+++ b/services/mcp/src/generated/ai_observability/api.ts@@ -1364,6 +1364,9 @@ export const EvaluationsCreateParams = () => zod.object({export const evaluationsCreateBodyNameMax = 400export const evaluationsCreateBodyEvaluationConfigThreeSourceDefault = `user_messages`+export const evaluationsCreateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsCreateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsCreateBodyOutputConfigStepExclusiveMin = 0export const evaluationsCreateBodyOutputConfigOptionsItemKeyMax = 128@@ -1461,6 +1464,14 @@ export const EvaluationsCreateBody = () => zod.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsCreateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsCreateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()@@ -1532,7 +1543,7 @@ export const EvaluationsCreateBody = () => zod}).optional().describe(- "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."+ "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."),conditions: zod.array(@@ -1827,6 +1838,9 @@ export const EvaluationsPartialUpdateParams = () => zod.object({export const evaluationsPartialUpdateBodyNameMax = 400export const evaluationsPartialUpdateBodyEvaluationConfigThreeSourceDefault = `user_messages`+export const evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsPartialUpdateBodyOutputConfigStepExclusiveMin = 0export const evaluationsPartialUpdateBodyOutputConfigOptionsItemKeyMax = 128@@ -1926,6 +1940,14 @@ export const EvaluationsPartialUpdateBody = () => zod.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsPartialUpdateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()@@ -1997,7 +2019,7 @@ export const EvaluationsPartialUpdateBody = () => zod}).optional().describe(- "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."+ "Output config. For 'boolean' output_type: {allows_na} to permit N\/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min\/max\/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}."),conditions: zod.array(@@ -2136,6 +2158,9 @@ export const EvaluationsTestHogCreateParams = () => zod.object({})export const evaluationsTestHogCreateBodyOutputTypeDefault = `boolean`+export const evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMin = 0+export const evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMax = 1+export const evaluationsTestHogCreateBodyOutputConfigStepExclusiveMin = 0export const evaluationsTestHogCreateBodyOutputConfigOptionsItemKeyMax = 128@@ -2178,6 +2203,14 @@ export const EvaluationsTestHogCreateBody = () => zod.object({.describe('Boolean output only. Omit for numeric, categorical, and sentiment output. Whether a true result means the evaluation found a problem. False (the default) suits pass\/fail evaluations, where a true result satisfied the criteria. Set it to true for detector-style evaluations, so a true result is counted and labeled as a fail.'),+ decision_threshold: zod+ .number()+ .min(evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMin)+ .max(evaluationsTestHogCreateBodyOutputConfigDecisionThresholdMax)+ .nullish()+ .describe(+ 'Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged.'+ ),min: zod.number().nullish()diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.jsonindex df93b06a08ce..7f94872293d0 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json@@ -176,12 +176,25 @@"type": "string"},"output_config": {- "description": "Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.",+ "description": "Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.","properties": {"allows_na": {"description": "Whether the evaluation can return N/A for non-applicable generations.","type": "boolean"},+ "decision_threshold": {+ "anyOf": [+ {+ "maximum": 1,+ "minimum": 0,+ "type": "number"+ },+ {+ "type": "null"+ }+ ],+ "description": "Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged."+ },"max": {"anyOf": [{diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-test-hog.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-test-hog.jsonindex 655cbd92d645..7a515f9b5f4b 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-test-hog.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-test-hog.json@@ -24,6 +24,19 @@"description": "Whether the evaluation can return N/A for non-applicable generations.","type": "boolean"},+ "decision_threshold": {+ "anyOf": [+ {+ "maximum": 1,+ "minimum": 0,+ "type": "number"+ },+ {+ "type": "null"+ }+ ],+ "description": "Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged."+ },"max": {"anyOf": [{diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.jsonindex 87f2a69ba186..5537211a1165 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json@@ -179,12 +179,25 @@"type": "string"},"output_config": {- "description": "Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.",+ "description": "Output config. For 'boolean' output_type: {allows_na} to permit N/A results, and {true_is_failure} to declare that a true result means the evaluation found a problem. For boolean decision-model judges, decision_threshold is a probability from 0 to 1 (default 0.5). For 'numeric': only min/max/step, allows_na, and passing_rule {operator: 'gte'|'lte', threshold}. For 'categorical': options [{key, label}], selection_mode (single or multiple), allows_na, and optional passing_rule {categories: [key]}. Do not send true_is_failure for numeric or categorical output. For 'sentiment': {}.","properties": {"allows_na": {"description": "Whether the evaluation can return N/A for non-applicable generations.","type": "boolean"},+ "decision_threshold": {+ "anyOf": [+ {+ "maximum": 1,+ "minimum": 0,+ "type": "number"+ },+ {+ "type": "null"+ }+ ],+ "description": "Boolean decision-model judges only. Minimum probability of true required for a true result, inclusive. Omit or set null to use 0.5. Applies to new runs; historical results are unchanged."+ },"max": {"anyOf": [{