:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): let llm judges set a reasoning level (#125840)
Bernat Torresd2a878248b70
diff --git a/docs/internal/ai-observability-judge-inputs.md b/docs/internal/ai-observability-judge-inputs.mdindex 617e39a89340..12b24733e73b 100644--- a/docs/internal/ai-observability-judge-inputs.md+++ b/docs/internal/ai-observability-judge-inputs.md@@ -180,6 +180,18 @@ Invalid probabilities, missing answers, and mismatched answer types skip the iteInputs rejected for exceeding the model's context window are skipped.See OpenAI's [Decisions guide](https://developers.openai.com/api/docs/guides/decisions), OpenRouter's [Decisions API reference](https://openrouter.ai/docs/api/api-reference/alphadecisions/submit-a-decisions-request), and TypeSafe's [API reference](https://docs.typesafe.ai/api) for their question and answer formats.+## Reasoning level++LLM judges on Anthropic and OpenAI reasoning models can set a reasoning level with the brain button next to the model selector.+The level is stored as `evaluation_config.reasoning_level` (`minimal`, `low`, `medium` or `high`). Without it, the judge uses the model's default.+Decision models and other providers do not support it, and the API rejects a level for them.+Prompt-only `evaluation_config` PATCH requests preserve the saved level; an explicit `reasoning_level: null` restores the model default.+Selecting a model without reasoning support clears the level in the editor.++Anthropic models before Claude 4.6 get a thinking budget of 1,024, 2,048, 4,096 or 8,192 tokens. Newer models get adaptive thinking with that effort, and `minimal` becomes `low`.+OpenAI models get `reasoning_effort`. Only the original GPT-5 models accept `minimal`; other models get `low` instead.+For Anthropic, the token limit of the reply grows by the thinking budget of the level, so the verdict still fits after the thinking.+## Model output limitsWhen the judge reply reaches the model's output limit, the evaluation skips that item with `output_limit_exceeded`.diff --git a/frontend/snapshots.yml b/frontend/snapshots.ymlindex 50d93b4c7e33..3975b426b985 100644--- a/frontend/snapshots.yml+++ b/frontend/snapshots.yml@@ -13443,9 +13443,9 @@ snapshots:scenes-app-llm-observability-evaluationmodelpicker--decision--light:hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.CM-LkhqTNhtSyn0O3Y6c6Yd_gLfSXe31n3Dxaj5uEZIscenes-app-llm-observability-evaluationmodelpicker--decision-narrow--dark:- hash: v1.k794b7964.8cf03ee1f25ccdee62cc5104d48589d196f85dd3dbad84d5a838cc4c8bfe9607.W_9QnaVZtdekweLtoPuF32I_r0G0fCdjRWUVyBg3kEI+ hash: v1.k794b7964.e75b3c58f2b1922e7ce4b3f8512fa7dc81021d0d988622599b3f52bb40f32eeb.PLEVbwWEAXbpJdC_PSjXyTKP-lLKLDdAkRcrP9jOOXUscenes-app-llm-observability-evaluationmodelpicker--decision-narrow--light:- hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.HuCQ8PyBI05CTQEjr38moKf85RyyPvF9SCxuwytomXQ+ hash: v1.k794b7964.8ae8ae696fb4cd22a44ccd18c901236eea8a0e6ac6273b686dc946821f3f7962.Dw7EaOMcP1cHXUw3o5SoYiAZ66QGAYk7WzhqDGcCrLYscenes-app-llm-observability-evaluationmodelpicker--llm--dark:hash: v1.k794b7964.a42ccc41fa5864dee67f21add5ecc3aa6838805fb494f6686b826688cc325e9e.tHDxeEbDTVatdZYPj2xVDQtJpukqzNyUHU9iScfQ2AYscenes-app-llm-observability-evaluationmodelpicker--llm--light:@@ -13455,13 +13455,17 @@ snapshots:scenes-app-llm-observability-evaluationmodelpicker--native-open-ai--light:hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.nTYWu7M1TfGyOlu52uGxd98cNvdj-nyoyzcWUovs6Ysscenes-app-llm-observability-evaluationmodelpicker--native-open-ai-narrow--dark:- hash: v1.k794b7964.8cf03ee1f25ccdee62cc5104d48589d196f85dd3dbad84d5a838cc4c8bfe9607.gZEs23POnv-eBuhQT-V2gxTuoBiA7R32dhshLPHHyVs+ hash: v1.k794b7964.e75b3c58f2b1922e7ce4b3f8512fa7dc81021d0d988622599b3f52bb40f32eeb.qdYv6zfUkJUHXikWSwr_1jyDM3PvnFOPTj8qFIdrqr8scenes-app-llm-observability-evaluationmodelpicker--native-open-ai-narrow--light:- hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.DO5O6V3lvkGYD6JJ5RfEBG2IbsSvb0aNeCME9dvKEWk+ hash: v1.k794b7964.8ae8ae696fb4cd22a44ccd18c901236eea8a0e6ac6273b686dc946821f3f7962._qdgUVltz2sd1ZHnkaFq8J5CXje346MVRlc4WOsz-pgscenes-app-llm-observability-evaluationmodelpicker--no-decision-models--dark:hash: v1.k794b7964.ed596389ff3b9a8a505c3b1ad5cbfabadc1cb83507c341f99dd54b004f8d257e.Jm_sWi9-7s4eTkxjyQUcGApiT5QcFzke8u5k2hK6gaEscenes-app-llm-observability-evaluationmodelpicker--no-decision-models--light:hash: v1.k794b7964.b10b433ed0507465953ef6174a8872546e4e695407914798a8e11a6e7f8aa38d.jQ8Xqfw0JjPdrTXWnuYodzsfn46fs4aKri7d1Pzl7QU+ scenes-app-llm-observability-evaluationmodelpicker--reasoning-level-narrow--dark:+ hash: v1.k794b7964.c135d25cde9ab12d1bbf35f7cc6cc87ab8c838586bdbd7316b62a59d54d85b9e.QVUXtySXrLSrHHvMht-mZvR9AIKvry19dFXdhLIHs30+ scenes-app-llm-observability-evaluationmodelpicker--reasoning-level-narrow--light:+ hash: v1.k794b7964.d0372da3513c43cccbb1d2c7e9b6212567f86ca3e20f33cfaf0fba84c2eb8072.1Lrr0sxzbqQN9iCbO_JrkZtixXdNV2MOgqu6TI5qjDwscenes-app-llm-observability-evaluationreportconfig--lemon-ui--dark:hash: v1.k794b7964.195aa4a5a39e8f947e17fe67f5c23303b7dd0579c278907761be24a08780be92.IkGQtAI95T60kJKVZcmmFBIWNCd4meuJv-bcNDoohwUscenes-app-llm-observability-evaluationreportconfig--lemon-ui--light:diff --git a/posthog/temporal/ai_observability/evaluation_llm_judge.py b/posthog/temporal/ai_observability/evaluation_llm_judge.pyindex 0825d945d7b0..312383fed0c0 100644--- a/posthog/temporal/ai_observability/evaluation_llm_judge.py+++ b/posthog/temporal/ai_observability/evaluation_llm_judge.py@@ -828,6 +828,7 @@ def call_llm_judge(messages=[{"role": "user", "content": user_prompt}],provider=provider,response_format=response_format,+ reasoning_level=evaluation["evaluation_config"].get("reasoning_level"),))except (DecisionEndpointBlockedError, ProviderConfigurationError) as e:diff --git a/posthog/temporal/ai_observability/test_run_evaluation.py b/posthog/temporal/ai_observability/test_run_evaluation.pyindex a1210eb83260..3366a68f9844 100644--- a/posthog/temporal/ai_observability/test_run_evaluation.py+++ b/posthog/temporal/ai_observability/test_run_evaluation.py@@ -373,6 +373,7 @@ def test_legacy_and_explicit_llm_methods_use_chat_for_a_dual_capability_model("team_id": 1,"evaluation_config": {"prompt": "Check the response.",+ "reasoning_level": "high",**({"judge_method": judge_method} if judge_method else {}),},},@@ -381,6 +382,7 @@ def test_legacy_and_explicit_llm_methods_use_chat_for_a_dual_capability_model(allows_na=False,)complete.assert_called_once()+ assert complete.call_args.args[0].reasoning_level == "high"decide.assert_not_called()native_decide.assert_not_called()assert result["verdict"] is Truediff --git a/products/ai_observability/backend/api/evaluations.py b/products/ai_observability/backend/api/evaluations.pyindex 940f9589e4fe..7798809a85bf 100644--- a/products/ai_observability/backend/api/evaluations.py+++ b/products/ai_observability/backend/api/evaluations.py@@ -58,7 +58,7 @@ from ..facade import ()from ..facade.contracts import Contractfrom ..hog import compile_ai_observability_hog-from ..llm import DEFAULT_MODEL_BY_PROVIDER+from ..llm import DEFAULT_MODEL_BY_PROVIDER, supports_reasoning_levelfrom ..llm.decisions import is_decision_modelfrom ..llm.errors import ProviderConnectionErrorfrom ..llm.providers.openai import OpenAIConfig@@ -126,6 +126,16 @@ logger = structlog.get_logger(__name__)"nullable": True,"description": "Judge method. Omit to keep the saved method; null restores automatic routing.",},+ "reasoning_level": {+ "type": "string",+ "enum": ["minimal", "low", "medium", "high"],+ "nullable": True,+ "description": (+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI "+ "reasoning models with the 'llm' judge method support it. Omit to keep the saved "+ "level; null uses the model's default."+ ),+ },},"additionalProperties": False,},@@ -591,15 +601,14 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSeconfig = data["evaluation_config"]if not isinstance(config, dict):raise serializers.ValidationError({"evaluation_config": "Must be an object."})- if (- self.partial- and self.instance is not None- and evaluation_type == EvaluationType.LLM_JUDGE.value- and "judge_method" not in config- and "judge_method" in self.instance.evaluation_config- ):- # A prompt-only PATCH must not reset the saved routing choice.- data["evaluation_config"] = {"judge_method": self.instance.evaluation_config["judge_method"], **config}+ if self.partial and self.instance is not None and evaluation_type == EvaluationType.LLM_JUDGE.value:+ # A prompt-only PATCH must not reset the saved judge settings.+ saved = {+ key: self.instance.evaluation_config[key]+ for key in ("judge_method", "reasoning_level")+ if key not in config and key in self.instance.evaluation_config+ }+ data["evaluation_config"] = {**saved, **config}if (self.instanceand output_type != self.instance.output_type@@ -639,6 +648,7 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSe).get("judge_method")if data.get("model_configuration") or data.get("enabled") or method_changed:self._validate_chat_model(data, uses_decision_model=uses_decision_model)+ self._validate_reasoning_level(data, uses_decision_model=uses_decision_model)should_validate_configs = (self.instance is None@@ -742,6 +752,21 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSe})+ def _validate_reasoning_level(self, data: dict, *, uses_decision_model: bool) -> None:+ evaluation_type = data.get("evaluation_type") or getattr(self.instance, "evaluation_type", None)+ if evaluation_type != EvaluationType.LLM_JUDGE.value:+ return+ evaluation_config = data.get("evaluation_config", getattr(self.instance, "evaluation_config", None))+ if not isinstance(evaluation_config, dict) or not evaluation_config.get("reasoning_level"):+ return+ model_config = self._effective_model_configuration(data) or {}+ if uses_decision_model or not supports_reasoning_level(model_config.get("provider"), model_config.get("model")):+ raise serializers.ValidationError(+ {+ "evaluation_config": "This judge model doesn't support a reasoning level. Choose an Anthropic or OpenAI reasoning model, or use the model default."+ }+ )+def _validate_can_run(self, data: dict) -> None:"""An eval being turned on — created enabled or re-enabled — must be able to resolve amodel at runtime; otherwise the caller gets a 200/201 and the first Temporal run silentlydiff --git a/products/ai_observability/backend/api/proxy.py b/products/ai_observability/backend/api/proxy.pyindex fd2d2678fc1f..993922756821 100644--- a/products/ai_observability/backend/api/proxy.py+++ b/products/ai_observability/backend/api/proxy.py@@ -56,6 +56,7 @@ from products.ai_observability.backend.llm import (ModelInfo,OutputSchema,get_playground_models,+ supports_reasoning_level,)from products.ai_observability.backend.llm.errors import (ProviderConfigurationError,@@ -147,6 +148,9 @@ class LLMProxyModelInfoSerializer(serializers.Serializer):supports_chat = serializers.BooleanField(default=True, help_text="Whether this model supports chat completions with written reasoning.")+ supports_reasoning = serializers.BooleanField(+ default=False, help_text="Whether an evaluation judge can set a reasoning level for this model."+ )class PlaygroundAccessPermission(BasePermission):@@ -568,6 +572,7 @@ class LLMProxyViewSet(viewsets.ViewSet):is_recommended=m in recommended,supports_decisions=provider_key.provider == LLMProvider.SYSTEM_ONE or m in decision_models,supports_chat=provider_key.provider != LLMProvider.SYSTEM_ONE and m not in non_chat_models,+ supports_reasoning=supports_reasoning_level(provider_key.provider, m),)for m in models]diff --git a/products/ai_observability/backend/api/test/test_evaluations.py b/products/ai_observability/backend/api/test/test_evaluations.pyindex 5fcc123c9561..8d155c62fc1b 100644--- a/products/ai_observability/backend/api/test/test_evaluations.py+++ b/products/ai_observability/backend/api/test/test_evaluations.py@@ -1,5 +1,6 @@import jsonfrom datetime import timedelta+from typing import Anyfrom uuid import UUID, uuid4from posthog.test.base import APIBaseTest, ClickhouseTestMixin, _create_event, flush_persons_and_events@@ -174,11 +175,53 @@ class TestModelConfigurationSerializer(SimpleTestCase):)self.assertEqual(evaluation.evaluation_config, stored_config)+ @parameterized.expand(+ [+ ("prompt_patch_keeps_level", {}, {"reasoning_level": "high"}),+ ("explicit_null_clears_level", {"evaluation_config": {"reasoning_level": None}}, {}),+ ("level_change", {"evaluation_config": {"reasoning_level": "minimal"}}, {"reasoning_level": "minimal"}),+ (+ "unsupported_model_rejected",+ {"model_configuration": {"provider": "gemini", "model": "gemini-2.5-flash"}},+ None,+ ),+ (+ "openai_reasoning_model_accepted",+ {"model_configuration": {"provider": "openai", "model": "gpt-5-mini"}},+ {"reasoning_level": "high"},+ ),+ ]+ )+ def test_reasoning_level_patch(+ self, _name: str, patch_data: dict[str, Any], expected_level: dict[str, str] | None+ ) -> None:+ evaluation = Evaluation(+ evaluation_type="llm_judge",+ evaluation_config={"prompt": "Check the response.", "reasoning_level": "high"},+ output_type="boolean",+ output_config={},+ model_configuration=LLMModelConfiguration(provider="anthropic", model="claude-haiku-5-5"),+ )+ data = {**patch_data}+ data["evaluation_config"] = {"prompt": "Check clarity.", **patch_data.get("evaluation_config", {})}+ serializer = EvaluationSerializer(+ instance=evaluation, data=data, partial=True, context={"get_team": lambda: Mock(id=1)}+ )+ # Avoid key resolution: this test checks the configuration boundary.+ with patch.object(serializer, "_validate_re_enable"), patch.object(serializer, "_validate_can_run"):+ valid = serializer.is_valid()+ if expected_level is None:+ self.assertFalse(valid)+ self.assertIn("evaluation_config", serializer.errors)+ return+ self.assertTrue(valid, serializer.errors)+ self.assertEqual(serializer.validated_data["evaluation_config"], {"prompt": "Check clarity.", **expected_level})+@parameterized.expand([("llm",), ("decision",)])def test_switch_to_sentiment_ignores_stored_judge_method(self, method: str) -> None:evaluation = Evaluation(evaluation_type="llm_judge",- evaluation_config={"prompt": "Check the response.", "judge_method": method},+ evaluation_config={"prompt": "Check the response.", "judge_method": method, "reasoning_level": "high"},output_type="boolean",output_config={},model_configuration=LLMModelConfiguration(provider="openrouter", model="example/model"),diff --git a/products/ai_observability/backend/llm/__init__.py b/products/ai_observability/backend/llm/__init__.pyindex a78e78948c70..fec2978e9f8e 100644--- a/products/ai_observability/backend/llm/__init__.py+++ b/products/ai_observability/backend/llm/__init__.py@@ -26,6 +26,7 @@ class ModelInfo(TypedDict):is_recommended: boolsupports_decisions: NotRequired[bool]supports_chat: NotRequired[bool]+ supports_reasoning: NotRequired[bool]# Single registry of providers. Add new providers here and everything else@@ -37,6 +38,18 @@ PROVIDERS: list[tuple[str, Any]] = []+# Providers whose `complete()` honors CompletionRequest.reasoning_level.+REASONING_LEVEL_CONFIGS: dict[str, type[AnthropicConfig] | type[OpenAIConfig]] = {+ "anthropic": AnthropicConfig,+ "openai": OpenAIConfig,+}+++def supports_reasoning_level(provider: str | None, model: str | None) -> bool:+ config = REASONING_LEVEL_CONFIGS.get(provider or "")+ return bool(config and model and config.supports_reasoning_level(model))++def _build_model_infos(provider: str, models: list[str]) -> list[ModelInfo]:return [{"id": m, "name": m, "provider": provider, "description": "", "is_recommended": True} for m in models]@@ -93,6 +106,7 @@ __all__ = ["DEFAULT_MODEL_BY_PROVIDER","get_default_models","get_playground_models",+ "supports_reasoning_level","OpenAIConfig","AnthropicConfig","GeminiConfig",diff --git a/products/ai_observability/backend/llm/providers/anthropic.py b/products/ai_observability/backend/llm/providers/anthropic.pyindex 83bec9ae74a7..306dbb9b0f97 100644--- a/products/ai_observability/backend/llm/providers/anthropic.py+++ b/products/ai_observability/backend/llm/providers/anthropic.py@@ -39,6 +39,7 @@ from products.ai_observability.backend.llm.types import (AnalyticsContext,CompletionRequest,CompletionResponse,+ ReasoningLevel,StreamChunk,Usage,)@@ -118,6 +119,20 @@ def _is_quota_or_billing_error(error: Exception) -> bool:class AnthropicConfig:MAX_TOKENS: int = 8192MAX_THINKING_TOKENS: int = 4096+ # Thinking tokens count against max_tokens, so each level also raises max_tokens by its budget.+ REASONING_BUDGET_TOKENS: dict[ReasoningLevel, int] = {"minimal": 1024, "low": 2048, "medium": 4096, "high": 8192}+ # Models before Claude 4.6 take a fixed thinking budget. Newer models reject it and take an effort level.+ BUDGET_THINKING_MODEL_PREFIXES: tuple[str, ...] = (+ "claude-opus-4-5",+ "claude-sonnet-4-5",+ "claude-haiku-4-5",+ "claude-opus-4-1",+ "claude-opus-4-0",+ "claude-sonnet-4-0",+ # Dated IDs of Claude 4.0, such as claude-sonnet-4-20250514.+ "claude-opus-4-20",+ "claude-sonnet-4-20",+ )# Timeout in seconds for API calls. Set high to accommodate slow reasoning models.# Note: Infrastructure-level timeouts (load balancers, proxies) may still limit actual request duration.TIMEOUT: float = 300.0@@ -170,6 +185,11 @@ class AnthropicConfig:"claude-sonnet-4-0",]+ @staticmethod+ def supports_reasoning_level(model: str) -> bool:+ # Every Claude model after the 3 family can think, including ones newer than our model lists.+ return model.startswith("claude-") and not model.startswith("claude-3")+class AnthropicAdapter:"""Anthropic provider implementing the unified Client interface."""@@ -237,6 +257,9 @@ class AnthropicAdapter:}}+ if request.reasoning_level and AnthropicConfig.supports_reasoning_level(request.model):+ self._apply_reasoning_level(create_kwargs, request.model, request.reasoning_level)+try:response = client.messages.create(**create_kwargs)@@ -273,6 +296,17 @@ class AnthropicAdapter:raise mapped from eraise+ @staticmethod+ def _apply_reasoning_level(create_kwargs: dict[str, Any], model: str, level: ReasoningLevel) -> None:+ budget = AnthropicConfig.REASONING_BUDGET_TOKENS[level]+ create_kwargs["max_tokens"] += budget+ if model.startswith(AnthropicConfig.BUDGET_THINKING_MODEL_PREFIXES):+ create_kwargs["thinking"] = {"type": "enabled", "budget_tokens": budget}+ return+ create_kwargs["thinking"] = {"type": "adaptive"}+ # Claude has no "minimal" effort level.+ create_kwargs.setdefault("output_config", {})["effort"] = "low" if level == "minimal" else level+def _mapped_error(self, error: Exception, model: str) -> LLMError | None:"""Normalize a provider exception into the shared taxonomy, or None when it isn't ours.diff --git a/products/ai_observability/backend/llm/providers/openai.py b/products/ai_observability/backend/llm/providers/openai.pyindex 898e2b853f88..5be139fd6a08 100644--- a/products/ai_observability/backend/llm/providers/openai.py+++ b/products/ai_observability/backend/llm/providers/openai.py@@ -11,6 +11,7 @@ from django.conf import settingsimport httpximport openaiimport posthoganalytics+from openai import omitfrom openai.types import CompletionUsage, ReasoningEffortfrom openai.types.chat import ChatCompletionDeveloperMessageParam, ChatCompletionSystemMessageParamfrom posthoganalytics.ai.openai import (@@ -39,6 +40,7 @@ from products.ai_observability.backend.llm.types import (AnalyticsContext,CompletionRequest,CompletionResponse,+ ReasoningLevel,StreamChunk,Usage,)@@ -109,6 +111,17 @@ class OpenAIConfig:"o3-mini",]+ @staticmethod+ def supports_reasoning_level(model: str) -> bool:+ return model.startswith(("gpt-5", "o3", "o4")) and "chat" not in model++ @staticmethod+ def reasoning_effort(model: str, level: ReasoningLevel) -> ReasoningEffort:+ # Only the original GPT-5 models accept "minimal". Later models and the o-series reject it.+ if level == "minimal" and not (model == "gpt-5" or model.startswith("gpt-5-")):+ return "low"+ return level+class OpenAIAdapter:"""OpenAI provider implementing the unified Client interface."""@@ -183,6 +196,11 @@ class OpenAIAdapter:model=request.model,messages=messages,response_format=request.response_format,+ reasoning_effort=(+ OpenAIConfig.reasoning_effort(request.model, request.reasoning_level)+ if request.reasoning_level and OpenAIConfig.supports_reasoning_level(request.model)+ else omit+ ),**(self._build_analytics_kwargs(analytics, client)),)parsed = response.choices[0].message.parsed@@ -380,7 +398,7 @@ Return ONLY the JSON object, no other text or markdown formatting."""if supports_reasoning:selected_effort: ReasoningEffort | None = Noneif request.reasoning_level in ("minimal", "low", "medium", "high"):- selected_effort = request.reasoning_level # type: ignore[assignment]+ selected_effort = request.reasoning_levelelif reasoning_on:selected_effort = OpenAIConfig.REASONING_EFFORTstream = client.chat.completions.create(diff --git a/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py b/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.pyindex 6a5c41d33003..a4fbe5b5ffbe 100644--- a/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py+++ b/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py@@ -102,6 +102,52 @@ class TestStructuredOutputComplete:assert "result" in schema["properties"]assert "reason" in schema["properties"]+ @parameterized.expand(+ [+ ("budget_model", "claude-haiku-4-5", "medium", {"type": "enabled", "budget_tokens": 4096}, None, 12288),+ (+ "dated_claude_4_0",+ "claude-sonnet-4-20250514",+ "low",+ {"type": "enabled", "budget_tokens": 2048},+ None,+ 10240,+ ),+ ("adaptive_model", "claude-haiku-5-5", "high", {"type": "adaptive"}, "high", 16384),+ ("adaptive_minimal_is_low", "claude-opus-4-7", "minimal", {"type": "adaptive"}, "low", 9216),+ ("unlisted_newer_model", "claude-sonnet-9", "low", {"type": "adaptive"}, "low", 10240),+ ("no_level", "claude-haiku-5-5", None, None, None, 8192),+ ("claude_3", "claude-3-haiku-20240307", "high", None, None, 8192),+ ]+ )+ @patch("products.ai_observability.backend.llm.providers.anthropic.settings")+ @patch("products.ai_observability.backend.llm.providers.anthropic.anthropic.Anthropic")+ def test_structured_output_applies_reasoning_level(+ self,+ _name,+ model,+ level,+ expected_thinking,+ expected_effort,+ expected_max_tokens,+ mock_anthropic_cls,+ mock_settings,+ ):+ mock_client = MagicMock()+ mock_anthropic_cls.return_value = mock_client+ mock_client.messages.create.return_value = self._make_mock_response('{"result": true, "reason": "ok"}')+ request = self._make_request(response_format=BooleanEvalResult)+ request.model = model+ request.reasoning_level = level++ AnthropicAdapter().complete(request, api_key="sk-ant-test", analytics=AnalyticsContext(capture=False))++ call_kwargs = mock_client.messages.create.call_args.kwargs+ assert call_kwargs.get("thinking") == expected_thinking+ assert call_kwargs["output_config"].get("effort") == expected_effort+ assert call_kwargs["output_config"]["format"]["type"] == "json_schema"+ assert call_kwargs["max_tokens"] == expected_max_tokens+@patch("products.ai_observability.backend.llm.providers.anthropic.settings")@patch("products.ai_observability.backend.llm.providers.anthropic.anthropic.Anthropic")def test_structured_output_parses_valid_json(self, mock_anthropic_cls, mock_settings):diff --git a/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py b/products/ai_observability/backend/llm/providers/test/test_openai_adapter.pyindex 1d77981e95d7..aa92919c8818 100644--- a/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py+++ b/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py@@ -177,6 +177,39 @@ class TestOpenAIStreamOutputSchema:assert "response_format" not in self._stream_kwargs(None)+class TestOpenAICompleteReasoningEffort:+ @parameterized.expand(+ [+ ("original_gpt5_keeps_minimal", "gpt-5-mini", "minimal", "minimal"),+ ("later_gpt5_maps_minimal_to_low", "gpt-5.1", "minimal", "low"),+ ("o_series_maps_minimal_to_low", "o4-mini", "minimal", "low"),+ ("level_passes_through", "gpt-5.1", "high", "high"),+ ("non_reasoning_model_omits_effort", "gpt-4.1", "high", None),+ ("chat_variant_omits_effort", "gpt-5-chat-latest", "high", None),+ ("no_level_omits_effort", "gpt-5-mini", None, None),+ ]+ )+ def test_structured_completion_sends_supported_reasoning_effort(+ self, _name: str, model: str, level: str | None, expected: str | None+ ) -> None:+ mock_client = MagicMock()+ mock_client.beta.chat.completions.parse.return_value.choices = [MagicMock(message=MagicMock(parsed=None))]+ request = CompletionRequest(+ model=model,+ system="s",+ messages=[{"role": "user", "content": "x"}],+ provider="openai",+ response_format=_Verdict,+ reasoning_level=level, # type: ignore[arg-type]+ )++ with patch("products.ai_observability.backend.llm.providers.openai.openai.OpenAI", return_value=mock_client):+ OpenAIAdapter().complete(request, api_key="sk-test", analytics=AnalyticsContext(capture=False))++ effort = mock_client.beta.chat.completions.parse.call_args.kwargs["reasoning_effort"]+ assert (None if effort is openai.omit else effort) == expected++class TestOpenAIAdapterErrorMapping:@pytest.fixturedef request_no_structured_output(self) -> CompletionRequest:diff --git a/products/ai_observability/backend/llm/types.py b/products/ai_observability/backend/llm/types.pyindex c0aec0240527..808eb04b3f64 100644--- a/products/ai_observability/backend/llm/types.py+++ b/products/ai_observability/backend/llm/types.py@@ -5,6 +5,8 @@ from pydantic import BaseModelfrom posthog.dataclasses import frozen+ReasoningLevel = Literal["minimal", "low", "medium", "high"]+@dataclassclass Usage:@@ -42,7 +44,7 @@ class CompletionRequest:response_format: type[BaseModel] | None = Noneoutput_schema: OutputSchema | None = Nonethinking: bool = False- reasoning_level: str | None = None+ reasoning_level: ReasoningLevel | None = None@dataclassdiff --git a/products/ai_observability/backend/models/evaluation_configs.py b/products/ai_observability/backend/models/evaluation_configs.pyindex 2af1f2c76b11..46f2bf5dca78 100644--- a/products/ai_observability/backend/models/evaluation_configs.py+++ b/products/ai_observability/backend/models/evaluation_configs.py@@ -30,6 +30,11 @@ class LLMJudgeConfig(BaseModel):judge_method: Literal["llm", "decision"] | None = Field(default=None, description="Judge method. Omit to keep the saved method; null restores automatic routing.")+ # Matches llm.types.ReasoningLevel; importing the llm package here would load provider SDKs at startup.+ reasoning_level: Literal["minimal", "low", "medium", "high"] | None = Field(+ default=None,+ description="How much the judge model reasons before it answers. Null uses the model's default.",+ )@field_validator("prompt")@classmethoddiff --git a/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsxindex 0d8cbc35c1c8..c81b7c793f19 100644--- a/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx+++ b/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx@@ -4,28 +4,40 @@ import { ByokModelPickerNotice } from '../../ByokModelPickerNotice'import { getModelPickerFooterLink, ModelPicker } from '../../ModelPicker'import { modelPickerLogic } from '../../modelPickerLogic'import { llmEvaluationLogic } from '../llmEvaluationLogic'+import { EvaluationReasoningLevelButton } from './EvaluationReasoningLevelButton'/** The judge model picker with its notices, for the models the current judge method can use. */export function EvaluationJudgeModelPicker(): JSX.Element {const { byokModelsLoading, providerKeysLoading } = useValues(modelPickerLogic)- const { selectedModel, selectedModelName, selectedPickerProviderKeyId, usesDecisionModel, judgeModelGroups } =- useValues(llmEvaluationLogic)+ const {+ selectedModel,+ selectedModelName,+ selectedPickerProviderKeyId,+ usesDecisionModel,+ judgeModelGroups,+ selectedModelSupportsReasoning,+ } = useValues(llmEvaluationLogic)const { selectModelFromPicker } = useActions(llmEvaluationLogic)const loading = byokModelsLoading || providerKeysLoadingreturn (<div>- <ModelPicker- model={selectedModel}- selectedProviderKeyId={selectedPickerProviderKeyId}- onSelect={selectModelFromPicker}- groups={judgeModelGroups}- loading={loading}- footerLink={getModelPickerFooterLink(judgeModelGroups.some((group) => !group.disabledReason))}- selectedModelName={selectedModelName}- data-attr="evaluation-model-selector"- />+ <div className="flex gap-2">+ <div className="flex-1 min-w-0">+ <ModelPicker+ model={selectedModel}+ selectedProviderKeyId={selectedPickerProviderKeyId}+ onSelect={selectModelFromPicker}+ groups={judgeModelGroups}+ loading={loading}+ footerLink={getModelPickerFooterLink(judgeModelGroups.some((group) => !group.disabledReason))}+ selectedModelName={selectedModelName}+ data-attr="evaluation-model-selector"+ />+ </div>+ {selectedModelSupportsReasoning && <EvaluationReasoningLevelButton />}+ </div><ByokModelPickerNotice forEvaluation />{usesDecisionModel && !loading && !judgeModelGroups.some((group) => group.models.length > 0) && (<p className="text-sm text-muted mt-2">diff --git a/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsxindex e84ceed4588f..1645b679a376 100644--- a/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx+++ b/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx@@ -4,16 +4,22 @@ import { BindLogic } from 'kea'import { useStorybookMocks } from '~/mocks/browser'import { llmEvaluationLogic } from '../llmEvaluationLogic'-import type { JudgeMethod } from '../types'+import type { JudgeMethod, ReasoningLevel } from '../types'import { EvaluationModelPicker } from './EvaluationModelPicker'-const meta: Meta<{ method?: JudgeMethod; narrow?: boolean; empty?: boolean; nativeOpenAI?: boolean }> = {+const meta: Meta<{+ method?: JudgeMethod+ narrow?: boolean+ empty?: boolean+ nativeOpenAI?: boolean+ reasoningLevel?: ReasoningLevel+}> = {title: 'Scenes-App/LLM observability/EvaluationModelPicker',component: EvaluationModelPicker,parameters: {layout: 'padded',},- render: ({ method = 'llm', narrow = false, empty = false, nativeOpenAI = false }) => {+ render: ({ method = 'llm', narrow = false, empty = false, nativeOpenAI = false, reasoningLevel }) => {useStorybookMocks({get: {'/api/environments/:team_id/llm_analytics/provider_keys/': {@@ -35,6 +41,7 @@ const meta: Meta<{ method?: JudgeMethod; narrow?: boolean; empty?: boolean; natiprovider: 'OpenAI',supports_chat: true,supports_decisions: false,+ supports_reasoning: true,},{id: 'gpt-6-luna',@@ -78,19 +85,25 @@ const meta: Meta<{ method?: JudgeMethod; narrow?: boolean; empty?: boolean; natiname: 'Response quality',enabled: false,evaluation_type: 'llm_judge',- evaluation_config: { prompt: 'Does the response answer the question?', judge_method: method },+ evaluation_config: {+ prompt: 'Does the response answer the question?',+ judge_method: method,+ reasoning_level: reasoningLevel,+ },output_type: 'boolean',output_config: {},conditions: [],target: 'generation',target_config: {},- model_configuration: null,+ model_configuration: reasoningLevel+ ? { provider: 'openai', model: 'gpt-5-mini', provider_key_id: 'example-openai-key' }+ : null,},},})return (<BindLogic logic={llmEvaluationLogic} props={{ evaluationId: 'example-evaluation' }}>- <div className={narrow ? 'max-w-lg' : 'max-w-4xl'}>+ <div className={narrow ? 'w-[32rem]' : 'max-w-4xl'}><EvaluationModelPicker /></div></BindLogic>@@ -106,3 +119,6 @@ export const DecisionNarrow: Story = { args: { method: 'decision', narrow: trueexport const NoDecisionModels: Story = { args: { method: 'decision', empty: true } }export const NativeOpenAI: Story = { args: { method: 'decision', nativeOpenAI: true } }export const NativeOpenAINarrow: Story = { args: { method: 'decision', nativeOpenAI: true, narrow: true } }+export const ReasoningLevelNarrow: Story = {+ args: { method: 'llm', nativeOpenAI: true, narrow: true, reasoningLevel: 'medium' },+}diff --git a/products/ai_observability/frontend/evaluations/components/EvaluationReasoningLevelButton.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationReasoningLevelButton.tsxnew file mode 100644index 000000000000..7a18e23cc9af--- /dev/null+++ b/products/ai_observability/frontend/evaluations/components/EvaluationReasoningLevelButton.tsx@@ -0,0 +1,43 @@+import { useActions, useValues } from 'kea'++import { IconBrain } from '@posthog/icons'+import { LemonButton } from '@posthog/lemon-ui'++import { LemonMenu } from 'lib/lemon-ui/LemonMenu'++import { llmEvaluationLogic } from '../llmEvaluationLogic'+import type { ReasoningLevel } from '../types'++const REASONING_LEVEL_OPTIONS: { value: ReasoningLevel | null; label: string }[] = [+ { value: null, label: 'Model default' },+ { value: 'minimal', label: 'Minimal' },+ { value: 'low', label: 'Low' },+ { value: 'medium', label: 'Medium' },+ { value: 'high', label: 'High' },+]++export function EvaluationReasoningLevelButton(): JSX.Element {+ const { reasoningLevel } = useValues(llmEvaluationLogic)+ const { setReasoningLevel } = useActions(llmEvaluationLogic)++ return (+ <LemonMenu+ placement="bottom-end"+ items={REASONING_LEVEL_OPTIONS.map(({ value, label }) => ({+ label,+ active: value === reasoningLevel,+ onClick: () => setReasoningLevel(value),+ }))}+ >+ <LemonButton+ type="secondary"+ icon={<IconBrain />}+ tooltip="Reasoning effort. Higher levels can judge harder cases better, but cost more and take longer."+ aria-label="Reasoning effort"+ data-attr="evaluation-reasoning-level"+ >+ {reasoningLevel && REASONING_LEVEL_OPTIONS.find((option) => option.value === reasoningLevel)?.label}+ </LemonButton>+ </LemonMenu>+ )+}diff --git a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.tsindex ecbc2abbb9c7..5b58de8384bc 100644--- a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts+++ b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts@@ -1954,6 +1954,47 @@ return result`,)})+ it.each([+ [true, 'high'],+ [false, null],+ ] as const)(+ 'keeps the reasoning level only when the next model supports it (supportsReasoning=%s)',+ async (supportsReasoning, expectedLevel) => {+ await expectLogic(logic).toDispatchActions(['loadEvaluationSuccess'])+ await expectLogic(keysLogic).toDispatchActions(['loadProviderKeysSuccess'])+ await expectLogic(modelPickerLogic).toFinishAllListeners()+ modelPickerLogic.actions.loadByokModelsSuccess([+ {+ id: 'example/reasoning-model',+ name: 'example/reasoning-model',+ description: '',+ provider: 'OpenAI',+ providerKeyId: 'key-1',+ supportsReasoning: true,+ },+ {+ id: 'example/next-model',+ name: 'example/next-model',+ description: '',+ provider: 'OpenAI',+ providerKeyId: 'key-1',+ supportsReasoning,+ },+ ])+ await expectLogic(logic, () =>+ logic.actions.selectModelFromPicker('example/reasoning-model', 'key-1')+ ).toFinishAllListeners()+ logic.actions.setReasoningLevel('high')+ expect(logic.values.selectedModelSupportsReasoning).toBe(true)++ await expectLogic(logic, () =>+ logic.actions.selectModelFromPicker('example/next-model', 'key-1')+ ).toFinishAllListeners()++ expect(logic.values.reasoningLevel).toBe(expectedLevel)+ }+ )+beforeEach(() => {logic = llmEvaluationLogic({ evaluationId: 'new' })logic.mount()diff --git a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.tsindex a7794abd62c9..502d0b0cc659 100644--- a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts+++ b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts@@ -78,6 +78,7 @@ import type {LLMJudgeEvaluation,JudgeMethod,ModelConfiguration,+ ReasoningLevel,SentimentEvaluation,} from './types'@@ -352,6 +353,7 @@ export interface llmEvaluationLogicValues {numericBoundsRequired: booleanoriginalEvaluation: EvaluationConfig | nulloutputConfigDrafts: Partial<Record<'boolean' | 'categorical' | 'numeric', EvaluationOutputConfig>>+ reasoningLevel: ReasoningLevel | nullrunsBackfill: EvaluationBackfillApi | nullrunsBackfillId: string | nullrunsBackfillLoading: boolean@@ -368,6 +370,8 @@ export interface llmEvaluationLogicValues {} | nullselectedModel: stringselectedModelName: string | undefined+ selectedModelOption: ModelOption | null+ selectedModelSupportsReasoning: booleanselectedPickerProviderKeyId: string | nullsidePanelContext: SidePanelSceneContext | nulltrendInsightUrl: string | null@@ -543,6 +547,9 @@ export interface llmEvaluationLogicActions {outputType: 'boolean' | 'categorical' | 'numeric'previousEvaluation: EvaluationConfig | null}+ setReasoningLevel: (reasoningLevel: ReasoningLevel | null) => {+ reasoningLevel: ReasoningLevel | null+ }setRunsBackfillId: (backfillId: string | null) => {backfillId: string | null}@@ -597,8 +604,11 @@ export interface llmEvaluationLogicMeta {originalEvaluation: EvaluationConfig | null,evaluationId: string) => boolean- judgeMethod: (evaluation: EvaluationConfig | null, byokModels: ModelOption[]) => JudgeMethod+ selectedModelOption: (evaluation: EvaluationConfig | null, byokModels: ModelOption[]) => ModelOption | null+ judgeMethod: (evaluation: EvaluationConfig | null, selectedModelOption: ModelOption | null) => JudgeMethodusesDecisionModel: (judgeMethod: JudgeMethod) => boolean+ reasoningLevel: (evaluation: EvaluationConfig | null) => ReasoningLevel | null+ selectedModelSupportsReasoning: (selectedModelOption: ModelOption | null, usesDecisionModel: boolean) => booleanselectedModelName: (byokModels: ModelOption[],selectedModel: string,@@ -694,6 +704,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([setTriggerConditions: (conditions: EvaluationConditionSet[]) => ({ conditions }),setModelConfiguration: (modelConfiguration: ModelConfiguration | null) => ({ modelConfiguration }),setJudgeMethod: (judgeMethod: JudgeMethod) => ({ judgeMethod }),+ setReasoningLevel: (reasoningLevel: ReasoningLevel | null) => ({ reasoningLevel }),setEvaluationType: (evaluationType: EvaluationType) => ({ evaluationType }),setEvaluationTarget: (target: EvaluationTarget) => ({ target }),setSettleStrategy: (strategy: EvaluationSettleStrategy) => ({ strategy }),@@ -1013,6 +1024,13 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([state && isLLMJudgeEvaluation(state)? { ...state, model_configuration: modelConfiguration }: state,+ setReasoningLevel: (state, { reasoningLevel }) =>+ state && isLLMJudgeEvaluation(state)+ ? {+ ...state,+ evaluation_config: { ...state.evaluation_config, reasoning_level: reasoningLevel },+ }+ : state,setJudgeMethod: (state, { judgeMethod }) =>state && isLLMJudgeEvaluation(state)? {@@ -1174,6 +1192,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([setTriggerConditions: () => true,setModelConfiguration: () => true,setJudgeMethod: () => true,+ setReasoningLevel: () => true,setEvaluationType: () => true,setEvaluationTarget: () => true,setSettleStrategy: () => true,@@ -1579,12 +1598,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([if (!config) {return}- const model = values.byokModels.find(- (model) =>- model.id === config.model &&- toLLMProvider(model.provider) === config.provider &&- (!config.provider_key_id || model.providerKeyId === config.provider_key_id)- )+ const model = values.selectedModelOptionconst compatible = model? values.judgeMethod === 'decision'? model.supportsDecisions@@ -1595,6 +1609,14 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([if (!compatible) {actions.setModelConfiguration(null)}+ if (values.reasoningLevel && !values.selectedModelSupportsReasoning) {+ actions.setReasoningLevel(null)+ }+ },+ setModelConfiguration: () => {+ if (values.reasoningLevel && !values.selectedModelSupportsReasoning) {+ actions.setReasoningLevel(null)+ }},setEvaluationType: () => {if (!values.isReportableEvaluation && values.activeTab === 'reports') {@@ -1723,28 +1745,47 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([},],- judgeMethod: [+ selectedModelOption: [(s) => [s.evaluation, s.byokModels],- (evaluation: EvaluationConfig | null, models: ModelOption[]): JudgeMethod => {+ (evaluation: EvaluationConfig | null, models: ModelOption[]): ModelOption | null => {+ const config = evaluation?.model_configuration+ if (!config) {+ return null+ }+ return (+ models.find(+ (model) =>+ model.id === config.model &&+ toLLMProvider(model.provider) === config.provider &&+ (!config.provider_key_id || model.providerKeyId === config.provider_key_id)+ ) ?? null+ )+ },+ ],+ judgeMethod: [+ (s) => [s.evaluation, s.selectedModelOption],+ (evaluation: EvaluationConfig | null, model: ModelOption | null): JudgeMethod => {if (isLLMJudgeEvaluation(evaluation) && evaluation.evaluation_config?.judge_method) {return evaluation.evaluation_config.judge_method}- const config = evaluation?.model_configuration- return config?.provider === 'system_one' ||- (config?.provider === 'openrouter' &&- models.some(- (model) =>- model.id === config.model &&- toLLMProvider(model.provider) === 'openrouter' &&- (!config.provider_key_id || model.providerKeyId === config.provider_key_id) &&- model.supportsDecisions &&- !model.supportsChat- ))+ const provider = evaluation?.model_configuration?.provider+ return provider === 'system_one' ||+ (provider === 'openrouter' && !!model?.supportsDecisions && !model.supportsChat)? 'decision': 'llm'},],usesDecisionModel: [(s) => [s.judgeMethod], (judgeMethod: JudgeMethod): boolean => judgeMethod === 'decision'],+ reasoningLevel: [+ (s) => [s.evaluation],+ (evaluation: EvaluationConfig | null): ReasoningLevel | null =>+ (isLLMJudgeEvaluation(evaluation) && evaluation.evaluation_config.reasoning_level) || null,+ ],+ selectedModelSupportsReasoning: [+ (s) => [s.selectedModelOption, s.usesDecisionModel],+ (model: ModelOption | null, usesDecisionModel: boolean): boolean =>+ !usesDecisionModel && !!model?.supportsReasoning,+ ],// Evals always run on the team's own provider key, so only BYOK models can be the selected one.selectedModelName: [(s) => [s.byokModels, s.selectedModel, s.selectedPickerProviderKeyId],diff --git a/products/ai_observability/frontend/evaluations/types.ts b/products/ai_observability/frontend/evaluations/types.tsindex 0f011c7cd732..348d86a96eb3 100644--- a/products/ai_observability/frontend/evaluations/types.ts+++ b/products/ai_observability/frontend/evaluations/types.ts@@ -53,6 +53,7 @@ export interface EvaluationTargetConfig {export type LLMJudgeEvaluationConfig = Extract<EvaluationApiEvaluationConfig, { prompt: string }>export type JudgeMethod = NonNullable<LLMJudgeEvaluationConfig['judge_method']>+export type ReasoningLevel = NonNullable<LLMJudgeEvaluationConfig['reasoning_level']>export interface HogEvaluationConfig {source: stringdiff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.tsindex 465014a8e56b..492fdfa88426 100644--- a/products/ai_observability/frontend/generated/api.schemas.ts+++ b/products/ai_observability/frontend/generated/api.schemas.ts@@ -2347,6 +2347,11 @@ export type EvaluationApiEvaluationConfig =* @nullable*/judge_method?: 'llm' | 'decision' | null+ /**+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.+ * @nullable+ */+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null}| {/**@@ -2676,6 +2681,11 @@ export type PatchedEvaluationApiEvaluationConfig =* @nullable*/judge_method?: 'llm' | 'decision' | null+ /**+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.+ * @nullable+ */+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null}| {/**diff --git a/products/ai_observability/frontend/generated/api.zod.ts b/products/ai_observability/frontend/generated/api.zod.tsindex 7f11e919e857..91e12dbbdce9 100644--- a/products/ai_observability/frontend/generated/api.zod.ts+++ b/products/ai_observability/frontend/generated/api.zod.ts@@ -620,6 +620,12 @@ export const EvaluationsCreateBody = /* @__PURE__ */ zod.enum(['llm', 'decision']).nullish().describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),+ reasoning_level: zod+ .enum(['minimal', 'low', 'medium', 'high'])+ .nullish()+ .describe(+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."+ ),}),zod.object({source: zod@@ -1015,6 +1021,12 @@ export const EvaluationsUpdateBody = /* @__PURE__ */ zod.enum(['llm', 'decision']).nullish().describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),+ reasoning_level: zod+ .enum(['minimal', 'low', 'medium', 'high'])+ .nullish()+ .describe(+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."+ ),}),zod.object({source: zod@@ -1313,6 +1325,12 @@ export const EvaluationsPartialUpdateBody = /* @__PURE__ */ zod.enum(['llm', 'decision']).nullish().describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),+ reasoning_level: zod+ .enum(['minimal', 'low', 'medium', 'high'])+ .nullish()+ .describe(+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."+ ),}),zod.object({source: zoddiff --git a/products/ai_observability/frontend/modelPickerLogic.test.ts b/products/ai_observability/frontend/modelPickerLogic.test.tsindex f511b858db92..1ada56d37351 100644--- a/products/ai_observability/frontend/modelPickerLogic.test.ts+++ b/products/ai_observability/frontend/modelPickerLogic.test.ts@@ -115,6 +115,7 @@ describe('modelPickerLogic', () => {isRecommended: true,supportsDecisions: false,supportsChat: true,+ supportsReasoning: false,providerKeyId: 'key-1',})))@@ -291,6 +292,7 @@ describe('modelPickerLogic', () => {providerKeyId: 'key-2',supportsDecisions: false,supportsChat: true,+ supportsReasoning: false,})))expect(logic.values.providerModelGroups.find((g) => g.providerKeyId === 'key-1')).toEqual({diff --git a/products/ai_observability/frontend/modelPickerLogic.ts b/products/ai_observability/frontend/modelPickerLogic.tsindex cd12b23d87b8..de80f4ea91ad 100644--- a/products/ai_observability/frontend/modelPickerLogic.ts+++ b/products/ai_observability/frontend/modelPickerLogic.ts@@ -22,6 +22,7 @@ export interface ModelOption {isRecommended?: booleansupportsDecisions?: booleansupportsChat?: boolean+ supportsReasoning?: boolean}export interface ProviderModelGroup {@@ -289,6 +290,7 @@ export const modelPickerLogic = kea<modelPickerLogicType>([is_recommended?: booleansupports_decisions?: booleansupports_chat?: boolean+ supports_reasoning?: boolean})[]return rawModels.map((m) => ({id: m.id,@@ -298,6 +300,7 @@ export const modelPickerLogic = kea<modelPickerLogicType>([isRecommended: m.is_recommended ?? false,supportsDecisions: m.supports_decisions ?? false,supportsChat: m.supports_chat ?? !m.supports_decisions,+ supportsReasoning: m.supports_reasoning ?? false,providerKeyId: key.id,}))} catch {diff --git a/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts b/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.tsindex 082ced8aca09..b2c58c21af5e 100644--- a/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts+++ b/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts@@ -800,6 +800,7 @@ describe('llmPlaygroundLogic', () => {providerKeyId: 'key-1',supportsDecisions: false,supportsChat: true,+ supportsReasoning: false,})))})diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.tsindex 8bac3d95957f..0d83508e42eb 100644--- a/products/desktop/packages/api-client/src/generated.ts+++ b/products/desktop/packages/api-client/src/generated.ts@@ -9228,6 +9228,9 @@ export namespace Schemas {judge_method?:| (("llm" | "decision") | null)| undefined;+ reasoning_level?:+ | (("minimal" | "low" | "medium" | "high") | null)+ | undefined;}| {/**@@ -19473,6 +19476,9 @@ export namespace Schemas {| {prompt: string;judge_method?: (("llm" | "decision") | null) | undefined;+ reasoning_level?:+ | (("minimal" | "low" | "medium" | "high") | null)+ | undefined;}| { source: string }| Partial<{ source: "user_messages" }>;diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 9c309a51aff1..0594669f6289 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -45312,6 +45312,11 @@ export namespace Schemas {* @nullable*/judge_method?: 'llm' | 'decision' | null;+ /**+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.+ * @nullable+ */+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null;} | {/*** Hog source code. Return a boolean, finite number, or category keys matching output_type. Categorical single selection accepts one key or a one-item list; multiple selection accepts a list, including []. Return null only for allowed N/A. Output settings determine which boolean counts as a failure.@@ -87677,6 +87682,11 @@ export namespace Schemas {* @nullable*/judge_method?: 'llm' | 'decision' | null;+ /**+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.+ * @nullable+ */+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null;} | {/*** Hog source code. Return a boolean, finite number, or category keys matching output_type. Categorical single selection accepts one key or a one-item list; multiple selection accepts a list, including []. Return null only for allowed N/A. Output settings determine which boolean counts as a failure.diff --git a/services/mcp/src/generated/ai_observability/api.ts b/services/mcp/src/generated/ai_observability/api.tsindex c4cda045eab7..7bdbe19643b3 100644--- a/services/mcp/src/generated/ai_observability/api.ts+++ b/services/mcp/src/generated/ai_observability/api.ts@@ -1435,6 +1435,12 @@ export const EvaluationsCreateBody = () => zod.enum(['llm', 'decision']).nullish().describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),+ reasoning_level: zod+ .enum(['minimal', 'low', 'medium', 'high'])+ .nullish()+ .describe(+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."+ ),}),zod.object({source: zod@@ -1910,6 +1916,12 @@ export const EvaluationsPartialUpdateBody = () => zod.enum(['llm', 'decision']).nullish().describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),+ reasoning_level: zod+ .enum(['minimal', 'low', 'medium', 'high'])+ .nullish()+ .describe(+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."+ ),}),zod.object({source: zoddiff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.jsonindex 7f94872293d0..2211a7d9905a 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json@@ -79,6 +79,18 @@"description": "Evaluation criteria for the LLM judge. Describe what makes a good vs bad response.","minLength": 1,"type": "string"+ },+ "reasoning_level": {+ "anyOf": [+ {+ "enum": ["minimal", "low", "medium", "high"],+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."}},"required": ["prompt"],diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.jsonindex 5537211a1165..d4eba03e9095 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json@@ -78,6 +78,18 @@"description": "Evaluation criteria for the LLM judge. Describe what makes a good vs bad response.","minLength": 1,"type": "string"+ },+ "reasoning_level": {+ "anyOf": [+ {+ "enum": ["minimal", "low", "medium", "high"],+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."}},"required": ["prompt"],