Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): let llm judges set a reasoning level (#125840)

Bernat Torresd2a878248b70
diff --git a/docs/internal/ai-observability-judge-inputs.md b/docs/internal/ai-observability-judge-inputs.md
index 617e39a89340..12b24733e73b 100644
--- a/docs/internal/ai-observability-judge-inputs.md
+++ b/docs/internal/ai-observability-judge-inputs.md
@@ -180,6 +180,18 @@ Invalid probabilities, missing answers, and mismatched answer types skip the ite
Inputs rejected for exceeding the model's context window are skipped.
See OpenAI's [Decisions guide](https://developers.openai.com/api/docs/guides/decisions), OpenRouter's [Decisions API reference](https://openrouter.ai/docs/api/api-reference/alphadecisions/submit-a-decisions-request), and TypeSafe's [API reference](https://docs.typesafe.ai/api) for their question and answer formats.
+## Reasoning level
+
+LLM judges on Anthropic and OpenAI reasoning models can set a reasoning level with the brain button next to the model selector.
+The level is stored as `evaluation_config.reasoning_level` (`minimal`, `low`, `medium` or `high`). Without it, the judge uses the model's default.
+Decision models and other providers do not support it, and the API rejects a level for them.
+Prompt-only `evaluation_config` PATCH requests preserve the saved level; an explicit `reasoning_level: null` restores the model default.
+Selecting a model without reasoning support clears the level in the editor.
+
+Anthropic models before Claude 4.6 get a thinking budget of 1,024, 2,048, 4,096 or 8,192 tokens. Newer models get adaptive thinking with that effort, and `minimal` becomes `low`.
+OpenAI models get `reasoning_effort`. Only the original GPT-5 models accept `minimal`; other models get `low` instead.
+For Anthropic, the token limit of the reply grows by the thinking budget of the level, so the verdict still fits after the thinking.
+
## Model output limits
When the judge reply reaches the model's output limit, the evaluation skips that item with `output_limit_exceeded`.
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index 50d93b4c7e33..3975b426b985 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -13443,9 +13443,9 @@ snapshots:
scenes-app-llm-observability-evaluationmodelpicker--decision--light:
hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.CM-LkhqTNhtSyn0O3Y6c6Yd_gLfSXe31n3Dxaj5uEZI
scenes-app-llm-observability-evaluationmodelpicker--decision-narrow--dark:
- hash: v1.k794b7964.8cf03ee1f25ccdee62cc5104d48589d196f85dd3dbad84d5a838cc4c8bfe9607.W_9QnaVZtdekweLtoPuF32I_r0G0fCdjRWUVyBg3kEI
+ hash: v1.k794b7964.e75b3c58f2b1922e7ce4b3f8512fa7dc81021d0d988622599b3f52bb40f32eeb.PLEVbwWEAXbpJdC_PSjXyTKP-lLKLDdAkRcrP9jOOXU
scenes-app-llm-observability-evaluationmodelpicker--decision-narrow--light:
- hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.HuCQ8PyBI05CTQEjr38moKf85RyyPvF9SCxuwytomXQ
+ hash: v1.k794b7964.8ae8ae696fb4cd22a44ccd18c901236eea8a0e6ac6273b686dc946821f3f7962.Dw7EaOMcP1cHXUw3o5SoYiAZ66QGAYk7WzhqDGcCrLY
scenes-app-llm-observability-evaluationmodelpicker--llm--dark:
hash: v1.k794b7964.a42ccc41fa5864dee67f21add5ecc3aa6838805fb494f6686b826688cc325e9e.tHDxeEbDTVatdZYPj2xVDQtJpukqzNyUHU9iScfQ2AY
scenes-app-llm-observability-evaluationmodelpicker--llm--light:
@@ -13455,13 +13455,17 @@ snapshots:
scenes-app-llm-observability-evaluationmodelpicker--native-open-ai--light:
hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.nTYWu7M1TfGyOlu52uGxd98cNvdj-nyoyzcWUovs6Ys
scenes-app-llm-observability-evaluationmodelpicker--native-open-ai-narrow--dark:
- hash: v1.k794b7964.8cf03ee1f25ccdee62cc5104d48589d196f85dd3dbad84d5a838cc4c8bfe9607.gZEs23POnv-eBuhQT-V2gxTuoBiA7R32dhshLPHHyVs
+ hash: v1.k794b7964.e75b3c58f2b1922e7ce4b3f8512fa7dc81021d0d988622599b3f52bb40f32eeb.qdYv6zfUkJUHXikWSwr_1jyDM3PvnFOPTj8qFIdrqr8
scenes-app-llm-observability-evaluationmodelpicker--native-open-ai-narrow--light:
- hash: v1.k794b7964.6b288367a286d1858e3032bab65a034553c5f5ae79dacd0274cb92d1f8bab818.DO5O6V3lvkGYD6JJ5RfEBG2IbsSvb0aNeCME9dvKEWk
+ hash: v1.k794b7964.8ae8ae696fb4cd22a44ccd18c901236eea8a0e6ac6273b686dc946821f3f7962._qdgUVltz2sd1ZHnkaFq8J5CXje346MVRlc4WOsz-pg
scenes-app-llm-observability-evaluationmodelpicker--no-decision-models--dark:
hash: v1.k794b7964.ed596389ff3b9a8a505c3b1ad5cbfabadc1cb83507c341f99dd54b004f8d257e.Jm_sWi9-7s4eTkxjyQUcGApiT5QcFzke8u5k2hK6gaE
scenes-app-llm-observability-evaluationmodelpicker--no-decision-models--light:
hash: v1.k794b7964.b10b433ed0507465953ef6174a8872546e4e695407914798a8e11a6e7f8aa38d.jQ8Xqfw0JjPdrTXWnuYodzsfn46fs4aKri7d1Pzl7QU
+ scenes-app-llm-observability-evaluationmodelpicker--reasoning-level-narrow--dark:
+ hash: v1.k794b7964.c135d25cde9ab12d1bbf35f7cc6cc87ab8c838586bdbd7316b62a59d54d85b9e.QVUXtySXrLSrHHvMht-mZvR9AIKvry19dFXdhLIHs30
+ scenes-app-llm-observability-evaluationmodelpicker--reasoning-level-narrow--light:
+ hash: v1.k794b7964.d0372da3513c43cccbb1d2c7e9b6212567f86ca3e20f33cfaf0fba84c2eb8072.1Lrr0sxzbqQN9iCbO_JrkZtixXdNV2MOgqu6TI5qjDw
scenes-app-llm-observability-evaluationreportconfig--lemon-ui--dark:
hash: v1.k794b7964.195aa4a5a39e8f947e17fe67f5c23303b7dd0579c278907761be24a08780be92.IkGQtAI95T60kJKVZcmmFBIWNCd4meuJv-bcNDoohwU
scenes-app-llm-observability-evaluationreportconfig--lemon-ui--light:
diff --git a/posthog/temporal/ai_observability/evaluation_llm_judge.py b/posthog/temporal/ai_observability/evaluation_llm_judge.py
index 0825d945d7b0..312383fed0c0 100644
--- a/posthog/temporal/ai_observability/evaluation_llm_judge.py
+++ b/posthog/temporal/ai_observability/evaluation_llm_judge.py
@@ -828,6 +828,7 @@ def call_llm_judge(
messages=[{"role": "user", "content": user_prompt}],
provider=provider,
response_format=response_format,
+ reasoning_level=evaluation["evaluation_config"].get("reasoning_level"),
)
)
except (DecisionEndpointBlockedError, ProviderConfigurationError) as e:
diff --git a/posthog/temporal/ai_observability/test_run_evaluation.py b/posthog/temporal/ai_observability/test_run_evaluation.py
index a1210eb83260..3366a68f9844 100644
--- a/posthog/temporal/ai_observability/test_run_evaluation.py
+++ b/posthog/temporal/ai_observability/test_run_evaluation.py
@@ -373,6 +373,7 @@ def test_legacy_and_explicit_llm_methods_use_chat_for_a_dual_capability_model(
"team_id": 1,
"evaluation_config": {
"prompt": "Check the response.",
+ "reasoning_level": "high",
**({"judge_method": judge_method} if judge_method else {}),
},
},
@@ -381,6 +382,7 @@ def test_legacy_and_explicit_llm_methods_use_chat_for_a_dual_capability_model(
allows_na=False,
)
complete.assert_called_once()
+ assert complete.call_args.args[0].reasoning_level == "high"
decide.assert_not_called()
native_decide.assert_not_called()
assert result["verdict"] is True
diff --git a/products/ai_observability/backend/api/evaluations.py b/products/ai_observability/backend/api/evaluations.py
index 940f9589e4fe..7798809a85bf 100644
--- a/products/ai_observability/backend/api/evaluations.py
+++ b/products/ai_observability/backend/api/evaluations.py
@@ -58,7 +58,7 @@ from ..facade import (
)
from ..facade.contracts import Contract
from ..hog import compile_ai_observability_hog
-from ..llm import DEFAULT_MODEL_BY_PROVIDER
+from ..llm import DEFAULT_MODEL_BY_PROVIDER, supports_reasoning_level
from ..llm.decisions import is_decision_model
from ..llm.errors import ProviderConnectionError
from ..llm.providers.openai import OpenAIConfig
@@ -126,6 +126,16 @@ logger = structlog.get_logger(__name__)
"nullable": True,
"description": "Judge method. Omit to keep the saved method; null restores automatic routing.",
},
+ "reasoning_level": {
+ "type": "string",
+ "enum": ["minimal", "low", "medium", "high"],
+ "nullable": True,
+ "description": (
+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI "
+ "reasoning models with the 'llm' judge method support it. Omit to keep the saved "
+ "level; null uses the model's default."
+ ),
+ },
},
"additionalProperties": False,
},
@@ -591,15 +601,14 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSe
config = data["evaluation_config"]
if not isinstance(config, dict):
raise serializers.ValidationError({"evaluation_config": "Must be an object."})
- if (
- self.partial
- and self.instance is not None
- and evaluation_type == EvaluationType.LLM_JUDGE.value
- and "judge_method" not in config
- and "judge_method" in self.instance.evaluation_config
- ):
- # A prompt-only PATCH must not reset the saved routing choice.
- data["evaluation_config"] = {"judge_method": self.instance.evaluation_config["judge_method"], **config}
+ if self.partial and self.instance is not None and evaluation_type == EvaluationType.LLM_JUDGE.value:
+ # A prompt-only PATCH must not reset the saved judge settings.
+ saved = {
+ key: self.instance.evaluation_config[key]
+ for key in ("judge_method", "reasoning_level")
+ if key not in config and key in self.instance.evaluation_config
+ }
+ data["evaluation_config"] = {**saved, **config}
if (
self.instance
and output_type != self.instance.output_type
@@ -639,6 +648,7 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSe
).get("judge_method")
if data.get("model_configuration") or data.get("enabled") or method_changed:
self._validate_chat_model(data, uses_decision_model=uses_decision_model)
+ self._validate_reasoning_level(data, uses_decision_model=uses_decision_model)
should_validate_configs = (
self.instance is None
@@ -742,6 +752,21 @@ class EvaluationSerializer(UserAccessControlSerializerMixin, serializers.ModelSe
}
)
+ def _validate_reasoning_level(self, data: dict, *, uses_decision_model: bool) -> None:
+ evaluation_type = data.get("evaluation_type") or getattr(self.instance, "evaluation_type", None)
+ if evaluation_type != EvaluationType.LLM_JUDGE.value:
+ return
+ evaluation_config = data.get("evaluation_config", getattr(self.instance, "evaluation_config", None))
+ if not isinstance(evaluation_config, dict) or not evaluation_config.get("reasoning_level"):
+ return
+ model_config = self._effective_model_configuration(data) or {}
+ if uses_decision_model or not supports_reasoning_level(model_config.get("provider"), model_config.get("model")):
+ raise serializers.ValidationError(
+ {
+ "evaluation_config": "This judge model doesn't support a reasoning level. Choose an Anthropic or OpenAI reasoning model, or use the model default."
+ }
+ )
+
def _validate_can_run(self, data: dict) -> None:
"""An eval being turned on — created enabled or re-enabled — must be able to resolve a
model at runtime; otherwise the caller gets a 200/201 and the first Temporal run silently
diff --git a/products/ai_observability/backend/api/proxy.py b/products/ai_observability/backend/api/proxy.py
index fd2d2678fc1f..993922756821 100644
--- a/products/ai_observability/backend/api/proxy.py
+++ b/products/ai_observability/backend/api/proxy.py
@@ -56,6 +56,7 @@ from products.ai_observability.backend.llm import (
ModelInfo,
OutputSchema,
get_playground_models,
+ supports_reasoning_level,
)
from products.ai_observability.backend.llm.errors import (
ProviderConfigurationError,
@@ -147,6 +148,9 @@ class LLMProxyModelInfoSerializer(serializers.Serializer):
supports_chat = serializers.BooleanField(
default=True, help_text="Whether this model supports chat completions with written reasoning."
)
+ supports_reasoning = serializers.BooleanField(
+ default=False, help_text="Whether an evaluation judge can set a reasoning level for this model."
+ )
class PlaygroundAccessPermission(BasePermission):
@@ -568,6 +572,7 @@ class LLMProxyViewSet(viewsets.ViewSet):
is_recommended=m in recommended,
supports_decisions=provider_key.provider == LLMProvider.SYSTEM_ONE or m in decision_models,
supports_chat=provider_key.provider != LLMProvider.SYSTEM_ONE and m not in non_chat_models,
+ supports_reasoning=supports_reasoning_level(provider_key.provider, m),
)
for m in models
]
diff --git a/products/ai_observability/backend/api/test/test_evaluations.py b/products/ai_observability/backend/api/test/test_evaluations.py
index 5fcc123c9561..8d155c62fc1b 100644
--- a/products/ai_observability/backend/api/test/test_evaluations.py
+++ b/products/ai_observability/backend/api/test/test_evaluations.py
@@ -1,5 +1,6 @@
import json
from datetime import timedelta
+from typing import Any
from uuid import UUID, uuid4
from posthog.test.base import APIBaseTest, ClickhouseTestMixin, _create_event, flush_persons_and_events
@@ -174,11 +175,53 @@ class TestModelConfigurationSerializer(SimpleTestCase):
)
self.assertEqual(evaluation.evaluation_config, stored_config)
+ @parameterized.expand(
+ [
+ ("prompt_patch_keeps_level", {}, {"reasoning_level": "high"}),
+ ("explicit_null_clears_level", {"evaluation_config": {"reasoning_level": None}}, {}),
+ ("level_change", {"evaluation_config": {"reasoning_level": "minimal"}}, {"reasoning_level": "minimal"}),
+ (
+ "unsupported_model_rejected",
+ {"model_configuration": {"provider": "gemini", "model": "gemini-2.5-flash"}},
+ None,
+ ),
+ (
+ "openai_reasoning_model_accepted",
+ {"model_configuration": {"provider": "openai", "model": "gpt-5-mini"}},
+ {"reasoning_level": "high"},
+ ),
+ ]
+ )
+ def test_reasoning_level_patch(
+ self, _name: str, patch_data: dict[str, Any], expected_level: dict[str, str] | None
+ ) -> None:
+ evaluation = Evaluation(
+ evaluation_type="llm_judge",
+ evaluation_config={"prompt": "Check the response.", "reasoning_level": "high"},
+ output_type="boolean",
+ output_config={},
+ model_configuration=LLMModelConfiguration(provider="anthropic", model="claude-haiku-5-5"),
+ )
+ data = {**patch_data}
+ data["evaluation_config"] = {"prompt": "Check clarity.", **patch_data.get("evaluation_config", {})}
+ serializer = EvaluationSerializer(
+ instance=evaluation, data=data, partial=True, context={"get_team": lambda: Mock(id=1)}
+ )
+ # Avoid key resolution: this test checks the configuration boundary.
+ with patch.object(serializer, "_validate_re_enable"), patch.object(serializer, "_validate_can_run"):
+ valid = serializer.is_valid()
+ if expected_level is None:
+ self.assertFalse(valid)
+ self.assertIn("evaluation_config", serializer.errors)
+ return
+ self.assertTrue(valid, serializer.errors)
+ self.assertEqual(serializer.validated_data["evaluation_config"], {"prompt": "Check clarity.", **expected_level})
+
@parameterized.expand([("llm",), ("decision",)])
def test_switch_to_sentiment_ignores_stored_judge_method(self, method: str) -> None:
evaluation = Evaluation(
evaluation_type="llm_judge",
- evaluation_config={"prompt": "Check the response.", "judge_method": method},
+ evaluation_config={"prompt": "Check the response.", "judge_method": method, "reasoning_level": "high"},
output_type="boolean",
output_config={},
model_configuration=LLMModelConfiguration(provider="openrouter", model="example/model"),
diff --git a/products/ai_observability/backend/llm/__init__.py b/products/ai_observability/backend/llm/__init__.py
index a78e78948c70..fec2978e9f8e 100644
--- a/products/ai_observability/backend/llm/__init__.py
+++ b/products/ai_observability/backend/llm/__init__.py
@@ -26,6 +26,7 @@ class ModelInfo(TypedDict):
is_recommended: bool
supports_decisions: NotRequired[bool]
supports_chat: NotRequired[bool]
+ supports_reasoning: NotRequired[bool]
# Single registry of providers. Add new providers here and everything else
@@ -37,6 +38,18 @@ PROVIDERS: list[tuple[str, Any]] = [
]
+# Providers whose `complete()` honors CompletionRequest.reasoning_level.
+REASONING_LEVEL_CONFIGS: dict[str, type[AnthropicConfig] | type[OpenAIConfig]] = {
+ "anthropic": AnthropicConfig,
+ "openai": OpenAIConfig,
+}
+
+
+def supports_reasoning_level(provider: str | None, model: str | None) -> bool:
+ config = REASONING_LEVEL_CONFIGS.get(provider or "")
+ return bool(config and model and config.supports_reasoning_level(model))
+
+
def _build_model_infos(provider: str, models: list[str]) -> list[ModelInfo]:
return [{"id": m, "name": m, "provider": provider, "description": "", "is_recommended": True} for m in models]
@@ -93,6 +106,7 @@ __all__ = [
"DEFAULT_MODEL_BY_PROVIDER",
"get_default_models",
"get_playground_models",
+ "supports_reasoning_level",
"OpenAIConfig",
"AnthropicConfig",
"GeminiConfig",
diff --git a/products/ai_observability/backend/llm/providers/anthropic.py b/products/ai_observability/backend/llm/providers/anthropic.py
index 83bec9ae74a7..306dbb9b0f97 100644
--- a/products/ai_observability/backend/llm/providers/anthropic.py
+++ b/products/ai_observability/backend/llm/providers/anthropic.py
@@ -39,6 +39,7 @@ from products.ai_observability.backend.llm.types import (
AnalyticsContext,
CompletionRequest,
CompletionResponse,
+ ReasoningLevel,
StreamChunk,
Usage,
)
@@ -118,6 +119,20 @@ def _is_quota_or_billing_error(error: Exception) -> bool:
class AnthropicConfig:
MAX_TOKENS: int = 8192
MAX_THINKING_TOKENS: int = 4096
+ # Thinking tokens count against max_tokens, so each level also raises max_tokens by its budget.
+ REASONING_BUDGET_TOKENS: dict[ReasoningLevel, int] = {"minimal": 1024, "low": 2048, "medium": 4096, "high": 8192}
+ # Models before Claude 4.6 take a fixed thinking budget. Newer models reject it and take an effort level.
+ BUDGET_THINKING_MODEL_PREFIXES: tuple[str, ...] = (
+ "claude-opus-4-5",
+ "claude-sonnet-4-5",
+ "claude-haiku-4-5",
+ "claude-opus-4-1",
+ "claude-opus-4-0",
+ "claude-sonnet-4-0",
+ # Dated IDs of Claude 4.0, such as claude-sonnet-4-20250514.
+ "claude-opus-4-20",
+ "claude-sonnet-4-20",
+ )
# Timeout in seconds for API calls. Set high to accommodate slow reasoning models.
# Note: Infrastructure-level timeouts (load balancers, proxies) may still limit actual request duration.
TIMEOUT: float = 300.0
@@ -170,6 +185,11 @@ class AnthropicConfig:
"claude-sonnet-4-0",
]
+ @staticmethod
+ def supports_reasoning_level(model: str) -> bool:
+ # Every Claude model after the 3 family can think, including ones newer than our model lists.
+ return model.startswith("claude-") and not model.startswith("claude-3")
+
class AnthropicAdapter:
"""Anthropic provider implementing the unified Client interface."""
@@ -237,6 +257,9 @@ class AnthropicAdapter:
}
}
+ if request.reasoning_level and AnthropicConfig.supports_reasoning_level(request.model):
+ self._apply_reasoning_level(create_kwargs, request.model, request.reasoning_level)
+
try:
response = client.messages.create(**create_kwargs)
@@ -273,6 +296,17 @@ class AnthropicAdapter:
raise mapped from e
raise
+ @staticmethod
+ def _apply_reasoning_level(create_kwargs: dict[str, Any], model: str, level: ReasoningLevel) -> None:
+ budget = AnthropicConfig.REASONING_BUDGET_TOKENS[level]
+ create_kwargs["max_tokens"] += budget
+ if model.startswith(AnthropicConfig.BUDGET_THINKING_MODEL_PREFIXES):
+ create_kwargs["thinking"] = {"type": "enabled", "budget_tokens": budget}
+ return
+ create_kwargs["thinking"] = {"type": "adaptive"}
+ # Claude has no "minimal" effort level.
+ create_kwargs.setdefault("output_config", {})["effort"] = "low" if level == "minimal" else level
+
def _mapped_error(self, error: Exception, model: str) -> LLMError | None:
"""Normalize a provider exception into the shared taxonomy, or None when it isn't ours.
diff --git a/products/ai_observability/backend/llm/providers/openai.py b/products/ai_observability/backend/llm/providers/openai.py
index 898e2b853f88..5be139fd6a08 100644
--- a/products/ai_observability/backend/llm/providers/openai.py
+++ b/products/ai_observability/backend/llm/providers/openai.py
@@ -11,6 +11,7 @@ from django.conf import settings
import httpx
import openai
import posthoganalytics
+from openai import omit
from openai.types import CompletionUsage, ReasoningEffort
from openai.types.chat import ChatCompletionDeveloperMessageParam, ChatCompletionSystemMessageParam
from posthoganalytics.ai.openai import (
@@ -39,6 +40,7 @@ from products.ai_observability.backend.llm.types import (
AnalyticsContext,
CompletionRequest,
CompletionResponse,
+ ReasoningLevel,
StreamChunk,
Usage,
)
@@ -109,6 +111,17 @@ class OpenAIConfig:
"o3-mini",
]
+ @staticmethod
+ def supports_reasoning_level(model: str) -> bool:
+ return model.startswith(("gpt-5", "o3", "o4")) and "chat" not in model
+
+ @staticmethod
+ def reasoning_effort(model: str, level: ReasoningLevel) -> ReasoningEffort:
+ # Only the original GPT-5 models accept "minimal". Later models and the o-series reject it.
+ if level == "minimal" and not (model == "gpt-5" or model.startswith("gpt-5-")):
+ return "low"
+ return level
+
class OpenAIAdapter:
"""OpenAI provider implementing the unified Client interface."""
@@ -183,6 +196,11 @@ class OpenAIAdapter:
model=request.model,
messages=messages,
response_format=request.response_format,
+ reasoning_effort=(
+ OpenAIConfig.reasoning_effort(request.model, request.reasoning_level)
+ if request.reasoning_level and OpenAIConfig.supports_reasoning_level(request.model)
+ else omit
+ ),
**(self._build_analytics_kwargs(analytics, client)),
)
parsed = response.choices[0].message.parsed
@@ -380,7 +398,7 @@ Return ONLY the JSON object, no other text or markdown formatting."""
if supports_reasoning:
selected_effort: ReasoningEffort | None = None
if request.reasoning_level in ("minimal", "low", "medium", "high"):
- selected_effort = request.reasoning_level # type: ignore[assignment]
+ selected_effort = request.reasoning_level
elif reasoning_on:
selected_effort = OpenAIConfig.REASONING_EFFORT
stream = client.chat.completions.create(
diff --git a/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py b/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py
index 6a5c41d33003..a4fbe5b5ffbe 100644
--- a/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py
+++ b/products/ai_observability/backend/llm/providers/test/test_anthropic_structured_output.py
@@ -102,6 +102,52 @@ class TestStructuredOutputComplete:
assert "result" in schema["properties"]
assert "reason" in schema["properties"]
+ @parameterized.expand(
+ [
+ ("budget_model", "claude-haiku-4-5", "medium", {"type": "enabled", "budget_tokens": 4096}, None, 12288),
+ (
+ "dated_claude_4_0",
+ "claude-sonnet-4-20250514",
+ "low",
+ {"type": "enabled", "budget_tokens": 2048},
+ None,
+ 10240,
+ ),
+ ("adaptive_model", "claude-haiku-5-5", "high", {"type": "adaptive"}, "high", 16384),
+ ("adaptive_minimal_is_low", "claude-opus-4-7", "minimal", {"type": "adaptive"}, "low", 9216),
+ ("unlisted_newer_model", "claude-sonnet-9", "low", {"type": "adaptive"}, "low", 10240),
+ ("no_level", "claude-haiku-5-5", None, None, None, 8192),
+ ("claude_3", "claude-3-haiku-20240307", "high", None, None, 8192),
+ ]
+ )
+ @patch("products.ai_observability.backend.llm.providers.anthropic.settings")
+ @patch("products.ai_observability.backend.llm.providers.anthropic.anthropic.Anthropic")
+ def test_structured_output_applies_reasoning_level(
+ self,
+ _name,
+ model,
+ level,
+ expected_thinking,
+ expected_effort,
+ expected_max_tokens,
+ mock_anthropic_cls,
+ mock_settings,
+ ):
+ mock_client = MagicMock()
+ mock_anthropic_cls.return_value = mock_client
+ mock_client.messages.create.return_value = self._make_mock_response('{"result": true, "reason": "ok"}')
+ request = self._make_request(response_format=BooleanEvalResult)
+ request.model = model
+ request.reasoning_level = level
+
+ AnthropicAdapter().complete(request, api_key="sk-ant-test", analytics=AnalyticsContext(capture=False))
+
+ call_kwargs = mock_client.messages.create.call_args.kwargs
+ assert call_kwargs.get("thinking") == expected_thinking
+ assert call_kwargs["output_config"].get("effort") == expected_effort
+ assert call_kwargs["output_config"]["format"]["type"] == "json_schema"
+ assert call_kwargs["max_tokens"] == expected_max_tokens
+
@patch("products.ai_observability.backend.llm.providers.anthropic.settings")
@patch("products.ai_observability.backend.llm.providers.anthropic.anthropic.Anthropic")
def test_structured_output_parses_valid_json(self, mock_anthropic_cls, mock_settings):
diff --git a/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py b/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py
index 1d77981e95d7..aa92919c8818 100644
--- a/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py
+++ b/products/ai_observability/backend/llm/providers/test/test_openai_adapter.py
@@ -177,6 +177,39 @@ class TestOpenAIStreamOutputSchema:
assert "response_format" not in self._stream_kwargs(None)
+class TestOpenAICompleteReasoningEffort:
+ @parameterized.expand(
+ [
+ ("original_gpt5_keeps_minimal", "gpt-5-mini", "minimal", "minimal"),
+ ("later_gpt5_maps_minimal_to_low", "gpt-5.1", "minimal", "low"),
+ ("o_series_maps_minimal_to_low", "o4-mini", "minimal", "low"),
+ ("level_passes_through", "gpt-5.1", "high", "high"),
+ ("non_reasoning_model_omits_effort", "gpt-4.1", "high", None),
+ ("chat_variant_omits_effort", "gpt-5-chat-latest", "high", None),
+ ("no_level_omits_effort", "gpt-5-mini", None, None),
+ ]
+ )
+ def test_structured_completion_sends_supported_reasoning_effort(
+ self, _name: str, model: str, level: str | None, expected: str | None
+ ) -> None:
+ mock_client = MagicMock()
+ mock_client.beta.chat.completions.parse.return_value.choices = [MagicMock(message=MagicMock(parsed=None))]
+ request = CompletionRequest(
+ model=model,
+ system="s",
+ messages=[{"role": "user", "content": "x"}],
+ provider="openai",
+ response_format=_Verdict,
+ reasoning_level=level, # type: ignore[arg-type]
+ )
+
+ with patch("products.ai_observability.backend.llm.providers.openai.openai.OpenAI", return_value=mock_client):
+ OpenAIAdapter().complete(request, api_key="sk-test", analytics=AnalyticsContext(capture=False))
+
+ effort = mock_client.beta.chat.completions.parse.call_args.kwargs["reasoning_effort"]
+ assert (None if effort is openai.omit else effort) == expected
+
+
class TestOpenAIAdapterErrorMapping:
@pytest.fixture
def request_no_structured_output(self) -> CompletionRequest:
diff --git a/products/ai_observability/backend/llm/types.py b/products/ai_observability/backend/llm/types.py
index c0aec0240527..808eb04b3f64 100644
--- a/products/ai_observability/backend/llm/types.py
+++ b/products/ai_observability/backend/llm/types.py
@@ -5,6 +5,8 @@ from pydantic import BaseModel
from posthog.dataclasses import frozen
+ReasoningLevel = Literal["minimal", "low", "medium", "high"]
+
@dataclass
class Usage:
@@ -42,7 +44,7 @@ class CompletionRequest:
response_format: type[BaseModel] | None = None
output_schema: OutputSchema | None = None
thinking: bool = False
- reasoning_level: str | None = None
+ reasoning_level: ReasoningLevel | None = None
@dataclass
diff --git a/products/ai_observability/backend/models/evaluation_configs.py b/products/ai_observability/backend/models/evaluation_configs.py
index 2af1f2c76b11..46f2bf5dca78 100644
--- a/products/ai_observability/backend/models/evaluation_configs.py
+++ b/products/ai_observability/backend/models/evaluation_configs.py
@@ -30,6 +30,11 @@ class LLMJudgeConfig(BaseModel):
judge_method: Literal["llm", "decision"] | None = Field(
default=None, description="Judge method. Omit to keep the saved method; null restores automatic routing."
)
+ # Matches llm.types.ReasoningLevel; importing the llm package here would load provider SDKs at startup.
+ reasoning_level: Literal["minimal", "low", "medium", "high"] | None = Field(
+ default=None,
+ description="How much the judge model reasons before it answers. Null uses the model's default.",
+ )
@field_validator("prompt")
@classmethod
diff --git a/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx
index 0d8cbc35c1c8..c81b7c793f19 100644
--- a/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx
+++ b/products/ai_observability/frontend/evaluations/components/EvaluationJudgeModelPicker.tsx
@@ -4,28 +4,40 @@ import { ByokModelPickerNotice } from '../../ByokModelPickerNotice'
import { getModelPickerFooterLink, ModelPicker } from '../../ModelPicker'
import { modelPickerLogic } from '../../modelPickerLogic'
import { llmEvaluationLogic } from '../llmEvaluationLogic'
+import { EvaluationReasoningLevelButton } from './EvaluationReasoningLevelButton'
/** The judge model picker with its notices, for the models the current judge method can use. */
export function EvaluationJudgeModelPicker(): JSX.Element {
const { byokModelsLoading, providerKeysLoading } = useValues(modelPickerLogic)
- const { selectedModel, selectedModelName, selectedPickerProviderKeyId, usesDecisionModel, judgeModelGroups } =
- useValues(llmEvaluationLogic)
+ const {
+ selectedModel,
+ selectedModelName,
+ selectedPickerProviderKeyId,
+ usesDecisionModel,
+ judgeModelGroups,
+ selectedModelSupportsReasoning,
+ } = useValues(llmEvaluationLogic)
const { selectModelFromPicker } = useActions(llmEvaluationLogic)
const loading = byokModelsLoading || providerKeysLoading
return (
<div>
- <ModelPicker
- model={selectedModel}
- selectedProviderKeyId={selectedPickerProviderKeyId}
- onSelect={selectModelFromPicker}
- groups={judgeModelGroups}
- loading={loading}
- footerLink={getModelPickerFooterLink(judgeModelGroups.some((group) => !group.disabledReason))}
- selectedModelName={selectedModelName}
- data-attr="evaluation-model-selector"
- />
+ <div className="flex gap-2">
+ <div className="flex-1 min-w-0">
+ <ModelPicker
+ model={selectedModel}
+ selectedProviderKeyId={selectedPickerProviderKeyId}
+ onSelect={selectModelFromPicker}
+ groups={judgeModelGroups}
+ loading={loading}
+ footerLink={getModelPickerFooterLink(judgeModelGroups.some((group) => !group.disabledReason))}
+ selectedModelName={selectedModelName}
+ data-attr="evaluation-model-selector"
+ />
+ </div>
+ {selectedModelSupportsReasoning && <EvaluationReasoningLevelButton />}
+ </div>
<ByokModelPickerNotice forEvaluation />
{usesDecisionModel && !loading && !judgeModelGroups.some((group) => group.models.length > 0) && (
<p className="text-sm text-muted mt-2">
diff --git a/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx
index e84ceed4588f..1645b679a376 100644
--- a/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx
+++ b/products/ai_observability/frontend/evaluations/components/EvaluationModelPicker.stories.tsx
@@ -4,16 +4,22 @@ import { BindLogic } from 'kea'
import { useStorybookMocks } from '~/mocks/browser'
import { llmEvaluationLogic } from '../llmEvaluationLogic'
-import type { JudgeMethod } from '../types'
+import type { JudgeMethod, ReasoningLevel } from '../types'
import { EvaluationModelPicker } from './EvaluationModelPicker'
-const meta: Meta<{ method?: JudgeMethod; narrow?: boolean; empty?: boolean; nativeOpenAI?: boolean }> = {
+const meta: Meta<{
+ method?: JudgeMethod
+ narrow?: boolean
+ empty?: boolean
+ nativeOpenAI?: boolean
+ reasoningLevel?: ReasoningLevel
+}> = {
title: 'Scenes-App/LLM observability/EvaluationModelPicker',
component: EvaluationModelPicker,
parameters: {
layout: 'padded',
},
- render: ({ method = 'llm', narrow = false, empty = false, nativeOpenAI = false }) => {
+ render: ({ method = 'llm', narrow = false, empty = false, nativeOpenAI = false, reasoningLevel }) => {
useStorybookMocks({
get: {
'/api/environments/:team_id/llm_analytics/provider_keys/': {
@@ -35,6 +41,7 @@ const meta: Meta<{ method?: JudgeMethod; narrow?: boolean; empty?: boolean; nati
provider: 'OpenAI',
supports_chat: true,
supports_decisions: false,
+ supports_reasoning: true,
},
{
id: 'gpt-6-luna',
@@ -78,19 +85,25 @@ const meta: Meta<{ method?: JudgeMethod; narrow?: boolean; empty?: boolean; nati
name: 'Response quality',
enabled: false,
evaluation_type: 'llm_judge',
- evaluation_config: { prompt: 'Does the response answer the question?', judge_method: method },
+ evaluation_config: {
+ prompt: 'Does the response answer the question?',
+ judge_method: method,
+ reasoning_level: reasoningLevel,
+ },
output_type: 'boolean',
output_config: {},
conditions: [],
target: 'generation',
target_config: {},
- model_configuration: null,
+ model_configuration: reasoningLevel
+ ? { provider: 'openai', model: 'gpt-5-mini', provider_key_id: 'example-openai-key' }
+ : null,
},
},
})
return (
<BindLogic logic={llmEvaluationLogic} props={{ evaluationId: 'example-evaluation' }}>
- <div className={narrow ? 'max-w-lg' : 'max-w-4xl'}>
+ <div className={narrow ? 'w-[32rem]' : 'max-w-4xl'}>
<EvaluationModelPicker />
</div>
</BindLogic>
@@ -106,3 +119,6 @@ export const DecisionNarrow: Story = { args: { method: 'decision', narrow: true
export const NoDecisionModels: Story = { args: { method: 'decision', empty: true } }
export const NativeOpenAI: Story = { args: { method: 'decision', nativeOpenAI: true } }
export const NativeOpenAINarrow: Story = { args: { method: 'decision', nativeOpenAI: true, narrow: true } }
+export const ReasoningLevelNarrow: Story = {
+ args: { method: 'llm', nativeOpenAI: true, narrow: true, reasoningLevel: 'medium' },
+}
diff --git a/products/ai_observability/frontend/evaluations/components/EvaluationReasoningLevelButton.tsx b/products/ai_observability/frontend/evaluations/components/EvaluationReasoningLevelButton.tsx
new file mode 100644
index 000000000000..7a18e23cc9af
--- /dev/null
+++ b/products/ai_observability/frontend/evaluations/components/EvaluationReasoningLevelButton.tsx
@@ -0,0 +1,43 @@
+import { useActions, useValues } from 'kea'
+
+import { IconBrain } from '@posthog/icons'
+import { LemonButton } from '@posthog/lemon-ui'
+
+import { LemonMenu } from 'lib/lemon-ui/LemonMenu'
+
+import { llmEvaluationLogic } from '../llmEvaluationLogic'
+import type { ReasoningLevel } from '../types'
+
+const REASONING_LEVEL_OPTIONS: { value: ReasoningLevel | null; label: string }[] = [
+ { value: null, label: 'Model default' },
+ { value: 'minimal', label: 'Minimal' },
+ { value: 'low', label: 'Low' },
+ { value: 'medium', label: 'Medium' },
+ { value: 'high', label: 'High' },
+]
+
+export function EvaluationReasoningLevelButton(): JSX.Element {
+ const { reasoningLevel } = useValues(llmEvaluationLogic)
+ const { setReasoningLevel } = useActions(llmEvaluationLogic)
+
+ return (
+ <LemonMenu
+ placement="bottom-end"
+ items={REASONING_LEVEL_OPTIONS.map(({ value, label }) => ({
+ label,
+ active: value === reasoningLevel,
+ onClick: () => setReasoningLevel(value),
+ }))}
+ >
+ <LemonButton
+ type="secondary"
+ icon={<IconBrain />}
+ tooltip="Reasoning effort. Higher levels can judge harder cases better, but cost more and take longer."
+ aria-label="Reasoning effort"
+ data-attr="evaluation-reasoning-level"
+ >
+ {reasoningLevel && REASONING_LEVEL_OPTIONS.find((option) => option.value === reasoningLevel)?.label}
+ </LemonButton>
+ </LemonMenu>
+ )
+}
diff --git a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts
index ecbc2abbb9c7..5b58de8384bc 100644
--- a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts
+++ b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.test.ts
@@ -1954,6 +1954,47 @@ return result`,
)
})
+ it.each([
+ [true, 'high'],
+ [false, null],
+ ] as const)(
+ 'keeps the reasoning level only when the next model supports it (supportsReasoning=%s)',
+ async (supportsReasoning, expectedLevel) => {
+ await expectLogic(logic).toDispatchActions(['loadEvaluationSuccess'])
+ await expectLogic(keysLogic).toDispatchActions(['loadProviderKeysSuccess'])
+ await expectLogic(modelPickerLogic).toFinishAllListeners()
+ modelPickerLogic.actions.loadByokModelsSuccess([
+ {
+ id: 'example/reasoning-model',
+ name: 'example/reasoning-model',
+ description: '',
+ provider: 'OpenAI',
+ providerKeyId: 'key-1',
+ supportsReasoning: true,
+ },
+ {
+ id: 'example/next-model',
+ name: 'example/next-model',
+ description: '',
+ provider: 'OpenAI',
+ providerKeyId: 'key-1',
+ supportsReasoning,
+ },
+ ])
+ await expectLogic(logic, () =>
+ logic.actions.selectModelFromPicker('example/reasoning-model', 'key-1')
+ ).toFinishAllListeners()
+ logic.actions.setReasoningLevel('high')
+ expect(logic.values.selectedModelSupportsReasoning).toBe(true)
+
+ await expectLogic(logic, () =>
+ logic.actions.selectModelFromPicker('example/next-model', 'key-1')
+ ).toFinishAllListeners()
+
+ expect(logic.values.reasoningLevel).toBe(expectedLevel)
+ }
+ )
+
beforeEach(() => {
logic = llmEvaluationLogic({ evaluationId: 'new' })
logic.mount()
diff --git a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts
index a7794abd62c9..502d0b0cc659 100644
--- a/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts
+++ b/products/ai_observability/frontend/evaluations/llmEvaluationLogic.ts
@@ -78,6 +78,7 @@ import type {
LLMJudgeEvaluation,
JudgeMethod,
ModelConfiguration,
+ ReasoningLevel,
SentimentEvaluation,
} from './types'
@@ -352,6 +353,7 @@ export interface llmEvaluationLogicValues {
numericBoundsRequired: boolean
originalEvaluation: EvaluationConfig | null
outputConfigDrafts: Partial<Record<'boolean' | 'categorical' | 'numeric', EvaluationOutputConfig>>
+ reasoningLevel: ReasoningLevel | null
runsBackfill: EvaluationBackfillApi | null
runsBackfillId: string | null
runsBackfillLoading: boolean
@@ -368,6 +370,8 @@ export interface llmEvaluationLogicValues {
} | null
selectedModel: string
selectedModelName: string | undefined
+ selectedModelOption: ModelOption | null
+ selectedModelSupportsReasoning: boolean
selectedPickerProviderKeyId: string | null
sidePanelContext: SidePanelSceneContext | null
trendInsightUrl: string | null
@@ -543,6 +547,9 @@ export interface llmEvaluationLogicActions {
outputType: 'boolean' | 'categorical' | 'numeric'
previousEvaluation: EvaluationConfig | null
}
+ setReasoningLevel: (reasoningLevel: ReasoningLevel | null) => {
+ reasoningLevel: ReasoningLevel | null
+ }
setRunsBackfillId: (backfillId: string | null) => {
backfillId: string | null
}
@@ -597,8 +604,11 @@ export interface llmEvaluationLogicMeta {
originalEvaluation: EvaluationConfig | null,
evaluationId: string
) => boolean
- judgeMethod: (evaluation: EvaluationConfig | null, byokModels: ModelOption[]) => JudgeMethod
+ selectedModelOption: (evaluation: EvaluationConfig | null, byokModels: ModelOption[]) => ModelOption | null
+ judgeMethod: (evaluation: EvaluationConfig | null, selectedModelOption: ModelOption | null) => JudgeMethod
usesDecisionModel: (judgeMethod: JudgeMethod) => boolean
+ reasoningLevel: (evaluation: EvaluationConfig | null) => ReasoningLevel | null
+ selectedModelSupportsReasoning: (selectedModelOption: ModelOption | null, usesDecisionModel: boolean) => boolean
selectedModelName: (
byokModels: ModelOption[],
selectedModel: string,
@@ -694,6 +704,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([
setTriggerConditions: (conditions: EvaluationConditionSet[]) => ({ conditions }),
setModelConfiguration: (modelConfiguration: ModelConfiguration | null) => ({ modelConfiguration }),
setJudgeMethod: (judgeMethod: JudgeMethod) => ({ judgeMethod }),
+ setReasoningLevel: (reasoningLevel: ReasoningLevel | null) => ({ reasoningLevel }),
setEvaluationType: (evaluationType: EvaluationType) => ({ evaluationType }),
setEvaluationTarget: (target: EvaluationTarget) => ({ target }),
setSettleStrategy: (strategy: EvaluationSettleStrategy) => ({ strategy }),
@@ -1013,6 +1024,13 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([
state && isLLMJudgeEvaluation(state)
? { ...state, model_configuration: modelConfiguration }
: state,
+ setReasoningLevel: (state, { reasoningLevel }) =>
+ state && isLLMJudgeEvaluation(state)
+ ? {
+ ...state,
+ evaluation_config: { ...state.evaluation_config, reasoning_level: reasoningLevel },
+ }
+ : state,
setJudgeMethod: (state, { judgeMethod }) =>
state && isLLMJudgeEvaluation(state)
? {
@@ -1174,6 +1192,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([
setTriggerConditions: () => true,
setModelConfiguration: () => true,
setJudgeMethod: () => true,
+ setReasoningLevel: () => true,
setEvaluationType: () => true,
setEvaluationTarget: () => true,
setSettleStrategy: () => true,
@@ -1579,12 +1598,7 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([
if (!config) {
return
}
- const model = values.byokModels.find(
- (model) =>
- model.id === config.model &&
- toLLMProvider(model.provider) === config.provider &&
- (!config.provider_key_id || model.providerKeyId === config.provider_key_id)
- )
+ const model = values.selectedModelOption
const compatible = model
? values.judgeMethod === 'decision'
? model.supportsDecisions
@@ -1595,6 +1609,14 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([
if (!compatible) {
actions.setModelConfiguration(null)
}
+ if (values.reasoningLevel && !values.selectedModelSupportsReasoning) {
+ actions.setReasoningLevel(null)
+ }
+ },
+ setModelConfiguration: () => {
+ if (values.reasoningLevel && !values.selectedModelSupportsReasoning) {
+ actions.setReasoningLevel(null)
+ }
},
setEvaluationType: () => {
if (!values.isReportableEvaluation && values.activeTab === 'reports') {
@@ -1723,28 +1745,47 @@ export const llmEvaluationLogic = kea<llmEvaluationLogicType>([
},
],
- judgeMethod: [
+ selectedModelOption: [
(s) => [s.evaluation, s.byokModels],
- (evaluation: EvaluationConfig | null, models: ModelOption[]): JudgeMethod => {
+ (evaluation: EvaluationConfig | null, models: ModelOption[]): ModelOption | null => {
+ const config = evaluation?.model_configuration
+ if (!config) {
+ return null
+ }
+ return (
+ models.find(
+ (model) =>
+ model.id === config.model &&
+ toLLMProvider(model.provider) === config.provider &&
+ (!config.provider_key_id || model.providerKeyId === config.provider_key_id)
+ ) ?? null
+ )
+ },
+ ],
+ judgeMethod: [
+ (s) => [s.evaluation, s.selectedModelOption],
+ (evaluation: EvaluationConfig | null, model: ModelOption | null): JudgeMethod => {
if (isLLMJudgeEvaluation(evaluation) && evaluation.evaluation_config?.judge_method) {
return evaluation.evaluation_config.judge_method
}
- const config = evaluation?.model_configuration
- return config?.provider === 'system_one' ||
- (config?.provider === 'openrouter' &&
- models.some(
- (model) =>
- model.id === config.model &&
- toLLMProvider(model.provider) === 'openrouter' &&
- (!config.provider_key_id || model.providerKeyId === config.provider_key_id) &&
- model.supportsDecisions &&
- !model.supportsChat
- ))
+ const provider = evaluation?.model_configuration?.provider
+ return provider === 'system_one' ||
+ (provider === 'openrouter' && !!model?.supportsDecisions && !model.supportsChat)
? 'decision'
: 'llm'
},
],
usesDecisionModel: [(s) => [s.judgeMethod], (judgeMethod: JudgeMethod): boolean => judgeMethod === 'decision'],
+ reasoningLevel: [
+ (s) => [s.evaluation],
+ (evaluation: EvaluationConfig | null): ReasoningLevel | null =>
+ (isLLMJudgeEvaluation(evaluation) && evaluation.evaluation_config.reasoning_level) || null,
+ ],
+ selectedModelSupportsReasoning: [
+ (s) => [s.selectedModelOption, s.usesDecisionModel],
+ (model: ModelOption | null, usesDecisionModel: boolean): boolean =>
+ !usesDecisionModel && !!model?.supportsReasoning,
+ ],
// Evals always run on the team's own provider key, so only BYOK models can be the selected one.
selectedModelName: [
(s) => [s.byokModels, s.selectedModel, s.selectedPickerProviderKeyId],
diff --git a/products/ai_observability/frontend/evaluations/types.ts b/products/ai_observability/frontend/evaluations/types.ts
index 0f011c7cd732..348d86a96eb3 100644
--- a/products/ai_observability/frontend/evaluations/types.ts
+++ b/products/ai_observability/frontend/evaluations/types.ts
@@ -53,6 +53,7 @@ export interface EvaluationTargetConfig {
export type LLMJudgeEvaluationConfig = Extract<EvaluationApiEvaluationConfig, { prompt: string }>
export type JudgeMethod = NonNullable<LLMJudgeEvaluationConfig['judge_method']>
+export type ReasoningLevel = NonNullable<LLMJudgeEvaluationConfig['reasoning_level']>
export interface HogEvaluationConfig {
source: string
diff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.ts
index 465014a8e56b..492fdfa88426 100644
--- a/products/ai_observability/frontend/generated/api.schemas.ts
+++ b/products/ai_observability/frontend/generated/api.schemas.ts
@@ -2347,6 +2347,11 @@ export type EvaluationApiEvaluationConfig =
* @nullable
*/
judge_method?: 'llm' | 'decision' | null
+ /**
+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.
+ * @nullable
+ */
+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null
}
| {
/**
@@ -2676,6 +2681,11 @@ export type PatchedEvaluationApiEvaluationConfig =
* @nullable
*/
judge_method?: 'llm' | 'decision' | null
+ /**
+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.
+ * @nullable
+ */
+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null
}
| {
/**
diff --git a/products/ai_observability/frontend/generated/api.zod.ts b/products/ai_observability/frontend/generated/api.zod.ts
index 7f11e919e857..91e12dbbdce9 100644
--- a/products/ai_observability/frontend/generated/api.zod.ts
+++ b/products/ai_observability/frontend/generated/api.zod.ts
@@ -620,6 +620,12 @@ export const EvaluationsCreateBody = /* @__PURE__ */ zod
.enum(['llm', 'decision'])
.nullish()
.describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),
+ reasoning_level: zod
+ .enum(['minimal', 'low', 'medium', 'high'])
+ .nullish()
+ .describe(
+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
+ ),
}),
zod.object({
source: zod
@@ -1015,6 +1021,12 @@ export const EvaluationsUpdateBody = /* @__PURE__ */ zod
.enum(['llm', 'decision'])
.nullish()
.describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),
+ reasoning_level: zod
+ .enum(['minimal', 'low', 'medium', 'high'])
+ .nullish()
+ .describe(
+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
+ ),
}),
zod.object({
source: zod
@@ -1313,6 +1325,12 @@ export const EvaluationsPartialUpdateBody = /* @__PURE__ */ zod
.enum(['llm', 'decision'])
.nullish()
.describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),
+ reasoning_level: zod
+ .enum(['minimal', 'low', 'medium', 'high'])
+ .nullish()
+ .describe(
+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
+ ),
}),
zod.object({
source: zod
diff --git a/products/ai_observability/frontend/modelPickerLogic.test.ts b/products/ai_observability/frontend/modelPickerLogic.test.ts
index f511b858db92..1ada56d37351 100644
--- a/products/ai_observability/frontend/modelPickerLogic.test.ts
+++ b/products/ai_observability/frontend/modelPickerLogic.test.ts
@@ -115,6 +115,7 @@ describe('modelPickerLogic', () => {
isRecommended: true,
supportsDecisions: false,
supportsChat: true,
+ supportsReasoning: false,
providerKeyId: 'key-1',
}))
)
@@ -291,6 +292,7 @@ describe('modelPickerLogic', () => {
providerKeyId: 'key-2',
supportsDecisions: false,
supportsChat: true,
+ supportsReasoning: false,
}))
)
expect(logic.values.providerModelGroups.find((g) => g.providerKeyId === 'key-1')).toEqual({
diff --git a/products/ai_observability/frontend/modelPickerLogic.ts b/products/ai_observability/frontend/modelPickerLogic.ts
index cd12b23d87b8..de80f4ea91ad 100644
--- a/products/ai_observability/frontend/modelPickerLogic.ts
+++ b/products/ai_observability/frontend/modelPickerLogic.ts
@@ -22,6 +22,7 @@ export interface ModelOption {
isRecommended?: boolean
supportsDecisions?: boolean
supportsChat?: boolean
+ supportsReasoning?: boolean
}
export interface ProviderModelGroup {
@@ -289,6 +290,7 @@ export const modelPickerLogic = kea<modelPickerLogicType>([
is_recommended?: boolean
supports_decisions?: boolean
supports_chat?: boolean
+ supports_reasoning?: boolean
})[]
return rawModels.map((m) => ({
id: m.id,
@@ -298,6 +300,7 @@ export const modelPickerLogic = kea<modelPickerLogicType>([
isRecommended: m.is_recommended ?? false,
supportsDecisions: m.supports_decisions ?? false,
supportsChat: m.supports_chat ?? !m.supports_decisions,
+ supportsReasoning: m.supports_reasoning ?? false,
providerKeyId: key.id,
}))
} catch {
diff --git a/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts b/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts
index 082ced8aca09..b2c58c21af5e 100644
--- a/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts
+++ b/products/ai_observability/frontend/playground/llmPlaygroundLogic.test.ts
@@ -800,6 +800,7 @@ describe('llmPlaygroundLogic', () => {
providerKeyId: 'key-1',
supportsDecisions: false,
supportsChat: true,
+ supportsReasoning: false,
}))
)
})
diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.ts
index 8bac3d95957f..0d83508e42eb 100644
--- a/products/desktop/packages/api-client/src/generated.ts
+++ b/products/desktop/packages/api-client/src/generated.ts
@@ -9228,6 +9228,9 @@ export namespace Schemas {
judge_method?:
| (("llm" | "decision") | null)
| undefined;
+ reasoning_level?:
+ | (("minimal" | "low" | "medium" | "high") | null)
+ | undefined;
}
| {
/**
@@ -19473,6 +19476,9 @@ export namespace Schemas {
| {
prompt: string;
judge_method?: (("llm" | "decision") | null) | undefined;
+ reasoning_level?:
+ | (("minimal" | "low" | "medium" | "high") | null)
+ | undefined;
}
| { source: string }
| Partial<{ source: "user_messages" }>;
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 9c309a51aff1..0594669f6289 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -45312,6 +45312,11 @@ export namespace Schemas {
* @nullable
*/
judge_method?: 'llm' | 'decision' | null;
+ /**
+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.
+ * @nullable
+ */
+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null;
} | {
/**
* Hog source code. Return a boolean, finite number, or category keys matching output_type. Categorical single selection accepts one key or a one-item list; multiple selection accepts a list, including []. Return null only for allowed N/A. Output settings determine which boolean counts as a failure.
@@ -87677,6 +87682,11 @@ export namespace Schemas {
* @nullable
*/
judge_method?: 'llm' | 'decision' | null;
+ /**
+ * How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default.
+ * @nullable
+ */
+ reasoning_level?: 'minimal' | 'low' | 'medium' | 'high' | null;
} | {
/**
* Hog source code. Return a boolean, finite number, or category keys matching output_type. Categorical single selection accepts one key or a one-item list; multiple selection accepts a list, including []. Return null only for allowed N/A. Output settings determine which boolean counts as a failure.
diff --git a/services/mcp/src/generated/ai_observability/api.ts b/services/mcp/src/generated/ai_observability/api.ts
index c4cda045eab7..7bdbe19643b3 100644
--- a/services/mcp/src/generated/ai_observability/api.ts
+++ b/services/mcp/src/generated/ai_observability/api.ts
@@ -1435,6 +1435,12 @@ export const EvaluationsCreateBody = () => zod
.enum(['llm', 'decision'])
.nullish()
.describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),
+ reasoning_level: zod
+ .enum(['minimal', 'low', 'medium', 'high'])
+ .nullish()
+ .describe(
+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
+ ),
}),
zod.object({
source: zod
@@ -1910,6 +1916,12 @@ export const EvaluationsPartialUpdateBody = () => zod
.enum(['llm', 'decision'])
.nullish()
.describe('Judge method. Omit to keep the saved method; null restores automatic routing.'),
+ reasoning_level: zod
+ .enum(['minimal', 'low', 'medium', 'high'])
+ .nullish()
+ .describe(
+ "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
+ ),
}),
zod.object({
source: zod
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json
index 7f94872293d0..2211a7d9905a 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-create.json
@@ -79,6 +79,18 @@
"description": "Evaluation criteria for the LLM judge. Describe what makes a good vs bad response.",
"minLength": 1,
"type": "string"
+ },
+ "reasoning_level": {
+ "anyOf": [
+ {
+ "enum": ["minimal", "low", "medium", "high"],
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
}
},
"required": ["prompt"],
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json
index 5537211a1165..d4eba03e9095 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-evaluation-update.json
@@ -78,6 +78,18 @@
"description": "Evaluation criteria for the LLM judge. Describe what makes a good vs bad response.",
"minLength": 1,
"type": "string"
+ },
+ "reasoning_level": {
+ "anyOf": [
+ {
+ "enum": ["minimal", "low", "medium", "high"],
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "How much the judge model reasons before it answers. Only Anthropic and OpenAI reasoning models with the 'llm' judge method support it. Omit to keep the saved level; null uses the model's default."
}
},
"required": ["prompt"],