Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): add score history breakdowns (#122787)

Radu Raiceaa9a57fa67da1
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 9f808559ebb1..97ac2d64a98b 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -633,6 +633,19 @@ Saving a new configuration checks the version observed when editing started. A c
The **More → Create new version** action can create an unchanged configuration version after confirmation. Save or discard a dirty draft before using this action.
The **Offline evals history** button in the editor opens a scorer's experiment timeline.
With the offline feature enabled, a scorer timeline offers upload state, source, date range, comparison, and exact version filters.
+The filters use compact controls. Selected and comparison period timestamps use the project timezone, matching the date range boundaries, chart labels, and coverage dates; hover a timestamp for full timezone details.
+**Break down by** splits score history by Model, Prompt version, Application version, Suite, or Run source.
+Model uses the item's optional `model_version`, falling back to the experiment's `model_version` when omitted or null.
+The scorer history `model_version` filter uses this resolved item model, with or without a breakdown. Scores and coverage include only matching items, even when the experiment's default model differs. Experiment list filters continue to match the experiment's model.
+The other breakdowns use experiment fields. Item reads expose the resolved model.
+Each point summarizes one experiment, scorer version, and breakdown value, with coverage and missing-result counts for that group.
+Unspecified values appear as **Not specified**. Different scorer versions, categories, and comparison periods remain separate.
+History pagination counts these groups when a breakdown is selected, so a page can contain multiple points from one experiment.
+Coverage aggregation includes only the page's experiment/breakdown groups, including their items without results.
+Pagination supports Unicode breakdown values up to the full 255-character identifier limit.
+The graph and table show the current page; partial history may omit other breakdown values.
+The breakdown is stored in the URL. Custom metadata breakdowns are not supported.
+
Its experiment table keeps names and coverage on one line, shows execution time in its own column, and opens run details with the arrow beside each name.
Compare against a previous or custom period.
Custom comparison periods ending now retain that end when their URL is shared or reloaded.
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index ff89099bb725..e2e3948b7201 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -113,17 +113,21 @@ snapshots:
ai-observability-offline-experiments-score-trends--numeric-passing-rule--light:
hash: v1.k794b7964.e5367e5503e0e7a14586290abedd1f8f25d9a938013d330a9b3783a9377e32cf.2SZhg6Ri2BlDEHtHrLx7RAbuPEUQFdP-D1FP3_tOPNI
ai-observability-offline-experiments-scorer-history--default--dark:
- hash: v1.k794b7964.de382c36f53d24e416c8a816a1cf194c9ac751b5eebfc7a373932035fecc1acb._coWDzE6lvhhThqoT_ASaYs0iiqQvERpqiwT5KhGunM
+ hash: v1.k794b7964.fa9847952affeb01cb37985844b373ce28cef402023df05737f8394189530b67.ccaDPhq_JGTa8n5cMvHqbAj9f2hkoNU9nUhCwFhfGXk
ai-observability-offline-experiments-scorer-history--default--light:
- hash: v1.k794b7964.15cbf9e2bdbbe89d7661c94ccd15d7732d828f564add36cdc067757e03a4a061.A0kjkj1ghKYsLEpla_XwYBMETI7KMu6YOB5BCiLbaxA
+ hash: v1.k794b7964.cc570ccb3a6b69c3e5a3add68e0b2faa3416bb81478ec97a3c763f988f582b67.Xbu9IZQNsowPb-o9i_RgTtvdOrXgTcERQBbaznPxFpc
ai-observability-offline-experiments-scorer-history--empty-history--dark:
- hash: v1.k794b7964.7fc63300ab3d7bf38a3058a285f6ecc6e36ffaae97b532deef09d13ed6cbafca.EaYco8jti78XKAqv4PEAm_z3n3MvWyTgBGVRhkVD5do
+ hash: v1.k794b7964.ae856e0b664f4e0029c23628c5e85098f42997e0526f24806e67dd3b433e6e7c.QuZHln7HMBUPuLH0qF-GLrrlmsgihcR5T4lBgf9UVfA
ai-observability-offline-experiments-scorer-history--empty-history--light:
- hash: v1.k794b7964.95f6a34d52a423471450e0a21f00dd8af7cfaf36281b07c02323b79d05e9fb3f.1cb98HkzcmAOIVK-QmbF1Yykej6T_sSVSuWGwRoUj-c
+ hash: v1.k794b7964.cda80a0c7f01b7e8c3ed4e811c189ccec024f1951abb8818167c431588e08ad4.6R0HRb2No1gsnW3c6FVXJpsKFmDRsRtI8b-TM7OzBvM
+ ai-observability-offline-experiments-scorer-history--model-breakdown--dark:
+ hash: v1.k794b7964.3232e0e44e04b8f402982aedb044a5ab7ef0bb8f666029eaceaafd342343df9c.UsoEzw4c1A3lspMlDgPae58EHaBJund2nYnv_OxtN1U
+ ai-observability-offline-experiments-scorer-history--model-breakdown--light:
+ hash: v1.k794b7964.81a77adc5d7de5fdb154ff4a3d35c2a3ba834a264da0708a6735acdd40c81269.oWsL-dscrLGH1bw3KivQnoZoOaN_hWDRWyUVROtLwsc
ai-observability-offline-experiments-scorer-history--narrow--dark:
- hash: v1.k794b7964.2d60979cab1b756742b3fef740e8c13a9b688f3eca76ddf7ae73dc08f2f58e64.48EDAeGamPjXu5HwCBjlgx4Z6130i2a_6kqwJ25p9jw
+ hash: v1.k794b7964.542a493a106271a9e78d761dd9fc7cc42108fff9c7336526ec687b98194d1063.t1b0Xq-37lNeB72sxqocIUjc_0vms7Cwa-wwyH9ub-M
ai-observability-offline-experiments-scorer-history--narrow--light:
- hash: v1.k794b7964.4842515b727957419dffc97a061cd561a640c5b3d12357415b2c6b0c0a5b2729.7EOWporG0PIqDbpe8ArXsT-fl8w_hNdckYUgKv4uHDY
+ hash: v1.k794b7964.ccfb0305fe7765645891a6e299e61794891ba11c48c093cdb498729210949a62.plGrGJumAXp8oFjajN09fPeLcqQpAZh9nUsT9A6uByw
ai-observability-scorer-editor--boolean-detector--dark:
hash: v1.k794b7964.bf295f71ee0750c9703c615cbc53f39849426b32b354fbf8ef0c8915f7f42e57.BVPu0uJuGhNx1_q4FgZ7vDXRQ7IllIygH2Hi6qKoTZw
ai-observability-scorer-editor--boolean-detector--light:
@@ -9173,9 +9177,9 @@ snapshots:
scenes-app-ai-observability-offline-experiments-overview--partial-history--light:
hash: v1.k794b7964.302ca88f07d59cdabb3dd8b286b13ff39c03b318a35bca99405683c8e3789d64.-Jd-kXQ46NBmqRsF90-2Y23EI8pzdxvbU9FADQWauYQ
scenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--dark:
- hash: v1.k794b7964.4c450840d8d01ddb2eb4817e1fea7f7fdbd288e93ea3c375555b4ceb1ae08311.jwIcdD5ZnyQniw64ArTCvFuX43hlRtSDbHAIU6qowFU
+ hash: v1.k794b7964.c475e74caf460fcfc280bd6983fedcdca879dcf011e058d9733b17ace2f0a348.nMKBGsSpB3g3PQXgiziItiY76qLFWGlhN3GLJlzBMUA
scenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--light:
- hash: v1.k794b7964.5b69ff2c107ab0450df0d7495d19da2d19293d8fe3b453224845365f8b402f3f.UkyFocgNR0N1T0QWFTOa6HcQYW4BdRMv5_qSzgjjFmI
+ hash: v1.k794b7964.568583aab6024942a42b957f625ce22201ee919609bc8ffa1b37e3bb90d0c0f7.c0AsgQZRr9uq6lC9Q3pghFN6ckTHFNdPYhnocXJ0Oig
scenes-app-ai-observability-offline-experiments-overview--recent-experiments--dark:
hash: v1.k794b7964.73de5e6dff8607fbad95d027b0b2856095e68f28922faa4f06315f6338c8012f.YbQpwN5pANDtCZDWgkZ8z9-XfMnI8dB5_HsfibRZ1z0
scenes-app-ai-observability-offline-experiments-overview--recent-experiments--light:
diff --git a/products/ai_observability/backend/api/offline_experiment_read_serializers.py b/products/ai_observability/backend/api/offline_experiment_read_serializers.py
index e2c138a1f291..8316e4b74af8 100644
--- a/products/ai_observability/backend/api/offline_experiment_read_serializers.py
+++ b/products/ai_observability/backend/api/offline_experiment_read_serializers.py
@@ -4,6 +4,7 @@ from uuid import UUID
from drf_spectacular.utils import extend_schema_serializer
from rest_framework import serializers
+from rest_framework_dataclasses.fields import EnumField
from rest_framework_dataclasses.serializers import DataclassSerializer
from posthog.api.shared import UserBasicSerializer
@@ -24,6 +25,7 @@ from products.ai_observability.backend.models.score_definitions import ScoreDefi
from products.ai_observability.backend.offline_evaluation_read_types import (
MAX_READ_SCORER_VERSIONS,
MAX_RESULT_CELL_ITEMS,
+ OfflineHistoryBreakdown,
OfflineReadQuery,
OfflineResultCellQuery,
)
@@ -211,6 +213,18 @@ class OfflineExperimentQuerySerializer(OfflineHistoryQuerySerializer):
fields = [*OfflineHistoryQuerySerializer.Meta.fields, "scorer_definition_id"]
+class OfflineScorerHistoryQuerySerializer(OfflineHistoryQuerySerializer):
+ breakdown = EnumField(
+ enum_class=OfflineHistoryBreakdown,
+ choices=OfflineHistoryBreakdown.choices,
+ required=False,
+ help_text="Split history by this property. Model uses each item's model, falling back to the experiment's model.",
+ )
+
+ class Meta(OfflineHistoryQuerySerializer.Meta):
+ fields = [*OfflineHistoryQuerySerializer.Meta.fields, "breakdown"]
+
+
class OfflineExperimentReadSerializer(serializers.Serializer):
created_by = UserBasicSerializer(
read_only=True, allow_null=True, help_text="User who submitted the experiment, or null for service credentials."
@@ -310,6 +324,9 @@ class OfflineResultCellsSerializer(serializers.Serializer):
class OfflineItemReadSerializer(serializers.Serializer):
+ model_version = serializers.CharField(
+ allow_null=True, required=False, help_text="Model used for the item, falling back to the experiment's model."
+ )
id = serializers.UUIDField(help_text="Stable item UUID.")
experiment_id = serializers.UUIDField(help_text="Owning experiment UUID.")
case_key = serializers.CharField(allow_null=True, help_text="Optional stable case identifier.")
@@ -427,6 +444,11 @@ class OfflineScorerSummarySerializer(serializers.Serializer):
class OfflineHistoryPointSerializer(serializers.Serializer):
+ breakdown_value = serializers.CharField(
+ allow_null=True,
+ required=False,
+ help_text="Breakdown value for this summary; null when unspecified or no breakdown is selected.",
+ )
experiment = OfflineExperimentReadSerializer(help_text="Experiment execution and cohort context.")
summary = OfflineScorerSummarySerializer(help_text="Complete summary for one experiment and scorer version.")
diff --git a/products/ai_observability/backend/api/offline_experiment_reads.py b/products/ai_observability/backend/api/offline_experiment_reads.py
index 9d66a168c018..1610b26dd242 100644
--- a/products/ai_observability/backend/api/offline_experiment_reads.py
+++ b/products/ai_observability/backend/api/offline_experiment_reads.py
@@ -31,7 +31,6 @@ from products.ai_observability.backend.api.offline_experiment_read_serializers i
OfflineExperimentQuerySerializer,
OfflineExperimentReadSerializer,
OfflineHistoryPageSerializer,
- OfflineHistoryQuerySerializer,
OfflineItemPageSerializer,
OfflineItemPayloadReadSerializer,
OfflineItemReadSerializer,
@@ -40,6 +39,7 @@ from products.ai_observability.backend.api.offline_experiment_read_serializers i
OfflineResultPageSerializer,
OfflineResultPayloadReadSerializer,
OfflineResultQuerySerializer,
+ OfflineScorerHistoryQuerySerializer,
OfflineSummaryPageSerializer,
OfflineSummaryQuerySerializer,
)
@@ -253,7 +253,7 @@ class OfflineExperimentReadViewSet(OfflineEvaluationReadViewSet):
class OfflineScorerViewSet(OfflineEvaluationReadViewSet):
@validated_request(
- query_serializer=OfflineHistoryQuerySerializer,
+ query_serializer=OfflineScorerHistoryQuerySerializer,
operation_id="ai_observability_offline_scorers_history_list",
parameters=[OpenApiParameter("id", UUID, OpenApiParameter.PATH, description="Scorer definition UUID.")],
responses={200: OfflineHistoryPageSerializer, **READ_ERROR_RESPONSES},
diff --git a/products/ai_observability/backend/api/offline_experiment_serializers.py b/products/ai_observability/backend/api/offline_experiment_serializers.py
index fc7c9ec7204e..01667456e233 100644
--- a/products/ai_observability/backend/api/offline_experiment_serializers.py
+++ b/products/ai_observability/backend/api/offline_experiment_serializers.py
@@ -247,6 +247,9 @@ class ExperimentSubmissionSerializer(_StrictDataclassSerializer[ExperimentSubmis
class ItemSubmissionSerializer(_StrictDataclassSerializer[ItemSubmission]):
id = _UUIDField(help_text="Caller-generated UUID for one input/output execution. Reuse for exact retries.")
+ model_version = _identifier_field(
+ "Model used for this item. Omitted or null values inherit the experiment's model."
+ )
case_key = _identifier_field("Stable case identifier for matching inputs across experiments.")
trial = _identifier_field("Identifier for a repeated execution of the same case.")
dataset_item_identifier = _identifier_field("Stable item identifier in an external dataset.")
diff --git a/products/ai_observability/backend/api/test/test_offline_experiment_reads.py b/products/ai_observability/backend/api/test/test_offline_experiment_reads.py
index 5a85684b44da..d360e33945c6 100644
--- a/products/ai_observability/backend/api/test/test_offline_experiment_reads.py
+++ b/products/ai_observability/backend/api/test/test_offline_experiment_reads.py
@@ -114,6 +114,19 @@ class TestOfflineExperimentReads(APIBaseTest):
self._history(),
]
+ def test_history_model_breakdown_uses_the_item_model(self) -> None:
+ self.experiment.model_version = "model-a"
+ self.experiment.save(update_fields=["model_version"])
+ self.item.model_version = "model-b"
+ self.item.save(update_fields=["model_version"])
+ response = self.client.get(self._history(), {"breakdown": "model_version", "model_version": "model-b"})
+ self.assertEqual(response.status_code, status.HTTP_200_OK, response.data)
+ self.assertEqual(response.data["results"][0]["breakdown_value"], "model-b")
+ self.assertEqual(response.data["results"][0]["summary"]["pass_rate"], 1.0)
+ response = self.client.get(self._history(), {"breakdown": "model_version", "model_version": "model-a"})
+ self.assertEqual(response.status_code, status.HTTP_200_OK, response.data)
+ self.assertEqual(response.data["count"], 0)
+
def test_author_filter_applies_to_experiments_and_history(self) -> None:
self.user.hedgehog_config = {
"version": 2,
@@ -531,6 +544,7 @@ class TestOfflineExperimentReads(APIBaseTest):
(self._endpoint(), {"limit": 101}, "limit"),
(self._endpoint(), {"created_by": "invalid"}, "created_by"),
(self._history(), {"created_by": 0}, "created_by"),
+ (self._history(), {"breakdown": "unsupported"}, "breakdown"),
(self._history(), {"created_by": self.user.id, "created_by_is_null": "true"}, "created_by"),
(self._endpoint(f"{self.experiment.id}/"), {"limit": 1}, "limit"),
(self._endpoint(f"{self.experiment.id}/items/{self.item.id}/"), {"limit": 1}, "limit"),
diff --git a/products/ai_observability/backend/migrations/0068_offline_item_model_version.py b/products/ai_observability/backend/migrations/0068_offline_item_model_version.py
new file mode 100644
index 000000000000..b260f2765fc4
--- /dev/null
+++ b/products/ai_observability/backend/migrations/0068_offline_item_model_version.py
@@ -0,0 +1,17 @@
+# Generated by Django 5.2.18 on 2026-10-23 14:27
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("ai_observability", "0067_scorer_short_name"),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name="offlineexperimentitem",
+ name="model_version",
+ field=models.CharField(blank=True, max_length=255, null=True),
+ ),
+ ]
diff --git a/products/ai_observability/backend/migrations/max_migration.txt b/products/ai_observability/backend/migrations/max_migration.txt
index eb7e75cc5f34..652d1094d6d4 100644
--- a/products/ai_observability/backend/migrations/max_migration.txt
+++ b/products/ai_observability/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0067_scorer_short_name
+0068_offline_item_model_version
diff --git a/products/ai_observability/backend/models/offline_evaluations.py b/products/ai_observability/backend/models/offline_evaluations.py
index c2304a7c8a61..a744b00363ab 100644
--- a/products/ai_observability/backend/models/offline_evaluations.py
+++ b/products/ai_observability/backend/models/offline_evaluations.py
@@ -123,6 +123,7 @@ class OfflineExperimentItem(OfflinePayloadOwner):
experiment = models.ForeignKey(OfflineExperiment, on_delete=models.CASCADE, related_name="items", db_index=False)
case_key = models.CharField(max_length=255, null=True, blank=True)
trial = models.CharField(max_length=255, null=True, blank=True)
+ model_version = models.CharField(max_length=255, null=True, blank=True)
dataset_item_identifier = models.CharField(max_length=255, null=True, blank=True)
dataset_item_version_identifier = models.CharField(max_length=255, null=True, blank=True)
dataset_item_version = models.ForeignKey(
diff --git a/products/ai_observability/backend/offline_evaluation_fingerprint.py b/products/ai_observability/backend/offline_evaluation_fingerprint.py
index 9117ebc6c755..5118001e74a3 100644
--- a/products/ai_observability/backend/offline_evaluation_fingerprint.py
+++ b/products/ai_observability/backend/offline_evaluation_fingerprint.py
@@ -27,5 +27,8 @@ def _normalize(value: object) -> object:
def submission_fingerprint(submission: ExperimentSubmission | ItemSubmission | ResultSubmission) -> str:
# This format must stay stable across deployments and after payload deletion.
- content = json.dumps(_normalize(asdict(submission)), sort_keys=True, separators=(",", ":"), allow_nan=False)
+ fields = asdict(submission)
+ if isinstance(submission, ItemSubmission) and submission.model_version is None:
+ fields.pop("model_version")
+ content = json.dumps(_normalize(fields), sort_keys=True, separators=(",", ":"), allow_nan=False)
return hashlib.sha256(b"offline-evaluation:v1:" + content.encode("utf-8")).hexdigest()
diff --git a/products/ai_observability/backend/offline_evaluation_read_service.py b/products/ai_observability/backend/offline_evaluation_read_service.py
index 8c54fd8bc1fb..1e74790f33ef 100644
--- a/products/ai_observability/backend/offline_evaluation_read_service.py
+++ b/products/ai_observability/backend/offline_evaluation_read_service.py
@@ -10,7 +10,8 @@ from typing import TYPE_CHECKING, cast
from uuid import UUID
from django.db import connections
-from django.db.models import Case, Count, Exists, F, OuterRef, Q, When, prefetch_related_objects
+from django.db.models import Case, Count, Exists, F, OuterRef, Q, Value, When, prefetch_related_objects
+from django.db.models.functions import Coalesce
from posthog.dataclasses import frozen
@@ -27,6 +28,7 @@ from products.ai_observability.backend.offline_evaluation_read_types import (
OfflineCategorySummary,
OfflineExperimentAuthor,
OfflineExperimentRead,
+ OfflineHistoryBreakdown,
OfflineHistoryPoint,
OfflineItemPage,
OfflineItemRead,
@@ -48,15 +50,23 @@ from products.ai_observability.backend.read_pagination import CursorPage, cursor
from products.ai_observability.backend.resource_tags import ResourceTags
if TYPE_CHECKING:
+ from typing import Annotated, TypedDict
+
from django.db.models import QuerySet
+ from django_stubs_ext import Annotations
+
from products.access_control.backend.facade.user_access_control import UserAccessControl
+ class _ItemAnnotations(TypedDict):
+ resolved_model_version: str | None
+
@frozen
class _SummaryIdentity:
experiment_id: UUID
version_id: UUID
+ breakdown_value: str = ""
@frozen
@@ -94,6 +104,7 @@ class _TimePosition:
started_at: datetime
experiment_id: UUID
version_id: UUID | None
+ breakdown_value: str = ""
class OfflineEvaluationReadService:
@@ -105,8 +116,14 @@ class OfflineEvaluationReadService:
def _experiments(self) -> QuerySet[OfflineExperiment]:
return OfflineExperiment.objects.for_team(self.team_id).select_related("created_by")
- def _items(self) -> QuerySet[OfflineExperimentItem]:
- return OfflineExperimentItem.objects.for_team(self.team_id)
+ @staticmethod
+ def _model_expression(*, item_prefix: str = "") -> Coalesce:
+ return Coalesce(F(f"{item_prefix}model_version"), F(f"{item_prefix}experiment__model_version"))
+
+ def _items(self) -> QuerySet[Annotated[OfflineExperimentItem, Annotations[_ItemAnnotations]]]:
+ return OfflineExperimentItem.objects.for_team(self.team_id).annotate(
+ resolved_model_version=self._model_expression()
+ )
def _definitions(self) -> QuerySet[ScoreDefinition]:
definitions = ScoreDefinition.objects.filter(team_id=self.team_id)
@@ -135,7 +152,9 @@ class OfflineEvaluationReadService:
if versions.count() != len(query.scorer_version_ids):
raise OfflineEvaluationNotFound
- def _results(self, query: OfflineReadQuery | None = None) -> QuerySet[OfflineEvaluationResult]:
+ def _results(
+ self, query: OfflineReadQuery | None = None, *, model_version: str | None = None
+ ) -> QuerySet[OfflineEvaluationResult]:
results = OfflineEvaluationResult.objects.for_team(self.team_id).filter(
scorer_definition__in=self._definitions()
)
@@ -144,6 +163,10 @@ class OfflineEvaluationReadService:
results = results.filter(scorer_definition_id=query.scorer_definition_id)
if query.scorer_version_ids:
results = results.filter(scorer_version_id__in=query.scorer_version_ids)
+ if model_version is not None:
+ return results.alias(resolved_model_version=self._model_expression(item_prefix="item__")).filter(
+ resolved_model_version=model_version
+ )
return results
def _require_experiment(self, experiment_id: UUID) -> OfflineExperiment:
@@ -152,7 +175,9 @@ class OfflineEvaluationReadService:
raise OfflineEvaluationNotFound
return experiment
- def _require_item(self, experiment_id: UUID, item_id: UUID) -> OfflineExperimentItem:
+ def _require_item(
+ self, experiment_id: UUID, item_id: UUID
+ ) -> Annotated[OfflineExperimentItem, Annotations[_ItemAnnotations]]:
item = self._items().filter(experiment_id=experiment_id, id=item_id).first()
if item is None:
raise OfflineEvaluationNotFound
@@ -184,12 +209,13 @@ class OfflineEvaluationReadService:
"dataset_identifier",
"dataset_revision_identifier",
"application_version",
- "model_version",
"prompt_version",
):
value = getattr(query, field)
if value is not None:
experiments = experiments.filter(**{field: value})
+ if query.model_version is not None and not history:
+ experiments = experiments.filter(model_version=query.model_version)
if query.scorer_definition_id is not None or query.scorer_version_ids:
experiments = experiments.filter(Exists(self._results(query).filter(item__experiment_id=OuterRef("id"))))
return experiments
@@ -198,6 +224,8 @@ class OfflineEvaluationReadService:
filters = asdict(query)
filters.pop("cursor")
filters.pop("limit")
+ if query.breakdown is None:
+ filters.pop("breakdown")
content = json.dumps([self.team_id, self.can_read_scores, scope, filters], default=str, sort_keys=True)
return hashlib.sha256(content.encode()).hexdigest()[:24]
@@ -223,7 +251,7 @@ class OfflineEvaluationReadService:
raise OfflineEvaluationValidationError("cursor", "Provide a valid continuation cursor.") from error
def _time_cursor(self, query: OfflineReadQuery, scope: str, *, history: bool = False) -> _TimePosition | None:
- parts = self._cursor(query, scope, 3 if history else 2)
+ parts = self._cursor(query, scope, (4 if query.breakdown else 3) if history else 2)
if parts is None:
return None
try:
@@ -231,7 +259,10 @@ class OfflineEvaluationReadService:
if started_at.tzinfo is None:
raise ValueError
return _TimePosition(
- started_at=started_at, experiment_id=UUID(parts[1]), version_id=UUID(parts[2]) if history else None
+ started_at=started_at,
+ experiment_id=UUID(parts[1]),
+ version_id=UUID(parts[2]) if history else None,
+ breakdown_value=parts[3] if history and query.breakdown else "",
)
except ValueError as error:
raise OfflineEvaluationValidationError("cursor", "Provide a valid continuation cursor.") from error
@@ -395,11 +426,14 @@ class OfflineEvaluationReadService:
)
def _item_read(
- self, item: OfflineExperimentItem, results: list[OfflineResultRead] | None = None
+ self,
+ item: Annotated[OfflineExperimentItem, Annotations[_ItemAnnotations]],
+ results: list[OfflineResultRead] | None = None,
) -> OfflineItemRead:
return OfflineItemRead(
id=item.id,
experiment_id=item.experiment_id,
+ model_version=item.resolved_model_version,
case_key=item.case_key,
trial=item.trial,
dataset_item_identifier=item.dataset_item_identifier,
@@ -557,12 +591,29 @@ class OfflineEvaluationReadService:
)
return None
+ @classmethod
+ def _breakdown_expression(cls, breakdown: OfflineHistoryBreakdown | None, *, item_prefix: str) -> Coalesce | Value:
+ if breakdown is None:
+ return Value("")
+ if breakdown == OfflineHistoryBreakdown.MODEL:
+ return Coalesce(cls._model_expression(item_prefix=item_prefix), Value(""))
+ return Coalesce(F(f"{item_prefix}experiment__{breakdown.value}"), Value(""))
+
def _aggregates(
- self, identities: list[_SummaryIdentity], *, versions: dict[UUID, ScoreDefinitionVersion]
+ self,
+ identities: list[_SummaryIdentity],
+ *,
+ versions: dict[UUID, ScoreDefinitionVersion],
+ breakdown: OfflineHistoryBreakdown | None = None,
+ model_version: str | None = None,
) -> dict[_SummaryIdentity, _Aggregate]:
selected = Q(pk__in=[])
for identity in identities:
- selected |= Q(item__experiment_id=identity.experiment_id, scorer_version_id=identity.version_id)
+ selected |= Q(
+ item__experiment_id=identity.experiment_id,
+ scorer_version_id=identity.version_id,
+ breakdown_value=identity.breakdown_value,
+ )
passing = Q(pk__in=[])
configured_versions = set()
for version_id, version in versions.items():
@@ -571,7 +622,8 @@ class OfflineEvaluationReadService:
configured_versions.add(version_id)
passing |= Q(scorer_version_id=version_id) & condition
results = (
- self._results()
+ self._results(model_version=model_version)
+ .annotate(breakdown_value=self._breakdown_expression(breakdown, item_prefix="item__"))
.filter(selected)
.order_by()
.annotate(
@@ -585,16 +637,31 @@ class OfflineEvaluationReadService:
"boolean_value",
"categorical_values",
"score_passes",
+ "breakdown_value",
)
)
sql, params = results.query.sql_with_params()
- item_table = OfflineExperimentItem._meta.db_table
+ selected_item_groups = Q(pk__in=[])
+ for experiment_id, breakdown_value in sorted(
+ {(identity.experiment_id, identity.breakdown_value) for identity in identities}
+ ):
+ selected_item_groups |= Q(experiment_id=experiment_id, breakdown_value=breakdown_value)
+ item_query = (
+ self._items()
+ .annotate(breakdown_value=self._breakdown_expression(breakdown, item_prefix=""))
+ .filter(selected_item_groups)
+ .order_by()
+ .values("experiment_id", "id", "case_key", "trial", "breakdown_value")
+ )
+ if model_version is not None:
+ item_query = item_query.filter(resolved_model_version=model_version)
+ item_sql, item_params = item_query.query.sql_with_params()
# The text conversion preserves float precision before numeric accumulation prevents overflow.
aggregate_sql = f"""
WITH selected AS (
{sql}
), aggregates AS (
- SELECT experiment_id, scorer_version_id, count(*) AS result_count,
+ SELECT experiment_id, scorer_version_id, breakdown_value, count(*) AS result_count,
count(*) FILTER (WHERE status = 'ok') AS ok_count,
count(*) FILTER (WHERE status = 'error') AS error_count,
count(*) FILTER (WHERE status = 'skipped') AS skipped_count,
@@ -603,35 +670,34 @@ class OfflineEvaluationReadService:
count(*) FILTER (WHERE status = 'ok' AND boolean_value IS TRUE) AS true_count,
count(*) FILTER (WHERE status = 'ok' AND boolean_value IS FALSE) AS false_count,
count(*) FILTER (WHERE status = 'ok' AND score_passes) AS pass_count
- FROM selected GROUP BY experiment_id, scorer_version_id
+ FROM selected GROUP BY experiment_id, scorer_version_id, breakdown_value
), category_counts AS (
- SELECT experiment_id, scorer_version_id, category, count(*) AS selected_count
+ SELECT experiment_id, scorer_version_id, breakdown_value, category, count(*) AS selected_count
FROM selected CROSS JOIN LATERAL unnest(categorical_values) AS category
WHERE status = 'ok'
- GROUP BY experiment_id, scorer_version_id, category
+ GROUP BY experiment_id, scorer_version_id, breakdown_value, category
), categories AS (
- SELECT experiment_id, scorer_version_id,
+ SELECT experiment_id, scorer_version_id, breakdown_value,
array_agg(category ORDER BY category) AS keys,
array_agg(selected_count ORDER BY category) AS counts
- FROM category_counts GROUP BY experiment_id, scorer_version_id
+ FROM category_counts GROUP BY experiment_id, scorer_version_id, breakdown_value
), coverage AS (
- SELECT experiment_id, count(*) AS observed_item_count, count(DISTINCT case_key) AS distinct_case_count,
+ SELECT experiment_id, breakdown_value, count(*) AS observed_item_count, count(DISTINCT case_key) AS distinct_case_count,
count(*) FILTER (WHERE case_key IS NOT NULL) AS items_with_case_key_count,
count(*) FILTER (WHERE case_key IS NULL) AS items_without_case_key_count,
count(*) FILTER (WHERE trial IS NOT NULL) AS trial_item_count,
count(DISTINCT (case_key, CASE WHEN case_key IS NULL THEN id END, trial))
FILTER (WHERE trial IS NOT NULL) AS distinct_trial_count
- FROM {item_table}
- WHERE team_id = %s AND experiment_id = ANY(%s)
- GROUP BY experiment_id
+ FROM ({item_sql}) AS selected_items
+ GROUP BY experiment_id, breakdown_value
)
SELECT a.experiment_id, a.scorer_version_id, a.result_count, a.ok_count, a.error_count,
a.skipped_count, a.not_applicable_count, a.mean, a.true_count, a.false_count,
c.observed_item_count, c.distinct_case_count, c.items_with_case_key_count,
c.items_without_case_key_count, c.trial_item_count, c.distinct_trial_count,
- categories.keys, categories.counts, a.pass_count
- FROM aggregates a JOIN coverage c USING (experiment_id)
- LEFT JOIN categories USING (experiment_id, scorer_version_id)
+ categories.keys, categories.counts, a.pass_count, a.breakdown_value
+ FROM aggregates a JOIN coverage c USING (experiment_id, breakdown_value)
+ LEFT JOIN categories USING (experiment_id, scorer_version_id, breakdown_value)
"""
# One statement keeps coverage and category denominators coherent during uploads.
with connections[results.db].cursor() as cursor:
@@ -639,12 +705,11 @@ class OfflineEvaluationReadService:
aggregate_sql,
[
*params,
- self.team_id,
- list({identity.experiment_id for identity in identities}),
+ *item_params,
],
)
return {
- _SummaryIdentity(experiment_id=row[0], version_id=row[1]): _Aggregate(
+ _SummaryIdentity(experiment_id=row[0], version_id=row[1], breakdown_value=row[19]): _Aggregate(
count=row[2],
status_counts=OfflineStatusCounts(ok=row[3], error=row[4], skipped=row[5], not_applicable=row[6]),
# Python rounds underflow to zero; PostgreSQL's float cast rejects it.
@@ -665,7 +730,13 @@ class OfflineEvaluationReadService:
for row in cursor.fetchall()
}
- def _summaries(self, identities: list[_SummaryIdentity]) -> dict[_SummaryIdentity, OfflineScorerSummary]:
+ def _summaries(
+ self,
+ identities: list[_SummaryIdentity],
+ *,
+ breakdown: OfflineHistoryBreakdown | None = None,
+ model_version: str | None = None,
+ ) -> dict[_SummaryIdentity, OfflineScorerSummary]:
if not identities:
return {}
versions = {
@@ -674,7 +745,7 @@ class OfflineEvaluationReadService:
.filter(id__in=[identity.version_id for identity in identities])
.select_related("definition")
}
- aggregates = self._aggregates(identities, versions=versions)
+ aggregates = self._aggregates(identities, versions=versions, breakdown=breakdown, model_version=model_version)
summaries = {}
for identity in identities:
version = versions.get(identity.version_id)
@@ -762,12 +833,15 @@ class OfflineEvaluationReadService:
query = replace(query, scorer_definition_id=definition_id)
self._validate_scorer_selection(query)
scope = f"history:{definition_id}"
- results = self._results(query).filter(
+ results = self._results(query, model_version=query.model_version).filter(
scorer_definition_id=definition_id, item__experiment__in=self._filter_experiments(query, history=True)
)
+ results = results.annotate(breakdown_value=self._breakdown_expression(query.breakdown, item_prefix="item__"))
groups = (
- results.order_by("-item__experiment__started_at", "item__experiment_id", "scorer_version_id")
- .values("item__experiment__started_at", "item__experiment_id", "scorer_version_id")
+ results.order_by(
+ "-item__experiment__started_at", "item__experiment_id", "scorer_version_id", "breakdown_value"
+ )
+ .values("item__experiment__started_at", "item__experiment_id", "scorer_version_id", "breakdown_value")
.distinct()
)
count = groups.count()
@@ -781,6 +855,12 @@ class OfflineEvaluationReadService:
item__experiment_id=position.experiment_id,
scorer_version_id__gt=position.version_id,
)
+ | Q(
+ item__experiment__started_at=position.started_at,
+ item__experiment_id=position.experiment_id,
+ scorer_version_id=position.version_id,
+ breakdown_value__gt=position.breakdown_value,
+ )
)
rows = list(groups[: query.limit + 1])
page = cursor_page(
@@ -794,14 +874,19 @@ class OfflineEvaluationReadService:
row["item__experiment__started_at"].isoformat(),
str(row["item__experiment_id"]),
str(row["scorer_version_id"]),
+ *([row["breakdown_value"]] if query.breakdown else []),
],
),
)
identities = [
- _SummaryIdentity(experiment_id=row["item__experiment_id"], version_id=row["scorer_version_id"])
+ _SummaryIdentity(
+ experiment_id=row["item__experiment_id"],
+ version_id=row["scorer_version_id"],
+ breakdown_value=row["breakdown_value"],
+ )
for row in page.results
]
- summary_by_id = self._summaries(identities)
+ summary_by_id = self._summaries(identities, breakdown=query.breakdown, model_version=query.model_version)
experiments = {
experiment.id: experiment
for experiment in self._experiment_reads(
@@ -812,7 +897,11 @@ class OfflineEvaluationReadService:
count=page.count,
next_cursor=page.next_cursor,
results=[
- OfflineHistoryPoint(experiment=experiments[identity.experiment_id], summary=summary_by_id[identity])
+ OfflineHistoryPoint(
+ experiment=experiments[identity.experiment_id],
+ summary=summary_by_id[identity],
+ breakdown_value=identity.breakdown_value or None,
+ )
for identity in identities
if identity in summary_by_id
],
diff --git a/products/ai_observability/backend/offline_evaluation_read_types.py b/products/ai_observability/backend/offline_evaluation_read_types.py
index 52b483c32482..cd908a8be257 100644
--- a/products/ai_observability/backend/offline_evaluation_read_types.py
+++ b/products/ai_observability/backend/offline_evaluation_read_types.py
@@ -3,6 +3,8 @@ from datetime import datetime
from typing import Literal
from uuid import UUID
+from django.db import models
+
from posthog.dataclasses import frozen
from products.ai_observability.backend.offline_evaluation_service import OfflineEvaluationValidationError
@@ -13,8 +15,17 @@ MAX_READ_SCORER_VERSIONS = 20
MAX_RESULT_CELL_ITEMS = 50
+class OfflineHistoryBreakdown(models.TextChoices):
+ MODEL = "model_version", "Model"
+ PROMPT = "prompt_version", "Prompt version"
+ APPLICATION = "application_version", "Application version"
+ SUITE = "suite_key", "Suite"
+ SOURCE = "run_source", "Run source"
+
+
@frozen
class OfflineReadQuery:
+ breakdown: OfflineHistoryBreakdown | None = None
limit: int = 50
cursor: str | None = None
date_from: datetime | None = None
@@ -143,6 +154,7 @@ class OfflineResultCells:
@frozen
class OfflineItemRead:
+ model_version: str | None = None
id: UUID
experiment_id: UUID
case_key: str | None
@@ -213,3 +225,4 @@ class OfflineScorerSummary:
class OfflineHistoryPoint:
experiment: OfflineExperimentRead
summary: OfflineScorerSummary
+ breakdown_value: str | None = None
diff --git a/products/ai_observability/backend/offline_evaluation_types.py b/products/ai_observability/backend/offline_evaluation_types.py
index 1388f139cddd..92f97cdd0ca3 100644
--- a/products/ai_observability/backend/offline_evaluation_types.py
+++ b/products/ai_observability/backend/offline_evaluation_types.py
@@ -32,6 +32,7 @@ class ExperimentSubmission:
@frozen
class ItemSubmission:
id: UUID
+ model_version: str | None = None
case_key: str | None = None
trial: str | None = None
dataset_item_identifier: str | None = None
diff --git a/products/ai_observability/backend/read_pagination.py b/products/ai_observability/backend/read_pagination.py
index 7fab5060f874..254e8a46ea8b 100644
--- a/products/ai_observability/backend/read_pagination.py
+++ b/products/ai_observability/backend/read_pagination.py
@@ -7,7 +7,11 @@ from posthog.dataclasses import frozen
def encode_cursor(values: list[str]) -> str:
- return base64.urlsafe_b64encode(json.dumps(values, separators=(",", ":")).encode()).decode().rstrip("=")
+ return (
+ base64.urlsafe_b64encode(json.dumps(values, separators=(",", ":"), ensure_ascii=False).encode())
+ .decode()
+ .rstrip("=")
+ )
def decode_cursor(cursor: str, expected_parts: int) -> list[str]:
diff --git a/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py b/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py
index 45728e53fca5..baa270743b27 100644
--- a/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py
+++ b/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py
@@ -1,4 +1,6 @@
-from dataclasses import replace
+import json
+import hashlib
+from dataclasses import asdict, replace
from datetime import datetime
from uuid import UUID
@@ -50,6 +52,21 @@ class TestOfflineEvaluationFingerprint(SimpleTestCase):
submission_fingerprint(ItemSubmission(id=ITEM_ID, payload=second)),
)
+ def test_item_model_preserves_legacy_retries_and_detects_model_changes(self) -> None:
+ item = ItemSubmission(id=ITEM_ID)
+ legacy_fields = asdict(item)
+ legacy_fields.pop("model_version")
+ legacy_fields["id"] = str(ITEM_ID)
+ legacy_hash = hashlib.sha256(
+ b"offline-evaluation:v1:" + json.dumps(legacy_fields, sort_keys=True, separators=(",", ":")).encode()
+ ).hexdigest()
+ self.assertEqual(submission_fingerprint(item), legacy_hash)
+ with_model = replace(item, model_version="model-a")
+ self.assertNotEqual(submission_fingerprint(with_model), legacy_hash)
+ self.assertNotEqual(
+ submission_fingerprint(with_model), submission_fingerprint(replace(item, model_version="model-b"))
+ )
+
def test_equivalent_timestamp_offsets_have_the_same_fingerprint(self) -> None:
submission = ExperimentSubmission(
id=ITEM_ID, name="Candidate model", started_at=datetime.fromisoformat("2026-09-24T10:00:00+00:00")
diff --git a/products/ai_observability/backend/test/test_offline_evaluation_reads.py b/products/ai_observability/backend/test/test_offline_evaluation_reads.py
index f0e249b9cf14..38d0aa365b86 100644
--- a/products/ai_observability/backend/test/test_offline_evaluation_reads.py
+++ b/products/ai_observability/backend/test/test_offline_evaluation_reads.py
@@ -12,6 +12,9 @@ from parameterized import parameterized
from posthog.models import Organization, Team
+from products.ai_observability.backend.api.offline_experiment_read_serializers import (
+ OfflineScorerHistoryQuerySerializer,
+)
from products.ai_observability.backend.models.offline_evaluations import (
OfflineEvaluationResult,
OfflineEvaluationResultPayload,
@@ -22,7 +25,11 @@ from products.ai_observability.backend.models.offline_evaluations import (
)
from products.ai_observability.backend.models.score_definitions import ScoreDefinition, ScoreDefinitionVersion
from products.ai_observability.backend.offline_evaluation_read_service import OfflineEvaluationReadService
-from products.ai_observability.backend.offline_evaluation_read_types import OfflineReadQuery, OfflineStatusCounts
+from products.ai_observability.backend.offline_evaluation_read_types import (
+ OfflineHistoryBreakdown,
+ OfflineReadQuery,
+ OfflineStatusCounts,
+)
from products.ai_observability.backend.offline_evaluation_service import (
OfflineEvaluationNotFound,
OfflineEvaluationValidationError,
@@ -102,13 +109,19 @@ class TestOfflineEvaluationReads(TestCase):
)
def _item(
- self, experiment: OfflineExperiment, *, case_key: str | None = None, trial: str | None = None
+ self,
+ experiment: OfflineExperiment,
+ *,
+ case_key: str | None = None,
+ trial: str | None = None,
+ model_version: str | None = None,
) -> OfflineExperimentItem:
return OfflineExperimentItem.objects.for_team(self.team.id).create(
id=uuid4(),
team=self.team,
experiment=experiment,
case_key=case_key,
+ model_version=model_version,
trial=trial,
submission_fingerprint="a" * 64,
)
@@ -334,6 +347,138 @@ class TestOfflineEvaluationReads(TestCase):
self.assertIsNone(older_page.next_cursor)
self.assertEqual(self.service.get_experiment(expected[0]).id, expected[0])
+ def test_model_breakdown_uses_item_overrides_and_group_coverage(self) -> None:
+ experiment = self._experiment()
+ experiment.model_version = "model-a"
+ experiment.save(update_fields=["model_version"])
+ inherited = self._item(experiment, case_key="one")
+ override = self._item(experiment, case_key="one", model_version="model-b")
+ error = self._item(experiment, case_key="two", model_version="model-b")
+ self._item(experiment, case_key="missing", model_version="model-b")
+ self._result(inherited, self.numeric, 1.0)
+ self._result(override, self.numeric, 0.0)
+ self._result(error, self.numeric, status="error")
+ self._result(inherited, self.boolean, True)
+ self._result(override, self.boolean, False)
+ self._result(inherited, self.categorical, ["a"])
+ self._result(override, self.categorical, ["b"])
+
+ for version in (self.numeric, self.boolean, self.categorical):
+ page = self.service.scorer_history(
+ version.definition_id, OfflineReadQuery(breakdown=OfflineHistoryBreakdown.MODEL)
+ )
+ self.assertEqual(page.count, 2)
+ groups = {point.breakdown_value: point.summary for point in page.results}
+ self.assertEqual(groups["model-a"].observed_item_count, 1)
+ self.assertEqual(groups["model-b"].observed_item_count, 3)
+ self.assertEqual(groups["model-a"].status_counts.ok, 1)
+ self.assertEqual(groups["model-b"].status_counts.ok, 1)
+ if version == self.numeric:
+ self.assertEqual(groups["model-a"].mean, 1.0)
+ self.assertEqual(groups["model-b"].mean, 0.0)
+ self.assertEqual(groups["model-b"].status_counts.error, 1)
+ self.assertEqual(groups["model-b"].missing_result_count, 1)
+ elif version == self.boolean:
+ self.assertEqual(groups["model-a"].pass_rate, 1.0)
+ self.assertEqual(groups["model-b"].pass_rate, 0.0)
+ else:
+ self.assertEqual([category.rate for category in groups["model-a"].categories], [1.0, 0.0, 0.0])
+ self.assertEqual([category.rate for category in groups["model-b"].categories], [0.0, 1.0, 0.0])
+
+ whole = self.service.scorer_history(self.numeric.definition_id, OfflineReadQuery()).results[0].summary
+ self.assertEqual((whole.mean, whole.observed_item_count, whole.missing_result_count), (0.5, 4, 1))
+ self.assertEqual(self.service.get_item(experiment.id, inherited.id).model_version, "model-a")
+ self.assertEqual(self.service.get_item(experiment.id, override.id).model_version, "model-b")
+
+ @parameterized.expand([(None,), (OfflineHistoryBreakdown.MODEL,), (OfflineHistoryBreakdown.SUITE,)])
+ def test_model_filter_matches_item_models_across_experiments(
+ self, breakdown: OfflineHistoryBreakdown | None
+ ) -> None:
+ experiments = []
+ for index, model in enumerate(("model-a", "model-b")):
+ experiment = self._experiment(started_at=self.now - timedelta(hours=index))
+ experiment.model_version = model
+ experiment.save(update_fields=["model_version"])
+ experiments.append(experiment)
+ matching_model = None if model == "model-a" else "model-a"
+ other_model = "model-b" if model == "model-a" else None
+ self._result(self._item(experiment, model_version=matching_model), self.numeric, float(index + 1))
+ self._item(experiment, model_version=matching_model)
+ self._result(self._item(experiment, model_version=other_model), self.numeric, 100.0)
+ self._item(experiment, model_version=other_model)
+
+ query = OfflineReadQuery(model_version="model-a", breakdown=breakdown, limit=1)
+ for index, experiment in enumerate(experiments):
+ page = self.service.scorer_history(self.numeric.definition_id, query)
+ self.assertEqual(page.count, 2)
+ self.assertEqual(len(page.results), 1)
+ point = page.results[0]
+ self.assertEqual(point.experiment.id, experiment.id)
+ self.assertEqual(point.breakdown_value, "model-a" if breakdown == OfflineHistoryBreakdown.MODEL else None)
+ self.assertEqual(point.summary.mean, float(index + 1))
+ self.assertEqual(point.summary.status_counts.ok, 1)
+ self.assertEqual(point.summary.observed_item_count, 2)
+ self.assertEqual(point.summary.missing_result_count, 1)
+ self.assertEqual(page.next_cursor is None, index == 1)
+ query = replace(query, cursor=page.next_cursor)
+
+ listed = self.service.list_experiments(OfflineReadQuery(model_version="model-a"))
+ self.assertEqual([experiment.id for experiment in listed.results], [experiments[0].id])
+
+ def test_model_breakdown_paginates_missing_and_named_models_without_duplicates(self) -> None:
+ experiment = self._experiment()
+ models = [None, "model-a", "model-b", "Not specified", "模" * 255, "😀" * 255, "😁" * 255]
+ for index, model in enumerate(models):
+ self._result(self._item(experiment, model_version=model, case_key="scored"), self.numeric, 1.0)
+ for missing in range(index + 1):
+ self._item(experiment, model_version=model, case_key=f"missing-{missing}")
+ query = OfflineReadQuery(breakdown=OfflineHistoryBreakdown.MODEL, limit=1)
+ values = []
+ while True:
+ page = self.service.scorer_history(self.numeric.definition_id, query)
+ self.assertEqual(page.count, len(models))
+ self.assertEqual(len(page.results), 1)
+ point = page.results[0]
+ values.append(point.breakdown_value)
+ missing_count = models.index(point.breakdown_value) + 1
+ self.assertEqual(point.summary.observed_item_count, missing_count + 1)
+ self.assertEqual(point.summary.distinct_case_count, missing_count + 1)
+ self.assertEqual(point.summary.missing_result_count, missing_count)
+ if not page.next_cursor:
+ break
+ serializer = OfflineScorerHistoryQuerySerializer(
+ data={"breakdown": "model_version", "limit": 1, "cursor": page.next_cursor}
+ )
+ self.assertTrue(serializer.is_valid(), serializer.errors)
+ with self.assertRaises(OfflineEvaluationValidationError):
+ self.service.scorer_history(
+ self.numeric.definition_id,
+ replace(query, cursor=page.next_cursor, breakdown=OfflineHistoryBreakdown.SUITE),
+ )
+ query = replace(query, cursor=page.next_cursor)
+ self.assertCountEqual(values, models)
+
+ @parameterized.expand(
+ [
+ (OfflineHistoryBreakdown.PROMPT, "prompt-a"),
+ (OfflineHistoryBreakdown.APPLICATION, "build-a"),
+ (OfflineHistoryBreakdown.SUITE, "suite-a"),
+ (OfflineHistoryBreakdown.SOURCE, "ci"),
+ ]
+ )
+ def test_experiment_breakdowns_keep_all_items_in_one_group(
+ self, breakdown: OfflineHistoryBreakdown, value: str
+ ) -> None:
+ experiment = self._experiment()
+ setattr(experiment, breakdown, value)
+ experiment.save(update_fields=[breakdown])
+ for model in ("model-a", "model-b"):
+ self._result(self._item(experiment, model_version=model), self.numeric, 1.0)
+ page = self.service.scorer_history(self.numeric.definition_id, OfflineReadQuery(breakdown=breakdown))
+ self.assertEqual(page.count, 1)
+ self.assertEqual(page.results[0].breakdown_value, value)
+ self.assertEqual(page.results[0].summary.observed_item_count, 2)
+
def test_history_pages_experiment_version_groups_by_execution_time(self) -> None:
older = self._experiment(started_at=self.now - timedelta(days=5), run_source="ci")
newer = self._experiment(run_source="local")
diff --git a/products/ai_observability/backend/test/test_offline_evaluation_service.py b/products/ai_observability/backend/test/test_offline_evaluation_service.py
index 0c1770f4e495..36c9227b8f9f 100644
--- a/products/ai_observability/backend/test/test_offline_evaluation_service.py
+++ b/products/ai_observability/backend/test/test_offline_evaluation_service.py
@@ -85,7 +85,10 @@ class TestOfflineEvaluationService(TestCase):
self.assertEqual(error.exception.code, "content_conflict")
self.assertEqual(OfflineExperiment.objects.for_team(self.team.id).count(), 1)
- def test_upload_retry_preserves_identity_acceptance_and_retention(self) -> None:
+ @parameterized.expand([(None,), ("model-a",)])
+ def test_upload_retry_preserves_identity_acceptance_and_retention(self, model_version: str | None) -> None:
+ self.item = replace(self.item, model_version=model_version)
+ self.submission = replace(self.submission, items=[self.item])
accepted = self.ingestion.upload(self.experiment.id, self.submission)
replay = self.ingestion.upload(self.experiment.id, self.submission)
@@ -95,6 +98,7 @@ class TestOfflineEvaluationService(TestCase):
self.assertTrue(accepted.results[0].created)
item = OfflineExperimentItem.objects.for_team(self.team.id).get(id=self.item.id)
result = OfflineEvaluationResult.objects.for_team(self.team.id).get(id=accepted.results[0].id)
+ self.assertEqual(item.model_version, model_version)
self.assertEqual(item.payload_expires_at, item.accepted_at + timedelta(days=30))
self.assertEqual(result.payload_expires_at, result.accepted_at + timedelta(days=30))
self.assertEqual(
diff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.ts
index 5a6edcfd7226..4897ec10e815 100644
--- a/products/ai_observability/frontend/generated/api.schemas.ts
+++ b/products/ai_observability/frontend/generated/api.schemas.ts
@@ -734,6 +734,11 @@ export interface OfflineResultCellApi {
}
export interface OfflineItemReadApi {
+ /**
+ * Model used for the item, falling back to the experiment's model.
+ * @nullable
+ */
+ model_version?: string | null
/** Stable item UUID. */
id: string
/** Owning experiment UUID. */
@@ -1208,6 +1213,12 @@ export interface OfflineSummaryPageApi {
export interface ItemSubmissionApi {
/** Caller-generated UUID for one input/output execution. Reuse for exact retries. */
id: string
+ /**
+ * Model used for this item. Omitted or null values inherit the experiment's model.
+ * @maxLength 255
+ * @nullable
+ */
+ model_version?: string | null
/**
* Stable case identifier for matching inputs across experiments.
* @maxLength 255
@@ -1326,6 +1337,11 @@ export interface UploadReceiptApi {
}
export interface OfflineHistoryPointApi {
+ /**
+ * Breakdown value for this summary; null when unspecified or no breakdown is selected.
+ * @nullable
+ */
+ breakdown_value?: string | null
/** Experiment execution and cohort context. */
experiment: OfflineExperimentReadApi
/** Complete summary for one experiment and scorer version. */
@@ -5316,6 +5332,17 @@ export type AiObservabilityOfflineScorersHistoryListParams = {
* @maxLength 255
*/
application_version?: string
+ /**
+ * Split history by this property. Model uses each item's model, falling back to the experiment's model.
+ *
+ * * `model_version` - Model
+ * * `prompt_version` - Prompt version
+ * * `application_version` - Application version
+ * * `suite_key` - Suite
+ * * `run_source` - Run source
+ * @minLength 1
+ */
+ breakdown?: AiObservabilityOfflineScorersHistoryListBreakdown
/**
* Filter experiments by the author's user ID.
* @minimum 1
@@ -5419,6 +5446,17 @@ export type AiObservabilityOfflineScorersHistoryListParams = {
tags?: string
}
+export type AiObservabilityOfflineScorersHistoryListBreakdown =
+ (typeof AiObservabilityOfflineScorersHistoryListBreakdown)[keyof typeof AiObservabilityOfflineScorersHistoryListBreakdown]
+
+export const AiObservabilityOfflineScorersHistoryListBreakdown = {
+ ModelVersion: 'model_version',
+ PromptVersion: 'prompt_version',
+ ApplicationVersion: 'application_version',
+ SuiteKey: 'suite_key',
+ RunSource: 'run_source',
+} as const
+
export type AiObservabilityOfflineScorersHistoryListTagMatch =
(typeof AiObservabilityOfflineScorersHistoryListTagMatch)[keyof typeof AiObservabilityOfflineScorersHistoryListTagMatch]
diff --git a/products/ai_observability/frontend/generated/api.zod.ts b/products/ai_observability/frontend/generated/api.zod.ts
index 0e3a707fcd5c..fb765b094829 100644
--- a/products/ai_observability/frontend/generated/api.zod.ts
+++ b/products/ai_observability/frontend/generated/api.zod.ts
@@ -141,6 +141,8 @@ export const AiObservabilityOfflineExperimentsPartialUpdateBody = /* @__PURE__ *
.describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
})
+export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax = 255
+
export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax = 255
export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemTrialMax = 255
@@ -168,6 +170,11 @@ export const AiObservabilityOfflineExperimentsUploadCreateBody = /* @__PURE__ */
id: zod
.uuid()
.describe('Caller-generated UUID for one input\/output execution. Reuse for exact retries.'),
+ model_version: zod
+ .string()
+ .max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax)
+ .nullish()
+ .describe("Model used for this item. Omitted or null values inherit the experiment's model."),
case_key: zod
.string()
.max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax)
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx
index daa307148145..7d534ca1154f 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx
@@ -71,6 +71,7 @@ export function OfflineItemInspector({
{item.id}
</span>
{item.trial && <span>{`Trial ${item.trial}`}</span>}
+ {item.model_version && <span>{`Model: ${item.model_version}`}</span>}
{item.application_trace_id && (
<Link to={urls.aiObservabilityTrace(item.application_trace_id)}>
Application trace
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx
index 0f35cf67c5e4..8269b2900ccf 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx
@@ -64,7 +64,7 @@ describe('OfflineOverviewTrend', () => {
).toBeInTheDocument()
expect(screen.getByText('Mean score')).toBeInTheDocument()
if (partial) {
- expect(screen.getByText(/30 of 200 experiment\/version results.*Partial history/)).toBeInTheDocument()
+ expect(screen.getByText(/30 of 200 history points.*Partial history/)).toBeInTheDocument()
expect(screen.getByText('Earlier experiments and scorer versions may not be shown.')).toBeInTheDocument()
} else {
expect(screen.queryByText(/Partial history/)).not.toBeInTheDocument()
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx
index a7f9f170ebfd..43c158dd0081 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx
@@ -10,6 +10,11 @@ import { cn } from 'lib/utils/css-classes'
import type { OfflineHistoryPointApi } from '../generated/api.schemas'
import { ScorerLabel } from '../scoreDefinitions/ScorerLabel'
import type { offlineExperimentsLogicType } from './offlineExperimentsLogic'
+import {
+ OFFLINE_HISTORY_BREAKDOWN_LABELS,
+ offlineBreakdownValueLabel,
+ type OfflineHistoryBreakdown,
+} from './offlineHistoryBreakdown'
import { createOfflineScoreTrendTickFormatter } from './offlineScoreTrendAxis'
import { OfflineScoreTrendCrosshair } from './OfflineScoreTrendCrosshair'
import { OfflineScoreTrendLines } from './OfflineScoreTrendLines'
@@ -31,6 +36,7 @@ export interface OfflineScoreTrendChartProps {
xDomain?: [number, number]
showHeading?: boolean
combineVersions?: boolean
+ breakdown?: OfflineHistoryBreakdown
}
export function OfflineScoreTrendChart({
@@ -43,6 +49,7 @@ export function OfflineScoreTrendChart({
xDomain,
showHeading = true,
combineVersions = false,
+ breakdown,
}: OfflineScoreTrendChartProps): JSX.Element {
const baseTheme = useChartTheme()
const theme = useMemo(
@@ -56,7 +63,7 @@ export function OfflineScoreTrendChart({
)
const panels = useMemo(
() =>
- buildOfflineTrendPanels(periods, combineVersions).map((panel) => {
+ buildOfflineTrendPanels(periods, combineVersions, breakdown).map((panel) => {
const domain = !panel.elapsed && xDomain ? xDomain : panel.xDomain
return {
...panel,
@@ -64,7 +71,7 @@ export function OfflineScoreTrendChart({
tickFormatter: createOfflineScoreTrendTickFormatter(domain, panel.elapsed, timezone),
}
}),
- [periods, xDomain, timezone, combineVersions]
+ [periods, xDomain, timezone, combineVersions, breakdown]
)
if (panels.length === 0) {
@@ -131,6 +138,9 @@ export function OfflineScoreTrendChart({
: String(point.y)}
</strong>
</div>
+ {breakdown && (
+ <div className="opacity-70">{`${OFFLINE_HISTORY_BREAKDOWN_LABELS[breakdown]}: ${offlineBreakdownValueLabel(meta.point.breakdown_value, breakdown)}`}</div>
+ )}
{periods.length > 1 && <div className="opacity-70">{meta.period}</div>}
{offlineScorePassingRuleLabel(summary.scorer) && (
<div className="opacity-70 mt-1">
@@ -172,7 +182,7 @@ export function OfflineScoreTrendChart({
['Upload state', experiment.status],
['Suite', experiment.suite_key],
['Application', experiment.application_version],
- ['Model', experiment.model_version],
+ ['Experiment model', experiment.model_version],
['Prompt', experiment.prompt_version],
['Dataset revision', experiment.dataset_revision_identifier],
]
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx
index 630b8a918b7d..d4fc2ff1d0c1 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx
@@ -17,6 +17,7 @@ import { AccessControlLevel, AccessControlResourceType } from '~/types'
import { ResourceTagFilter } from '../components/ResourceTagFilter'
import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'
import { offlineExperimentUrl } from './offlineExperimentPresentation'
+import { OFFLINE_HISTORY_BREAKDOWN_OPTIONS, type OfflineHistoryBreakdown } from './offlineHistoryBreakdown'
import { offlineScorerHistoryLogic, type OfflineScorerHistoryProps } from './offlineScorerHistoryLogic'
import { OfflineScorerHistoryTable } from './OfflineScorerHistoryTable'
import { OfflineScoreTrendChart } from './OfflineScoreTrendChart'
@@ -112,6 +113,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div>
<LemonLabel>Upload state</LemonLabel>
<LemonSelect
+ size="small"
value={filters.statuses}
onChange={(statuses) => setFilters({ statuses })}
options={[
@@ -125,6 +127,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div>
<LemonLabel>Source</LemonLabel>
<LemonSelect
+ size="small"
value={filters.run_source}
onChange={(run_source) => setFilters({ run_source })}
options={[
@@ -138,13 +141,13 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
</div>
<div>
<LemonLabel>Tags</LemonLabel>
- <ResourceTagFilter filters={filters} onChange={setFilters} />
+ <ResourceTagFilter size="small" filters={filters} onChange={setFilters} />
</div>
<div data-attr="offline-history-author-filter">
<LemonLabel>Author</LemonLabel>
<MemberSelect
value={filters.created_by ?? null}
- size="medium"
+ size="small"
defaultLabel="Any author"
extraOptions={[
{
@@ -159,7 +162,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
>
{filters.created_by_is_null
? () => (
- <LemonButton type="secondary" size="medium">
+ <LemonButton type="secondary" size="small">
No author
</LemonButton>
)
@@ -169,6 +172,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div>
<LemonLabel>Version</LemonLabel>
<LemonSelect
+ size="small"
value={filters.version}
options={versionOptions}
onChange={(version) => setFilters({ version })}
@@ -178,6 +182,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
</div>
{versions?.next_cursor && (
<LemonButton
+ size="small"
type="secondary"
loading={versionsLoading}
onClick={() => loadOfflineHistoryVersions({ cursor: versions.next_cursor! })}
@@ -188,6 +193,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div>
<LemonLabel>Date range</LemonLabel>
<DateFilter
+ size="small"
dateFrom={filters.date_from}
dateTo={filters.date_to}
onChange={(date_from, date_to) =>
@@ -199,6 +205,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div>
<LemonLabel>Compare</LemonLabel>
<LemonSelect
+ size="small"
value={filters.compare}
onChange={(compare) => setFilters({ compare })}
options={[
@@ -220,6 +227,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div>
<LemonLabel>Comparison range</LemonLabel>
<DateFilter
+ size="small"
dateFrom={filters.compare_from}
dateTo={filters.compare_to}
onChange={(compare_from, compare_to) =>
@@ -230,6 +238,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
</div>
)}
<LemonButton
+ size="small"
type="secondary"
loading={primaryPageLoading || comparisonPageLoading}
onClick={refreshHistory}
@@ -265,7 +274,22 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
) : (
<>
<LemonCard hoverEffect={false} className="min-w-0">
- <div className="font-semibold mb-2">Score history</div>
+ <div className="flex flex-wrap items-center justify-between gap-2 mb-2">
+ <div className="font-semibold">Score history</div>
+ <div className="flex items-center gap-2">
+ <LemonLabel>Break down by</LemonLabel>
+ <LemonSelect<OfflineHistoryBreakdown | ''>
+ size="small"
+ value={filters.breakdown}
+ onChange={(breakdown) => setFilters({ breakdown })}
+ options={[
+ { value: '', label: 'None' },
+ ...OFFLINE_HISTORY_BREAKDOWN_OPTIONS,
+ ]}
+ data-attr="offline-history-breakdown"
+ />
+ </div>
+ </div>
{primaryPageLoading || comparisonPageLoading || (!primaryPage && !primaryError) ? (
<Spinner />
) : (
@@ -278,6 +302,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
{trendPeriods.length > 0 && (
<OfflineScoreTrendChart
periods={trendPeriods}
+ breakdown={filters.breakdown || undefined}
timezone={timezone}
onPointClick={(point) =>
router.actions.push(offlineExperimentUrl(point))
@@ -287,8 +312,9 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
</>
)}
<p className="text-muted text-xs mt-2 mb-0">
- Each point is one experiment and scorer version. Points use successful results
- only. Different periods and versions remain separate.
+ {filters.breakdown
+ ? 'Each point is one experiment, scorer version, and breakdown value. Points use successful results only. Different periods and versions remain separate.'
+ : 'Each point is one experiment and scorer version. Points use successful results only. Different periods and versions remain separate.'}
</p>
</LemonCard>
{[
@@ -321,14 +347,27 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
<div className="font-semibold">{period.title}</div>
<div className="text-xs text-muted flex flex-wrap gap-1">
{period.range?.dateFrom ? (
- <TZLabel time={period.range.dateFrom} displayTimezone={timezone} />
+ <TZLabel
+ time={period.range.dateFrom}
+ displayTimezone={timezone}
+ formatDate="MMM D, YYYY"
+ formatTime="HH:mm"
+ timestampStyle="absolute"
+ />
) : (
<span>All time</span>
)}
<span>to</span>
{period.range?.dateTo && (
- <TZLabel time={period.range.dateTo} displayTimezone={timezone} />
+ <TZLabel
+ time={period.range.dateTo}
+ displayTimezone={timezone}
+ formatDate="MMM D, YYYY"
+ formatTime="HH:mm"
+ timestampStyle="absolute"
+ />
)}
+ <span>{`(${timezone})`}</span>
</div>
{period.error ? (
<LemonBanner
@@ -345,6 +384,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
</div>
)}
<OfflineScorerHistoryTable
+ breakdown={filters.breakdown || undefined}
periodKey={period.key}
page={period.page}
loading={period.loading}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx
index 9b0f40552c75..2fc98af979aa 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx
@@ -73,3 +73,42 @@ export const EmptyHistory: Story = {
}),
],
}
+
+export const ModelBreakdown: Story = {
+ parameters: {
+ pageUrl: `${urls.aiObservabilityOfflineScorerHistory(OFFLINE_STORY_DEFINITION.id)}?breakdown=model_version`,
+ },
+ decorators: [
+ mswDecorator({
+ get: {
+ '/api/projects/:project/ai_observability/offline_scorers/:definition/history/': ({ request }) => {
+ const breakdown = new URL(request.url).searchParams.get('breakdown')
+ const results =
+ breakdown === 'model_version'
+ ? OFFLINE_STORY_POINTS.flatMap((point) => [
+ { ...point, breakdown_value: 'model-a' },
+ {
+ ...point,
+ breakdown_value: 'model-b',
+ summary: { ...point.summary, mean: (point.summary.mean ?? 0) / 2 },
+ },
+ ])
+ : OFFLINE_STORY_POINTS.map((point) => ({
+ ...point,
+ breakdown_value:
+ breakdown === 'suite_key'
+ ? point.experiment.suite_key
+ : breakdown === 'run_source'
+ ? point.experiment.run_source
+ : breakdown === 'prompt_version'
+ ? point.experiment.prompt_version
+ : breakdown === 'application_version'
+ ? point.experiment.application_version
+ : null,
+ }))
+ return { results, count: results.length, next_cursor: null }
+ },
+ },
+ }),
+ ],
+}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
index d1d83974700d..9c271fec0508 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
@@ -6,17 +6,25 @@ import { TZLabel } from 'lib/components/TZLabel'
import type { OfflineHistoryPageApi, OfflineHistoryPointApi } from '../generated/api.schemas'
import { offlineExperimentUrl } from './offlineExperimentPresentation'
+import {
+ OFFLINE_HISTORY_BREAKDOWN_LABELS,
+ offlineBreakdownValueLabel,
+ offlineHistoryPointKey,
+ type OfflineHistoryBreakdown,
+} from './offlineHistoryBreakdown'
import { OfflineScorerHistoryRunDetails } from './OfflineScorerHistoryRunDetails'
import { OfflineScoreSummaryDisplay } from './OfflineScoreSummaryDisplay'
export function OfflineScorerHistoryTable({
periodKey,
+ breakdown,
page,
loading,
expandedRuns,
onToggleRun,
}: {
periodKey: string
+ breakdown?: OfflineHistoryBreakdown
page: OfflineHistoryPageApi | null
loading: boolean
expandedRuns: ReadonlySet<string>
@@ -25,7 +33,7 @@ export function OfflineScorerHistoryTable({
return (
<LemonTable<OfflineHistoryPointApi>
className="@max-[50rem]/scorer-history:[&_colgroup]:hidden @max-[50rem]/scorer-history:[&_table]:table-auto"
- rowKey={(point) => `${point.experiment.id}:${point.summary.scorer.id}`}
+ rowKey={offlineHistoryPointKey}
dataSource={page?.results || []}
loading={loading || !page}
size="small"
@@ -35,9 +43,10 @@ export function OfflineScorerHistoryTable({
{
title: 'Experiment',
key: 'name',
- width: '28%',
+ className: '@max-[50rem]/scorer-history:w-2/5 @max-[50rem]/scorer-history:max-w-0',
+ width: breakdown ? '24%' : '28%',
render: (_, point) => {
- const key = `${periodKey}:${point.experiment.id}:${point.summary.scorer.id}`
+ const key = `${periodKey}:${offlineHistoryPointKey(point)}`
return (
<div className="flex items-center gap-1 min-w-0">
<LemonButton
@@ -60,10 +69,27 @@ export function OfflineScorerHistoryTable({
)
},
},
+ ...(breakdown
+ ? [
+ {
+ title: OFFLINE_HISTORY_BREAKDOWN_LABELS[breakdown],
+ key: 'breakdown',
+ className: '@max-[50rem]/scorer-history:w-1/5 @max-[50rem]/scorer-history:max-w-0',
+ width: '16%',
+ render: (_: unknown, point: OfflineHistoryPointApi) => (
+ <Tooltip title={offlineBreakdownValueLabel(point.breakdown_value, breakdown)}>
+ <div className="truncate">
+ {offlineBreakdownValueLabel(point.breakdown_value, breakdown)}
+ </div>
+ </Tooltip>
+ ),
+ },
+ ]
+ : []),
{
title: 'Tags',
key: 'tags',
- width: '24%',
+ width: breakdown ? '18%' : '24%',
render: (_, point) => (
<div className="min-w-0 max-w-64">
<ObjectTags tags={point.experiment.tags} staticOnly maxVisibleTags={2} wrap />
@@ -74,7 +100,7 @@ export function OfflineScorerHistoryTable({
title: 'Execution',
key: 'execution',
className: '@max-[50rem]/scorer-history:hidden',
- width: '16%',
+ width: breakdown ? '14%' : '16%',
render: (_, point) => (
<div className="truncate text-xs">
<TZLabel time={point.experiment.started_at} />
@@ -84,14 +110,14 @@ export function OfflineScorerHistoryTable({
{
title: 'Score',
key: 'score',
- width: '12%',
+ width: breakdown ? '14%' : '12%',
render: (_, point) => <OfflineScoreSummaryDisplay summary={point.summary} />,
},
{
title: 'Coverage',
key: 'coverage',
className: '@max-[40rem]/scorer-history:hidden',
- width: '20%',
+ width: breakdown ? '14%' : '20%',
render: (_, { summary }) => (
<Tooltip
title={`${summary.status_counts.ok} / ${summary.observed_item_count} scored items · ${summary.status_counts.error} errors · ${summary.status_counts.skipped} skipped · ${summary.status_counts.not_applicable} not applicable · ${summary.missing_result_count} missing`}
@@ -110,8 +136,7 @@ export function OfflineScorerHistoryTable({
expandable={{
noIndent: true,
showRowExpansionToggle: false,
- isRowExpanded: (point) =>
- expandedRuns.has(`${periodKey}:${point.experiment.id}:${point.summary.scorer.id}`),
+ isRowExpanded: (point) => expandedRuns.has(`${periodKey}:${offlineHistoryPointKey(point)}`),
expandedRowRender: (point) => <OfflineScorerHistoryRunDetails point={point} />,
}}
/>
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineHistoryBreakdown.ts b/products/ai_observability/frontend/offline-evaluations/offlineHistoryBreakdown.ts
new file mode 100644
index 000000000000..e304866b4ac1
--- /dev/null
+++ b/products/ai_observability/frontend/offline-evaluations/offlineHistoryBreakdown.ts
@@ -0,0 +1,34 @@
+import {
+ AiObservabilityOfflineScorersHistoryListBreakdown,
+ type AiObservabilityOfflineScorersHistoryListParams,
+ type OfflineHistoryPointApi,
+} from '../generated/api.schemas'
+import { offlineRunSourceLabel } from './offlineOverviewState'
+
+export type OfflineHistoryBreakdown = NonNullable<AiObservabilityOfflineScorersHistoryListParams['breakdown']>
+
+export const OFFLINE_HISTORY_BREAKDOWN_LABELS: Record<OfflineHistoryBreakdown, string> = {
+ model_version: 'Model',
+ prompt_version: 'Prompt version',
+ application_version: 'Application version',
+ suite_key: 'Suite',
+ run_source: 'Run source',
+}
+
+export const OFFLINE_HISTORY_BREAKDOWN_OPTIONS = Object.values(AiObservabilityOfflineScorersHistoryListBreakdown).map(
+ (value) => ({ value, label: OFFLINE_HISTORY_BREAKDOWN_LABELS[value] })
+)
+
+export function offlineBreakdownValueLabel(
+ value: string | null | undefined,
+ breakdown: OfflineHistoryBreakdown
+): string {
+ if (value == null) {
+ return 'Not specified'
+ }
+ return breakdown === 'run_source' ? offlineRunSourceLabel(value) : value
+}
+
+export function offlineHistoryPointKey(point: OfflineHistoryPointApi): string {
+ return JSON.stringify([point.experiment.id, point.summary.scorer.id, point.breakdown_value ?? null])
+}
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts
index 2ca49f5f5507..6301502b5839 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts
@@ -10,6 +10,42 @@ import {
import { makeOfflineHistoryPoint } from './offlineScoreTrends.fixtures'
describe('offline score trends', () => {
+ it('separates breakdown values, missing values, versions, and periods', () => {
+ const point = makeOfflineHistoryPoint(1, 0.8)
+ point.summary.scorer = { ...point.summary.scorer, version: 1 }
+ const points = [
+ { ...point, breakdown_value: 'model-a' },
+ { ...point, breakdown_value: 'model-b' },
+ { ...point, breakdown_value: null },
+ { ...point, breakdown_value: 'Not specified' },
+ {
+ ...point,
+ breakdown_value: 'model-a',
+ summary: { ...point.summary, scorer: { ...point.summary.scorer, id: 'v2', version: 2 } },
+ },
+ ]
+ const [panel] = buildOfflineTrendPanels(
+ [
+ { key: 'primary', label: 'Selected', points, dateFrom: '2026-01-01', dateTo: '2026-01-10' },
+ {
+ key: 'previous',
+ label: 'Previous',
+ points: [points[0]],
+ dateFrom: '2025-12-23',
+ dateTo: '2026-01-01',
+ },
+ ],
+ false,
+ 'model_version'
+ )
+ expect(panel.series).toHaveLength(6)
+ expect(new Set(panel.series.map((series) => series.key)).size).toBe(6)
+ expect(panel.series[0].label).toBe('model-a · Selected · v1')
+ expect(panel.series[4].label).toBe('model-a · Selected · v2')
+ expect(panel.series[5].label).toBe('model-a · Previous · v1')
+ expect(panel.series.every((series) => series.points.length === 1)).toBe(true)
+ })
+
it.each([
['gte', 0.9999999, 1, '0.9999999'],
['lte', 1.0000001, 1, '1.0000001'],
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts
index 40a9f2d92d29..cc1039054f27 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts
@@ -9,6 +9,7 @@ import type {
OfflineScorerSummaryApi,
OfflineScorerVersionReadApi,
} from '../generated/api.schemas'
+import { offlineBreakdownValueLabel, type OfflineHistoryBreakdown } from './offlineHistoryBreakdown'
import {
offlineBooleanPolarity,
offlineCategoricalPassingRule,
@@ -238,10 +239,14 @@ export function getOfflineHistoryCoverage(page: OfflineHistoryPageApi, timezone:
.tz(timezone)
.format('MMM D, YYYY')}`
: ''
- return `${page.results.length} of ${page.count} experiment/version results${span}${page.count > page.results.length ? ' · Partial history' : ''}`
+ return `${page.results.length} of ${page.count} history points${span}${page.count > page.results.length ? ' · Partial history' : ''}`
}
-export function buildOfflineTrendPanels(periods: OfflineTrendPeriod[], combineVersions = false): OfflineTrendPanel[] {
+export function buildOfflineTrendPanels(
+ periods: OfflineTrendPeriod[],
+ combineVersions = false,
+ breakdown?: OfflineHistoryBreakdown
+): OfflineTrendPanel[] {
const durations = periods.map((period) =>
period.dateFrom && period.dateTo ? Date.parse(period.dateTo) - Date.parse(period.dateFrom) : null
)
@@ -298,12 +303,17 @@ export function buildOfflineTrendPanels(periods: OfflineTrendPeriod[], combineVe
},
]
for (const metric of metrics) {
- const key = `${period.key}:${scorer.id}:${metric.key}`
+ const key = JSON.stringify([
+ period.key,
+ scorer.id,
+ metric.key,
+ breakdown ? (point.breakdown_value ?? null) : null,
+ ])
let series = panel.series.find((candidate) => candidate.key === key)
if (!series) {
series = {
key,
- label: `${periods.length > 1 ? `${period.label} · ` : ''}v${scorer.version}${scorer.kind === 'categorical' ? ` · ${metric.label}` : ''}`,
+ label: `${breakdown ? `${offlineBreakdownValueLabel(point.breakdown_value, breakdown)} · ` : ''}${periods.length > 1 ? `${period.label} · ` : ''}v${scorer.version}${scorer.kind === 'categorical' ? ` · ${metric.label}` : ''}`,
points: [],
shape: periodIndex === 0 ? 'circle' : 'square',
}
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts
index 7afe36046410..651cf213c358 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts
@@ -37,6 +37,29 @@ describe('offlineScorerHistoryLogic', () => {
})
})
+ it('restores breakdowns from the URL and resets pagination when they change', async () => {
+ const props = { scorerId: OFFLINE_STORY_DEFINITION.id, teamId: 1 }
+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId), { breakdown: 'model_version' })
+ const logic = offlineScorerHistoryLogic(props)
+ logic.mount()
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.query.breakdown).toBe('model_version')
+ await expectLogic(logic, () => logic.actions.changePage('primary', 'older')).toFinishAllListeners()
+ await expectLogic(logic, () =>
+ logic.actions.setFilters({ breakdown: 'suite_key', compare: 'previous' })
+ ).toFinishAllListeners()
+ expect(logic.values.primaryCursors).toEqual([null])
+ expect(router.values.searchParams.breakdown).toBe('suite_key')
+ expect(api.aiObservabilityOfflineScorersHistoryList).toHaveBeenLastCalledWith(
+ '1',
+ props.scorerId,
+ expect.objectContaining({ breakdown: 'suite_key', cursor: undefined })
+ )
+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId), { breakdown: 'unknown' })
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.query.breakdown).toBeUndefined()
+ })
+
it.each([{ created_by: 42 }, { created_by_is_null: true }])(
'preserves author and tag filters across periods and clears them through the URL (%j)',
async (authorFilters) => {
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts
index 00d34bf374d3..570b633afa99 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts
@@ -37,6 +37,7 @@ import type {
ScoreDefinitionVersionApi,
ScoreDefinitionVersionPageApi,
} from '../generated/api.schemas'
+import { OFFLINE_HISTORY_BREAKDOWN_LABELS, type OfflineHistoryBreakdown } from './offlineHistoryBreakdown'
import { OFFLINE_ALL_UPLOAD_STATES, offlineFiltersFromUrl } from './offlineOverviewState'
import {
previousOfflinePeriod,
@@ -54,6 +55,7 @@ export interface OfflineHistoryFilters
ResourceTagFilters,
Pick<AiObservabilityOfflineScorersHistoryListParams, 'created_by' | 'created_by_is_null'> {
version: string
+ breakdown: OfflineHistoryBreakdown | ''
date_from: string
date_to: string | null
compare: 'none' | 'previous' | 'custom'
@@ -65,6 +67,7 @@ export interface OfflineHistoryFilters
const DEFAULT_FILTERS: OfflineHistoryFilters = {
version: '',
+ breakdown: '',
date_from: '-30d',
date_to: null,
compare: 'none',
@@ -87,6 +90,9 @@ function readFilters(search: Record<string, unknown>): OfflineHistoryFilters {
if (filters.compare_to === 'now') {
filters.compare_to = null
}
+ if (!Object.hasOwn(OFFLINE_HISTORY_BREAKDOWN_LABELS, filters.breakdown)) {
+ filters.breakdown = ''
+ }
if (!['none', 'previous', 'custom'].includes(filters.compare)) {
filters.compare = 'none'
}
@@ -526,6 +532,7 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([
(s) => [s.filters],
(filters: OfflineHistoryFilters): AiObservabilityOfflineScorersHistoryListParams => ({
scorer_version_ids: filters.version,
+ breakdown: filters.breakdown || undefined,
limit: 50,
statuses: filters.statuses,
run_source: filters.run_source || undefined,
diff --git a/services/mcp/schema/tool-inputs.json b/services/mcp/schema/tool-inputs.json
index 070a353b16d6..ba766c685162 100644
--- a/services/mcp/schema/tool-inputs.json
+++ b/services/mcp/schema/tool-inputs.json
@@ -810,6 +810,18 @@
"type": "string",
"description": "Caller-generated UUID for one input/output execution. Reuse for exact retries."
},
+ "model_version": {
+ "description": "Model used for this item. Omitted or null values inherit the experiment's model.",
+ "anyOf": [
+ {
+ "type": "string",
+ "maxLength": 255
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
"case_key": {
"description": "Stable case identifier for matching inputs across experiments.",
"anyOf": [
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index cde3bd4b47ef..166804156d5c 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -65202,6 +65202,12 @@ export namespace Schemas {
export interface ItemSubmission {
/** Caller-generated UUID for one input/output execution. Reuse for exact retries. */
id: string;
+ /**
+ * Model used for this item. Omitted or null values inherit the experiment's model.
+ * @maxLength 255
+ * @nullable
+ */
+ model_version?: string | null;
/**
* Stable case identifier for matching inputs across experiments.
* @maxLength 255
@@ -74010,6 +74016,11 @@ export namespace Schemas {
}
export interface OfflineHistoryPoint {
+ /**
+ * Breakdown value for this summary; null when unspecified or no breakdown is selected.
+ * @nullable
+ */
+ breakdown_value?: string | null;
/** Experiment execution and cohort context. */
experiment: OfflineExperimentRead;
/** Complete summary for one experiment and scorer version. */
@@ -74089,6 +74100,11 @@ export namespace Schemas {
}
export interface OfflineItemRead {
+ /**
+ * Model used for the item, falling back to the experiment's model.
+ * @nullable
+ */
+ model_version?: string | null;
/** Stable item UUID. */
id: string;
/** Owning experiment UUID. */
@@ -127888,6 +127904,17 @@ export namespace Schemas {
* @maxLength 255
*/
application_version?: string;
+ /**
+ * Split history by this property. Model uses each item's model, falling back to the experiment's model.
+ *
+ * * `model_version` - Model
+ * * `prompt_version` - Prompt version
+ * * `application_version` - Application version
+ * * `suite_key` - Suite
+ * * `run_source` - Run source
+ * @minLength 1
+ */
+ breakdown?: AiObservabilityOfflineScorersHistoryListBreakdown;
/**
* Filter experiments by the author's user ID.
* @minimum 1
@@ -127991,6 +128018,17 @@ export namespace Schemas {
tags?: string;
};
+ export type AiObservabilityOfflineScorersHistoryListBreakdown = typeof AiObservabilityOfflineScorersHistoryListBreakdown[keyof typeof AiObservabilityOfflineScorersHistoryListBreakdown];
+
+
+ export const AiObservabilityOfflineScorersHistoryListBreakdown = {
+ ModelVersion: 'model_version',
+ PromptVersion: 'prompt_version',
+ ApplicationVersion: 'application_version',
+ SuiteKey: 'suite_key',
+ RunSource: 'run_source',
+ } as const;
+
export type AiObservabilityOfflineScorersHistoryListTagMatch = typeof AiObservabilityOfflineScorersHistoryListTagMatch[keyof typeof AiObservabilityOfflineScorersHistoryListTagMatch];
diff --git a/services/mcp/src/generated/ai_observability/api.ts b/services/mcp/src/generated/ai_observability/api.ts
index b789533a214d..c4cda045eab7 100644
--- a/services/mcp/src/generated/ai_observability/api.ts
+++ b/services/mcp/src/generated/ai_observability/api.ts
@@ -532,6 +532,8 @@ export const AiObservabilityOfflineExperimentsUploadCreateParams = () => zod.obj
),
})
+export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax = 255
+
export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax = 255
export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemTrialMax = 255
@@ -559,6 +561,11 @@ export const AiObservabilityOfflineExperimentsUploadCreateBody = () => zod.objec
id: zod
.string()
.describe('Caller-generated UUID for one input\/output execution. Reuse for exact retries.'),
+ model_version: zod
+ .string()
+ .max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax)
+ .nullish()
+ .describe("Model used for this item. Omitted or null values inherit the experiment's model."),
case_key: zod
.string()
.max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax)
@@ -739,6 +746,12 @@ export const AiObservabilityOfflineScorersHistoryListQueryParams = () => zod.obj
.max(aiObservabilityOfflineScorersHistoryListQueryApplicationVersionMax)
.optional()
.describe('Exact application revision.'),
+ breakdown: zod
+ .enum(['model_version', 'prompt_version', 'application_version', 'suite_key', 'run_source'])
+ .optional()
+ .describe(
+ "Split history by this property. Model uses each item's model, falling back to the experiment's model.\n\n\* `model_version` - Model\n\* `prompt_version` - Prompt version\n\* `application_version` - Application version\n\* `suite_key` - Suite\n\* `run_source` - Run source"
+ ),
created_by: zod.number().min(1).optional().describe("Filter experiments by the author's user ID."),
created_by_is_null: zod.boolean().optional().describe('Only include experiments without an author.'),
cursor: zod
diff --git a/services/mcp/src/tools/generated/ai_observability.ts b/services/mcp/src/tools/generated/ai_observability.ts
index c31d051b7f5d..1d0ed7c8d70f 100644
--- a/services/mcp/src/tools/generated/ai_observability.ts
+++ b/services/mcp/src/tools/generated/ai_observability.ts
@@ -2078,6 +2078,7 @@ const llmaOfflineScorerHistory = (): ToolBase<
path: `/api/projects/${encodeURIComponent(String(projectId))}/ai_observability/offline_scorers/${encodeURIComponent(String(params.id))}/history/`,
query: {
application_version: params.application_version,
+ breakdown: params.breakdown,
created_by: params.created_by,
created_by_is_null: params.created_by_is_null,
cursor: params.cursor,
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json
index 31c3964e71c7..1a95ce5e69be 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json
@@ -72,6 +72,18 @@
"description": "Caller-generated UUID for one input/output execution. Reuse for exact retries.",
"type": "string"
},
+ "model_version": {
+ "anyOf": [
+ {
+ "maxLength": 255,
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Model used for this item. Omitted or null values inherit the experiment's model."
+ },
"payload": {
"additionalProperties": false,
"description": "Optional input/output payload, up to 1 MiB and 32 JSON levels. Omission, {} and null properties differ.",
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json
index ac613c023cb0..6f6adf23ceca 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json
@@ -7,6 +7,11 @@
"minLength": 1,
"type": "string"
},
+ "breakdown": {
+ "description": "Split history by this property. Model uses each item's model, falling back to the experiment's model.\n\n* `model_version` - Model\n* `prompt_version` - Prompt version\n* `application_version` - Application version\n* `suite_key` - Suite\n* `run_source` - Run source",
+ "enum": ["model_version", "prompt_version", "application_version", "suite_key", "run_source"],
+ "type": "string"
+ },
"created_by": {
"description": "Filter experiments by the author's user ID.",
"minimum": 1,