:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): add score history breakdowns (#122787)
Radu Raiceaa9a57fa67da1
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 9f808559ebb1..97ac2d64a98b 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -633,6 +633,19 @@ Saving a new configuration checks the version observed when editing started. A cThe **More → Create new version** action can create an unchanged configuration version after confirmation. Save or discard a dirty draft before using this action.The **Offline evals history** button in the editor opens a scorer's experiment timeline.With the offline feature enabled, a scorer timeline offers upload state, source, date range, comparison, and exact version filters.+The filters use compact controls. Selected and comparison period timestamps use the project timezone, matching the date range boundaries, chart labels, and coverage dates; hover a timestamp for full timezone details.+**Break down by** splits score history by Model, Prompt version, Application version, Suite, or Run source.+Model uses the item's optional `model_version`, falling back to the experiment's `model_version` when omitted or null.+The scorer history `model_version` filter uses this resolved item model, with or without a breakdown. Scores and coverage include only matching items, even when the experiment's default model differs. Experiment list filters continue to match the experiment's model.+The other breakdowns use experiment fields. Item reads expose the resolved model.+Each point summarizes one experiment, scorer version, and breakdown value, with coverage and missing-result counts for that group.+Unspecified values appear as **Not specified**. Different scorer versions, categories, and comparison periods remain separate.+History pagination counts these groups when a breakdown is selected, so a page can contain multiple points from one experiment.+Coverage aggregation includes only the page's experiment/breakdown groups, including their items without results.+Pagination supports Unicode breakdown values up to the full 255-character identifier limit.+The graph and table show the current page; partial history may omit other breakdown values.+The breakdown is stored in the URL. Custom metadata breakdowns are not supported.+Its experiment table keeps names and coverage on one line, shows execution time in its own column, and opens run details with the arrow beside each name.Compare against a previous or custom period.Custom comparison periods ending now retain that end when their URL is shared or reloaded.diff --git a/frontend/snapshots.yml b/frontend/snapshots.ymlindex ff89099bb725..e2e3948b7201 100644--- a/frontend/snapshots.yml+++ b/frontend/snapshots.yml@@ -113,17 +113,21 @@ snapshots:ai-observability-offline-experiments-score-trends--numeric-passing-rule--light:hash: v1.k794b7964.e5367e5503e0e7a14586290abedd1f8f25d9a938013d330a9b3783a9377e32cf.2SZhg6Ri2BlDEHtHrLx7RAbuPEUQFdP-D1FP3_tOPNIai-observability-offline-experiments-scorer-history--default--dark:- hash: v1.k794b7964.de382c36f53d24e416c8a816a1cf194c9ac751b5eebfc7a373932035fecc1acb._coWDzE6lvhhThqoT_ASaYs0iiqQvERpqiwT5KhGunM+ hash: v1.k794b7964.fa9847952affeb01cb37985844b373ce28cef402023df05737f8394189530b67.ccaDPhq_JGTa8n5cMvHqbAj9f2hkoNU9nUhCwFhfGXkai-observability-offline-experiments-scorer-history--default--light:- hash: v1.k794b7964.15cbf9e2bdbbe89d7661c94ccd15d7732d828f564add36cdc067757e03a4a061.A0kjkj1ghKYsLEpla_XwYBMETI7KMu6YOB5BCiLbaxA+ hash: v1.k794b7964.cc570ccb3a6b69c3e5a3add68e0b2faa3416bb81478ec97a3c763f988f582b67.Xbu9IZQNsowPb-o9i_RgTtvdOrXgTcERQBbaznPxFpcai-observability-offline-experiments-scorer-history--empty-history--dark:- hash: v1.k794b7964.7fc63300ab3d7bf38a3058a285f6ecc6e36ffaae97b532deef09d13ed6cbafca.EaYco8jti78XKAqv4PEAm_z3n3MvWyTgBGVRhkVD5do+ hash: v1.k794b7964.ae856e0b664f4e0029c23628c5e85098f42997e0526f24806e67dd3b433e6e7c.QuZHln7HMBUPuLH0qF-GLrrlmsgihcR5T4lBgf9UVfAai-observability-offline-experiments-scorer-history--empty-history--light:- hash: v1.k794b7964.95f6a34d52a423471450e0a21f00dd8af7cfaf36281b07c02323b79d05e9fb3f.1cb98HkzcmAOIVK-QmbF1Yykej6T_sSVSuWGwRoUj-c+ hash: v1.k794b7964.cda80a0c7f01b7e8c3ed4e811c189ccec024f1951abb8818167c431588e08ad4.6R0HRb2No1gsnW3c6FVXJpsKFmDRsRtI8b-TM7OzBvM+ ai-observability-offline-experiments-scorer-history--model-breakdown--dark:+ hash: v1.k794b7964.3232e0e44e04b8f402982aedb044a5ab7ef0bb8f666029eaceaafd342343df9c.UsoEzw4c1A3lspMlDgPae58EHaBJund2nYnv_OxtN1U+ ai-observability-offline-experiments-scorer-history--model-breakdown--light:+ hash: v1.k794b7964.81a77adc5d7de5fdb154ff4a3d35c2a3ba834a264da0708a6735acdd40c81269.oWsL-dscrLGH1bw3KivQnoZoOaN_hWDRWyUVROtLwscai-observability-offline-experiments-scorer-history--narrow--dark:- hash: v1.k794b7964.2d60979cab1b756742b3fef740e8c13a9b688f3eca76ddf7ae73dc08f2f58e64.48EDAeGamPjXu5HwCBjlgx4Z6130i2a_6kqwJ25p9jw+ hash: v1.k794b7964.542a493a106271a9e78d761dd9fc7cc42108fff9c7336526ec687b98194d1063.t1b0Xq-37lNeB72sxqocIUjc_0vms7Cwa-wwyH9ub-Mai-observability-offline-experiments-scorer-history--narrow--light:- hash: v1.k794b7964.4842515b727957419dffc97a061cd561a640c5b3d12357415b2c6b0c0a5b2729.7EOWporG0PIqDbpe8ArXsT-fl8w_hNdckYUgKv4uHDY+ hash: v1.k794b7964.ccfb0305fe7765645891a6e299e61794891ba11c48c093cdb498729210949a62.plGrGJumAXp8oFjajN09fPeLcqQpAZh9nUsT9A6uBywai-observability-scorer-editor--boolean-detector--dark:hash: v1.k794b7964.bf295f71ee0750c9703c615cbc53f39849426b32b354fbf8ef0c8915f7f42e57.BVPu0uJuGhNx1_q4FgZ7vDXRQ7IllIygH2Hi6qKoTZwai-observability-scorer-editor--boolean-detector--light:@@ -9173,9 +9177,9 @@ snapshots:scenes-app-ai-observability-offline-experiments-overview--partial-history--light:hash: v1.k794b7964.302ca88f07d59cdabb3dd8b286b13ff39c03b318a35bca99405683c8e3789d64.-Jd-kXQ46NBmqRsF90-2Y23EI8pzdxvbU9FADQWauYQscenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--dark:- hash: v1.k794b7964.4c450840d8d01ddb2eb4817e1fea7f7fdbd288e93ea3c375555b4ceb1ae08311.jwIcdD5ZnyQniw64ArTCvFuX43hlRtSDbHAIU6qowFU+ hash: v1.k794b7964.c475e74caf460fcfc280bd6983fedcdca879dcf011e058d9733b17ace2f0a348.nMKBGsSpB3g3PQXgiziItiY76qLFWGlhN3GLJlzBMUAscenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--light:- hash: v1.k794b7964.5b69ff2c107ab0450df0d7495d19da2d19293d8fe3b453224845365f8b402f3f.UkyFocgNR0N1T0QWFTOa6HcQYW4BdRMv5_qSzgjjFmI+ hash: v1.k794b7964.568583aab6024942a42b957f625ce22201ee919609bc8ffa1b37e3bb90d0c0f7.c0AsgQZRr9uq6lC9Q3pghFN6ckTHFNdPYhnocXJ0Oigscenes-app-ai-observability-offline-experiments-overview--recent-experiments--dark:hash: v1.k794b7964.73de5e6dff8607fbad95d027b0b2856095e68f28922faa4f06315f6338c8012f.YbQpwN5pANDtCZDWgkZ8z9-XfMnI8dB5_HsfibRZ1z0scenes-app-ai-observability-offline-experiments-overview--recent-experiments--light:diff --git a/products/ai_observability/backend/api/offline_experiment_read_serializers.py b/products/ai_observability/backend/api/offline_experiment_read_serializers.pyindex e2c138a1f291..8316e4b74af8 100644--- a/products/ai_observability/backend/api/offline_experiment_read_serializers.py+++ b/products/ai_observability/backend/api/offline_experiment_read_serializers.py@@ -4,6 +4,7 @@ from uuid import UUIDfrom drf_spectacular.utils import extend_schema_serializerfrom rest_framework import serializers+from rest_framework_dataclasses.fields import EnumFieldfrom rest_framework_dataclasses.serializers import DataclassSerializerfrom posthog.api.shared import UserBasicSerializer@@ -24,6 +25,7 @@ from products.ai_observability.backend.models.score_definitions import ScoreDefifrom products.ai_observability.backend.offline_evaluation_read_types import (MAX_READ_SCORER_VERSIONS,MAX_RESULT_CELL_ITEMS,+ OfflineHistoryBreakdown,OfflineReadQuery,OfflineResultCellQuery,)@@ -211,6 +213,18 @@ class OfflineExperimentQuerySerializer(OfflineHistoryQuerySerializer):fields = [*OfflineHistoryQuerySerializer.Meta.fields, "scorer_definition_id"]+class OfflineScorerHistoryQuerySerializer(OfflineHistoryQuerySerializer):+ breakdown = EnumField(+ enum_class=OfflineHistoryBreakdown,+ choices=OfflineHistoryBreakdown.choices,+ required=False,+ help_text="Split history by this property. Model uses each item's model, falling back to the experiment's model.",+ )++ class Meta(OfflineHistoryQuerySerializer.Meta):+ fields = [*OfflineHistoryQuerySerializer.Meta.fields, "breakdown"]++class OfflineExperimentReadSerializer(serializers.Serializer):created_by = UserBasicSerializer(read_only=True, allow_null=True, help_text="User who submitted the experiment, or null for service credentials."@@ -310,6 +324,9 @@ class OfflineResultCellsSerializer(serializers.Serializer):class OfflineItemReadSerializer(serializers.Serializer):+ model_version = serializers.CharField(+ allow_null=True, required=False, help_text="Model used for the item, falling back to the experiment's model."+ )id = serializers.UUIDField(help_text="Stable item UUID.")experiment_id = serializers.UUIDField(help_text="Owning experiment UUID.")case_key = serializers.CharField(allow_null=True, help_text="Optional stable case identifier.")@@ -427,6 +444,11 @@ class OfflineScorerSummarySerializer(serializers.Serializer):class OfflineHistoryPointSerializer(serializers.Serializer):+ breakdown_value = serializers.CharField(+ allow_null=True,+ required=False,+ help_text="Breakdown value for this summary; null when unspecified or no breakdown is selected.",+ )experiment = OfflineExperimentReadSerializer(help_text="Experiment execution and cohort context.")summary = OfflineScorerSummarySerializer(help_text="Complete summary for one experiment and scorer version.")diff --git a/products/ai_observability/backend/api/offline_experiment_reads.py b/products/ai_observability/backend/api/offline_experiment_reads.pyindex 9d66a168c018..1610b26dd242 100644--- a/products/ai_observability/backend/api/offline_experiment_reads.py+++ b/products/ai_observability/backend/api/offline_experiment_reads.py@@ -31,7 +31,6 @@ from products.ai_observability.backend.api.offline_experiment_read_serializers iOfflineExperimentQuerySerializer,OfflineExperimentReadSerializer,OfflineHistoryPageSerializer,- OfflineHistoryQuerySerializer,OfflineItemPageSerializer,OfflineItemPayloadReadSerializer,OfflineItemReadSerializer,@@ -40,6 +39,7 @@ from products.ai_observability.backend.api.offline_experiment_read_serializers iOfflineResultPageSerializer,OfflineResultPayloadReadSerializer,OfflineResultQuerySerializer,+ OfflineScorerHistoryQuerySerializer,OfflineSummaryPageSerializer,OfflineSummaryQuerySerializer,)@@ -253,7 +253,7 @@ class OfflineExperimentReadViewSet(OfflineEvaluationReadViewSet):class OfflineScorerViewSet(OfflineEvaluationReadViewSet):@validated_request(- query_serializer=OfflineHistoryQuerySerializer,+ query_serializer=OfflineScorerHistoryQuerySerializer,operation_id="ai_observability_offline_scorers_history_list",parameters=[OpenApiParameter("id", UUID, OpenApiParameter.PATH, description="Scorer definition UUID.")],responses={200: OfflineHistoryPageSerializer, **READ_ERROR_RESPONSES},diff --git a/products/ai_observability/backend/api/offline_experiment_serializers.py b/products/ai_observability/backend/api/offline_experiment_serializers.pyindex fc7c9ec7204e..01667456e233 100644--- a/products/ai_observability/backend/api/offline_experiment_serializers.py+++ b/products/ai_observability/backend/api/offline_experiment_serializers.py@@ -247,6 +247,9 @@ class ExperimentSubmissionSerializer(_StrictDataclassSerializer[ExperimentSubmisclass ItemSubmissionSerializer(_StrictDataclassSerializer[ItemSubmission]):id = _UUIDField(help_text="Caller-generated UUID for one input/output execution. Reuse for exact retries.")+ model_version = _identifier_field(+ "Model used for this item. Omitted or null values inherit the experiment's model."+ )case_key = _identifier_field("Stable case identifier for matching inputs across experiments.")trial = _identifier_field("Identifier for a repeated execution of the same case.")dataset_item_identifier = _identifier_field("Stable item identifier in an external dataset.")diff --git a/products/ai_observability/backend/api/test/test_offline_experiment_reads.py b/products/ai_observability/backend/api/test/test_offline_experiment_reads.pyindex 5a85684b44da..d360e33945c6 100644--- a/products/ai_observability/backend/api/test/test_offline_experiment_reads.py+++ b/products/ai_observability/backend/api/test/test_offline_experiment_reads.py@@ -114,6 +114,19 @@ class TestOfflineExperimentReads(APIBaseTest):self._history(),]+ def test_history_model_breakdown_uses_the_item_model(self) -> None:+ self.experiment.model_version = "model-a"+ self.experiment.save(update_fields=["model_version"])+ self.item.model_version = "model-b"+ self.item.save(update_fields=["model_version"])+ response = self.client.get(self._history(), {"breakdown": "model_version", "model_version": "model-b"})+ self.assertEqual(response.status_code, status.HTTP_200_OK, response.data)+ self.assertEqual(response.data["results"][0]["breakdown_value"], "model-b")+ self.assertEqual(response.data["results"][0]["summary"]["pass_rate"], 1.0)+ response = self.client.get(self._history(), {"breakdown": "model_version", "model_version": "model-a"})+ self.assertEqual(response.status_code, status.HTTP_200_OK, response.data)+ self.assertEqual(response.data["count"], 0)+def test_author_filter_applies_to_experiments_and_history(self) -> None:self.user.hedgehog_config = {"version": 2,@@ -531,6 +544,7 @@ class TestOfflineExperimentReads(APIBaseTest):(self._endpoint(), {"limit": 101}, "limit"),(self._endpoint(), {"created_by": "invalid"}, "created_by"),(self._history(), {"created_by": 0}, "created_by"),+ (self._history(), {"breakdown": "unsupported"}, "breakdown"),(self._history(), {"created_by": self.user.id, "created_by_is_null": "true"}, "created_by"),(self._endpoint(f"{self.experiment.id}/"), {"limit": 1}, "limit"),(self._endpoint(f"{self.experiment.id}/items/{self.item.id}/"), {"limit": 1}, "limit"),diff --git a/products/ai_observability/backend/migrations/0068_offline_item_model_version.py b/products/ai_observability/backend/migrations/0068_offline_item_model_version.pynew file mode 100644index 000000000000..b260f2765fc4--- /dev/null+++ b/products/ai_observability/backend/migrations/0068_offline_item_model_version.py@@ -0,0 +1,17 @@+# Generated by Django 5.2.18 on 2026-10-23 14:27++from django.db import migrations, models+++class Migration(migrations.Migration):+ dependencies = [+ ("ai_observability", "0067_scorer_short_name"),+ ]++ operations = [+ migrations.AddField(+ model_name="offlineexperimentitem",+ name="model_version",+ field=models.CharField(blank=True, max_length=255, null=True),+ ),+ ]diff --git a/products/ai_observability/backend/migrations/max_migration.txt b/products/ai_observability/backend/migrations/max_migration.txtindex eb7e75cc5f34..652d1094d6d4 100644--- a/products/ai_observability/backend/migrations/max_migration.txt+++ b/products/ai_observability/backend/migrations/max_migration.txt@@ -1 +1 @@-0067_scorer_short_name+0068_offline_item_model_versiondiff --git a/products/ai_observability/backend/models/offline_evaluations.py b/products/ai_observability/backend/models/offline_evaluations.pyindex c2304a7c8a61..a744b00363ab 100644--- a/products/ai_observability/backend/models/offline_evaluations.py+++ b/products/ai_observability/backend/models/offline_evaluations.py@@ -123,6 +123,7 @@ class OfflineExperimentItem(OfflinePayloadOwner):experiment = models.ForeignKey(OfflineExperiment, on_delete=models.CASCADE, related_name="items", db_index=False)case_key = models.CharField(max_length=255, null=True, blank=True)trial = models.CharField(max_length=255, null=True, blank=True)+ model_version = models.CharField(max_length=255, null=True, blank=True)dataset_item_identifier = models.CharField(max_length=255, null=True, blank=True)dataset_item_version_identifier = models.CharField(max_length=255, null=True, blank=True)dataset_item_version = models.ForeignKey(diff --git a/products/ai_observability/backend/offline_evaluation_fingerprint.py b/products/ai_observability/backend/offline_evaluation_fingerprint.pyindex 9117ebc6c755..5118001e74a3 100644--- a/products/ai_observability/backend/offline_evaluation_fingerprint.py+++ b/products/ai_observability/backend/offline_evaluation_fingerprint.py@@ -27,5 +27,8 @@ def _normalize(value: object) -> object:def submission_fingerprint(submission: ExperimentSubmission | ItemSubmission | ResultSubmission) -> str:# This format must stay stable across deployments and after payload deletion.- content = json.dumps(_normalize(asdict(submission)), sort_keys=True, separators=(",", ":"), allow_nan=False)+ fields = asdict(submission)+ if isinstance(submission, ItemSubmission) and submission.model_version is None:+ fields.pop("model_version")+ content = json.dumps(_normalize(fields), sort_keys=True, separators=(",", ":"), allow_nan=False)return hashlib.sha256(b"offline-evaluation:v1:" + content.encode("utf-8")).hexdigest()diff --git a/products/ai_observability/backend/offline_evaluation_read_service.py b/products/ai_observability/backend/offline_evaluation_read_service.pyindex 8c54fd8bc1fb..1e74790f33ef 100644--- a/products/ai_observability/backend/offline_evaluation_read_service.py+++ b/products/ai_observability/backend/offline_evaluation_read_service.py@@ -10,7 +10,8 @@ from typing import TYPE_CHECKING, castfrom uuid import UUIDfrom django.db import connections-from django.db.models import Case, Count, Exists, F, OuterRef, Q, When, prefetch_related_objects+from django.db.models import Case, Count, Exists, F, OuterRef, Q, Value, When, prefetch_related_objects+from django.db.models.functions import Coalescefrom posthog.dataclasses import frozen@@ -27,6 +28,7 @@ from products.ai_observability.backend.offline_evaluation_read_types import (OfflineCategorySummary,OfflineExperimentAuthor,OfflineExperimentRead,+ OfflineHistoryBreakdown,OfflineHistoryPoint,OfflineItemPage,OfflineItemRead,@@ -48,15 +50,23 @@ from products.ai_observability.backend.read_pagination import CursorPage, cursorfrom products.ai_observability.backend.resource_tags import ResourceTagsif TYPE_CHECKING:+ from typing import Annotated, TypedDict+from django.db.models import QuerySet+ from django_stubs_ext import Annotations+from products.access_control.backend.facade.user_access_control import UserAccessControl+ class _ItemAnnotations(TypedDict):+ resolved_model_version: str | None+@frozenclass _SummaryIdentity:experiment_id: UUIDversion_id: UUID+ breakdown_value: str = ""@frozen@@ -94,6 +104,7 @@ class _TimePosition:started_at: datetimeexperiment_id: UUIDversion_id: UUID | None+ breakdown_value: str = ""class OfflineEvaluationReadService:@@ -105,8 +116,14 @@ class OfflineEvaluationReadService:def _experiments(self) -> QuerySet[OfflineExperiment]:return OfflineExperiment.objects.for_team(self.team_id).select_related("created_by")- def _items(self) -> QuerySet[OfflineExperimentItem]:- return OfflineExperimentItem.objects.for_team(self.team_id)+ @staticmethod+ def _model_expression(*, item_prefix: str = "") -> Coalesce:+ return Coalesce(F(f"{item_prefix}model_version"), F(f"{item_prefix}experiment__model_version"))++ def _items(self) -> QuerySet[Annotated[OfflineExperimentItem, Annotations[_ItemAnnotations]]]:+ return OfflineExperimentItem.objects.for_team(self.team_id).annotate(+ resolved_model_version=self._model_expression()+ )def _definitions(self) -> QuerySet[ScoreDefinition]:definitions = ScoreDefinition.objects.filter(team_id=self.team_id)@@ -135,7 +152,9 @@ class OfflineEvaluationReadService:if versions.count() != len(query.scorer_version_ids):raise OfflineEvaluationNotFound- def _results(self, query: OfflineReadQuery | None = None) -> QuerySet[OfflineEvaluationResult]:+ def _results(+ self, query: OfflineReadQuery | None = None, *, model_version: str | None = None+ ) -> QuerySet[OfflineEvaluationResult]:results = OfflineEvaluationResult.objects.for_team(self.team_id).filter(scorer_definition__in=self._definitions())@@ -144,6 +163,10 @@ class OfflineEvaluationReadService:results = results.filter(scorer_definition_id=query.scorer_definition_id)if query.scorer_version_ids:results = results.filter(scorer_version_id__in=query.scorer_version_ids)+ if model_version is not None:+ return results.alias(resolved_model_version=self._model_expression(item_prefix="item__")).filter(+ resolved_model_version=model_version+ )return resultsdef _require_experiment(self, experiment_id: UUID) -> OfflineExperiment:@@ -152,7 +175,9 @@ class OfflineEvaluationReadService:raise OfflineEvaluationNotFoundreturn experiment- def _require_item(self, experiment_id: UUID, item_id: UUID) -> OfflineExperimentItem:+ def _require_item(+ self, experiment_id: UUID, item_id: UUID+ ) -> Annotated[OfflineExperimentItem, Annotations[_ItemAnnotations]]:item = self._items().filter(experiment_id=experiment_id, id=item_id).first()if item is None:raise OfflineEvaluationNotFound@@ -184,12 +209,13 @@ class OfflineEvaluationReadService:"dataset_identifier","dataset_revision_identifier","application_version",- "model_version","prompt_version",):value = getattr(query, field)if value is not None:experiments = experiments.filter(**{field: value})+ if query.model_version is not None and not history:+ experiments = experiments.filter(model_version=query.model_version)if query.scorer_definition_id is not None or query.scorer_version_ids:experiments = experiments.filter(Exists(self._results(query).filter(item__experiment_id=OuterRef("id"))))return experiments@@ -198,6 +224,8 @@ class OfflineEvaluationReadService:filters = asdict(query)filters.pop("cursor")filters.pop("limit")+ if query.breakdown is None:+ filters.pop("breakdown")content = json.dumps([self.team_id, self.can_read_scores, scope, filters], default=str, sort_keys=True)return hashlib.sha256(content.encode()).hexdigest()[:24]@@ -223,7 +251,7 @@ class OfflineEvaluationReadService:raise OfflineEvaluationValidationError("cursor", "Provide a valid continuation cursor.") from errordef _time_cursor(self, query: OfflineReadQuery, scope: str, *, history: bool = False) -> _TimePosition | None:- parts = self._cursor(query, scope, 3 if history else 2)+ parts = self._cursor(query, scope, (4 if query.breakdown else 3) if history else 2)if parts is None:return Nonetry:@@ -231,7 +259,10 @@ class OfflineEvaluationReadService:if started_at.tzinfo is None:raise ValueErrorreturn _TimePosition(- started_at=started_at, experiment_id=UUID(parts[1]), version_id=UUID(parts[2]) if history else None+ started_at=started_at,+ experiment_id=UUID(parts[1]),+ version_id=UUID(parts[2]) if history else None,+ breakdown_value=parts[3] if history and query.breakdown else "",)except ValueError as error:raise OfflineEvaluationValidationError("cursor", "Provide a valid continuation cursor.") from error@@ -395,11 +426,14 @@ class OfflineEvaluationReadService:)def _item_read(- self, item: OfflineExperimentItem, results: list[OfflineResultRead] | None = None+ self,+ item: Annotated[OfflineExperimentItem, Annotations[_ItemAnnotations]],+ results: list[OfflineResultRead] | None = None,) -> OfflineItemRead:return OfflineItemRead(id=item.id,experiment_id=item.experiment_id,+ model_version=item.resolved_model_version,case_key=item.case_key,trial=item.trial,dataset_item_identifier=item.dataset_item_identifier,@@ -557,12 +591,29 @@ class OfflineEvaluationReadService:)return None+ @classmethod+ def _breakdown_expression(cls, breakdown: OfflineHistoryBreakdown | None, *, item_prefix: str) -> Coalesce | Value:+ if breakdown is None:+ return Value("")+ if breakdown == OfflineHistoryBreakdown.MODEL:+ return Coalesce(cls._model_expression(item_prefix=item_prefix), Value(""))+ return Coalesce(F(f"{item_prefix}experiment__{breakdown.value}"), Value(""))+def _aggregates(- self, identities: list[_SummaryIdentity], *, versions: dict[UUID, ScoreDefinitionVersion]+ self,+ identities: list[_SummaryIdentity],+ *,+ versions: dict[UUID, ScoreDefinitionVersion],+ breakdown: OfflineHistoryBreakdown | None = None,+ model_version: str | None = None,) -> dict[_SummaryIdentity, _Aggregate]:selected = Q(pk__in=[])for identity in identities:- selected |= Q(item__experiment_id=identity.experiment_id, scorer_version_id=identity.version_id)+ selected |= Q(+ item__experiment_id=identity.experiment_id,+ scorer_version_id=identity.version_id,+ breakdown_value=identity.breakdown_value,+ )passing = Q(pk__in=[])configured_versions = set()for version_id, version in versions.items():@@ -571,7 +622,8 @@ class OfflineEvaluationReadService:configured_versions.add(version_id)passing |= Q(scorer_version_id=version_id) & conditionresults = (- self._results()+ self._results(model_version=model_version)+ .annotate(breakdown_value=self._breakdown_expression(breakdown, item_prefix="item__")).filter(selected).order_by().annotate(@@ -585,16 +637,31 @@ class OfflineEvaluationReadService:"boolean_value","categorical_values","score_passes",+ "breakdown_value",))sql, params = results.query.sql_with_params()- item_table = OfflineExperimentItem._meta.db_table+ selected_item_groups = Q(pk__in=[])+ for experiment_id, breakdown_value in sorted(+ {(identity.experiment_id, identity.breakdown_value) for identity in identities}+ ):+ selected_item_groups |= Q(experiment_id=experiment_id, breakdown_value=breakdown_value)+ item_query = (+ self._items()+ .annotate(breakdown_value=self._breakdown_expression(breakdown, item_prefix=""))+ .filter(selected_item_groups)+ .order_by()+ .values("experiment_id", "id", "case_key", "trial", "breakdown_value")+ )+ if model_version is not None:+ item_query = item_query.filter(resolved_model_version=model_version)+ item_sql, item_params = item_query.query.sql_with_params()# The text conversion preserves float precision before numeric accumulation prevents overflow.aggregate_sql = f"""WITH selected AS ({sql}), aggregates AS (- SELECT experiment_id, scorer_version_id, count(*) AS result_count,+ SELECT experiment_id, scorer_version_id, breakdown_value, count(*) AS result_count,count(*) FILTER (WHERE status = 'ok') AS ok_count,count(*) FILTER (WHERE status = 'error') AS error_count,count(*) FILTER (WHERE status = 'skipped') AS skipped_count,@@ -603,35 +670,34 @@ class OfflineEvaluationReadService:count(*) FILTER (WHERE status = 'ok' AND boolean_value IS TRUE) AS true_count,count(*) FILTER (WHERE status = 'ok' AND boolean_value IS FALSE) AS false_count,count(*) FILTER (WHERE status = 'ok' AND score_passes) AS pass_count- FROM selected GROUP BY experiment_id, scorer_version_id+ FROM selected GROUP BY experiment_id, scorer_version_id, breakdown_value), category_counts AS (- SELECT experiment_id, scorer_version_id, category, count(*) AS selected_count+ SELECT experiment_id, scorer_version_id, breakdown_value, category, count(*) AS selected_countFROM selected CROSS JOIN LATERAL unnest(categorical_values) AS categoryWHERE status = 'ok'- GROUP BY experiment_id, scorer_version_id, category+ GROUP BY experiment_id, scorer_version_id, breakdown_value, category), categories AS (- SELECT experiment_id, scorer_version_id,+ SELECT experiment_id, scorer_version_id, breakdown_value,array_agg(category ORDER BY category) AS keys,array_agg(selected_count ORDER BY category) AS counts- FROM category_counts GROUP BY experiment_id, scorer_version_id+ FROM category_counts GROUP BY experiment_id, scorer_version_id, breakdown_value), coverage AS (- SELECT experiment_id, count(*) AS observed_item_count, count(DISTINCT case_key) AS distinct_case_count,+ SELECT experiment_id, breakdown_value, count(*) AS observed_item_count, count(DISTINCT case_key) AS distinct_case_count,count(*) FILTER (WHERE case_key IS NOT NULL) AS items_with_case_key_count,count(*) FILTER (WHERE case_key IS NULL) AS items_without_case_key_count,count(*) FILTER (WHERE trial IS NOT NULL) AS trial_item_count,count(DISTINCT (case_key, CASE WHEN case_key IS NULL THEN id END, trial))FILTER (WHERE trial IS NOT NULL) AS distinct_trial_count- FROM {item_table}- WHERE team_id = %s AND experiment_id = ANY(%s)- GROUP BY experiment_id+ FROM ({item_sql}) AS selected_items+ GROUP BY experiment_id, breakdown_value)SELECT a.experiment_id, a.scorer_version_id, a.result_count, a.ok_count, a.error_count,a.skipped_count, a.not_applicable_count, a.mean, a.true_count, a.false_count,c.observed_item_count, c.distinct_case_count, c.items_with_case_key_count,c.items_without_case_key_count, c.trial_item_count, c.distinct_trial_count,- categories.keys, categories.counts, a.pass_count- FROM aggregates a JOIN coverage c USING (experiment_id)- LEFT JOIN categories USING (experiment_id, scorer_version_id)+ categories.keys, categories.counts, a.pass_count, a.breakdown_value+ FROM aggregates a JOIN coverage c USING (experiment_id, breakdown_value)+ LEFT JOIN categories USING (experiment_id, scorer_version_id, breakdown_value)"""# One statement keeps coverage and category denominators coherent during uploads.with connections[results.db].cursor() as cursor:@@ -639,12 +705,11 @@ class OfflineEvaluationReadService:aggregate_sql,[*params,- self.team_id,- list({identity.experiment_id for identity in identities}),+ *item_params,],)return {- _SummaryIdentity(experiment_id=row[0], version_id=row[1]): _Aggregate(+ _SummaryIdentity(experiment_id=row[0], version_id=row[1], breakdown_value=row[19]): _Aggregate(count=row[2],status_counts=OfflineStatusCounts(ok=row[3], error=row[4], skipped=row[5], not_applicable=row[6]),# Python rounds underflow to zero; PostgreSQL's float cast rejects it.@@ -665,7 +730,13 @@ class OfflineEvaluationReadService:for row in cursor.fetchall()}- def _summaries(self, identities: list[_SummaryIdentity]) -> dict[_SummaryIdentity, OfflineScorerSummary]:+ def _summaries(+ self,+ identities: list[_SummaryIdentity],+ *,+ breakdown: OfflineHistoryBreakdown | None = None,+ model_version: str | None = None,+ ) -> dict[_SummaryIdentity, OfflineScorerSummary]:if not identities:return {}versions = {@@ -674,7 +745,7 @@ class OfflineEvaluationReadService:.filter(id__in=[identity.version_id for identity in identities]).select_related("definition")}- aggregates = self._aggregates(identities, versions=versions)+ aggregates = self._aggregates(identities, versions=versions, breakdown=breakdown, model_version=model_version)summaries = {}for identity in identities:version = versions.get(identity.version_id)@@ -762,12 +833,15 @@ class OfflineEvaluationReadService:query = replace(query, scorer_definition_id=definition_id)self._validate_scorer_selection(query)scope = f"history:{definition_id}"- results = self._results(query).filter(+ results = self._results(query, model_version=query.model_version).filter(scorer_definition_id=definition_id, item__experiment__in=self._filter_experiments(query, history=True))+ results = results.annotate(breakdown_value=self._breakdown_expression(query.breakdown, item_prefix="item__"))groups = (- results.order_by("-item__experiment__started_at", "item__experiment_id", "scorer_version_id")- .values("item__experiment__started_at", "item__experiment_id", "scorer_version_id")+ results.order_by(+ "-item__experiment__started_at", "item__experiment_id", "scorer_version_id", "breakdown_value"+ )+ .values("item__experiment__started_at", "item__experiment_id", "scorer_version_id", "breakdown_value").distinct())count = groups.count()@@ -781,6 +855,12 @@ class OfflineEvaluationReadService:item__experiment_id=position.experiment_id,scorer_version_id__gt=position.version_id,)+ | Q(+ item__experiment__started_at=position.started_at,+ item__experiment_id=position.experiment_id,+ scorer_version_id=position.version_id,+ breakdown_value__gt=position.breakdown_value,+ ))rows = list(groups[: query.limit + 1])page = cursor_page(@@ -794,14 +874,19 @@ class OfflineEvaluationReadService:row["item__experiment__started_at"].isoformat(),str(row["item__experiment_id"]),str(row["scorer_version_id"]),+ *([row["breakdown_value"]] if query.breakdown else []),],),)identities = [- _SummaryIdentity(experiment_id=row["item__experiment_id"], version_id=row["scorer_version_id"])+ _SummaryIdentity(+ experiment_id=row["item__experiment_id"],+ version_id=row["scorer_version_id"],+ breakdown_value=row["breakdown_value"],+ )for row in page.results]- summary_by_id = self._summaries(identities)+ summary_by_id = self._summaries(identities, breakdown=query.breakdown, model_version=query.model_version)experiments = {experiment.id: experimentfor experiment in self._experiment_reads(@@ -812,7 +897,11 @@ class OfflineEvaluationReadService:count=page.count,next_cursor=page.next_cursor,results=[- OfflineHistoryPoint(experiment=experiments[identity.experiment_id], summary=summary_by_id[identity])+ OfflineHistoryPoint(+ experiment=experiments[identity.experiment_id],+ summary=summary_by_id[identity],+ breakdown_value=identity.breakdown_value or None,+ )for identity in identitiesif identity in summary_by_id],diff --git a/products/ai_observability/backend/offline_evaluation_read_types.py b/products/ai_observability/backend/offline_evaluation_read_types.pyindex 52b483c32482..cd908a8be257 100644--- a/products/ai_observability/backend/offline_evaluation_read_types.py+++ b/products/ai_observability/backend/offline_evaluation_read_types.py@@ -3,6 +3,8 @@ from datetime import datetimefrom typing import Literalfrom uuid import UUID+from django.db import models+from posthog.dataclasses import frozenfrom products.ai_observability.backend.offline_evaluation_service import OfflineEvaluationValidationError@@ -13,8 +15,17 @@ MAX_READ_SCORER_VERSIONS = 20MAX_RESULT_CELL_ITEMS = 50+class OfflineHistoryBreakdown(models.TextChoices):+ MODEL = "model_version", "Model"+ PROMPT = "prompt_version", "Prompt version"+ APPLICATION = "application_version", "Application version"+ SUITE = "suite_key", "Suite"+ SOURCE = "run_source", "Run source"++@frozenclass OfflineReadQuery:+ breakdown: OfflineHistoryBreakdown | None = Nonelimit: int = 50cursor: str | None = Nonedate_from: datetime | None = None@@ -143,6 +154,7 @@ class OfflineResultCells:@frozenclass OfflineItemRead:+ model_version: str | None = Noneid: UUIDexperiment_id: UUIDcase_key: str | None@@ -213,3 +225,4 @@ class OfflineScorerSummary:class OfflineHistoryPoint:experiment: OfflineExperimentReadsummary: OfflineScorerSummary+ breakdown_value: str | None = Nonediff --git a/products/ai_observability/backend/offline_evaluation_types.py b/products/ai_observability/backend/offline_evaluation_types.pyindex 1388f139cddd..92f97cdd0ca3 100644--- a/products/ai_observability/backend/offline_evaluation_types.py+++ b/products/ai_observability/backend/offline_evaluation_types.py@@ -32,6 +32,7 @@ class ExperimentSubmission:@frozenclass ItemSubmission:id: UUID+ model_version: str | None = Nonecase_key: str | None = Nonetrial: str | None = Nonedataset_item_identifier: str | None = Nonediff --git a/products/ai_observability/backend/read_pagination.py b/products/ai_observability/backend/read_pagination.pyindex 7fab5060f874..254e8a46ea8b 100644--- a/products/ai_observability/backend/read_pagination.py+++ b/products/ai_observability/backend/read_pagination.py@@ -7,7 +7,11 @@ from posthog.dataclasses import frozendef encode_cursor(values: list[str]) -> str:- return base64.urlsafe_b64encode(json.dumps(values, separators=(",", ":")).encode()).decode().rstrip("=")+ return (+ base64.urlsafe_b64encode(json.dumps(values, separators=(",", ":"), ensure_ascii=False).encode())+ .decode()+ .rstrip("=")+ )def decode_cursor(cursor: str, expected_parts: int) -> list[str]:diff --git a/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py b/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.pyindex 45728e53fca5..baa270743b27 100644--- a/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py+++ b/products/ai_observability/backend/test/test_offline_evaluation_fingerprint.py@@ -1,4 +1,6 @@-from dataclasses import replace+import json+import hashlib+from dataclasses import asdict, replacefrom datetime import datetimefrom uuid import UUID@@ -50,6 +52,21 @@ class TestOfflineEvaluationFingerprint(SimpleTestCase):submission_fingerprint(ItemSubmission(id=ITEM_ID, payload=second)),)+ def test_item_model_preserves_legacy_retries_and_detects_model_changes(self) -> None:+ item = ItemSubmission(id=ITEM_ID)+ legacy_fields = asdict(item)+ legacy_fields.pop("model_version")+ legacy_fields["id"] = str(ITEM_ID)+ legacy_hash = hashlib.sha256(+ b"offline-evaluation:v1:" + json.dumps(legacy_fields, sort_keys=True, separators=(",", ":")).encode()+ ).hexdigest()+ self.assertEqual(submission_fingerprint(item), legacy_hash)+ with_model = replace(item, model_version="model-a")+ self.assertNotEqual(submission_fingerprint(with_model), legacy_hash)+ self.assertNotEqual(+ submission_fingerprint(with_model), submission_fingerprint(replace(item, model_version="model-b"))+ )+def test_equivalent_timestamp_offsets_have_the_same_fingerprint(self) -> None:submission = ExperimentSubmission(id=ITEM_ID, name="Candidate model", started_at=datetime.fromisoformat("2026-09-24T10:00:00+00:00")diff --git a/products/ai_observability/backend/test/test_offline_evaluation_reads.py b/products/ai_observability/backend/test/test_offline_evaluation_reads.pyindex f0e249b9cf14..38d0aa365b86 100644--- a/products/ai_observability/backend/test/test_offline_evaluation_reads.py+++ b/products/ai_observability/backend/test/test_offline_evaluation_reads.py@@ -12,6 +12,9 @@ from parameterized import parameterizedfrom posthog.models import Organization, Team+from products.ai_observability.backend.api.offline_experiment_read_serializers import (+ OfflineScorerHistoryQuerySerializer,+)from products.ai_observability.backend.models.offline_evaluations import (OfflineEvaluationResult,OfflineEvaluationResultPayload,@@ -22,7 +25,11 @@ from products.ai_observability.backend.models.offline_evaluations import ()from products.ai_observability.backend.models.score_definitions import ScoreDefinition, ScoreDefinitionVersionfrom products.ai_observability.backend.offline_evaluation_read_service import OfflineEvaluationReadService-from products.ai_observability.backend.offline_evaluation_read_types import OfflineReadQuery, OfflineStatusCounts+from products.ai_observability.backend.offline_evaluation_read_types import (+ OfflineHistoryBreakdown,+ OfflineReadQuery,+ OfflineStatusCounts,+)from products.ai_observability.backend.offline_evaluation_service import (OfflineEvaluationNotFound,OfflineEvaluationValidationError,@@ -102,13 +109,19 @@ class TestOfflineEvaluationReads(TestCase):)def _item(- self, experiment: OfflineExperiment, *, case_key: str | None = None, trial: str | None = None+ self,+ experiment: OfflineExperiment,+ *,+ case_key: str | None = None,+ trial: str | None = None,+ model_version: str | None = None,) -> OfflineExperimentItem:return OfflineExperimentItem.objects.for_team(self.team.id).create(id=uuid4(),team=self.team,experiment=experiment,case_key=case_key,+ model_version=model_version,trial=trial,submission_fingerprint="a" * 64,)@@ -334,6 +347,138 @@ class TestOfflineEvaluationReads(TestCase):self.assertIsNone(older_page.next_cursor)self.assertEqual(self.service.get_experiment(expected[0]).id, expected[0])+ def test_model_breakdown_uses_item_overrides_and_group_coverage(self) -> None:+ experiment = self._experiment()+ experiment.model_version = "model-a"+ experiment.save(update_fields=["model_version"])+ inherited = self._item(experiment, case_key="one")+ override = self._item(experiment, case_key="one", model_version="model-b")+ error = self._item(experiment, case_key="two", model_version="model-b")+ self._item(experiment, case_key="missing", model_version="model-b")+ self._result(inherited, self.numeric, 1.0)+ self._result(override, self.numeric, 0.0)+ self._result(error, self.numeric, status="error")+ self._result(inherited, self.boolean, True)+ self._result(override, self.boolean, False)+ self._result(inherited, self.categorical, ["a"])+ self._result(override, self.categorical, ["b"])++ for version in (self.numeric, self.boolean, self.categorical):+ page = self.service.scorer_history(+ version.definition_id, OfflineReadQuery(breakdown=OfflineHistoryBreakdown.MODEL)+ )+ self.assertEqual(page.count, 2)+ groups = {point.breakdown_value: point.summary for point in page.results}+ self.assertEqual(groups["model-a"].observed_item_count, 1)+ self.assertEqual(groups["model-b"].observed_item_count, 3)+ self.assertEqual(groups["model-a"].status_counts.ok, 1)+ self.assertEqual(groups["model-b"].status_counts.ok, 1)+ if version == self.numeric:+ self.assertEqual(groups["model-a"].mean, 1.0)+ self.assertEqual(groups["model-b"].mean, 0.0)+ self.assertEqual(groups["model-b"].status_counts.error, 1)+ self.assertEqual(groups["model-b"].missing_result_count, 1)+ elif version == self.boolean:+ self.assertEqual(groups["model-a"].pass_rate, 1.0)+ self.assertEqual(groups["model-b"].pass_rate, 0.0)+ else:+ self.assertEqual([category.rate for category in groups["model-a"].categories], [1.0, 0.0, 0.0])+ self.assertEqual([category.rate for category in groups["model-b"].categories], [0.0, 1.0, 0.0])++ whole = self.service.scorer_history(self.numeric.definition_id, OfflineReadQuery()).results[0].summary+ self.assertEqual((whole.mean, whole.observed_item_count, whole.missing_result_count), (0.5, 4, 1))+ self.assertEqual(self.service.get_item(experiment.id, inherited.id).model_version, "model-a")+ self.assertEqual(self.service.get_item(experiment.id, override.id).model_version, "model-b")++ @parameterized.expand([(None,), (OfflineHistoryBreakdown.MODEL,), (OfflineHistoryBreakdown.SUITE,)])+ def test_model_filter_matches_item_models_across_experiments(+ self, breakdown: OfflineHistoryBreakdown | None+ ) -> None:+ experiments = []+ for index, model in enumerate(("model-a", "model-b")):+ experiment = self._experiment(started_at=self.now - timedelta(hours=index))+ experiment.model_version = model+ experiment.save(update_fields=["model_version"])+ experiments.append(experiment)+ matching_model = None if model == "model-a" else "model-a"+ other_model = "model-b" if model == "model-a" else None+ self._result(self._item(experiment, model_version=matching_model), self.numeric, float(index + 1))+ self._item(experiment, model_version=matching_model)+ self._result(self._item(experiment, model_version=other_model), self.numeric, 100.0)+ self._item(experiment, model_version=other_model)++ query = OfflineReadQuery(model_version="model-a", breakdown=breakdown, limit=1)+ for index, experiment in enumerate(experiments):+ page = self.service.scorer_history(self.numeric.definition_id, query)+ self.assertEqual(page.count, 2)+ self.assertEqual(len(page.results), 1)+ point = page.results[0]+ self.assertEqual(point.experiment.id, experiment.id)+ self.assertEqual(point.breakdown_value, "model-a" if breakdown == OfflineHistoryBreakdown.MODEL else None)+ self.assertEqual(point.summary.mean, float(index + 1))+ self.assertEqual(point.summary.status_counts.ok, 1)+ self.assertEqual(point.summary.observed_item_count, 2)+ self.assertEqual(point.summary.missing_result_count, 1)+ self.assertEqual(page.next_cursor is None, index == 1)+ query = replace(query, cursor=page.next_cursor)++ listed = self.service.list_experiments(OfflineReadQuery(model_version="model-a"))+ self.assertEqual([experiment.id for experiment in listed.results], [experiments[0].id])++ def test_model_breakdown_paginates_missing_and_named_models_without_duplicates(self) -> None:+ experiment = self._experiment()+ models = [None, "model-a", "model-b", "Not specified", "模" * 255, "😀" * 255, "😁" * 255]+ for index, model in enumerate(models):+ self._result(self._item(experiment, model_version=model, case_key="scored"), self.numeric, 1.0)+ for missing in range(index + 1):+ self._item(experiment, model_version=model, case_key=f"missing-{missing}")+ query = OfflineReadQuery(breakdown=OfflineHistoryBreakdown.MODEL, limit=1)+ values = []+ while True:+ page = self.service.scorer_history(self.numeric.definition_id, query)+ self.assertEqual(page.count, len(models))+ self.assertEqual(len(page.results), 1)+ point = page.results[0]+ values.append(point.breakdown_value)+ missing_count = models.index(point.breakdown_value) + 1+ self.assertEqual(point.summary.observed_item_count, missing_count + 1)+ self.assertEqual(point.summary.distinct_case_count, missing_count + 1)+ self.assertEqual(point.summary.missing_result_count, missing_count)+ if not page.next_cursor:+ break+ serializer = OfflineScorerHistoryQuerySerializer(+ data={"breakdown": "model_version", "limit": 1, "cursor": page.next_cursor}+ )+ self.assertTrue(serializer.is_valid(), serializer.errors)+ with self.assertRaises(OfflineEvaluationValidationError):+ self.service.scorer_history(+ self.numeric.definition_id,+ replace(query, cursor=page.next_cursor, breakdown=OfflineHistoryBreakdown.SUITE),+ )+ query = replace(query, cursor=page.next_cursor)+ self.assertCountEqual(values, models)++ @parameterized.expand(+ [+ (OfflineHistoryBreakdown.PROMPT, "prompt-a"),+ (OfflineHistoryBreakdown.APPLICATION, "build-a"),+ (OfflineHistoryBreakdown.SUITE, "suite-a"),+ (OfflineHistoryBreakdown.SOURCE, "ci"),+ ]+ )+ def test_experiment_breakdowns_keep_all_items_in_one_group(+ self, breakdown: OfflineHistoryBreakdown, value: str+ ) -> None:+ experiment = self._experiment()+ setattr(experiment, breakdown, value)+ experiment.save(update_fields=[breakdown])+ for model in ("model-a", "model-b"):+ self._result(self._item(experiment, model_version=model), self.numeric, 1.0)+ page = self.service.scorer_history(self.numeric.definition_id, OfflineReadQuery(breakdown=breakdown))+ self.assertEqual(page.count, 1)+ self.assertEqual(page.results[0].breakdown_value, value)+ self.assertEqual(page.results[0].summary.observed_item_count, 2)+def test_history_pages_experiment_version_groups_by_execution_time(self) -> None:older = self._experiment(started_at=self.now - timedelta(days=5), run_source="ci")newer = self._experiment(run_source="local")diff --git a/products/ai_observability/backend/test/test_offline_evaluation_service.py b/products/ai_observability/backend/test/test_offline_evaluation_service.pyindex 0c1770f4e495..36c9227b8f9f 100644--- a/products/ai_observability/backend/test/test_offline_evaluation_service.py+++ b/products/ai_observability/backend/test/test_offline_evaluation_service.py@@ -85,7 +85,10 @@ class TestOfflineEvaluationService(TestCase):self.assertEqual(error.exception.code, "content_conflict")self.assertEqual(OfflineExperiment.objects.for_team(self.team.id).count(), 1)- def test_upload_retry_preserves_identity_acceptance_and_retention(self) -> None:+ @parameterized.expand([(None,), ("model-a",)])+ def test_upload_retry_preserves_identity_acceptance_and_retention(self, model_version: str | None) -> None:+ self.item = replace(self.item, model_version=model_version)+ self.submission = replace(self.submission, items=[self.item])accepted = self.ingestion.upload(self.experiment.id, self.submission)replay = self.ingestion.upload(self.experiment.id, self.submission)@@ -95,6 +98,7 @@ class TestOfflineEvaluationService(TestCase):self.assertTrue(accepted.results[0].created)item = OfflineExperimentItem.objects.for_team(self.team.id).get(id=self.item.id)result = OfflineEvaluationResult.objects.for_team(self.team.id).get(id=accepted.results[0].id)+ self.assertEqual(item.model_version, model_version)self.assertEqual(item.payload_expires_at, item.accepted_at + timedelta(days=30))self.assertEqual(result.payload_expires_at, result.accepted_at + timedelta(days=30))self.assertEqual(diff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.tsindex 5a6edcfd7226..4897ec10e815 100644--- a/products/ai_observability/frontend/generated/api.schemas.ts+++ b/products/ai_observability/frontend/generated/api.schemas.ts@@ -734,6 +734,11 @@ export interface OfflineResultCellApi {}export interface OfflineItemReadApi {+ /**+ * Model used for the item, falling back to the experiment's model.+ * @nullable+ */+ model_version?: string | null/** Stable item UUID. */id: string/** Owning experiment UUID. */@@ -1208,6 +1213,12 @@ export interface OfflineSummaryPageApi {export interface ItemSubmissionApi {/** Caller-generated UUID for one input/output execution. Reuse for exact retries. */id: string+ /**+ * Model used for this item. Omitted or null values inherit the experiment's model.+ * @maxLength 255+ * @nullable+ */+ model_version?: string | null/*** Stable case identifier for matching inputs across experiments.* @maxLength 255@@ -1326,6 +1337,11 @@ export interface UploadReceiptApi {}export interface OfflineHistoryPointApi {+ /**+ * Breakdown value for this summary; null when unspecified or no breakdown is selected.+ * @nullable+ */+ breakdown_value?: string | null/** Experiment execution and cohort context. */experiment: OfflineExperimentReadApi/** Complete summary for one experiment and scorer version. */@@ -5316,6 +5332,17 @@ export type AiObservabilityOfflineScorersHistoryListParams = {* @maxLength 255*/application_version?: string+ /**+ * Split history by this property. Model uses each item's model, falling back to the experiment's model.+ *+ * * `model_version` - Model+ * * `prompt_version` - Prompt version+ * * `application_version` - Application version+ * * `suite_key` - Suite+ * * `run_source` - Run source+ * @minLength 1+ */+ breakdown?: AiObservabilityOfflineScorersHistoryListBreakdown/*** Filter experiments by the author's user ID.* @minimum 1@@ -5419,6 +5446,17 @@ export type AiObservabilityOfflineScorersHistoryListParams = {tags?: string}+export type AiObservabilityOfflineScorersHistoryListBreakdown =+ (typeof AiObservabilityOfflineScorersHistoryListBreakdown)[keyof typeof AiObservabilityOfflineScorersHistoryListBreakdown]++export const AiObservabilityOfflineScorersHistoryListBreakdown = {+ ModelVersion: 'model_version',+ PromptVersion: 'prompt_version',+ ApplicationVersion: 'application_version',+ SuiteKey: 'suite_key',+ RunSource: 'run_source',+} as const+export type AiObservabilityOfflineScorersHistoryListTagMatch =(typeof AiObservabilityOfflineScorersHistoryListTagMatch)[keyof typeof AiObservabilityOfflineScorersHistoryListTagMatch]diff --git a/products/ai_observability/frontend/generated/api.zod.ts b/products/ai_observability/frontend/generated/api.zod.tsindex 0e3a707fcd5c..fb765b094829 100644--- a/products/ai_observability/frontend/generated/api.zod.ts+++ b/products/ai_observability/frontend/generated/api.zod.ts@@ -141,6 +141,8 @@ export const AiObservabilityOfflineExperimentsPartialUpdateBody = /* @__PURE__ *.describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),})+export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax = 255+export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax = 255export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemTrialMax = 255@@ -168,6 +170,11 @@ export const AiObservabilityOfflineExperimentsUploadCreateBody = /* @__PURE__ */id: zod.uuid().describe('Caller-generated UUID for one input\/output execution. Reuse for exact retries.'),+ model_version: zod+ .string()+ .max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax)+ .nullish()+ .describe("Model used for this item. Omitted or null values inherit the experiment's model."),case_key: zod.string().max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax)diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsxindex daa307148145..7d534ca1154f 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx@@ -71,6 +71,7 @@ export function OfflineItemInspector({{item.id}</span>{item.trial && <span>{`Trial ${item.trial}`}</span>}+ {item.model_version && <span>{`Model: ${item.model_version}`}</span>}{item.application_trace_id && (<Link to={urls.aiObservabilityTrace(item.application_trace_id)}>Application tracediff --git a/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsxindex 0f35cf67c5e4..8269b2900ccf 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineOverviewTrend.test.tsx@@ -64,7 +64,7 @@ describe('OfflineOverviewTrend', () => {).toBeInTheDocument()expect(screen.getByText('Mean score')).toBeInTheDocument()if (partial) {- expect(screen.getByText(/30 of 200 experiment\/version results.*Partial history/)).toBeInTheDocument()+ expect(screen.getByText(/30 of 200 history points.*Partial history/)).toBeInTheDocument()expect(screen.getByText('Earlier experiments and scorer versions may not be shown.')).toBeInTheDocument()} else {expect(screen.queryByText(/Partial history/)).not.toBeInTheDocument()diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsxindex a7f9f170ebfd..43c158dd0081 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScoreTrendChart.tsx@@ -10,6 +10,11 @@ import { cn } from 'lib/utils/css-classes'import type { OfflineHistoryPointApi } from '../generated/api.schemas'import { ScorerLabel } from '../scoreDefinitions/ScorerLabel'import type { offlineExperimentsLogicType } from './offlineExperimentsLogic'+import {+ OFFLINE_HISTORY_BREAKDOWN_LABELS,+ offlineBreakdownValueLabel,+ type OfflineHistoryBreakdown,+} from './offlineHistoryBreakdown'import { createOfflineScoreTrendTickFormatter } from './offlineScoreTrendAxis'import { OfflineScoreTrendCrosshair } from './OfflineScoreTrendCrosshair'import { OfflineScoreTrendLines } from './OfflineScoreTrendLines'@@ -31,6 +36,7 @@ export interface OfflineScoreTrendChartProps {xDomain?: [number, number]showHeading?: booleancombineVersions?: boolean+ breakdown?: OfflineHistoryBreakdown}export function OfflineScoreTrendChart({@@ -43,6 +49,7 @@ export function OfflineScoreTrendChart({xDomain,showHeading = true,combineVersions = false,+ breakdown,}: OfflineScoreTrendChartProps): JSX.Element {const baseTheme = useChartTheme()const theme = useMemo(@@ -56,7 +63,7 @@ export function OfflineScoreTrendChart({)const panels = useMemo(() =>- buildOfflineTrendPanels(periods, combineVersions).map((panel) => {+ buildOfflineTrendPanels(periods, combineVersions, breakdown).map((panel) => {const domain = !panel.elapsed && xDomain ? xDomain : panel.xDomainreturn {...panel,@@ -64,7 +71,7 @@ export function OfflineScoreTrendChart({tickFormatter: createOfflineScoreTrendTickFormatter(domain, panel.elapsed, timezone),}}),- [periods, xDomain, timezone, combineVersions]+ [periods, xDomain, timezone, combineVersions, breakdown])if (panels.length === 0) {@@ -131,6 +138,9 @@ export function OfflineScoreTrendChart({: String(point.y)}</strong></div>+ {breakdown && (+ <div className="opacity-70">{`${OFFLINE_HISTORY_BREAKDOWN_LABELS[breakdown]}: ${offlineBreakdownValueLabel(meta.point.breakdown_value, breakdown)}`}</div>+ )}{periods.length > 1 && <div className="opacity-70">{meta.period}</div>}{offlineScorePassingRuleLabel(summary.scorer) && (<div className="opacity-70 mt-1">@@ -172,7 +182,7 @@ export function OfflineScoreTrendChart({['Upload state', experiment.status],['Suite', experiment.suite_key],['Application', experiment.application_version],- ['Model', experiment.model_version],+ ['Experiment model', experiment.model_version],['Prompt', experiment.prompt_version],['Dataset revision', experiment.dataset_revision_identifier],]diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsxindex 630b8a918b7d..d4fc2ff1d0c1 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx@@ -17,6 +17,7 @@ import { AccessControlLevel, AccessControlResourceType } from '~/types'import { ResourceTagFilter } from '../components/ResourceTagFilter'import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'import { offlineExperimentUrl } from './offlineExperimentPresentation'+import { OFFLINE_HISTORY_BREAKDOWN_OPTIONS, type OfflineHistoryBreakdown } from './offlineHistoryBreakdown'import { offlineScorerHistoryLogic, type OfflineScorerHistoryProps } from './offlineScorerHistoryLogic'import { OfflineScorerHistoryTable } from './OfflineScorerHistoryTable'import { OfflineScoreTrendChart } from './OfflineScoreTrendChart'@@ -112,6 +113,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div><LemonLabel>Upload state</LemonLabel><LemonSelect+ size="small"value={filters.statuses}onChange={(statuses) => setFilters({ statuses })}options={[@@ -125,6 +127,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div><LemonLabel>Source</LemonLabel><LemonSelect+ size="small"value={filters.run_source}onChange={(run_source) => setFilters({ run_source })}options={[@@ -138,13 +141,13 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem</div><div><LemonLabel>Tags</LemonLabel>- <ResourceTagFilter filters={filters} onChange={setFilters} />+ <ResourceTagFilter size="small" filters={filters} onChange={setFilters} /></div><div data-attr="offline-history-author-filter"><LemonLabel>Author</LemonLabel><MemberSelectvalue={filters.created_by ?? null}- size="medium"+ size="small"defaultLabel="Any author"extraOptions={[{@@ -159,7 +162,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem>{filters.created_by_is_null? () => (- <LemonButton type="secondary" size="medium">+ <LemonButton type="secondary" size="small">No author</LemonButton>)@@ -169,6 +172,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div><LemonLabel>Version</LemonLabel><LemonSelect+ size="small"value={filters.version}options={versionOptions}onChange={(version) => setFilters({ version })}@@ -178,6 +182,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem</div>{versions?.next_cursor && (<LemonButton+ size="small"type="secondary"loading={versionsLoading}onClick={() => loadOfflineHistoryVersions({ cursor: versions.next_cursor! })}@@ -188,6 +193,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div><LemonLabel>Date range</LemonLabel><DateFilter+ size="small"dateFrom={filters.date_from}dateTo={filters.date_to}onChange={(date_from, date_to) =>@@ -199,6 +205,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div><LemonLabel>Compare</LemonLabel><LemonSelect+ size="small"value={filters.compare}onChange={(compare) => setFilters({ compare })}options={[@@ -220,6 +227,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div><LemonLabel>Comparison range</LemonLabel><DateFilter+ size="small"dateFrom={filters.compare_from}dateTo={filters.compare_to}onChange={(compare_from, compare_to) =>@@ -230,6 +238,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem</div>)}<LemonButton+ size="small"type="secondary"loading={primaryPageLoading || comparisonPageLoading}onClick={refreshHistory}@@ -265,7 +274,22 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem) : (<><LemonCard hoverEffect={false} className="min-w-0">- <div className="font-semibold mb-2">Score history</div>+ <div className="flex flex-wrap items-center justify-between gap-2 mb-2">+ <div className="font-semibold">Score history</div>+ <div className="flex items-center gap-2">+ <LemonLabel>Break down by</LemonLabel>+ <LemonSelect<OfflineHistoryBreakdown | ''>+ size="small"+ value={filters.breakdown}+ onChange={(breakdown) => setFilters({ breakdown })}+ options={[+ { value: '', label: 'None' },+ ...OFFLINE_HISTORY_BREAKDOWN_OPTIONS,+ ]}+ data-attr="offline-history-breakdown"+ />+ </div>+ </div>{primaryPageLoading || comparisonPageLoading || (!primaryPage && !primaryError) ? (<Spinner />) : (@@ -278,6 +302,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem{trendPeriods.length > 0 && (<OfflineScoreTrendChartperiods={trendPeriods}+ breakdown={filters.breakdown || undefined}timezone={timezone}onPointClick={(point) =>router.actions.push(offlineExperimentUrl(point))@@ -287,8 +312,9 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem</>)}<p className="text-muted text-xs mt-2 mb-0">- Each point is one experiment and scorer version. Points use successful results- only. Different periods and versions remain separate.+ {filters.breakdown+ ? 'Each point is one experiment, scorer version, and breakdown value. Points use successful results only. Different periods and versions remain separate.'+ : 'Each point is one experiment and scorer version. Points use successful results only. Different periods and versions remain separate.'}</p></LemonCard>{[@@ -321,14 +347,27 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem<div className="font-semibold">{period.title}</div><div className="text-xs text-muted flex flex-wrap gap-1">{period.range?.dateFrom ? (- <TZLabel time={period.range.dateFrom} displayTimezone={timezone} />+ <TZLabel+ time={period.range.dateFrom}+ displayTimezone={timezone}+ formatDate="MMM D, YYYY"+ formatTime="HH:mm"+ timestampStyle="absolute"+ />) : (<span>All time</span>)}<span>to</span>{period.range?.dateTo && (- <TZLabel time={period.range.dateTo} displayTimezone={timezone} />+ <TZLabel+ time={period.range.dateTo}+ displayTimezone={timezone}+ formatDate="MMM D, YYYY"+ formatTime="HH:mm"+ timestampStyle="absolute"+ />)}+ <span>{`(${timezone})`}</span></div>{period.error ? (<LemonBanner@@ -345,6 +384,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem</div>)}<OfflineScorerHistoryTable+ breakdown={filters.breakdown || undefined}periodKey={period.key}page={period.page}loading={period.loading}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsxindex 9b0f40552c75..2fc98af979aa 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx@@ -73,3 +73,42 @@ export const EmptyHistory: Story = {}),],}++export const ModelBreakdown: Story = {+ parameters: {+ pageUrl: `${urls.aiObservabilityOfflineScorerHistory(OFFLINE_STORY_DEFINITION.id)}?breakdown=model_version`,+ },+ decorators: [+ mswDecorator({+ get: {+ '/api/projects/:project/ai_observability/offline_scorers/:definition/history/': ({ request }) => {+ const breakdown = new URL(request.url).searchParams.get('breakdown')+ const results =+ breakdown === 'model_version'+ ? OFFLINE_STORY_POINTS.flatMap((point) => [+ { ...point, breakdown_value: 'model-a' },+ {+ ...point,+ breakdown_value: 'model-b',+ summary: { ...point.summary, mean: (point.summary.mean ?? 0) / 2 },+ },+ ])+ : OFFLINE_STORY_POINTS.map((point) => ({+ ...point,+ breakdown_value:+ breakdown === 'suite_key'+ ? point.experiment.suite_key+ : breakdown === 'run_source'+ ? point.experiment.run_source+ : breakdown === 'prompt_version'+ ? point.experiment.prompt_version+ : breakdown === 'application_version'+ ? point.experiment.application_version+ : null,+ }))+ return { results, count: results.length, next_cursor: null }+ },+ },+ }),+ ],+}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsxindex d1d83974700d..9c271fec0508 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx@@ -6,17 +6,25 @@ import { TZLabel } from 'lib/components/TZLabel'import type { OfflineHistoryPageApi, OfflineHistoryPointApi } from '../generated/api.schemas'import { offlineExperimentUrl } from './offlineExperimentPresentation'+import {+ OFFLINE_HISTORY_BREAKDOWN_LABELS,+ offlineBreakdownValueLabel,+ offlineHistoryPointKey,+ type OfflineHistoryBreakdown,+} from './offlineHistoryBreakdown'import { OfflineScorerHistoryRunDetails } from './OfflineScorerHistoryRunDetails'import { OfflineScoreSummaryDisplay } from './OfflineScoreSummaryDisplay'export function OfflineScorerHistoryTable({periodKey,+ breakdown,page,loading,expandedRuns,onToggleRun,}: {periodKey: string+ breakdown?: OfflineHistoryBreakdownpage: OfflineHistoryPageApi | nullloading: booleanexpandedRuns: ReadonlySet<string>@@ -25,7 +33,7 @@ export function OfflineScorerHistoryTable({return (<LemonTable<OfflineHistoryPointApi>className="@max-[50rem]/scorer-history:[&_colgroup]:hidden @max-[50rem]/scorer-history:[&_table]:table-auto"- rowKey={(point) => `${point.experiment.id}:${point.summary.scorer.id}`}+ rowKey={offlineHistoryPointKey}dataSource={page?.results || []}loading={loading || !page}size="small"@@ -35,9 +43,10 @@ export function OfflineScorerHistoryTable({{title: 'Experiment',key: 'name',- width: '28%',+ className: '@max-[50rem]/scorer-history:w-2/5 @max-[50rem]/scorer-history:max-w-0',+ width: breakdown ? '24%' : '28%',render: (_, point) => {- const key = `${periodKey}:${point.experiment.id}:${point.summary.scorer.id}`+ const key = `${periodKey}:${offlineHistoryPointKey(point)}`return (<div className="flex items-center gap-1 min-w-0"><LemonButton@@ -60,10 +69,27 @@ export function OfflineScorerHistoryTable({)},},+ ...(breakdown+ ? [+ {+ title: OFFLINE_HISTORY_BREAKDOWN_LABELS[breakdown],+ key: 'breakdown',+ className: '@max-[50rem]/scorer-history:w-1/5 @max-[50rem]/scorer-history:max-w-0',+ width: '16%',+ render: (_: unknown, point: OfflineHistoryPointApi) => (+ <Tooltip title={offlineBreakdownValueLabel(point.breakdown_value, breakdown)}>+ <div className="truncate">+ {offlineBreakdownValueLabel(point.breakdown_value, breakdown)}+ </div>+ </Tooltip>+ ),+ },+ ]+ : []),{title: 'Tags',key: 'tags',- width: '24%',+ width: breakdown ? '18%' : '24%',render: (_, point) => (<div className="min-w-0 max-w-64"><ObjectTags tags={point.experiment.tags} staticOnly maxVisibleTags={2} wrap />@@ -74,7 +100,7 @@ export function OfflineScorerHistoryTable({title: 'Execution',key: 'execution',className: '@max-[50rem]/scorer-history:hidden',- width: '16%',+ width: breakdown ? '14%' : '16%',render: (_, point) => (<div className="truncate text-xs"><TZLabel time={point.experiment.started_at} />@@ -84,14 +110,14 @@ export function OfflineScorerHistoryTable({{title: 'Score',key: 'score',- width: '12%',+ width: breakdown ? '14%' : '12%',render: (_, point) => <OfflineScoreSummaryDisplay summary={point.summary} />,},{title: 'Coverage',key: 'coverage',className: '@max-[40rem]/scorer-history:hidden',- width: '20%',+ width: breakdown ? '14%' : '20%',render: (_, { summary }) => (<Tooltiptitle={`${summary.status_counts.ok} / ${summary.observed_item_count} scored items · ${summary.status_counts.error} errors · ${summary.status_counts.skipped} skipped · ${summary.status_counts.not_applicable} not applicable · ${summary.missing_result_count} missing`}@@ -110,8 +136,7 @@ export function OfflineScorerHistoryTable({expandable={{noIndent: true,showRowExpansionToggle: false,- isRowExpanded: (point) =>- expandedRuns.has(`${periodKey}:${point.experiment.id}:${point.summary.scorer.id}`),+ isRowExpanded: (point) => expandedRuns.has(`${periodKey}:${offlineHistoryPointKey(point)}`),expandedRowRender: (point) => <OfflineScorerHistoryRunDetails point={point} />,}}/>diff --git a/products/ai_observability/frontend/offline-evaluations/offlineHistoryBreakdown.ts b/products/ai_observability/frontend/offline-evaluations/offlineHistoryBreakdown.tsnew file mode 100644index 000000000000..e304866b4ac1--- /dev/null+++ b/products/ai_observability/frontend/offline-evaluations/offlineHistoryBreakdown.ts@@ -0,0 +1,34 @@+import {+ AiObservabilityOfflineScorersHistoryListBreakdown,+ type AiObservabilityOfflineScorersHistoryListParams,+ type OfflineHistoryPointApi,+} from '../generated/api.schemas'+import { offlineRunSourceLabel } from './offlineOverviewState'++export type OfflineHistoryBreakdown = NonNullable<AiObservabilityOfflineScorersHistoryListParams['breakdown']>++export const OFFLINE_HISTORY_BREAKDOWN_LABELS: Record<OfflineHistoryBreakdown, string> = {+ model_version: 'Model',+ prompt_version: 'Prompt version',+ application_version: 'Application version',+ suite_key: 'Suite',+ run_source: 'Run source',+}++export const OFFLINE_HISTORY_BREAKDOWN_OPTIONS = Object.values(AiObservabilityOfflineScorersHistoryListBreakdown).map(+ (value) => ({ value, label: OFFLINE_HISTORY_BREAKDOWN_LABELS[value] })+)++export function offlineBreakdownValueLabel(+ value: string | null | undefined,+ breakdown: OfflineHistoryBreakdown+): string {+ if (value == null) {+ return 'Not specified'+ }+ return breakdown === 'run_source' ? offlineRunSourceLabel(value) : value+}++export function offlineHistoryPointKey(point: OfflineHistoryPointApi): string {+ return JSON.stringify([point.experiment.id, point.summary.scorer.id, point.breakdown_value ?? null])+}diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.tsindex 2ca49f5f5507..6301502b5839 100644--- a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts+++ b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.test.ts@@ -10,6 +10,42 @@ import {import { makeOfflineHistoryPoint } from './offlineScoreTrends.fixtures'describe('offline score trends', () => {+ it('separates breakdown values, missing values, versions, and periods', () => {+ const point = makeOfflineHistoryPoint(1, 0.8)+ point.summary.scorer = { ...point.summary.scorer, version: 1 }+ const points = [+ { ...point, breakdown_value: 'model-a' },+ { ...point, breakdown_value: 'model-b' },+ { ...point, breakdown_value: null },+ { ...point, breakdown_value: 'Not specified' },+ {+ ...point,+ breakdown_value: 'model-a',+ summary: { ...point.summary, scorer: { ...point.summary.scorer, id: 'v2', version: 2 } },+ },+ ]+ const [panel] = buildOfflineTrendPanels(+ [+ { key: 'primary', label: 'Selected', points, dateFrom: '2026-01-01', dateTo: '2026-01-10' },+ {+ key: 'previous',+ label: 'Previous',+ points: [points[0]],+ dateFrom: '2025-12-23',+ dateTo: '2026-01-01',+ },+ ],+ false,+ 'model_version'+ )+ expect(panel.series).toHaveLength(6)+ expect(new Set(panel.series.map((series) => series.key)).size).toBe(6)+ expect(panel.series[0].label).toBe('model-a · Selected · v1')+ expect(panel.series[4].label).toBe('model-a · Selected · v2')+ expect(panel.series[5].label).toBe('model-a · Previous · v1')+ expect(panel.series.every((series) => series.points.length === 1)).toBe(true)+ })+it.each([['gte', 0.9999999, 1, '0.9999999'],['lte', 1.0000001, 1, '1.0000001'],diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.tsindex 40a9f2d92d29..cc1039054f27 100644--- a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts+++ b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.ts@@ -9,6 +9,7 @@ import type {OfflineScorerSummaryApi,OfflineScorerVersionReadApi,} from '../generated/api.schemas'+import { offlineBreakdownValueLabel, type OfflineHistoryBreakdown } from './offlineHistoryBreakdown'import {offlineBooleanPolarity,offlineCategoricalPassingRule,@@ -238,10 +239,14 @@ export function getOfflineHistoryCoverage(page: OfflineHistoryPageApi, timezone:.tz(timezone).format('MMM D, YYYY')}`: ''- return `${page.results.length} of ${page.count} experiment/version results${span}${page.count > page.results.length ? ' · Partial history' : ''}`+ return `${page.results.length} of ${page.count} history points${span}${page.count > page.results.length ? ' · Partial history' : ''}`}-export function buildOfflineTrendPanels(periods: OfflineTrendPeriod[], combineVersions = false): OfflineTrendPanel[] {+export function buildOfflineTrendPanels(+ periods: OfflineTrendPeriod[],+ combineVersions = false,+ breakdown?: OfflineHistoryBreakdown+): OfflineTrendPanel[] {const durations = periods.map((period) =>period.dateFrom && period.dateTo ? Date.parse(period.dateTo) - Date.parse(period.dateFrom) : null)@@ -298,12 +303,17 @@ export function buildOfflineTrendPanels(periods: OfflineTrendPeriod[], combineVe},]for (const metric of metrics) {- const key = `${period.key}:${scorer.id}:${metric.key}`+ const key = JSON.stringify([+ period.key,+ scorer.id,+ metric.key,+ breakdown ? (point.breakdown_value ?? null) : null,+ ])let series = panel.series.find((candidate) => candidate.key === key)if (!series) {series = {key,- label: `${periods.length > 1 ? `${period.label} · ` : ''}v${scorer.version}${scorer.kind === 'categorical' ? ` · ${metric.label}` : ''}`,+ label: `${breakdown ? `${offlineBreakdownValueLabel(point.breakdown_value, breakdown)} · ` : ''}${periods.length > 1 ? `${period.label} · ` : ''}v${scorer.version}${scorer.kind === 'categorical' ? ` · ${metric.label}` : ''}`,points: [],shape: periodIndex === 0 ? 'circle' : 'square',}diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.tsindex 7afe36046410..651cf213c358 100644--- a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts+++ b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts@@ -37,6 +37,29 @@ describe('offlineScorerHistoryLogic', () => {})})+ it('restores breakdowns from the URL and resets pagination when they change', async () => {+ const props = { scorerId: OFFLINE_STORY_DEFINITION.id, teamId: 1 }+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId), { breakdown: 'model_version' })+ const logic = offlineScorerHistoryLogic(props)+ logic.mount()+ await expectLogic(logic).toFinishAllListeners()+ expect(logic.values.query.breakdown).toBe('model_version')+ await expectLogic(logic, () => logic.actions.changePage('primary', 'older')).toFinishAllListeners()+ await expectLogic(logic, () =>+ logic.actions.setFilters({ breakdown: 'suite_key', compare: 'previous' })+ ).toFinishAllListeners()+ expect(logic.values.primaryCursors).toEqual([null])+ expect(router.values.searchParams.breakdown).toBe('suite_key')+ expect(api.aiObservabilityOfflineScorersHistoryList).toHaveBeenLastCalledWith(+ '1',+ props.scorerId,+ expect.objectContaining({ breakdown: 'suite_key', cursor: undefined })+ )+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId), { breakdown: 'unknown' })+ await expectLogic(logic).toFinishAllListeners()+ expect(logic.values.query.breakdown).toBeUndefined()+ })+it.each([{ created_by: 42 }, { created_by_is_null: true }])('preserves author and tag filters across periods and clears them through the URL (%j)',async (authorFilters) => {diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.tsindex 00d34bf374d3..570b633afa99 100644--- a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts+++ b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts@@ -37,6 +37,7 @@ import type {ScoreDefinitionVersionApi,ScoreDefinitionVersionPageApi,} from '../generated/api.schemas'+import { OFFLINE_HISTORY_BREAKDOWN_LABELS, type OfflineHistoryBreakdown } from './offlineHistoryBreakdown'import { OFFLINE_ALL_UPLOAD_STATES, offlineFiltersFromUrl } from './offlineOverviewState'import {previousOfflinePeriod,@@ -54,6 +55,7 @@ export interface OfflineHistoryFiltersResourceTagFilters,Pick<AiObservabilityOfflineScorersHistoryListParams, 'created_by' | 'created_by_is_null'> {version: string+ breakdown: OfflineHistoryBreakdown | ''date_from: stringdate_to: string | nullcompare: 'none' | 'previous' | 'custom'@@ -65,6 +67,7 @@ export interface OfflineHistoryFiltersconst DEFAULT_FILTERS: OfflineHistoryFilters = {version: '',+ breakdown: '',date_from: '-30d',date_to: null,compare: 'none',@@ -87,6 +90,9 @@ function readFilters(search: Record<string, unknown>): OfflineHistoryFilters {if (filters.compare_to === 'now') {filters.compare_to = null}+ if (!Object.hasOwn(OFFLINE_HISTORY_BREAKDOWN_LABELS, filters.breakdown)) {+ filters.breakdown = ''+ }if (!['none', 'previous', 'custom'].includes(filters.compare)) {filters.compare = 'none'}@@ -526,6 +532,7 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([(s) => [s.filters],(filters: OfflineHistoryFilters): AiObservabilityOfflineScorersHistoryListParams => ({scorer_version_ids: filters.version,+ breakdown: filters.breakdown || undefined,limit: 50,statuses: filters.statuses,run_source: filters.run_source || undefined,diff --git a/services/mcp/schema/tool-inputs.json b/services/mcp/schema/tool-inputs.jsonindex 070a353b16d6..ba766c685162 100644--- a/services/mcp/schema/tool-inputs.json+++ b/services/mcp/schema/tool-inputs.json@@ -810,6 +810,18 @@"type": "string","description": "Caller-generated UUID for one input/output execution. Reuse for exact retries."},+ "model_version": {+ "description": "Model used for this item. Omitted or null values inherit the experiment's model.",+ "anyOf": [+ {+ "type": "string",+ "maxLength": 255+ },+ {+ "type": "null"+ }+ ]+ },"case_key": {"description": "Stable case identifier for matching inputs across experiments.","anyOf": [diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex cde3bd4b47ef..166804156d5c 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -65202,6 +65202,12 @@ export namespace Schemas {export interface ItemSubmission {/** Caller-generated UUID for one input/output execution. Reuse for exact retries. */id: string;+ /**+ * Model used for this item. Omitted or null values inherit the experiment's model.+ * @maxLength 255+ * @nullable+ */+ model_version?: string | null;/*** Stable case identifier for matching inputs across experiments.* @maxLength 255@@ -74010,6 +74016,11 @@ export namespace Schemas {}export interface OfflineHistoryPoint {+ /**+ * Breakdown value for this summary; null when unspecified or no breakdown is selected.+ * @nullable+ */+ breakdown_value?: string | null;/** Experiment execution and cohort context. */experiment: OfflineExperimentRead;/** Complete summary for one experiment and scorer version. */@@ -74089,6 +74100,11 @@ export namespace Schemas {}export interface OfflineItemRead {+ /**+ * Model used for the item, falling back to the experiment's model.+ * @nullable+ */+ model_version?: string | null;/** Stable item UUID. */id: string;/** Owning experiment UUID. */@@ -127888,6 +127904,17 @@ export namespace Schemas {* @maxLength 255*/application_version?: string;+ /**+ * Split history by this property. Model uses each item's model, falling back to the experiment's model.+ *+ * * `model_version` - Model+ * * `prompt_version` - Prompt version+ * * `application_version` - Application version+ * * `suite_key` - Suite+ * * `run_source` - Run source+ * @minLength 1+ */+ breakdown?: AiObservabilityOfflineScorersHistoryListBreakdown;/*** Filter experiments by the author's user ID.* @minimum 1@@ -127991,6 +128018,17 @@ export namespace Schemas {tags?: string;};+ export type AiObservabilityOfflineScorersHistoryListBreakdown = typeof AiObservabilityOfflineScorersHistoryListBreakdown[keyof typeof AiObservabilityOfflineScorersHistoryListBreakdown];+++ export const AiObservabilityOfflineScorersHistoryListBreakdown = {+ ModelVersion: 'model_version',+ PromptVersion: 'prompt_version',+ ApplicationVersion: 'application_version',+ SuiteKey: 'suite_key',+ RunSource: 'run_source',+ } as const;+export type AiObservabilityOfflineScorersHistoryListTagMatch = typeof AiObservabilityOfflineScorersHistoryListTagMatch[keyof typeof AiObservabilityOfflineScorersHistoryListTagMatch];diff --git a/services/mcp/src/generated/ai_observability/api.ts b/services/mcp/src/generated/ai_observability/api.tsindex b789533a214d..c4cda045eab7 100644--- a/services/mcp/src/generated/ai_observability/api.ts+++ b/services/mcp/src/generated/ai_observability/api.ts@@ -532,6 +532,8 @@ export const AiObservabilityOfflineExperimentsUploadCreateParams = () => zod.obj),})+export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax = 255+export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax = 255export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemTrialMax = 255@@ -559,6 +561,11 @@ export const AiObservabilityOfflineExperimentsUploadCreateBody = () => zod.objecid: zod.string().describe('Caller-generated UUID for one input\/output execution. Reuse for exact retries.'),+ model_version: zod+ .string()+ .max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemModelVersionMax)+ .nullish()+ .describe("Model used for this item. Omitted or null values inherit the experiment's model."),case_key: zod.string().max(aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax)@@ -739,6 +746,12 @@ export const AiObservabilityOfflineScorersHistoryListQueryParams = () => zod.obj.max(aiObservabilityOfflineScorersHistoryListQueryApplicationVersionMax).optional().describe('Exact application revision.'),+ breakdown: zod+ .enum(['model_version', 'prompt_version', 'application_version', 'suite_key', 'run_source'])+ .optional()+ .describe(+ "Split history by this property. Model uses each item's model, falling back to the experiment's model.\n\n\* `model_version` - Model\n\* `prompt_version` - Prompt version\n\* `application_version` - Application version\n\* `suite_key` - Suite\n\* `run_source` - Run source"+ ),created_by: zod.number().min(1).optional().describe("Filter experiments by the author's user ID."),created_by_is_null: zod.boolean().optional().describe('Only include experiments without an author.'),cursor: zoddiff --git a/services/mcp/src/tools/generated/ai_observability.ts b/services/mcp/src/tools/generated/ai_observability.tsindex c31d051b7f5d..1d0ed7c8d70f 100644--- a/services/mcp/src/tools/generated/ai_observability.ts+++ b/services/mcp/src/tools/generated/ai_observability.ts@@ -2078,6 +2078,7 @@ const llmaOfflineScorerHistory = (): ToolBase<path: `/api/projects/${encodeURIComponent(String(projectId))}/ai_observability/offline_scorers/${encodeURIComponent(String(params.id))}/history/`,query: {application_version: params.application_version,+ breakdown: params.breakdown,created_by: params.created_by,created_by_is_null: params.created_by_is_null,cursor: params.cursor,diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.jsonindex 31c3964e71c7..1a95ce5e69be 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-upload.json@@ -72,6 +72,18 @@"description": "Caller-generated UUID for one input/output execution. Reuse for exact retries.","type": "string"},+ "model_version": {+ "anyOf": [+ {+ "maxLength": 255,+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Model used for this item. Omitted or null values inherit the experiment's model."+ },"payload": {"additionalProperties": false,"description": "Optional input/output payload, up to 1 MiB and 32 JSON levels. Omission, {} and null properties differ.",diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.jsonindex ac613c023cb0..6f6adf23ceca 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json@@ -7,6 +7,11 @@"minLength": 1,"type": "string"},+ "breakdown": {+ "description": "Split history by this property. Model uses each item's model, falling back to the experiment's model.\n\n* `model_version` - Model\n* `prompt_version` - Prompt version\n* `application_version` - Application version\n* `suite_key` - Suite\n* `run_source` - Run source",+ "enum": ["model_version", "prompt_version", "application_version", "suite_key", "run_source"],+ "type": "string"+ },"created_by": {"description": "Filter experiments by the author's user ID.","minimum": 1,