:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(aio): fill evaluation usage tracking gaps (#123640)
Radu Raicea3e3e92a23419
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 31521cacf264..1f1ce97fcc7f 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -401,6 +401,31 @@ Large payloads have separate storage and 30-day deadlines anchored to first acceRetries neither extend deadlines nor restore deleted payloads.Automatic payload deletion and usage billing are not enabled by these endpoints.+### Product usage events++The API reports these product usage events after the database transaction commits:++| Events | When they fire |+| -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `llma offline experiment created` | An experiment is first accepted. |+| `llma offline experiment results uploaded` | A batch accepts new items or results. `created_item_count` and `created_result_count` exclude retries. |+| `llma offline experiment completed`, `llma offline experiment failed` | An uploading experiment changes to the corresponding terminal state. Repeated closure emits no event. |+| `llma dataset created`, `llma dataset updated`, `llma dataset archived`, `llma dataset restored` | A dataset is created or changes. Updates include `changed_fields`; unchanged updates and repeated archive/restore requests emit no event. |+| `llma dataset item created`, `llma dataset item updated`, `llma dataset item archived`, `llma dataset item restored` | A new item version is accepted. Events include item and dataset IDs, version and dataset revision; restoration also includes the requested `source_version`. |+| `llma dataset export requested`, `llma dataset export downloaded` | An export starts successfully or its content is served. Events include the export ID and pinned dataset revision. Each successful download request is counted. |+| `llma scorer created`, `llma scorer updated`, `llma scorer version created` | A scorer is created, changed, or versioned. Archive and unarchive use `llma scorer updated` with `archived_new_value`. |++Events include the acting user or service principal, project groups, and request context.+Experiment lifecycle events include dataset source, run source, status, and expected and accepted counts.+Dataset source is classified as `posthog` for hosted revisions, `external` for a supplied external source, or `unknown` when omitted; the original source string is never captured.+Dataset and experiment usage events exclude input, output, reasoning, metadata values, and resource names.+Rejected requests, unchanged item updates, and exact ingestion retries emit no success event.+Dataset and experiment reporting failures are logged without failing successful API operations.++Frontend controls use `data-attr` for autocapture action matching.+Dataset row actions retain their existing resource-specific `data-attr` values and have stable `data-attr` values on their menu wrappers for aggregate matching.+Existing `llma` event names and attributes remain unchanged so saved actions and dashboards continue to match.+### Durable usage recordingNewly accepted Postgres results produce durable `OfflineEvaluationUsageLedger` entries.diff --git a/products/ai_observability/backend/api/analytics.py b/products/ai_observability/backend/api/analytics.pynew file mode 100644index 000000000000..56f00da0b8fd--- /dev/null+++ b/products/ai_observability/backend/api/analytics.py@@ -0,0 +1,10 @@+from collections.abc import Callable++from django.db import transaction+++def report_on_commit(callback: Callable[[], None]) -> None:+ def report() -> None:+ callback()++ transaction.on_commit(report, robust=True)diff --git a/products/ai_observability/backend/api/datasets.py b/products/ai_observability/backend/api/datasets.pyindex debe34f372b3..befbc29f9154 100644--- a/products/ai_observability/backend/api/datasets.py+++ b/products/ai_observability/backend/api/datasets.py@@ -1,3 +1,4 @@+from functools import partialfrom typing import NoReturn, castfrom uuid import UUID@@ -40,6 +41,7 @@ from products.access_control.backend.presentation.access_control import (AccessControlViewSetMixin,UserAccessControlSerializerMixin,)+from products.ai_observability.backend.api.analytics import report_on_commitfrom products.ai_observability.backend.api.dataset_exports import (DatasetExportCreateSerializer,DatasetExportErrorSerializer,@@ -62,6 +64,7 @@ from products.ai_observability.backend.dataset_queries import ()from products.ai_observability.backend.dataset_service import (UNSET,+ DatasetItemMutationResult,DatasetLimitExceeded,DatasetMutationConflict,DatasetValidationError,@@ -667,6 +670,23 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericVreturn DatasetRevisionReadSerializerreturn DatasetReadSerializer+ def _report_action(self, event: str, dataset: Dataset, properties: dict[str, object] | None = None) -> None:+ report_on_commit(+ partial(+ report_user_action,+ self.request.user,+ event,+ {+ "dataset_id": str(dataset.id),+ "has_description": bool(dataset.description),+ "has_metadata": bool(dataset.metadata),+ **(properties or {}),+ },+ team=self.team,+ request=self.request,+ )+ )+@extend_schema(parameters=[DatasetListQuerySerializer,@@ -762,17 +782,7 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericVexcept (DatasetMutationConflict, DatasetLimitExceeded) as error:return _conflict_response(error)- report_user_action(- request.user,- "llma dataset created",- {- "dataset_id": str(dataset.id),- "has_description": bool(dataset.description),- "has_metadata": bool(dataset.metadata),- },- team=self.team,- request=request,- )+ self._report_action("llma dataset created", dataset)return Response(DatasetReadSerializer(dataset, context=self.get_serializer_context()).data,status=status.HTTP_201_CREATED,@@ -796,15 +806,21 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericVdata = serializer.validated_datatry:with transaction.atomic():- dataset = update_dataset(+ result = update_dataset(team_id=self.team.id,dataset_id=current.id,name=data.get("name", UNSET),description=data.get("description", UNSET),metadata=data.get("metadata", UNSET),)+ dataset = result.dataset+ changed_fields = list(result.changed_fields)if "tags" in data:+ if set(data["tags"]) != set(ResourceTags.names(dataset)):+ changed_fields.append("tags")ResourceTags.set(dataset, data["tags"])+ if changed_fields:+ self._report_action("llma dataset updated", dataset, {"changed_fields": changed_fields})except DatasetValidationError as error:_validation_error_response(error)except DatasetMutationConflict as error:@@ -822,8 +838,10 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV@action(detail=True, methods=["post"])def archive(self, request: Request, *args: object, **kwargs: object) -> Response:current = self.get_object()- dataset = archive_dataset(team_id=self.team.id, dataset_id=current.id)- return Response(DatasetReadSerializer(dataset, context=self.get_serializer_context()).data)+ result = archive_dataset(team_id=self.team.id, dataset_id=current.id)+ if result.changed_fields:+ self._report_action("llma dataset archived", result.dataset)+ return Response(DatasetReadSerializer(result.dataset, context=self.get_serializer_context()).data)@extend_schema(operation_id="datasets_restore",@@ -835,8 +853,10 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV@action(detail=True, methods=["post"])def restore(self, request: Request, *args: object, **kwargs: object) -> Response:current = self.get_object()- dataset = restore_dataset(team_id=self.team.id, dataset_id=current.id)- return Response(DatasetReadSerializer(dataset, context=self.get_serializer_context()).data)+ result = restore_dataset(team_id=self.team.id, dataset_id=current.id)+ if result.changed_fields:+ self._report_action("llma dataset restored", result.dataset)+ return Response(DatasetReadSerializer(result.dataset, context=self.get_serializer_context()).data)@extend_schema(operation_id="datasets_revisions_list",@@ -888,6 +908,11 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV)except DatasetExportUnavailableError as error:return Response({"detail": str(error)}, status=status.HTTP_409_CONFLICT)+ self._report_action(+ "llma dataset export requested",+ dataset,+ {"export_id": asset.id, "dataset_revision": (asset.export_context or {}).get("dataset_revision")},+ )return Response(DatasetExportReadSerializer(asset).data, status=status.HTTP_201_CREATED)@extend_schema(@@ -949,7 +974,13 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV{"detail": "The export is still being prepared."},status=status.HTTP_409_CONFLICT,)- return get_export_asset_content_response(asset=asset, download=True)+ response = get_export_asset_content_response(asset=asset, download=True)+ self._report_action(+ "llma dataset export downloaded",+ dataset,+ {"export_id": asset.id, "dataset_revision": (asset.export_context or {}).get("dataset_revision")},+ )+ return responseclass DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):@@ -1028,6 +1059,31 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):self.check_object_permissions(self.request, dataset)return dataset+ def _report_mutation(+ self, event: str, result: DatasetItemMutationResult, properties: dict[str, object] | None = None+ ) -> None:+ if not result.created:+ return+ report_on_commit(+ partial(+ report_user_action,+ self.request.user,+ event,+ {+ "dataset_item_id": str(result.item.id),+ "dataset_id": str(result.item.dataset_id),+ "version": result.version.version,+ "dataset_revision": result.version.dataset_revision.revision,+ "has_expected_output": result.version.expected_output is not None,+ "has_source_output": result.version.source_output is not None,+ "has_provenance": result.version.source_trace_id is not None,+ **(properties or {}),+ },+ team=self.team,+ request=self.request,+ )+ )+@extend_schema(parameters=[DatasetItemListQuerySerializer],responses=DatasetItemReadSerializer(many=True),@@ -1157,20 +1213,7 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):except (DatasetMutationConflict, DatasetLimitExceeded) as error:return _conflict_response(error)- if result.created:- report_user_action(- request.user,- "llma dataset item created",- {- "dataset_item_id": str(result.item.id),- "dataset_id": str(dataset.id),- "has_expected_output": result.version.expected_output is not None,- "has_source_output": result.version.source_output is not None,- "has_provenance": result.version.source_trace_id is not None,- },- team=self.team,- request=request,- )+ self._report_mutation("llma dataset item created", result)response_status = status.HTTP_201_CREATED if result.created else status.HTTP_200_OKreturn Response(DatasetItemReadSerializer(result.version).data, status=response_status)@@ -1205,6 +1248,7 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):_validation_error_response(error)except (DatasetMutationConflict, DatasetLimitExceeded) as error:return _conflict_response(error)+ self._report_mutation("llma dataset item updated", result)return Response(DatasetItemReadSerializer(result.version).data)@extend_schema(@@ -1232,6 +1276,7 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):)except (DatasetMutationConflict, DatasetLimitExceeded) as error:return _conflict_response(error)+ self._report_mutation("llma dataset item archived", result)return Response(DatasetItemReadSerializer(result.version).data)@extend_schema(@@ -1262,6 +1307,9 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):raise Http404("Dataset item version not found.") from errorexcept (DatasetMutationConflict, DatasetLimitExceeded) as error:return _conflict_response(error)+ self._report_mutation(+ "llma dataset item restored", result, {"source_version": serializer.validated_data.get("source_version")}+ )return Response(DatasetItemReadSerializer(result.version).data)@extend_schema(diff --git a/products/ai_observability/backend/api/offline_experiments.py b/products/ai_observability/backend/api/offline_experiments.pyindex 045028d24f86..85b3fc241d0a 100644--- a/products/ai_observability/backend/api/offline_experiments.py+++ b/products/ai_observability/backend/api/offline_experiments.py@@ -1,4 +1,5 @@from dataclasses import asdict+from functools import partialfrom typing import TYPE_CHECKING, castfrom uuid import UUID@@ -15,8 +16,10 @@ from rest_framework.throttling import BaseThrottlefrom posthog.api.mixins import TypedRequest, ValidatedRequest, validated_requestfrom posthog.api.monitoring import monitor+from posthog.event_usage import report_user_actionfrom posthog.permissions import is_service_auth+from products.ai_observability.backend.api.analytics import report_on_commitfrom products.ai_observability.backend.api.metrics import llma_track_latencyfrom products.ai_observability.backend.api.offline_evaluation_base import OfflineEvaluationViewSetfrom products.ai_observability.backend.api.offline_experiment_access import (@@ -153,6 +156,31 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):def _reference_access_control(self) -> "UserAccessControl | None":return None if is_service_auth(self.request) else self.user_access_control+ def _report_experiment_action(self, event: str, receipt: ExperimentReceipt) -> None:+ experiment = receipt.experiment+ if experiment.dataset_revision_id is not None:+ dataset_source = "posthog"+ elif experiment.dataset_source:+ dataset_source = "external"+ else:+ dataset_source = "unknown"+ report_on_commit(+ partial(+ report_user_action,+ self.request.user,+ event,+ {+ "experiment_id": str(experiment.id),+ "dataset_source": dataset_source,+ "run_source": experiment.run_source,+ "status": experiment.status,+ **asdict(receipt.counts),+ },+ team=self.team,+ request=self.request,+ )+ )+def handle_exception(self, exc: Exception) -> Response:action_name = getattr(self, "action", None)if action_name in self.scope_object_read_actions or action_name in {"metadata", "partial_update"}:@@ -203,6 +231,8 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):request.validated_data,created_by_id=None if is_service_auth(request) else request.user.pk,)+ if receipt.created:+ self._report_experiment_action("llma offline experiment created", receipt)return _experiment_response(receipt, status=201 if receipt.created else 200)def _experiment_id(self) -> UUID:@@ -217,13 +247,31 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):@llma_track_latency("aio_offline_experiment_upload")@monitor(feature=None, endpoint="aio_offline_experiment_upload", method="POST")def upload(self, request: TypedRequest[UploadSubmission], **kwargs: object) -> Response:+ experiment_id = self._experiment_id()receipt = OfflineEvaluationIngestionService(team_id=self.team_id, user_access_control=self._reference_access_control()- ).upload(self._experiment_id(), request.validated_data)+ ).upload(experiment_id, request.validated_data)for created in (True, False):OFFLINE_UPLOAD_RESULTS.labels(outcome="created" if created else "duplicate").inc(sum(result.created == created for result in receipt.results))+ created_item_count = sum(item.created for item in receipt.items)+ created_result_count = sum(result.created for result in receipt.results)+ if created_item_count or created_result_count:+ report_on_commit(+ partial(+ report_user_action,+ request.user,+ "llma offline experiment results uploaded",+ {+ "experiment_id": str(experiment_id),+ "created_item_count": created_item_count,+ "created_result_count": created_result_count,+ },+ team=self.team,+ request=request,+ )+ )return Response(UploadReceiptSerializer(receipt).data)@validated_request(@@ -235,11 +283,12 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):@llma_track_latency("aio_offline_experiment_complete")@monitor(feature=None, endpoint="aio_offline_experiment_complete", method="POST")def complete(self, request: Request, **kwargs: object) -> Response:- return _experiment_response(- OfflineExperimentService(team_id=self.team_id, user_access_control=self._reference_access_control()).close(- self._experiment_id(), status="completed"- )- )+ receipt = OfflineExperimentService(+ team_id=self.team_id, user_access_control=self._reference_access_control()+ ).close(self._experiment_id(), status="completed")+ if receipt.status_changed:+ self._report_experiment_action("llma offline experiment completed", receipt)+ return _experiment_response(receipt)@validated_request(EmptyOfflineExperimentSerializer,@@ -250,8 +299,9 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):@llma_track_latency("aio_offline_experiment_fail")@monitor(feature=None, endpoint="aio_offline_experiment_fail", method="POST")def fail(self, request: Request, **kwargs: object) -> Response:- return _experiment_response(- OfflineExperimentService(team_id=self.team_id, user_access_control=self._reference_access_control()).close(- self._experiment_id(), status="failed"- )- )+ receipt = OfflineExperimentService(+ team_id=self.team_id, user_access_control=self._reference_access_control()+ ).close(self._experiment_id(), status="failed")+ if receipt.status_changed:+ self._report_experiment_action("llma offline experiment failed", receipt)+ return _experiment_response(receipt)diff --git a/products/ai_observability/backend/api/test/test_datasets.py b/products/ai_observability/backend/api/test/test_datasets.pyindex 7cc8f90cc928..29df1fa75cf9 100644--- a/products/ai_observability/backend/api/test/test_datasets.py+++ b/products/ai_observability/backend/api/test/test_datasets.py@@ -3,7 +3,7 @@ from datetime import timedeltafrom functools import partialfrom posthog.test.base import APIBaseTest-from unittest.mock import DEFAULT, AsyncMock, MagicMock, patch+from unittest.mock import ANY, DEFAULT, AsyncMock, MagicMock, Mock, patchfrom django.core.cache import cachefrom django.db import IntegrityError, connection, transaction@@ -68,8 +68,10 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)- def test_dataset_lifecycle_and_name_uniqueness(self) -> None:- dataset = self._create_dataset()+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_dataset_lifecycle_and_name_uniqueness(self, report_action: Mock) -> None:+ with self.captureOnCommitCallbacks(execute=True):+ dataset = self._create_dataset()duplicate_response = self.client.post(self.datasets_url,@@ -79,8 +81,11 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(duplicate_response.status_code, status.HTTP_409_CONFLICT)self.assertEqual(duplicate_response.data["code"], "dataset_name_conflict")- archive_response = self.client.post(f"{self.datasets_url}{dataset['id']}/archive/")+ with self.captureOnCommitCallbacks(execute=True):+ archive_response = self.client.post(f"{self.datasets_url}{dataset['id']}/archive/")+ archive_retry = self.client.post(f"{self.datasets_url}{dataset['id']}/archive/")self.assertEqual(archive_response.status_code, status.HTTP_200_OK)+ self.assertEqual(archive_retry.status_code, status.HTTP_200_OK)self.assertTrue(archive_response.data["archived"])active_list_response = self.client.get(self.datasets_url)@@ -101,9 +106,93 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(blocked_item_response.status_code, status.HTTP_409_CONFLICT)self.assertEqual(blocked_item_response.data["code"], "dataset_archived")- restore_response = self.client.post(f"{self.datasets_url}{dataset['id']}/restore/")+ with self.captureOnCommitCallbacks(execute=True):+ restore_response = self.client.post(f"{self.datasets_url}{dataset['id']}/restore/")+ restore_retry = self.client.post(f"{self.datasets_url}{dataset['id']}/restore/")self.assertEqual(restore_response.status_code, status.HTTP_200_OK)+ self.assertEqual(restore_retry.status_code, status.HTTP_200_OK)self.assertFalse(restore_response.data["archived"])+ self.assertEqual(+ [call.args[1] for call in report_action.call_args_list],+ ["llma dataset created", "llma dataset archived", "llma dataset restored"],+ )+ report_action.assert_called_with(+ self.user,+ "llma dataset restored",+ {"dataset_id": dataset["id"], "has_description": True, "has_metadata": True},+ team=self.team,+ request=ANY,+ )++ @parameterized.expand(+ [+ ("name", {"name": "Updated name"}, ["name"]),+ (+ "metadata",+ {"description": "Updated description", "metadata": {"reviewed": True}},+ ["description", "metadata"],+ ),+ ("tags", {"tags": ["evaluation"]}, ["tags"]),+ ]+ )+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_dataset_update_reports_only_committed_changes(+ self, _name: str, payload: dict[str, object], changed_fields: list[str], report_action: Mock+ ) -> None:+ dataset = self._create_dataset()+ url = f"{self.datasets_url}{dataset['id']}/"+ with self.captureOnCommitCallbacks(execute=True):+ with self.assertRaises(RuntimeError), transaction.atomic():+ response = self.client.patch(url, payload, format="json")+ self.assertEqual(response.status_code, status.HTTP_200_OK)+ raise RuntimeError("Roll back the mutation")+ report_action.assert_not_called()++ with self.captureOnCommitCallbacks(execute=True):+ response = self.client.patch(url, payload, format="json")+ unchanged = self.client.patch(url, payload, format="json")+ self.assertEqual(response.status_code, status.HTTP_200_OK)+ self.assertEqual(unchanged.status_code, status.HTTP_200_OK)+ report_action.assert_called_once_with(+ self.user,+ "llma dataset updated",+ {+ "dataset_id": dataset["id"],+ "has_description": True,+ "has_metadata": True,+ "changed_fields": changed_fields,+ },+ team=self.team,+ request=ANY,+ )++ @parameterized.expand([("malformed_referer",), ("capture_failure",)])+ @patch("posthog.event_usage.posthoganalytics.capture")+ def test_analytics_failure_does_not_interrupt_mutations(self, failure: str, capture: Mock) -> None:+ if failure == "malformed_referer":+ self.client.credentials(HTTP_REFERER="http://[invalid")+ else:+ capture.side_effect = RuntimeError("Analytics unavailable")++ with self.assertLogs("django.test", level="ERROR"), self.captureOnCommitCallbacks(execute=True):+ dataset = self._create_dataset()+ item = self._create_item(dataset["id"])+ updated = self.client.patch(+ f"{self.items_url}{item['id']}/",+ {"base_version": 1, "input": "Updated input"},+ format="json",+ )+ self.assertEqual(updated.status_code, status.HTTP_200_OK)+ self.assertEqual(updated.data["version"], 2)++ with self.assertLogs("django.test", level="ERROR"), self.captureOnCommitCallbacks(execute=True):+ next_update = self.client.patch(+ f"{self.items_url}{item['id']}/",+ {"base_version": 2, "input": "Next input"},+ format="json",+ )+ self.assertEqual(next_update.status_code, status.HTTP_200_OK)+ self.assertEqual(next_update.data["version"], 3)def test_dataset_list_filters_by_comma_separated_ids(self) -> None:first_dataset = self._create_dataset("First dataset")@@ -142,7 +231,8 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(item["input"], input_value)- def test_item_create_handles_existing_client_item_id(self) -> None:+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_item_create_handles_existing_client_item_id(self, report_action: Mock) -> None:dataset = self._create_dataset()payload = {"dataset": dataset["id"],@@ -154,13 +244,29 @@ class TestDatasetsApi(APIBaseTest):"source_timestamp": "2026-07-30T12:00:00Z",}- first_response = self.client.post(self.items_url, payload, format="json")- retry_response = self.client.post(self.items_url, payload, format="json")+ with self.captureOnCommitCallbacks(execute=True):+ first_response = self.client.post(self.items_url, payload, format="json")+ retry_response = self.client.post(self.items_url, payload, format="json")self.assertEqual(first_response.status_code, status.HTTP_201_CREATED)self.assertEqual(retry_response.status_code, status.HTTP_200_OK)self.assertEqual(retry_response.data["id"], first_response.data["id"])self.assertEqual(retry_response.data["client_item_id"], payload["client_item_id"])+ report_action.assert_called_once_with(+ self.user,+ "llma dataset item created",+ {+ "dataset_item_id": first_response.data["id"],+ "dataset_id": dataset["id"],+ "version": 1,+ "dataset_revision": 1,+ "has_expected_output": False,+ "has_source_output": True,+ "has_provenance": True,+ },+ team=self.team,+ request=ANY,+ )self.assertEqual(DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset["id"]).count(),1,@@ -261,53 +367,75 @@ class TestDatasetsApi(APIBaseTest):)self.assertEqual(immutable_source_response.status_code, status.HTTP_400_BAD_REQUEST)- def test_stale_update_does_not_create_a_revision(self) -> None:+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_stale_update_does_not_create_a_revision(self, report_action: Mock) -> None:dataset = self._create_dataset()item = self._create_item(dataset["id"])- first_update = self.client.patch(- f"{self.items_url}{item['id']}/",- {"base_version": 1, "input": "new"},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ first_update = self.client.patch(+ f"{self.items_url}{item['id']}/",+ {"base_version": 1, "input": "new"},+ format="json",+ )self.assertEqual(first_update.status_code, status.HTTP_200_OK)- stale_update = self.client.patch(- f"{self.items_url}{item['id']}/",- {"base_version": 1, "input": "stale"},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ stale_update = self.client.patch(+ f"{self.items_url}{item['id']}/",+ {"base_version": 1, "input": "stale"},+ format="json",+ )self.assertEqual(stale_update.status_code, status.HTTP_409_CONFLICT)self.assertEqual(stale_update.data["code"], "stale_version")self.assertEqual(stale_update.data["current_version"], 2)+ report_action.assert_called_once_with(+ self.user,+ "llma dataset item updated",+ {+ "dataset_item_id": item["id"],+ "dataset_id": dataset["id"],+ "version": 2,+ "dataset_revision": 2,+ "has_expected_output": True,+ "has_source_output": False,+ "has_provenance": False,+ },+ team=self.team,+ request=ANY,+ )self.assertEqual(DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset["id"]).count(),2,)- def test_unchanged_update_does_not_create_a_revision(self) -> None:+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_unchanged_update_does_not_create_a_revision(self, report_action: Mock) -> None:dataset = self._create_dataset()item = self._create_item(dataset["id"])- response = self.client.patch(- f"{self.items_url}{item['id']}/",- {- "base_version": 1,- "input": item["input"],- "expected_output": item["expected_output"],- "metadata": item["metadata"],- },- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ response = self.client.patch(+ f"{self.items_url}{item['id']}/",+ {+ "base_version": 1,+ "input": item["input"],+ "expected_output": item["expected_output"],+ "metadata": item["metadata"],+ },+ format="json",+ )self.assertEqual(response.status_code, status.HTTP_200_OK)self.assertEqual(response.data["version"], 1)+ report_action.assert_not_called()self.assertEqual(DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset["id"]).count(),1,)- def test_archive_and_restore_create_versions_with_explicit_transitions(self) -> None:+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_archive_and_restore_create_versions_with_explicit_transitions(self, report_action: Mock) -> None:dataset = self._create_dataset()item = self._create_item(dataset["id"], input="original")update_response = self.client.patch(@@ -317,11 +445,12 @@ class TestDatasetsApi(APIBaseTest):)self.assertEqual(update_response.status_code, status.HTTP_200_OK)- archive_response = self.client.post(- f"{self.items_url}{item['id']}/archive/",- {"base_version": 2},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ archive_response = self.client.post(+ f"{self.items_url}{item['id']}/archive/",+ {"base_version": 2},+ format="json",+ )self.assertEqual(archive_response.status_code, status.HTTP_200_OK)self.assertTrue(archive_response.data["archived"])self.assertEqual(archive_response.data["version"], 3)@@ -334,31 +463,54 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(active_list.data["results"], [])self.assertEqual([result["id"] for result in archived_list.data["results"]], [item["id"]])- duplicate_archive = self.client.post(- f"{self.items_url}{item['id']}/archive/",- {"base_version": 3},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ duplicate_archive = self.client.post(+ f"{self.items_url}{item['id']}/archive/",+ {"base_version": 3},+ format="json",+ )self.assertEqual(duplicate_archive.status_code, status.HTTP_409_CONFLICT)self.assertEqual(duplicate_archive.data["code"], "dataset_item_archived")- restore_response = self.client.post(- f"{self.items_url}{item['id']}/restore/",- {"base_version": 3, "source_version": 1},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ restore_response = self.client.post(+ f"{self.items_url}{item['id']}/restore/",+ {"base_version": 3, "source_version": 1},+ format="json",+ )self.assertEqual(restore_response.status_code, status.HTTP_200_OK)self.assertFalse(restore_response.data["archived"])self.assertEqual(restore_response.data["input"], "original")self.assertEqual(restore_response.data["version"], 4)- duplicate_restore = self.client.post(- f"{self.items_url}{item['id']}/restore/",- {"base_version": 4},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ duplicate_restore = self.client.post(+ f"{self.items_url}{item['id']}/restore/",+ {"base_version": 4},+ format="json",+ )self.assertEqual(duplicate_restore.status_code, status.HTTP_409_CONFLICT)self.assertEqual(duplicate_restore.data["code"], "dataset_item_active")+ self.assertEqual(+ [call.args[1] for call in report_action.call_args_list],+ ["llma dataset item archived", "llma dataset item restored"],+ )+ report_action.assert_called_with(+ self.user,+ "llma dataset item restored",+ {+ "dataset_item_id": item["id"],+ "dataset_id": dataset["id"],+ "version": 4,+ "dataset_revision": 4,+ "has_expected_output": True,+ "has_source_output": False,+ "has_provenance": False,+ "source_version": 1,+ },+ team=self.team,+ request=ANY,+ )def test_revision_list_reconstructs_exact_snapshots(self) -> None:dataset = self._create_dataset()@@ -1013,15 +1165,19 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(len(response.data["results"]), 2)@patch("products.exports.backend.facade.api.async_connect", new_callable=AsyncMock)- def test_dataset_export_pins_revision_and_scopes_status_and_content(self, async_connect: AsyncMock) -> None:+ @patch("products.ai_observability.backend.api.datasets.report_user_action")+ def test_dataset_export_pins_revision_and_scopes_status_and_content(+ self, report_action: Mock, async_connect: AsyncMock+ ) -> None:dataset = self._create_dataset()first_item = self._create_item(dataset["id"], input={"question": "First"})- create_response = self.client.post(- f"{self.datasets_url}{dataset['id']}/exports/",- {},- format="json",- )+ with self.captureOnCommitCallbacks(execute=True):+ create_response = self.client.post(+ f"{self.datasets_url}{dataset['id']}/exports/",+ {},+ format="json",+ )self.assertEqual(create_response.status_code, status.HTTP_201_CREATED)self.assertEqual(create_response.data["status"], "pending")@@ -1043,7 +1199,10 @@ class TestDatasetsApi(APIBaseTest):mismatched_parent_response = self.client.get(f"{self.datasets_url}{other_dataset['id']}/exports/{export_id}/")self.assertEqual(mismatched_parent_response.status_code, status.HTTP_404_NOT_FOUND)- pending_content_response = self.client.get(f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/")+ with self.captureOnCommitCallbacks(execute=True):+ pending_content_response = self.client.get(+ f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/"+ )self.assertEqual(pending_content_response.status_code, status.HTTP_409_CONFLICT)asset = ExportedAsset.objects.get(id=export_id)@@ -1051,11 +1210,29 @@ class TestDatasetsApi(APIBaseTest):self.assertEqual(asset.export_context["kind"], DATASET_EXPORT_KIND)asset.content = b'{"input":{"question":"First"}}\n'asset.save(update_fields=["content"])- content_response = self.client.get(f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/")+ with self.captureOnCommitCallbacks(execute=True):+ content_response = self.client.get(f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/")self.assertEqual(content_response.status_code, status.HTTP_200_OK)self.assertEqual(content_response.content, asset.content)self.assertIn("attachment", content_response["Content-Disposition"])+ self.assertEqual(+ [call.args[1] for call in report_action.call_args_list],+ ["llma dataset export requested", "llma dataset export downloaded"],+ )+ report_action.assert_called_with(+ self.user,+ "llma dataset export downloaded",+ {+ "dataset_id": dataset["id"],+ "has_description": True,+ "has_metadata": True,+ "export_id": export_id,+ "dataset_revision": 1,+ },+ team=self.team,+ request=ANY,+ )@patch("posthog.rate_limit.is_rate_limit_enabled", return_value=True)@patch("products.ai_observability.backend.api.datasets.DatasetExportRateThrottle.rate", new="1/day")diff --git a/products/ai_observability/backend/api/test/test_offline_experiments.py b/products/ai_observability/backend/api/test/test_offline_experiments.pyindex 1bd770ba0a7c..c4bafdf877b6 100644--- a/products/ai_observability/backend/api/test/test_offline_experiments.py+++ b/products/ai_observability/backend/api/test/test_offline_experiments.py@@ -3,7 +3,7 @@ from uuid import uuid4import pytestfrom posthog.test.base import APIBaseTest-from unittest.mock import patch+from unittest.mock import Mock, patchfrom django.db import OperationalError, connectionfrom django.test import SimpleTestCase@@ -118,13 +118,27 @@ class TestOfflineExperimentsAPI(APIBaseTest):input="What is 2 + 2?",)- @parameterized.expand([("session",), ("personal_key",), ("project_key",)])- def test_ingestion_and_both_lifecycle_actions(self, auth_kind: str) -> None:+ @parameterized.expand([("session", False), ("personal_key", False), ("project_key", False), ("session", True)])+ @patch("products.ai_observability.backend.api.offline_experiments.report_user_action")+ def test_ingestion_and_both_lifecycle_actions(+ self, auth_kind: str, analytics_failure: bool, report_action: Mock+ ) -> None:+ if analytics_failure:+ report_action.side_effect = RuntimeError("Analytics unavailable")definition = ScoreDefinition.objects.create(team=self.team, name="Correct", kind="boolean")version = definition.create_new_version(config={"true_label": "Yes", "false_label": "No"}, created_by=self.user)+ other_definition = ScoreDefinition.objects.create(team=self.team, name="Relevant", kind="boolean")+ other_version = other_definition.create_new_version(+ config={"true_label": "Yes", "false_label": "No"}, created_by=self.user+ )self._authenticate(auth_kind)- body = self._experiment_body() | {"expected_item_count": 1, "expected_result_count": 1}- created = self.client.post(self._endpoint(), body, format="json")+ body = self._experiment_body() | {+ "expected_item_count": 2,+ "expected_result_count": 4,+ "dataset_source": "https://private.example.com/datasets/customer-evals",+ }+ with self.captureOnCommitCallbacks(execute=True):+ created = self.client.post(self._endpoint(), body, format="json")self.assertEqual(created.status_code, status.HTTP_201_CREATED, created.data)experiment_id = str(body["id"])self.assertEqual(created.data["id"], experiment_id)@@ -132,49 +146,89 @@ class TestOfflineExperimentsAPI(APIBaseTest):self.assertEqual(OfflineExperiment.objects.for_team(self.team.id).get(pk=experiment_id).created_by_id, expected_author)- incomplete = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")+ with self.captureOnCommitCallbacks(execute=True):+ incomplete = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")self.assertEqual(incomplete.status_code, status.HTTP_409_CONFLICT, incomplete.data)self.assertEqual(incomplete.data["code"], "expected_count_mismatch")- self.assertEqual(incomplete.data["expected_item_count"], 1)- self.assertEqual(incomplete.data["expected_result_count"], 1)+ self.assertEqual(incomplete.data["expected_item_count"], 2)+ self.assertEqual(incomplete.data["expected_result_count"], 4)self.assertEqual(incomplete.data["accepted_item_count"], 0)self.assertEqual(incomplete.data["accepted_result_count"], 0)- item_id = str(uuid4())- uploaded = self.client.post(- self._endpoint(experiment_id, "upload"),- {- "items": [{"id": item_id, "payload": {"input": "What is 2 + 2?", "output": "5"}}],- "results": [{"item_id": item_id, "scorer_version_id": str(version.id), "status": "ok", "value": False}],- },- format="json",- )+ item_ids = [str(uuid4()), str(uuid4())]+ upload_body = {+ "items": [{"id": item_id, "payload": {"input": "What is 2 + 2?", "output": "5"}} for item_id in item_ids],+ "results": [+ {"item_id": item_id, "scorer_version_id": str(scorer_version.id), "status": "ok", "value": False}+ for item_id in item_ids+ for scorer_version in (version, other_version)+ ],+ }+ with self.captureOnCommitCallbacks(execute=True):+ uploaded = self.client.post(self._endpoint(experiment_id, "upload"), upload_body, format="json")self.assertEqual(uploaded.status_code, status.HTTP_200_OK, uploaded.data)result = OfflineEvaluationResult.objects.for_team(self.team.id).get(pk=uploaded.data["results"][0]["id"])self.assertIs(result.boolean_value, False)self.assertEqual(result.scorer_version_id, version.id)- completed = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")+ with self.captureOnCommitCallbacks(execute=True):+ completed = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")self.assertEqual(completed.status_code, status.HTTP_200_OK, completed.data)self.assertEqual(completed.data["status"], "completed")- self.assertEqual(completed.data["accepted_item_count"], 1)- self.assertEqual(completed.data["accepted_result_count"], 1)+ self.assertEqual(completed.data["accepted_item_count"], 2)+ self.assertEqual(completed.data["accepted_result_count"], 4)failed_body = self._experiment_body() | {"expected_result_count": 10}- self.assertEqual(- self.client.post(self._endpoint(), failed_body, format="json").status_code, status.HTTP_201_CREATED- )- failed = self.client.post(self._endpoint(str(failed_body["id"]), "fail"), {}, format="json")+ with self.captureOnCommitCallbacks(execute=True):+ self.assertEqual(+ self.client.post(self._endpoint(), failed_body, format="json").status_code, status.HTTP_201_CREATED+ )+ failed = self.client.post(self._endpoint(str(failed_body["id"]), "fail"), {}, format="json")self.assertEqual(failed.status_code, status.HTTP_200_OK, failed.data)self.assertEqual(failed.data["status"], "failed")self.assertEqual(failed.data["accepted_result_count"], 0)self._authenticate("project_key" if auth_kind != "project_key" else "session")- retried = self.client.post(self._endpoint(), body, format="json")+ with self.captureOnCommitCallbacks(execute=True):+ retried = self.client.post(self._endpoint(), body, format="json")+ upload_retry = self.client.post(self._endpoint(experiment_id, "upload"), upload_body, format="json")+ complete_retry = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")+ fail_retry = self.client.post(self._endpoint(str(failed_body["id"]), "fail"), {}, format="json")self.assertEqual(retried.status_code, status.HTTP_200_OK)+ self.assertEqual(upload_retry.status_code, status.HTTP_200_OK)+ self.assertEqual(complete_retry.status_code, status.HTTP_200_OK)+ self.assertEqual(fail_retry.status_code, status.HTTP_200_OK)self.assertEqual(OfflineExperiment.objects.for_team(self.team.id).get(pk=experiment_id).created_by_id, expected_author)+ self.assertEqual(+ [call.args[1] for call in report_action.call_args_list],+ [+ "llma offline experiment created",+ "llma offline experiment results uploaded",+ "llma offline experiment completed",+ "llma offline experiment created",+ "llma offline experiment failed",+ ],+ )+ upload_event = report_action.call_args_list[1]+ self.assertEqual(+ upload_event.args[2],+ {"experiment_id": experiment_id, "created_item_count": 2, "created_result_count": 4},+ )+ completed_event = report_action.call_args_list[2]+ self.assertEqual(report_action.call_args_list[0].args[2]["dataset_source"], "external")+ self.assertEqual(completed_event.args[2]["dataset_source"], "external")+ self.assertEqual(report_action.call_args_list[3].args[2]["dataset_source"], "unknown")+ self.assertEqual(report_action.call_args_list[4].args[2]["dataset_source"], "unknown")+ for call in report_action.call_args_list:+ self.assertNotIn(body["dataset_source"], call.args[2].values())+ self.assertEqual(completed_event.args[2]["status"], "completed")+ self.assertEqual(completed_event.args[2]["accepted_item_count"], 2)+ self.assertEqual(completed_event.args[2]["accepted_result_count"], 4)+ self.assertEqual(completed_event.kwargs["team"], self.team)+ self.assertEqual(completed_event.args[0], completed_event.kwargs["request"].user)+ self.assertTrue(completed_event.args[0].distinct_id)def test_ledger_failure_returns_server_error_and_mixed_upload_can_retry(self) -> None:definition = ScoreDefinition.objects.create(team=self.team, name="Correct", kind="boolean")@@ -304,8 +358,15 @@ class TestOfflineExperimentsAPI(APIBaseTest):("project_key", "project_key", "none", "none", True),])+ @patch("products.ai_observability.backend.api.offline_experiments.report_user_action")def test_references_require_viewer_access(- self, _name: str, auth_kind: str, resource_access: str, object_access: str | None, allowed: bool+ self,+ _name: str,+ auth_kind: str,+ resource_access: str,+ object_access: str | None,+ allowed: bool,+ report_action: Mock,) -> None:membership = self._evaluation_member()dataset_version = self._dataset_item_version()@@ -326,10 +387,15 @@ class TestOfflineExperimentsAPI(APIBaseTest):self._authenticate(auth_kind, user=membership.user)body = self._experiment_body() | {"dataset_revision_id": str(dataset_version.dataset_revision_id)}- created = self.client.post(self._endpoint(), body, format="json")+ with self.captureOnCommitCallbacks(execute=True):+ created = self.client.post(self._endpoint(), body, format="json")self.assertEqual(created.status_code, status.HTTP_201_CREATED if allowed else status.HTTP_400_BAD_REQUEST, created.data)+ if allowed:+ self.assertEqual(report_action.call_args.args[2]["dataset_source"], "posthog")+ else:+ report_action.assert_not_called()if not allowed:missing = self.client.post(self._endpoint(), body | {"dataset_revision_id": str(uuid4())}, format="json")self.assertEqual(created.data, missing.data)diff --git a/products/ai_observability/backend/dataset_service.py b/products/ai_observability/backend/dataset_service.pyindex ee8b99b1e96d..a3f06e7cba43 100644--- a/products/ai_observability/backend/dataset_service.py+++ b/products/ai_observability/backend/dataset_service.py@@ -11,6 +11,7 @@ from django.db.models import Maximport posthoganalytics+from posthog.dataclasses import frozenfrom posthog.exceptions_capture import capture_exceptionfrom posthog.models import Team, User@@ -125,6 +126,12 @@ class DatasetItemMutationResult:created: bool = True+@frozen+class DatasetMutationResult:+ dataset: Dataset+ changed_fields: tuple[str, ...] = ()++def _json_size(value: JSONValue | dict[str, JSONValue], *, field: str, limit: int) -> None:try:payload_size = len(@@ -486,7 +493,7 @@ def update_dataset(name: str | _Unset = UNSET,description: str | _Unset = UNSET,metadata: dict[str, JSONValue] | _Unset = UNSET,-) -> Dataset:+) -> DatasetMutationResult:dataset = _lock_dataset(team_id=team_id, dataset_id=dataset_id)if dataset.archived:raise DatasetMutationConflict(@@ -503,14 +510,21 @@ def update_dataset(metadata=next_metadata,)+ changed_fields: list[str] = []update_fields = ["updated_at"]if not isinstance(name, _Unset):+ if dataset.name != normalized_name:+ changed_fields.append("name")dataset.name = normalized_nameupdate_fields.append("name")if not isinstance(description, _Unset):+ if dataset.description != description:+ changed_fields.append("description")dataset.description = descriptionupdate_fields.append("description")if not isinstance(metadata, _Unset):+ if not _json_values_equal(dataset.metadata, metadata):+ changed_fields.append("metadata")dataset.metadata = metadataupdate_fields.append("metadata")try:@@ -520,27 +534,27 @@ def update_dataset(code="dataset_name_conflict",detail="A dataset with this name already exists.",) from error- return dataset+ return DatasetMutationResult(dataset=dataset, changed_fields=tuple(changed_fields))@transaction.atomic-def archive_dataset(*, team_id: int, dataset_id: UUID) -> Dataset:+def archive_dataset(*, team_id: int, dataset_id: UUID) -> DatasetMutationResult:dataset = _lock_dataset(team_id=team_id, dataset_id=dataset_id)if dataset.archived:- return dataset+ return DatasetMutationResult(dataset=dataset)dataset.archived = Truedataset.save(update_fields=["archived", "updated_at"])- return dataset+ return DatasetMutationResult(dataset=dataset, changed_fields=("archived",))@transaction.atomic-def restore_dataset(*, team_id: int, dataset_id: UUID) -> Dataset:+def restore_dataset(*, team_id: int, dataset_id: UUID) -> DatasetMutationResult:dataset = _lock_dataset(team_id=team_id, dataset_id=dataset_id)if not dataset.archived:- return dataset+ return DatasetMutationResult(dataset=dataset)dataset.archived = Falsedataset.save(update_fields=["archived", "updated_at"])- return dataset+ return DatasetMutationResult(dataset=dataset, changed_fields=("archived",))@transaction.atomicdiff --git a/products/ai_observability/backend/offline_evaluation_service.py b/products/ai_observability/backend/offline_evaluation_service.pyindex 85cc9cba02c4..eaa6efd92de3 100644--- a/products/ai_observability/backend/offline_evaluation_service.py+++ b/products/ai_observability/backend/offline_evaluation_service.py@@ -55,6 +55,7 @@ class ExperimentReceipt:experiment: OfflineExperimentcounts: ExperimentCountscreated: bool = False+ status_changed: bool = False@frozen@@ -261,7 +262,7 @@ class OfflineExperimentService(_OfflineEvaluationService):experiment.status = statusexperiment.finished_at = timezone.now()experiment.save(update_fields=["status", "finished_at"])- receipt = ExperimentReceipt(experiment=experiment, counts=counts)+ receipt = ExperimentReceipt(experiment=experiment, counts=counts, status_changed=True)return receiptdiff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsxindex 03815c7aea88..77181290a7e4 100644--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx@@ -48,6 +48,7 @@ export function AIObservabilityDatasetsScene({ navigation }: { navigation?: Reacreturn (<Linkto={datasetUrl(dataset.id)}+ data-attr="dataset-open"data-testid="dataset-link"className="block max-w-full @max-[50rem]/datasets:max-w-48 truncate font-semibold">@@ -129,38 +130,44 @@ export function AIObservabilityDatasetsScene({ navigation }: { navigation?: Reac<Moreoverlay={<>- <LemonButton- to={datasetUrl(dataset.id)}- data-attr={`dataset-item-${dataset.id}-dropdown-view`}- fullWidth- >- View- </LemonButton>-- <AccessControlAction- resourceType={AccessControlResourceType.LlmAnalytics}- minAccessLevel={AccessControlLevel.Editor}- userAccessLevel={toAccessControlLevel(dataset.user_access_level)}- >+ <div data-attr="dataset-dropdown-view"><LemonButton- status={dataset.archived ? undefined : 'danger'}- onClick={() =>- dataset.archived ? restoreDataset(dataset.id) : archiveDataset(dataset.id)- }- loading={archivingDatasetId === dataset.id}- disabledReason={- archivingDatasetId && archivingDatasetId !== dataset.id- ? 'Another dataset is being archived'- : undefined- }- data-attr={`dataset-${dataset.id}-dropdown-${- dataset.archived ? 'unarchive' : 'archive'- }`}+ to={datasetUrl(dataset.id)}+ data-attr={`dataset-item-${dataset.id}-dropdown-view`}fullWidth>- {dataset.archived ? 'Unarchive' : 'Archive'}+ View</LemonButton>- </AccessControlAction>+ </div>++ <div data-attr={`dataset-dropdown-${dataset.archived ? 'unarchive' : 'archive'}`}>+ <AccessControlAction+ resourceType={AccessControlResourceType.LlmAnalytics}+ minAccessLevel={AccessControlLevel.Editor}+ userAccessLevel={toAccessControlLevel(dataset.user_access_level)}+ >+ <LemonButton+ status={dataset.archived ? undefined : 'danger'}+ onClick={() =>+ dataset.archived+ ? restoreDataset(dataset.id)+ : archiveDataset(dataset.id)+ }+ loading={archivingDatasetId === dataset.id}+ disabledReason={+ archivingDatasetId && archivingDatasetId !== dataset.id+ ? 'Another dataset is being archived'+ : undefined+ }+ data-attr={`dataset-${dataset.id}-dropdown-${+ dataset.archived ? 'unarchive' : 'archive'+ }`}+ fullWidth+ >+ {dataset.archived ? 'Unarchive' : 'Archive'}+ </LemonButton>+ </AccessControlAction>+ </div></>}/>diff --git a/products/ai_observability/frontend/datasets/DatasetItemModal.tsx b/products/ai_observability/frontend/datasets/DatasetItemModal.tsxindex 4c229380dfe3..079f27fec4fe 100644--- a/products/ai_observability/frontend/datasets/DatasetItemModal.tsx+++ b/products/ai_observability/frontend/datasets/DatasetItemModal.tsx@@ -110,6 +110,7 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({logic={datasetItemModalLogic}props={logicProps}formKey="datasetItemForm"+ data-attr="dataset-item-form"enableFormOnSubmit={!isDatasetItemFormReadOnly}className="flex flex-col overflow-y-hidden">@@ -198,11 +199,20 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({<LemonModalFooter>{readOnly || loadError || loading ? (<>- <LemonButton type="secondary" onClick={() => onClose(refetchDatasetItems)}>+ <LemonButton+ type="secondary"+ onClick={() => onClose(refetchDatasetItems)}+ data-attr="dataset-item-close"+ >Close</LemonButton>{onUnarchive && storedDatasetItem?.archived && (- <LemonButton type="primary" onClick={onUnarchive} loading={unarchiving}>+ <LemonButton+ type="primary"+ onClick={onUnarchive}+ loading={unarchiving}+ data-attr="dataset-item-unarchive"+ >Unarchive</LemonButton>)}@@ -213,6 +223,7 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({<LemonButtontype="secondary"loading={isDatasetItemFormSubmitting}+ data-attr="dataset-item-save-and-add-another"disabledReason={datasetItemFormSubmitDisabledReason}htmlType="submit"onClick={(e) => {@@ -227,6 +238,7 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({<LemonButtontype="primary"htmlType="submit"+ data-attr="dataset-item-save"loading={isDatasetItemFormSubmitting}disabledReason={datasetItemFormSubmitDisabledReason}>@@ -321,6 +333,7 @@ function DatasetItemHistory({type="secondary"size="small"onClick={() => onRestore(version.version)}+ data-attr="dataset-item-restore-version"loading={restoringVersion === version.version}disabledReason={restoreDisabledReason ??@@ -348,6 +361,7 @@ function DatasetItemHistory({type="secondary"size="small"onClick={() => onPageChange(page - 1)}+ data-attr="dataset-item-history-previous"disabledReason={page <= 1 ? 'No previous page' : undefined}>Previous@@ -356,6 +370,7 @@ function DatasetItemHistory({type="secondary"size="small"onClick={() => onPageChange(page + 1)}+ data-attr="dataset-item-history-next"disabledReason={page >= pageCount ? 'No next page' : undefined}>Nextdiff --git a/products/ai_observability/frontend/datasets/DatasetItems.tsx b/products/ai_observability/frontend/datasets/DatasetItems.tsxindex e802bf47dc4f..f755edf0251c 100644--- a/products/ai_observability/frontend/datasets/DatasetItems.tsx+++ b/products/ai_observability/frontend/datasets/DatasetItems.tsx@@ -66,7 +66,10 @@ export function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {width: '10%',render: function renderID(_, item) {return (- <Link to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}>+ <Link+ to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}+ data-attr="dataset-item-open"+ >{truncateValue(item.id)}</Link>)@@ -162,35 +165,39 @@ export function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {<Moreoverlay={<>- <LemonButton- to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}- data-attr={`dataset-item-${item.id}-dropdown-edit`}- fullWidth- >- {canEditDataset && !item.archived ? 'Edit' : 'View'}- </LemonButton>-- {canManageDataset && !dataset.archived && !isHistoricalRevision && (+ <div data-attr="dataset-item-dropdown-edit"><LemonButton- status={item.archived ? undefined : 'danger'}- onClick={() =>- item.archived- ? restoreDatasetItem(item.id, item.version)- : archiveDatasetItem(item.id, item.version)- }- loading={archivingDatasetItemId === item.id}- disabledReason={- archivingDatasetItemId && archivingDatasetItemId !== item.id- ? 'Another dataset item is being archived'- : undefined- }- data-attr={`dataset-item-${item.id}-dropdown-${- item.archived ? 'unarchive' : 'archive'- }`}+ to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}+ data-attr={`dataset-item-${item.id}-dropdown-edit`}fullWidth>- {item.archived ? 'Unarchive' : 'Archive'}+ {canEditDataset && !item.archived ? 'Edit' : 'View'}</LemonButton>+ </div>++ {canManageDataset && !dataset.archived && !isHistoricalRevision && (+ <div data-attr={`dataset-item-dropdown-${item.archived ? 'unarchive' : 'archive'}`}>+ <LemonButton+ status={item.archived ? undefined : 'danger'}+ onClick={() =>+ item.archived+ ? restoreDatasetItem(item.id, item.version)+ : archiveDatasetItem(item.id, item.version)+ }+ loading={archivingDatasetItemId === item.id}+ disabledReason={+ archivingDatasetItemId && archivingDatasetItemId !== item.id+ ? 'Another dataset item is being archived'+ : undefined+ }+ data-attr={`dataset-item-${item.id}-dropdown-${+ item.archived ? 'unarchive' : 'archive'+ }`}+ fullWidth+ >+ {item.archived ? 'Unarchive' : 'Archive'}+ </LemonButton>+ </div>)}</>}diff --git a/products/ai_observability/frontend/datasets/EditDatasetForm.tsx b/products/ai_observability/frontend/datasets/EditDatasetForm.tsxindex bd74b586d870..948f8ae799c2 100644--- a/products/ai_observability/frontend/datasets/EditDatasetForm.tsx+++ b/products/ai_observability/frontend/datasets/EditDatasetForm.tsx@@ -27,6 +27,7 @@ export function EditDatasetForm(): JSX.Element {<LemonField name="description" label="Description" showOptional><LemonTextAreaclassName="ph-ignore-input"+ data-attr="dataset-description"placeholder="Describe what this dataset contains"value={datasetForm.description}onChange={(value) => setDatasetFormValue('description', value)}diff --git a/products/ai_observability/frontend/datasets/RefreshButton.tsx b/products/ai_observability/frontend/datasets/RefreshButton.tsxindex 94f9a08b8ae5..5287ed0429a2 100644--- a/products/ai_observability/frontend/datasets/RefreshButton.tsx+++ b/products/ai_observability/frontend/datasets/RefreshButton.tsx@@ -13,6 +13,7 @@ export function RefreshButton({ onClick, isRefreshing }: RefreshButtonProps): JS<div className="relative"><LemonButtononClick={onClick}+ data-attr="dataset-refresh"type="secondary"icon={isRefreshing ? <Spinner textColored /> : <IconRefresh />}size="small"diff --git a/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx b/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsxindex 35d2e1ab30da..b1ffab940910 100644--- a/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx+++ b/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx@@ -68,9 +68,11 @@ export const SaveToDatasetButton = React.memo(function SaveToDatasetButton({><LemonButtontype="secondary"+ data-attr="save-to-dataset-open"size="xsmall"icon={<IconDatabase />}sideAction={{+ 'data-attr': 'save-to-dataset-edit',icon: <IconPencil />,onClick: () => {if (!dropdownVisible) {@@ -117,7 +119,7 @@ function OverlayMenu(): JSX.Element {return (<Form logic={saveToDatasetButtonLogic} formKey="searchForm" className="w-xs" enableFormOnSubmit><LemonField name="search" label="Search" labelClassName="sr-only">- <LemonInput placeholder="Find a dataset" autoFocus />+ <LemonInput placeholder="Find a dataset" autoFocus data-attr="save-to-dataset-search" /></LemonField><LemonDivider className="my-0 mt-2" /><div@@ -206,6 +208,7 @@ function OverlayMenu(): JSX.Element {fullWidthsize="small"to={urls.aiObservabilityDataset('new')}+ data-attr="save-to-dataset-create"sideIcon={<IconExternal />}targetBlankonClick={() => {diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsxindex a1a7adadb933..32cacab56bb3 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx@@ -19,6 +19,7 @@ export function OfflineEvaluationsTabs({ activeTab }: { activeTab: OfflineEvaluaoptions={[{value: 'experiments',+ 'data-attr': 'offline-evaluations-experiments-tab',label: (<span className="flex items-center gap-1"><span>Experiments</span>@@ -38,6 +39,7 @@ export function OfflineEvaluationsTabs({ activeTab }: { activeTab: OfflineEvalua},{value: 'datasets',+ 'data-attr': 'offline-evaluations-datasets-tab',label: 'Datasets',disabledReason: !datasetsEnabled? 'Datasets are coming soon.'diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsxindex 155fbd814181..e343266c8038 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx@@ -42,7 +42,11 @@ export function OfflineExperimentContent(props: OfflineExperimentLogicProps): JSreturn (<div className="flex flex-col gap-4 min-w-0"><div className="flex flex-wrap items-center justify-between gap-2">- <LemonButton to={urls.aiObservabilityOfflineEvaluations()} icon={<IconArrowLeft />}>+ <LemonButton+ to={urls.aiObservabilityOfflineEvaluations()}+ icon={<IconArrowLeft />}+ data-attr="offline-experiment-back"+ >Experiments</LemonButton><LemonButton@@ -182,6 +186,7 @@ export function OfflineExperimentContent(props: OfflineExperimentLogicProps): JS<LinkclassName="break-words"to={offlineScorerHistoryUrl(summary.scorer, experiment, timezone)}+ data-attr="offline-experiment-scorer-history"><ScorerLabel scorer={summary.scorer} /></Link>@@ -225,7 +230,12 @@ export function OfflineExperimentContent(props: OfflineExperimentLogicProps): JStitle="Mark experiment as completed?"footer={<>- <LemonButton type="secondary" onClick={hideCompletion} disabled={completedLoading}>+ <LemonButton+ type="secondary"+ onClick={hideCompletion}+ disabled={completedLoading}+ data-attr="offline-experiment-cancel-complete"+ >Cancel</LemonButton><LemonButtondiff --git a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsxindex 7d534ca1154f..bd35b3df7f09 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx@@ -73,7 +73,10 @@ export function OfflineItemInspector({{item.trial && <span>{`Trial ${item.trial}`}</span>}{item.model_version && <span>{`Model: ${item.model_version}`}</span>}{item.application_trace_id && (- <Link to={urls.aiObservabilityTrace(item.application_trace_id)}>+ <Link+ to={urls.aiObservabilityTrace(item.application_trace_id)}+ data-attr="offline-item-application-trace"+ >Application trace</Link>)}@@ -138,6 +141,7 @@ export function OfflineItemInspector({to={urls.aiObservabilityTrace(selectedResult.evaluator_trace_id)}+ data-attr="offline-result-evaluator-trace">Evaluator trace</Link>@@ -219,6 +223,7 @@ export function OfflineItemInspector({className="max-w-full"active={result.id === selectedResult?.id}onClick={() => onSelectResult(result)}+ data-attr="offline-item-select-result"><ScorerLabel scorer={result.scorer} /></LemonButton>@@ -247,6 +252,7 @@ export function OfflineItemInspector({size="small"disabled={!resultCursors.length || resultsLoading}onClick={previousResults}+ data-attr="offline-item-results-previous">Previous</LemonButton>@@ -254,6 +260,7 @@ export function OfflineItemInspector({size="small"disabled={!!resultsError || !results?.next_cursor || resultsLoading}onClick={() => results?.next_cursor && nextResults(results.next_cursor)}+ data-attr="offline-item-results-next">Next</LemonButton>@@ -263,7 +270,7 @@ export function OfflineItemInspector({</div></LemonModalContent><LemonModalFooter>- <LemonButton type="secondary" onClick={onClose}>+ <LemonButton type="secondary" onClick={onClose} data-attr="offline-item-close">Close</LemonButton></LemonModalFooter>diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsxindex da8ef6a12f21..dbb0a5c1e95d 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx@@ -32,7 +32,9 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.Elwidth={720}footer={<>- <LemonButton onClick={closeChooser}>Cancel</LemonButton>+ <LemonButton onClick={closeChooser} data-attr="offline-score-selection-cancel">+ Cancel+ </LemonButton><LemonButton type="primary" onClick={saveScorers} data-attr="offline-score-selection-save">Save choices</LemonButton>@@ -48,6 +50,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.Eltype="search"value={scorerSearch}onChange={setScorerSearch}+ data-attr="offline-score-selection-search"placeholder="Search scorers"/><div className="h-48 @min-[32rem]:h-64 overflow-y-auto space-y-3">@@ -65,6 +68,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El{scorerOptions.results.map((scorer) => (<LemonCheckboxkey={scorer.id}+ data-attr="offline-score-selection-toggle"checked={draftScorerIds.includes(scorer.id)}label={`${scorer.name} (${scorer.kind})${scorer.archived ? ' · Archived' : ''}`}onChange={(checked) =>@@ -91,6 +95,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.Elsize="small"disabledReason={scorerOffset === 0 ? 'First page' : undefined}onClick={() => setScorerOffset(scorerOffset - 50)}+ data-attr="offline-score-selection-previous">Previous</LemonButton>@@ -100,6 +105,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.ElscorerOffset + 50 >= scorerOptions.count ? 'Last page' : undefined}onClick={() => setScorerOffset(scorerOffset + 50)}+ data-attr="offline-score-selection-next">Next</LemonButton>@@ -125,6 +131,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.Elsize="xsmall"disabledReason={index === 0 ? 'Already first' : undefined}onClick={() => move(index, -1)}+ data-attr="offline-score-selection-move-up"/><LemonButtonicon={<IconChevronDown />}@@ -134,10 +141,12 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.Elindex === draftScorerIds.length - 1 ? 'Already last' : undefined}onClick={() => move(index, 1)}+ data-attr="offline-score-selection-move-down"/><LemonButtonicon={<IconX />}aria-label="Remove score"+ data-attr="offline-score-selection-remove"size="xsmall"onClick={() =>setDraftScorerIds(draftScorerIds.filter((selected) => selected !== id))diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsxindex 9c271fec0508..4079af7f9c89 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx@@ -61,7 +61,11 @@ export function OfflineScorerHistoryTable({onClick={() => onToggleRun(key)}/><Tooltip title={point.experiment.name}>- <Link to={offlineExperimentUrl(point)} className="truncate min-w-0">+ <Link+ to={offlineExperimentUrl(point)}+ className="truncate min-w-0"+ data-attr="offline-history-open-experiment"+ >{point.experiment.name}</Link></Tooltip>diff --git a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsxindex df3d7752dabc..e692c397ca6a 100644--- a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx+++ b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx@@ -84,7 +84,9 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {render: function renderName(_, scoreDefinition) {return (<div className="font-semibold max-w-64 @max-[50rem]/scorers:max-w-32 @max-[40rem]/scorers:max-w-24 truncate">- <Link to={urls.aiObservabilityScorer(scoreDefinition.id)}>{scoreDefinition.name}</Link>+ <Link to={urls.aiObservabilityScorer(scoreDefinition.id)} data-attr="llma-scorer-open">+ {scoreDefinition.name}+ </Link></div>)},diff --git a/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx b/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsxindex ae1a422c94c8..021f4c788df1 100644--- a/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx+++ b/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx@@ -32,7 +32,13 @@ export function ScoreDefinitionActions({'data-attr': 'llma-scorer-edit-config',},...(showHistory- ? [{ label: 'Offline history', to: urls.aiObservabilityOfflineScorerHistory(definition.id) }]+ ? [+ {+ label: 'Offline history',+ to: urls.aiObservabilityOfflineScorerHistory(definition.id),+ 'data-attr': 'llma-scorer-offline-history',+ },+ ]: []),{label: 'Duplicate',@@ -53,6 +59,7 @@ export function ScoreDefinitionActions({icon={<IconEllipsis />}size="small"aria-label={`Actions for ${definition.name}`}+ data-attr="llma-scorer-actions"loading={archiving}/></LemonMenu>diff --git a/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts b/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.tsindex bdeb5d2bc1ac..f6969666c634 100644--- a/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts+++ b/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts@@ -10,7 +10,11 @@ export function openCreateScorerVersionDialog({LemonDialog.open({title: 'Create a new scorer version?',content: `Version ${currentVersion + 1} will keep the current configuration. Use this when your external evaluator changes.`,- primaryButton: { children: 'Create version', onClick: createVersion },- secondaryButton: { children: 'Cancel' },+ primaryButton: {+ children: 'Create version',+ onClick: createVersion,+ 'data-attr': 'llma-scorer-confirm-new-version',+ },+ secondaryButton: { children: 'Cancel', 'data-attr': 'llma-scorer-cancel-new-version' },})}