Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(aio): fill evaluation usage tracking gaps (#123640)

Radu Raicea3e3e92a23419
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 31521cacf264..1f1ce97fcc7f 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -401,6 +401,31 @@ Large payloads have separate storage and 30-day deadlines anchored to first acce
Retries neither extend deadlines nor restore deleted payloads.
Automatic payload deletion and usage billing are not enabled by these endpoints.
+### Product usage events
+
+The API reports these product usage events after the database transaction commits:
+
+| Events | When they fire |
+| -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `llma offline experiment created` | An experiment is first accepted. |
+| `llma offline experiment results uploaded` | A batch accepts new items or results. `created_item_count` and `created_result_count` exclude retries. |
+| `llma offline experiment completed`, `llma offline experiment failed` | An uploading experiment changes to the corresponding terminal state. Repeated closure emits no event. |
+| `llma dataset created`, `llma dataset updated`, `llma dataset archived`, `llma dataset restored` | A dataset is created or changes. Updates include `changed_fields`; unchanged updates and repeated archive/restore requests emit no event. |
+| `llma dataset item created`, `llma dataset item updated`, `llma dataset item archived`, `llma dataset item restored` | A new item version is accepted. Events include item and dataset IDs, version and dataset revision; restoration also includes the requested `source_version`. |
+| `llma dataset export requested`, `llma dataset export downloaded` | An export starts successfully or its content is served. Events include the export ID and pinned dataset revision. Each successful download request is counted. |
+| `llma scorer created`, `llma scorer updated`, `llma scorer version created` | A scorer is created, changed, or versioned. Archive and unarchive use `llma scorer updated` with `archived_new_value`. |
+
+Events include the acting user or service principal, project groups, and request context.
+Experiment lifecycle events include dataset source, run source, status, and expected and accepted counts.
+Dataset source is classified as `posthog` for hosted revisions, `external` for a supplied external source, or `unknown` when omitted; the original source string is never captured.
+Dataset and experiment usage events exclude input, output, reasoning, metadata values, and resource names.
+Rejected requests, unchanged item updates, and exact ingestion retries emit no success event.
+Dataset and experiment reporting failures are logged without failing successful API operations.
+
+Frontend controls use `data-attr` for autocapture action matching.
+Dataset row actions retain their existing resource-specific `data-attr` values and have stable `data-attr` values on their menu wrappers for aggregate matching.
+Existing `llma` event names and attributes remain unchanged so saved actions and dashboards continue to match.
+
### Durable usage recording
Newly accepted Postgres results produce durable `OfflineEvaluationUsageLedger` entries.
diff --git a/products/ai_observability/backend/api/analytics.py b/products/ai_observability/backend/api/analytics.py
new file mode 100644
index 000000000000..56f00da0b8fd
--- /dev/null
+++ b/products/ai_observability/backend/api/analytics.py
@@ -0,0 +1,10 @@
+from collections.abc import Callable
+
+from django.db import transaction
+
+
+def report_on_commit(callback: Callable[[], None]) -> None:
+ def report() -> None:
+ callback()
+
+ transaction.on_commit(report, robust=True)
diff --git a/products/ai_observability/backend/api/datasets.py b/products/ai_observability/backend/api/datasets.py
index debe34f372b3..befbc29f9154 100644
--- a/products/ai_observability/backend/api/datasets.py
+++ b/products/ai_observability/backend/api/datasets.py
@@ -1,3 +1,4 @@
+from functools import partial
from typing import NoReturn, cast
from uuid import UUID
@@ -40,6 +41,7 @@ from products.access_control.backend.presentation.access_control import (
AccessControlViewSetMixin,
UserAccessControlSerializerMixin,
)
+from products.ai_observability.backend.api.analytics import report_on_commit
from products.ai_observability.backend.api.dataset_exports import (
DatasetExportCreateSerializer,
DatasetExportErrorSerializer,
@@ -62,6 +64,7 @@ from products.ai_observability.backend.dataset_queries import (
)
from products.ai_observability.backend.dataset_service import (
UNSET,
+ DatasetItemMutationResult,
DatasetLimitExceeded,
DatasetMutationConflict,
DatasetValidationError,
@@ -667,6 +670,23 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
return DatasetRevisionReadSerializer
return DatasetReadSerializer
+ def _report_action(self, event: str, dataset: Dataset, properties: dict[str, object] | None = None) -> None:
+ report_on_commit(
+ partial(
+ report_user_action,
+ self.request.user,
+ event,
+ {
+ "dataset_id": str(dataset.id),
+ "has_description": bool(dataset.description),
+ "has_metadata": bool(dataset.metadata),
+ **(properties or {}),
+ },
+ team=self.team,
+ request=self.request,
+ )
+ )
+
@extend_schema(
parameters=[
DatasetListQuerySerializer,
@@ -762,17 +782,7 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
except (DatasetMutationConflict, DatasetLimitExceeded) as error:
return _conflict_response(error)
- report_user_action(
- request.user,
- "llma dataset created",
- {
- "dataset_id": str(dataset.id),
- "has_description": bool(dataset.description),
- "has_metadata": bool(dataset.metadata),
- },
- team=self.team,
- request=request,
- )
+ self._report_action("llma dataset created", dataset)
return Response(
DatasetReadSerializer(dataset, context=self.get_serializer_context()).data,
status=status.HTTP_201_CREATED,
@@ -796,15 +806,21 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
data = serializer.validated_data
try:
with transaction.atomic():
- dataset = update_dataset(
+ result = update_dataset(
team_id=self.team.id,
dataset_id=current.id,
name=data.get("name", UNSET),
description=data.get("description", UNSET),
metadata=data.get("metadata", UNSET),
)
+ dataset = result.dataset
+ changed_fields = list(result.changed_fields)
if "tags" in data:
+ if set(data["tags"]) != set(ResourceTags.names(dataset)):
+ changed_fields.append("tags")
ResourceTags.set(dataset, data["tags"])
+ if changed_fields:
+ self._report_action("llma dataset updated", dataset, {"changed_fields": changed_fields})
except DatasetValidationError as error:
_validation_error_response(error)
except DatasetMutationConflict as error:
@@ -822,8 +838,10 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
@action(detail=True, methods=["post"])
def archive(self, request: Request, *args: object, **kwargs: object) -> Response:
current = self.get_object()
- dataset = archive_dataset(team_id=self.team.id, dataset_id=current.id)
- return Response(DatasetReadSerializer(dataset, context=self.get_serializer_context()).data)
+ result = archive_dataset(team_id=self.team.id, dataset_id=current.id)
+ if result.changed_fields:
+ self._report_action("llma dataset archived", result.dataset)
+ return Response(DatasetReadSerializer(result.dataset, context=self.get_serializer_context()).data)
@extend_schema(
operation_id="datasets_restore",
@@ -835,8 +853,10 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
@action(detail=True, methods=["post"])
def restore(self, request: Request, *args: object, **kwargs: object) -> Response:
current = self.get_object()
- dataset = restore_dataset(team_id=self.team.id, dataset_id=current.id)
- return Response(DatasetReadSerializer(dataset, context=self.get_serializer_context()).data)
+ result = restore_dataset(team_id=self.team.id, dataset_id=current.id)
+ if result.changed_fields:
+ self._report_action("llma dataset restored", result.dataset)
+ return Response(DatasetReadSerializer(result.dataset, context=self.get_serializer_context()).data)
@extend_schema(
operation_id="datasets_revisions_list",
@@ -888,6 +908,11 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
)
except DatasetExportUnavailableError as error:
return Response({"detail": str(error)}, status=status.HTTP_409_CONFLICT)
+ self._report_action(
+ "llma dataset export requested",
+ dataset,
+ {"export_id": asset.id, "dataset_revision": (asset.export_context or {}).get("dataset_revision")},
+ )
return Response(DatasetExportReadSerializer(asset).data, status=status.HTTP_201_CREATED)
@extend_schema(
@@ -949,7 +974,13 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
{"detail": "The export is still being prepared."},
status=status.HTTP_409_CONFLICT,
)
- return get_export_asset_content_response(asset=asset, download=True)
+ response = get_export_asset_content_response(asset=asset, download=True)
+ self._report_action(
+ "llma dataset export downloaded",
+ dataset,
+ {"export_id": asset.id, "dataset_revision": (asset.export_context or {}).get("dataset_revision")},
+ )
+ return response
class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
@@ -1028,6 +1059,31 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
self.check_object_permissions(self.request, dataset)
return dataset
+ def _report_mutation(
+ self, event: str, result: DatasetItemMutationResult, properties: dict[str, object] | None = None
+ ) -> None:
+ if not result.created:
+ return
+ report_on_commit(
+ partial(
+ report_user_action,
+ self.request.user,
+ event,
+ {
+ "dataset_item_id": str(result.item.id),
+ "dataset_id": str(result.item.dataset_id),
+ "version": result.version.version,
+ "dataset_revision": result.version.dataset_revision.revision,
+ "has_expected_output": result.version.expected_output is not None,
+ "has_source_output": result.version.source_output is not None,
+ "has_provenance": result.version.source_trace_id is not None,
+ **(properties or {}),
+ },
+ team=self.team,
+ request=self.request,
+ )
+ )
+
@extend_schema(
parameters=[DatasetItemListQuerySerializer],
responses=DatasetItemReadSerializer(many=True),
@@ -1157,20 +1213,7 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
except (DatasetMutationConflict, DatasetLimitExceeded) as error:
return _conflict_response(error)
- if result.created:
- report_user_action(
- request.user,
- "llma dataset item created",
- {
- "dataset_item_id": str(result.item.id),
- "dataset_id": str(dataset.id),
- "has_expected_output": result.version.expected_output is not None,
- "has_source_output": result.version.source_output is not None,
- "has_provenance": result.version.source_trace_id is not None,
- },
- team=self.team,
- request=request,
- )
+ self._report_mutation("llma dataset item created", result)
response_status = status.HTTP_201_CREATED if result.created else status.HTTP_200_OK
return Response(DatasetItemReadSerializer(result.version).data, status=response_status)
@@ -1205,6 +1248,7 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
_validation_error_response(error)
except (DatasetMutationConflict, DatasetLimitExceeded) as error:
return _conflict_response(error)
+ self._report_mutation("llma dataset item updated", result)
return Response(DatasetItemReadSerializer(result.version).data)
@extend_schema(
@@ -1232,6 +1276,7 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
)
except (DatasetMutationConflict, DatasetLimitExceeded) as error:
return _conflict_response(error)
+ self._report_mutation("llma dataset item archived", result)
return Response(DatasetItemReadSerializer(result.version).data)
@extend_schema(
@@ -1262,6 +1307,9 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
raise Http404("Dataset item version not found.") from error
except (DatasetMutationConflict, DatasetLimitExceeded) as error:
return _conflict_response(error)
+ self._report_mutation(
+ "llma dataset item restored", result, {"source_version": serializer.validated_data.get("source_version")}
+ )
return Response(DatasetItemReadSerializer(result.version).data)
@extend_schema(
diff --git a/products/ai_observability/backend/api/offline_experiments.py b/products/ai_observability/backend/api/offline_experiments.py
index 045028d24f86..85b3fc241d0a 100644
--- a/products/ai_observability/backend/api/offline_experiments.py
+++ b/products/ai_observability/backend/api/offline_experiments.py
@@ -1,4 +1,5 @@
from dataclasses import asdict
+from functools import partial
from typing import TYPE_CHECKING, cast
from uuid import UUID
@@ -15,8 +16,10 @@ from rest_framework.throttling import BaseThrottle
from posthog.api.mixins import TypedRequest, ValidatedRequest, validated_request
from posthog.api.monitoring import monitor
+from posthog.event_usage import report_user_action
from posthog.permissions import is_service_auth
+from products.ai_observability.backend.api.analytics import report_on_commit
from products.ai_observability.backend.api.metrics import llma_track_latency
from products.ai_observability.backend.api.offline_evaluation_base import OfflineEvaluationViewSet
from products.ai_observability.backend.api.offline_experiment_access import (
@@ -153,6 +156,31 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
def _reference_access_control(self) -> "UserAccessControl | None":
return None if is_service_auth(self.request) else self.user_access_control
+ def _report_experiment_action(self, event: str, receipt: ExperimentReceipt) -> None:
+ experiment = receipt.experiment
+ if experiment.dataset_revision_id is not None:
+ dataset_source = "posthog"
+ elif experiment.dataset_source:
+ dataset_source = "external"
+ else:
+ dataset_source = "unknown"
+ report_on_commit(
+ partial(
+ report_user_action,
+ self.request.user,
+ event,
+ {
+ "experiment_id": str(experiment.id),
+ "dataset_source": dataset_source,
+ "run_source": experiment.run_source,
+ "status": experiment.status,
+ **asdict(receipt.counts),
+ },
+ team=self.team,
+ request=self.request,
+ )
+ )
+
def handle_exception(self, exc: Exception) -> Response:
action_name = getattr(self, "action", None)
if action_name in self.scope_object_read_actions or action_name in {"metadata", "partial_update"}:
@@ -203,6 +231,8 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
request.validated_data,
created_by_id=None if is_service_auth(request) else request.user.pk,
)
+ if receipt.created:
+ self._report_experiment_action("llma offline experiment created", receipt)
return _experiment_response(receipt, status=201 if receipt.created else 200)
def _experiment_id(self) -> UUID:
@@ -217,13 +247,31 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
@llma_track_latency("aio_offline_experiment_upload")
@monitor(feature=None, endpoint="aio_offline_experiment_upload", method="POST")
def upload(self, request: TypedRequest[UploadSubmission], **kwargs: object) -> Response:
+ experiment_id = self._experiment_id()
receipt = OfflineEvaluationIngestionService(
team_id=self.team_id, user_access_control=self._reference_access_control()
- ).upload(self._experiment_id(), request.validated_data)
+ ).upload(experiment_id, request.validated_data)
for created in (True, False):
OFFLINE_UPLOAD_RESULTS.labels(outcome="created" if created else "duplicate").inc(
sum(result.created == created for result in receipt.results)
)
+ created_item_count = sum(item.created for item in receipt.items)
+ created_result_count = sum(result.created for result in receipt.results)
+ if created_item_count or created_result_count:
+ report_on_commit(
+ partial(
+ report_user_action,
+ request.user,
+ "llma offline experiment results uploaded",
+ {
+ "experiment_id": str(experiment_id),
+ "created_item_count": created_item_count,
+ "created_result_count": created_result_count,
+ },
+ team=self.team,
+ request=request,
+ )
+ )
return Response(UploadReceiptSerializer(receipt).data)
@validated_request(
@@ -235,11 +283,12 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
@llma_track_latency("aio_offline_experiment_complete")
@monitor(feature=None, endpoint="aio_offline_experiment_complete", method="POST")
def complete(self, request: Request, **kwargs: object) -> Response:
- return _experiment_response(
- OfflineExperimentService(team_id=self.team_id, user_access_control=self._reference_access_control()).close(
- self._experiment_id(), status="completed"
- )
- )
+ receipt = OfflineExperimentService(
+ team_id=self.team_id, user_access_control=self._reference_access_control()
+ ).close(self._experiment_id(), status="completed")
+ if receipt.status_changed:
+ self._report_experiment_action("llma offline experiment completed", receipt)
+ return _experiment_response(receipt)
@validated_request(
EmptyOfflineExperimentSerializer,
@@ -250,8 +299,9 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
@llma_track_latency("aio_offline_experiment_fail")
@monitor(feature=None, endpoint="aio_offline_experiment_fail", method="POST")
def fail(self, request: Request, **kwargs: object) -> Response:
- return _experiment_response(
- OfflineExperimentService(team_id=self.team_id, user_access_control=self._reference_access_control()).close(
- self._experiment_id(), status="failed"
- )
- )
+ receipt = OfflineExperimentService(
+ team_id=self.team_id, user_access_control=self._reference_access_control()
+ ).close(self._experiment_id(), status="failed")
+ if receipt.status_changed:
+ self._report_experiment_action("llma offline experiment failed", receipt)
+ return _experiment_response(receipt)
diff --git a/products/ai_observability/backend/api/test/test_datasets.py b/products/ai_observability/backend/api/test/test_datasets.py
index 7cc8f90cc928..29df1fa75cf9 100644
--- a/products/ai_observability/backend/api/test/test_datasets.py
+++ b/products/ai_observability/backend/api/test/test_datasets.py
@@ -3,7 +3,7 @@ from datetime import timedelta
from functools import partial
from posthog.test.base import APIBaseTest
-from unittest.mock import DEFAULT, AsyncMock, MagicMock, patch
+from unittest.mock import ANY, DEFAULT, AsyncMock, MagicMock, Mock, patch
from django.core.cache import cache
from django.db import IntegrityError, connection, transaction
@@ -68,8 +68,10 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
- def test_dataset_lifecycle_and_name_uniqueness(self) -> None:
- dataset = self._create_dataset()
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_dataset_lifecycle_and_name_uniqueness(self, report_action: Mock) -> None:
+ with self.captureOnCommitCallbacks(execute=True):
+ dataset = self._create_dataset()
duplicate_response = self.client.post(
self.datasets_url,
@@ -79,8 +81,11 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(duplicate_response.status_code, status.HTTP_409_CONFLICT)
self.assertEqual(duplicate_response.data["code"], "dataset_name_conflict")
- archive_response = self.client.post(f"{self.datasets_url}{dataset['id']}/archive/")
+ with self.captureOnCommitCallbacks(execute=True):
+ archive_response = self.client.post(f"{self.datasets_url}{dataset['id']}/archive/")
+ archive_retry = self.client.post(f"{self.datasets_url}{dataset['id']}/archive/")
self.assertEqual(archive_response.status_code, status.HTTP_200_OK)
+ self.assertEqual(archive_retry.status_code, status.HTTP_200_OK)
self.assertTrue(archive_response.data["archived"])
active_list_response = self.client.get(self.datasets_url)
@@ -101,9 +106,93 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(blocked_item_response.status_code, status.HTTP_409_CONFLICT)
self.assertEqual(blocked_item_response.data["code"], "dataset_archived")
- restore_response = self.client.post(f"{self.datasets_url}{dataset['id']}/restore/")
+ with self.captureOnCommitCallbacks(execute=True):
+ restore_response = self.client.post(f"{self.datasets_url}{dataset['id']}/restore/")
+ restore_retry = self.client.post(f"{self.datasets_url}{dataset['id']}/restore/")
self.assertEqual(restore_response.status_code, status.HTTP_200_OK)
+ self.assertEqual(restore_retry.status_code, status.HTTP_200_OK)
self.assertFalse(restore_response.data["archived"])
+ self.assertEqual(
+ [call.args[1] for call in report_action.call_args_list],
+ ["llma dataset created", "llma dataset archived", "llma dataset restored"],
+ )
+ report_action.assert_called_with(
+ self.user,
+ "llma dataset restored",
+ {"dataset_id": dataset["id"], "has_description": True, "has_metadata": True},
+ team=self.team,
+ request=ANY,
+ )
+
+ @parameterized.expand(
+ [
+ ("name", {"name": "Updated name"}, ["name"]),
+ (
+ "metadata",
+ {"description": "Updated description", "metadata": {"reviewed": True}},
+ ["description", "metadata"],
+ ),
+ ("tags", {"tags": ["evaluation"]}, ["tags"]),
+ ]
+ )
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_dataset_update_reports_only_committed_changes(
+ self, _name: str, payload: dict[str, object], changed_fields: list[str], report_action: Mock
+ ) -> None:
+ dataset = self._create_dataset()
+ url = f"{self.datasets_url}{dataset['id']}/"
+ with self.captureOnCommitCallbacks(execute=True):
+ with self.assertRaises(RuntimeError), transaction.atomic():
+ response = self.client.patch(url, payload, format="json")
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ raise RuntimeError("Roll back the mutation")
+ report_action.assert_not_called()
+
+ with self.captureOnCommitCallbacks(execute=True):
+ response = self.client.patch(url, payload, format="json")
+ unchanged = self.client.patch(url, payload, format="json")
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ self.assertEqual(unchanged.status_code, status.HTTP_200_OK)
+ report_action.assert_called_once_with(
+ self.user,
+ "llma dataset updated",
+ {
+ "dataset_id": dataset["id"],
+ "has_description": True,
+ "has_metadata": True,
+ "changed_fields": changed_fields,
+ },
+ team=self.team,
+ request=ANY,
+ )
+
+ @parameterized.expand([("malformed_referer",), ("capture_failure",)])
+ @patch("posthog.event_usage.posthoganalytics.capture")
+ def test_analytics_failure_does_not_interrupt_mutations(self, failure: str, capture: Mock) -> None:
+ if failure == "malformed_referer":
+ self.client.credentials(HTTP_REFERER="http://[invalid")
+ else:
+ capture.side_effect = RuntimeError("Analytics unavailable")
+
+ with self.assertLogs("django.test", level="ERROR"), self.captureOnCommitCallbacks(execute=True):
+ dataset = self._create_dataset()
+ item = self._create_item(dataset["id"])
+ updated = self.client.patch(
+ f"{self.items_url}{item['id']}/",
+ {"base_version": 1, "input": "Updated input"},
+ format="json",
+ )
+ self.assertEqual(updated.status_code, status.HTTP_200_OK)
+ self.assertEqual(updated.data["version"], 2)
+
+ with self.assertLogs("django.test", level="ERROR"), self.captureOnCommitCallbacks(execute=True):
+ next_update = self.client.patch(
+ f"{self.items_url}{item['id']}/",
+ {"base_version": 2, "input": "Next input"},
+ format="json",
+ )
+ self.assertEqual(next_update.status_code, status.HTTP_200_OK)
+ self.assertEqual(next_update.data["version"], 3)
def test_dataset_list_filters_by_comma_separated_ids(self) -> None:
first_dataset = self._create_dataset("First dataset")
@@ -142,7 +231,8 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(item["input"], input_value)
- def test_item_create_handles_existing_client_item_id(self) -> None:
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_item_create_handles_existing_client_item_id(self, report_action: Mock) -> None:
dataset = self._create_dataset()
payload = {
"dataset": dataset["id"],
@@ -154,13 +244,29 @@ class TestDatasetsApi(APIBaseTest):
"source_timestamp": "2026-07-30T12:00:00Z",
}
- first_response = self.client.post(self.items_url, payload, format="json")
- retry_response = self.client.post(self.items_url, payload, format="json")
+ with self.captureOnCommitCallbacks(execute=True):
+ first_response = self.client.post(self.items_url, payload, format="json")
+ retry_response = self.client.post(self.items_url, payload, format="json")
self.assertEqual(first_response.status_code, status.HTTP_201_CREATED)
self.assertEqual(retry_response.status_code, status.HTTP_200_OK)
self.assertEqual(retry_response.data["id"], first_response.data["id"])
self.assertEqual(retry_response.data["client_item_id"], payload["client_item_id"])
+ report_action.assert_called_once_with(
+ self.user,
+ "llma dataset item created",
+ {
+ "dataset_item_id": first_response.data["id"],
+ "dataset_id": dataset["id"],
+ "version": 1,
+ "dataset_revision": 1,
+ "has_expected_output": False,
+ "has_source_output": True,
+ "has_provenance": True,
+ },
+ team=self.team,
+ request=ANY,
+ )
self.assertEqual(
DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset["id"]).count(),
1,
@@ -261,53 +367,75 @@ class TestDatasetsApi(APIBaseTest):
)
self.assertEqual(immutable_source_response.status_code, status.HTTP_400_BAD_REQUEST)
- def test_stale_update_does_not_create_a_revision(self) -> None:
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_stale_update_does_not_create_a_revision(self, report_action: Mock) -> None:
dataset = self._create_dataset()
item = self._create_item(dataset["id"])
- first_update = self.client.patch(
- f"{self.items_url}{item['id']}/",
- {"base_version": 1, "input": "new"},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ first_update = self.client.patch(
+ f"{self.items_url}{item['id']}/",
+ {"base_version": 1, "input": "new"},
+ format="json",
+ )
self.assertEqual(first_update.status_code, status.HTTP_200_OK)
- stale_update = self.client.patch(
- f"{self.items_url}{item['id']}/",
- {"base_version": 1, "input": "stale"},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ stale_update = self.client.patch(
+ f"{self.items_url}{item['id']}/",
+ {"base_version": 1, "input": "stale"},
+ format="json",
+ )
self.assertEqual(stale_update.status_code, status.HTTP_409_CONFLICT)
self.assertEqual(stale_update.data["code"], "stale_version")
self.assertEqual(stale_update.data["current_version"], 2)
+ report_action.assert_called_once_with(
+ self.user,
+ "llma dataset item updated",
+ {
+ "dataset_item_id": item["id"],
+ "dataset_id": dataset["id"],
+ "version": 2,
+ "dataset_revision": 2,
+ "has_expected_output": True,
+ "has_source_output": False,
+ "has_provenance": False,
+ },
+ team=self.team,
+ request=ANY,
+ )
self.assertEqual(
DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset["id"]).count(),
2,
)
- def test_unchanged_update_does_not_create_a_revision(self) -> None:
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_unchanged_update_does_not_create_a_revision(self, report_action: Mock) -> None:
dataset = self._create_dataset()
item = self._create_item(dataset["id"])
- response = self.client.patch(
- f"{self.items_url}{item['id']}/",
- {
- "base_version": 1,
- "input": item["input"],
- "expected_output": item["expected_output"],
- "metadata": item["metadata"],
- },
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ response = self.client.patch(
+ f"{self.items_url}{item['id']}/",
+ {
+ "base_version": 1,
+ "input": item["input"],
+ "expected_output": item["expected_output"],
+ "metadata": item["metadata"],
+ },
+ format="json",
+ )
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["version"], 1)
+ report_action.assert_not_called()
self.assertEqual(
DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset["id"]).count(),
1,
)
- def test_archive_and_restore_create_versions_with_explicit_transitions(self) -> None:
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_archive_and_restore_create_versions_with_explicit_transitions(self, report_action: Mock) -> None:
dataset = self._create_dataset()
item = self._create_item(dataset["id"], input="original")
update_response = self.client.patch(
@@ -317,11 +445,12 @@ class TestDatasetsApi(APIBaseTest):
)
self.assertEqual(update_response.status_code, status.HTTP_200_OK)
- archive_response = self.client.post(
- f"{self.items_url}{item['id']}/archive/",
- {"base_version": 2},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ archive_response = self.client.post(
+ f"{self.items_url}{item['id']}/archive/",
+ {"base_version": 2},
+ format="json",
+ )
self.assertEqual(archive_response.status_code, status.HTTP_200_OK)
self.assertTrue(archive_response.data["archived"])
self.assertEqual(archive_response.data["version"], 3)
@@ -334,31 +463,54 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(active_list.data["results"], [])
self.assertEqual([result["id"] for result in archived_list.data["results"]], [item["id"]])
- duplicate_archive = self.client.post(
- f"{self.items_url}{item['id']}/archive/",
- {"base_version": 3},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ duplicate_archive = self.client.post(
+ f"{self.items_url}{item['id']}/archive/",
+ {"base_version": 3},
+ format="json",
+ )
self.assertEqual(duplicate_archive.status_code, status.HTTP_409_CONFLICT)
self.assertEqual(duplicate_archive.data["code"], "dataset_item_archived")
- restore_response = self.client.post(
- f"{self.items_url}{item['id']}/restore/",
- {"base_version": 3, "source_version": 1},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ restore_response = self.client.post(
+ f"{self.items_url}{item['id']}/restore/",
+ {"base_version": 3, "source_version": 1},
+ format="json",
+ )
self.assertEqual(restore_response.status_code, status.HTTP_200_OK)
self.assertFalse(restore_response.data["archived"])
self.assertEqual(restore_response.data["input"], "original")
self.assertEqual(restore_response.data["version"], 4)
- duplicate_restore = self.client.post(
- f"{self.items_url}{item['id']}/restore/",
- {"base_version": 4},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ duplicate_restore = self.client.post(
+ f"{self.items_url}{item['id']}/restore/",
+ {"base_version": 4},
+ format="json",
+ )
self.assertEqual(duplicate_restore.status_code, status.HTTP_409_CONFLICT)
self.assertEqual(duplicate_restore.data["code"], "dataset_item_active")
+ self.assertEqual(
+ [call.args[1] for call in report_action.call_args_list],
+ ["llma dataset item archived", "llma dataset item restored"],
+ )
+ report_action.assert_called_with(
+ self.user,
+ "llma dataset item restored",
+ {
+ "dataset_item_id": item["id"],
+ "dataset_id": dataset["id"],
+ "version": 4,
+ "dataset_revision": 4,
+ "has_expected_output": True,
+ "has_source_output": False,
+ "has_provenance": False,
+ "source_version": 1,
+ },
+ team=self.team,
+ request=ANY,
+ )
def test_revision_list_reconstructs_exact_snapshots(self) -> None:
dataset = self._create_dataset()
@@ -1013,15 +1165,19 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(len(response.data["results"]), 2)
@patch("products.exports.backend.facade.api.async_connect", new_callable=AsyncMock)
- def test_dataset_export_pins_revision_and_scopes_status_and_content(self, async_connect: AsyncMock) -> None:
+ @patch("products.ai_observability.backend.api.datasets.report_user_action")
+ def test_dataset_export_pins_revision_and_scopes_status_and_content(
+ self, report_action: Mock, async_connect: AsyncMock
+ ) -> None:
dataset = self._create_dataset()
first_item = self._create_item(dataset["id"], input={"question": "First"})
- create_response = self.client.post(
- f"{self.datasets_url}{dataset['id']}/exports/",
- {},
- format="json",
- )
+ with self.captureOnCommitCallbacks(execute=True):
+ create_response = self.client.post(
+ f"{self.datasets_url}{dataset['id']}/exports/",
+ {},
+ format="json",
+ )
self.assertEqual(create_response.status_code, status.HTTP_201_CREATED)
self.assertEqual(create_response.data["status"], "pending")
@@ -1043,7 +1199,10 @@ class TestDatasetsApi(APIBaseTest):
mismatched_parent_response = self.client.get(f"{self.datasets_url}{other_dataset['id']}/exports/{export_id}/")
self.assertEqual(mismatched_parent_response.status_code, status.HTTP_404_NOT_FOUND)
- pending_content_response = self.client.get(f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/")
+ with self.captureOnCommitCallbacks(execute=True):
+ pending_content_response = self.client.get(
+ f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/"
+ )
self.assertEqual(pending_content_response.status_code, status.HTTP_409_CONFLICT)
asset = ExportedAsset.objects.get(id=export_id)
@@ -1051,11 +1210,29 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(asset.export_context["kind"], DATASET_EXPORT_KIND)
asset.content = b'{"input":{"question":"First"}}\n'
asset.save(update_fields=["content"])
- content_response = self.client.get(f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/")
+ with self.captureOnCommitCallbacks(execute=True):
+ content_response = self.client.get(f"{self.datasets_url}{dataset['id']}/exports/{export_id}/content/")
self.assertEqual(content_response.status_code, status.HTTP_200_OK)
self.assertEqual(content_response.content, asset.content)
self.assertIn("attachment", content_response["Content-Disposition"])
+ self.assertEqual(
+ [call.args[1] for call in report_action.call_args_list],
+ ["llma dataset export requested", "llma dataset export downloaded"],
+ )
+ report_action.assert_called_with(
+ self.user,
+ "llma dataset export downloaded",
+ {
+ "dataset_id": dataset["id"],
+ "has_description": True,
+ "has_metadata": True,
+ "export_id": export_id,
+ "dataset_revision": 1,
+ },
+ team=self.team,
+ request=ANY,
+ )
@patch("posthog.rate_limit.is_rate_limit_enabled", return_value=True)
@patch("products.ai_observability.backend.api.datasets.DatasetExportRateThrottle.rate", new="1/day")
diff --git a/products/ai_observability/backend/api/test/test_offline_experiments.py b/products/ai_observability/backend/api/test/test_offline_experiments.py
index 1bd770ba0a7c..c4bafdf877b6 100644
--- a/products/ai_observability/backend/api/test/test_offline_experiments.py
+++ b/products/ai_observability/backend/api/test/test_offline_experiments.py
@@ -3,7 +3,7 @@ from uuid import uuid4
import pytest
from posthog.test.base import APIBaseTest
-from unittest.mock import patch
+from unittest.mock import Mock, patch
from django.db import OperationalError, connection
from django.test import SimpleTestCase
@@ -118,13 +118,27 @@ class TestOfflineExperimentsAPI(APIBaseTest):
input="What is 2 + 2?",
)
- @parameterized.expand([("session",), ("personal_key",), ("project_key",)])
- def test_ingestion_and_both_lifecycle_actions(self, auth_kind: str) -> None:
+ @parameterized.expand([("session", False), ("personal_key", False), ("project_key", False), ("session", True)])
+ @patch("products.ai_observability.backend.api.offline_experiments.report_user_action")
+ def test_ingestion_and_both_lifecycle_actions(
+ self, auth_kind: str, analytics_failure: bool, report_action: Mock
+ ) -> None:
+ if analytics_failure:
+ report_action.side_effect = RuntimeError("Analytics unavailable")
definition = ScoreDefinition.objects.create(team=self.team, name="Correct", kind="boolean")
version = definition.create_new_version(config={"true_label": "Yes", "false_label": "No"}, created_by=self.user)
+ other_definition = ScoreDefinition.objects.create(team=self.team, name="Relevant", kind="boolean")
+ other_version = other_definition.create_new_version(
+ config={"true_label": "Yes", "false_label": "No"}, created_by=self.user
+ )
self._authenticate(auth_kind)
- body = self._experiment_body() | {"expected_item_count": 1, "expected_result_count": 1}
- created = self.client.post(self._endpoint(), body, format="json")
+ body = self._experiment_body() | {
+ "expected_item_count": 2,
+ "expected_result_count": 4,
+ "dataset_source": "https://private.example.com/datasets/customer-evals",
+ }
+ with self.captureOnCommitCallbacks(execute=True):
+ created = self.client.post(self._endpoint(), body, format="json")
self.assertEqual(created.status_code, status.HTTP_201_CREATED, created.data)
experiment_id = str(body["id"])
self.assertEqual(created.data["id"], experiment_id)
@@ -132,49 +146,89 @@ class TestOfflineExperimentsAPI(APIBaseTest):
self.assertEqual(
OfflineExperiment.objects.for_team(self.team.id).get(pk=experiment_id).created_by_id, expected_author
)
- incomplete = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")
+ with self.captureOnCommitCallbacks(execute=True):
+ incomplete = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")
self.assertEqual(incomplete.status_code, status.HTTP_409_CONFLICT, incomplete.data)
self.assertEqual(incomplete.data["code"], "expected_count_mismatch")
- self.assertEqual(incomplete.data["expected_item_count"], 1)
- self.assertEqual(incomplete.data["expected_result_count"], 1)
+ self.assertEqual(incomplete.data["expected_item_count"], 2)
+ self.assertEqual(incomplete.data["expected_result_count"], 4)
self.assertEqual(incomplete.data["accepted_item_count"], 0)
self.assertEqual(incomplete.data["accepted_result_count"], 0)
- item_id = str(uuid4())
- uploaded = self.client.post(
- self._endpoint(experiment_id, "upload"),
- {
- "items": [{"id": item_id, "payload": {"input": "What is 2 + 2?", "output": "5"}}],
- "results": [{"item_id": item_id, "scorer_version_id": str(version.id), "status": "ok", "value": False}],
- },
- format="json",
- )
+ item_ids = [str(uuid4()), str(uuid4())]
+ upload_body = {
+ "items": [{"id": item_id, "payload": {"input": "What is 2 + 2?", "output": "5"}} for item_id in item_ids],
+ "results": [
+ {"item_id": item_id, "scorer_version_id": str(scorer_version.id), "status": "ok", "value": False}
+ for item_id in item_ids
+ for scorer_version in (version, other_version)
+ ],
+ }
+ with self.captureOnCommitCallbacks(execute=True):
+ uploaded = self.client.post(self._endpoint(experiment_id, "upload"), upload_body, format="json")
self.assertEqual(uploaded.status_code, status.HTTP_200_OK, uploaded.data)
result = OfflineEvaluationResult.objects.for_team(self.team.id).get(pk=uploaded.data["results"][0]["id"])
self.assertIs(result.boolean_value, False)
self.assertEqual(result.scorer_version_id, version.id)
- completed = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")
+ with self.captureOnCommitCallbacks(execute=True):
+ completed = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")
self.assertEqual(completed.status_code, status.HTTP_200_OK, completed.data)
self.assertEqual(completed.data["status"], "completed")
- self.assertEqual(completed.data["accepted_item_count"], 1)
- self.assertEqual(completed.data["accepted_result_count"], 1)
+ self.assertEqual(completed.data["accepted_item_count"], 2)
+ self.assertEqual(completed.data["accepted_result_count"], 4)
failed_body = self._experiment_body() | {"expected_result_count": 10}
- self.assertEqual(
- self.client.post(self._endpoint(), failed_body, format="json").status_code, status.HTTP_201_CREATED
- )
- failed = self.client.post(self._endpoint(str(failed_body["id"]), "fail"), {}, format="json")
+ with self.captureOnCommitCallbacks(execute=True):
+ self.assertEqual(
+ self.client.post(self._endpoint(), failed_body, format="json").status_code, status.HTTP_201_CREATED
+ )
+ failed = self.client.post(self._endpoint(str(failed_body["id"]), "fail"), {}, format="json")
self.assertEqual(failed.status_code, status.HTTP_200_OK, failed.data)
self.assertEqual(failed.data["status"], "failed")
self.assertEqual(failed.data["accepted_result_count"], 0)
self._authenticate("project_key" if auth_kind != "project_key" else "session")
- retried = self.client.post(self._endpoint(), body, format="json")
+ with self.captureOnCommitCallbacks(execute=True):
+ retried = self.client.post(self._endpoint(), body, format="json")
+ upload_retry = self.client.post(self._endpoint(experiment_id, "upload"), upload_body, format="json")
+ complete_retry = self.client.post(self._endpoint(experiment_id, "complete"), {}, format="json")
+ fail_retry = self.client.post(self._endpoint(str(failed_body["id"]), "fail"), {}, format="json")
self.assertEqual(retried.status_code, status.HTTP_200_OK)
+ self.assertEqual(upload_retry.status_code, status.HTTP_200_OK)
+ self.assertEqual(complete_retry.status_code, status.HTTP_200_OK)
+ self.assertEqual(fail_retry.status_code, status.HTTP_200_OK)
self.assertEqual(
OfflineExperiment.objects.for_team(self.team.id).get(pk=experiment_id).created_by_id, expected_author
)
+ self.assertEqual(
+ [call.args[1] for call in report_action.call_args_list],
+ [
+ "llma offline experiment created",
+ "llma offline experiment results uploaded",
+ "llma offline experiment completed",
+ "llma offline experiment created",
+ "llma offline experiment failed",
+ ],
+ )
+ upload_event = report_action.call_args_list[1]
+ self.assertEqual(
+ upload_event.args[2],
+ {"experiment_id": experiment_id, "created_item_count": 2, "created_result_count": 4},
+ )
+ completed_event = report_action.call_args_list[2]
+ self.assertEqual(report_action.call_args_list[0].args[2]["dataset_source"], "external")
+ self.assertEqual(completed_event.args[2]["dataset_source"], "external")
+ self.assertEqual(report_action.call_args_list[3].args[2]["dataset_source"], "unknown")
+ self.assertEqual(report_action.call_args_list[4].args[2]["dataset_source"], "unknown")
+ for call in report_action.call_args_list:
+ self.assertNotIn(body["dataset_source"], call.args[2].values())
+ self.assertEqual(completed_event.args[2]["status"], "completed")
+ self.assertEqual(completed_event.args[2]["accepted_item_count"], 2)
+ self.assertEqual(completed_event.args[2]["accepted_result_count"], 4)
+ self.assertEqual(completed_event.kwargs["team"], self.team)
+ self.assertEqual(completed_event.args[0], completed_event.kwargs["request"].user)
+ self.assertTrue(completed_event.args[0].distinct_id)
def test_ledger_failure_returns_server_error_and_mixed_upload_can_retry(self) -> None:
definition = ScoreDefinition.objects.create(team=self.team, name="Correct", kind="boolean")
@@ -304,8 +358,15 @@ class TestOfflineExperimentsAPI(APIBaseTest):
("project_key", "project_key", "none", "none", True),
]
)
+ @patch("products.ai_observability.backend.api.offline_experiments.report_user_action")
def test_references_require_viewer_access(
- self, _name: str, auth_kind: str, resource_access: str, object_access: str | None, allowed: bool
+ self,
+ _name: str,
+ auth_kind: str,
+ resource_access: str,
+ object_access: str | None,
+ allowed: bool,
+ report_action: Mock,
) -> None:
membership = self._evaluation_member()
dataset_version = self._dataset_item_version()
@@ -326,10 +387,15 @@ class TestOfflineExperimentsAPI(APIBaseTest):
self._authenticate(auth_kind, user=membership.user)
body = self._experiment_body() | {"dataset_revision_id": str(dataset_version.dataset_revision_id)}
- created = self.client.post(self._endpoint(), body, format="json")
+ with self.captureOnCommitCallbacks(execute=True):
+ created = self.client.post(self._endpoint(), body, format="json")
self.assertEqual(
created.status_code, status.HTTP_201_CREATED if allowed else status.HTTP_400_BAD_REQUEST, created.data
)
+ if allowed:
+ self.assertEqual(report_action.call_args.args[2]["dataset_source"], "posthog")
+ else:
+ report_action.assert_not_called()
if not allowed:
missing = self.client.post(self._endpoint(), body | {"dataset_revision_id": str(uuid4())}, format="json")
self.assertEqual(created.data, missing.data)
diff --git a/products/ai_observability/backend/dataset_service.py b/products/ai_observability/backend/dataset_service.py
index ee8b99b1e96d..a3f06e7cba43 100644
--- a/products/ai_observability/backend/dataset_service.py
+++ b/products/ai_observability/backend/dataset_service.py
@@ -11,6 +11,7 @@ from django.db.models import Max
import posthoganalytics
+from posthog.dataclasses import frozen
from posthog.exceptions_capture import capture_exception
from posthog.models import Team, User
@@ -125,6 +126,12 @@ class DatasetItemMutationResult:
created: bool = True
+@frozen
+class DatasetMutationResult:
+ dataset: Dataset
+ changed_fields: tuple[str, ...] = ()
+
+
def _json_size(value: JSONValue | dict[str, JSONValue], *, field: str, limit: int) -> None:
try:
payload_size = len(
@@ -486,7 +493,7 @@ def update_dataset(
name: str | _Unset = UNSET,
description: str | _Unset = UNSET,
metadata: dict[str, JSONValue] | _Unset = UNSET,
-) -> Dataset:
+) -> DatasetMutationResult:
dataset = _lock_dataset(team_id=team_id, dataset_id=dataset_id)
if dataset.archived:
raise DatasetMutationConflict(
@@ -503,14 +510,21 @@ def update_dataset(
metadata=next_metadata,
)
+ changed_fields: list[str] = []
update_fields = ["updated_at"]
if not isinstance(name, _Unset):
+ if dataset.name != normalized_name:
+ changed_fields.append("name")
dataset.name = normalized_name
update_fields.append("name")
if not isinstance(description, _Unset):
+ if dataset.description != description:
+ changed_fields.append("description")
dataset.description = description
update_fields.append("description")
if not isinstance(metadata, _Unset):
+ if not _json_values_equal(dataset.metadata, metadata):
+ changed_fields.append("metadata")
dataset.metadata = metadata
update_fields.append("metadata")
try:
@@ -520,27 +534,27 @@ def update_dataset(
code="dataset_name_conflict",
detail="A dataset with this name already exists.",
) from error
- return dataset
+ return DatasetMutationResult(dataset=dataset, changed_fields=tuple(changed_fields))
@transaction.atomic
-def archive_dataset(*, team_id: int, dataset_id: UUID) -> Dataset:
+def archive_dataset(*, team_id: int, dataset_id: UUID) -> DatasetMutationResult:
dataset = _lock_dataset(team_id=team_id, dataset_id=dataset_id)
if dataset.archived:
- return dataset
+ return DatasetMutationResult(dataset=dataset)
dataset.archived = True
dataset.save(update_fields=["archived", "updated_at"])
- return dataset
+ return DatasetMutationResult(dataset=dataset, changed_fields=("archived",))
@transaction.atomic
-def restore_dataset(*, team_id: int, dataset_id: UUID) -> Dataset:
+def restore_dataset(*, team_id: int, dataset_id: UUID) -> DatasetMutationResult:
dataset = _lock_dataset(team_id=team_id, dataset_id=dataset_id)
if not dataset.archived:
- return dataset
+ return DatasetMutationResult(dataset=dataset)
dataset.archived = False
dataset.save(update_fields=["archived", "updated_at"])
- return dataset
+ return DatasetMutationResult(dataset=dataset, changed_fields=("archived",))
@transaction.atomic
diff --git a/products/ai_observability/backend/offline_evaluation_service.py b/products/ai_observability/backend/offline_evaluation_service.py
index 85cc9cba02c4..eaa6efd92de3 100644
--- a/products/ai_observability/backend/offline_evaluation_service.py
+++ b/products/ai_observability/backend/offline_evaluation_service.py
@@ -55,6 +55,7 @@ class ExperimentReceipt:
experiment: OfflineExperiment
counts: ExperimentCounts
created: bool = False
+ status_changed: bool = False
@frozen
@@ -261,7 +262,7 @@ class OfflineExperimentService(_OfflineEvaluationService):
experiment.status = status
experiment.finished_at = timezone.now()
experiment.save(update_fields=["status", "finished_at"])
- receipt = ExperimentReceipt(experiment=experiment, counts=counts)
+ receipt = ExperimentReceipt(experiment=experiment, counts=counts, status_changed=True)
return receipt
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
index 03815c7aea88..77181290a7e4 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
@@ -48,6 +48,7 @@ export function AIObservabilityDatasetsScene({ navigation }: { navigation?: Reac
return (
<Link
to={datasetUrl(dataset.id)}
+ data-attr="dataset-open"
data-testid="dataset-link"
className="block max-w-full @max-[50rem]/datasets:max-w-48 truncate font-semibold"
>
@@ -129,38 +130,44 @@ export function AIObservabilityDatasetsScene({ navigation }: { navigation?: Reac
<More
overlay={
<>
- <LemonButton
- to={datasetUrl(dataset.id)}
- data-attr={`dataset-item-${dataset.id}-dropdown-view`}
- fullWidth
- >
- View
- </LemonButton>
-
- <AccessControlAction
- resourceType={AccessControlResourceType.LlmAnalytics}
- minAccessLevel={AccessControlLevel.Editor}
- userAccessLevel={toAccessControlLevel(dataset.user_access_level)}
- >
+ <div data-attr="dataset-dropdown-view">
<LemonButton
- status={dataset.archived ? undefined : 'danger'}
- onClick={() =>
- dataset.archived ? restoreDataset(dataset.id) : archiveDataset(dataset.id)
- }
- loading={archivingDatasetId === dataset.id}
- disabledReason={
- archivingDatasetId && archivingDatasetId !== dataset.id
- ? 'Another dataset is being archived'
- : undefined
- }
- data-attr={`dataset-${dataset.id}-dropdown-${
- dataset.archived ? 'unarchive' : 'archive'
- }`}
+ to={datasetUrl(dataset.id)}
+ data-attr={`dataset-item-${dataset.id}-dropdown-view`}
fullWidth
>
- {dataset.archived ? 'Unarchive' : 'Archive'}
+ View
</LemonButton>
- </AccessControlAction>
+ </div>
+
+ <div data-attr={`dataset-dropdown-${dataset.archived ? 'unarchive' : 'archive'}`}>
+ <AccessControlAction
+ resourceType={AccessControlResourceType.LlmAnalytics}
+ minAccessLevel={AccessControlLevel.Editor}
+ userAccessLevel={toAccessControlLevel(dataset.user_access_level)}
+ >
+ <LemonButton
+ status={dataset.archived ? undefined : 'danger'}
+ onClick={() =>
+ dataset.archived
+ ? restoreDataset(dataset.id)
+ : archiveDataset(dataset.id)
+ }
+ loading={archivingDatasetId === dataset.id}
+ disabledReason={
+ archivingDatasetId && archivingDatasetId !== dataset.id
+ ? 'Another dataset is being archived'
+ : undefined
+ }
+ data-attr={`dataset-${dataset.id}-dropdown-${
+ dataset.archived ? 'unarchive' : 'archive'
+ }`}
+ fullWidth
+ >
+ {dataset.archived ? 'Unarchive' : 'Archive'}
+ </LemonButton>
+ </AccessControlAction>
+ </div>
</>
}
/>
diff --git a/products/ai_observability/frontend/datasets/DatasetItemModal.tsx b/products/ai_observability/frontend/datasets/DatasetItemModal.tsx
index 4c229380dfe3..079f27fec4fe 100644
--- a/products/ai_observability/frontend/datasets/DatasetItemModal.tsx
+++ b/products/ai_observability/frontend/datasets/DatasetItemModal.tsx
@@ -110,6 +110,7 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({
logic={datasetItemModalLogic}
props={logicProps}
formKey="datasetItemForm"
+ data-attr="dataset-item-form"
enableFormOnSubmit={!isDatasetItemFormReadOnly}
className="flex flex-col overflow-y-hidden"
>
@@ -198,11 +199,20 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({
<LemonModalFooter>
{readOnly || loadError || loading ? (
<>
- <LemonButton type="secondary" onClick={() => onClose(refetchDatasetItems)}>
+ <LemonButton
+ type="secondary"
+ onClick={() => onClose(refetchDatasetItems)}
+ data-attr="dataset-item-close"
+ >
Close
</LemonButton>
{onUnarchive && storedDatasetItem?.archived && (
- <LemonButton type="primary" onClick={onUnarchive} loading={unarchiving}>
+ <LemonButton
+ type="primary"
+ onClick={onUnarchive}
+ loading={unarchiving}
+ data-attr="dataset-item-unarchive"
+ >
Unarchive
</LemonButton>
)}
@@ -213,6 +223,7 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({
<LemonButton
type="secondary"
loading={isDatasetItemFormSubmitting}
+ data-attr="dataset-item-save-and-add-another"
disabledReason={datasetItemFormSubmitDisabledReason}
htmlType="submit"
onClick={(e) => {
@@ -227,6 +238,7 @@ export const DatasetItemModal = React.memo(function DatasetItemModal({
<LemonButton
type="primary"
htmlType="submit"
+ data-attr="dataset-item-save"
loading={isDatasetItemFormSubmitting}
disabledReason={datasetItemFormSubmitDisabledReason}
>
@@ -321,6 +333,7 @@ function DatasetItemHistory({
type="secondary"
size="small"
onClick={() => onRestore(version.version)}
+ data-attr="dataset-item-restore-version"
loading={restoringVersion === version.version}
disabledReason={
restoreDisabledReason ??
@@ -348,6 +361,7 @@ function DatasetItemHistory({
type="secondary"
size="small"
onClick={() => onPageChange(page - 1)}
+ data-attr="dataset-item-history-previous"
disabledReason={page <= 1 ? 'No previous page' : undefined}
>
Previous
@@ -356,6 +370,7 @@ function DatasetItemHistory({
type="secondary"
size="small"
onClick={() => onPageChange(page + 1)}
+ data-attr="dataset-item-history-next"
disabledReason={page >= pageCount ? 'No next page' : undefined}
>
Next
diff --git a/products/ai_observability/frontend/datasets/DatasetItems.tsx b/products/ai_observability/frontend/datasets/DatasetItems.tsx
index e802bf47dc4f..f755edf0251c 100644
--- a/products/ai_observability/frontend/datasets/DatasetItems.tsx
+++ b/products/ai_observability/frontend/datasets/DatasetItems.tsx
@@ -66,7 +66,10 @@ export function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {
width: '10%',
render: function renderID(_, item) {
return (
- <Link to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}>
+ <Link
+ to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}
+ data-attr="dataset-item-open"
+ >
{truncateValue(item.id)}
</Link>
)
@@ -162,35 +165,39 @@ export function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {
<More
overlay={
<>
- <LemonButton
- to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}
- data-attr={`dataset-item-${item.id}-dropdown-edit`}
- fullWidth
- >
- {canEditDataset && !item.archived ? 'Edit' : 'View'}
- </LemonButton>
-
- {canManageDataset && !dataset.archived && !isHistoricalRevision && (
+ <div data-attr="dataset-item-dropdown-edit">
<LemonButton
- status={item.archived ? undefined : 'danger'}
- onClick={() =>
- item.archived
- ? restoreDatasetItem(item.id, item.version)
- : archiveDatasetItem(item.id, item.version)
- }
- loading={archivingDatasetItemId === item.id}
- disabledReason={
- archivingDatasetItemId && archivingDatasetItemId !== item.id
- ? 'Another dataset item is being archived'
- : undefined
- }
- data-attr={`dataset-item-${item.id}-dropdown-${
- item.archived ? 'unarchive' : 'archive'
- }`}
+ to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}
+ data-attr={`dataset-item-${item.id}-dropdown-edit`}
fullWidth
>
- {item.archived ? 'Unarchive' : 'Archive'}
+ {canEditDataset && !item.archived ? 'Edit' : 'View'}
</LemonButton>
+ </div>
+
+ {canManageDataset && !dataset.archived && !isHistoricalRevision && (
+ <div data-attr={`dataset-item-dropdown-${item.archived ? 'unarchive' : 'archive'}`}>
+ <LemonButton
+ status={item.archived ? undefined : 'danger'}
+ onClick={() =>
+ item.archived
+ ? restoreDatasetItem(item.id, item.version)
+ : archiveDatasetItem(item.id, item.version)
+ }
+ loading={archivingDatasetItemId === item.id}
+ disabledReason={
+ archivingDatasetItemId && archivingDatasetItemId !== item.id
+ ? 'Another dataset item is being archived'
+ : undefined
+ }
+ data-attr={`dataset-item-${item.id}-dropdown-${
+ item.archived ? 'unarchive' : 'archive'
+ }`}
+ fullWidth
+ >
+ {item.archived ? 'Unarchive' : 'Archive'}
+ </LemonButton>
+ </div>
)}
</>
}
diff --git a/products/ai_observability/frontend/datasets/EditDatasetForm.tsx b/products/ai_observability/frontend/datasets/EditDatasetForm.tsx
index bd74b586d870..948f8ae799c2 100644
--- a/products/ai_observability/frontend/datasets/EditDatasetForm.tsx
+++ b/products/ai_observability/frontend/datasets/EditDatasetForm.tsx
@@ -27,6 +27,7 @@ export function EditDatasetForm(): JSX.Element {
<LemonField name="description" label="Description" showOptional>
<LemonTextArea
className="ph-ignore-input"
+ data-attr="dataset-description"
placeholder="Describe what this dataset contains"
value={datasetForm.description}
onChange={(value) => setDatasetFormValue('description', value)}
diff --git a/products/ai_observability/frontend/datasets/RefreshButton.tsx b/products/ai_observability/frontend/datasets/RefreshButton.tsx
index 94f9a08b8ae5..5287ed0429a2 100644
--- a/products/ai_observability/frontend/datasets/RefreshButton.tsx
+++ b/products/ai_observability/frontend/datasets/RefreshButton.tsx
@@ -13,6 +13,7 @@ export function RefreshButton({ onClick, isRefreshing }: RefreshButtonProps): JS
<div className="relative">
<LemonButton
onClick={onClick}
+ data-attr="dataset-refresh"
type="secondary"
icon={isRefreshing ? <Spinner textColored /> : <IconRefresh />}
size="small"
diff --git a/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx b/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx
index 35d2e1ab30da..b1ffab940910 100644
--- a/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx
+++ b/products/ai_observability/frontend/datasets/SaveToDatasetButton.tsx
@@ -68,9 +68,11 @@ export const SaveToDatasetButton = React.memo(function SaveToDatasetButton({
>
<LemonButton
type="secondary"
+ data-attr="save-to-dataset-open"
size="xsmall"
icon={<IconDatabase />}
sideAction={{
+ 'data-attr': 'save-to-dataset-edit',
icon: <IconPencil />,
onClick: () => {
if (!dropdownVisible) {
@@ -117,7 +119,7 @@ function OverlayMenu(): JSX.Element {
return (
<Form logic={saveToDatasetButtonLogic} formKey="searchForm" className="w-xs" enableFormOnSubmit>
<LemonField name="search" label="Search" labelClassName="sr-only">
- <LemonInput placeholder="Find a dataset" autoFocus />
+ <LemonInput placeholder="Find a dataset" autoFocus data-attr="save-to-dataset-search" />
</LemonField>
<LemonDivider className="my-0 mt-2" />
<div
@@ -206,6 +208,7 @@ function OverlayMenu(): JSX.Element {
fullWidth
size="small"
to={urls.aiObservabilityDataset('new')}
+ data-attr="save-to-dataset-create"
sideIcon={<IconExternal />}
targetBlank
onClick={() => {
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx
index a1a7adadb933..32cacab56bb3 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx
@@ -19,6 +19,7 @@ export function OfflineEvaluationsTabs({ activeTab }: { activeTab: OfflineEvalua
options={[
{
value: 'experiments',
+ 'data-attr': 'offline-evaluations-experiments-tab',
label: (
<span className="flex items-center gap-1">
<span>Experiments</span>
@@ -38,6 +39,7 @@ export function OfflineEvaluationsTabs({ activeTab }: { activeTab: OfflineEvalua
},
{
value: 'datasets',
+ 'data-attr': 'offline-evaluations-datasets-tab',
label: 'Datasets',
disabledReason: !datasetsEnabled
? 'Datasets are coming soon.'
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx
index 155fbd814181..e343266c8038 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentContent.tsx
@@ -42,7 +42,11 @@ export function OfflineExperimentContent(props: OfflineExperimentLogicProps): JS
return (
<div className="flex flex-col gap-4 min-w-0">
<div className="flex flex-wrap items-center justify-between gap-2">
- <LemonButton to={urls.aiObservabilityOfflineEvaluations()} icon={<IconArrowLeft />}>
+ <LemonButton
+ to={urls.aiObservabilityOfflineEvaluations()}
+ icon={<IconArrowLeft />}
+ data-attr="offline-experiment-back"
+ >
Experiments
</LemonButton>
<LemonButton
@@ -182,6 +186,7 @@ export function OfflineExperimentContent(props: OfflineExperimentLogicProps): JS
<Link
className="break-words"
to={offlineScorerHistoryUrl(summary.scorer, experiment, timezone)}
+ data-attr="offline-experiment-scorer-history"
>
<ScorerLabel scorer={summary.scorer} />
</Link>
@@ -225,7 +230,12 @@ export function OfflineExperimentContent(props: OfflineExperimentLogicProps): JS
title="Mark experiment as completed?"
footer={
<>
- <LemonButton type="secondary" onClick={hideCompletion} disabled={completedLoading}>
+ <LemonButton
+ type="secondary"
+ onClick={hideCompletion}
+ disabled={completedLoading}
+ data-attr="offline-experiment-cancel-complete"
+ >
Cancel
</LemonButton>
<LemonButton
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx
index 7d534ca1154f..bd35b3df7f09 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineItemInspector.tsx
@@ -73,7 +73,10 @@ export function OfflineItemInspector({
{item.trial && <span>{`Trial ${item.trial}`}</span>}
{item.model_version && <span>{`Model: ${item.model_version}`}</span>}
{item.application_trace_id && (
- <Link to={urls.aiObservabilityTrace(item.application_trace_id)}>
+ <Link
+ to={urls.aiObservabilityTrace(item.application_trace_id)}
+ data-attr="offline-item-application-trace"
+ >
Application trace
</Link>
)}
@@ -138,6 +141,7 @@ export function OfflineItemInspector({
to={urls.aiObservabilityTrace(
selectedResult.evaluator_trace_id
)}
+ data-attr="offline-result-evaluator-trace"
>
Evaluator trace
</Link>
@@ -219,6 +223,7 @@ export function OfflineItemInspector({
className="max-w-full"
active={result.id === selectedResult?.id}
onClick={() => onSelectResult(result)}
+ data-attr="offline-item-select-result"
>
<ScorerLabel scorer={result.scorer} />
</LemonButton>
@@ -247,6 +252,7 @@ export function OfflineItemInspector({
size="small"
disabled={!resultCursors.length || resultsLoading}
onClick={previousResults}
+ data-attr="offline-item-results-previous"
>
Previous
</LemonButton>
@@ -254,6 +260,7 @@ export function OfflineItemInspector({
size="small"
disabled={!!resultsError || !results?.next_cursor || resultsLoading}
onClick={() => results?.next_cursor && nextResults(results.next_cursor)}
+ data-attr="offline-item-results-next"
>
Next
</LemonButton>
@@ -263,7 +270,7 @@ export function OfflineItemInspector({
</div>
</LemonModalContent>
<LemonModalFooter>
- <LemonButton type="secondary" onClick={onClose}>
+ <LemonButton type="secondary" onClick={onClose} data-attr="offline-item-close">
Close
</LemonButton>
</LemonModalFooter>
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx
index da8ef6a12f21..dbb0a5c1e95d 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScoreChooser.tsx
@@ -32,7 +32,9 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
width={720}
footer={
<>
- <LemonButton onClick={closeChooser}>Cancel</LemonButton>
+ <LemonButton onClick={closeChooser} data-attr="offline-score-selection-cancel">
+ Cancel
+ </LemonButton>
<LemonButton type="primary" onClick={saveScorers} data-attr="offline-score-selection-save">
Save choices
</LemonButton>
@@ -48,6 +50,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
type="search"
value={scorerSearch}
onChange={setScorerSearch}
+ data-attr="offline-score-selection-search"
placeholder="Search scorers"
/>
<div className="h-48 @min-[32rem]:h-64 overflow-y-auto space-y-3">
@@ -65,6 +68,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
{scorerOptions.results.map((scorer) => (
<LemonCheckbox
key={scorer.id}
+ data-attr="offline-score-selection-toggle"
checked={draftScorerIds.includes(scorer.id)}
label={`${scorer.name} (${scorer.kind})${scorer.archived ? ' · Archived' : ''}`}
onChange={(checked) =>
@@ -91,6 +95,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
size="small"
disabledReason={scorerOffset === 0 ? 'First page' : undefined}
onClick={() => setScorerOffset(scorerOffset - 50)}
+ data-attr="offline-score-selection-previous"
>
Previous
</LemonButton>
@@ -100,6 +105,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
scorerOffset + 50 >= scorerOptions.count ? 'Last page' : undefined
}
onClick={() => setScorerOffset(scorerOffset + 50)}
+ data-attr="offline-score-selection-next"
>
Next
</LemonButton>
@@ -125,6 +131,7 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
size="xsmall"
disabledReason={index === 0 ? 'Already first' : undefined}
onClick={() => move(index, -1)}
+ data-attr="offline-score-selection-move-up"
/>
<LemonButton
icon={<IconChevronDown />}
@@ -134,10 +141,12 @@ export function OfflineScoreChooser(props: OfflineExperimentsLogicProps): JSX.El
index === draftScorerIds.length - 1 ? 'Already last' : undefined
}
onClick={() => move(index, 1)}
+ data-attr="offline-score-selection-move-down"
/>
<LemonButton
icon={<IconX />}
aria-label="Remove score"
+ data-attr="offline-score-selection-remove"
size="xsmall"
onClick={() =>
setDraftScorerIds(draftScorerIds.filter((selected) => selected !== id))
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
index 9c271fec0508..4079af7f9c89 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
@@ -61,7 +61,11 @@ export function OfflineScorerHistoryTable({
onClick={() => onToggleRun(key)}
/>
<Tooltip title={point.experiment.name}>
- <Link to={offlineExperimentUrl(point)} className="truncate min-w-0">
+ <Link
+ to={offlineExperimentUrl(point)}
+ className="truncate min-w-0"
+ data-attr="offline-history-open-experiment"
+ >
{point.experiment.name}
</Link>
</Tooltip>
diff --git a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx
index df3d7752dabc..e692c397ca6a 100644
--- a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx
+++ b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx
@@ -84,7 +84,9 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {
render: function renderName(_, scoreDefinition) {
return (
<div className="font-semibold max-w-64 @max-[50rem]/scorers:max-w-32 @max-[40rem]/scorers:max-w-24 truncate">
- <Link to={urls.aiObservabilityScorer(scoreDefinition.id)}>{scoreDefinition.name}</Link>
+ <Link to={urls.aiObservabilityScorer(scoreDefinition.id)} data-attr="llma-scorer-open">
+ {scoreDefinition.name}
+ </Link>
</div>
)
},
diff --git a/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx b/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx
index ae1a422c94c8..021f4c788df1 100644
--- a/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx
+++ b/products/ai_observability/frontend/scoreDefinitions/ScoreDefinitionActions.tsx
@@ -32,7 +32,13 @@ export function ScoreDefinitionActions({
'data-attr': 'llma-scorer-edit-config',
},
...(showHistory
- ? [{ label: 'Offline history', to: urls.aiObservabilityOfflineScorerHistory(definition.id) }]
+ ? [
+ {
+ label: 'Offline history',
+ to: urls.aiObservabilityOfflineScorerHistory(definition.id),
+ 'data-attr': 'llma-scorer-offline-history',
+ },
+ ]
: []),
{
label: 'Duplicate',
@@ -53,6 +59,7 @@ export function ScoreDefinitionActions({
icon={<IconEllipsis />}
size="small"
aria-label={`Actions for ${definition.name}`}
+ data-attr="llma-scorer-actions"
loading={archiving}
/>
</LemonMenu>
diff --git a/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts b/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts
index bdeb5d2bc1ac..f6969666c634 100644
--- a/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts
+++ b/products/ai_observability/frontend/scoreDefinitions/openCreateScorerVersionDialog.ts
@@ -10,7 +10,11 @@ export function openCreateScorerVersionDialog({
LemonDialog.open({
title: 'Create a new scorer version?',
content: `Version ${currentVersion + 1} will keep the current configuration. Use this when your external evaluator changes.`,
- primaryButton: { children: 'Create version', onClick: createVersion },
- secondaryButton: { children: 'Cancel' },
+ primaryButton: {
+ children: 'Create version',
+ onClick: createVersion,
+ 'data-attr': 'llma-scorer-confirm-new-version',
+ },
+ secondaryButton: { children: 'Cancel', 'data-attr': 'llma-scorer-cancel-new-version' },
})
}