Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): expire unused datasets and historical versions (#128324)

Radu Raicea57187cab0fc1
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 919779c2216d..30035c66eb90 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -492,6 +492,24 @@ The reporting integration uses the existing ledger schema without an additional
Temporal computes the offline ledger's exclusive day end inside the aggregation activity.
The shared workflow window and already serialized reports are unchanged.
+## Dataset retention
+
+Datasets expire after inactivity: 90 days for active datasets and their current revision, and 30 days for archived datasets and historical revisions. Creating a dataset starts its first retention period. Archiving or restoring a dataset restarts its period, and replacing the current revision starts a new 30-day period for that historical revision. Existing data gets a full period from the retention migration, rather than being judged by its creation date.
+
+Opening a dataset, reading its items through the UI or API, editing it, exporting it, and using it in an offline evaluation count as use. Reading a particular snapshot refreshes that revision and its dataset; it does not refresh unrelated historical revisions. Reading item version history preserves the returned item versions. Dataset lists, revision lists, and export status polling do not refresh usage.
+
+Usage is recorded in Postgres at most once per 12 hours per resource on read paths. Requests with a recent timestamp do not issue a usage write. Expiry includes an extra 12 hours so coalescing cannot shorten the promised inactivity period. Usage writes and cleanup lock the same dataset row; cleanup rechecks eligibility under that lock. Uploading experiments protect their referenced snapshots while snapshot use is within the 30-day history window, including the 12-hour margin. Uploads of new items or results refresh usage, as does closing an experiment. An abandoned upload cannot protect an unused snapshot indefinitely.
+
+The hourly cleanup marks expired datasets and snapshots unavailable before deleting item versions in bounded batches. A version is removable only when no retained snapshot needs it, it is not the item's current version, and it has not been read recently. Archive markers follow the same rule so pruning cannot make an archived item reappear. Expiring a whole dataset immediately releases its name for reuse and eventually removes all its item versions, items, and revisions. Names remain unique across active and archived datasets that have not expired. Completed experiments, evaluation items, scores, scorer versions, and usage ledger rows remain; optional dataset foreign keys become null in bounded batches while their provenance identifiers remain intact. Versions and revisions are deleted only after their references have been detached. Completed evaluation reads do not keep source datasets alive.
+
+A failed dataset cleanup rolls back its content changes but preserves its attempt timestamp, so later candidates can still be processed. The next sweep rotates past the failed dataset before retrying it. Task time limits still stop the sweep. Cleanup reports failed dataset counts and uses `completed_with_errors` when it finishes with individual failures.
+
+Dataset and revision responses expose `expires_at` and `expiring_soon`. The UI shows **Expiring soon** in the final seven days, replacing the Active or Archived status label, without an extension button. Normal use refreshes retention. The deadline is the earliest cleanup eligibility time, not a promise of immediate deletion. Cleanup emits a `dataset retention run` event with expiry and deletion counts, duration, completion status, and whether the time budget was exhausted.
+
+Staff can pause or resume cleanup under **Organization tools > AI observability dataset retention** in Django admin (`/admin/posthog/organization/ai-observability-dataset-retention/`). The `AI_OBSERVABILITY_DATASET_RETENTION_ENABLED` instance setting defaults to enabled and applies separately to each region. Disabling stops both new expiry and unfinished deletion; a running sweep checks the setting before each dataset and finishes its current dataset before stopping. Re-enabling resumes the hourly schedule. Disabling does not restore deleted content or make already expired datasets available. Successful expiry/deletion logs include the team and dataset IDs after the transaction commits, and setting changes log the staff user and old/new state.
+
+Candidate selection skips expired revisions that are retained only for shared versions or evaluation provenance, while continuing unfinished version and reference deletion. HogQL's `system.datasets` and `system.dataset_revisions` expose `expired_at`; queries must check the dataset and requested snapshot are available. Do not filter out a version solely because its introducing revision expired: retained snapshots can still need that version. HogQL queries and listing metadata do not refresh usage.
+
## Resource tags
Datasets, stable scorer definitions, and offline experiments use the shared project tag vocabulary.
diff --git a/posthog/admin/admins/organization_admin.py b/posthog/admin/admins/organization_admin.py
index 4c63758efe32..57b42a5da6fa 100644
--- a/posthog/admin/admins/organization_admin.py
+++ b/posthog/admin/admins/organization_admin.py
@@ -12,12 +12,16 @@ from django.core.exceptions import PermissionDenied
from django.core.management import call_command
from django.core.management.base import CommandError
from django.db import transaction
+from django.http import HttpRequest, HttpResponse
from django.shortcuts import redirect, render
from django.template.loader import render_to_string
+from django.template.response import TemplateResponse
from django.urls import path, reverse
from django.utils import timezone
+from django.utils.decorators import method_decorator
from django.utils.html import format_html
from django.utils.safestring import mark_safe
+from django.views.decorators.http import require_http_methods
from posthog.admin.ai_training_opt_in_history import get_ai_training_opt_in_history
from posthog.admin.authorization import can_trigger_admin_deletion
@@ -35,6 +39,7 @@ from posthog.person_db_router import PERSONS_DB_MODELS
from posthog.tasks.ai_observability_usage_report import internal_reporting_team_id
from posthog.utils import pluralize
+from products.ai_observability.backend.facade.admin import dataset_retention_enabled, set_dataset_retention_enabled
from products.batch_exports.backend.facade import api as batch_exports_api
from products.batch_exports.backend.facade.contracts import BATCH_EXPORT_MODEL_LABEL
@@ -133,6 +138,15 @@ class UsageReportForm(forms.Form):
return report_date
+class DatasetRetentionForm(forms.Form):
+ enabled = forms.TypedChoiceField(
+ label="Dataset cleanup",
+ choices=[("enabled", "Enabled"), ("disabled", "Disabled")],
+ coerce=lambda value: value == "enabled",
+ widget=forms.RadioSelect,
+ )
+
+
class BulkDeactivateOrganizationsForm(forms.Form):
CUSTOM_REASON = "__custom__"
MAX_ORGANIZATIONS = 100
@@ -624,6 +638,11 @@ class OrganizationAdmin(admin.ModelAdmin):
self.admin_site.admin_view(self.send_ai_observability_usage_report_view),
name="send-ai-observability-usage-report",
),
+ path(
+ "ai-observability-dataset-retention/",
+ self.admin_site.admin_view(self.dataset_retention_view),
+ name="ai-observability-dataset-retention",
+ ),
path(
"bulk-deactivate/",
self.admin_site.admin_view(self.bulk_deactivate_view),
@@ -822,6 +841,34 @@ class OrganizationAdmin(admin.ModelAdmin):
{"form": form, "can_skip_existing_reports": internal_reporting_team_id() is not None},
)
+ @method_decorator(require_http_methods(["GET", "POST"]))
+ def dataset_retention_view(self, request: HttpRequest) -> HttpResponse:
+ user_id = request.user.pk
+ if user_id is None or not request.user.has_perm("posthog.change_instancesetting"):
+ raise PermissionDenied
+ enabled = dataset_retention_enabled()
+ form = DatasetRetentionForm(
+ request.POST if request.method == "POST" else None,
+ initial={"enabled": "enabled" if enabled else "disabled"},
+ )
+ if request.method == "POST" and form.is_valid():
+ new_enabled = form.cleaned_data["enabled"]
+ set_dataset_retention_enabled(enabled=new_enabled, user_id=user_id)
+ messages.success(
+ request, f"Dataset cleanup is now {'enabled' if new_enabled else 'disabled'} in this region."
+ )
+ return redirect(reverse("admin:ai-observability-dataset-retention"))
+ return TemplateResponse(
+ request,
+ "admin/posthog/organization/ai_observability_dataset_retention.html",
+ {
+ **self.admin_site.each_context(request),
+ "title": "AI observability dataset retention",
+ "form": form,
+ "enabled": enabled,
+ },
+ )
+
def change_view(self, request, object_id, form_url="", extra_context=None):
extra_context = extra_context or {}
# Store request for access in display methods (needed for CSRF tokens in templates)
diff --git a/posthog/hogql/database/schema/system.py b/posthog/hogql/database/schema/system.py
index fe1f39049099..e60b098e8241 100644
--- a/posthog/hogql/database/schema/system.py
+++ b/posthog/hogql/database/schema/system.py
@@ -485,6 +485,9 @@ datasets: PostgresTable = PostgresTable(
"description": StringDatabaseField(name="description", description="What the dataset contains."),
"metadata": StringJSONDatabaseField(name="metadata", description="JSON dataset metadata."),
"archived": BooleanDatabaseField(name="archived", description="Whether the dataset is archived."),
+ "expired_at": DateTimeDatabaseField(
+ name="expired_at", nullable=True, description="When the dataset expired; NULL while available."
+ ),
"current_revision_id": UUIDDatabaseField(
name="current_revision_id",
nullable=True,
@@ -511,6 +514,11 @@ dataset_revisions: PostgresTable = PostgresTable(
"team_id": IntegerDatabaseField(name="team_id"),
"dataset_id": UUIDDatabaseField(name="dataset_id", description="Parent dataset; joins to datasets.id."),
"revision": IntegerDatabaseField(name="revision", description="Monotonic revision number within the dataset."),
+ "expired_at": DateTimeDatabaseField(
+ name="expired_at",
+ nullable=True,
+ description="When the snapshot expired; NULL while available. Expired snapshots may be incomplete.",
+ ),
"created_by_id": IntegerDatabaseField(
name="created_by_id", nullable=True, description="User who committed the revision."
),
diff --git a/posthog/hogql/database/test/__snapshots__/test_database.ambr b/posthog/hogql/database/test/__snapshots__/test_database.ambr
index 3e941711b2f6..ace6a1233b38 100644
--- a/posthog/hogql/database/test/__snapshots__/test_database.ambr
+++ b/posthog/hogql/database/test/__snapshots__/test_database.ambr
@@ -5229,6 +5229,16 @@
"table": null,
"type": "integer"
},
+ "expired_at": {
+ "chain": null,
+ "fields": null,
+ "hogql_value": "expired_at",
+ "id": null,
+ "name": "expired_at",
+ "schema_valid": true,
+ "table": null,
+ "type": "datetime"
+ },
"created_by_id": {
"chain": null,
"fields": null,
@@ -5308,6 +5318,16 @@
"table": null,
"type": "boolean"
},
+ "expired_at": {
+ "chain": null,
+ "fields": null,
+ "hogql_value": "expired_at",
+ "id": null,
+ "name": "expired_at",
+ "schema_valid": true,
+ "table": null,
+ "type": "datetime"
+ },
"current_revision_id": {
"chain": null,
"fields": null,
@@ -20185,6 +20205,16 @@
"table": null,
"type": "integer"
},
+ "expired_at": {
+ "chain": null,
+ "fields": null,
+ "hogql_value": "expired_at",
+ "id": null,
+ "name": "expired_at",
+ "schema_valid": true,
+ "table": null,
+ "type": "datetime"
+ },
"created_by_id": {
"chain": null,
"fields": null,
@@ -20264,6 +20294,16 @@
"table": null,
"type": "boolean"
},
+ "expired_at": {
+ "chain": null,
+ "fields": null,
+ "hogql_value": "expired_at",
+ "id": null,
+ "name": "expired_at",
+ "schema_valid": true,
+ "table": null,
+ "type": "datetime"
+ },
"current_revision_id": {
"chain": null,
"fields": null,
diff --git a/posthog/settings/dynamic_settings.py b/posthog/settings/dynamic_settings.py
index a888f7ceea4b..e90dda2579f3 100644
--- a/posthog/settings/dynamic_settings.py
+++ b/posthog/settings/dynamic_settings.py
@@ -9,6 +9,11 @@ CONSTANCE_DATABASE_PREFIX = "constance:posthog:"
# To edit, visit: ${SITE_URL}/admin/posthog/instancesetting/
CONSTANCE_CONFIG = {
+ "AI_OBSERVABILITY_DATASET_RETENTION_ENABLED": (
+ True,
+ "Enable expiry and deletion of unused AI observability datasets and historical versions in this region.",
+ bool,
+ ),
"WAREHOUSE_PERSON_PROPERTY_SET_RATE_PER_SEC": (
5000,
"Global max rate (events/sec) at which the warehouse person-property consumer sends $set "
diff --git a/posthog/tasks/scheduled.py b/posthog/tasks/scheduled.py
index c29d83d8c339..4f293b658cdd 100644
--- a/posthog/tasks/scheduled.py
+++ b/posthog/tasks/scheduled.py
@@ -78,7 +78,7 @@ from posthog.utils import get_crontab, get_instance_region
from products.account_audits.backend.facade.tasks import reconcile_account_audits
from products.aeo.backend.facade.tasks import run_aeo_citation_checks_task
-from products.ai_observability.backend.facade.tasks import expire_offline_evaluation_payloads
+from products.ai_observability.backend.facade.tasks import expire_datasets, expire_offline_evaluation_payloads
from products.ai_training.backend.facade.api import privacy_enabled
from products.ai_training.backend.facade.tasks import process_ai_training_privacy_requests
from products.approvals.backend.tasks import (
@@ -276,6 +276,7 @@ def add_periodic_task_with_expiry(
def setup_periodic_tasks(sender: Celery, **kwargs: Any) -> None:
+ add_periodic_task_with_expiry(sender, crontab(minute="37"), expire_datasets.s(), name="expire unused datasets")
add_periodic_task_with_expiry(
sender,
crontab(minute="23"),
diff --git a/posthog/templates/organization_tools/app.html b/posthog/templates/organization_tools/app.html
index 5177ed076cf5..6afd39c13333 100644
--- a/posthog/templates/organization_tools/app.html
+++ b/posthog/templates/organization_tools/app.html
@@ -26,6 +26,14 @@
<td>Deactivate organizations from pasted IDs after a preview step</td>
</tr>
{% endif %}
+ {% if perms.posthog.change_instancesetting %}
+ <tr class="model-aiobservabilitydatasetretention">
+ <th scope="row">
+ <a href="{% url 'admin:ai-observability-dataset-retention' %}">AI observability dataset retention</a>
+ </th>
+ <td>Enable or disable automatic dataset cleanup in this region</td>
+ </tr>
+ {% endif %}
</table>
</div>
{% endif %}
diff --git a/products/ai_observability/backend/admin.py b/products/ai_observability/backend/admin.py
new file mode 100644
index 000000000000..9743f2ec14d8
--- /dev/null
+++ b/products/ai_observability/backend/admin.py
@@ -0,0 +1,15 @@
+import structlog
+
+from posthog.models.instance_setting import set_instance_setting
+
+from products.ai_observability.backend.dataset_retention import RETENTION_ENABLED_SETTING, dataset_retention_enabled
+
+logger = structlog.get_logger(__name__)
+
+
+def set_dataset_retention_enabled(*, enabled: bool, user_id: int) -> None:
+ previous_enabled = dataset_retention_enabled()
+ set_instance_setting(RETENTION_ENABLED_SETTING, enabled)
+ logger.info(
+ "dataset_retention_setting_changed", user_id=user_id, previous_enabled=previous_enabled, enabled=enabled
+ )
diff --git a/products/ai_observability/backend/api/dataset_exports.py b/products/ai_observability/backend/api/dataset_exports.py
index 289b4ef6cad6..2492e525f031 100644
--- a/products/ai_observability/backend/api/dataset_exports.py
+++ b/products/ai_observability/backend/api/dataset_exports.py
@@ -11,7 +11,7 @@ from posthog.models import Team, User
from posthog.models.activity_logging.activity_log import Change, Detail, log_activity
from products.ai_observability.backend.api.dataset_serializers import StrictDatasetSerializer
-from products.ai_observability.backend.dataset_queries import latest_dataset_revision
+from products.ai_observability.backend.dataset_retention import DatasetUsage
from products.ai_observability.backend.models.datasets import Dataset, DatasetRevision
from products.exports.backend.facade.api import (
DATASET_EXPORT_KIND,
@@ -83,21 +83,13 @@ def create_dataset_export(
was_impersonated: bool,
revision: int | None,
) -> ExportedAsset:
- selected_revision = revision
- if selected_revision is None:
- latest_revision = latest_dataset_revision(team_id=team.id, dataset_id=dataset.id)
- if latest_revision is None:
- raise DatasetExportUnavailableError("Add an item before exporting this dataset.")
- selected_revision = latest_revision.revision
- elif (
- not DatasetRevision.objects.for_team(team.id, canonical=True)
- .filter(
- dataset_id=dataset.id,
- revision=selected_revision,
- )
- .exists()
- ):
- raise DatasetExportUnavailableError("This dataset revision does not exist.")
+ try:
+ selected = DatasetUsage.read_revision(dataset, revision)
+ except (Dataset.DoesNotExist, DatasetRevision.DoesNotExist) as error:
+ raise DatasetExportUnavailableError("This dataset revision is no longer available.") from error
+ if selected is None:
+ raise DatasetExportUnavailableError("Add an item before exporting this dataset.")
+ selected_revision = selected.revision
asset = create_export_asset_async(
team=team,
diff --git a/products/ai_observability/backend/api/datasets.py b/products/ai_observability/backend/api/datasets.py
index befbc29f9154..cfcb8ce547f8 100644
--- a/products/ai_observability/backend/api/datasets.py
+++ b/products/ai_observability/backend/api/datasets.py
@@ -1,3 +1,4 @@
+from datetime import datetime
from functools import partial
from typing import NoReturn, cast
from uuid import UUID
@@ -62,6 +63,7 @@ from products.ai_observability.backend.dataset_queries import (
consistent_dataset_item_versions,
dataset_item_versions_at_revision,
)
+from products.ai_observability.backend.dataset_retention import DatasetRetentionPolicy, DatasetUsage
from products.ai_observability.backend.dataset_service import (
UNSET,
DatasetItemMutationResult,
@@ -199,6 +201,8 @@ def _resolved_dataset_revision_id(dataset: Dataset) -> UUID | None:
class DatasetReadSerializer(ResourceTagsReadSerializer, UserAccessControlSerializerMixin, serializers.ModelSerializer):
+ expires_at = serializers.SerializerMethodField(help_text="Earliest cleanup time if the dataset remains unused.")
+ expiring_soon = serializers.SerializerMethodField(help_text="Whether the dataset is within seven days of expiry.")
created_by = UserBasicSerializer(read_only=True, allow_null=True)
metadata = JSONObjectField(read_only=True, help_text="JSON object with descriptive dataset metadata.")
current_revision = serializers.SerializerMethodField(
@@ -218,6 +222,8 @@ class DatasetReadSerializer(ResourceTagsReadSerializer, UserAccessControlSeriali
"metadata",
"tags",
"archived",
+ "expires_at",
+ "expiring_soon",
"current_revision",
"current_revision_id",
"created_at",
@@ -228,6 +234,14 @@ class DatasetReadSerializer(ResourceTagsReadSerializer, UserAccessControlSeriali
]
read_only_fields = fields
+ @extend_schema_field(serializers.DateTimeField(allow_null=True))
+ def get_expires_at(self, instance: Dataset) -> datetime | None:
+ return DatasetRetentionPolicy.dataset_expires_at(instance)
+
+ @extend_schema_field(serializers.BooleanField())
+ def get_expiring_soon(self, instance: Dataset) -> bool:
+ return DatasetRetentionPolicy.expiring_soon(self.get_expires_at(instance))
+
@extend_schema_field(serializers.IntegerField(allow_null=True))
def get_current_revision(self, instance: Dataset) -> int | None:
return _resolved_dataset_revision(instance)
@@ -305,15 +319,25 @@ class DatasetListQuerySerializer(ResourceTagFilterSerializer):
class DatasetRevisionReadSerializer(serializers.ModelSerializer):
+ expires_at = serializers.SerializerMethodField(help_text="Earliest expiry time of this dataset snapshot if unused.")
+ expiring_soon = serializers.SerializerMethodField(help_text="Whether this snapshot is within seven days of expiry.")
created_by = UserBasicSerializer(read_only=True, allow_null=True)
dataset_id = serializers.UUIDField(read_only=True, help_text="Dataset this revision belongs to.")
team_id = serializers.IntegerField(read_only=True, help_text="Project that owns the revision.")
class Meta:
model = DatasetRevision
- fields = ["id", "dataset_id", "revision", "created_at", "created_by", "team_id"]
+ fields = ["id", "dataset_id", "revision", "created_at", "created_by", "team_id", "expires_at", "expiring_soon"]
read_only_fields = fields
+ @extend_schema_field(serializers.DateTimeField(allow_null=True))
+ def get_expires_at(self, instance: DatasetRevision) -> datetime | None:
+ return DatasetRetentionPolicy.revision_expires_at(instance)
+
+ @extend_schema_field(serializers.BooleanField())
+ def get_expiring_soon(self, instance: DatasetRevision) -> bool:
+ return DatasetRetentionPolicy.expiring_soon(self.get_expires_at(instance))
+
class DatasetItemReadSerializer(serializers.ModelSerializer):
id = serializers.UUIDField(
@@ -595,7 +619,9 @@ def _conflict_response(error: DatasetMutationConflict | DatasetLimitExceeded) ->
def _item_version_queryset() -> QuerySet[DatasetItemVersion, DatasetItemVersion]:
return consistent_dataset_item_versions(
- DatasetItemVersion.objects.unscoped().select_related(
+ DatasetItemVersion.objects.unscoped()
+ .filter(dataset__expired_at__isnull=True)
+ .select_related(
"created_by",
"dataset",
"dataset_revision",
@@ -609,6 +635,7 @@ def _dataset_queryset() -> QuerySet[Dataset, Dataset]:
latest_revision = DatasetRevision.objects.unscoped().filter(dataset_id=OuterRef("id")).order_by("-revision")
return (
Dataset.objects.unscoped()
+ .filter(expired_at__isnull=True)
.select_related("created_by", "current_revision")
.prefetch_related(ResourceTags.prefetch())
.annotate(
@@ -743,7 +770,12 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
@llma_track_latency("llma_datasets_retrieve")
@monitor(feature=None, endpoint="llma_datasets_retrieve", method="GET")
def retrieve(self, request: Request, *args: object, **kwargs: object) -> Response:
- return Response(DatasetReadSerializer(self.get_object(), context=self.get_serializer_context()).data)
+ dataset = self.get_object()
+ try:
+ DatasetUsage.record(dataset)
+ except Dataset.DoesNotExist as error:
+ raise Http404("Dataset not found.") from error
+ return Response(DatasetReadSerializer(dataset, context=self.get_serializer_context()).data)
@extend_schema(
request=DatasetCreateSerializer,
@@ -869,8 +901,8 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
dataset = self.get_object()
queryset = (
DatasetRevision.objects.for_team(self.team.id, canonical=True)
- .filter(dataset_id=dataset.id)
- .select_related("created_by")
+ .filter(dataset_id=dataset.id, expired_at__isnull=True)
+ .select_related("created_by", "dataset")
.order_by("-revision")
)
page = self.paginate_queryset(queryset)
@@ -974,6 +1006,10 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
{"detail": "The export is still being prepared."},
status=status.HTTP_409_CONFLICT,
)
+ try:
+ DatasetUsage.read_revision(dataset, int((asset.export_context or {})["dataset_revision"]))
+ except (Dataset.DoesNotExist, DatasetRevision.DoesNotExist) as error:
+ raise Http404("Dataset revision not found.") from error
response = get_export_asset_content_response(asset=asset, download=True)
self._report_action(
"llma dataset export downloaded",
@@ -1100,22 +1136,16 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
archived: bool = query["archived"]
revision: int | None = query.get("revision")
+ try:
+ DatasetUsage.read_revision(dataset, revision)
+ except (Dataset.DoesNotExist, DatasetRevision.DoesNotExist) as error:
+ raise Http404("Dataset revision not found.") from error
if revision is None:
queryset = self.get_queryset().filter(
dataset_id=dataset.id,
archived=archived,
)
else:
- if (
- not DatasetRevision.objects.for_team(self.team.id, canonical=True)
- .filter(
- dataset_id=dataset.id,
- revision=revision,
- )
- .exists()
- ):
- raise Http404("Dataset revision not found.")
-
version_ids = dataset_item_versions_at_revision(
team_id=self.team.id,
dataset_id=dataset.id,
@@ -1145,20 +1175,14 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
query_serializer.is_valid(raise_exception=True)
revision: int | None = query_serializer.validated_data.get("revision")
current = self.get_object()
+ try:
+ DatasetUsage.read_revision(current.dataset, revision)
+ except (Dataset.DoesNotExist, DatasetRevision.DoesNotExist) as error:
+ raise Http404("Dataset revision not found.") from error
if revision is None:
return Response(DatasetItemReadSerializer(current).data)
dataset = current.dataset
- if (
- not DatasetRevision.objects.for_team(self.team.id, canonical=True)
- .filter(
- dataset_id=dataset.id,
- revision=revision,
- )
- .exists()
- ):
- raise Http404("Dataset revision not found.")
-
version = (
_item_version_queryset()
.filter(
@@ -1326,6 +1350,11 @@ class DatasetItemViewSet(TeamAndOrgViewSetMixin, GenericViewSet):
dataset_item_id=current.dataset_item_id,
)
page = self.paginate_queryset(queryset)
+ versions = page if page is not None else list(queryset)
+ try:
+ DatasetUsage.record_versions(current.dataset, versions)
+ except (Dataset.DoesNotExist, DatasetItemVersion.DoesNotExist) as error:
+ raise Http404("Dataset item version not found.") from error
if page is not None:
- return self.get_paginated_response(DatasetItemReadSerializer(page, many=True).data)
- return Response(DatasetItemReadSerializer(queryset, many=True).data)
+ return self.get_paginated_response(DatasetItemReadSerializer(versions, many=True).data)
+ return Response(DatasetItemReadSerializer(versions, many=True).data)
diff --git a/products/ai_observability/backend/api/test/test_datasets.py b/products/ai_observability/backend/api/test/test_datasets.py
index 29df1fa75cf9..ce969493b1fa 100644
--- a/products/ai_observability/backend/api/test/test_datasets.py
+++ b/products/ai_observability/backend/api/test/test_datasets.py
@@ -22,7 +22,7 @@ from posthog.models.utils import generate_random_token_personal, hash_key_value
from products.ai_observability.backend.api.dataset_exports import DATASET_EXPORT_STUCK_MESSAGE
from products.ai_observability.backend.dataset_service import create_dataset
from products.ai_observability.backend.models.datasets import Dataset, DatasetItem, DatasetItemVersion, DatasetRevision
-from products.exports.backend.facade.api import DATASET_EXPORT_KIND, ExportedAsset
+from products.exports.backend.facade.api import DATASET_EXPORT_KIND, JSONL_EXPORT_FORMAT, ExportedAsset
class TestDatasetsApi(APIBaseTest):
@@ -68,6 +68,73 @@ class TestDatasetsApi(APIBaseTest):
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
+ def test_lists_do_not_refresh_usage_but_snapshot_reads_do(self) -> None:
+ dataset = self._create_dataset()
+ first = self._create_item(dataset["id"])
+ self._create_item(dataset["id"], input="Second item")
+ old = now() - timedelta(days=27)
+ datasets = Dataset.objects.for_team(self.team.id, canonical=True).filter(id=dataset["id"])
+ revisions = DatasetRevision.objects.for_team(self.team.id, canonical=True).filter(dataset_id=dataset["id"])
+ datasets.update(last_used_at=old, archived=True)
+ revisions.update(last_used_at=old)
+
+ response = self.client.get(self.datasets_url, {"archived": "true"})
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ self.assertTrue(response.data["results"][0]["expiring_soon"])
+ response = self.client.get(f"{self.datasets_url}{dataset['id']}/revisions/")
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ self.assertTrue(all(revision["expiring_soon"] for revision in response.data["results"]))
+ self.assertEqual(datasets.get().last_used_at, old)
+ self.assertTrue(all(revision.last_used_at == old for revision in revisions))
+
+ response = self.client.get(self.items_url, {"dataset": dataset["id"], "revision": first["dataset_revision"]})
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ last_used_at = datasets.get().last_used_at
+ revision_last_used_at = revisions.get(revision=first["dataset_revision"]).last_used_at
+ assert last_used_at is not None
+ assert revision_last_used_at is not None
+ self.assertGreater(last_used_at, old)
+ self.assertGreater(revision_last_used_at, old)
+ self.assertEqual(revisions.get(revision=2).last_used_at, old)
+
+ def test_expired_snapshots_and_datasets_are_unavailable(self) -> None:
+ dataset = self._create_dataset()
+ first = self._create_item(dataset["id"])
+ self._create_item(dataset["id"], input="Second item")
+ asset = ExportedAsset.objects.create(
+ team=self.team,
+ created_by=self.user,
+ export_format=JSONL_EXPORT_FORMAT,
+ export_context={"kind": DATASET_EXPORT_KIND, "dataset_id": dataset["id"], "dataset_revision": 1},
+ content=b'{"input":"Expired snapshot"}\n',
+ )
+ DatasetRevision.objects.for_team(self.team.id, canonical=True).filter(id=first["dataset_revision_id"]).update(
+ expired_at=now()
+ )
+ for url, query in [
+ (self.items_url, {"dataset": dataset["id"], "revision": 1}),
+ (f"{self.items_url}{first['id']}/", {"revision": 1}),
+ (f"{self.datasets_url}{dataset['id']}/exports/{asset.id}/content/", {}),
+ ]:
+ self.assertEqual(self.client.get(url, query).status_code, status.HTTP_404_NOT_FOUND)
+ revisions = self.client.get(f"{self.datasets_url}{dataset['id']}/revisions/")
+ self.assertEqual([revision["revision"] for revision in revisions.data["results"]], [2])
+ Dataset.objects.for_team(self.team.id, canonical=True).filter(id=dataset["id"]).update(expired_at=now())
+ self.assertEqual(self.client.get(f"{self.datasets_url}{dataset['id']}/").status_code, status.HTTP_404_NOT_FOUND)
+ self.assertEqual(
+ self.client.get(self.items_url, {"dataset": dataset["id"]}).status_code, status.HTTP_404_NOT_FOUND
+ )
+ self.assertEqual(self.client.get(f"{self.items_url}{first['id']}/").status_code, status.HTTP_404_NOT_FOUND)
+
+ def test_dataset_detail_renews_usage_and_hides_expiring_soon(self) -> None:
+ dataset = self._create_dataset()
+ Dataset.objects.for_team(self.team.id, canonical=True).filter(id=dataset["id"]).update(
+ last_used_at=now() - timedelta(days=88)
+ )
+ response = self.client.get(f"{self.datasets_url}{dataset['id']}/")
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ self.assertFalse(response.data["expiring_soon"])
+
@patch("products.ai_observability.backend.api.datasets.report_user_action")
def test_dataset_lifecycle_and_name_uniqueness(self, report_action: Mock) -> None:
with self.captureOnCommitCallbacks(execute=True):
diff --git a/products/ai_observability/backend/dataset_export.py b/products/ai_observability/backend/dataset_export.py
index f10d3d8e4c83..a6cbe0a50864 100644
--- a/products/ai_observability/backend/dataset_export.py
+++ b/products/ai_observability/backend/dataset_export.py
@@ -14,6 +14,7 @@ from posthog.permissions import posthog_feature_flag_value
from products.access_control.backend.facade.user_access_control import UserAccessControl
from products.ai_observability.backend.dataset_limits import MAX_DATASET_EXPORT_BYTES, MAX_DATASET_EXPORT_MEGABYTES
from products.ai_observability.backend.dataset_queries import dataset_item_versions_at_revision
+from products.ai_observability.backend.dataset_retention import DatasetUsage
from products.ai_observability.backend.models.datasets import Dataset, DatasetItemVersion, DatasetRevision
from products.exports.backend.facade.api import (
ExportedAsset,
@@ -75,6 +76,16 @@ def _iter_versions_in_batches(
last_item_id = last_version.dataset_item_id
+def _read_export_revision(dataset: Dataset, revision: int) -> DatasetRevision:
+ try:
+ selected_revision = DatasetUsage.read_revision(dataset, revision)
+ except (Dataset.DoesNotExist, DatasetRevision.DoesNotExist) as error:
+ raise DatasetExportError("The selected dataset revision is no longer available.") from error
+ if selected_revision is None:
+ raise DatasetExportError("The selected dataset revision is no longer available.")
+ return selected_revision
+
+
def export_dataset_jsonl(asset: ExportedAsset) -> None:
context = asset.export_context or {}
try:
@@ -86,7 +97,9 @@ def export_dataset_jsonl(asset: ExportedAsset) -> None:
if not asset.is_dataset_export or revision < 1 or asset.created_by is None:
raise DatasetExportError("The dataset export configuration is invalid.", reportable=True)
- dataset = Dataset.objects.for_team(asset.team_id, canonical=True).filter(id=dataset_id).first()
+ dataset = (
+ Dataset.objects.for_team(asset.team_id, canonical=True).filter(id=dataset_id, expired_at__isnull=True).first()
+ )
if dataset is None:
raise DatasetExportError("The dataset is no longer available.")
if not UserAccessControl(user=asset.created_by, team=asset.team).check_access_level_for_object(dataset, "viewer"):
@@ -101,16 +114,7 @@ def export_dataset_jsonl(asset: ExportedAsset) -> None:
raise RetryableExportError("Couldn't verify whether dataset exports are available. Try again.")
if not feature_flag_value:
raise DatasetExportError("Dataset exports are not available for this project.")
- selected_revision = (
- DatasetRevision.objects.for_team(asset.team_id, canonical=True)
- .filter(
- dataset_id=dataset.id,
- revision=revision,
- )
- .first()
- )
- if selected_revision is None:
- raise DatasetExportError("The selected dataset revision is no longer available.")
+ selected_revision = _read_export_revision(dataset, revision)
versions = (
dataset_item_versions_at_revision(
diff --git a/products/ai_observability/backend/dataset_queries.py b/products/ai_observability/backend/dataset_queries.py
index 902f8a77078e..ac1032089a55 100644
--- a/products/ai_observability/backend/dataset_queries.py
+++ b/products/ai_observability/backend/dataset_queries.py
@@ -20,7 +20,7 @@ def consistent_dataset_item_versions(
def latest_dataset_revision(*, team_id: int, dataset_id: UUID) -> DatasetRevision | None:
return (
DatasetRevision.objects.for_team(team_id, canonical=True)
- .filter(dataset_id=dataset_id)
+ .filter(dataset_id=dataset_id, expired_at__isnull=True, dataset__expired_at__isnull=True)
.order_by("-revision")
.first()
)
diff --git a/products/ai_observability/backend/dataset_retention.py b/products/ai_observability/backend/dataset_retention.py
new file mode 100644
index 000000000000..0f42ec1b41e1
--- /dev/null
+++ b/products/ai_observability/backend/dataset_retention.py
@@ -0,0 +1,379 @@
+import time
+from collections.abc import Sequence
+from datetime import datetime, timedelta
+from functools import partial
+from uuid import UUID
+
+from django.db import transaction
+from django.db.models import Exists, F, OuterRef, Q, QuerySet, Subquery
+from django.db.models.functions import Coalesce
+from django.utils import timezone
+
+import structlog
+from celery.exceptions import SoftTimeLimitExceeded
+
+from posthog.models.instance_setting import get_instance_settings
+
+from products.ai_observability.backend.models.datasets import Dataset, DatasetItem, DatasetItemVersion, DatasetRevision
+from products.ai_observability.backend.models.offline_evaluations import OfflineExperiment, OfflineExperimentItem
+
+logger = structlog.get_logger(__name__)
+
+ACTIVE_RETENTION = timedelta(days=90)
+HISTORY_RETENTION = timedelta(days=30)
+USAGE_REFRESH_INTERVAL = timedelta(hours=12)
+EXPIRING_SOON_WINDOW = timedelta(days=7)
+BATCH_SIZE = 500
+MAX_DATASETS_PER_RUN = 100
+TIME_BUDGET_SECONDS = 180
+RETENTION_ENABLED_SETTING = "AI_OBSERVABILITY_DATASET_RETENTION_ENABLED"
+
+
+def dataset_retention_enabled() -> bool:
+ return get_instance_settings([RETENTION_ENABLED_SETTING])[RETENTION_ENABLED_SETTING] is True
+
+
+class DatasetRetentionPolicy:
+ @staticmethod
+ def dataset_expires_at(dataset: Dataset) -> datetime | None:
+ if dataset.last_used_at is None:
+ return None
+ return (
+ dataset.last_used_at
+ + (HISTORY_RETENTION if dataset.archived else ACTIVE_RETENTION)
+ + USAGE_REFRESH_INTERVAL
+ )
+
+ @staticmethod
+ def revision_expires_at(revision: DatasetRevision) -> datetime | None:
+ dataset = revision.dataset
+ if dataset.current_revision_id == revision.id:
+ return DatasetRetentionPolicy.dataset_expires_at(dataset)
+ if revision.last_used_at is None:
+ return None
+ expires_at = revision.last_used_at + HISTORY_RETENTION + USAGE_REFRESH_INTERVAL
+ dataset_expires_at = DatasetRetentionPolicy.dataset_expires_at(dataset)
+ return min(expires_at, dataset_expires_at) if dataset_expires_at else expires_at
+
+ @staticmethod
+ def expiring_soon(expires_at: datetime | None) -> bool:
+ return expires_at is not None and expires_at <= timezone.now() + EXPIRING_SOON_WINDOW
+
+
+class DatasetUsage:
+ @staticmethod
+ def record(dataset: Dataset, revision: DatasetRevision | None = None) -> None:
+ if dataset.expired_at is not None:
+ raise Dataset.DoesNotExist
+ if revision is not None and (revision.expired_at is not None or revision.dataset_id != dataset.id):
+ raise DatasetRevision.DoesNotExist
+ now = timezone.now()
+ cutoff = now - USAGE_REFRESH_INTERVAL
+ if dataset.last_used_at is not None and dataset.last_used_at > cutoff:
+ if revision is None or (revision.last_used_at is not None and revision.last_used_at > cutoff):
+ return
+ # Usage and cleanup serialize on the dataset, including accesses to its older revisions.
+ with transaction.atomic():
+ locked = (
+ Dataset.objects.for_team(dataset.team_id, canonical=True)
+ .select_for_update()
+ .get(id=dataset.id, expired_at__isnull=True)
+ )
+ if revision is not None:
+ revisions = DatasetRevision.objects.for_team(dataset.team_id, canonical=True).filter(
+ id=revision.id, dataset_id=dataset.id, expired_at__isnull=True
+ )
+ if not revisions.exists():
+ raise DatasetRevision.DoesNotExist
+ if revisions.filter(Q(last_used_at__isnull=True) | Q(last_used_at__lte=cutoff)).update(
+ last_used_at=now
+ ):
+ revision.last_used_at = now
+ else:
+ revision.last_used_at = revisions.values_list("last_used_at", flat=True).get()
+ if locked.last_used_at is None or locked.last_used_at <= cutoff:
+ Dataset.objects.for_team(dataset.team_id, canonical=True).filter(id=dataset.id).update(last_used_at=now)
+ locked.last_used_at = now
+ dataset.last_used_at = locked.last_used_at
+
+ @staticmethod
+ def record_versions(dataset: Dataset, versions: Sequence[DatasetItemVersion]) -> None:
+ now = timezone.now()
+ cutoff = now - USAGE_REFRESH_INTERVAL
+ stale_ids = [v.id for v in versions if v.last_used_at is None or v.last_used_at <= cutoff]
+ DatasetUsage.record(dataset)
+ if not stale_ids:
+ return
+ with transaction.atomic():
+ Dataset.objects.for_team(dataset.team_id, canonical=True).select_for_update().get(
+ id=dataset.id, expired_at__isnull=True
+ )
+ queryset = DatasetItemVersion.objects.for_team(dataset.team_id, canonical=True).filter(
+ dataset_id=dataset.id, id__in=stale_ids
+ )
+ if queryset.count() != len(stale_ids):
+ raise DatasetItemVersion.DoesNotExist
+ queryset.filter(Q(last_used_at__isnull=True) | Q(last_used_at__lte=cutoff)).update(last_used_at=now)
+
+ @staticmethod
+ def read_revision(dataset: Dataset, number: int | None = None) -> DatasetRevision | None:
+ revisions = DatasetRevision.objects.for_team(dataset.team_id, canonical=True).filter(
+ dataset_id=dataset.id, expired_at__isnull=True
+ )
+ revision: DatasetRevision | None
+ if number is not None:
+ revision = revisions.get(revision=number)
+ elif dataset.current_revision_id is not None:
+ revision = revisions.get(id=dataset.current_revision_id)
+ else:
+ revision = revisions.order_by("-revision").first()
+ DatasetUsage.record(dataset, revision)
+ return revision
+
+
+class DatasetRetention:
+ def __init__(self) -> None:
+ self.datasets_expired = 0
+ self.revisions_expired = 0
+ self.versions_deleted = 0
+ self.datasets_deleted = 0
+ self.datasets_failed = 0
+ self.budget_exhausted = False
+ self.disabled = False
+
+ @staticmethod
+ def _uploading_revisions(team_id: int) -> QuerySet[OfflineExperiment]:
+ cutoff = timezone.now() - HISTORY_RETENTION - USAGE_REFRESH_INTERVAL
+ return (
+ OfflineExperiment.objects.for_team(team_id)
+ .filter(status=OfflineExperiment.Status.UPLOADING)
+ .filter(Q(dataset_revision__last_used_at__isnull=True) | Q(dataset_revision__last_used_at__gt=cutoff))
+ )
+
+ @staticmethod
+ def _unused_versions(
+ versions: QuerySet[DatasetItemVersion], revisions: QuerySet[DatasetRevision], cutoff: datetime
+ ) -> QuerySet[DatasetItemVersion]:
+ next_revision = (
+ versions.filter(dataset_item_id=OuterRef("dataset_item_id"), version__gt=OuterRef("version"))
+ .order_by("version")
+ .values("dataset_revision__revision")[:1]
+ )
+ retained_revision = revisions.filter(
+ dataset_id=OuterRef("dataset_id"),
+ expired_at__isnull=True,
+ revision__gte=OuterRef("dataset_revision__revision"),
+ revision__lt=OuterRef("next_revision_number"),
+ )
+ # The next version bounds the snapshots that need this one, including archive markers.
+ return (
+ versions.filter(last_used_at__lte=cutoff)
+ .exclude(id=F("dataset_item__current_version_id"))
+ .annotate(next_revision_number=Coalesce(Subquery(next_revision), 2_147_483_647))
+ .filter(~Exists(retained_revision))
+ )
+
+ def _delete_versions(self, dataset: Dataset, ids: list[UUID]) -> None:
+ references = OfflineExperimentItem.objects.for_team(dataset.team_id).filter(dataset_item_version_id__in=ids)
+ reference_ids = list(references.order_by("id").values_list("id", flat=True)[:BATCH_SIZE])
+ references.filter(id__in=reference_ids).update(dataset_item_version=None)
+ versions = DatasetItemVersion.objects.for_team(dataset.team_id, canonical=True).filter(
+ dataset_id=dataset.id, id__in=ids
+ )
+ removable_ids = list(
+ versions.filter(~Exists(references.filter(dataset_item_version_id=OuterRef("id")))).values_list(
+ "id", flat=True
+ )
+ )
+ if dataset.expired_at is not None:
+ DatasetItem.objects.for_team(dataset.team_id, canonical=True).filter(
+ dataset_id=dataset.id, current_version_id__in=removable_ids
+ ).update(current_version=None)
+ versions.filter(id__in=removable_ids).only("id").delete()
+ self.versions_deleted += len(removable_ids)
+
+ @staticmethod
+ def _delete_revisions(dataset: Dataset, revisions: QuerySet[DatasetRevision], ids: list[UUID]) -> None:
+ references = OfflineExperiment.objects.for_team(dataset.team_id).filter(dataset_revision_id__in=ids)
+ reference_ids = list(references.order_by("id").values_list("id", flat=True)[:BATCH_SIZE])
+ references.filter(id__in=reference_ids).update(dataset_revision=None)
+ removable_ids = list(
+ revisions.filter(id__in=ids)
+ .filter(~Exists(references.filter(dataset_revision_id=OuterRef("id"))))
+ .values_list("id", flat=True)
+ )
+ Dataset.objects.for_team(dataset.team_id, canonical=True).filter(
+ id=dataset.id, current_revision_id__in=removable_ids
+ ).update(current_revision=None)
+ revisions.filter(id__in=removable_ids).delete()
+
+ def _delete_expired_dataset_batch(self, dataset: Dataset) -> None:
+ versions = DatasetItemVersion.objects.for_team(dataset.team_id, canonical=True).filter(dataset_id=dataset.id)
+ ids = list(versions.order_by("id").values_list("id", flat=True)[:BATCH_SIZE])
+ if ids:
+ self._delete_versions(dataset, ids)
+ return
+ revisions = DatasetRevision.objects.for_team(dataset.team_id, canonical=True).filter(dataset_id=dataset.id)
+ revision_ids = list(revisions.order_by("id").values_list("id", flat=True)[:BATCH_SIZE])
+ if revision_ids:
+ self._delete_revisions(dataset, revisions, revision_ids)
+ return
+ dataset.delete()
+ self.datasets_deleted += 1
+
+ def _clean_dataset_contents(self, dataset: Dataset) -> None:
+ team_id = dataset.team_id
+ now = timezone.now()
+ uploading = self._uploading_revisions(team_id).filter(dataset_revision__dataset_id=dataset.id)
+ if dataset.expired_at is not None:
+ self._delete_expired_dataset_batch(dataset)
+ return
+ if dataset.last_used_at is None:
+ Dataset.objects.for_team(team_id, canonical=True).filter(id=dataset.id).update(last_used_at=now)
+ return
+ expires_at = DatasetRetentionPolicy.dataset_expires_at(dataset)
+ if expires_at is not None and expires_at <= now and not uploading.exists():
+ dataset.expired_at = now
+ dataset.save(update_fields=["expired_at"])
+ self._delete_expired_dataset_batch(dataset)
+ self.datasets_expired += 1
+ return
+ cutoff = now - HISTORY_RETENTION - USAGE_REFRESH_INTERVAL
+ revisions = DatasetRevision.objects.for_team(team_id, canonical=True).filter(dataset_id=dataset.id)
+ current_revision_id = dataset.current_revision_id
+ if current_revision_id is None:
+ current_revision_id = revisions.order_by("-revision").values_list("id", flat=True).first()
+ uninitialized_revisions = list(
+ revisions.filter(last_used_at__isnull=True).values_list("id", flat=True)[:BATCH_SIZE]
+ )
+ revisions.filter(id__in=uninitialized_revisions).update(last_used_at=now)
+ versions = DatasetItemVersion.objects.for_team(team_id, canonical=True).filter(dataset_id=dataset.id)
+ uninitialized_versions = list(
+ versions.filter(last_used_at__isnull=True).values_list("id", flat=True)[:BATCH_SIZE]
+ )
+ versions.filter(id__in=uninitialized_versions).update(last_used_at=now)
+ eligible_revisions = revisions.filter(expired_at__isnull=True, last_used_at__lte=cutoff)
+ if current_revision_id is not None:
+ eligible_revisions = eligible_revisions.exclude(id=current_revision_id)
+ revision_ids = list(
+ eligible_revisions.exclude(id__in=uploading.values("dataset_revision_id"))
+ .order_by("last_used_at", "id")
+ .values_list("id", flat=True)[:BATCH_SIZE]
+ )
+ expired_count = revisions.filter(id__in=revision_ids).update(expired_at=now)
+ ids = list(
+ self._unused_versions(versions, revisions, cutoff).order_by("id").values_list("id", flat=True)[:BATCH_SIZE]
+ )
+ self._delete_versions(dataset, ids)
+ # Keep revision identities while a surviving version or an evaluation still references them.
+ removable_revisions = revisions.filter(expired_at__isnull=False).filter(
+ ~Exists(
+ DatasetItemVersion.objects.for_team(team_id, canonical=True).filter(dataset_revision_id=OuterRef("id"))
+ ),
+ ~Exists(OfflineExperiment.objects.for_team(team_id).filter(dataset_revision_id=OuterRef("id"))),
+ )
+ empty_ids = list(removable_revisions.order_by("id").values_list("id", flat=True)[:BATCH_SIZE])
+ revisions.filter(id__in=empty_ids).delete()
+ self.revisions_expired += expired_count
+
+ def _clean_dataset(self, team_id: int, dataset_id: UUID) -> None:
+ failure: Exception | None = None
+ before = (self.datasets_expired, self.revisions_expired, self.versions_deleted, self.datasets_deleted)
+ with transaction.atomic():
+ dataset = (
+ Dataset.objects.for_team(team_id, canonical=True)
+ .select_for_update(skip_locked=True)
+ .filter(id=dataset_id)
+ .first()
+ )
+ if dataset is None:
+ return
+ Dataset.objects.for_team(team_id, canonical=True).filter(id=dataset.id).update(
+ retention_checked_at=timezone.now()
+ )
+ # A savepoint rolls back deletion without losing the attempt timestamp that rotates candidates.
+ try:
+ with transaction.atomic():
+ self._clean_dataset_contents(dataset)
+ counts = {
+ "datasets_expired": self.datasets_expired - before[0],
+ "revisions_expired": self.revisions_expired - before[1],
+ "versions_deleted": self.versions_deleted - before[2],
+ "datasets_deleted": self.datasets_deleted - before[3],
+ }
+ if any(counts.values()):
+ transaction.on_commit(
+ partial(
+ logger.info,
+ "dataset_retention_dataset_cleaned",
+ team_id=team_id,
+ dataset_id=str(dataset_id),
+ **counts,
+ )
+ )
+ except Exception as error:
+ failure = error
+ if failure is not None:
+ raise failure
+
+ def run(self) -> None:
+ if not dataset_retention_enabled():
+ self.disabled = True
+ return
+ deadline = time.monotonic() + TIME_BUDGET_SECONDS
+ now = timezone.now()
+ history_cutoff = now - HISTORY_RETENTION - USAGE_REFRESH_INTERVAL
+ active_cutoff = now - ACTIVE_RETENTION - USAGE_REFRESH_INTERVAL
+ old_revisions = (
+ DatasetRevision.objects.unscoped()
+ .filter(expired_at__isnull=True)
+ .filter(Q(last_used_at__isnull=True) | Q(last_used_at__lte=history_cutoff))
+ .exclude(id=F("dataset__current_revision_id"))
+ )
+ unused_versions = self._unused_versions(
+ DatasetItemVersion.objects.unscoped(), DatasetRevision.objects.unscoped(), history_cutoff
+ )
+ empty_revisions = (
+ DatasetRevision.objects.unscoped()
+ .filter(expired_at__isnull=False)
+ .filter(
+ ~Exists(DatasetItemVersion.objects.unscoped().filter(dataset_revision_id=OuterRef("id"))),
+ ~Exists(OfflineExperiment.objects.unscoped().filter(dataset_revision_id=OuterRef("id"))),
+ )
+ )
+ candidates = (
+ Dataset.objects.unscoped()
+ .filter(
+ Q(expired_at__isnull=False)
+ | Q(last_used_at__isnull=True)
+ | Q(archived=True, last_used_at__lte=history_cutoff)
+ | Q(archived=False, last_used_at__lte=active_cutoff)
+ | Q(Exists(old_revisions.filter(dataset_id=OuterRef("id"))))
+ | Q(Exists(unused_versions.filter(dataset_id=OuterRef("id"))))
+ | Q(Exists(empty_revisions.filter(dataset_id=OuterRef("id"))))
+ )
+ .order_by(F("retention_checked_at").asc(nulls_first=True), "id")
+ .values_list("id", "team_id")[:MAX_DATASETS_PER_RUN]
+ )
+ for dataset_id, team_id in candidates:
+ if not dataset_retention_enabled():
+ self.disabled = True
+ break
+ if time.monotonic() >= deadline:
+ self.budget_exhausted = True
+ break
+ datasets_expired = self.datasets_expired
+ revisions_expired = self.revisions_expired
+ versions_deleted = self.versions_deleted
+ datasets_deleted = self.datasets_deleted
+ try:
+ self._clean_dataset(team_id, dataset_id)
+ except Exception as error:
+ self.datasets_expired = datasets_expired
+ self.revisions_expired = revisions_expired
+ self.versions_deleted = versions_deleted
+ self.datasets_deleted = datasets_deleted
+ self.datasets_failed += 1
+ if isinstance(error, SoftTimeLimitExceeded):
+ raise
+ logger.exception("dataset_retention_dataset_failed", team_id=team_id, dataset_id=str(dataset_id))
diff --git a/products/ai_observability/backend/dataset_service.py b/products/ai_observability/backend/dataset_service.py
index a3f06e7cba43..26b7a0515d11 100644
--- a/products/ai_observability/backend/dataset_service.py
+++ b/products/ai_observability/backend/dataset_service.py
@@ -8,6 +8,7 @@ from uuid import UUID
from django.db import IntegrityError, connection, transaction
from django.db.models import Max
+from django.utils import timezone
import posthoganalytics
@@ -17,6 +18,7 @@ from posthog.models import Team, User
from products.ai_observability.backend.dataset_limits import MAX_DATASET_ITEM_PAYLOAD_BYTES, MAX_ITEMS_PER_DATASET
from products.ai_observability.backend.dataset_queries import latest_dataset_revision
+from products.ai_observability.backend.dataset_retention import DatasetUsage
from products.ai_observability.backend.models.datasets import Dataset, DatasetItem, DatasetItemVersion, DatasetRevision
type JSONValue = None | bool | int | float | str | list[JSONValue] | dict[str, JSONValue]
@@ -257,7 +259,7 @@ def _lock_dataset(*, team_id: int, dataset_id: UUID) -> Dataset:
Dataset.objects.for_team(team_id, canonical=True)
.select_for_update(of=("self",))
.select_related("current_revision")
- .get(id=dataset_id)
+ .get(id=dataset_id, expired_at__isnull=True)
)
current_revision = dataset.current_revision
if current_revision is None:
@@ -330,6 +332,10 @@ def _check_base_version(*, current_version: DatasetItemVersion, base_version: in
def _create_revision(*, dataset: Dataset, created_by: User | None) -> DatasetRevision:
+ if dataset.current_revision_id is not None:
+ DatasetRevision.objects.for_team(dataset.team_id, canonical=True).filter(id=dataset.current_revision_id).update(
+ last_used_at=timezone.now()
+ )
latest_revision_number = (
DatasetRevision.objects.for_team(dataset.team_id, canonical=True)
.filter(dataset_id=dataset.id)
@@ -404,7 +410,8 @@ def _create_item_version(
item.current_version = version
item.save(update_fields=["current_version", "updated_at"])
dataset.current_revision = revision
- dataset.save(update_fields=["current_revision", "updated_at"])
+ dataset.last_used_at = timezone.now()
+ dataset.save(update_fields=["current_revision", "updated_at", "last_used_at"])
return version
@@ -459,7 +466,7 @@ def create_dataset(
normalized_name = _validate_dataset_fields(name=name, description=description, metadata=normalized_metadata)
with connection.cursor() as cursor:
cursor.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", [f"aio-dataset-allocation:{team.id}"])
- current_count = Dataset.objects.for_team(team.id, canonical=True).count()
+ current_count = Dataset.objects.for_team(team.id, canonical=True).filter(expired_at__isnull=True).count()
if current_count >= max_datasets:
raise DatasetLimitExceeded(
resource="datasets",
@@ -511,7 +518,8 @@ def update_dataset(
)
changed_fields: list[str] = []
- update_fields = ["updated_at"]
+ dataset.last_used_at = timezone.now()
+ update_fields = ["updated_at", "last_used_at"]
if not isinstance(name, _Unset):
if dataset.name != normalized_name:
changed_fields.append("name")
@@ -543,7 +551,8 @@ def archive_dataset(*, team_id: int, dataset_id: UUID) -> DatasetMutationResult:
if dataset.archived:
return DatasetMutationResult(dataset=dataset)
dataset.archived = True
- dataset.save(update_fields=["archived", "updated_at"])
+ dataset.last_used_at = timezone.now()
+ dataset.save(update_fields=["archived", "updated_at", "last_used_at"])
return DatasetMutationResult(dataset=dataset, changed_fields=("archived",))
@@ -553,7 +562,8 @@ def restore_dataset(*, team_id: int, dataset_id: UUID) -> DatasetMutationResult:
if not dataset.archived:
return DatasetMutationResult(dataset=dataset)
dataset.archived = False
- dataset.save(update_fields=["archived", "updated_at"])
+ dataset.last_used_at = timezone.now()
+ dataset.save(update_fields=["archived", "updated_at", "last_used_at"])
return DatasetMutationResult(dataset=dataset, changed_fields=("archived",))
@@ -609,6 +619,7 @@ def create_dataset_item(
current_item_id=existing_item.id,
)
if _item_contents_equal(_DatasetItemContent.from_version(current_version), content):
+ DatasetUsage.record(dataset, dataset.current_revision)
return DatasetItemMutationResult(
item=existing_item,
version=current_version,
@@ -689,6 +700,7 @@ def update_dataset_item(
_validate_item_content(content=next_content)
if _item_contents_equal(current_content, next_content):
+ DatasetUsage.record(dataset, dataset.current_revision)
return DatasetItemMutationResult(
item=item,
version=current_version,
diff --git a/products/ai_observability/backend/facade/admin.py b/products/ai_observability/backend/facade/admin.py
new file mode 100644
index 000000000000..4dc9eaaf3309
--- /dev/null
+++ b/products/ai_observability/backend/facade/admin.py
@@ -0,0 +1,2 @@
+from products.ai_observability.backend.admin import set_dataset_retention_enabled as set_dataset_retention_enabled
+from products.ai_observability.backend.dataset_retention import dataset_retention_enabled as dataset_retention_enabled
diff --git a/products/ai_observability/backend/facade/tasks.py b/products/ai_observability/backend/facade/tasks.py
index 109cd1364bcb..bd9fcbac8197 100644
--- a/products/ai_observability/backend/facade/tasks.py
+++ b/products/ai_observability/backend/facade/tasks.py
@@ -1,3 +1,4 @@
+from products.ai_observability.backend.tasks.dataset_retention import expire_datasets
from products.ai_observability.backend.tasks.offline_payload_retention import expire_offline_evaluation_payloads
-__all__ = ["expire_offline_evaluation_payloads"]
+__all__ = ["expire_datasets", "expire_offline_evaluation_payloads"]
diff --git a/products/ai_observability/backend/migrations/0071_dataset_retention.py b/products/ai_observability/backend/migrations/0071_dataset_retention.py
new file mode 100644
index 000000000000..6eb37326b7d3
--- /dev/null
+++ b/products/ai_observability/backend/migrations/0071_dataset_retention.py
@@ -0,0 +1,43 @@
+# Generated by Django 6.1.2 on 2026-11-03 15:08
+
+import django.utils.timezone
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("ai_observability", "0070_offline_payload_expiry_indexes"),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name="dataset",
+ name="expired_at",
+ field=models.DateTimeField(blank=True, null=True),
+ ),
+ migrations.AddField(
+ model_name="dataset",
+ name="last_used_at",
+ field=models.DateTimeField(default=django.utils.timezone.now, null=True),
+ ),
+ migrations.AddField(
+ model_name="dataset",
+ name="retention_checked_at",
+ field=models.DateTimeField(blank=True, null=True),
+ ),
+ migrations.AddField(
+ model_name="datasetitemversion",
+ name="last_used_at",
+ field=models.DateTimeField(default=django.utils.timezone.now, null=True),
+ ),
+ migrations.AddField(
+ model_name="datasetrevision",
+ name="expired_at",
+ field=models.DateTimeField(blank=True, null=True),
+ ),
+ migrations.AddField(
+ model_name="datasetrevision",
+ name="last_used_at",
+ field=models.DateTimeField(default=django.utils.timezone.now, null=True),
+ ),
+ ]
diff --git a/products/ai_observability/backend/migrations/0072_dataset_retention_indexes.py b/products/ai_observability/backend/migrations/0072_dataset_retention_indexes.py
new file mode 100644
index 000000000000..d342b6e39924
--- /dev/null
+++ b/products/ai_observability/backend/migrations/0072_dataset_retention_indexes.py
@@ -0,0 +1,24 @@
+from django.db import migrations, models
+
+from posthog.migration_helpers import SafeAddIndexConcurrently
+
+
+class Migration(migrations.Migration):
+ atomic = False
+
+ dependencies = [("ai_observability", "0071_dataset_retention")]
+
+ operations = [
+ SafeAddIndexConcurrently(
+ model_name="dataset",
+ index=models.Index(fields=["last_used_at", "id"], name="llma_dataset_retention_idx"),
+ ),
+ SafeAddIndexConcurrently(
+ model_name="dataset",
+ index=models.Index(fields=["retention_checked_at", "id"], name="llma_dataset_retention_chk_idx"),
+ ),
+ SafeAddIndexConcurrently(
+ model_name="datasetrevision",
+ index=models.Index(fields=["last_used_at", "id"], name="llma_revision_retention_idx"),
+ ),
+ ]
diff --git a/products/ai_observability/backend/migrations/0073_dataset_live_name_uniqueness.py b/products/ai_observability/backend/migrations/0073_dataset_live_name_uniqueness.py
new file mode 100644
index 000000000000..4ac331d047e5
--- /dev/null
+++ b/products/ai_observability/backend/migrations/0073_dataset_live_name_uniqueness.py
@@ -0,0 +1,37 @@
+# Generated by Django 6.1.2 on 2026-11-03 16:17
+
+from django.db import migrations, models
+
+from posthog.migration_helpers import CreateIndexConcurrently
+
+
+class Migration(migrations.Migration):
+ atomic = False
+
+ dependencies = [
+ ("ai_observability", "0072_dataset_retention_indexes"),
+ ]
+
+ operations = [
+ migrations.SeparateDatabaseAndState(
+ database_operations=[
+ CreateIndexConcurrently(
+ index_name="uniq_llma_dataset_v2_live_name",
+ table_name="llm_analytics_dataset_v2",
+ columns="(team_id, name)",
+ unique=True,
+ where="WHERE expired_at IS NULL",
+ ),
+ ],
+ state_operations=[
+ migrations.AddConstraint(
+ model_name="dataset",
+ constraint=models.UniqueConstraint(
+ condition=models.Q(expired_at__isnull=True),
+ fields=("team", "name"),
+ name="uniq_llma_dataset_v2_live_name",
+ ),
+ ),
+ ],
+ ),
+ ]
diff --git a/products/ai_observability/backend/migrations/0074_release_expired_dataset_names.py b/products/ai_observability/backend/migrations/0074_release_expired_dataset_names.py
new file mode 100644
index 000000000000..818557666c7e
--- /dev/null
+++ b/products/ai_observability/backend/migrations/0074_release_expired_dataset_names.py
@@ -0,0 +1,12 @@
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+ dependencies = [("ai_observability", "0073_dataset_live_name_uniqueness")]
+
+ operations = [
+ migrations.RemoveConstraint(
+ model_name="dataset",
+ name="uniq_llma_dataset_v2_team_name",
+ ),
+ ]
diff --git a/products/ai_observability/backend/migrations/max_migration.txt b/products/ai_observability/backend/migrations/max_migration.txt
index 0317898d12ac..8ea568e02f5e 100644
--- a/products/ai_observability/backend/migrations/max_migration.txt
+++ b/products/ai_observability/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0070_offline_payload_expiry_indexes
+0074_release_expired_dataset_names
diff --git a/products/ai_observability/backend/models/datasets.py b/products/ai_observability/backend/models/datasets.py
index 36d31484de09..179e0b948d66 100644
--- a/products/ai_observability/backend/models/datasets.py
+++ b/products/ai_observability/backend/models/datasets.py
@@ -5,6 +5,7 @@ from django.db import models
from django.db.models import F, Func, Q, Value
from django.db.models.expressions import NegatedExpression
from django.db.models.lookups import Exact
+from django.utils import timezone
from posthog.models.scoping.manager import TeamScopedManager
from posthog.models.tagged_items_relation import Taggable
@@ -34,6 +35,9 @@ class Dataset(Taggable, UUIDModel, CreatedMetaFields, UpdatedMetaFields):
description = models.TextField(blank=True, default="")
metadata = models.JSONField(blank=True, default=dict)
archived = models.BooleanField(default=False)
+ last_used_at = models.DateTimeField(default=timezone.now, null=True)
+ expired_at = models.DateTimeField(null=True, blank=True)
+ retention_checked_at = models.DateTimeField(null=True, blank=True)
current_revision = models.ForeignKey(
"DatasetRevision",
on_delete=models.RESTRICT,
@@ -46,13 +50,19 @@ class Dataset(Taggable, UUIDModel, CreatedMetaFields, UpdatedMetaFields):
db_table = "llm_analytics_dataset_v2"
ordering = ["-created_at", "id"]
constraints = [
- models.UniqueConstraint(fields=["team", "name"], name="uniq_llma_dataset_v2_team_name"),
+ models.UniqueConstraint(
+ fields=["team", "name"],
+ condition=Q(expired_at__isnull=True),
+ name="uniq_llma_dataset_v2_live_name",
+ ),
models.CheckConstraint(
condition=_json_type_is("metadata", "object"),
name="llma_dataset_v2_metadata_object",
),
]
indexes = [
+ models.Index(fields=["last_used_at", "id"], name="llma_dataset_retention_idx"),
+ models.Index(fields=["retention_checked_at", "id"], name="llma_dataset_retention_chk_idx"),
models.Index(
fields=["team", "archived", "-created_at", "id"],
name="llma_dataset_v2_team_arch_idx",
@@ -80,10 +90,13 @@ class DatasetRevision(UUIDModel, CreatedMetaFields):
dataset = models.ForeignKey(Dataset, on_delete=models.CASCADE, related_name="revisions")
revision = models.PositiveIntegerField()
+ last_used_at = models.DateTimeField(default=timezone.now, null=True)
+ expired_at = models.DateTimeField(null=True, blank=True)
class Meta:
db_table = "llm_analytics_datasetrevision_v2"
ordering = ["-revision"]
+ indexes = [models.Index(fields=["last_used_at", "id"], name="llma_revision_retention_idx")]
constraints = [
models.UniqueConstraint(
fields=["dataset", "revision"],
@@ -156,6 +169,7 @@ class DatasetItemVersion(UUIDModel, CreatedMetaFields):
related_name="item_versions",
)
version = models.PositiveIntegerField()
+ last_used_at = models.DateTimeField(default=timezone.now, null=True)
archived = models.BooleanField(default=False)
input = models.JSONField()
expected_output = models.JSONField(null=True, blank=True)
diff --git a/products/ai_observability/backend/offline_evaluation_service.py b/products/ai_observability/backend/offline_evaluation_service.py
index 29f53c73f7f2..66f685e8b9a2 100644
--- a/products/ai_observability/backend/offline_evaluation_service.py
+++ b/products/ai_observability/backend/offline_evaluation_service.py
@@ -17,6 +17,7 @@ from posthog.exceptions import QuotaLimitExceeded
from posthog.models.team import Team
from products.ai_observability.backend.dataset_queries import dataset_item_versions_at_revision
+from products.ai_observability.backend.dataset_retention import DatasetUsage
from products.ai_observability.backend.models.datasets import Dataset, DatasetItemVersion, DatasetRevision
from products.ai_observability.backend.models.offline_evaluations import (
OfflineEvaluationResult,
@@ -173,7 +174,11 @@ class _OfflineEvaluationService:
def _dataset_revisions(self) -> QuerySet[DatasetRevision]:
# Skip parent resolution because hosted dataset references belong to the submitted environment.
- revisions = DatasetRevision.objects.for_team(self.team_id, canonical=True)
+ revisions = (
+ DatasetRevision.objects.for_team(self.team_id, canonical=True)
+ .filter(expired_at__isnull=True, dataset__expired_at__isnull=True)
+ .select_related("dataset")
+ )
if self._user_access_control is not None:
datasets = self._user_access_control.filter_queryset_by_access_level(
Dataset.objects.for_team(self.team_id, canonical=True), include_all_if_admin=True, resource="dataset"
@@ -213,6 +218,12 @@ class OfflineExperimentService(_OfflineEvaluationService):
errors.update(error.errors)
if errors:
raise OfflineEvaluationValidationError.from_errors(errors)
+ try:
+ DatasetUsage.record(revision.dataset, revision)
+ except (Dataset.DoesNotExist, DatasetRevision.DoesNotExist) as error:
+ raise OfflineEvaluationValidationError(
+ "dataset_revision_id", "Select an existing dataset revision."
+ ) from error
return fields
def create(self, submission: ExperimentSubmission, *, created_by_id: int | None = None) -> ExperimentReceipt:
@@ -251,6 +262,10 @@ class OfflineExperimentService(_OfflineEvaluationService):
return ExperimentReceipt(experiment=experiment, counts=counts)
if experiment.status != OfflineExperiment.Status.UPLOADING:
raise OfflineEvaluationConflict("experiment_closed", "This experiment is already closed.")
+ if experiment.dataset_revision_id is not None:
+ revision = self._dataset_revisions().filter(id=experiment.dataset_revision_id).first()
+ if revision is not None:
+ DatasetUsage.record(revision.dataset, revision)
if status == "completed" and (
counts.expected_item_count is not None
and counts.expected_item_count != counts.accepted_item_count
@@ -306,6 +321,7 @@ class OfflineEvaluationIngestionService(_OfflineEvaluationService):
revision = self._dataset_revisions().filter(id=revision_id).first()
if revision is None:
raise OfflineEvaluationValidationError("items", "The experiment's dataset revision is no longer available.")
+ DatasetUsage.record(revision.dataset, revision)
return {
version.id: version
for version in dataset_item_versions_at_revision(
@@ -506,6 +522,10 @@ class OfflineEvaluationIngestionService(_OfflineEvaluationService):
results[ResultIdentity(item_id=result.item_id, scorer_version_id=result.scorer_version_id)] = row
if errors:
raise OfflineEvaluationValidationError.from_errors(errors)
+ if new_results and not new_item_submissions and experiment.dataset_revision_id is not None:
+ revision = self._dataset_revisions().filter(id=experiment.dataset_revision_id).first()
+ if revision is not None:
+ DatasetUsage.record(revision.dataset, revision)
OfflineExperimentItem.objects.for_team(self.team_id).bulk_create(new_items)
new_item_ids = {item.id for item in new_items}
OfflineExperimentItemPayload.objects.for_team(self.team_id).bulk_create(
diff --git a/products/ai_observability/backend/tasks/__init__.py b/products/ai_observability/backend/tasks/__init__.py
index b69cd3fdf549..b08fe61941ca 100644
--- a/products/ai_observability/backend/tasks/__init__.py
+++ b/products/ai_observability/backend/tasks/__init__.py
@@ -1,3 +1,4 @@
+from .dataset_retention import expire_datasets
from .offline_payload_retention import expire_offline_evaluation_payloads
-__all__ = ["expire_offline_evaluation_payloads"]
+__all__ = ["expire_datasets", "expire_offline_evaluation_payloads"]
diff --git a/products/ai_observability/backend/tasks/dataset_retention.py b/products/ai_observability/backend/tasks/dataset_retention.py
new file mode 100644
index 000000000000..106c1cee3861
--- /dev/null
+++ b/products/ai_observability/backend/tasks/dataset_retention.py
@@ -0,0 +1,43 @@
+import time
+from uuid import uuid4
+
+import structlog
+from celery import shared_task
+
+from posthog.ph_client import ph_scoped_capture
+from posthog.scoping_audit import skip_team_scope_audit
+from posthog.utils import get_instance_region
+
+from products.ai_observability.backend.dataset_retention import DatasetRetention
+
+logger = structlog.get_logger(__name__)
+
+
+@shared_task(ignore_result=True, soft_time_limit=240, time_limit=300)
+@skip_team_scope_audit
+def expire_datasets() -> None:
+ sweep = DatasetRetention()
+ started = time.monotonic()
+ status = "failed"
+ try:
+ sweep.run()
+ status = "disabled" if sweep.disabled else "completed_with_errors" if sweep.datasets_failed else "completed"
+ finally:
+ properties = {
+ "run_id": str(uuid4()),
+ "status": status,
+ "datasets_expired": sweep.datasets_expired,
+ "revisions_expired": sweep.revisions_expired,
+ "versions_deleted": sweep.versions_deleted,
+ "datasets_deleted": sweep.datasets_deleted,
+ "datasets_failed": sweep.datasets_failed,
+ "budget_exhausted": sweep.budget_exhausted,
+ "duration_seconds": time.monotonic() - started,
+ "$process_person_profile": False,
+ }
+ logger.info("dataset_retention_run", **properties)
+ try:
+ with ph_scoped_capture(region=get_instance_region() or "US") as capture:
+ capture(distinct_id="dataset-retention", event="dataset retention run", properties=properties)
+ except Exception:
+ logger.exception("dataset_retention_capture_failed")
diff --git a/products/ai_observability/backend/templates/admin/posthog/organization/ai_observability_dataset_retention.html b/products/ai_observability/backend/templates/admin/posthog/organization/ai_observability_dataset_retention.html
new file mode 100644
index 000000000000..363b0d412195
--- /dev/null
+++ b/products/ai_observability/backend/templates/admin/posthog/organization/ai_observability_dataset_retention.html
@@ -0,0 +1,24 @@
+{% extends "admin/base_site.html" %}
+{% block content %}
+ <div id="content-main">
+ <p>Dataset cleanup is currently <strong>{{ enabled|yesno:"enabled,disabled" }}</strong> in this region.</p>
+ <p>
+ Cleanup permanently deletes unused active datasets after 90 days and archived datasets and historical
+ revisions after 30 days, with a 12-hour usage recording margin. Evaluation results and scores are preserved.
+ </p>
+ <p>
+ Disabling cleanup stops new expiry and deletion, including deletion already in progress. A running sweep
+ finishes its current dataset before stopping. Expired datasets remain unavailable, and deleted content is not restored.
+ </p>
+ <p>Enabling cleanup resumes the hourly schedule. Change this setting separately in each region.</p>
+ <form method="post">
+ {% csrf_token %}
+ <fieldset class="module aligned">
+ {{ form.as_div }}
+ </fieldset>
+ <div class="submit-row">
+ <input type="submit" value="Save dataset cleanup setting" class="default">
+ </div>
+ </form>
+ </div>
+{% endblock %}
diff --git a/products/ai_observability/backend/test/test_dataset_export.py b/products/ai_observability/backend/test/test_dataset_export.py
index d578a2ce3237..944db5eafc3d 100644
--- a/products/ai_observability/backend/test/test_dataset_export.py
+++ b/products/ai_observability/backend/test/test_dataset_export.py
@@ -1,8 +1,11 @@
import json
+import time_machine
from posthog.test.base import APIBaseTest
from unittest.mock import patch
+from django.utils import timezone
+
from parameterized import parameterized
from posthog.storage import object_storage
@@ -15,7 +18,7 @@ from products.ai_observability.backend.dataset_service import (
create_dataset_item,
update_dataset_item,
)
-from products.ai_observability.backend.models.datasets import Dataset
+from products.ai_observability.backend.models.datasets import Dataset, DatasetRevision
from products.exports.backend.facade.api import DATASET_EXPORT_KIND, ExportedAsset, RetryableExportError
from products.exports.backend.tasks.failure_handler import EXCEPTIONS_TO_RETRY
@@ -204,6 +207,29 @@ class TestDatasetExport(APIBaseTest):
asset.refresh_from_db()
self.assertFalse(asset.has_content)
+ @parameterized.expand([("expired", 1), ("missing", 2)])
+ @time_machine.travel("2026-11-03T12:00:00Z", tick=False)
+ @patch("products.ai_observability.backend.dataset_export.posthog_feature_flag_value", return_value=True)
+ def test_export_rejects_unavailable_revisions(self, _name: str, revision: int, _feature_flag: object) -> None:
+ dataset, _result = self._create_dataset_and_item()
+ DatasetRevision.objects.for_team(self.team.id).filter(dataset_id=dataset.id).update(expired_at=timezone.now())
+ asset = ExportedAsset.objects.create(
+ team=self.team,
+ created_by=self.user,
+ export_format=ExportedAsset.ExportFormat.JSONL,
+ export_context={
+ "kind": DATASET_EXPORT_KIND,
+ "dataset_id": str(dataset.id),
+ "dataset_revision": revision,
+ },
+ )
+
+ with self.assertRaisesRegex(DatasetExportError, "The selected dataset revision is no longer available"):
+ export_dataset_jsonl(asset)
+
+ asset.refresh_from_db()
+ self.assertFalse(asset.has_content)
+
@patch("products.ai_observability.backend.dataset_export.posthog_feature_flag_value", return_value=None)
def test_export_retries_when_feature_flag_evaluation_is_unavailable(self, _feature_flag: object) -> None:
dataset, _result = self._create_dataset_and_item()
diff --git a/products/ai_observability/backend/test/test_dataset_retention.py b/products/ai_observability/backend/test/test_dataset_retention.py
new file mode 100644
index 000000000000..059fa38fd71b
--- /dev/null
+++ b/products/ai_observability/backend/test/test_dataset_retention.py
@@ -0,0 +1,685 @@
+from concurrent.futures import ThreadPoolExecutor
+from datetime import timedelta
+from uuid import UUID, uuid4
+
+import time_machine
+from unittest.mock import patch
+
+from django.db import DatabaseError, IntegrityError, connection, transaction
+from django.db.migrations.executor import MigrationExecutor
+from django.test import TestCase, TransactionTestCase
+from django.utils import timezone
+
+from celery.exceptions import SoftTimeLimitExceeded
+from parameterized import parameterized
+
+from posthog.models import Organization, Team
+from posthog.models.instance_setting import override_instance_config, set_instance_setting
+
+from products.ai_observability.backend.dataset_queries import dataset_item_versions_at_revision
+from products.ai_observability.backend.dataset_retention import (
+ ACTIVE_RETENTION,
+ HISTORY_RETENTION,
+ RETENTION_ENABLED_SETTING,
+ USAGE_REFRESH_INTERVAL,
+ DatasetRetention,
+ DatasetRetentionPolicy,
+ DatasetUsage,
+)
+from products.ai_observability.backend.dataset_service import (
+ DatasetItemMutationResult,
+ DatasetMutationConflict,
+ archive_dataset,
+ archive_dataset_item,
+ create_dataset,
+ create_dataset_item,
+ restore_dataset_item,
+ update_dataset_item,
+)
+from products.ai_observability.backend.models.datasets import Dataset, DatasetItemVersion, DatasetRevision
+from products.ai_observability.backend.models.offline_evaluations import (
+ OfflineEvaluationResult,
+ OfflineEvaluationUsageLedger,
+ OfflineExperiment,
+ OfflineExperimentItem,
+)
+from products.ai_observability.backend.models.score_definitions import ScoreDefinition
+from products.ai_observability.backend.offline_evaluation_service import (
+ OfflineEvaluationIngestionService,
+ OfflineExperimentService,
+)
+from products.ai_observability.backend.offline_evaluation_types import (
+ ExperimentSubmission,
+ ItemSubmission,
+ ResultSubmission,
+ UploadSubmission,
+)
+
+
+@time_machine.travel("2026-11-03T12:00:00Z", tick=False)
+class TestDatasetRetention(TestCase):
+ team: Team
+
+ @classmethod
+ def setUpTestData(cls) -> None:
+ _, _, cls.team = Organization.objects.bootstrap(None)
+
+ def setUp(self) -> None:
+ self.dataset = Dataset.objects.for_team(self.team.id, canonical=True).create(
+ team=self.team, name="Retention fixture"
+ )
+ self.old = timezone.now() - ACTIVE_RETENTION - USAGE_REFRESH_INTERVAL - timedelta(days=1)
+
+ def create_item(self, value: str = "First input") -> DatasetItemMutationResult:
+ result = create_dataset_item(team_id=self.team.id, dataset_id=self.dataset.id, created_by=None, input=value)
+ self.dataset.refresh_from_db()
+ return result
+
+ def update_item(self, result: DatasetItemMutationResult, value: str = "Updated input") -> DatasetItemMutationResult:
+ updated = update_dataset_item(
+ team_id=self.team.id,
+ dataset_id=self.dataset.id,
+ item_id=result.item.id,
+ created_by=None,
+ base_version=result.version.version,
+ input=value,
+ )
+ self.dataset.refresh_from_db()
+ return updated
+
+ def age_history(self) -> None:
+ DatasetRevision.objects.for_team(self.team.id, canonical=True).filter(dataset=self.dataset).update(
+ last_used_at=self.old
+ )
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(dataset=self.dataset).update(
+ last_used_at=self.old
+ )
+
+ @parameterized.expand([(False, ACTIVE_RETENTION), (True, HISTORY_RETENTION)])
+ def test_expiry_includes_the_usage_recording_margin(self, archived: bool, retention: timedelta) -> None:
+ now = timezone.now()
+ self.dataset.archived = archived
+ self.dataset.last_used_at = now - retention
+ self.dataset.save()
+ DatasetRetention().run()
+ self.assertTrue(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=self.dataset.id).exists())
+ self.assertEqual(DatasetRetentionPolicy.dataset_expires_at(self.dataset), now + USAGE_REFRESH_INTERVAL)
+
+ self.dataset.last_used_at -= USAGE_REFRESH_INTERVAL + timedelta(seconds=1)
+ self.dataset.save()
+ DatasetRetention().run()
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=self.dataset.id).exists())
+
+ def test_repeated_usage_needs_no_database_queries(self) -> None:
+ item = self.create_item()
+ with self.assertNumQueries(0):
+ DatasetUsage.record(self.dataset, item.version.dataset_revision)
+ self.age_history()
+ self.dataset.last_used_at = self.old
+ self.dataset.save()
+ item.version.dataset_revision.refresh_from_db()
+ DatasetUsage.record(self.dataset, item.version.dataset_revision)
+ with self.assertNumQueries(0):
+ DatasetUsage.record(self.dataset, item.version.dataset_revision)
+ assert self.dataset.last_used_at is not None
+ assert item.version.dataset_revision.last_used_at is not None
+ self.assertGreater(self.dataset.last_used_at, self.old)
+ self.assertGreater(item.version.dataset_revision.last_used_at, self.old)
+
+ def test_shared_version_survives_expired_introducing_revision(self) -> None:
+ first = self.create_item()
+ second = self.create_item("Second item")
+ self.update_item(first)
+ self.age_history()
+ second.version.dataset_revision.last_used_at = timezone.now()
+ second.version.dataset_revision.save(update_fields=["last_used_at"])
+ snapshot = dataset_item_versions_at_revision(
+ team_id=self.team.id,
+ dataset_id=self.dataset.id,
+ revision=second.version.dataset_revision.revision,
+ archived=False,
+ )
+ before = list(snapshot.values_list("id", flat=True))
+ DatasetRetention().run()
+
+ first.version.dataset_revision.refresh_from_db()
+ self.assertIsNotNone(first.version.dataset_revision.expired_at)
+ self.assertTrue(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=first.version.id).exists()
+ )
+ snapshot = dataset_item_versions_at_revision(
+ team_id=self.team.id,
+ dataset_id=self.dataset.id,
+ revision=second.version.dataset_revision.revision,
+ archived=False,
+ )
+ self.assertCountEqual(snapshot.values_list("id", flat=True), [first.version.id, second.version.id])
+ self.assertCountEqual(snapshot.values_list("id", flat=True), before)
+ with self.assertRaises(DatasetRevision.DoesNotExist):
+ DatasetUsage.read_revision(self.dataset, first.version.dataset_revision.revision)
+
+ def test_obsolete_versions_are_deleted_and_numbers_are_not_reused(self) -> None:
+ first = self.create_item()
+ second = self.update_item(first)
+ self.age_history()
+ sweep = DatasetRetention()
+ sweep.run()
+ self.assertEqual(sweep.versions_deleted, 1)
+ self.assertFalse(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=first.version.id).exists()
+ )
+ third = self.update_item(second, "Third input")
+ self.assertEqual(third.version.version, 3)
+ self.assertEqual(third.version.dataset_revision.revision, 3)
+
+ def test_archive_marker_prevents_resurrection_in_retained_snapshot(self) -> None:
+ first = self.create_item()
+ archived = archive_dataset_item(
+ team_id=self.team.id, dataset_id=self.dataset.id, item_id=first.item.id, created_by=None, base_version=1
+ )
+ second = self.create_item("Second item")
+ restore_dataset_item(
+ team_id=self.team.id, dataset_id=self.dataset.id, item_id=first.item.id, created_by=None, base_version=2
+ )
+ self.dataset.refresh_from_db()
+ self.age_history()
+ first.version.dataset_revision.last_used_at = timezone.now()
+ first.version.dataset_revision.save(update_fields=["last_used_at"])
+ second.version.dataset_revision.last_used_at = timezone.now()
+ second.version.dataset_revision.save(update_fields=["last_used_at"])
+ before = list(
+ dataset_item_versions_at_revision(
+ team_id=self.team.id,
+ dataset_id=self.dataset.id,
+ revision=second.version.dataset_revision.revision,
+ archived=False,
+ ).values_list("id", flat=True)
+ )
+ DatasetRetention().run()
+ self.assertTrue(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=archived.version.id).exists()
+ )
+ snapshot = dataset_item_versions_at_revision(
+ team_id=self.team.id,
+ dataset_id=self.dataset.id,
+ revision=second.version.dataset_revision.revision,
+ archived=False,
+ )
+ self.assertEqual(list(snapshot.values_list("id", flat=True)), [second.version.id])
+ self.assertEqual(list(snapshot.values_list("id", flat=True)), before)
+
+ def test_using_history_protects_only_the_requested_revision(self) -> None:
+ first = self.create_item()
+ second = self.update_item(first)
+ self.update_item(second, "Third input")
+ self.age_history()
+ DatasetUsage.read_revision(self.dataset, first.version.dataset_revision.revision)
+ DatasetRetention().run()
+ self.assertTrue(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=first.version.id).exists()
+ )
+ self.assertFalse(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=second.version.id).exists()
+ )
+
+ def test_reading_item_history_preserves_its_payload(self) -> None:
+ first = self.create_item()
+ self.update_item(first)
+ self.age_history()
+ first.version.refresh_from_db()
+ DatasetUsage.record_versions(self.dataset, [first.version])
+ DatasetRetention().run()
+ self.assertTrue(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=first.version.id).exists()
+ )
+
+ def test_archiving_and_superseding_start_a_full_history_period(self) -> None:
+ first = self.create_item()
+ self.age_history()
+ self.update_item(first)
+ first.version.dataset_revision.refresh_from_db()
+ assert first.version.dataset_revision.last_used_at is not None
+ self.assertGreater(first.version.dataset_revision.last_used_at, timezone.now() - timedelta(seconds=5))
+ self.dataset.last_used_at = self.old
+ self.dataset.save()
+ archive_dataset(team_id=self.team.id, dataset_id=self.dataset.id)
+ self.dataset.refresh_from_db()
+ expires_at = DatasetRetentionPolicy.dataset_expires_at(self.dataset)
+ assert expires_at is not None
+ self.assertGreater(expires_at, timezone.now() + HISTORY_RETENTION)
+
+ def test_missing_usage_gets_a_full_grace_period(self) -> None:
+ self.dataset.last_used_at = None
+ self.dataset.save()
+ DatasetRetention().run()
+ self.dataset.refresh_from_db()
+ self.assertIsNotNone(self.dataset.last_used_at)
+ self.assertIsNone(self.dataset.expired_at)
+
+ def test_uploading_evaluation_pins_snapshot_then_completed_history_survives_cleanup(self) -> None:
+ first = self.create_item()
+ lifecycle = OfflineExperimentService(team_id=self.team.id, user_access_control=None)
+ experiment = lifecycle.create(
+ ExperimentSubmission(
+ id=uuid4(),
+ name="Retention run",
+ started_at=timezone.now(),
+ dataset_revision_id=first.version.dataset_revision_id,
+ )
+ ).experiment
+ definition = ScoreDefinition.objects.create(team=self.team, name="Quality", kind="numeric")
+ scorer = definition.create_new_version(config={}, created_by=None)
+ item_id = uuid4()
+ OfflineEvaluationIngestionService(team_id=self.team.id, user_access_control=None).upload(
+ experiment.id,
+ UploadSubmission(
+ items=[ItemSubmission(id=item_id, dataset_item_version_id=first.version.id)],
+ results=[ResultSubmission(item_id=item_id, scorer_version_id=scorer.id, status="ok", value=1.0)],
+ ),
+ )
+ self.update_item(first)
+ self.age_history()
+ first.version.dataset_revision.last_used_at = timezone.now()
+ first.version.dataset_revision.save(update_fields=["last_used_at"])
+ self.dataset.last_used_at = self.old
+ self.dataset.save()
+ DatasetRetention().run()
+ self.dataset.refresh_from_db()
+ self.assertIsNone(self.dataset.expired_at)
+ self.assertTrue(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=first.version.id).exists()
+ )
+ lifecycle.close(experiment.id, status="completed")
+ self.dataset.refresh_from_db()
+ self.assertGreater(self.dataset.last_used_at, self.old)
+ self.dataset.last_used_at = self.old
+ self.dataset.save()
+ for _ in range(3):
+ DatasetRetention().run()
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=self.dataset.id).exists())
+ experiment.refresh_from_db()
+ item = OfflineExperimentItem.objects.for_team(self.team.id).get(id=item_id)
+ self.assertIsNone(experiment.dataset_revision_id)
+ self.assertEqual(experiment.dataset_revision_identifier, str(first.version.dataset_revision_id))
+ self.assertIsNone(item.dataset_item_version_id)
+ self.assertEqual(item.dataset_item_version_identifier, str(first.version.id))
+ self.assertTrue(OfflineEvaluationResult.objects.for_team(self.team.id).filter(item=item).exists())
+ self.assertTrue(
+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).filter(experiment_id=experiment.id).exists()
+ )
+ scorer.refresh_from_db()
+
+ def test_cleanup_is_bounded_and_cannot_be_starved_by_pinned_history(self) -> None:
+ first = self.create_item()
+ self.update_item(first)
+ OfflineExperimentService(team_id=self.team.id, user_access_control=None).create(
+ ExperimentSubmission(
+ id=uuid4(),
+ name="Pinned snapshot",
+ started_at=timezone.now(),
+ dataset_revision_id=first.version.dataset_revision_id,
+ )
+ )
+ self.age_history()
+ self.dataset.retention_checked_at = None
+ self.dataset.save()
+ other = Dataset.objects.for_team(self.team.id, canonical=True).create(
+ team=self.team, name="Other", last_used_at=self.old, retention_checked_at=timezone.now() - timedelta(days=1)
+ )
+ with patch("products.ai_observability.backend.dataset_retention.MAX_DATASETS_PER_RUN", 1):
+ DatasetRetention().run()
+ self.assertTrue(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=other.id).exists())
+ DatasetRetention().run()
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=other.id).exists())
+
+ def test_expired_shared_history_no_longer_uses_a_cleanup_slot(self) -> None:
+ first = self.create_item()
+ self.create_item("Second item")
+ self.age_history()
+ DatasetRetention().run()
+ self.dataset.refresh_from_db()
+ first.version.dataset_revision.refresh_from_db()
+ self.assertIsNotNone(first.version.dataset_revision.expired_at)
+ other = Dataset.objects.for_team(self.team.id, canonical=True).create(
+ team=self.team, name="Needs cleanup", last_used_at=self.old, retention_checked_at=timezone.now()
+ )
+ checked_at = self.dataset.retention_checked_at
+ with patch("products.ai_observability.backend.dataset_retention.MAX_DATASETS_PER_RUN", 1):
+ DatasetRetention().run()
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=other.id).exists())
+ self.dataset.refresh_from_db()
+ self.assertEqual(self.dataset.retention_checked_at, checked_at)
+
+ @parameterized.expand([("active", False), ("abandoned", True)])
+ def test_upload_protection_uses_snapshot_activity(self, _name: str, abandoned: bool) -> None:
+ first = self.create_item()
+ experiment = (
+ OfflineExperimentService(team_id=self.team.id, user_access_control=None)
+ .create(
+ ExperimentSubmission(
+ id=uuid4(),
+ name="Upload activity",
+ started_at=self.old,
+ dataset_revision_id=first.version.dataset_revision_id,
+ )
+ )
+ .experiment
+ )
+ OfflineExperiment.objects.for_team(self.team.id).filter(id=experiment.id).update(created_at=self.old)
+ self.dataset.last_used_at = self.old
+ self.dataset.save(update_fields=["last_used_at"])
+ if abandoned:
+ self.age_history()
+ DatasetRetention().run()
+ self.dataset.refresh_from_db()
+ self.assertEqual(self.dataset.expired_at is not None, abandoned)
+ experiment.refresh_from_db()
+ self.assertEqual(experiment.status, OfflineExperiment.Status.UPLOADING)
+
+ def test_result_only_upload_refreshes_snapshot_activity(self) -> None:
+ first = self.create_item()
+ experiment = (
+ OfflineExperimentService(team_id=self.team.id, user_access_control=None)
+ .create(
+ ExperimentSubmission(
+ id=uuid4(),
+ name="Result batches",
+ started_at=self.old,
+ dataset_revision_id=first.version.dataset_revision_id,
+ )
+ )
+ .experiment
+ )
+ ingestion = OfflineEvaluationIngestionService(team_id=self.team.id, user_access_control=None)
+ item_id = uuid4()
+ ingestion.upload(
+ experiment.id,
+ UploadSubmission(items=[ItemSubmission(id=item_id, dataset_item_version_id=first.version.id)], results=[]),
+ )
+ definition = ScoreDefinition.objects.create(team=self.team, name="Quality", kind="numeric")
+ scorer = definition.create_new_version(config={}, created_by=None)
+ self.age_history()
+ self.dataset.last_used_at = self.old
+ self.dataset.save(update_fields=["last_used_at"])
+ ingestion.upload(
+ experiment.id,
+ UploadSubmission(
+ items=[],
+ results=[ResultSubmission(item_id=item_id, scorer_version_id=scorer.id, status="ok", value=1.0)],
+ ),
+ )
+ DatasetRetention().run()
+ first.version.dataset_revision.refresh_from_db()
+ self.dataset.refresh_from_db()
+ self.assertIsNone(self.dataset.expired_at)
+ self.assertIsNone(first.version.dataset_revision.expired_at)
+ assert first.version.dataset_revision.last_used_at is not None
+ self.assertGreater(first.version.dataset_revision.last_used_at, self.old)
+
+ def test_disabled_cleanup_preserves_pending_deletions_and_can_resume(self) -> None:
+ self.create_item()
+ self.dataset.expired_at = timezone.now()
+ self.dataset.save(update_fields=["expired_at"])
+ with override_instance_config(RETENTION_ENABLED_SETTING, False):
+ sweep = DatasetRetention()
+ sweep.run()
+ self.assertTrue(sweep.disabled)
+ self.assertTrue(DatasetItemVersion.objects.for_team(self.team.id).filter(dataset=self.dataset).exists())
+ set_instance_setting(RETENTION_ENABLED_SETTING, True)
+ DatasetRetention().run()
+ self.assertFalse(DatasetItemVersion.objects.for_team(self.team.id).filter(dataset=self.dataset).exists())
+
+ def test_disabling_during_a_sweep_stops_before_the_next_dataset(self) -> None:
+ other = Dataset.objects.for_team(self.team.id, canonical=True).create(
+ team=self.team, name="Next dataset", last_used_at=self.old, retention_checked_at=timezone.now()
+ )
+ self.dataset.last_used_at = self.old
+ self.dataset.save(update_fields=["last_used_at"])
+ clean_dataset = DatasetRetention._clean_dataset
+
+ def disable_after_cleanup(sweep: DatasetRetention, team_id: int, dataset_id: UUID) -> None:
+ clean_dataset(sweep, team_id, dataset_id)
+ set_instance_setting(RETENTION_ENABLED_SETTING, False)
+
+ with (
+ override_instance_config(RETENTION_ENABLED_SETTING, True),
+ patch.object(DatasetRetention, "_clean_dataset", disable_after_cleanup),
+ ):
+ sweep = DatasetRetention()
+ sweep.run()
+ self.assertTrue(sweep.disabled)
+ self.assertFalse(Dataset.objects.for_team(self.team.id).filter(id=self.dataset.id).exists())
+ self.assertTrue(Dataset.objects.for_team(self.team.id).filter(id=other.id).exists())
+
+ @parameterized.expand([("dataset", True), ("history", False)])
+ def test_evaluation_references_are_detached_in_batches(self, _name: str, expire_dataset: bool) -> None:
+ first = self.create_item()
+ lifecycle = OfflineExperimentService(team_id=self.team.id, user_access_control=None)
+ ingestion = OfflineEvaluationIngestionService(team_id=self.team.id, user_access_control=None)
+ experiment_ids = []
+ for index in range(3):
+ experiment = lifecycle.create(
+ ExperimentSubmission(
+ id=uuid4(),
+ name=f"Repeated evaluation {index}",
+ started_at=timezone.now(),
+ dataset_revision_id=first.version.dataset_revision_id,
+ )
+ ).experiment
+ experiment_ids.append(experiment.id)
+ ingestion.upload(
+ experiment.id,
+ UploadSubmission(
+ items=[ItemSubmission(id=uuid4(), dataset_item_version_id=first.version.id)], results=[]
+ ),
+ )
+ lifecycle.close(experiment.id, status="completed")
+ if not expire_dataset:
+ self.update_item(first)
+ self.age_history()
+ if expire_dataset:
+ self.dataset.last_used_at = self.old
+ self.dataset.save(update_fields=["last_used_at"])
+
+ items = OfflineExperimentItem.objects.for_team(self.team.id).filter(experiment_id__in=experiment_ids)
+ experiments = OfflineExperiment.objects.for_team(self.team.id).filter(id__in=experiment_ids)
+ versions = DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(id=first.version.id)
+ with patch("products.ai_observability.backend.dataset_retention.BATCH_SIZE", 2):
+ DatasetRetention().run()
+ self.assertEqual(items.filter(dataset_item_version_id=first.version.id).count(), 1)
+ self.assertTrue(versions.exists())
+ self.assertEqual(experiments.filter(dataset_revision_id=first.version.dataset_revision_id).count(), 3)
+ with patch("django.db.models.query.QuerySet.delete", side_effect=DatabaseError("Injected failure")):
+ failed_sweep = DatasetRetention()
+ failed_sweep.run()
+ self.assertEqual(failed_sweep.datasets_failed, 1)
+ self.assertEqual(items.filter(dataset_item_version_id=first.version.id).count(), 1)
+ self.assertTrue(versions.exists())
+ DatasetRetention().run()
+ self.assertFalse(versions.exists())
+ self.assertEqual(items.filter(dataset_item_version_id__isnull=True).count(), 3)
+ DatasetRetention().run()
+ self.assertEqual(
+ experiments.filter(dataset_revision_id=first.version.dataset_revision_id).count(),
+ 1 if expire_dataset else 3,
+ )
+ self.assertTrue(
+ DatasetRevision.objects.for_team(self.team.id, canonical=True)
+ .filter(id=first.version.dataset_revision_id)
+ .exists()
+ )
+ DatasetRetention().run()
+ self.assertEqual(experiments.filter(dataset_revision_id__isnull=True).count(), 3 if expire_dataset else 0)
+ self.assertEqual(items.filter(dataset_item_version_identifier=str(first.version.id)).count(), 3)
+ self.assertEqual(
+ experiments.filter(dataset_revision_identifier=str(first.version.dataset_revision_id)).count(), 3
+ )
+
+ def test_dataset_payload_deletion_is_batched_and_hidden_until_finished(self) -> None:
+ first = self.create_item()
+ second = self.create_item("Second item")
+ self.dataset.last_used_at = self.old
+ self.dataset.save()
+ with patch("products.ai_observability.backend.dataset_retention.BATCH_SIZE", 1):
+ sweep = DatasetRetention()
+ with patch("products.ai_observability.backend.dataset_retention.logger.info") as log:
+ with self.captureOnCommitCallbacks(execute=True):
+ sweep.run()
+ log.assert_not_called()
+ log.assert_called_once_with(
+ "dataset_retention_dataset_cleaned",
+ team_id=self.team.id,
+ dataset_id=str(self.dataset.id),
+ datasets_expired=1,
+ revisions_expired=0,
+ versions_deleted=1,
+ datasets_deleted=0,
+ )
+ self.assertEqual(sweep.versions_deleted, 1)
+ self.dataset.refresh_from_db()
+ self.assertIsNotNone(self.dataset.expired_at)
+ replacement = create_dataset(team=self.team, created_by=None, name=self.dataset.name)
+ self.assertNotEqual(replacement.id, self.dataset.id)
+ with self.assertRaises(DatasetMutationConflict):
+ create_dataset(team=self.team, created_by=None, name=self.dataset.name)
+ self.assertEqual(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True).filter(dataset=self.dataset).count(),
+ 1,
+ )
+ for _ in range(4):
+ DatasetRetention().run()
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=self.dataset.id).exists())
+ self.assertFalse(
+ DatasetItemVersion.objects.for_team(self.team.id, canonical=True)
+ .filter(id__in=[first.version.id, second.version.id])
+ .exists()
+ )
+
+ @parameterized.expand([(1, RuntimeError), (100, DatabaseError), (100, SoftTimeLimitExceeded)])
+ def test_deletion_failure_rolls_back_content_without_starving_cleanup(
+ self, max_datasets: int, error_type: type[Exception]
+ ) -> None:
+ first = self.create_item()
+ self.dataset.last_used_at = self.old
+ self.dataset.save()
+ other = Dataset.objects.for_team(self.team.id, canonical=True).create(
+ team=self.team, name="Another eligible dataset", last_used_at=self.old, retention_checked_at=self.old
+ )
+ sweep = DatasetRetention()
+ with (
+ patch("products.ai_observability.backend.dataset_retention.MAX_DATASETS_PER_RUN", max_datasets),
+ patch("django.db.models.query.QuerySet.delete", side_effect=error_type("Injected failure")),
+ patch("products.ai_observability.backend.dataset_retention.logger.info") as log,
+ self.captureOnCommitCallbacks(execute=True),
+ ):
+ if error_type is SoftTimeLimitExceeded:
+ with self.assertRaises(SoftTimeLimitExceeded):
+ sweep.run()
+ else:
+ sweep.run()
+ self.assertFalse(any(call.kwargs.get("dataset_id") == str(self.dataset.id) for call in log.call_args_list))
+ self.assertEqual(sweep.datasets_failed, 1)
+ self.assertEqual(sweep.versions_deleted, 0)
+ self.dataset.refresh_from_db()
+ first.item.refresh_from_db()
+ self.assertIsNone(self.dataset.expired_at)
+ self.assertEqual(first.item.current_version_id, first.version.id)
+ self.assertIsNotNone(self.dataset.retention_checked_at)
+ if error_type is DatabaseError:
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=other.id).exists())
+ else:
+ with patch("products.ai_observability.backend.dataset_retention.MAX_DATASETS_PER_RUN", 1):
+ DatasetRetention().run()
+ self.assertFalse(Dataset.objects.for_team(self.team.id, canonical=True).filter(id=other.id).exists())
+
+ @parameterized.expand([(8, False), (7, True), (1, True)])
+ def test_expiring_soon_window(self, days: int, expected: bool) -> None:
+ self.assertEqual(DatasetRetentionPolicy.expiring_soon(timezone.now() + timedelta(days=days)), expected)
+
+
+class TestDatasetRetentionConcurrency(TransactionTestCase):
+ def setUp(self) -> None:
+ _, _, self.team = Organization.objects.bootstrap(None)
+ self.dataset = Dataset.objects.for_team(self.team.id, canonical=True).create(
+ team=self.team, name="Concurrent retention", last_used_at=timezone.now() - timedelta(days=100)
+ )
+
+ @staticmethod
+ def run_cleanup() -> None:
+ try:
+ with connection.cursor() as cursor:
+ cursor.execute("SET lock_timeout = '2s'")
+ DatasetRetention().run()
+ finally:
+ connection.close()
+
+ def test_cleanup_skips_a_usage_transaction_then_rechecks_the_new_timestamp(self) -> None:
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ with transaction.atomic():
+ DatasetUsage.record(self.dataset)
+ executor.submit(self.run_cleanup).result(timeout=10)
+ executor.submit(self.run_cleanup).result(timeout=10)
+ self.dataset.refresh_from_db()
+ self.assertIsNone(self.dataset.expired_at)
+
+ def test_usage_after_cleanup_cannot_resurrect_an_expired_dataset(self) -> None:
+ create_dataset_item(team_id=self.team.id, dataset_id=self.dataset.id, created_by=None, input="Example")
+ Dataset.objects.for_team(self.team.id, canonical=True).filter(id=self.dataset.id).update(
+ last_used_at=self.dataset.last_used_at
+ )
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ executor.submit(self.run_cleanup).result(timeout=10)
+ with self.assertRaises(Dataset.DoesNotExist):
+ DatasetUsage.record(self.dataset)
+ self.dataset.refresh_from_db()
+ self.assertIsNotNone(self.dataset.expired_at)
+
+
+@time_machine.travel("2026-11-03T12:00:00Z", tick=False)
+class TestDatasetRetentionMigration(TransactionTestCase):
+ def test_existing_content_receives_a_full_retention_period(self) -> None:
+ _, _, team = Organization.objects.bootstrap(None)
+ executor = MigrationExecutor(connection)
+ before = [("ai_observability", "0070_offline_payload_expiry_indexes")]
+ after = [("ai_observability", "0074_release_expired_dataset_names")]
+ executor.migrate(before)
+ try:
+ apps = executor.loader.project_state(before).apps
+ legacy_dataset = apps.get_model("ai_observability", "Dataset").objects.create(
+ team_id=team.id, name="Existing dataset"
+ )
+ legacy_revision = apps.get_model("ai_observability", "DatasetRevision").objects.create(
+ team_id=team.id, dataset_id=legacy_dataset.id, revision=1
+ )
+ legacy_item = apps.get_model("ai_observability", "DatasetItem").objects.create(
+ team_id=team.id, dataset_id=legacy_dataset.id
+ )
+ legacy_version = apps.get_model("ai_observability", "DatasetItemVersion").objects.create(
+ team_id=team.id,
+ dataset_id=legacy_dataset.id,
+ dataset_item_id=legacy_item.id,
+ dataset_revision_id=legacy_revision.id,
+ version=1,
+ input="Existing input",
+ )
+ started = timezone.now()
+ MigrationExecutor(connection).migrate(after)
+ dataset = Dataset.objects.for_team(team.id, canonical=True).get(id=legacy_dataset.id)
+ revision = DatasetRevision.objects.for_team(team.id, canonical=True).get(id=legacy_revision.id)
+ version = DatasetItemVersion.objects.for_team(team.id, canonical=True).get(id=legacy_version.id)
+ assert dataset.last_used_at is not None
+ assert revision.last_used_at is not None
+ assert version.last_used_at is not None
+ self.assertGreaterEqual(dataset.last_used_at, started)
+ self.assertGreaterEqual(revision.last_used_at, started)
+ self.assertGreaterEqual(version.last_used_at, started)
+ DatasetRetention().run()
+ self.assertTrue(DatasetItemVersion.objects.for_team(team.id, canonical=True).filter(id=version.id).exists())
+ with self.assertRaises(IntegrityError), transaction.atomic():
+ Dataset.objects.for_team(team.id, canonical=True).create(team_id=team.id, name=dataset.name)
+ dataset.expired_at = timezone.now()
+ dataset.save(update_fields=["expired_at"])
+ replacement = Dataset.objects.for_team(team.id, canonical=True).create(team_id=team.id, name=dataset.name)
+ self.assertNotEqual(dataset.id, replacement.id)
+ finally:
+ MigrationExecutor(connection).migrate(after)
diff --git a/products/ai_observability/backend/test/test_dataset_retention_admin.py b/products/ai_observability/backend/test/test_dataset_retention_admin.py
new file mode 100644
index 000000000000..2c66d1a71e01
--- /dev/null
+++ b/products/ai_observability/backend/test/test_dataset_retention_admin.py
@@ -0,0 +1,41 @@
+from posthog.test.base import BaseTest
+
+from django.urls import reverse
+
+from parameterized import parameterized
+
+from posthog.models.instance_setting import override_instance_config
+
+from products.ai_observability.backend.dataset_retention import RETENTION_ENABLED_SETTING, dataset_retention_enabled
+
+
+class TestDatasetRetentionAdmin(BaseTest):
+ @parameterized.expand([("enabled", True), ("disabled", False)])
+ def test_staff_can_change_cleanup_setting(self, value: str, expected: bool) -> None:
+ self.user.is_staff = True
+ self.user.save(update_fields=["is_staff"])
+ self.client.force_login(self.user)
+ url = reverse("admin:ai-observability-dataset-retention")
+ with override_instance_config(RETENTION_ENABLED_SETTING, not expected):
+ response = self.client.get(url)
+ self.assertContains(response, "AI observability dataset retention")
+ self.assertEqual(dataset_retention_enabled(), not expected)
+ response = self.client.post(url, {"enabled": value}, follow=True)
+ self.assertEqual(response.status_code, 200)
+ self.assertEqual(dataset_retention_enabled(), expected)
+
+ def test_nonstaff_cannot_change_cleanup_setting(self) -> None:
+ self.client.force_login(self.user)
+ with override_instance_config(RETENTION_ENABLED_SETTING, True):
+ response = self.client.post(reverse("admin:ai-observability-dataset-retention"), {"enabled": "disabled"})
+ self.assertEqual(response.status_code, 302)
+ self.assertTrue(dataset_retention_enabled())
+
+ def test_invalid_setting_does_not_disable_cleanup(self) -> None:
+ self.user.is_staff = True
+ self.user.save(update_fields=["is_staff"])
+ self.client.force_login(self.user)
+ with override_instance_config(RETENTION_ENABLED_SETTING, True):
+ response = self.client.post(reverse("admin:ai-observability-dataset-retention"), {"enabled": "invalid"})
+ self.assertEqual(response.status_code, 200)
+ self.assertTrue(dataset_retention_enabled())
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx
index 709afd51fa93..c27d569c58a7 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx
@@ -19,6 +19,8 @@ const dataset: DatasetReadApi = {
metadata: { owner: 'support', language: 'en' },
tags: [],
archived: false,
+ expires_at: '2024-04-01T12:00:00Z',
+ expiring_soon: false,
current_revision: 1,
current_revision_id: 'example-revision',
created_at: '2024-01-01T00:00:00Z',
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx
index 45b787a53cea..655e47bca1b4 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx
@@ -273,9 +273,9 @@ export function AIObservabilityDatasetScene(): JSX.Element {
<h2 className="text-base font-semibold m-0 truncate" title={datasetForm.name}>
{isNewDataset ? 'New dataset' : (isDataset(dataset) && dataset.name) || 'Dataset'}
</h2>
- {isDataset(dataset) && dataset.archived && (
- <LemonTag type="muted" size="small">
- Archived
+ {isDataset(dataset) && (dataset.expiring_soon || dataset.archived) && (
+ <LemonTag type={dataset.expiring_soon ? 'warning' : 'muted'} size="small">
+ {dataset.expiring_soon ? 'Expiring soon' : 'Archived'}
</LemonTag>
)}
</nav>
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
index 77181290a7e4..f98e302c11ec 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
@@ -88,10 +88,11 @@ export function AIObservabilityDatasetsScene({ navigation }: { navigation?: Reac
{
title: 'Status',
key: 'status',
+ width: 112,
render: function renderStatus(_, dataset) {
return (
- <LemonTag type={dataset.archived ? 'muted' : 'success'}>
- {dataset.archived ? 'Archived' : 'Active'}
+ <LemonTag type={dataset.expiring_soon ? 'warning' : dataset.archived ? 'muted' : 'success'}>
+ {dataset.expiring_soon ? 'Expiring soon' : dataset.archived ? 'Archived' : 'Active'}
</LemonTag>
)
},
diff --git a/products/ai_observability/frontend/datasets/DatasetItems.test.tsx b/products/ai_observability/frontend/datasets/DatasetItems.test.tsx
index 067ae3493acb..9706b8fa0e64 100644
--- a/products/ai_observability/frontend/datasets/DatasetItems.test.tsx
+++ b/products/ai_observability/frontend/datasets/DatasetItems.test.tsx
@@ -41,6 +41,8 @@ const DATASET: Dataset = {
updated_at: TIMESTAMP,
created_by: null,
archived: false,
+ expires_at: null,
+ expiring_soon: false,
current_revision: null,
current_revision_id: null,
user_access_level: 'editor',
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
index 311bd009e18c..62049623815a 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
@@ -58,6 +58,8 @@ describe('aiObservabilityDatasetLogic', () => {
updated_at: '2024-01-01T00:00:00Z',
created_by: null,
archived: false,
+ expires_at: null,
+ expiring_soon: false,
current_revision: null,
current_revision_id: null,
user_access_level: 'editor',
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts
index 4e018875b096..0b855c775eb8 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts
@@ -986,7 +986,10 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(
{ value: 'latest', label: 'Latest' },
...revisionNumbers
.sort((a, b) => b - a)
- .map((revision) => ({ value: String(revision), label: `Revision ${revision}` })),
+ .map((revision) => ({
+ value: String(revision),
+ label: `Revision ${revision}${revisions.results.find((entry) => entry.revision === revision)?.expiring_soon ? ' · Expiring soon' : ''}`,
+ })),
]
},
],
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
index b948a52f776a..fb660000826e 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
@@ -40,6 +40,8 @@ describe('aiObservabilityDatasetsLogic', () => {
updated_at: '2024-01-01T00:00:00Z',
created_by: null,
archived: false,
+ expires_at: null,
+ expiring_soon: false,
current_revision: null,
current_revision_id: null,
user_access_level: 'editor',
@@ -56,6 +58,8 @@ describe('aiObservabilityDatasetsLogic', () => {
updated_at: '2024-01-02T00:00:00Z',
created_by: null,
archived: false,
+ expires_at: null,
+ expiring_soon: false,
current_revision: null,
current_revision_id: null,
user_access_level: 'editor',
diff --git a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
index cd712c5a2b90..ef43eb8cb079 100644
--- a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
@@ -41,6 +41,8 @@ describe('saveToDatasetButtonLogic', () => {
updated_at: '2024-01-01T00:00:00Z',
created_by: null,
archived: false,
+ expires_at: null,
+ expiring_soon: false,
current_revision: null,
current_revision_id: null,
user_access_level: 'editor',
@@ -57,6 +59,8 @@ describe('saveToDatasetButtonLogic', () => {
updated_at: '2024-01-02T00:00:00Z',
created_by: null,
archived: false,
+ expires_at: null,
+ expiring_soon: false,
current_revision: null,
current_revision_id: null,
user_access_level: 'editor',
diff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.ts
index 1d6f1b50ec26..fa2923423782 100644
--- a/products/ai_observability/frontend/generated/api.schemas.ts
+++ b/products/ai_observability/frontend/generated/api.schemas.ts
@@ -2111,6 +2111,13 @@ export interface DatasetReadApi {
/** Project tag names assigned to this resource. */
readonly tags: readonly string[]
readonly archived: boolean
+ /**
+ * Earliest cleanup time if the dataset remains unused.
+ * @nullable
+ */
+ readonly expires_at: string | null
+ /** Whether the dataset is within seven days of expiry. */
+ readonly expiring_soon: boolean
/**
* Latest dataset revision, or null before the first item is added.
* @nullable
@@ -2237,6 +2244,13 @@ export interface DatasetRevisionReadApi {
readonly created_by: UserBasicApi | null
/** Project that owns the revision. */
readonly team_id: number
+ /**
+ * Earliest expiry time of this dataset snapshot if unused.
+ * @nullable
+ */
+ readonly expires_at: string | null
+ /** Whether this snapshot is within seven days of expiry. */
+ readonly expiring_soon: boolean
}
export interface PaginatedDatasetRevisionReadListApi {
diff --git a/products/ai_observability/mcp/tools.yaml b/products/ai_observability/mcp/tools.yaml
index d516dae5b5a3..63845e730f1f 100644
--- a/products/ai_observability/mcp/tools.yaml
+++ b/products/ai_observability/mcp/tools.yaml
@@ -121,8 +121,9 @@ tools:
enrich_url: '{id}'
title: Archive dataset
description: >
- Archive a dataset by ID. Its revisions and items remain readable, but item mutations are rejected until the
- dataset is restored. Archiving an already archived dataset leaves it unchanged.
+ Archive a dataset by ID. Its revisions and items remain readable until expiry, but item mutations are
+ rejected until the dataset is restored. Archived datasets and historical revisions expire after 30 days of
+ inactivity; active datasets expire after 90 days. Archiving an already archived dataset leaves it unchanged.
feature_flag: llm-analytics-datasets
url_prefix: /ai-evals/datasets
response:
@@ -305,9 +306,10 @@ tools:
list: true
title: List dataset item versions
description: >
- List every immutable version of a dataset item, newest first. The response includes version metadata only.
- To retrieve a version's content, call `llma-dataset-item-get` with the item ID and its `dataset_revision` as
- `revision`.
+ List retained immutable versions of a dataset item, newest first. Unused history expires after 30 days. The
+ response includes version metadata only. To read content, call `llma-dataset-item-get` with a retained
+ revision containing the version. Its introducing `dataset_revision` may have expired even when a later
+ snapshot still needs the version; expired revisions return 404.
feature_flag: llm-analytics-datasets
response:
include:
@@ -337,7 +339,8 @@ tools:
description: >
List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set
`archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all
- (AND).
+ (AND). Active datasets expire after 90 days of inactivity; archived datasets expire after 30 days. Listing
+ datasets does not refresh retention. Expired datasets are omitted.
feature_flag: llm-analytics-datasets
url_prefix: /ai-evals/datasets
response:
@@ -348,6 +351,8 @@ tools:
- tags
- archived
- current_revision
+ - expires_at
+ - expiring_soon
- created_at
- updated_at
- user_access_level
@@ -386,8 +391,9 @@ tools:
list: true
title: List dataset revisions
description: >
- List immutable revisions for a dataset by UUID, newest first. Use a revision number with the dataset item
- list tool to inspect that exact snapshot.
+ List available immutable revisions for a dataset by UUID, newest first. Historical revisions expire after 30
+ days of inactivity and are omitted. Listing does not refresh retention. Use a revision number with the
+ dataset item list tool to inspect that exact snapshot and refresh its retention.
feature_flag: llm-analytics-datasets
url_prefix: /ai-evals/datasets
response:
@@ -395,6 +401,8 @@ tools:
- id
- dataset_id
- revision
+ - expires_at
+ - expiring_soon
- created_at
- created_by
informational_wrapper:
diff --git a/products/posthog_ai/skills/querying-posthog-data/references/models-datasets.md.j2 b/products/posthog_ai/skills/querying-posthog-data/references/models-datasets.md.j2
index 33afefe5103f..192fa054f49d 100644
--- a/products/posthog_ai/skills/querying-posthog-data/references/models-datasets.md.j2
+++ b/products/posthog_ai/skills/querying-posthog-data/references/models-datasets.md.j2
@@ -4,6 +4,13 @@ Datasets hold curated inputs, expected outputs, and optional trace provenance fo
Dataset items have stable IDs and immutable content versions.
Every item mutation creates a dataset revision, which makes prior dataset contents queryable as exact snapshots.
+Active datasets expire after 90 days of inactivity; archived datasets and historical revisions expire after 30 days,
+with a 12-hour usage recording margin. Expired datasets and revisions may remain in these tables during cleanup.
+Only datasets and requested snapshots with `expired_at IS NULL` are available; expired snapshots can be incomplete.
+HogQL reads do not refresh retention. Use the dataset API or MCP item-read tools to refresh a snapshot's retention.
+Expired snapshots return 404 through those tools. Dataset names can be reused after expiry, so select by UUID or
+filter out expired datasets when selecting by name.
+
## Dataset (`system.datasets`)
{{ schema_columns('system.datasets') }}
@@ -52,8 +59,10 @@ SELECT
v.source_output,
v.metadata
FROM system.dataset_items AS i
+INNER JOIN system.datasets AS d ON d.id = i.dataset_id
INNER JOIN system.dataset_item_versions AS v ON v.id = i.current_version_id
WHERE i.dataset_id = '01234567-89ab-cdef-0123-456789abcdef'
+ AND d.expired_at IS NULL
AND v.archived = false
ORDER BY i.created_at DESC
LIMIT 100
@@ -61,6 +70,9 @@ LIMIT 100
Reconstruct active items at dataset revision 7:
+Check the requested snapshot's expiry, not the expiry of each version's introducing revision. A version introduced
+by an expired revision can still be needed by an available snapshot.
+
```sql
SELECT
i.id,
@@ -71,28 +83,34 @@ SELECT
tupleElement(argMax(tuple(v.source_output), r.revision), 1) AS source_output,
argMax(v.metadata, r.revision) AS metadata
FROM system.dataset_items AS i
+INNER JOIN system.datasets AS d ON d.id = i.dataset_id
+INNER JOIN system.dataset_revisions AS snapshot ON snapshot.dataset_id = i.dataset_id AND snapshot.revision = 7
INNER JOIN system.dataset_item_versions AS v ON v.dataset_item_id = i.id
INNER JOIN system.dataset_revisions AS r ON r.id = v.dataset_revision_id
WHERE i.dataset_id = '01234567-89ab-cdef-0123-456789abcdef'
+ AND d.expired_at IS NULL
+ AND snapshot.expired_at IS NULL
AND r.revision <= 7
GROUP BY i.id, i.client_item_id
HAVING argMax(v.archived, r.revision) = false
ORDER BY i.id
```
-List the immutable history of one item:
+List the retained immutable history of one item in an available dataset:
```sql
SELECT
- version,
- archived,
- input,
- expected_output,
- metadata,
- dataset_revision_id,
- created_by_id,
- created_at
-FROM system.dataset_item_versions
-WHERE dataset_item_id = '01234567-89ab-cdef-0123-456789abcdef'
-ORDER BY version DESC
+ v.version,
+ v.archived,
+ v.input,
+ v.expected_output,
+ v.metadata,
+ v.dataset_revision_id,
+ v.created_by_id,
+ v.created_at
+FROM system.dataset_item_versions AS v
+INNER JOIN system.datasets AS d ON d.id = v.dataset_id
+WHERE v.dataset_item_id = '01234567-89ab-cdef-0123-456789abcdef'
+ AND d.expired_at IS NULL
+ORDER BY v.version DESC
```
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index 50d1d6fee604..8ab0cfbbc73c 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -8759,7 +8759,7 @@
}
},
"llma-dataset-archive": {
- "description": "Archive a dataset by ID. Its revisions and items remain readable, but item mutations are rejected until the dataset is restored. Archiving an already archived dataset leaves it unchanged.",
+ "description": "Archive a dataset by ID. Its revisions and items remain readable until expiry, but item mutations are rejected until the dataset is restored. Archived datasets and historical revisions expire after 30 days of inactivity; active datasets expire after 90 days. Archiving an already archived dataset leaves it unchanged.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Archive dataset",
@@ -8903,7 +8903,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-item-version-list": {
- "description": "List every immutable version of a dataset item, newest first. The response includes version metadata only. To retrieve a version's content, call `llma-dataset-item-get` with the item ID and its `dataset_revision` as `revision`.",
+ "description": "List retained immutable versions of a dataset item, newest first. Unused history expires after 30 days. The response includes version metadata only. To read content, call `llma-dataset-item-get` with a retained revision containing the version. Its introducing `dataset_revision` may have expired even when a later snapshot still needs the version; expired revisions return 404.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List dataset item versions",
@@ -8919,7 +8919,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-list": {
- "description": "List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND).",
+ "description": "List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND). Active datasets expire after 90 days of inactivity; archived datasets expire after 30 days. Listing datasets does not refresh retention. Expired datasets are omitted.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List datasets",
@@ -8951,7 +8951,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-revision-list": {
- "description": "List immutable revisions for a dataset by UUID, newest first. Use a revision number with the dataset item list tool to inspect that exact snapshot.",
+ "description": "List available immutable revisions for a dataset by UUID, newest first. Historical revisions expire after 30 days of inactivity and are omitted. Listing does not refresh retention. Use a revision number with the dataset item list tool to inspect that exact snapshot and refresh its retention.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List dataset revisions",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index c6ed92ba23e0..f648462b5f6d 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -9025,7 +9025,7 @@
}
},
"llma-dataset-archive": {
- "description": "Archive a dataset by ID. Its revisions and items remain readable, but item mutations are rejected until the dataset is restored. Archiving an already archived dataset leaves it unchanged.",
+ "description": "Archive a dataset by ID. Its revisions and items remain readable until expiry, but item mutations are rejected until the dataset is restored. Archived datasets and historical revisions expire after 30 days of inactivity; active datasets expire after 90 days. Archiving an already archived dataset leaves it unchanged.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Archive dataset",
@@ -9169,7 +9169,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-item-version-list": {
- "description": "List every immutable version of a dataset item, newest first. The response includes version metadata only. To retrieve a version's content, call `llma-dataset-item-get` with the item ID and its `dataset_revision` as `revision`.",
+ "description": "List retained immutable versions of a dataset item, newest first. Unused history expires after 30 days. The response includes version metadata only. To read content, call `llma-dataset-item-get` with a retained revision containing the version. Its introducing `dataset_revision` may have expired even when a later snapshot still needs the version; expired revisions return 404.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List dataset item versions",
@@ -9185,7 +9185,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-list": {
- "description": "List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND).",
+ "description": "List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND). Active datasets expire after 90 days of inactivity; archived datasets expire after 30 days. Listing datasets does not refresh retention. Expired datasets are omitted.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List datasets",
@@ -9217,7 +9217,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-revision-list": {
- "description": "List immutable revisions for a dataset by UUID, newest first. Use a revision number with the dataset item list tool to inspect that exact snapshot.",
+ "description": "List available immutable revisions for a dataset by UUID, newest first. Historical revisions expire after 30 days of inactivity and are omitted. Listing does not refresh retention. Use a revision number with the dataset item list tool to inspect that exact snapshot and refresh its retention.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List dataset revisions",
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 7bb51a70c910..7fb573aabe9f 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -33948,6 +33948,13 @@ export namespace Schemas {
/** Project tag names assigned to this resource. */
readonly tags: readonly string[];
readonly archived: boolean;
+ /**
+ * Earliest cleanup time if the dataset remains unused.
+ * @nullable
+ */
+ readonly expires_at: string | null;
+ /** Whether the dataset is within seven days of expiry. */
+ readonly expiring_soon: boolean;
/**
* Latest dataset revision, or null before the first item is added.
* @nullable
@@ -33980,6 +33987,13 @@ export namespace Schemas {
readonly created_by: UserBasic | null;
/** Project that owns the revision. */
readonly team_id: number;
+ /**
+ * Earliest expiry time of this dataset snapshot if unused.
+ * @nullable
+ */
+ readonly expires_at: string | null;
+ /** Whether this snapshot is within seven days of expiry. */
+ readonly expiring_soon: boolean;
}
export interface DayItem {
diff --git a/services/mcp/src/tools/generated/ai_observability.ts b/services/mcp/src/tools/generated/ai_observability.ts
index ea3b6f69a3e6..50163a2e3c74 100644
--- a/services/mcp/src/tools/generated/ai_observability.ts
+++ b/services/mcp/src/tools/generated/ai_observability.ts
@@ -610,6 +610,8 @@ const llmaDatasetList = (): ToolBase<
'tags',
'archived',
'current_revision',
+ 'expires_at',
+ 'expiring_soon',
'created_at',
'updated_at',
'user_access_level',
@@ -674,7 +676,15 @@ const llmaDatasetRevisionList = (): ToolBase<
const filtered = {
...result,
results: (result.results ?? []).map((item: any) =>
- pickResponseFields(item, ['id', 'dataset_id', 'revision', 'created_at', 'created_by'])
+ pickResponseFields(item, [
+ 'id',
+ 'dataset_id',
+ 'revision',
+ 'expires_at',
+ 'expiring_soon',
+ 'created_at',
+ 'created_by',
+ ])
),
} as typeof result
return withInformationalResponse(