:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): report offline evaluation usage (#124862)
Radu Raiceaf57bf86b8003
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 1f1ce97fcc7f..c74f74c302a3 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -391,6 +391,8 @@ Each upload must contain a result, and every declared item must be referenced byRequests exceeding the body limit return HTTP 413.Per-caller and shared project limits allow 60 requests per minute and 1,000 per hour; HTTP 429 responses include retry guidance.The shared project limits apply across the parent project and all its child environments, regardless of which credentials each request uses.+Uploads containing new results also check the shared AI observability billing quota before writing items, results, payloads, or usage.+While the quota is active, these uploads return HTTP 402 with code `quota_limit_exceeded`; exact retries remain available and add no usage.The optional `run_source` accepts `ci`, `local`, `scheduled`, or null; empty strings are invalid.For hosted datasets, set `dataset_revision_id` to the UUID of a hosted dataset revision when you create the experiment.@@ -399,7 +401,7 @@ Local and external datasets do not require hosted links; they can use the `*_ideLarge payloads have separate storage and 30-day deadlines anchored to first acceptance.Retries neither extend deadlines nor restore deleted payloads.-Automatic payload deletion and usage billing are not enabled by these endpoints.+Automatic payload deletion is not enabled by these endpoints.### Product usage events@@ -435,13 +437,60 @@ Exact retries add no usage, including retries split or regrouped into differentMixed uploads count only new results, and changed content still returns a conflict.Each entry stores the environment's team ID, the owning organization's ID at acceptance, the experiment ID, and the same server acceptance timestamp as its new results.-These identifiers are stored independently of the product records, so deleting results, experiments, or their owners does not remove usage.+These identifiers are stored independently of the product records, so deleting results, experiments, or their owners does not remove ledger entries.Moving a project to another organization does not rewrite historical attribution.-The ownership snapshots and timestamp preserve the information needed for future aggregation by organization, team, and UTC day.Recording begins with newly accepted results after deployment; there is no historical backfill, compaction, or ledger cleanup.-The ledger has no editing API or editable admin interface.-Organization usage reporting, billing, actual charging, and quota enforcement remain unchanged and do not consume this ledger.+The ledger has no editing API, admin registration, or production update/delete path.+The database enforces positive quantities and result identity uniqueness; experiment row locks serialize overlapping uploads, and database transactions include results, payloads, and the ledger write.+Independent scalar identifiers prevent product deletion cascades from deleting usage.+`editable=False`, environment scoping, and server-assigned fields are application protections, not database immutability: privileged SQL and ORM bulk updates/deletes can still change the ledger.++Retry deduplication uses retained result identities, not the batch ledger. Payload expiry preserves these identities.+There is no result/experiment deletion endpoint. Privileged deletion of a result followed by reupload while its experiment is still open creates a new result and another usage unit; the ledger cannot identify the old metric after deletion.+Operators must not delete durable result identities and then replay uploads as a billing-neutral operation. Any future identity cleanup needs an explicit replay policy before deployment.++### Usage reporting and shared AIO meter++The `organization usage report` exposes:++| Property | Meaning |+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ |+| `ai_ingested_event_count_in_period` | Billing-eligible ingested AIO events, preserving wallet-billed gateway and sponsored-event exemptions. |+| `ai_offline_eval_metric_count_in_period` | Sum of eligible offline ledger quantities accepted in the reporting window. |+| `ai_event_count_in_period` | The sum of the two components. |++Organization reports include the same breakdown in each `teams` entry, including existing environments that have moved. A moved environment can appear under both organizations, with only the usage attributed to each.+`team_count` continues to count live billable teams; historical entries do not increase it.+The existing `ai_event_count_in_period` → `llm_events` contract remains the shared AIO meter. Reporting does not deduct the standard 100,000-unit monthly allowance or determine prices or custom allowances; billing owns those calculations.++The `llm analytics usage` event adds `ai_offline_eval_metric_count_in_period` from the same offline ledger aggregation. Its existing `ai_metric_count`, event-type analytics, traces, costs, tokens, and dimension breakdowns remain ClickHouse-derived. Billing-eligible ingested counts and combined billing totals belong to the organization usage report; analytics reporting does not run an additional billing scan.++`get_offline_evaluation_usage` performs database-side `SUM(quantity)` grouped by the stored organization and environment, with `accepted_at >= period_start AND accepted_at < period_end`.+Eligibility follows current team records, like ingested AI events: deleted teams, demo teams, and teams in internal-metrics organizations are excluded. The stored organization must also still exist and not be internal-metrics.+The ledger retains all accepted usage; reporting applies these exclusions without recording a separate eligibility field.+It uses timezone-aware UTC windows and never counts ledger rows or current results.+Scheduled daily reports and current-day quota refresh query the offline ledger with midnight-to-midnight windows, including the final microsecond of the day.+The exclusive day end applies only to the offline ledger query; shared report metadata and other usage queries retain their existing windows.+The existing acceptance-time index supports the range filter. Team and organization eligibility use SQL subqueries in the same aggregation query. Only grouped totals reach Python; there is no query per metric, team, or organization.++Celery and Temporal compose offline quantities after selecting the ingested counter source, so source switching and shadow comparisons remain about ingested events only.+Temporal queries the ledger in its aggregation activity before emitting chunks; existing query files and activity inputs need no new serialized counter.+Offline-only organizations reach the normal delivery path when they have eligible usage. Stored attribution can include an organization whose team has since moved elsewhere.+Deleted teams' usage is omitted from newly generated reports, even though its ledger entries remain. Deleted organizations are also omitted; no customer or organization is recreated.++Current-day quota refresh adds eligible offline usage to `llm_events.todays_usage` by the stored organization, using the same current-team exclusions.+It leaves the prior-period `usage` received from billing intact. Ledger failures abort before quota updates or report delivery, allowing existing retries without publishing an incomplete total.+Organizations included only through offline usage are read in bounded, buffered pages so database connection cleanup between quota batches cannot interrupt the scan.+Reports are absolute totals: rereading an unchanged window does not consume or modify ledger entries. Existing analytics dispatch claims and already-reported checks still apply.+Separately scheduled reports can observe different late-arriving data; identical windows and ledger snapshots produce matching offline metric totals.++### Deployment++Offline usage reporting uses the requested reporting window without a separate charging-start date or activation setting.+The reporting integration uses the existing ledger schema without an additional migration.+Temporal computes the offline ledger's exclusive day end inside the aggregation activity.+The shared workflow window and already serialized reports are unchanged.## Resource tagsdiff --git a/ee/billing/quota_limiting.py b/ee/billing/quota_limiting.pyindex fa4c0d1b1dfc..00756b64b408 100644--- a/ee/billing/quota_limiting.py+++ b/ee/billing/quota_limiting.py@@ -1,6 +1,6 @@import copyimport json-from collections.abc import Callable, Collection, Iterable, Mapping, Sequence+from collections.abc import Callable, Collection, Iterable, Iterator, Mapping, Sequencefrom datetime import datetime, timedeltafrom enum import Enumfrom functools import lru_cache, partial@@ -41,6 +41,8 @@ from posthog.tasks.usage_report import (from posthog.usage_counters import UsageCounter, UsageCounterService, log_usage_counter_comparisonsfrom posthog.utils import get_current_day+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage+from ee.billing.quota_usage import QuotaUsageBatch, retry_quota_database_operationlogger = structlog.get_logger(__name__)@@ -1245,7 +1247,13 @@ class _QuotaUsageReports:team_tokens: dict[str, list[str]]-def _aggregate_quota_usage(teams: Sequence[Team], all_data: dict[str, dict[int, int]]) -> _QuotaUsageReports:+def _aggregate_quota_usage(+ teams: Sequence[Team],+ all_data: dict[str, dict[int, int]],+ *,+ offline_usage: dict[str, dict[int, int]] | None = None,+ organizations: Sequence[Organization] = (),+) -> _QuotaUsageReports:todays_usage_report: dict[str, UsageCounters] = {}orgs_by_id: dict[str, Organization] = {}teams_by_org: dict[str, list[str]] = {}@@ -1298,9 +1306,38 @@ def _aggregate_quota_usage(teams: Sequence[Team], all_data: dict[str, dict[int,if team.api_token:teams_by_org.setdefault(org_id, []).append(team.api_token)+ for organization in organizations:+ org_id = str(organization.id)+ orgs_by_id.setdefault(org_id, organization)+ todays_usage_report.setdefault(org_id, cast(UsageCounters, dict.fromkeys(UsageCounters.__annotations__, 0)))+ for org_id, report in todays_usage_report.items():+ report["llm_events"] += sum((offline_usage or {}).get(org_id, {}).values())return _QuotaUsageReports(usage=todays_usage_report, organizations=orgs_by_id, team_tokens=teams_by_org)+def _iter_quota_organizations(+ teams: Iterable[Team], offline_usage: dict[str, dict[int, int]]+) -> Iterator[tuple[Organization, list[Team]]]:+ seen: set[str] = set()+ for organization_id, group in groupby(teams, key=lambda team: team.organization_id):+ org_teams = list(group)+ seen.add(str(organization_id))+ yield org_teams[0].organization, org_teams+ organizations = (+ Organization.objects.filter(id__in=offline_usage.keys() - seen)+ .only("id", "usage", "created_at", "never_drop_data", "customer_trust_scores", "customer_id")+ .order_by("id")+ )+ # Buffer each page because quota updates close database connections between batches.+ page = organizations+ while batch := list(page[:QUOTA_ORGANIZATION_BATCH_SIZE]):+ for organization in batch:+ yield organization, []+ if len(batch) < QUOTA_ORGANIZATION_BATCH_SIZE:+ return+ page = organizations.filter(id__gt=batch[-1].id)++def update_all_orgs_billing_quotas(dry_run: bool = False,progress_callback: Callable[[str, str, str], None] | None = None,@@ -1345,6 +1382,9 @@ def update_all_orgs_billing_quotas(tag_queries(product=Product.BILLING, feature=Feature.QUOTA_LIMITING)logger.info("quota_limiting_run", phase="queries", status="start")queries_start = time()+ offline_usage = _timed_query(+ "offline_evaluation_usage", get_offline_evaluation_usage, period.start, period.start + timedelta(days=1)+ )counter_report = _timed_query("usage_counters", service.fetch_report, period, plan=plan)log_usage_counter_comparisons(counter_report, caller="quota_limiting", date=period.start.date().isoformat())@@ -1467,12 +1507,14 @@ def update_all_orgs_billing_quotas(quota_limiting_suspended_orgs[resource.value].pop(org_id, None)# The shared iterator keeps an organization's teams contiguous, including across read pages.- organization_teams = (list(group) for _, group in groupby(teams_iterator, key=lambda team: team.organization_id))+ organization_teams = _iter_quota_organizations(teams_iterator, offline_usage)for team_groups in batched(organization_teams, QUOTA_ORGANIZATION_BATCH_SIZE, strict=False):if not settings.TEST:close_old_connections()- teams = [team for group in team_groups for team in group]- reports = _aggregate_quota_usage(teams, all_data)+ teams = [team for _, group in team_groups for team in group]+ reports = _aggregate_quota_usage(+ teams, all_data, offline_usage=offline_usage, organizations=[org for org, _ in team_groups]+ )orgs_by_id = reports.organizationsteams_by_org = reports.team_tokenscandidates_start = time()diff --git a/ee/billing/test/test_quota_limiting.py b/ee/billing/test/test_quota_limiting.pyindex 2776c77ef9ff..c6709f0fa1eb 100644--- a/ee/billing/test/test_quota_limiting.py+++ b/ee/billing/test/test_quota_limiting.py@@ -23,6 +23,8 @@ from posthog.redis import get_clientfrom posthog.test.fixtures import create_app_metric2from posthog.usage_counters import UsageRecordTotal, UsageRecordTotals+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger+from ee.billing.quota_limiting import (INFORMATIONAL_USAGE_RESOURCES,QUOTA_LIMIT_DATA_RETENTION_FLAG,@@ -87,6 +89,30 @@ class TestQuotaLimiting(BaseTest):with self.captureOnCommitCallbacks(execute=True):return update_all_orgs_billing_quotas()+ @parameterized.expand([(False,), (True,)])+ @time_machine.travel("2026-10-31T23:59:59.999999Z", tick=False)+ def test_offline_usage_updates_today_without_recounting_billing_history(self, deleted_team: bool) -> None:+ self.organization.usage = {+ "period": ["2026-10-01T00:00:00Z", "2026-11-01T00:00:00Z"],+ "llm_events": {"usage": 50, "todays_usage": 0, "limit": 1000},+ }+ self.organization.save(update_fields=["usage"])+ for accepted_at, quantity in ((now() - datetime.timedelta(days=1), 50), (now(), 20), (now(), 10)):+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).create(+ team_id=self.team.id,+ organization_id=self.organization.id,+ experiment_id=uuid4(),+ accepted_at=accepted_at,+ quantity=quantity,+ )+ if deleted_team:+ self.team.delete()+ for _ in range(2):+ self._run_quota_limiting()+ self.organization.refresh_from_db()+ self.assertEqual(self.organization.usage["llm_events"]["usage"], 50)+ self.assertEqual(self.organization.usage["llm_events"]["todays_usage"], 0 if deleted_team else 30)+@patch("posthoganalytics.capture")@patch("posthoganalytics.feature_enabled", return_value=True)@time_machine.travel("2021-01-25T23:59:59Z", tick=False)@@ -264,7 +290,7 @@ class TestQuotaLimiting(BaseTest):self.organization.save()flush_persons_and_events()- with self.assertNumQueries(FuzzyInt(3, 10)):+ with self.assertNumQueries(FuzzyInt(4, 11)):result = self._run_quota_limiting()assert patch_capture.call_count == 0 # No events should be captured since org won't be limitedassert result.quota_limited_orgs["events"] == {}diff --git a/posthog/tasks/ai_observability_usage_report.py b/posthog/tasks/ai_observability_usage_report.pyindex 7861243174be..c03ddd27e98d 100644--- a/posthog/tasks/ai_observability_usage_report.py+++ b/posthog/tasks/ai_observability_usage_report.py@@ -16,6 +16,7 @@ from posthog.constants import AI_EVENT_NAME_PREFIXfrom posthog.exceptions_capture import capture_exceptionfrom posthog.logging.timing import timed_logfrom posthog.models.event.new_events_schema import events_read_table, use_new_events_schema+from posthog.models.organization import Organizationfrom posthog.models.property.util import get_property_string_exprfrom posthog.models.team.team import Teamfrom posthog.ph_client import PH_US_API_KEY@@ -24,6 +25,8 @@ from posthog.tasks.report_utils import capture_eventfrom posthog.tasks.utils import CeleryQueuefrom posthog.utils import DayRange, get_instance_region, get_previous_day+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage+logger = structlog.get_logger(__name__)@@ -917,6 +920,7 @@ def _get_all_ai_observability_reports(dict mapping organization_id to usage data"""logger.info("Querying AI observability usage data")+ offline_usage = get_offline_evaluation_usage(period.start, period.start + timedelta(days=1))# Phase 1: Get all team_ids with report trigger events (fast query)try:@@ -933,7 +937,7 @@ def _get_all_ai_observability_reports(# Re-raise so Celery's autoretry_for=(Exception,) kicks in. Do not swallow.raise- if not team_ids:+ if not team_ids and not offline_usage:logger.info("No teams with AI observability trigger events found")return {}@@ -942,7 +946,7 @@ def _get_all_ai_observability_reports(# Phase 2: Get all metrics in a single combined querylogger.info("Querying all AI metrics")try:- all_metrics = get_all_ai_metrics(period.start, period.end, team_ids)+ all_metrics = get_all_ai_metrics(period.start, period.end, team_ids) if team_ids else {}except Exception:logger.warning("[AIO Usage Error] metrics query failed",@@ -957,7 +961,7 @@ def _get_all_ai_observability_reports(# Phase 3: Get distinct trace countslogger.info("Querying AI trace counts")try:- ai_trace_counts = get_ai_trace_counts(period.start, period.end, team_ids)+ ai_trace_counts = get_ai_trace_counts(period.start, period.end, team_ids) if team_ids else {}except Exception:logger.warning("[AIO Usage Error] trace counts query failed",@@ -973,7 +977,9 @@ def _get_all_ai_observability_reports(llm_prompt_fetched_counts: dict[int, int] = {}try:logger.info("Querying LLM prompt fetched counts")- llm_prompt_fetched_counts = get_llm_prompt_fetched_counts(period.start, period.end, team_ids)+ llm_prompt_fetched_counts = (+ get_llm_prompt_fetched_counts(period.start, period.end, team_ids) if team_ids else {}+ )logger.info(f"Retrieved prompt fetched counts for {len(llm_prompt_fetched_counts)} teams")except Exception as err:logger.warning(@@ -984,7 +990,7 @@ def _get_all_ai_observability_reports(# Phase 5: Get all dimension breakdowns in a single combined querylogger.info("Querying all AI dimension breakdowns")try:- all_breakdowns = get_all_ai_dimension_breakdowns(period.start, period.end, team_ids)+ all_breakdowns = get_all_ai_dimension_breakdowns(period.start, period.end, team_ids) if team_ids else {}except Exception:logger.warning("[AIO Usage Error] breakdowns query failed",@@ -999,7 +1005,7 @@ def _get_all_ai_observability_reports(# Phase 6: Get LLM feedback survey metricslogger.info("Querying LLM feedback survey metrics")try:- survey_metrics = get_llm_feedback_survey_metrics(period.start, period.end, team_ids)+ survey_metrics = get_llm_feedback_survey_metrics(period.start, period.end, team_ids) if team_ids else {}except Exception:logger.warning("[AIO Usage Error] surveys query failed",@@ -1015,17 +1021,21 @@ def _get_all_ai_observability_reports(teams = Team.objects.filter(id__in=team_ids).select_related("organization")team_to_org: dict[int, str] = {team.id: str(team.organization_id) for team in teams}org_id_to_name: dict[str, str] = {str(team.organization_id): team.organization.name for team in teams}+ org_id_to_name.update(+ (str(org.id), org.name) for org in Organization.objects.filter(id__in=offline_usage).only("id", "name")+ )# Aggregate by organizationorg_reports: dict[str, dict[str, Any]] = {}- for team_id, org_id in team_to_org.items():+ for org_id in org_id_to_name:if org_id not in org_reports:org_reports[org_id] = {"organization_id": org_id,"organization_name": org_id_to_name.get(org_id, ""),"period_start": period.start.isoformat(),"period_end": period.end.isoformat(),+ "ai_offline_eval_metric_count_in_period": sum(offline_usage.get(org_id, {}).values()),"ai_generation_count": 0,"ai_embedding_count": 0,"ai_span_count": 0,@@ -1069,6 +1079,7 @@ def _get_all_ai_observability_reports("cost_model_provider_breakdown": {},}+ for team_id, org_id in team_to_org.items():report = org_reports[org_id]# Add metrics from TeamMetrics dataclassdiff --git a/posthog/tasks/test/test_ai_observability_usage_report.py b/posthog/tasks/test/test_ai_observability_usage_report.pyindex 8a97e8b97ac7..325b637d35ac 100644--- a/posthog/tasks/test/test_ai_observability_usage_report.py+++ b/posthog/tasks/test/test_ai_observability_usage_report.py@@ -1,3 +1,6 @@+import dataclasses+from collections import defaultdict+from collections.abc import Callablefrom datetime import UTC, datetime, timedeltafrom uuid import uuid4@@ -12,6 +15,7 @@ from posthog.test.base import ()from unittest.mock import MagicMock, patch+from django.db import DatabaseError, connectionfrom django.test import SimpleTestCasefrom parameterized import parameterized@@ -35,7 +39,27 @@ from posthog.tasks.ai_observability_usage_report import (get_teams_with_ai_events,send_ai_observability_usage_reports,)-from posthog.utils import get_previous_day+from posthog.tasks.usage_report import _get_all_org_reports, capture_report, has_non_zero_usage+from posthog.temporal.usage_report.aggregator import iter_org_reports+from posthog.temporal.usage_report.types import WorkflowContext+from posthog.usage_counters import UsageCounterService+from posthog.utils import DayRange, get_previous_day++from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger+from products.ai_observability.backend.models.score_definitions import ScoreDefinition+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage+from products.ai_observability.backend.offline_evaluation_service import (+ OfflineEvaluationIngestionService,+ OfflineExperimentService,+)+from products.ai_observability.backend.offline_evaluation_types import (+ ExperimentSubmission,+ ItemSubmission,+ ResultSubmission,+ UploadSubmission,+)++from ee.billing.quota_limiting import _aggregate_quota_usage@time_machine.travel("2022-01-10T00:01:00Z", tick=False)@@ -522,6 +546,180 @@ class TestAIObservabilityUsageReport(APIBaseTest, ClickhouseTestMixin, Clickhousassert metrics.active_survey_count == 2 # survey_id_1 and survey_id_2assert metrics.response_count == 5 # 3 + 2 (only "survey sent" events)+ @parameterized.expand([(50, 30), (0, 30), (50, 0), (0, 0)])+ @time_machine.travel("2026-10-31T23:59:59.999999Z", tick=False)+ def test_upload_to_both_reports_and_quota(self, ingested: int, offline: int) -> None:+ period = get_previous_day(at=datetime(2026, 11, 1, tzinfo=UTC))+ if ingested:+ self._create_ai_events(self.team, "billing-test", "$ai_generation", ingested, timestamp=period.start)+ if offline:+ experiment = (+ OfflineExperimentService(team_id=self.team.id, user_access_control=None)+ .create(ExperimentSubmission(id=uuid4(), name="Billing fixture", started_at=period.start))+ .experiment+ )+ scorer = ScoreDefinition.objects.create(team=self.team, name="Accuracy", kind="numeric")+ versions = [scorer.create_new_version(config={}, created_by=None) for _ in range(6)]+ items = [ItemSubmission(id=uuid4()) for _ in range(5)]+ results = [+ ResultSubmission(item_id=item.id, scorer_version_id=version.id, status="ok", value=0.8)+ for item in items+ for version in versions+ ]+ ingestion = OfflineEvaluationIngestionService(team_id=self.team.id, user_access_control=None)+ for batch in (results[:20], results[20:]):+ submission = UploadSubmission(+ items=[item for item in items if item.id in {result.item_id for result in batch}], results=batch+ )+ ingestion.upload(experiment.id, submission)+ ingestion.upload(experiment.id, submission)+ ingestion.upload(+ experiment.id, UploadSubmission(items=list(reversed(items)), results=list(reversed(results)))+ )+ self.assertEqual(+ sorted(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).values_list("quantity", flat=True)),+ [10, 20],+ )+ rejected_item = ItemSubmission(id=uuid4(), payload={"input": "Rejected transaction"})++ def fail_ledger_insert(+ execute: Callable[..., object], sql: str, params: object, many: bool, context: object+ ) -> object:+ if sql.startswith("INSERT INTO") and OfflineEvaluationUsageLedger._meta.db_table in sql:+ raise DatabaseError("Ledger write unavailable")+ return execute(sql, params, many, context)++ with (+ connection.execute_wrapper(fail_ledger_insert),+ self.assertRaisesMessage(DatabaseError, "Ledger write unavailable"),+ ):+ ingestion.upload(+ experiment.id,+ UploadSubmission(+ items=[rejected_item], results=[dataclasses.replace(results[0], item_id=rejected_item.id)]+ ),+ )++ counter_report = UsageCounterService().fetch_report(period)+ all_data = defaultdict(dict, {key: dict(rows) for key, rows in counter_report.counts.items()})+ org_id = str(self.organization.id)+ before = list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).order_by("id").values())+ celery_report = _get_all_org_reports(period=period, counter_report=counter_report)[org_id]+ ctx = WorkflowContext(+ run_id="offline-usage", date_str="2026-10-31", period_start=period.start, period_end=period.end+ )+ temporal_report = next(+ report for report in iter_org_reports(all_data, ctx, {}) if report.organization_id == org_id+ )+ expected = {+ "ai_ingested_event_count_in_period": ingested,+ "ai_offline_eval_metric_count_in_period": offline,+ "ai_event_count_in_period": ingested + offline,+ }+ for report in (celery_report, temporal_report):+ self.assertEqual({key: getattr(report, key) for key in expected}, expected)+ self.assertEqual({key: getattr(report.teams[str(self.team.id)], key) for key in expected}, expected)+ self.assertEqual(has_non_zero_usage(report), bool(ingested + offline))++ with patch(+ "posthog.tasks.usage_report.sync_execute",+ side_effect=AssertionError("Analytics reporting must not execute billing queries"),+ ):+ analytics_reports = _get_all_ai_observability_reports(period=period)+ if ingested + offline:+ analytics_report = analytics_reports[org_id]+ self.assertEqual(analytics_report["ai_offline_eval_metric_count_in_period"], offline)+ self.assertEqual(analytics_report["ai_generation_count"], ingested)+ self.assertEqual(analytics_report["ai_metric_count"], 0)+ self.assertNotIn("ai_ingested_event_count_in_period", analytics_report)+ self.assertNotIn("ai_event_count_in_period", analytics_report)+ with (+ patch("posthog.tasks.usage_report.get_ph_client"),+ patch("posthog.tasks.usage_report.capture_event") as org_capture,+ patch("posthog.tasks.ai_observability_usage_report.get_ph_client"),+ patch("posthog.tasks.ai_observability_usage_report.capture_event") as ai_capture,+ ):+ capture_report(organization_id=org_id, full_report_dict=dataclasses.asdict(celery_report))+ capture_ai_observability_report(organization_id=org_id, report_dict=analytics_report)+ org_payload = next(+ call.kwargs["properties"]+ for call in org_capture.call_args_list+ if call.kwargs["name"] == "organization usage report"+ )+ self.assertEqual({key: org_payload[key] for key in expected}, expected)+ self.assertEqual(+ ai_capture.call_args.kwargs["properties"]["ai_offline_eval_metric_count_in_period"], offline+ )+ else:+ self.assertEqual(analytics_reports, {})+ quotas = _aggregate_quota_usage(+ [self.team],+ all_data,+ offline_usage=get_offline_evaluation_usage(period.start, period.start + timedelta(days=1)),+ )+ self.assertEqual(quotas.usage[org_id]["llm_events"], ingested + offline)+ self.assertEqual(+ list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).order_by("id").values()), before+ )++ @parameterized.expand([(False,), (True,)])+ @time_machine.travel("2026-11-01T00:01:00Z", tick=False)+ def test_both_reports_preserve_moved_team_usage_but_exclude_deleted_teams(self, deleted_team: bool) -> None:+ period = DayRange(start=datetime(2026, 10, 31, tzinfo=UTC), end=datetime(2026, 11, 1, tzinfo=UTC))+ new_org = Organization.objects.create(name="New owner")+ for org, quantity in ((self.organization, 30), (new_org, 4)):+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).create(+ team_id=self.team.id,+ organization_id=org.id,+ experiment_id=uuid4(),+ accepted_at=period.start,+ quantity=quantity,+ )+ Team.objects.filter(id=self.team.id).update(organization=new_org)+ if deleted_team:+ self.team.delete()+ analytics = _get_all_ai_observability_reports(period=period)+ reports = _get_all_org_reports(period=period)+ for org, expected in ((self.organization, 30), (new_org, 4)):+ org_id = str(org.id)+ if deleted_team:+ self.assertNotIn(org_id, analytics)+ self.assertNotIn(org_id, reports)+ continue+ self.assertEqual(analytics[org_id]["ai_offline_eval_metric_count_in_period"], expected)+ self.assertEqual(reports[org_id].ai_event_count_in_period, expected)++ @time_machine.travel("2026-11-01T00:01:00Z", tick=False)+ def test_failed_ledger_read_does_not_claim_the_analytics_dispatch(self) -> None:+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).create(+ team_id=self.team.id,+ organization_id=self.organization.id,+ experiment_id=uuid4(),+ accepted_at=datetime(2026, 10, 31, tzinfo=UTC),+ quantity=30,+ )++ def fail_ledger_read(+ execute: Callable[..., object], sql: str, params: object, many: bool, context: object+ ) -> object:+ if sql.startswith("SELECT") and OfflineEvaluationUsageLedger._meta.db_table in sql:+ raise DatabaseError("Ledger unavailable")+ return execute(sql, params, many, context)++ with (+ patch("posthoganalytics.feature_enabled", return_value=False),+ patch("posthog.tasks.ai_observability_usage_report.capture_ai_observability_report.delay") as dispatch,+ ):+ with (+ connection.execute_wrapper(fail_ledger_read),+ self.assertRaisesMessage(DatabaseError, "Ledger unavailable"),+ ):+ send_ai_observability_usage_reports()+ dispatch.assert_not_called()+ send_ai_observability_usage_reports()+ self.assertEqual(dispatch.call_count, 1)+ self.assertEqual(dispatch.call_args.kwargs["report_dict"]["ai_offline_eval_metric_count_in_period"], 30)+def test_full_ai_observability_report(self) -> None:"""Test the full AI observability report generation."""# Create second organization and team@@ -1374,7 +1572,10 @@ class TestAIObservabilityUsageReportTaskWiring(SimpleTestCase):task = send_ai_observability_usage_reportstask.push_request(retries=retries, called_directly=called_directly, is_eager=True)try:- with pytest.raises(Exception, match="CH down"):+ with (+ patch("posthog.tasks.ai_observability_usage_report.get_offline_evaluation_usage", return_value={}),+ pytest.raises(Exception, match="CH down"),+ ):task.run()finally:task.pop_request()diff --git a/posthog/tasks/usage_report.py b/posthog/tasks/usage_report.pyindex d1dfe979ef89..39108726b446 100644--- a/posthog/tasks/usage_report.py+++ b/posthog/tasks/usage_report.py@@ -63,6 +63,7 @@ from posthog.usage_counters import ()from posthog.utils import DayRange, get_helm_info_env, get_instance_realm, get_instance_region, get_previous_day+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usagefrom products.batch_exports.backend.facade import api as batch_exports_apifrom products.cdp.backend.models.hog_functions.hog_function import HogFunction, HogFunctionTypefrom products.cdp.backend.models.plugin import PluginConfig@@ -297,6 +298,8 @@ class UsageReportCounters:# AI observabilityai_event_count_in_period: int+ ai_ingested_event_count_in_period: int = dataclasses.field(default=0, kw_only=True)+ ai_offline_eval_metric_count_in_period: int = dataclasses.field(default=0, kw_only=True)# AI Billing Credits (PostHog AI feature usage)ai_credits_used_in_period: int@@ -3579,6 +3582,7 @@ def _get_team_report(all_data: dict[str, Any], team: Team) -> UsageReportCounterroblox_events_count_in_period=all_data["teams_with_roblox_events_count_in_period"].get(team.id, 0),rust_events_count_in_period=all_data["teams_with_rust_events_count_in_period"].get(team.id, 0),ai_event_count_in_period=all_data["teams_with_ai_event_count_in_period"].get(team.id, 0),+ ai_ingested_event_count_in_period=all_data["teams_with_ai_event_count_in_period"].get(team.id, 0),ai_credits_used_in_period=all_data["teams_with_ai_credits_used_in_period"].get(team.id, 0),signals_credits_used_in_period=all_data["teams_with_signals_credits_used_in_period"].get(team.id, 0),posthog_code_credits_used_in_period=combine_posthog_code_credits(@@ -3682,9 +3686,63 @@ def _add_team_report_to_org_reports()+def iter_reports_with_offline_evaluation_usage(+ reports: Iterable[OrgReport],+ offline_usage: dict[str, dict[int, int]],+ period_start: datetime,+ *,+ organization_ids: Sequence[str] | None = None,+) -> Iterator[OrgReport]:+ def add_usage(report: OrgReport) -> OrgReport:+ for team_id, quantity in offline_usage.get(report.organization_id, {}).items():+ team_key = str(team_id)+ if team_key not in report.teams:+ report.teams[team_key] = UsageReportCounters(+ **{field.name: 0 for field in dataclasses.fields(UsageReportCounters)}+ )+ team_report = report.teams[team_key]+ team_report.ai_offline_eval_metric_count_in_period = quantity+ team_report.ai_event_count_in_period = team_report.ai_ingested_event_count_in_period + quantity+ report.ai_offline_eval_metric_count_in_period = sum(offline_usage.get(report.organization_id, {}).values())+ report.ai_event_count_in_period = (+ report.ai_ingested_event_count_in_period + report.ai_offline_eval_metric_count_in_period+ )+ return report++ seen: set[str] = set()+ for report in reports:+ seen.add(report.organization_id)+ yield add_usage(report)+ missing = offline_usage.keys() - seen+ if organization_ids:+ missing &= set(organization_ids)+ user_counts = dict(+ OrganizationMembership.objects.filter(organization_id__in=missing)+ .values("organization_id")+ .annotate(count=Count("id"))+ .values_list("organization_id", "count")+ )+ for organization in Organization.objects.filter(id__in=missing).only("id", "name", "created_at").iterator():+ yield add_usage(+ OrgReport(+ date=period_start.strftime("%Y-%m-%d"),+ organization_id=str(organization.id),+ organization_name=organization.name,+ organization_created_at=organization.created_at.isoformat(),+ organization_user_count=user_counts.get(organization.id, 0),+ team_count=0,+ teams={},+ **dataclasses.asdict(+ UsageReportCounters(**{field.name: 0 for field in dataclasses.fields(UsageReportCounters)})+ ),+ )+ )++def _get_all_org_reports(*, period: DayRange, counter_report: UsageCounterReport | None = None) -> dict[str, OrgReport]:logger.info("Querying all org reports", period_start=period.start, period_end=period.end)+ offline_usage = get_offline_evaluation_usage(period.start, period.start + timedelta(days=1))all_data = _get_all_usage_data_as_team_rows(period.start, period.end, counter_report)org_reports: dict[str, OrgReport] = {}@@ -3699,7 +3757,10 @@ def _get_all_org_reports(*, period: DayRange, counter_report: UsageCounterReportlogger.info("Generating org reports complete", teams_count=teams_count, org_reports_count=len(org_reports))- return org_reports+ return {+ report.organization_id: report+ for report in iter_reports_with_offline_evaluation_usage(org_reports.values(), offline_usage, period.start)+ }def _get_full_org_usage_report(org_report: OrgReport, instance_metadata: InstanceMetadata) -> FullUsageReport:@@ -3772,12 +3833,24 @@ def iter_reports_with_usage_counter_metadata(report_teams: set[int] = set()missing: dict[str, dict[str, UsageCounterComparison]] = {}wanted = set(organization_ids) if organization_ids else None+ comparison_team_owners: dict[int, str] | None = Nonefor report in org_reports:report.usage_sources = counter_report.usage_sourcesreport.counter_comparisons = {} if counter_report.counter_comparisons is not None else Noneif counter_report.counter_comparisons is not None:org_id = report.organization_id- teams = [int(team_id) for team_id in report.teams]+ if report.ai_offline_eval_metric_count_in_period and comparison_team_owners is None:+ comparison_team_owners = {+ team_id: str(owner)+ for team_id, owner in Team.objects.filter(+ is_demo=False, organization__for_internal_metrics=False+ ).values_list("id", "organization_id")+ }+ teams = [+ int(team_id)+ for team_id in report.teams+ if comparison_team_owners is None or comparison_team_owners.get(int(team_id)) == org_id+ ]report_orgs.add(org_id)report_teams.update(teams)reported = has_non_zero_usage(report)diff --git a/posthog/temporal/tests/usage_report/test_aggregate_activity.py b/posthog/temporal/tests/usage_report/test_aggregate_activity.pyindex 76c72efca938..0aa3fdca3528 100644--- a/posthog/temporal/tests/usage_report/test_aggregate_activity.py+++ b/posthog/temporal/tests/usage_report/test_aggregate_activity.py@@ -29,6 +29,8 @@ from posthog.temporal.usage_report.queries import QUERIES, QUERY_INDEXfrom posthog.temporal.usage_report.storage import queries_key, write_jsonfrom posthog.temporal.usage_report.types import AggregateInputs, RunQueryToS3Result, WorkflowContext+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger+@sync_to_asyncdef _make_org(name: str) -> Organization:@@ -212,6 +214,19 @@ async def test_aggregate_writes_chunks_and_manifest(minio_workflow_ctx: Workflow# pre-seeded orgs that would otherwise inflate the manifest counts.ctx = minio_workflow_ctx.model_copy(update={"organization_ids": [str(org_a.id), str(org_b.id)]})query_results = _seed_query_results(ctx, team_a.id, team_b.id)+ write_json(queries_key(ctx, "teams_with_ai_event_count_in_period"), [[team_a.id, 50]])+ await sync_to_async(OfflineEvaluationUsageLedger.objects.for_team(team_a.id).bulk_create)(+ [+ OfflineEvaluationUsageLedger(+ team_id=team_a.id,+ organization_id=org_a.id,+ experiment_id=uuid.uuid4(),+ accepted_at=ctx.period_start,+ quantity=quantity,+ )+ for quantity in (20, 10)+ ]+ )with patch("posthog.temporal.usage_report.activities.get_instance_metadata",@@ -243,6 +258,10 @@ async def test_aggregate_writes_chunks_and_manifest(minio_workflow_ctx: Workflowassert len(rows) == 2by_org = {row["organization_id"]: row["usage_report"] for row in rows}assert set(by_org.keys()) == {str(org_a.id), str(org_b.id)}+ assert by_org[str(org_a.id)]["ai_ingested_event_count_in_period"] == 50+ assert by_org[str(org_a.id)]["ai_offline_eval_metric_count_in_period"] == 30+ assert by_org[str(org_a.id)]["ai_event_count_in_period"] == 80+ assert by_org[str(org_a.id)]["teams"][str(team_a.id)]["ai_event_count_in_period"] == 80# multi-key fan-out: web events from `all_event_metrics` landed in the# destination key on each org's report.@@ -443,6 +462,13 @@ async def test_aggregate_filters_by_organization_ids(minio_workflow_ctx: Workfloctx = minio_workflow_ctx.model_copy(update={"organization_ids": [str(org_a.id)]})query_results = _seed_query_results(ctx, team_a.id, team_b.id)+ await sync_to_async(OfflineEvaluationUsageLedger.objects.for_team(team_b.id).create)(+ team_id=team_b.id,+ organization_id=org_b.id,+ experiment_id=uuid.uuid4(),+ accepted_at=ctx.period_start,+ quantity=7,+ )with patch("posthog.temporal.usage_report.activities.get_instance_metadata",diff --git a/posthog/temporal/usage_report/aggregator.py b/posthog/temporal/usage_report/aggregator.pyindex 132d9c3866ad..5cc4f4551e3c 100644--- a/posthog/temporal/usage_report/aggregator.py+++ b/posthog/temporal/usage_report/aggregator.py@@ -11,7 +11,7 @@ for Temporal. Activities import from here.import dataclassesfrom collections.abc import Callable, Iterable, Iterator-from datetime import datetime+from datetime import datetime, timedeltafrom itertools import chain, groupby, islicefrom typing import Any@@ -29,6 +29,7 @@ from posthog.tasks.usage_report import (_get_teams_for_usage_reports,convert_team_usage_rows_to_dict,has_non_zero_usage,+ iter_reports_with_offline_evaluation_usage,serialize_full_org_report,)from posthog.temporal.usage_report.queries import QUERY_INDEX@@ -49,6 +50,8 @@ from posthog.temporal.usage_report.types import ()from posthog.usage_counters import UsageCounterReport+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage+_SANDBOX_COMPUTE_QUERY_NAME = "sandbox_compute_usage"_SANDBOX_COMPUTE_DESTINATION_KEYS = ("teams_with_sandbox_compute_credits_used_in_period",@@ -214,7 +217,7 @@ def _add_team_report_to_org_reports()-def iter_org_reports(+def _iter_ingested_org_reports(all_data: dict[str, dict[int, int]],ctx: WorkflowContext,org_user_counts: dict[str, int],@@ -229,6 +232,20 @@ def iter_org_reports(yield org_reports[org_id]+def iter_org_reports(+ all_data: dict[str, dict[int, int]],+ ctx: WorkflowContext,+ org_user_counts: dict[str, int],+) -> Iterator[OrgReport]:+ offline_usage = get_offline_evaluation_usage(ctx.period_start, ctx.period_start + timedelta(days=1))+ yield from iter_reports_with_offline_evaluation_usage(+ _iter_ingested_org_reports(all_data, ctx, org_user_counts),+ offline_usage,+ ctx.period_start,+ organization_ids=ctx.organization_ids,+ )++def write_org_report_chunks(ctx: WorkflowContext,org_reports: Iterable[OrgReport],diff --git a/posthog/usage_counters.py b/posthog/usage_counters.pyindex 9a3a9e9f13db..1752bf04d839 100644--- a/posthog/usage_counters.py+++ b/posthog/usage_counters.py@@ -38,6 +38,7 @@ class UsageCounter(StrEnum):MOBILE_RECORDINGS = "teams_with_mobile_recording_count_in_period"MOBILE_BILLABLE_RECORDINGS = "teams_with_mobile_billable_recording_count_in_period"SURVEY_RESPONSES = "teams_with_survey_responses_count_in_period"+ # Ingested AIO events only. Offline usage is composed after organization attribution.AI_EVENTS = "teams_with_ai_event_count_in_period"EXCEPTIONS = "teams_with_exceptions_captured_in_period"CDP_INVOCATIONS = "teams_with_cdp_billable_invocations_in_period"diff --git a/products/ai_observability/backend/api/offline_experiments.py b/products/ai_observability/backend/api/offline_experiments.pyindex 85b3fc241d0a..2a3916ad492d 100644--- a/products/ai_observability/backend/api/offline_experiments.py+++ b/products/ai_observability/backend/api/offline_experiments.py@@ -241,7 +241,7 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):@validated_request(UploadSubmissionSerializer,parameters=[EXPERIMENT_ID_PARAMETER],- responses={200: UploadReceiptSerializer, **ERROR_RESPONSES},+ responses={200: UploadReceiptSerializer, 402: OfflineEvaluationErrorSerializer, **ERROR_RESPONSES},)@action(detail=True, methods=["post"])@llma_track_latency("aio_offline_experiment_upload")diff --git a/products/ai_observability/backend/api/test/test_offline_experiments.py b/products/ai_observability/backend/api/test/test_offline_experiments.pyindex c4bafdf877b6..42439dea5053 100644--- a/products/ai_observability/backend/api/test/test_offline_experiments.py+++ b/products/ai_observability/backend/api/test/test_offline_experiments.py@@ -20,6 +20,7 @@ from posthog.models import OrganizationMembership, Project, Userfrom posthog.models.personal_api_key import PersonalAPIKeyfrom posthog.models.project_secret_api_key import ProjectSecretAPIKeyfrom posthog.models.utils import generate_random_token_personal, hash_key_value+from posthog.redis import get_clientfrom products.access_control.backend.models.access_control import AccessControlfrom products.ai_observability.backend.api.offline_experiment_reads import OfflineScorerViewSet@@ -230,7 +231,12 @@ class TestOfflineExperimentsAPI(APIBaseTest):self.assertEqual(completed_event.args[0], completed_event.kwargs["request"].user)self.assertTrue(completed_event.args[0].distinct_id)- def test_ledger_failure_returns_server_error_and_mixed_upload_can_retry(self) -> None:+ @parameterized.expand(+ [("ledger", status.HTTP_500_INTERNAL_SERVER_ERROR), ("quota", status.HTTP_402_PAYMENT_REQUIRED)]+ )+ def test_rejected_mixed_upload_preserves_existing_results_and_can_retry(+ self, rejection: str, expected_status: int+ ) -> None:definition = ScoreDefinition.objects.create(team=self.team, name="Correct", kind="boolean")version = definition.create_new_version(config={}, created_by=self.user)self._authenticate("project_key")@@ -273,10 +279,22 @@ class TestOfflineExperimentsAPI(APIBaseTest):return execute(sql, params, many, context)upload = {"items": items, "results": results}- with connection.execute_wrapper(fail_ledger_insert):+ if rejection == "ledger":+ with connection.execute_wrapper(fail_ledger_insert):+ failed = self.client.post(endpoint, upload, format="json")+ else:+ redis_client = get_client()+ quota_key = "@posthog/quota-limits/llm_events"+ redis_client.zadd(quota_key, {self.team.api_token: timezone.now().timestamp() + 3600})+ self.addCleanup(redis_client.zrem, quota_key, self.team.api_token)+ replayed = self.client.post(endpoint, {"items": items[:1], "results": results[:1]}, format="json")+ self.assertEqual(replayed.status_code, status.HTTP_200_OK, replayed.data)+ self.assertEqual(replayed.data["results"][0], accepted.data["results"][0] | {"created": False})failed = self.client.post(endpoint, upload, format="json")+ self.assertEqual(failed.data["code"], "quota_limit_exceeded", failed.data)+ redis_client.zadd(quota_key, {self.team.api_token: timezone.now().timestamp() - 1})- self.assertEqual(failed.status_code, status.HTTP_500_INTERNAL_SERVER_ERROR, failed.data)+ self.assertEqual(failed.status_code, expected_status, failed.data)for model, rows in rows_before_failure.items():self.assertEqual(list(model.objects.for_team(self.team.id).values()), rows)diff --git a/products/ai_observability/backend/offline_evaluation_billing.py b/products/ai_observability/backend/offline_evaluation_billing.pynew file mode 100644index 000000000000..8396aa24184e--- /dev/null+++ b/products/ai_observability/backend/offline_evaluation_billing.py@@ -0,0 +1,30 @@+from datetime import UTC, datetime++from django.db.models import Sum+from django.utils import timezone++from posthog.models import Organization, Team++from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger+++def get_offline_evaluation_usage(period_start: datetime, period_end: datetime) -> dict[str, dict[int, int]]:+ """Billable quantities by acceptance organization and environment, without consuming the ledger."""+ if timezone.is_naive(period_start) or timezone.is_naive(period_end) or period_start >= period_end:+ raise ValueError("Offline evaluation usage requires an aware, increasing reporting window")++ rows = (+ OfflineEvaluationUsageLedger.objects.unscoped()+ .filter(+ accepted_at__gte=period_start.astimezone(UTC),+ accepted_at__lt=period_end.astimezone(UTC),+ team_id__in=Team.objects.filter(is_demo=False, organization__for_internal_metrics=False).values("id"),+ organization_id__in=Organization.objects.filter(for_internal_metrics=False).values("id"),+ )+ .values("organization_id", "team_id")+ .annotate(total_quantity=Sum("quantity"))+ )+ usage: dict[str, dict[int, int]] = {}+ for row in rows:+ usage.setdefault(str(row["organization_id"]), {})[row["team_id"]] = int(row["total_quantity"])+ return usagediff --git a/products/ai_observability/backend/offline_evaluation_service.py b/products/ai_observability/backend/offline_evaluation_service.pyindex eaa6efd92de3..29f53c73f7f2 100644--- a/products/ai_observability/backend/offline_evaluation_service.py+++ b/products/ai_observability/backend/offline_evaluation_service.py@@ -13,6 +13,7 @@ from django.db.models import Qfrom django.utils import timezonefrom posthog.dataclasses import frozen+from posthog.exceptions import QuotaLimitExceededfrom posthog.models.team import Teamfrom products.ai_observability.backend.dataset_queries import dataset_item_versions_at_revision@@ -36,6 +37,8 @@ from products.ai_observability.backend.offline_evaluation_types import ()from products.ai_observability.backend.score_validation import validate_score_value+from ee.billing.quota_limiting import QuotaLimitingCaches, QuotaResource, is_team_limited+if TYPE_CHECKING:from django.db.models import QuerySet@@ -444,6 +447,13 @@ class OfflineEvaluationIngestionService(_OfflineEvaluationService):new_item_submissions = [item for item in submission.items if item.id not in items]if (new_item_submissions or new_results) and experiment.status != OfflineExperiment.Status.UPLOADING:raise OfflineEvaluationConflict("experiment_closed", "This experiment is closed to new items and results.")+ if new_results:+ api_token = Team.objects.values_list("api_token", flat=True).get(id=self.team_id)+ if is_team_limited(api_token, QuotaResource.LLM_EVENTS, QuotaLimitingCaches.QUOTA_LIMITER_CACHE_KEY):+ raise QuotaLimitExceeded(+ "Your organization reached its AI observability usage limit. Increase the limit in Billing settings, "+ "or ask an org admin to do so."+ )errors: dict[str, str] = {}try:dataset_versions = self._dataset_versions(experiment, new_item_submissions)diff --git a/products/ai_observability/backend/test/test_offline_evaluation_billing.py b/products/ai_observability/backend/test/test_offline_evaluation_billing.pynew file mode 100644index 000000000000..abf8bdc03195--- /dev/null+++ b/products/ai_observability/backend/test/test_offline_evaluation_billing.py@@ -0,0 +1,234 @@+from collections import defaultdict+from collections.abc import Callable+from datetime import UTC, datetime, timedelta+from itertools import batched+from uuid import UUID, uuid4++import time_machine+from unittest.mock import patch++from django.db import DatabaseError, connection+from django.test import TestCase++from parameterized import parameterized++from posthog.models import Organization, Team+from posthog.tasks.ai_observability_usage_report import _get_all_ai_observability_reports+from posthog.tasks.usage_report import _get_all_org_reports, iter_reports_with_usage_counter_metadata+from posthog.temporal.usage_report.aggregator import iter_org_reports+from posthog.temporal.usage_report.types import WorkflowContext+from posthog.usage_counters import UsageCounterComparisonRows, UsageCounterMode, UsageCounterReport+from posthog.utils import DayRange++from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage++from ee.billing.quota_limiting import _aggregate_quota_usage, _iter_quota_organizations, update_all_orgs_billing_quotas+++@time_machine.travel("2026-10-31T12:00:00Z", tick=False)+class TestOfflineEvaluationBilling(TestCase):+ organization: Organization+ other_organization: Organization+ team: Team+ other_team: Team++ @classmethod+ def setUpTestData(cls) -> None:+ cls.organization, _, cls.team = Organization.objects.bootstrap(None)+ cls.other_organization, _, cls.other_team = Organization.objects.bootstrap(None)++ def setUp(self) -> None:+ self.period = DayRange(start=datetime(2026, 10, 31, tzinfo=UTC), end=datetime(2026, 11, 1, tzinfo=UTC))+ self.ctx = WorkflowContext(+ run_id="offline-usage-test",+ date_str="2026-10-31",+ period_start=self.period.start,+ period_end=self.period.end,+ )++ def record(self, team: Team, quantity: int, *, at: datetime | None = None) -> OfflineEvaluationUsageLedger:+ return OfflineEvaluationUsageLedger.objects.for_team(team.id).create(+ team_id=team.id,+ organization_id=team.organization_id,+ experiment_id=uuid4(),+ accepted_at=at or self.period.start,+ quantity=quantity,+ )++ @parameterized.expand([(-1, 0), (0, 7), (86_399_999_999, 7), (86_400_000_000, 0)])+ def test_utc_half_open_month_boundary(self, microseconds: int, expected: int) -> None:+ self.record(self.team, 7, at=self.period.start + timedelta(microseconds=microseconds))+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)+ self.assertEqual(sum(usage.get(str(self.organization.id), {}).values()), expected)++ @parameterized.expand([(False,), (True,)])+ def test_sums_quantities_for_moved_teams_but_excludes_deleted_teams(self, deleted_team: bool) -> None:+ child = Team.objects.create(organization=self.organization, parent_team=self.team)+ self.record(child, 20)+ self.record(child, 10)+ self.record(self.team, 5)+ self.record(self.other_team, 9)+ child.organization = self.other_organization+ child.save(update_fields=["organization"])+ self.record(child, 4)+ child_id = child.id+ if deleted_team:+ child.delete()+ expected = {+ str(self.organization.id): {self.team.id: 5},+ str(self.other_organization.id): {self.other_team.id: 9},+ }+ if not deleted_team:+ expected[str(self.organization.id)][child_id] = 30+ expected[str(self.other_organization.id)][child_id] = 4+ before = list(OfflineEvaluationUsageLedger.objects.unscoped().order_by("id").values())+ for _ in range(2):+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)+ self.assertEqual(usage, expected)+ reports = {report.organization_id: report for report in iter_org_reports(defaultdict(dict), self.ctx, {})}+ self.assertEqual(reports[str(self.organization.id)].ai_event_count_in_period, 5 if deleted_team else 35)+ self.assertEqual(+ reports[str(self.other_organization.id)].ai_event_count_in_period, 9 if deleted_team else 13+ )+ if deleted_team:+ self.assertNotIn(str(child_id), reports[str(self.organization.id)].teams)+ else:+ self.assertEqual(reports[str(self.organization.id)].teams[str(child_id)].ai_event_count_in_period, 30)+ self.assertEqual(list(OfflineEvaluationUsageLedger.objects.unscoped().order_by("id").values()), before)++ @parameterized.expand([(False,), (True,)])+ def test_organization_without_live_teams_has_no_reported_usage(self, demo: bool) -> None:+ self.record(self.team, 30)+ Team.objects.filter(id=self.team.id).update(is_demo=demo)+ self.team.delete()+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)+ self.assertEqual(usage, {})+ reports = {report.organization_id: report for report in iter_org_reports(defaultdict(dict), self.ctx, {})}+ self.assertNotIn(str(self.organization.id), reports)+ self.assertEqual(list(_iter_quota_organizations([], usage)), [])+ quotas = _aggregate_quota_usage([], defaultdict(dict), offline_usage=usage)+ self.assertEqual(quotas.usage, {})+ self.assertEqual(OfflineEvaluationUsageLedger.objects.unscoped().get().quantity, 30)++ @parameterized.expand([(1,), (2,)])+ def test_offline_quota_organizations_survive_closed_server_cursors(self, batch_size: int) -> None:+ historical_organizations = [self.organization, Organization.objects.create(), Organization.objects.create()]+ for organization in historical_organizations:+ self.team.organization = organization+ self.team.save(update_fields=["organization"])+ self.record(self.team, 10)+ self.team.organization = self.other_organization+ self.team.save(update_fields=["organization"])+ self.record(self.team, 7)+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)++ organizations: list[tuple[UUID, list[int]]] = []+ with (+ patch.dict(connection.settings_dict, DISABLE_SERVER_SIDE_CURSORS=False),+ patch("ee.billing.quota_limiting.QUOTA_ORGANIZATION_BATCH_SIZE", batch_size),+ ):+ for batch in batched(+ _iter_quota_organizations([self.team, self.other_team], usage), batch_size, strict=False+ ):+ with connection.cursor() as cursor:+ cursor.execute("CLOSE ALL")+ organizations.extend((organization.id, [team.id for team in teams]) for organization, teams in batch)++ self.assertEqual(+ organizations,+ [+ (self.other_organization.id, [self.team.id, self.other_team.id]),+ *[(organization.id, []) for organization in sorted(historical_organizations, key=lambda org: org.id)],+ ],+ )++ def test_grouped_query_cost_does_not_grow_with_upload_count(self) -> None:+ for _ in range(20):+ self.record(self.team, 3)+ with self.assertNumQueries(1):+ self.assertEqual(+ get_offline_evaluation_usage(self.period.start, self.period.end),+ {str(self.organization.id): {self.team.id: 60}},+ )++ def test_historical_team_does_not_duplicate_ingested_shadow_comparisons(self) -> None:+ self.record(self.team, 30)+ Team.objects.filter(id=self.team.id).update(organization=self.other_organization)+ field = "teams_with_ai_event_count_in_period"+ counter_report = UsageCounterReport(+ counts={field: [(self.team.id, 50)]},+ usage_sources={field: UsageCounterMode.BOTH},+ counter_comparisons={+ field: UsageCounterComparisonRows(+ legacy_by_team={self.team.id: 50},+ realtime_by_team={self.team.id: {str(self.other_organization.id): 55}},+ )+ },+ )+ reports = {+ report.organization_id: report+ for report in iter_reports_with_usage_counter_metadata(+ iter_org_reports(defaultdict(dict, {field: {self.team.id: 50}}), self.ctx, {}),+ counter_report,+ caller="usage_reports_v2",+ date="2026-10-31",+ )+ }+ old = reports[str(self.organization.id)]+ new = reports[str(self.other_organization.id)]+ self.assertEqual(old.ai_event_count_in_period, 30)+ self.assertEqual(new.ai_event_count_in_period, 50)+ self.assertEqual(old.counter_comparisons, {field: {"legacy": 0, "realtime": 0}})+ self.assertEqual(new.counter_comparisons, {field: {"legacy": 50, "realtime": 55}})++ def test_current_demo_and_internal_status_excludes_usage(self) -> None:+ demo_team = Team.objects.create(organization=self.organization, is_demo=True)+ self.record(demo_team, 30)+ self.record(self.team, 10)+ self.record(self.other_team, 9)+ self.other_organization.for_internal_metrics = True+ self.other_organization.save(update_fields=["for_internal_metrics"])+ self.assertEqual(+ get_offline_evaluation_usage(self.period.start, self.period.end),+ {str(self.organization.id): {self.team.id: 10}},+ )+ Team.objects.filter(id=self.team.id).update(is_demo=True)+ Team.objects.filter(id=demo_team.id).update(is_demo=False)+ self.assertEqual(+ get_offline_evaluation_usage(self.period.start, self.period.end),+ {str(self.organization.id): {demo_team.id: 30}},+ )++ def test_deleted_organization_is_not_recreated(self) -> None:+ self.record(self.team, 30)+ self.organization.delete()+ self.assertEqual(get_offline_evaluation_usage(self.period.start, self.period.end), {})+ self.assertEqual(OfflineEvaluationUsageLedger.objects.unscoped().get().quantity, 30)++ @parameterized.expand([("celery",), ("temporal",), ("analytics",), ("quota",)])+ def test_ledger_failure_prevents_incomplete_reports_and_quota_updates(self, path: str) -> None:+ self.organization.usage = {"llm_events": {"usage": 50, "todays_usage": 30, "limit": 100}}+ self.organization.save(update_fields=["usage"])++ def fail_ledger_read(+ execute: Callable[..., object], sql: str, params: object, many: bool, context: object+ ) -> object:+ if sql.startswith("SELECT") and OfflineEvaluationUsageLedger._meta.db_table in sql:+ raise DatabaseError("Ledger unavailable")+ return execute(sql, params, many, context)++ with (+ connection.execute_wrapper(fail_ledger_read),+ self.assertRaisesMessage(DatabaseError, "Ledger unavailable"),+ ):+ if path == "celery":+ _get_all_org_reports(period=self.period)+ elif path == "temporal":+ list(iter_org_reports(defaultdict(dict), self.ctx, {}))+ elif path == "analytics":+ _get_all_ai_observability_reports(period=self.period)+ else:+ update_all_orgs_billing_quotas()+ self.organization.refresh_from_db()+ self.assertEqual(self.organization.usage["llm_events"], {"usage": 50, "todays_usage": 30, "limit": 100})diff --git a/products/ai_observability/backend/test/test_offline_evaluation_service.py b/products/ai_observability/backend/test/test_offline_evaluation_service.pyindex 1536c5b54c96..1f5277f6f600 100644--- a/products/ai_observability/backend/test/test_offline_evaluation_service.py+++ b/products/ai_observability/backend/test/test_offline_evaluation_service.py@@ -194,6 +194,16 @@ class TestOfflineEvaluationService(TestCase):self.assertEqual(new_ledger.organization_id, other_organization.id)self.assertEqual(new_ledger.quantity, 1)+ def test_privileged_result_deletion_removes_retry_identity_but_not_recorded_usage(self) -> None:+ self.ingestion.upload(self.experiment.id, self.submission)+ OfflineEvaluationResult.objects.for_team(self.team.id).all().delete()+ replay = self.ingestion.upload(self.experiment.id, self.submission)+ self.assertTrue(replay.results[0].created)+ self.assertEqual(+ list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).values_list("quantity", flat=True)),+ [1, 1],+ )+def test_child_environment_keeps_the_entire_experiment_graph_and_hosted_links(self) -> None:child = Team.objects.create(organization=self.team.organization, parent_team=self.team)lifecycle = OfflineExperimentService(team_id=child.id, user_access_control=None)@@ -591,23 +601,34 @@ class TestOfflineEvaluationServiceConcurrency(TransactionTestCase):finally:connection.close()- def test_concurrent_duplicate_uploads_return_the_same_result(self) -> None:+ @parameterized.expand([(False,), (True,)])+ def test_concurrent_duplicate_uploads_return_the_same_result(self, overlapping: bool) -> None:+ second_submission = self.submission+ if overlapping:+ new_item = replace(self.submission.items[0], id=uuid4())+ second_submission = UploadSubmission(+ items=[new_item],+ results=[replace(self.submission.results[0], item_id=new_item.id), self.submission.results[0]],+ )second_started = Event()with ThreadPoolExecutor(max_workers=1) as executor:with transaction.atomic():first = self.ingestion.upload(self.experiment.id, self.submission)pending = executor.submit(self._run_second_operation,- lambda: self.ingestion.upload(self.experiment.id, self.submission),+ lambda: self.ingestion.upload(self.experiment.id, second_submission),second_started,)self.assertTrue(second_started.wait(timeout=10))second = pending.result(timeout=10)assert isinstance(second, UploadReceipt)- self.assertEqual(second.results, [replace(first.results[0], created=False)])- self.assertEqual(OfflineEvaluationResult.objects.for_team(self.team.id).count(), 1)- self.assertEqual(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).get().quantity, 1)+ self.assertEqual(second.results[-1], replace(first.results[0], created=False))+ self.assertEqual(OfflineEvaluationResult.objects.for_team(self.team.id).count(), 2 if overlapping else 1)+ self.assertEqual(+ list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).values_list("quantity", flat=True)),+ [1, 1] if overlapping else [1],+ )def test_concurrent_experiment_creation_returns_one_identity(self) -> None:submission = ExperimentSubmission(id=uuid4(), name="Retried creation", started_at=self.experiment.started_at)diff --git a/tach.toml b/tach.tomlindex ab8f165ead23..c487de8ed1f9 100644--- a/tach.toml+++ b/tach.toml@@ -1745,6 +1745,13 @@ expose = ["backend\\.llm.*","backend\\.markdown_outline.*","backend\\.models.*",+ "backend\\.offline_evaluation_billing.get_offline_evaluation_usage",+ "backend\\.offline_evaluation_service.OfflineEvaluationIngestionService",+ "backend\\.offline_evaluation_service.OfflineExperimentService",+ "backend\\.offline_evaluation_types.ExperimentSubmission",+ "backend\\.offline_evaluation_types.ItemSubmission",+ "backend\\.offline_evaluation_types.ResultSubmission",+ "backend\\.offline_evaluation_types.UploadSubmission","backend\\.prompt_references.*","backend\\.queries.*","backend\\.summarization.*",