Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): report offline evaluation usage (#124862)

Radu Raiceaf57bf86b8003
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 1f1ce97fcc7f..c74f74c302a3 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -391,6 +391,8 @@ Each upload must contain a result, and every declared item must be referenced by
Requests exceeding the body limit return HTTP 413.
Per-caller and shared project limits allow 60 requests per minute and 1,000 per hour; HTTP 429 responses include retry guidance.
The shared project limits apply across the parent project and all its child environments, regardless of which credentials each request uses.
+Uploads containing new results also check the shared AI observability billing quota before writing items, results, payloads, or usage.
+While the quota is active, these uploads return HTTP 402 with code `quota_limit_exceeded`; exact retries remain available and add no usage.
The optional `run_source` accepts `ci`, `local`, `scheduled`, or null; empty strings are invalid.
For hosted datasets, set `dataset_revision_id` to the UUID of a hosted dataset revision when you create the experiment.
@@ -399,7 +401,7 @@ Local and external datasets do not require hosted links; they can use the `*_ide
Large payloads have separate storage and 30-day deadlines anchored to first acceptance.
Retries neither extend deadlines nor restore deleted payloads.
-Automatic payload deletion and usage billing are not enabled by these endpoints.
+Automatic payload deletion is not enabled by these endpoints.
### Product usage events
@@ -435,13 +437,60 @@ Exact retries add no usage, including retries split or regrouped into different
Mixed uploads count only new results, and changed content still returns a conflict.
Each entry stores the environment's team ID, the owning organization's ID at acceptance, the experiment ID, and the same server acceptance timestamp as its new results.
-These identifiers are stored independently of the product records, so deleting results, experiments, or their owners does not remove usage.
+These identifiers are stored independently of the product records, so deleting results, experiments, or their owners does not remove ledger entries.
Moving a project to another organization does not rewrite historical attribution.
-The ownership snapshots and timestamp preserve the information needed for future aggregation by organization, team, and UTC day.
Recording begins with newly accepted results after deployment; there is no historical backfill, compaction, or ledger cleanup.
-The ledger has no editing API or editable admin interface.
-Organization usage reporting, billing, actual charging, and quota enforcement remain unchanged and do not consume this ledger.
+The ledger has no editing API, admin registration, or production update/delete path.
+The database enforces positive quantities and result identity uniqueness; experiment row locks serialize overlapping uploads, and database transactions include results, payloads, and the ledger write.
+Independent scalar identifiers prevent product deletion cascades from deleting usage.
+`editable=False`, environment scoping, and server-assigned fields are application protections, not database immutability: privileged SQL and ORM bulk updates/deletes can still change the ledger.
+
+Retry deduplication uses retained result identities, not the batch ledger. Payload expiry preserves these identities.
+There is no result/experiment deletion endpoint. Privileged deletion of a result followed by reupload while its experiment is still open creates a new result and another usage unit; the ledger cannot identify the old metric after deletion.
+Operators must not delete durable result identities and then replay uploads as a billing-neutral operation. Any future identity cleanup needs an explicit replay policy before deployment.
+
+### Usage reporting and shared AIO meter
+
+The `organization usage report` exposes:
+
+| Property | Meaning |
+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| `ai_ingested_event_count_in_period` | Billing-eligible ingested AIO events, preserving wallet-billed gateway and sponsored-event exemptions. |
+| `ai_offline_eval_metric_count_in_period` | Sum of eligible offline ledger quantities accepted in the reporting window. |
+| `ai_event_count_in_period` | The sum of the two components. |
+
+Organization reports include the same breakdown in each `teams` entry, including existing environments that have moved. A moved environment can appear under both organizations, with only the usage attributed to each.
+`team_count` continues to count live billable teams; historical entries do not increase it.
+The existing `ai_event_count_in_period` → `llm_events` contract remains the shared AIO meter. Reporting does not deduct the standard 100,000-unit monthly allowance or determine prices or custom allowances; billing owns those calculations.
+
+The `llm analytics usage` event adds `ai_offline_eval_metric_count_in_period` from the same offline ledger aggregation. Its existing `ai_metric_count`, event-type analytics, traces, costs, tokens, and dimension breakdowns remain ClickHouse-derived. Billing-eligible ingested counts and combined billing totals belong to the organization usage report; analytics reporting does not run an additional billing scan.
+
+`get_offline_evaluation_usage` performs database-side `SUM(quantity)` grouped by the stored organization and environment, with `accepted_at >= period_start AND accepted_at < period_end`.
+Eligibility follows current team records, like ingested AI events: deleted teams, demo teams, and teams in internal-metrics organizations are excluded. The stored organization must also still exist and not be internal-metrics.
+The ledger retains all accepted usage; reporting applies these exclusions without recording a separate eligibility field.
+It uses timezone-aware UTC windows and never counts ledger rows or current results.
+Scheduled daily reports and current-day quota refresh query the offline ledger with midnight-to-midnight windows, including the final microsecond of the day.
+The exclusive day end applies only to the offline ledger query; shared report metadata and other usage queries retain their existing windows.
+The existing acceptance-time index supports the range filter. Team and organization eligibility use SQL subqueries in the same aggregation query. Only grouped totals reach Python; there is no query per metric, team, or organization.
+
+Celery and Temporal compose offline quantities after selecting the ingested counter source, so source switching and shadow comparisons remain about ingested events only.
+Temporal queries the ledger in its aggregation activity before emitting chunks; existing query files and activity inputs need no new serialized counter.
+Offline-only organizations reach the normal delivery path when they have eligible usage. Stored attribution can include an organization whose team has since moved elsewhere.
+Deleted teams' usage is omitted from newly generated reports, even though its ledger entries remain. Deleted organizations are also omitted; no customer or organization is recreated.
+
+Current-day quota refresh adds eligible offline usage to `llm_events.todays_usage` by the stored organization, using the same current-team exclusions.
+It leaves the prior-period `usage` received from billing intact. Ledger failures abort before quota updates or report delivery, allowing existing retries without publishing an incomplete total.
+Organizations included only through offline usage are read in bounded, buffered pages so database connection cleanup between quota batches cannot interrupt the scan.
+Reports are absolute totals: rereading an unchanged window does not consume or modify ledger entries. Existing analytics dispatch claims and already-reported checks still apply.
+Separately scheduled reports can observe different late-arriving data; identical windows and ledger snapshots produce matching offline metric totals.
+
+### Deployment
+
+Offline usage reporting uses the requested reporting window without a separate charging-start date or activation setting.
+The reporting integration uses the existing ledger schema without an additional migration.
+Temporal computes the offline ledger's exclusive day end inside the aggregation activity.
+The shared workflow window and already serialized reports are unchanged.
## Resource tags
diff --git a/ee/billing/quota_limiting.py b/ee/billing/quota_limiting.py
index fa4c0d1b1dfc..00756b64b408 100644
--- a/ee/billing/quota_limiting.py
+++ b/ee/billing/quota_limiting.py
@@ -1,6 +1,6 @@
import copy
import json
-from collections.abc import Callable, Collection, Iterable, Mapping, Sequence
+from collections.abc import Callable, Collection, Iterable, Iterator, Mapping, Sequence
from datetime import datetime, timedelta
from enum import Enum
from functools import lru_cache, partial
@@ -41,6 +41,8 @@ from posthog.tasks.usage_report import (
from posthog.usage_counters import UsageCounter, UsageCounterService, log_usage_counter_comparisons
from posthog.utils import get_current_day
+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage
+
from ee.billing.quota_usage import QuotaUsageBatch, retry_quota_database_operation
logger = structlog.get_logger(__name__)
@@ -1245,7 +1247,13 @@ class _QuotaUsageReports:
team_tokens: dict[str, list[str]]
-def _aggregate_quota_usage(teams: Sequence[Team], all_data: dict[str, dict[int, int]]) -> _QuotaUsageReports:
+def _aggregate_quota_usage(
+ teams: Sequence[Team],
+ all_data: dict[str, dict[int, int]],
+ *,
+ offline_usage: dict[str, dict[int, int]] | None = None,
+ organizations: Sequence[Organization] = (),
+) -> _QuotaUsageReports:
todays_usage_report: dict[str, UsageCounters] = {}
orgs_by_id: dict[str, Organization] = {}
teams_by_org: dict[str, list[str]] = {}
@@ -1298,9 +1306,38 @@ def _aggregate_quota_usage(teams: Sequence[Team], all_data: dict[str, dict[int,
if team.api_token:
teams_by_org.setdefault(org_id, []).append(team.api_token)
+ for organization in organizations:
+ org_id = str(organization.id)
+ orgs_by_id.setdefault(org_id, organization)
+ todays_usage_report.setdefault(org_id, cast(UsageCounters, dict.fromkeys(UsageCounters.__annotations__, 0)))
+ for org_id, report in todays_usage_report.items():
+ report["llm_events"] += sum((offline_usage or {}).get(org_id, {}).values())
return _QuotaUsageReports(usage=todays_usage_report, organizations=orgs_by_id, team_tokens=teams_by_org)
+def _iter_quota_organizations(
+ teams: Iterable[Team], offline_usage: dict[str, dict[int, int]]
+) -> Iterator[tuple[Organization, list[Team]]]:
+ seen: set[str] = set()
+ for organization_id, group in groupby(teams, key=lambda team: team.organization_id):
+ org_teams = list(group)
+ seen.add(str(organization_id))
+ yield org_teams[0].organization, org_teams
+ organizations = (
+ Organization.objects.filter(id__in=offline_usage.keys() - seen)
+ .only("id", "usage", "created_at", "never_drop_data", "customer_trust_scores", "customer_id")
+ .order_by("id")
+ )
+ # Buffer each page because quota updates close database connections between batches.
+ page = organizations
+ while batch := list(page[:QUOTA_ORGANIZATION_BATCH_SIZE]):
+ for organization in batch:
+ yield organization, []
+ if len(batch) < QUOTA_ORGANIZATION_BATCH_SIZE:
+ return
+ page = organizations.filter(id__gt=batch[-1].id)
+
+
def update_all_orgs_billing_quotas(
dry_run: bool = False,
progress_callback: Callable[[str, str, str], None] | None = None,
@@ -1345,6 +1382,9 @@ def update_all_orgs_billing_quotas(
tag_queries(product=Product.BILLING, feature=Feature.QUOTA_LIMITING)
logger.info("quota_limiting_run", phase="queries", status="start")
queries_start = time()
+ offline_usage = _timed_query(
+ "offline_evaluation_usage", get_offline_evaluation_usage, period.start, period.start + timedelta(days=1)
+ )
counter_report = _timed_query("usage_counters", service.fetch_report, period, plan=plan)
log_usage_counter_comparisons(counter_report, caller="quota_limiting", date=period.start.date().isoformat())
@@ -1467,12 +1507,14 @@ def update_all_orgs_billing_quotas(
quota_limiting_suspended_orgs[resource.value].pop(org_id, None)
# The shared iterator keeps an organization's teams contiguous, including across read pages.
- organization_teams = (list(group) for _, group in groupby(teams_iterator, key=lambda team: team.organization_id))
+ organization_teams = _iter_quota_organizations(teams_iterator, offline_usage)
for team_groups in batched(organization_teams, QUOTA_ORGANIZATION_BATCH_SIZE, strict=False):
if not settings.TEST:
close_old_connections()
- teams = [team for group in team_groups for team in group]
- reports = _aggregate_quota_usage(teams, all_data)
+ teams = [team for _, group in team_groups for team in group]
+ reports = _aggregate_quota_usage(
+ teams, all_data, offline_usage=offline_usage, organizations=[org for org, _ in team_groups]
+ )
orgs_by_id = reports.organizations
teams_by_org = reports.team_tokens
candidates_start = time()
diff --git a/ee/billing/test/test_quota_limiting.py b/ee/billing/test/test_quota_limiting.py
index 2776c77ef9ff..c6709f0fa1eb 100644
--- a/ee/billing/test/test_quota_limiting.py
+++ b/ee/billing/test/test_quota_limiting.py
@@ -23,6 +23,8 @@ from posthog.redis import get_client
from posthog.test.fixtures import create_app_metric2
from posthog.usage_counters import UsageRecordTotal, UsageRecordTotals
+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger
+
from ee.billing.quota_limiting import (
INFORMATIONAL_USAGE_RESOURCES,
QUOTA_LIMIT_DATA_RETENTION_FLAG,
@@ -87,6 +89,30 @@ class TestQuotaLimiting(BaseTest):
with self.captureOnCommitCallbacks(execute=True):
return update_all_orgs_billing_quotas()
+ @parameterized.expand([(False,), (True,)])
+ @time_machine.travel("2026-10-31T23:59:59.999999Z", tick=False)
+ def test_offline_usage_updates_today_without_recounting_billing_history(self, deleted_team: bool) -> None:
+ self.organization.usage = {
+ "period": ["2026-10-01T00:00:00Z", "2026-11-01T00:00:00Z"],
+ "llm_events": {"usage": 50, "todays_usage": 0, "limit": 1000},
+ }
+ self.organization.save(update_fields=["usage"])
+ for accepted_at, quantity in ((now() - datetime.timedelta(days=1), 50), (now(), 20), (now(), 10)):
+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ organization_id=self.organization.id,
+ experiment_id=uuid4(),
+ accepted_at=accepted_at,
+ quantity=quantity,
+ )
+ if deleted_team:
+ self.team.delete()
+ for _ in range(2):
+ self._run_quota_limiting()
+ self.organization.refresh_from_db()
+ self.assertEqual(self.organization.usage["llm_events"]["usage"], 50)
+ self.assertEqual(self.organization.usage["llm_events"]["todays_usage"], 0 if deleted_team else 30)
+
@patch("posthoganalytics.capture")
@patch("posthoganalytics.feature_enabled", return_value=True)
@time_machine.travel("2021-01-25T23:59:59Z", tick=False)
@@ -264,7 +290,7 @@ class TestQuotaLimiting(BaseTest):
self.organization.save()
flush_persons_and_events()
- with self.assertNumQueries(FuzzyInt(3, 10)):
+ with self.assertNumQueries(FuzzyInt(4, 11)):
result = self._run_quota_limiting()
assert patch_capture.call_count == 0 # No events should be captured since org won't be limited
assert result.quota_limited_orgs["events"] == {}
diff --git a/posthog/tasks/ai_observability_usage_report.py b/posthog/tasks/ai_observability_usage_report.py
index 7861243174be..c03ddd27e98d 100644
--- a/posthog/tasks/ai_observability_usage_report.py
+++ b/posthog/tasks/ai_observability_usage_report.py
@@ -16,6 +16,7 @@ from posthog.constants import AI_EVENT_NAME_PREFIX
from posthog.exceptions_capture import capture_exception
from posthog.logging.timing import timed_log
from posthog.models.event.new_events_schema import events_read_table, use_new_events_schema
+from posthog.models.organization import Organization
from posthog.models.property.util import get_property_string_expr
from posthog.models.team.team import Team
from posthog.ph_client import PH_US_API_KEY
@@ -24,6 +25,8 @@ from posthog.tasks.report_utils import capture_event
from posthog.tasks.utils import CeleryQueue
from posthog.utils import DayRange, get_instance_region, get_previous_day
+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage
+
logger = structlog.get_logger(__name__)
@@ -917,6 +920,7 @@ def _get_all_ai_observability_reports(
dict mapping organization_id to usage data
"""
logger.info("Querying AI observability usage data")
+ offline_usage = get_offline_evaluation_usage(period.start, period.start + timedelta(days=1))
# Phase 1: Get all team_ids with report trigger events (fast query)
try:
@@ -933,7 +937,7 @@ def _get_all_ai_observability_reports(
# Re-raise so Celery's autoretry_for=(Exception,) kicks in. Do not swallow.
raise
- if not team_ids:
+ if not team_ids and not offline_usage:
logger.info("No teams with AI observability trigger events found")
return {}
@@ -942,7 +946,7 @@ def _get_all_ai_observability_reports(
# Phase 2: Get all metrics in a single combined query
logger.info("Querying all AI metrics")
try:
- all_metrics = get_all_ai_metrics(period.start, period.end, team_ids)
+ all_metrics = get_all_ai_metrics(period.start, period.end, team_ids) if team_ids else {}
except Exception:
logger.warning(
"[AIO Usage Error] metrics query failed",
@@ -957,7 +961,7 @@ def _get_all_ai_observability_reports(
# Phase 3: Get distinct trace counts
logger.info("Querying AI trace counts")
try:
- ai_trace_counts = get_ai_trace_counts(period.start, period.end, team_ids)
+ ai_trace_counts = get_ai_trace_counts(period.start, period.end, team_ids) if team_ids else {}
except Exception:
logger.warning(
"[AIO Usage Error] trace counts query failed",
@@ -973,7 +977,9 @@ def _get_all_ai_observability_reports(
llm_prompt_fetched_counts: dict[int, int] = {}
try:
logger.info("Querying LLM prompt fetched counts")
- llm_prompt_fetched_counts = get_llm_prompt_fetched_counts(period.start, period.end, team_ids)
+ llm_prompt_fetched_counts = (
+ get_llm_prompt_fetched_counts(period.start, period.end, team_ids) if team_ids else {}
+ )
logger.info(f"Retrieved prompt fetched counts for {len(llm_prompt_fetched_counts)} teams")
except Exception as err:
logger.warning(
@@ -984,7 +990,7 @@ def _get_all_ai_observability_reports(
# Phase 5: Get all dimension breakdowns in a single combined query
logger.info("Querying all AI dimension breakdowns")
try:
- all_breakdowns = get_all_ai_dimension_breakdowns(period.start, period.end, team_ids)
+ all_breakdowns = get_all_ai_dimension_breakdowns(period.start, period.end, team_ids) if team_ids else {}
except Exception:
logger.warning(
"[AIO Usage Error] breakdowns query failed",
@@ -999,7 +1005,7 @@ def _get_all_ai_observability_reports(
# Phase 6: Get LLM feedback survey metrics
logger.info("Querying LLM feedback survey metrics")
try:
- survey_metrics = get_llm_feedback_survey_metrics(period.start, period.end, team_ids)
+ survey_metrics = get_llm_feedback_survey_metrics(period.start, period.end, team_ids) if team_ids else {}
except Exception:
logger.warning(
"[AIO Usage Error] surveys query failed",
@@ -1015,17 +1021,21 @@ def _get_all_ai_observability_reports(
teams = Team.objects.filter(id__in=team_ids).select_related("organization")
team_to_org: dict[int, str] = {team.id: str(team.organization_id) for team in teams}
org_id_to_name: dict[str, str] = {str(team.organization_id): team.organization.name for team in teams}
+ org_id_to_name.update(
+ (str(org.id), org.name) for org in Organization.objects.filter(id__in=offline_usage).only("id", "name")
+ )
# Aggregate by organization
org_reports: dict[str, dict[str, Any]] = {}
- for team_id, org_id in team_to_org.items():
+ for org_id in org_id_to_name:
if org_id not in org_reports:
org_reports[org_id] = {
"organization_id": org_id,
"organization_name": org_id_to_name.get(org_id, ""),
"period_start": period.start.isoformat(),
"period_end": period.end.isoformat(),
+ "ai_offline_eval_metric_count_in_period": sum(offline_usage.get(org_id, {}).values()),
"ai_generation_count": 0,
"ai_embedding_count": 0,
"ai_span_count": 0,
@@ -1069,6 +1079,7 @@ def _get_all_ai_observability_reports(
"cost_model_provider_breakdown": {},
}
+ for team_id, org_id in team_to_org.items():
report = org_reports[org_id]
# Add metrics from TeamMetrics dataclass
diff --git a/posthog/tasks/test/test_ai_observability_usage_report.py b/posthog/tasks/test/test_ai_observability_usage_report.py
index 8a97e8b97ac7..325b637d35ac 100644
--- a/posthog/tasks/test/test_ai_observability_usage_report.py
+++ b/posthog/tasks/test/test_ai_observability_usage_report.py
@@ -1,3 +1,6 @@
+import dataclasses
+from collections import defaultdict
+from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from uuid import uuid4
@@ -12,6 +15,7 @@ from posthog.test.base import (
)
from unittest.mock import MagicMock, patch
+from django.db import DatabaseError, connection
from django.test import SimpleTestCase
from parameterized import parameterized
@@ -35,7 +39,27 @@ from posthog.tasks.ai_observability_usage_report import (
get_teams_with_ai_events,
send_ai_observability_usage_reports,
)
-from posthog.utils import get_previous_day
+from posthog.tasks.usage_report import _get_all_org_reports, capture_report, has_non_zero_usage
+from posthog.temporal.usage_report.aggregator import iter_org_reports
+from posthog.temporal.usage_report.types import WorkflowContext
+from posthog.usage_counters import UsageCounterService
+from posthog.utils import DayRange, get_previous_day
+
+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger
+from products.ai_observability.backend.models.score_definitions import ScoreDefinition
+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage
+from products.ai_observability.backend.offline_evaluation_service import (
+ OfflineEvaluationIngestionService,
+ OfflineExperimentService,
+)
+from products.ai_observability.backend.offline_evaluation_types import (
+ ExperimentSubmission,
+ ItemSubmission,
+ ResultSubmission,
+ UploadSubmission,
+)
+
+from ee.billing.quota_limiting import _aggregate_quota_usage
@time_machine.travel("2022-01-10T00:01:00Z", tick=False)
@@ -522,6 +546,180 @@ class TestAIObservabilityUsageReport(APIBaseTest, ClickhouseTestMixin, Clickhous
assert metrics.active_survey_count == 2 # survey_id_1 and survey_id_2
assert metrics.response_count == 5 # 3 + 2 (only "survey sent" events)
+ @parameterized.expand([(50, 30), (0, 30), (50, 0), (0, 0)])
+ @time_machine.travel("2026-10-31T23:59:59.999999Z", tick=False)
+ def test_upload_to_both_reports_and_quota(self, ingested: int, offline: int) -> None:
+ period = get_previous_day(at=datetime(2026, 11, 1, tzinfo=UTC))
+ if ingested:
+ self._create_ai_events(self.team, "billing-test", "$ai_generation", ingested, timestamp=period.start)
+ if offline:
+ experiment = (
+ OfflineExperimentService(team_id=self.team.id, user_access_control=None)
+ .create(ExperimentSubmission(id=uuid4(), name="Billing fixture", started_at=period.start))
+ .experiment
+ )
+ scorer = ScoreDefinition.objects.create(team=self.team, name="Accuracy", kind="numeric")
+ versions = [scorer.create_new_version(config={}, created_by=None) for _ in range(6)]
+ items = [ItemSubmission(id=uuid4()) for _ in range(5)]
+ results = [
+ ResultSubmission(item_id=item.id, scorer_version_id=version.id, status="ok", value=0.8)
+ for item in items
+ for version in versions
+ ]
+ ingestion = OfflineEvaluationIngestionService(team_id=self.team.id, user_access_control=None)
+ for batch in (results[:20], results[20:]):
+ submission = UploadSubmission(
+ items=[item for item in items if item.id in {result.item_id for result in batch}], results=batch
+ )
+ ingestion.upload(experiment.id, submission)
+ ingestion.upload(experiment.id, submission)
+ ingestion.upload(
+ experiment.id, UploadSubmission(items=list(reversed(items)), results=list(reversed(results)))
+ )
+ self.assertEqual(
+ sorted(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).values_list("quantity", flat=True)),
+ [10, 20],
+ )
+ rejected_item = ItemSubmission(id=uuid4(), payload={"input": "Rejected transaction"})
+
+ def fail_ledger_insert(
+ execute: Callable[..., object], sql: str, params: object, many: bool, context: object
+ ) -> object:
+ if sql.startswith("INSERT INTO") and OfflineEvaluationUsageLedger._meta.db_table in sql:
+ raise DatabaseError("Ledger write unavailable")
+ return execute(sql, params, many, context)
+
+ with (
+ connection.execute_wrapper(fail_ledger_insert),
+ self.assertRaisesMessage(DatabaseError, "Ledger write unavailable"),
+ ):
+ ingestion.upload(
+ experiment.id,
+ UploadSubmission(
+ items=[rejected_item], results=[dataclasses.replace(results[0], item_id=rejected_item.id)]
+ ),
+ )
+
+ counter_report = UsageCounterService().fetch_report(period)
+ all_data = defaultdict(dict, {key: dict(rows) for key, rows in counter_report.counts.items()})
+ org_id = str(self.organization.id)
+ before = list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).order_by("id").values())
+ celery_report = _get_all_org_reports(period=period, counter_report=counter_report)[org_id]
+ ctx = WorkflowContext(
+ run_id="offline-usage", date_str="2026-10-31", period_start=period.start, period_end=period.end
+ )
+ temporal_report = next(
+ report for report in iter_org_reports(all_data, ctx, {}) if report.organization_id == org_id
+ )
+ expected = {
+ "ai_ingested_event_count_in_period": ingested,
+ "ai_offline_eval_metric_count_in_period": offline,
+ "ai_event_count_in_period": ingested + offline,
+ }
+ for report in (celery_report, temporal_report):
+ self.assertEqual({key: getattr(report, key) for key in expected}, expected)
+ self.assertEqual({key: getattr(report.teams[str(self.team.id)], key) for key in expected}, expected)
+ self.assertEqual(has_non_zero_usage(report), bool(ingested + offline))
+
+ with patch(
+ "posthog.tasks.usage_report.sync_execute",
+ side_effect=AssertionError("Analytics reporting must not execute billing queries"),
+ ):
+ analytics_reports = _get_all_ai_observability_reports(period=period)
+ if ingested + offline:
+ analytics_report = analytics_reports[org_id]
+ self.assertEqual(analytics_report["ai_offline_eval_metric_count_in_period"], offline)
+ self.assertEqual(analytics_report["ai_generation_count"], ingested)
+ self.assertEqual(analytics_report["ai_metric_count"], 0)
+ self.assertNotIn("ai_ingested_event_count_in_period", analytics_report)
+ self.assertNotIn("ai_event_count_in_period", analytics_report)
+ with (
+ patch("posthog.tasks.usage_report.get_ph_client"),
+ patch("posthog.tasks.usage_report.capture_event") as org_capture,
+ patch("posthog.tasks.ai_observability_usage_report.get_ph_client"),
+ patch("posthog.tasks.ai_observability_usage_report.capture_event") as ai_capture,
+ ):
+ capture_report(organization_id=org_id, full_report_dict=dataclasses.asdict(celery_report))
+ capture_ai_observability_report(organization_id=org_id, report_dict=analytics_report)
+ org_payload = next(
+ call.kwargs["properties"]
+ for call in org_capture.call_args_list
+ if call.kwargs["name"] == "organization usage report"
+ )
+ self.assertEqual({key: org_payload[key] for key in expected}, expected)
+ self.assertEqual(
+ ai_capture.call_args.kwargs["properties"]["ai_offline_eval_metric_count_in_period"], offline
+ )
+ else:
+ self.assertEqual(analytics_reports, {})
+ quotas = _aggregate_quota_usage(
+ [self.team],
+ all_data,
+ offline_usage=get_offline_evaluation_usage(period.start, period.start + timedelta(days=1)),
+ )
+ self.assertEqual(quotas.usage[org_id]["llm_events"], ingested + offline)
+ self.assertEqual(
+ list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).order_by("id").values()), before
+ )
+
+ @parameterized.expand([(False,), (True,)])
+ @time_machine.travel("2026-11-01T00:01:00Z", tick=False)
+ def test_both_reports_preserve_moved_team_usage_but_exclude_deleted_teams(self, deleted_team: bool) -> None:
+ period = DayRange(start=datetime(2026, 10, 31, tzinfo=UTC), end=datetime(2026, 11, 1, tzinfo=UTC))
+ new_org = Organization.objects.create(name="New owner")
+ for org, quantity in ((self.organization, 30), (new_org, 4)):
+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ organization_id=org.id,
+ experiment_id=uuid4(),
+ accepted_at=period.start,
+ quantity=quantity,
+ )
+ Team.objects.filter(id=self.team.id).update(organization=new_org)
+ if deleted_team:
+ self.team.delete()
+ analytics = _get_all_ai_observability_reports(period=period)
+ reports = _get_all_org_reports(period=period)
+ for org, expected in ((self.organization, 30), (new_org, 4)):
+ org_id = str(org.id)
+ if deleted_team:
+ self.assertNotIn(org_id, analytics)
+ self.assertNotIn(org_id, reports)
+ continue
+ self.assertEqual(analytics[org_id]["ai_offline_eval_metric_count_in_period"], expected)
+ self.assertEqual(reports[org_id].ai_event_count_in_period, expected)
+
+ @time_machine.travel("2026-11-01T00:01:00Z", tick=False)
+ def test_failed_ledger_read_does_not_claim_the_analytics_dispatch(self) -> None:
+ OfflineEvaluationUsageLedger.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ organization_id=self.organization.id,
+ experiment_id=uuid4(),
+ accepted_at=datetime(2026, 10, 31, tzinfo=UTC),
+ quantity=30,
+ )
+
+ def fail_ledger_read(
+ execute: Callable[..., object], sql: str, params: object, many: bool, context: object
+ ) -> object:
+ if sql.startswith("SELECT") and OfflineEvaluationUsageLedger._meta.db_table in sql:
+ raise DatabaseError("Ledger unavailable")
+ return execute(sql, params, many, context)
+
+ with (
+ patch("posthoganalytics.feature_enabled", return_value=False),
+ patch("posthog.tasks.ai_observability_usage_report.capture_ai_observability_report.delay") as dispatch,
+ ):
+ with (
+ connection.execute_wrapper(fail_ledger_read),
+ self.assertRaisesMessage(DatabaseError, "Ledger unavailable"),
+ ):
+ send_ai_observability_usage_reports()
+ dispatch.assert_not_called()
+ send_ai_observability_usage_reports()
+ self.assertEqual(dispatch.call_count, 1)
+ self.assertEqual(dispatch.call_args.kwargs["report_dict"]["ai_offline_eval_metric_count_in_period"], 30)
+
def test_full_ai_observability_report(self) -> None:
"""Test the full AI observability report generation."""
# Create second organization and team
@@ -1374,7 +1572,10 @@ class TestAIObservabilityUsageReportTaskWiring(SimpleTestCase):
task = send_ai_observability_usage_reports
task.push_request(retries=retries, called_directly=called_directly, is_eager=True)
try:
- with pytest.raises(Exception, match="CH down"):
+ with (
+ patch("posthog.tasks.ai_observability_usage_report.get_offline_evaluation_usage", return_value={}),
+ pytest.raises(Exception, match="CH down"),
+ ):
task.run()
finally:
task.pop_request()
diff --git a/posthog/tasks/usage_report.py b/posthog/tasks/usage_report.py
index d1dfe979ef89..39108726b446 100644
--- a/posthog/tasks/usage_report.py
+++ b/posthog/tasks/usage_report.py
@@ -63,6 +63,7 @@ from posthog.usage_counters import (
)
from posthog.utils import DayRange, get_helm_info_env, get_instance_realm, get_instance_region, get_previous_day
+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage
from products.batch_exports.backend.facade import api as batch_exports_api
from products.cdp.backend.models.hog_functions.hog_function import HogFunction, HogFunctionType
from products.cdp.backend.models.plugin import PluginConfig
@@ -297,6 +298,8 @@ class UsageReportCounters:
# AI observability
ai_event_count_in_period: int
+ ai_ingested_event_count_in_period: int = dataclasses.field(default=0, kw_only=True)
+ ai_offline_eval_metric_count_in_period: int = dataclasses.field(default=0, kw_only=True)
# AI Billing Credits (PostHog AI feature usage)
ai_credits_used_in_period: int
@@ -3579,6 +3582,7 @@ def _get_team_report(all_data: dict[str, Any], team: Team) -> UsageReportCounter
roblox_events_count_in_period=all_data["teams_with_roblox_events_count_in_period"].get(team.id, 0),
rust_events_count_in_period=all_data["teams_with_rust_events_count_in_period"].get(team.id, 0),
ai_event_count_in_period=all_data["teams_with_ai_event_count_in_period"].get(team.id, 0),
+ ai_ingested_event_count_in_period=all_data["teams_with_ai_event_count_in_period"].get(team.id, 0),
ai_credits_used_in_period=all_data["teams_with_ai_credits_used_in_period"].get(team.id, 0),
signals_credits_used_in_period=all_data["teams_with_signals_credits_used_in_period"].get(team.id, 0),
posthog_code_credits_used_in_period=combine_posthog_code_credits(
@@ -3682,9 +3686,63 @@ def _add_team_report_to_org_reports(
)
+def iter_reports_with_offline_evaluation_usage(
+ reports: Iterable[OrgReport],
+ offline_usage: dict[str, dict[int, int]],
+ period_start: datetime,
+ *,
+ organization_ids: Sequence[str] | None = None,
+) -> Iterator[OrgReport]:
+ def add_usage(report: OrgReport) -> OrgReport:
+ for team_id, quantity in offline_usage.get(report.organization_id, {}).items():
+ team_key = str(team_id)
+ if team_key not in report.teams:
+ report.teams[team_key] = UsageReportCounters(
+ **{field.name: 0 for field in dataclasses.fields(UsageReportCounters)}
+ )
+ team_report = report.teams[team_key]
+ team_report.ai_offline_eval_metric_count_in_period = quantity
+ team_report.ai_event_count_in_period = team_report.ai_ingested_event_count_in_period + quantity
+ report.ai_offline_eval_metric_count_in_period = sum(offline_usage.get(report.organization_id, {}).values())
+ report.ai_event_count_in_period = (
+ report.ai_ingested_event_count_in_period + report.ai_offline_eval_metric_count_in_period
+ )
+ return report
+
+ seen: set[str] = set()
+ for report in reports:
+ seen.add(report.organization_id)
+ yield add_usage(report)
+ missing = offline_usage.keys() - seen
+ if organization_ids:
+ missing &= set(organization_ids)
+ user_counts = dict(
+ OrganizationMembership.objects.filter(organization_id__in=missing)
+ .values("organization_id")
+ .annotate(count=Count("id"))
+ .values_list("organization_id", "count")
+ )
+ for organization in Organization.objects.filter(id__in=missing).only("id", "name", "created_at").iterator():
+ yield add_usage(
+ OrgReport(
+ date=period_start.strftime("%Y-%m-%d"),
+ organization_id=str(organization.id),
+ organization_name=organization.name,
+ organization_created_at=organization.created_at.isoformat(),
+ organization_user_count=user_counts.get(organization.id, 0),
+ team_count=0,
+ teams={},
+ **dataclasses.asdict(
+ UsageReportCounters(**{field.name: 0 for field in dataclasses.fields(UsageReportCounters)})
+ ),
+ )
+ )
+
+
def _get_all_org_reports(*, period: DayRange, counter_report: UsageCounterReport | None = None) -> dict[str, OrgReport]:
logger.info("Querying all org reports", period_start=period.start, period_end=period.end)
+ offline_usage = get_offline_evaluation_usage(period.start, period.start + timedelta(days=1))
all_data = _get_all_usage_data_as_team_rows(period.start, period.end, counter_report)
org_reports: dict[str, OrgReport] = {}
@@ -3699,7 +3757,10 @@ def _get_all_org_reports(*, period: DayRange, counter_report: UsageCounterReport
logger.info("Generating org reports complete", teams_count=teams_count, org_reports_count=len(org_reports))
- return org_reports
+ return {
+ report.organization_id: report
+ for report in iter_reports_with_offline_evaluation_usage(org_reports.values(), offline_usage, period.start)
+ }
def _get_full_org_usage_report(org_report: OrgReport, instance_metadata: InstanceMetadata) -> FullUsageReport:
@@ -3772,12 +3833,24 @@ def iter_reports_with_usage_counter_metadata(
report_teams: set[int] = set()
missing: dict[str, dict[str, UsageCounterComparison]] = {}
wanted = set(organization_ids) if organization_ids else None
+ comparison_team_owners: dict[int, str] | None = None
for report in org_reports:
report.usage_sources = counter_report.usage_sources
report.counter_comparisons = {} if counter_report.counter_comparisons is not None else None
if counter_report.counter_comparisons is not None:
org_id = report.organization_id
- teams = [int(team_id) for team_id in report.teams]
+ if report.ai_offline_eval_metric_count_in_period and comparison_team_owners is None:
+ comparison_team_owners = {
+ team_id: str(owner)
+ for team_id, owner in Team.objects.filter(
+ is_demo=False, organization__for_internal_metrics=False
+ ).values_list("id", "organization_id")
+ }
+ teams = [
+ int(team_id)
+ for team_id in report.teams
+ if comparison_team_owners is None or comparison_team_owners.get(int(team_id)) == org_id
+ ]
report_orgs.add(org_id)
report_teams.update(teams)
reported = has_non_zero_usage(report)
diff --git a/posthog/temporal/tests/usage_report/test_aggregate_activity.py b/posthog/temporal/tests/usage_report/test_aggregate_activity.py
index 76c72efca938..0aa3fdca3528 100644
--- a/posthog/temporal/tests/usage_report/test_aggregate_activity.py
+++ b/posthog/temporal/tests/usage_report/test_aggregate_activity.py
@@ -29,6 +29,8 @@ from posthog.temporal.usage_report.queries import QUERIES, QUERY_INDEX
from posthog.temporal.usage_report.storage import queries_key, write_json
from posthog.temporal.usage_report.types import AggregateInputs, RunQueryToS3Result, WorkflowContext
+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger
+
@sync_to_async
def _make_org(name: str) -> Organization:
@@ -212,6 +214,19 @@ async def test_aggregate_writes_chunks_and_manifest(minio_workflow_ctx: Workflow
# pre-seeded orgs that would otherwise inflate the manifest counts.
ctx = minio_workflow_ctx.model_copy(update={"organization_ids": [str(org_a.id), str(org_b.id)]})
query_results = _seed_query_results(ctx, team_a.id, team_b.id)
+ write_json(queries_key(ctx, "teams_with_ai_event_count_in_period"), [[team_a.id, 50]])
+ await sync_to_async(OfflineEvaluationUsageLedger.objects.for_team(team_a.id).bulk_create)(
+ [
+ OfflineEvaluationUsageLedger(
+ team_id=team_a.id,
+ organization_id=org_a.id,
+ experiment_id=uuid.uuid4(),
+ accepted_at=ctx.period_start,
+ quantity=quantity,
+ )
+ for quantity in (20, 10)
+ ]
+ )
with patch(
"posthog.temporal.usage_report.activities.get_instance_metadata",
@@ -243,6 +258,10 @@ async def test_aggregate_writes_chunks_and_manifest(minio_workflow_ctx: Workflow
assert len(rows) == 2
by_org = {row["organization_id"]: row["usage_report"] for row in rows}
assert set(by_org.keys()) == {str(org_a.id), str(org_b.id)}
+ assert by_org[str(org_a.id)]["ai_ingested_event_count_in_period"] == 50
+ assert by_org[str(org_a.id)]["ai_offline_eval_metric_count_in_period"] == 30
+ assert by_org[str(org_a.id)]["ai_event_count_in_period"] == 80
+ assert by_org[str(org_a.id)]["teams"][str(team_a.id)]["ai_event_count_in_period"] == 80
# multi-key fan-out: web events from `all_event_metrics` landed in the
# destination key on each org's report.
@@ -443,6 +462,13 @@ async def test_aggregate_filters_by_organization_ids(minio_workflow_ctx: Workflo
ctx = minio_workflow_ctx.model_copy(update={"organization_ids": [str(org_a.id)]})
query_results = _seed_query_results(ctx, team_a.id, team_b.id)
+ await sync_to_async(OfflineEvaluationUsageLedger.objects.for_team(team_b.id).create)(
+ team_id=team_b.id,
+ organization_id=org_b.id,
+ experiment_id=uuid.uuid4(),
+ accepted_at=ctx.period_start,
+ quantity=7,
+ )
with patch(
"posthog.temporal.usage_report.activities.get_instance_metadata",
diff --git a/posthog/temporal/usage_report/aggregator.py b/posthog/temporal/usage_report/aggregator.py
index 132d9c3866ad..5cc4f4551e3c 100644
--- a/posthog/temporal/usage_report/aggregator.py
+++ b/posthog/temporal/usage_report/aggregator.py
@@ -11,7 +11,7 @@ for Temporal. Activities import from here.
import dataclasses
from collections.abc import Callable, Iterable, Iterator
-from datetime import datetime
+from datetime import datetime, timedelta
from itertools import chain, groupby, islice
from typing import Any
@@ -29,6 +29,7 @@ from posthog.tasks.usage_report import (
_get_teams_for_usage_reports,
convert_team_usage_rows_to_dict,
has_non_zero_usage,
+ iter_reports_with_offline_evaluation_usage,
serialize_full_org_report,
)
from posthog.temporal.usage_report.queries import QUERY_INDEX
@@ -49,6 +50,8 @@ from posthog.temporal.usage_report.types import (
)
from posthog.usage_counters import UsageCounterReport
+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage
+
_SANDBOX_COMPUTE_QUERY_NAME = "sandbox_compute_usage"
_SANDBOX_COMPUTE_DESTINATION_KEYS = (
"teams_with_sandbox_compute_credits_used_in_period",
@@ -214,7 +217,7 @@ def _add_team_report_to_org_reports(
)
-def iter_org_reports(
+def _iter_ingested_org_reports(
all_data: dict[str, dict[int, int]],
ctx: WorkflowContext,
org_user_counts: dict[str, int],
@@ -229,6 +232,20 @@ def iter_org_reports(
yield org_reports[org_id]
+def iter_org_reports(
+ all_data: dict[str, dict[int, int]],
+ ctx: WorkflowContext,
+ org_user_counts: dict[str, int],
+) -> Iterator[OrgReport]:
+ offline_usage = get_offline_evaluation_usage(ctx.period_start, ctx.period_start + timedelta(days=1))
+ yield from iter_reports_with_offline_evaluation_usage(
+ _iter_ingested_org_reports(all_data, ctx, org_user_counts),
+ offline_usage,
+ ctx.period_start,
+ organization_ids=ctx.organization_ids,
+ )
+
+
def write_org_report_chunks(
ctx: WorkflowContext,
org_reports: Iterable[OrgReport],
diff --git a/posthog/usage_counters.py b/posthog/usage_counters.py
index 9a3a9e9f13db..1752bf04d839 100644
--- a/posthog/usage_counters.py
+++ b/posthog/usage_counters.py
@@ -38,6 +38,7 @@ class UsageCounter(StrEnum):
MOBILE_RECORDINGS = "teams_with_mobile_recording_count_in_period"
MOBILE_BILLABLE_RECORDINGS = "teams_with_mobile_billable_recording_count_in_period"
SURVEY_RESPONSES = "teams_with_survey_responses_count_in_period"
+ # Ingested AIO events only. Offline usage is composed after organization attribution.
AI_EVENTS = "teams_with_ai_event_count_in_period"
EXCEPTIONS = "teams_with_exceptions_captured_in_period"
CDP_INVOCATIONS = "teams_with_cdp_billable_invocations_in_period"
diff --git a/products/ai_observability/backend/api/offline_experiments.py b/products/ai_observability/backend/api/offline_experiments.py
index 85b3fc241d0a..2a3916ad492d 100644
--- a/products/ai_observability/backend/api/offline_experiments.py
+++ b/products/ai_observability/backend/api/offline_experiments.py
@@ -241,7 +241,7 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
@validated_request(
UploadSubmissionSerializer,
parameters=[EXPERIMENT_ID_PARAMETER],
- responses={200: UploadReceiptSerializer, **ERROR_RESPONSES},
+ responses={200: UploadReceiptSerializer, 402: OfflineEvaluationErrorSerializer, **ERROR_RESPONSES},
)
@action(detail=True, methods=["post"])
@llma_track_latency("aio_offline_experiment_upload")
diff --git a/products/ai_observability/backend/api/test/test_offline_experiments.py b/products/ai_observability/backend/api/test/test_offline_experiments.py
index c4bafdf877b6..42439dea5053 100644
--- a/products/ai_observability/backend/api/test/test_offline_experiments.py
+++ b/products/ai_observability/backend/api/test/test_offline_experiments.py
@@ -20,6 +20,7 @@ from posthog.models import OrganizationMembership, Project, User
from posthog.models.personal_api_key import PersonalAPIKey
from posthog.models.project_secret_api_key import ProjectSecretAPIKey
from posthog.models.utils import generate_random_token_personal, hash_key_value
+from posthog.redis import get_client
from products.access_control.backend.models.access_control import AccessControl
from products.ai_observability.backend.api.offline_experiment_reads import OfflineScorerViewSet
@@ -230,7 +231,12 @@ class TestOfflineExperimentsAPI(APIBaseTest):
self.assertEqual(completed_event.args[0], completed_event.kwargs["request"].user)
self.assertTrue(completed_event.args[0].distinct_id)
- def test_ledger_failure_returns_server_error_and_mixed_upload_can_retry(self) -> None:
+ @parameterized.expand(
+ [("ledger", status.HTTP_500_INTERNAL_SERVER_ERROR), ("quota", status.HTTP_402_PAYMENT_REQUIRED)]
+ )
+ def test_rejected_mixed_upload_preserves_existing_results_and_can_retry(
+ self, rejection: str, expected_status: int
+ ) -> None:
definition = ScoreDefinition.objects.create(team=self.team, name="Correct", kind="boolean")
version = definition.create_new_version(config={}, created_by=self.user)
self._authenticate("project_key")
@@ -273,10 +279,22 @@ class TestOfflineExperimentsAPI(APIBaseTest):
return execute(sql, params, many, context)
upload = {"items": items, "results": results}
- with connection.execute_wrapper(fail_ledger_insert):
+ if rejection == "ledger":
+ with connection.execute_wrapper(fail_ledger_insert):
+ failed = self.client.post(endpoint, upload, format="json")
+ else:
+ redis_client = get_client()
+ quota_key = "@posthog/quota-limits/llm_events"
+ redis_client.zadd(quota_key, {self.team.api_token: timezone.now().timestamp() + 3600})
+ self.addCleanup(redis_client.zrem, quota_key, self.team.api_token)
+ replayed = self.client.post(endpoint, {"items": items[:1], "results": results[:1]}, format="json")
+ self.assertEqual(replayed.status_code, status.HTTP_200_OK, replayed.data)
+ self.assertEqual(replayed.data["results"][0], accepted.data["results"][0] | {"created": False})
failed = self.client.post(endpoint, upload, format="json")
+ self.assertEqual(failed.data["code"], "quota_limit_exceeded", failed.data)
+ redis_client.zadd(quota_key, {self.team.api_token: timezone.now().timestamp() - 1})
- self.assertEqual(failed.status_code, status.HTTP_500_INTERNAL_SERVER_ERROR, failed.data)
+ self.assertEqual(failed.status_code, expected_status, failed.data)
for model, rows in rows_before_failure.items():
self.assertEqual(list(model.objects.for_team(self.team.id).values()), rows)
diff --git a/products/ai_observability/backend/offline_evaluation_billing.py b/products/ai_observability/backend/offline_evaluation_billing.py
new file mode 100644
index 000000000000..8396aa24184e
--- /dev/null
+++ b/products/ai_observability/backend/offline_evaluation_billing.py
@@ -0,0 +1,30 @@
+from datetime import UTC, datetime
+
+from django.db.models import Sum
+from django.utils import timezone
+
+from posthog.models import Organization, Team
+
+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger
+
+
+def get_offline_evaluation_usage(period_start: datetime, period_end: datetime) -> dict[str, dict[int, int]]:
+ """Billable quantities by acceptance organization and environment, without consuming the ledger."""
+ if timezone.is_naive(period_start) or timezone.is_naive(period_end) or period_start >= period_end:
+ raise ValueError("Offline evaluation usage requires an aware, increasing reporting window")
+
+ rows = (
+ OfflineEvaluationUsageLedger.objects.unscoped()
+ .filter(
+ accepted_at__gte=period_start.astimezone(UTC),
+ accepted_at__lt=period_end.astimezone(UTC),
+ team_id__in=Team.objects.filter(is_demo=False, organization__for_internal_metrics=False).values("id"),
+ organization_id__in=Organization.objects.filter(for_internal_metrics=False).values("id"),
+ )
+ .values("organization_id", "team_id")
+ .annotate(total_quantity=Sum("quantity"))
+ )
+ usage: dict[str, dict[int, int]] = {}
+ for row in rows:
+ usage.setdefault(str(row["organization_id"]), {})[row["team_id"]] = int(row["total_quantity"])
+ return usage
diff --git a/products/ai_observability/backend/offline_evaluation_service.py b/products/ai_observability/backend/offline_evaluation_service.py
index eaa6efd92de3..29f53c73f7f2 100644
--- a/products/ai_observability/backend/offline_evaluation_service.py
+++ b/products/ai_observability/backend/offline_evaluation_service.py
@@ -13,6 +13,7 @@ from django.db.models import Q
from django.utils import timezone
from posthog.dataclasses import frozen
+from posthog.exceptions import QuotaLimitExceeded
from posthog.models.team import Team
from products.ai_observability.backend.dataset_queries import dataset_item_versions_at_revision
@@ -36,6 +37,8 @@ from products.ai_observability.backend.offline_evaluation_types import (
)
from products.ai_observability.backend.score_validation import validate_score_value
+from ee.billing.quota_limiting import QuotaLimitingCaches, QuotaResource, is_team_limited
+
if TYPE_CHECKING:
from django.db.models import QuerySet
@@ -444,6 +447,13 @@ class OfflineEvaluationIngestionService(_OfflineEvaluationService):
new_item_submissions = [item for item in submission.items if item.id not in items]
if (new_item_submissions or new_results) and experiment.status != OfflineExperiment.Status.UPLOADING:
raise OfflineEvaluationConflict("experiment_closed", "This experiment is closed to new items and results.")
+ if new_results:
+ api_token = Team.objects.values_list("api_token", flat=True).get(id=self.team_id)
+ if is_team_limited(api_token, QuotaResource.LLM_EVENTS, QuotaLimitingCaches.QUOTA_LIMITER_CACHE_KEY):
+ raise QuotaLimitExceeded(
+ "Your organization reached its AI observability usage limit. Increase the limit in Billing settings, "
+ "or ask an org admin to do so."
+ )
errors: dict[str, str] = {}
try:
dataset_versions = self._dataset_versions(experiment, new_item_submissions)
diff --git a/products/ai_observability/backend/test/test_offline_evaluation_billing.py b/products/ai_observability/backend/test/test_offline_evaluation_billing.py
new file mode 100644
index 000000000000..abf8bdc03195
--- /dev/null
+++ b/products/ai_observability/backend/test/test_offline_evaluation_billing.py
@@ -0,0 +1,234 @@
+from collections import defaultdict
+from collections.abc import Callable
+from datetime import UTC, datetime, timedelta
+from itertools import batched
+from uuid import UUID, uuid4
+
+import time_machine
+from unittest.mock import patch
+
+from django.db import DatabaseError, connection
+from django.test import TestCase
+
+from parameterized import parameterized
+
+from posthog.models import Organization, Team
+from posthog.tasks.ai_observability_usage_report import _get_all_ai_observability_reports
+from posthog.tasks.usage_report import _get_all_org_reports, iter_reports_with_usage_counter_metadata
+from posthog.temporal.usage_report.aggregator import iter_org_reports
+from posthog.temporal.usage_report.types import WorkflowContext
+from posthog.usage_counters import UsageCounterComparisonRows, UsageCounterMode, UsageCounterReport
+from posthog.utils import DayRange
+
+from products.ai_observability.backend.models.offline_evaluations import OfflineEvaluationUsageLedger
+from products.ai_observability.backend.offline_evaluation_billing import get_offline_evaluation_usage
+
+from ee.billing.quota_limiting import _aggregate_quota_usage, _iter_quota_organizations, update_all_orgs_billing_quotas
+
+
+@time_machine.travel("2026-10-31T12:00:00Z", tick=False)
+class TestOfflineEvaluationBilling(TestCase):
+ organization: Organization
+ other_organization: Organization
+ team: Team
+ other_team: Team
+
+ @classmethod
+ def setUpTestData(cls) -> None:
+ cls.organization, _, cls.team = Organization.objects.bootstrap(None)
+ cls.other_organization, _, cls.other_team = Organization.objects.bootstrap(None)
+
+ def setUp(self) -> None:
+ self.period = DayRange(start=datetime(2026, 10, 31, tzinfo=UTC), end=datetime(2026, 11, 1, tzinfo=UTC))
+ self.ctx = WorkflowContext(
+ run_id="offline-usage-test",
+ date_str="2026-10-31",
+ period_start=self.period.start,
+ period_end=self.period.end,
+ )
+
+ def record(self, team: Team, quantity: int, *, at: datetime | None = None) -> OfflineEvaluationUsageLedger:
+ return OfflineEvaluationUsageLedger.objects.for_team(team.id).create(
+ team_id=team.id,
+ organization_id=team.organization_id,
+ experiment_id=uuid4(),
+ accepted_at=at or self.period.start,
+ quantity=quantity,
+ )
+
+ @parameterized.expand([(-1, 0), (0, 7), (86_399_999_999, 7), (86_400_000_000, 0)])
+ def test_utc_half_open_month_boundary(self, microseconds: int, expected: int) -> None:
+ self.record(self.team, 7, at=self.period.start + timedelta(microseconds=microseconds))
+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)
+ self.assertEqual(sum(usage.get(str(self.organization.id), {}).values()), expected)
+
+ @parameterized.expand([(False,), (True,)])
+ def test_sums_quantities_for_moved_teams_but_excludes_deleted_teams(self, deleted_team: bool) -> None:
+ child = Team.objects.create(organization=self.organization, parent_team=self.team)
+ self.record(child, 20)
+ self.record(child, 10)
+ self.record(self.team, 5)
+ self.record(self.other_team, 9)
+ child.organization = self.other_organization
+ child.save(update_fields=["organization"])
+ self.record(child, 4)
+ child_id = child.id
+ if deleted_team:
+ child.delete()
+ expected = {
+ str(self.organization.id): {self.team.id: 5},
+ str(self.other_organization.id): {self.other_team.id: 9},
+ }
+ if not deleted_team:
+ expected[str(self.organization.id)][child_id] = 30
+ expected[str(self.other_organization.id)][child_id] = 4
+ before = list(OfflineEvaluationUsageLedger.objects.unscoped().order_by("id").values())
+ for _ in range(2):
+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)
+ self.assertEqual(usage, expected)
+ reports = {report.organization_id: report for report in iter_org_reports(defaultdict(dict), self.ctx, {})}
+ self.assertEqual(reports[str(self.organization.id)].ai_event_count_in_period, 5 if deleted_team else 35)
+ self.assertEqual(
+ reports[str(self.other_organization.id)].ai_event_count_in_period, 9 if deleted_team else 13
+ )
+ if deleted_team:
+ self.assertNotIn(str(child_id), reports[str(self.organization.id)].teams)
+ else:
+ self.assertEqual(reports[str(self.organization.id)].teams[str(child_id)].ai_event_count_in_period, 30)
+ self.assertEqual(list(OfflineEvaluationUsageLedger.objects.unscoped().order_by("id").values()), before)
+
+ @parameterized.expand([(False,), (True,)])
+ def test_organization_without_live_teams_has_no_reported_usage(self, demo: bool) -> None:
+ self.record(self.team, 30)
+ Team.objects.filter(id=self.team.id).update(is_demo=demo)
+ self.team.delete()
+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)
+ self.assertEqual(usage, {})
+ reports = {report.organization_id: report for report in iter_org_reports(defaultdict(dict), self.ctx, {})}
+ self.assertNotIn(str(self.organization.id), reports)
+ self.assertEqual(list(_iter_quota_organizations([], usage)), [])
+ quotas = _aggregate_quota_usage([], defaultdict(dict), offline_usage=usage)
+ self.assertEqual(quotas.usage, {})
+ self.assertEqual(OfflineEvaluationUsageLedger.objects.unscoped().get().quantity, 30)
+
+ @parameterized.expand([(1,), (2,)])
+ def test_offline_quota_organizations_survive_closed_server_cursors(self, batch_size: int) -> None:
+ historical_organizations = [self.organization, Organization.objects.create(), Organization.objects.create()]
+ for organization in historical_organizations:
+ self.team.organization = organization
+ self.team.save(update_fields=["organization"])
+ self.record(self.team, 10)
+ self.team.organization = self.other_organization
+ self.team.save(update_fields=["organization"])
+ self.record(self.team, 7)
+ usage = get_offline_evaluation_usage(self.period.start, self.period.end)
+
+ organizations: list[tuple[UUID, list[int]]] = []
+ with (
+ patch.dict(connection.settings_dict, DISABLE_SERVER_SIDE_CURSORS=False),
+ patch("ee.billing.quota_limiting.QUOTA_ORGANIZATION_BATCH_SIZE", batch_size),
+ ):
+ for batch in batched(
+ _iter_quota_organizations([self.team, self.other_team], usage), batch_size, strict=False
+ ):
+ with connection.cursor() as cursor:
+ cursor.execute("CLOSE ALL")
+ organizations.extend((organization.id, [team.id for team in teams]) for organization, teams in batch)
+
+ self.assertEqual(
+ organizations,
+ [
+ (self.other_organization.id, [self.team.id, self.other_team.id]),
+ *[(organization.id, []) for organization in sorted(historical_organizations, key=lambda org: org.id)],
+ ],
+ )
+
+ def test_grouped_query_cost_does_not_grow_with_upload_count(self) -> None:
+ for _ in range(20):
+ self.record(self.team, 3)
+ with self.assertNumQueries(1):
+ self.assertEqual(
+ get_offline_evaluation_usage(self.period.start, self.period.end),
+ {str(self.organization.id): {self.team.id: 60}},
+ )
+
+ def test_historical_team_does_not_duplicate_ingested_shadow_comparisons(self) -> None:
+ self.record(self.team, 30)
+ Team.objects.filter(id=self.team.id).update(organization=self.other_organization)
+ field = "teams_with_ai_event_count_in_period"
+ counter_report = UsageCounterReport(
+ counts={field: [(self.team.id, 50)]},
+ usage_sources={field: UsageCounterMode.BOTH},
+ counter_comparisons={
+ field: UsageCounterComparisonRows(
+ legacy_by_team={self.team.id: 50},
+ realtime_by_team={self.team.id: {str(self.other_organization.id): 55}},
+ )
+ },
+ )
+ reports = {
+ report.organization_id: report
+ for report in iter_reports_with_usage_counter_metadata(
+ iter_org_reports(defaultdict(dict, {field: {self.team.id: 50}}), self.ctx, {}),
+ counter_report,
+ caller="usage_reports_v2",
+ date="2026-10-31",
+ )
+ }
+ old = reports[str(self.organization.id)]
+ new = reports[str(self.other_organization.id)]
+ self.assertEqual(old.ai_event_count_in_period, 30)
+ self.assertEqual(new.ai_event_count_in_period, 50)
+ self.assertEqual(old.counter_comparisons, {field: {"legacy": 0, "realtime": 0}})
+ self.assertEqual(new.counter_comparisons, {field: {"legacy": 50, "realtime": 55}})
+
+ def test_current_demo_and_internal_status_excludes_usage(self) -> None:
+ demo_team = Team.objects.create(organization=self.organization, is_demo=True)
+ self.record(demo_team, 30)
+ self.record(self.team, 10)
+ self.record(self.other_team, 9)
+ self.other_organization.for_internal_metrics = True
+ self.other_organization.save(update_fields=["for_internal_metrics"])
+ self.assertEqual(
+ get_offline_evaluation_usage(self.period.start, self.period.end),
+ {str(self.organization.id): {self.team.id: 10}},
+ )
+ Team.objects.filter(id=self.team.id).update(is_demo=True)
+ Team.objects.filter(id=demo_team.id).update(is_demo=False)
+ self.assertEqual(
+ get_offline_evaluation_usage(self.period.start, self.period.end),
+ {str(self.organization.id): {demo_team.id: 30}},
+ )
+
+ def test_deleted_organization_is_not_recreated(self) -> None:
+ self.record(self.team, 30)
+ self.organization.delete()
+ self.assertEqual(get_offline_evaluation_usage(self.period.start, self.period.end), {})
+ self.assertEqual(OfflineEvaluationUsageLedger.objects.unscoped().get().quantity, 30)
+
+ @parameterized.expand([("celery",), ("temporal",), ("analytics",), ("quota",)])
+ def test_ledger_failure_prevents_incomplete_reports_and_quota_updates(self, path: str) -> None:
+ self.organization.usage = {"llm_events": {"usage": 50, "todays_usage": 30, "limit": 100}}
+ self.organization.save(update_fields=["usage"])
+
+ def fail_ledger_read(
+ execute: Callable[..., object], sql: str, params: object, many: bool, context: object
+ ) -> object:
+ if sql.startswith("SELECT") and OfflineEvaluationUsageLedger._meta.db_table in sql:
+ raise DatabaseError("Ledger unavailable")
+ return execute(sql, params, many, context)
+
+ with (
+ connection.execute_wrapper(fail_ledger_read),
+ self.assertRaisesMessage(DatabaseError, "Ledger unavailable"),
+ ):
+ if path == "celery":
+ _get_all_org_reports(period=self.period)
+ elif path == "temporal":
+ list(iter_org_reports(defaultdict(dict), self.ctx, {}))
+ elif path == "analytics":
+ _get_all_ai_observability_reports(period=self.period)
+ else:
+ update_all_orgs_billing_quotas()
+ self.organization.refresh_from_db()
+ self.assertEqual(self.organization.usage["llm_events"], {"usage": 50, "todays_usage": 30, "limit": 100})
diff --git a/products/ai_observability/backend/test/test_offline_evaluation_service.py b/products/ai_observability/backend/test/test_offline_evaluation_service.py
index 1536c5b54c96..1f5277f6f600 100644
--- a/products/ai_observability/backend/test/test_offline_evaluation_service.py
+++ b/products/ai_observability/backend/test/test_offline_evaluation_service.py
@@ -194,6 +194,16 @@ class TestOfflineEvaluationService(TestCase):
self.assertEqual(new_ledger.organization_id, other_organization.id)
self.assertEqual(new_ledger.quantity, 1)
+ def test_privileged_result_deletion_removes_retry_identity_but_not_recorded_usage(self) -> None:
+ self.ingestion.upload(self.experiment.id, self.submission)
+ OfflineEvaluationResult.objects.for_team(self.team.id).all().delete()
+ replay = self.ingestion.upload(self.experiment.id, self.submission)
+ self.assertTrue(replay.results[0].created)
+ self.assertEqual(
+ list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).values_list("quantity", flat=True)),
+ [1, 1],
+ )
+
def test_child_environment_keeps_the_entire_experiment_graph_and_hosted_links(self) -> None:
child = Team.objects.create(organization=self.team.organization, parent_team=self.team)
lifecycle = OfflineExperimentService(team_id=child.id, user_access_control=None)
@@ -591,23 +601,34 @@ class TestOfflineEvaluationServiceConcurrency(TransactionTestCase):
finally:
connection.close()
- def test_concurrent_duplicate_uploads_return_the_same_result(self) -> None:
+ @parameterized.expand([(False,), (True,)])
+ def test_concurrent_duplicate_uploads_return_the_same_result(self, overlapping: bool) -> None:
+ second_submission = self.submission
+ if overlapping:
+ new_item = replace(self.submission.items[0], id=uuid4())
+ second_submission = UploadSubmission(
+ items=[new_item],
+ results=[replace(self.submission.results[0], item_id=new_item.id), self.submission.results[0]],
+ )
second_started = Event()
with ThreadPoolExecutor(max_workers=1) as executor:
with transaction.atomic():
first = self.ingestion.upload(self.experiment.id, self.submission)
pending = executor.submit(
self._run_second_operation,
- lambda: self.ingestion.upload(self.experiment.id, self.submission),
+ lambda: self.ingestion.upload(self.experiment.id, second_submission),
second_started,
)
self.assertTrue(second_started.wait(timeout=10))
second = pending.result(timeout=10)
assert isinstance(second, UploadReceipt)
- self.assertEqual(second.results, [replace(first.results[0], created=False)])
- self.assertEqual(OfflineEvaluationResult.objects.for_team(self.team.id).count(), 1)
- self.assertEqual(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).get().quantity, 1)
+ self.assertEqual(second.results[-1], replace(first.results[0], created=False))
+ self.assertEqual(OfflineEvaluationResult.objects.for_team(self.team.id).count(), 2 if overlapping else 1)
+ self.assertEqual(
+ list(OfflineEvaluationUsageLedger.objects.for_team(self.team.id).values_list("quantity", flat=True)),
+ [1, 1] if overlapping else [1],
+ )
def test_concurrent_experiment_creation_returns_one_identity(self) -> None:
submission = ExperimentSubmission(id=uuid4(), name="Retried creation", started_at=self.experiment.started_at)
diff --git a/tach.toml b/tach.toml
index ab8f165ead23..c487de8ed1f9 100644
--- a/tach.toml
+++ b/tach.toml
@@ -1745,6 +1745,13 @@ expose = [
"backend\\.llm.*",
"backend\\.markdown_outline.*",
"backend\\.models.*",
+ "backend\\.offline_evaluation_billing.get_offline_evaluation_usage",
+ "backend\\.offline_evaluation_service.OfflineEvaluationIngestionService",
+ "backend\\.offline_evaluation_service.OfflineExperimentService",
+ "backend\\.offline_evaluation_types.ExperimentSubmission",
+ "backend\\.offline_evaluation_types.ItemSubmission",
+ "backend\\.offline_evaluation_types.ResultSubmission",
+ "backend\\.offline_evaluation_types.UploadSubmission",
"backend\\.prompt_references.*",
"backend\\.queries.*",
"backend\\.summarization.*",