:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(aio): run online evaluations independently of property rules (#125555)
Radu Raicead4323c2d9017
diff --git a/.github/new-events-schema-targets.txt b/.github/new-events-schema-targets.txtindex 58ea04a7c0e9..7f544d587b9a 100644--- a/.github/new-events-schema-targets.txt+++ b/.github/new-events-schema-targets.txt@@ -63,6 +63,8 @@ posthog/tasks/test/test_ai_observability_usage_report.pyposthog/tasks/test/test_usage_report.pyposthog/temporal/tests/test_clickhouse.pyposthog/temporal/ai_observability/eval_reports/test/test_activities.py+posthog/temporal/ai_observability/test_run_trace_evaluation.py+posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.pyproducts/autoresearch/backend/inference/test_person_property.pyproducts/batch_exports/backend/tests/api/test_serializers.pyproducts/batch_exports/backend/tests/temporal/destinations/test_http_batch_export_workflow.pydiff --git a/docs/internal/ai-observability-judge-inputs.md b/docs/internal/ai-observability-judge-inputs.mdindex f8bb56f21263..617e39a89340 100644--- a/docs/internal/ai-observability-judge-inputs.md+++ b/docs/internal/ai-observability-judge-inputs.md@@ -12,6 +12,11 @@ They do not change with the selected judge model.## Trace and session formatting+Online trace and session evaluations run as project workflows. Their event reads include all+properties regardless of default, role, or member property access rules and AI observability+product access. Evaluation previews and interactive trace/session queries continue to apply+the requesting user's permissions.+Trace evaluations skip the LLM judge when a trace has no child events and no readable trace-level input or output.Message role headers, empty OpenTelemetry parts, and whitespace-only message bodies do not count as readable content.Tool calls embedded in output content count as readable content, even when the adjacent text is whitespace.@@ -274,6 +279,28 @@ Deploy all evaluation workers before enabling the flag.## Run history and reports+Online evaluations and report result queries run as project workflows with complete event properties. Evaluation editors+are trusted to configure workflows that read properties hidden from ordinary users. Grant Evaluations editor access only+to people authorized to process the full inputs. AI Observability and Evaluations have separate RBAC settings.++Source-property rules do not automatically apply to evaluation outputs. To keep message content private:++- Keep the source-property rules and separately restrict `$ai_evaluation_reasoning`.+- Restrict `$ai_clusters` when evaluation cluster titles and descriptions may contain private content.+- For report events, restrict `$ai_report_content`, `$ai_report_title`, and `$ai_report_citations`.+- If outcomes are also sensitive, restrict `$ai_evaluation_result`, `$ai_evaluation_numeric_result`, and+ `$ai_evaluation_categorical_result`. Apply separate rules to sentiment properties when using sentiment evaluations.+- Restrict Evaluations viewer access to the intended report audience. Stored report API responses follow evaluation RBAC;+ property rules on report events protect the event copy, not the stored report narrative.+- Choose report email recipients and Slack channels with the same audience in mind. Deliveries contain report text and do+ not check each recipient's PostHog property permissions.+- Evaluation-report Inbox summaries are suppressed when the project has any event-property denial or a denial of access+ to the evaluation, including default, member, and role rules. Inbox copies cannot enforce those source permissions.++User-facing event queries continue to apply the configured property rules. Background report queries can read evaluation+results even when the project's default rules hide those result properties from users.+Report agents also read complete source conversations when inspecting trace or session evaluation results.+The evaluation's Runs tab defaults to the last seven days.Its date filter applies to both the run list and summary statistics, supports custom ranges and All time, and is preserved in the URL.Opening a specific backfill shows all runs from that backfill, regardless of the date filter.diff --git a/posthog/hogql/ACCESS_CONTROL.md b/posthog/hogql/ACCESS_CONTROL.mdindex a0f93d8a0a91..d22be43db44a 100644--- a/posthog/hogql/ACCESS_CONTROL.md+++ b/posthog/hogql/ACCESS_CONTROL.md@@ -245,12 +245,18 @@ There is the asymmetry with the warehouse access control, which bypasses entirel### AI previews and summaries-AI evaluation and tagger previews read event properties with the requesting user's permissions, including properties available to Hog scripts.-Background trace and session evaluations skip with `property_access_restricted` when project-default rules deny any event property.-They skip before reading content or running a Hog or LLM judge, because grading masked data can change the verdict.-Hog scripts can read arbitrary event properties, so this check covers all event-property denials, not only input and output.-Person/group restrictions and member-only rules do not trigger this skip.-Generation evaluations keep using the original Kafka event payload.+Interactive AI reads retain the requesting user's property permissions. Online trace and session evaluations read full events+as project workflows, using an explicit property-access bypass and separate query cache keys. Generation evaluations keep+using the original Kafka event payload. Report result queries also run as project workflows.++Evaluation editors are trusted to configure project workflows with full inputs, even when their own property rules hide+some of those inputs. Evaluation RBAC controls authoring and access to stored reports; it is separate from AI Observability+RBAC. Source-property restrictions do not propagate to generated outputs. Configure separate property rules for evaluation+reasoning, report event content, and any sensitive outcomes. Event queries apply those rules to the generated properties,+while background report result queries explicitly bypass them. Stored report narratives and report delivery do not inherit+event-property rules, so configure report access, recipients, and channels for the intended audience. Inbox report summaries+are suppressed when the project has an event-property denial or a denial of access to the evaluation, including member and role rules.+See [evaluation access configuration](../../docs/internal/ai-observability-judge-inputs.md#run-history-and-reports).Summary caches use the caller's current property restrictions, including cached titles and summaries read by PostHog AI.When the caller has event-property restrictions, summaries generated from client-supplied data refetch the source with the caller's permissions.diff --git a/posthog/hogql_queries/ai/ai_table_resolver.py b/posthog/hogql_queries/ai/ai_table_resolver.pyindex 6057586bc290..bda5e7a938cc 100644--- a/posthog/hogql_queries/ai/ai_table_resolver.py+++ b/posthog/hogql_queries/ai/ai_table_resolver.py@@ -85,6 +85,7 @@ def query_ai_events(settings: HogQLGlobalSettings | None = None,workload: Workload | None = None,posthog_tables_only: bool = False,+ bypass_property_access_control: bool = False,) -> Any:"""Execute a query written against the dedicated ai_events table.@@ -107,6 +108,8 @@ def query_ai_events(Pass the requesting ``user`` for interactive reads so member and role property rulesapply on every read path. Userless background reads apply the project's default rules.+ Online evaluations opt into ``bypass_property_access_control`` to grade complete events+ as a project workflow, independently of user permissions.`workload` should be specified explicitly for batch / scheduled callers (e.g. usagereports). Inside a Celery task the `task_prerun` signal sets `Workload.OFFLINE` on@@ -130,12 +133,13 @@ def query_ai_events(kwargs["settings"] = settingsif workload is not None:kwargs["workload"] = workload- if posthog_tables_only:+ if posthog_tables_only or bypass_property_access_control:kwargs["context"] = HogQLContext(team_id=team.pk,team=team,user=user,- database=Database.create_for_posthog_tables(team, modifiers=modifiers),+ database=Database.create_for_posthog_tables(team, modifiers=modifiers) if posthog_tables_only else None,+ restricted_properties=set() if bypass_property_access_control else None,)with tags_context(product=Product.LLM_ANALYTICS):@@ -160,6 +164,7 @@ def query_ai_events(user=user,use_new_events_schema=events_schema,property_type_overrides=EVENTS_FALLBACK_PROPERTY_TYPE_OVERRIDES,+ restricted_properties=set() if bypass_property_access_control else None,database=Database.create_for_posthog_tables(team, modifiers=modifiers) if posthog_tables_only else None,),}diff --git a/posthog/hogql_queries/ai/session_query_runner.py b/posthog/hogql_queries/ai/session_query_runner.pyindex 410a6f36c14c..275ce2ebdc75 100644--- a/posthog/hogql_queries/ai/session_query_runner.py+++ b/posthog/hogql_queries/ai/session_query_runner.py@@ -65,11 +65,18 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):cached_response: CachedSessionQueryResponsepaginator: HogQLHasMorePaginator- def __init__(self, *args: Any, for_evaluation: bool = False, **kwargs: Any) -> None:+ def __init__(+ self,+ *args: Any,+ for_evaluation: bool = False,+ bypass_property_access_control: bool = False,+ **kwargs: Any,+ ) -> None:# Online evaluations need a time-bounded primary query and no events fallback: an# aged-out session must surface as "expired", not as a stripped events scan whose rows# lack input/output_choices, which would have the judge grade an empty transcript.self.for_evaluation = for_evaluation+ self._bypass_property_access_control = bypass_property_access_controlsuper().__init__(*args, **kwargs)limit = self.query.limitif self.limit_context == LimitContext.EXPORT:@@ -105,7 +112,12 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):timings=self.timings,modifiers=self.modifiers,limit_context=self.limit_context,- context=HogQLContext(team_id=self.team.pk, user=self.user, database=database),+ context=HogQLContext(+ team_id=self.team.pk,+ user=self.user,+ database=database,+ restricted_properties=set() if self._bypass_property_access_control else None,+ ),)if not query_result.results and not self.for_evaluation:@@ -133,7 +145,11 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):modifiers=self.modifiers,limit_context=self.limit_context,context=HogQLContext(- team_id=self.team.pk, user=self.user, use_new_events_schema=events_schema, database=database+ team_id=self.team.pk,+ user=self.user,+ use_new_events_schema=events_schema,+ database=database,+ restricted_properties=set() if self._bypass_property_access_control else None,),)@@ -289,6 +305,8 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):# with a plain read. Keyed only when on, so plain reads keep their cache entries.if self.for_evaluation:payload["for_evaluation"] = True+ if self._bypass_property_access_control:+ payload["bypass_property_access_control"] = Truereturn payloaddef cache_target_age(self, last_refresh: Optional[datetime], lazy: bool = False) -> Optional[datetime]:diff --git a/posthog/hogql_queries/ai/trace_query_runner.py b/posthog/hogql_queries/ai/trace_query_runner.pyindex 2090b5b79592..09e209fdea1f 100644--- a/posthog/hogql_queries/ai/trace_query_runner.py+++ b/posthog/hogql_queries/ai/trace_query_runner.py@@ -90,8 +90,13 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):cached_response: CachedTraceQueryResponsedef __init__(- self, *args: Any, bound_events_to_date_range: bool = False, posthog_tables_only: bool = False, **kwargs: Any- ):+ self,+ *args: Any,+ bound_events_to_date_range: bool = False,+ posthog_tables_only: bool = False,+ bypass_property_access_control: bool = False,+ **kwargs: Any,+ ) -> None:super().__init__(*args, **kwargs)# The trace view wants the whole trace whatever the date picker says, so the default reads# `ai_events` unbounded. A caller that grades a trace "as of" an instant opts in here, which@@ -100,6 +105,7 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):# See query_ai_events. The trace view keeps the full catalog because its property filters# and modifiers can reference warehouse tables and group types.self._posthog_tables_only = posthog_tables_only+ self._bypass_property_access_control = bypass_property_access_controldef _calculate(self):query_result = query_ai_events(@@ -114,6 +120,7 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):modifiers=self.modifiers,limit_context=self.limit_context,posthog_tables_only=self._posthog_tables_only,+ bypass_property_access_control=self._bypass_property_access_control,)columns: list[str] = query_result.columns or []@@ -264,8 +271,8 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):)return cast(ast.SelectQuery, query)- def get_cache_payload(self):- return {+ def get_cache_payload(self) -> dict[str, Any]:+ payload = {**super().get_cache_payload(),# When the response schema changes, increment this version to invalidate the cache."schema_version": 11,@@ -273,6 +280,9 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):# a bounded and an unbounded read of the same trace must not share a cache entry."bound_events_to_date_range": self._bound_events_to_date_range,}+ if self._bypass_property_access_control:+ payload["bypass_property_access_control"] = True+ return payload@cached_propertydef _date_range(self):diff --git a/posthog/temporal/ai_observability/eval_reports/emit_signal.py b/posthog/temporal/ai_observability/eval_reports/emit_signal.pyindex 37a2a77aa052..14318805a484 100644--- a/posthog/temporal/ai_observability/eval_reports/emit_signal.py+++ b/posthog/temporal/ai_observability/eval_reports/emit_signal.py@@ -21,6 +21,8 @@ from dataclasses import dataclassfrom datetime import timedeltafrom typing import Any+from django.db.models import Q+import structlogimport temporalioimport temporalio.activity@@ -28,10 +30,13 @@ import temporalio.workflowfrom pydantic import BaseModel, Fieldfrom temporalio.common import RetryPolicy+from posthog.models import PropertyDefinitionfrom posthog.models.team import Teamfrom posthog.sync import database_sync_to_asyncfrom posthog.temporal.common.base import PostHogWorkflow+from products.access_control.backend.models.access_control import AccessControl+from products.access_control.backend.models.property_access_control import PropertyAccessControlfrom products.ai_observability.backend.models.evaluation_reports import EvaluationReportRunfrom products.signals.backend.models import SignalSourceConfigfrom products.signals.backend.temporal.llm import call_llm@@ -176,6 +181,22 @@ async def emit_eval_report_signal_activity(inputs: EmitEvalReportSignalInputs) -SignalSourceConfig.SourceType.EVALUATION_REPORT,):return None+ # Inbox copies cannot enforce the source's property or evaluation permissions.+ # Include member and role denials, since Inbox readers need not share the same access.+ if (+ PropertyAccessControl.objects.filter(+ team_id=inputs.team_id,+ property_definition__type=PropertyDefinition.Type.EVENT,+ access_level="none",+ ).exists()+ or AccessControl.objects.filter(+ Q(resource_id__isnull=True) | Q(resource_id=inputs.evaluation_id),+ team_id=inputs.team_id,+ resource="evaluation",+ access_level="none",+ ).exists()+ ):+ return Nonereturn Team.objects.get(id=inputs.team_id)team = await database_sync_to_async(_is_eval_report_source_enabled, thread_sensitive=False)()diff --git a/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py b/posthog/temporal/ai_observability/eval_reports/report_agent/tools.pyindex 0fb27b6c7cf3..741c67fd5a27 100644--- a/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py+++ b/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py@@ -23,6 +23,8 @@ from langchain_core.tools import BaseTool, toolfrom langgraph.prebuilt import InjectedStatefrom posthog.hogql import ast+from posthog.hogql.context import HogQLContext+from posthog.hogql.database.database import Databasefrom posthog.dataclasses import frozenfrom posthog.errors import CH_TRANSIENT_ERRORS@@ -341,6 +343,12 @@ def _execute_hogql(team_id: int, query_str: str, placeholders: dict | None = Nonquery=query,placeholders=placeholders or {},team=team,+ context=HogQLContext(+ team_id=team.pk,+ team=team,+ database=Database.create_for_posthog_tables(team),+ restricted_properties=set(),+ ),),query_type="EvalReportAgent",)@@ -375,6 +383,8 @@ def _execute_hogql_via_ai_events(team: "Team", query_str: str, placeholders: dicteam=team,query_type="EvalReportAgent",fall_back_to_events=True,+ posthog_tables_only=True,+ bypass_property_access_control=True,),query_type="EvalReportAgent",)@@ -1294,6 +1304,7 @@ def _fetch_trace_detail(state: dict, trace_id: object, max_length: int) -> dict:max_length=max_length,max_trace_events=MAX_TRACE_EVENTS_LIMIT,max_raw_trace_size=MAX_TRACE_PROPERTIES_SIZE,+ bypass_property_access_control=True,)except Exception as error:logger.exception(diff --git a/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py b/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.pyindex 558446433650..83ec4ce0695f 100644--- a/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py+++ b/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py@@ -6,7 +6,7 @@ from unittest.mock import AsyncMock, patchimport pytest_asynciofrom asgiref.sync import sync_to_async-from posthog.models import Organization, Team+from posthog.models import Organization, PropertyDefinition, Team, Userfrom posthog.temporal.ai_observability.eval_reports.emit_signal import (EmitEvalReportSignalInputs,EvalReportSignalSummary,@@ -15,6 +15,9 @@ from posthog.temporal.ai_observability.eval_reports.emit_signal import (summarize_report_for_signal,)+from products.access_control.backend.models.access_control import AccessControl+from products.access_control.backend.models.property_access_control import PropertyAccessControl+from products.access_control.backend.models.role import Rolefrom products.signals.backend.models import SignalSourceConfig@@ -25,6 +28,7 @@ async def aorganization():is_ai_data_processing_approved=True,)yield organization+ await sync_to_async(User.objects.filter(organization_membership__organization=organization).delete)()await sync_to_async(organization.delete)()@@ -168,7 +172,63 @@ class TestEmitEvalReportSignalActivity:mock_summarize.assert_not_called()mock_emit.assert_not_called()- async def test_emits_signal_when_gates_pass(self, ateam):+ @pytest.mark.parametrize(+ "rule,subject,should_emit",+ [+ ("none", "default", True),+ ("event", "default", False),+ ("event", "member", False),+ ("event", "role", False),+ ("evaluation", "default", False),+ ("evaluation", "member", False),+ ("evaluation", "role", False),+ ("evaluation_project", "default", False),+ ("other_evaluation", "default", True),+ ("person", "default", True),+ ("event_read", "default", True),+ ],+ )+ async def test_emits_only_for_unrestricted_audiences(+ self, ateam: Team, rule: str, subject: str, should_emit: bool+ ) -> None:+ def configure_access() -> None:+ member = None+ role = None+ if subject == "member":+ user = User.objects.create_and_join(+ ateam.organization, f"reader-{ateam.pk}@example.com", "test-password"+ )+ member = user.organization_memberships.get(organization=ateam.organization)+ elif subject == "role":+ role = Role.objects.create(organization=ateam.organization, name="Report readers")+ if rule in ("event", "person", "event_read"):+ definition = PropertyDefinition.objects.create(+ team=ateam,+ name="$ai_evaluation_reasoning",+ type=PropertyDefinition.Type.PERSON if rule == "person" else PropertyDefinition.Type.EVENT,+ )+ PropertyAccessControl.objects.create(+ team=ateam,+ property_definition=definition,+ access_level="read" if rule == "event_read" else "none",+ organization_member=member,+ role=role,+ )+ elif rule != "none":+ AccessControl.objects.create(+ team=ateam,+ resource="evaluation",+ resource_id=None+ if rule == "evaluation_project"+ else "other-eval"+ if rule == "other_evaluation"+ else "eval-123",+ access_level="none",+ organization_member=member,+ role=role,+ )++ await sync_to_async(configure_access)()await sync_to_async(SignalSourceConfig.objects.create)(team=ateam,source_product=SignalSourceConfig.SourceProduct.LLM_ANALYTICS,@@ -186,7 +246,7 @@ class TestEmitEvalReportSignalActivity:"posthog.temporal.ai_observability.eval_reports.emit_signal.summarize_report_for_signal",new_callable=AsyncMock,return_value=summary,- ),+ ) as mock_summarize,patch("posthog.temporal.ai_observability.eval_reports.emit_signal.EvaluationReportRun.objects.values_list") as mock_values_list,@@ -195,6 +255,12 @@ class TestEmitEvalReportSignalActivity:mock_values_list.return_value.get.return_value = _make_content()await emit_eval_report_signal_activity(inputs)+ if not should_emit:+ mock_summarize.assert_not_called()+ mock_values_list.assert_not_called()+ mock_emit.assert_not_called()+ return+mock_emit.assert_called_once()call_kwargs = mock_emit.call_args.kwargsassert call_kwargs["source_product"] == "llm_analytics"diff --git a/posthog/temporal/ai_observability/run_session_evaluation.py b/posthog/temporal/ai_observability/run_session_evaluation.pyindex 7358c08a632c..565cb4784ab8 100644--- a/posthog/temporal/ai_observability/run_session_evaluation.py+++ b/posthog/temporal/ai_observability/run_session_evaluation.py@@ -46,10 +46,6 @@ from posthog.temporal.ai_observability.message_utils import extract_text_from_mefrom posthog.temporal.ai_observability.run_trace_evaluation import TRACE_EVENTS_LOOKBACKfrom posthog.temporal.common.utils import close_db_connections-from products.access_control.backend.facade.api import (- get_restricted_properties_with_group_type_index_for_team,- split_restricted_property_names,-)from products.ai_observability.backend.models.evaluation_configs import (EVALUATION_TEST_LOOKBACK_DAYS,MAX_SESSION_EVAL_EVENTS,@@ -74,10 +70,6 @@ if TYPE_CHECKING:_SESSION_SKIP_REASONING = {- "property_access_restricted": (- "Default project permissions hide event properties, so this session was not evaluated. "- "Review the project's property access rules before running this evaluation."- ),"session_not_found": "No session events were found within the evaluation window; evaluation skipped.","session_has_no_traces": ("This session's events have no trace id, so there was nothing to evaluate. Set $ai_trace_id "@@ -175,6 +167,7 @@ def _sum_session_payload_bytes(team=team,user=user,query_type="SessionEvaluationPayloadBytes",+ bypass_property_access_control=user is None,posthog_tables_only=True,fall_back_to_events=False,workload=Workload.OFFLINE,@@ -230,6 +223,7 @@ def _count_session_events(team=team,user=user,query_type="SessionEvaluationEventCount",+ bypass_property_access_control=user is None,posthog_tables_only=True,fall_back_to_events=False,workload=Workload.OFFLINE,@@ -260,14 +254,6 @@ def fetch_session_for_evaluation(window is chosen."""team = Team.objects.get(id=team_id)- # Hog evaluations can read any event property, so masking can change the verdict even outside input/output.- if (- user is None- and split_restricted_property_names(- get_restricted_properties_with_group_type_index_for_team(user=None, team_id=team_id)- ).event- ):- return SessionFetchOutcome(traces=None, skip_reason="property_access_restricted", event_count=0)retention_floor = window_start - timedelta(days=AI_EVENTS_RETENTION_DAYS)date_to = window_end or datetime.now(UTC)@@ -306,6 +292,7 @@ def fetch_session_for_evaluation(limit=MAX_SELECT_TRACES_LIMIT_EXPORT,),for_evaluation=True,+ bypass_property_access_control=user is None,)response = runner.calculate()if not response.results:diff --git a/posthog/temporal/ai_observability/run_trace_evaluation.py b/posthog/temporal/ai_observability/run_trace_evaluation.pyindex f645eeae5ae1..fe92da4f6bf8 100644--- a/posthog/temporal/ai_observability/run_trace_evaluation.py+++ b/posthog/temporal/ai_observability/run_trace_evaluation.py@@ -81,10 +81,6 @@ from posthog.temporal.ai_observability.team_capture import (from posthog.temporal.common.base import PostHogWorkflowfrom posthog.temporal.common.utils import close_db_connections-from products.access_control.backend.facade.api import (- get_restricted_properties_with_group_type_index_for_team,- split_restricted_property_names,-)from products.ai_observability.backend.models.evaluation_configs import (EVALUATION_TEST_LOOKBACK_DAYS,TRACE_EVAL_DEFAULT_WINDOW_SECONDS,@@ -144,10 +140,6 @@ HAVING event_count > 0_SKIP_REASONING = {- "property_access_restricted": (- "Default project permissions hide event properties, so this trace was not evaluated. "- "Review the project's property access rules before running this evaluation."- ),"trace_not_found": "No trace events were found within the evaluation window; evaluation skipped.","trace_too_large": (f"Trace exceeds {MAX_TRACE_EVAL_EVENTS} events — likely a shared or runaway trace id; evaluation skipped."@@ -224,6 +216,7 @@ def _count_trace_events(team=team,user=user,query_type="TraceEvaluationEventCount",+ bypass_property_access_control=user is None,posthog_tables_only=True,fall_back_to_events=True,)@@ -255,6 +248,7 @@ def _sum_trace_payload_bytes(team=team,user=user,query_type="TraceEvaluationPayloadBytes",+ bypass_property_access_control=user is None,posthog_tables_only=True,fall_back_to_events=False,)@@ -322,6 +316,7 @@ def _fetch_trace(# early events of the trace, which live runs do grade.bound_events_to_date_range=bound_to_date_to,posthog_tables_only=True,+ bypass_property_access_control=user is None,)response = runner.calculate()if not response.results:@@ -339,11 +334,6 @@ def fetch_trace_for_evaluation() -> TraceFetchOutcome:"""Fetch the full trace for an online evaluation run, looking back from the workflow start."""team = Team.objects.get(id=team_id)- # Hog evaluations can read any event property, so masking can change the verdict even outside input/output.- if split_restricted_property_names(- get_restricted_properties_with_group_type_index_for_team(user=None, team_id=team_id)- ).event:- return TraceFetchOutcome(trace=None, skip_reason="property_access_restricted", event_count=0)date_from = window_start - TRACE_EVENTS_LOOKBACKdate_to = window_end or datetime.now(UTC)return _fetch_trace(team, trace_id, date_from, date_to, bound_to_date_to=window_end is not None)diff --git a/posthog/temporal/ai_observability/test_run_session_evaluation.py b/posthog/temporal/ai_observability/test_run_session_evaluation.pyindex 4b7438a329db..e32bd6ec8d0b 100644--- a/posthog/temporal/ai_observability/test_run_session_evaluation.py+++ b/posthog/temporal/ai_observability/test_run_session_evaluation.py@@ -1,5 +1,4 @@import uuid-from collections.abc import Iteratorfrom datetime import UTC, datetime, timedeltaimport pytest@@ -293,14 +292,6 @@ class TestRunHogEvalOverRecentSessions:class TestFetchSessionForEvaluation:- @pytest.fixture(autouse=True)- def unrestricted_project_defaults(self) -> Iterator[None]:- with patch(- "posthog.temporal.ai_observability.run_session_evaluation.get_restricted_properties_with_group_type_index_for_team",- return_value=set(),- ):- yield-def test_queries_in_evaluation_mode_with_both_date_bounds(self):user = Mock()with (@@ -598,7 +589,7 @@ class TestExecuteSessionActivities:"session_too_large","session_payload_too_large","session_truncated",- "property_access_restricted",+ "session_has_no_traces",],)@pytest.mark.parametrize("output_type", ["boolean", "numeric"])@@ -640,7 +631,7 @@ class TestExecuteSessionActivities:)@pytest.mark.parametrize("output_type", ["boolean", "numeric"])- @pytest.mark.parametrize("skip_reason", ["session_truncated", "property_access_restricted"])+ @pytest.mark.parametrize("skip_reason", ["session_truncated", "session_has_no_traces"])def test_judge_skips_without_judging(self, output_type: str, skip_reason: str) -> None:with (patch(@@ -668,8 +659,7 @@ class TestExecuteSessionActivities:assert "score" not in resultif output_type == "numeric":assert "verdict" not in result- if skip_reason == "property_access_restricted":- assert "property access rules" in result["reasoning"]+ assert result["reasoning"]# The whole point of the truncation-as-skip choice: never grade a partial transcript.mock_call_llm_judge.assert_not_called()diff --git a/posthog/temporal/ai_observability/test_run_trace_evaluation.py b/posthog/temporal/ai_observability/test_run_trace_evaluation.pyindex 6d606e1d2546..a78b2f6f9b12 100644--- a/posthog/temporal/ai_observability/test_run_trace_evaluation.py+++ b/posthog/temporal/ai_observability/test_run_trace_evaluation.py@@ -10,25 +10,32 @@ from unittest.mock import MagicMock, patchfrom asgiref.sync import async_to_sync-from posthog.schema import LLMTrace, LLMTraceEvent+from posthog.schema import DateRange, LLMTrace, LLMTraceEvent, SessionQuery, TraceQueryfrom posthog.hogql import astfrom posthog.hogql.database.database import Database+from posthog.hogql.query import execute_hogql_queryfrom posthog.api.capture import CaptureInternalErrorfrom posthog.constants import AvailableFeature+from posthog.hogql_queries.ai.session_query_runner import SessionQueryRunner+from posthog.hogql_queries.ai.trace_query_runner import TraceQueryRunnerfrom posthog.models import Organization, PropertyDefinition, Team, Userfrom posthog.models.ai_events.test_util import bulk_create_ai_eventsfrom posthog.models.event.util import bulk_create_eventsfrom posthog.temporal.ai_observability.run_session_evaluation import fetch_session_for_evaluation+from posthog.temporal.ai_observability.trace_summarization.fetch_and_format import _fetch_and_format_tracefrom products.access_control.backend.facade.contracts import PropertyAccessLevel+from products.access_control.backend.models.access_control import AccessControlfrom products.access_control.backend.models.property_access_control import PropertyAccessControl+from products.access_control.backend.property_access_control import restriction_cache_scopefrom products.ai_observability.backend.models.evaluation_config import EvaluationConfigfrom products.ai_observability.backend.models.evaluations import Evaluationfrom products.ai_observability.backend.models.provider_keys import LLMProviderKeyfrom products.cdp.backend.validation import compile_hog+from .eval_reports.report_agent.tools import _execute_hogql, get_trace_detailfrom .evaluation_hog import execute_hog_eval_bytecodefrom .evaluation_llm_judge import BooleanEvalResultfrom .evaluation_types import EvaluationActivityResult@@ -338,59 +345,175 @@ class TestBackgroundEvaluationPropertyAccess:@pytest.mark.django_db@pytest.mark.parametrize("target", ["trace", "session"])@pytest.mark.parametrize(- "property_type,member_only,access_level,should_skip",+ "property_type,member_only,access_level",[- (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.NONE, True),- (PropertyDefinition.Type.EVENT, True, PropertyAccessLevel.NONE, False),- (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.READ, False),- (PropertyDefinition.Type.PERSON, False, PropertyAccessLevel.NONE, False),- (PropertyDefinition.Type.GROUP, False, PropertyAccessLevel.NONE, False),+ (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.NONE),+ (PropertyDefinition.Type.EVENT, True, PropertyAccessLevel.NONE),+ (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.READ),+ (PropertyDefinition.Type.PERSON, False, PropertyAccessLevel.NONE),+ (PropertyDefinition.Type.GROUP, False, PropertyAccessLevel.NONE),],)- def test_only_default_event_denials_skip_before_reading_content(+ @time_machine.travel(FROZEN_NOW, tick=False)+ def test_online_evaluations_read_properties_independently_of_user_permissions(self,setup_data: dict[str, object],target: str,property_type: int,member_only: bool,access_level: PropertyAccessLevel,- should_skip: bool,) -> None:organization = cast(Organization, setup_data["organization"])team = cast(Team, setup_data["team"])organization.available_product_features = [- {"name": AvailableFeature.PROPERTY_ACCESS_CONTROL, "key": AvailableFeature.PROPERTY_ACCESS_CONTROL}+ {"name": feature, "key": feature}+ for feature in (+ AvailableFeature.PROPERTY_ACCESS_CONTROL,+ AvailableFeature.ACCESS_CONTROL,+ AvailableFeature.ROLE_BASED_ACCESS,+ )]organization.save()- member = None- if member_only:- user = User.objects.create_and_join(organization, "[email protected]", "test-password")- member = user.organization_memberships.get(organization=organization)- definition = PropertyDefinition.objects.create(- team=team,- name="$ai_input",- type=property_type,- group_type_index=0 if property_type == PropertyDefinition.Type.GROUP else None,+ user = User.objects.create_and_join(organization, "[email protected]", "test-password")+ member = user.organization_memberships.get(organization=organization)+ AccessControl.objects.create(team=team, resource="llm_analytics", access_level="none")+ properties = {+ "$ai_input": [{"role": "user", "content": "What is 2 plus 2?"}],+ "$ai_output": "The answer is four.",+ "$ai_output_choices": [{"role": "assistant", "content": "The answer is four."}],+ "evaluation_context": "arithmetic",+ }+ for name in properties:+ definition = PropertyDefinition.objects.create(+ team=team,+ name=name,+ type=property_type,+ group_type_index=0 if property_type == PropertyDefinition.Type.GROUP else None,+ )+ PropertyAccessControl.objects.create(+ team=team,+ property_definition=definition,+ organization_member=member if member_only else None,+ access_level=access_level.value,+ )+ unit_id = str(uuid.uuid4())+ bulk_create_ai_events(+ [+ {+ "event": "$ai_generation",+ "team": team,+ "distinct_id": "test-person",+ "timestamp": FROZEN_NOW - timedelta(minutes=1),+ "properties": {"$ai_trace_id": unit_id, "$ai_session_id": unit_id, **properties},+ }+ ])- PropertyAccessControl.objects.create(- team=team,- property_definition=definition,- organization_member=member,- access_level=access_level.value,+ fetch = fetch_trace_for_evaluation if target == "trace" else fetch_session_for_evaluation+ outcome = fetch(team.id, unit_id, FROZEN_NOW)++ assert outcome.skip_reason is None+ assert outcome.event_count == 1+ if isinstance(outcome, TraceFetchOutcome):+ assert outcome.trace is not None+ trace = outcome.trace+ else:+ assert outcome.traces is not None+ trace = outcome.traces[0]+ assert len(trace.events) == 1+ for name, value in properties.items():+ assert trace.events[0].properties[name] == value++ date_range = DateRange(date_from=(FROZEN_NOW - timedelta(hours=1)).isoformat(), date_to=FROZEN_NOW.isoformat())+ report_detail = json.loads(+ get_trace_detail.invoke(+ {"state": {"team_id": team.id, "trace_id_allowlist": [unit_id]}, "trace_id": unit_id}+ ))- with patch(- f"posthog.temporal.ai_observability.run_{target}_evaluation.query_ai_events",- return_value=MagicMock(results=[]),- ) as mock_query:- fetch = fetch_trace_for_evaluation if target == "trace" else fetch_session_for_evaluation- outcome = fetch(team.id, "missing-unit", FROZEN_NOW)-- assert outcome.skip_reason == ("property_access_restricted" if should_skip else f"{target}_not_found")- assert outcome.event_count == 0- if should_skip:- mock_query.assert_not_called()+ summary = _fetch_and_format_trace(+ unit_id, team.id, (FROZEN_NOW - timedelta(hours=1)).isoformat(), FROZEN_NOW.isoformat()+ )+ assert report_detail["event_count"] == 1+ assert summary is not None and summary.text_repr is not None+ for text in ("What is 2 plus 2?", "The answer is four."):+ assert text in report_detail["text"]+ if (+ property_type == PropertyDefinition.Type.EVENT+ and access_level == PropertyAccessLevel.NONE+ and not member_only+ ):+ assert text not in summary.text_repr+ else:+ assert text in summary.text_repr++ runner = (+ TraceQueryRunner(team=team, user=user, query=TraceQuery(traceId=unit_id, dateRange=date_range))+ if target == "trace"+ else SessionQueryRunner(team=team, user=user, query=SessionQuery(sessionId=unit_id, dateRange=date_range))+ )+ system_runner = type(runner)(team=team, query=runner.query, bypass_property_access_control=True)+ default_runner = type(runner)(team=team, query=runner.query)+ assert system_runner.get_cache_key() != default_runner.get_cache_key()+ user_trace = runner.calculate().results[0]+ for name, value in properties.items():+ if property_type == PropertyDefinition.Type.EVENT and access_level == PropertyAccessLevel.NONE:+ assert name not in user_trace.events[0].properties+ else:+ assert user_trace.events[0].properties[name] == value++ bytecode = compile_hog("print(evaluation_events[1].properties.evaluation_context); return true", "destination")+ result = execute_hog_eval_bytecode(bytecode, build_trace_hog_globals(trace, unit_id, bytecode=bytecode), False)+ assert result["reasoning"] == "arithmetic"+ bulk_create_events(+ [+ {+ "event": "$ai_evaluation",+ "team": team,+ "distinct_id": "test-person",+ "timestamp": FROZEN_NOW,+ "properties": {+ "$ai_trace_id": unit_id,+ "$ai_evaluation_reasoning": result["reasoning"],+ "$ai_evaluation_result": result["verdict"],+ },+ }+ ]+ )+ query = "SELECT properties.$ai_evaluation_reasoning, properties.$ai_evaluation_result, properties FROM events WHERE event = '$ai_evaluation' AND properties.$ai_trace_id = {trace_id}"+ response = execute_hogql_query(+ query, team=team, user=user, placeholders={"trace_id": ast.Constant(value=unit_id)}+ )+ assert response.results[0][0] == "arithmetic"++ if property_type == PropertyDefinition.Type.EVENT and access_level == PropertyAccessLevel.NONE:+ for name in ("$ai_evaluation_reasoning", "$ai_evaluation_result"):+ definition = PropertyDefinition.objects.create(team=team, name=name, type=PropertyDefinition.Type.EVENT)+ PropertyAccessControl.objects.create(+ team=team,+ property_definition=definition,+ organization_member=member if member_only else None,+ access_level=PropertyAccessLevel.NONE.value,+ )++ response = execute_hogql_query(+ query, team=team, user=user, placeholders={"trace_id": ast.Constant(value=unit_id)}+ )+ reasoning, verdict, blob = response.results[0]+ assert _execute_hogql(team.id, query, {"trace_id": ast.Constant(value=unit_id)})[0][0] == "arithmetic"+ if property_type == PropertyDefinition.Type.EVENT and access_level == PropertyAccessLevel.NONE:+ assert reasoning is None+ assert verdict is None+ assert "$ai_evaluation_reasoning" not in json.loads(blob)+ assert "$ai_evaluation_result" not in json.loads(blob)else:- mock_query.assert_called_once()+ assert reasoning == "arithmetic"++ # A newly granted reader sees the existing result without rerunning the evaluation.+ PropertyAccessControl.objects.filter(team=team).update(access_level=PropertyAccessLevel.READ.value)+ with restriction_cache_scope():+ response = execute_hogql_query(+ query, team=team, user=user, placeholders={"trace_id": ast.Constant(value=unit_id)}+ )+ assert response.results[0][0] == "arithmetic"class TestFetchTraceForEvaluation:@@ -498,6 +621,17 @@ class TestFetchTraceForEvaluation:@pytest.mark.parametrize("insert_events", [bulk_create_ai_events, bulk_create_events])def test_reads_the_trace_without_building_the_full_catalog(self, setup_data, insert_events):team = setup_data["team"]+ organization = setup_data["organization"]+ organization.available_product_features = [+ {"name": AvailableFeature.PROPERTY_ACCESS_CONTROL, "key": AvailableFeature.PROPERTY_ACCESS_CONTROL}+ ]+ organization.save()+ definition = PropertyDefinition.objects.create(+ team=team, name="evaluation_context", type=PropertyDefinition.Type.EVENT+ )+ PropertyAccessControl.objects.create(+ team=team, property_definition=definition, access_level=PropertyAccessLevel.NONE.value+ )trace_id = f"trace-{uuid.uuid4()}"timestamp = FROZEN_NOW - timedelta(minutes=1)insert_events(@@ -518,6 +652,7 @@ class TestFetchTraceForEvaluation:"$ai_trace_id": trace_id,"$ai_input": [{"role": "user", "content": "What is the weather?"}],"$ai_output_choices": [{"role": "assistant", "content": "Sunny all day."}],+ "evaluation_context": "weather",},},]@@ -532,6 +667,7 @@ class TestFetchTraceForEvaluation:assert outcome.skip_reason is Noneassert outcome.trace is not Noneassert [event.event for event in outcome.trace.events] == ["$ai_generation"]+ assert outcome.trace.events[0].properties["evaluation_context"] == "weather"class TestRunHogEvalOverRecentTraces:@@ -788,7 +924,7 @@ class TestExecuteTraceLLMJudgeActivity:assert result["result_type"] == output_typemock_client_class.assert_not_called()- @pytest.mark.parametrize("skip_reason", ["trace_not_found", "property_access_restricted"])+ @pytest.mark.parametrize("skip_reason", ["trace_not_found", "trace_too_large"])def test_skips_without_llm_call(self, skip_reason: str) -> None:with patch("posthog.temporal.ai_observability.run_trace_evaluation.fetch_trace_for_evaluation",@@ -811,8 +947,7 @@ class TestExecuteTraceLLMJudgeActivity:assert result["skipped"] is Trueassert result["skip_reason"] == skip_reason- if skip_reason == "property_access_restricted":- assert "property access rules" in result["reasoning"]+ assert result["reasoning"]mock_client_class.assert_not_called()@@ -878,7 +1013,7 @@ class TestExecuteTraceHogEvalActivity:assert not result.get("skipped")@pytest.mark.asyncio- @pytest.mark.parametrize("skip_reason", ["trace_too_large", "property_access_restricted"])+ @pytest.mark.parametrize("skip_reason", ["trace_too_large", "trace_not_found"])async def test_skips_without_running_hog(self, skip_reason: str) -> None:bytecode = compile_hog("return true", "destination")evaluation = {diff --git a/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py b/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.pyindex 2dc4418b61dc..6706b09abcad 100644--- a/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py+++ b/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py@@ -44,6 +44,7 @@ def _fetch_and_format_trace(*,max_trace_events: int | None = None,max_raw_trace_size: int | None = None,+ bypass_property_access_control: bool = False,) -> FetchResult | None:"""Fetch trace data and format text representation.@@ -53,7 +54,7 @@ def _fetch_and_format_trace(raw_size_limit = max_raw_trace_size if max_raw_trace_size is not None else MAX_RAW_TRACE_SIZE# A multi-GB trace runs the worker out of memory during the load, before the checks below.- trace_size = fetch_trace_size(team, trace_id)+ trace_size = fetch_trace_size(team, trace_id, bypass_property_access_control=bypass_property_access_control)if trace_size.payload_chars > raw_size_limit or (max_trace_events is not None and trace_size.event_count > max_trace_events):@@ -68,7 +69,9 @@ def _fetch_and_format_trace()return FetchResult(text_repr=None, event_count=trace_size.event_count)- llm_trace = fetch_trace(team, trace_id, window_start, window_end)+ llm_trace = fetch_trace(+ team, trace_id, window_start, window_end, bypass_property_access_control=bypass_property_access_control+ )if llm_trace is None:return Nonediff --git a/posthog/temporal/ai_observability/trace_summarization/queries.py b/posthog/temporal/ai_observability/trace_summarization/queries.pyindex f44a161ea262..85dbb8366c56 100644--- a/posthog/temporal/ai_observability/trace_summarization/queries.py+++ b/posthog/temporal/ai_observability/trace_summarization/queries.py@@ -48,7 +48,7 @@ class TraceSize:payload_chars: int-def fetch_trace_size(team: Team, trace_id: str) -> TraceSize:+def fetch_trace_size(team: Team, trace_id: str, *, bypass_property_access_control: bool = False) -> TraceSize:"""Measure a trace in ClickHouse without loading its payload.A trace ID that a client reuses for every run grows without limit, and `fetch_trace` loads all of@@ -60,12 +60,20 @@ def fetch_trace_size(team: Team, trace_id: str) -> TraceSize:placeholders={"trace_id": ast.Constant(value=trace_id)},team=team,query_type="TraceSummarizationTraceSize",+ bypass_property_access_control=bypass_property_access_control,)event_count, payload_chars = result.results[0]return TraceSize(event_count=int(event_count or 0), payload_chars=int(payload_chars or 0))-def fetch_trace(team: Team, trace_id: str, window_start: str, window_end: str) -> LLMTrace | None:+def fetch_trace(+ team: Team,+ trace_id: str,+ window_start: str,+ window_end: str,+ *,+ bypass_property_access_control: bool = False,+) -> LLMTrace | None:"""Fetch a single trace by ID via the migrated `TraceQueryRunner`.The runner reads `ai_events` with no timestamp bounds, so this loads the whole trace whatever@@ -81,7 +89,9 @@ def fetch_trace(team: Team, trace_id: str, window_start: str, window_end: str) -dateRange=DateRange(date_from=window_start, date_to=window_end),)with tags_context(product=Product.LLM_ANALYTICS, feature=Feature.QUERY, team_id=team.id):- runner = TraceQueryRunner(team=team, query=trace_query)+ runner = TraceQueryRunner(+ team=team, query=trace_query, bypass_property_access_control=bypass_property_access_control+ )response = runner.calculate()if not response.results:diff --git a/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py b/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.pyindex a836b2a14657..1bff55b5e9ce 100644--- a/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py+++ b/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py@@ -10,6 +10,8 @@ from unittest.mock import MagicMock, patchfrom parameterized import parameterized+from posthog.constants import AvailableFeature+from posthog.models import PropertyDefinitionfrom posthog.models.ai_events.test_util import bulk_create_ai_eventsfrom posthog.temporal.ai_observability.trace_summarization.fetch_and_format import (_fetch_and_format_trace,@@ -19,6 +21,8 @@ from posthog.temporal.ai_observability.trace_summarization.fetch_and_format impofrom posthog.temporal.ai_observability.trace_summarization.models import FetchAndFormatInputfrom posthog.temporal.ai_observability.trace_summarization.queries import TraceSize+from products.access_control.backend.models.property_access_control import PropertyAccessControl+@asynccontextmanagerasync def _noop_heartbeater(*args, **kwargs):@@ -141,10 +145,11 @@ class TestFetchAndFormatTrace:class TestFetchAndFormatTraceSizePreflight(ClickhouseTestMixin, BaseTest):@parameterized.expand([- ("heavy_content_outside_window", 5_000, None, True),- ("small_trace", 10, None, False),- ("root_and_duplicate_rows_not_counted", 10, 2, False),- ("over_event_limit", 10, 1, True),+ ("heavy_content_outside_window", 5_000, None, True, False),+ ("restricted_heavy_content_outside_window", 5_000, None, True, True),+ ("small_trace", 10, None, False, False),+ ("root_and_duplicate_rows_not_counted", 10, 2, False, False),+ ("over_event_limit", 10, 1, True, False),])@patch("posthog.temporal.ai_observability.trace_summarization.fetch_and_format.fetch_trace", return_value=None)@@ -154,8 +159,18 @@ class TestFetchAndFormatTraceSizePreflight(ClickhouseTestMixin, BaseTest):old_input_length: int,max_trace_events: int | None,expect_skip: bool,+ restricted_input: bool,mock_fetch: MagicMock,) -> None:+ if restricted_input:+ self.organization.available_product_features = [+ {"name": AvailableFeature.PROPERTY_ACCESS_CONTROL, "key": AvailableFeature.PROPERTY_ACCESS_CONTROL}+ ]+ self.organization.save()+ definition = PropertyDefinition.objects.create(+ team=self.team, name="$ai_input", type=PropertyDefinition.Type.EVENT+ )+ PropertyAccessControl.objects.create(team=self.team, property_definition=definition, access_level="none")trace_id = str(uuid.uuid4())window_end = datetime.now(UTC)window_start = window_end - timedelta(hours=1)@@ -186,6 +201,7 @@ class TestFetchAndFormatTraceSizePreflight(ClickhouseTestMixin, BaseTest):window_end=window_end.isoformat(),max_raw_trace_size=1_000,max_trace_events=max_trace_events,+ bypass_property_access_control=restricted_input,)if expect_skip: