Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(aio): run online evaluations independently of property rules (#125555)

Radu Raicead4323c2d9017
diff --git a/.github/new-events-schema-targets.txt b/.github/new-events-schema-targets.txt
index 58ea04a7c0e9..7f544d587b9a 100644
--- a/.github/new-events-schema-targets.txt
+++ b/.github/new-events-schema-targets.txt
@@ -63,6 +63,8 @@ posthog/tasks/test/test_ai_observability_usage_report.py
posthog/tasks/test/test_usage_report.py
posthog/temporal/tests/test_clickhouse.py
posthog/temporal/ai_observability/eval_reports/test/test_activities.py
+posthog/temporal/ai_observability/test_run_trace_evaluation.py
+posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py
products/autoresearch/backend/inference/test_person_property.py
products/batch_exports/backend/tests/api/test_serializers.py
products/batch_exports/backend/tests/temporal/destinations/test_http_batch_export_workflow.py
diff --git a/docs/internal/ai-observability-judge-inputs.md b/docs/internal/ai-observability-judge-inputs.md
index f8bb56f21263..617e39a89340 100644
--- a/docs/internal/ai-observability-judge-inputs.md
+++ b/docs/internal/ai-observability-judge-inputs.md
@@ -12,6 +12,11 @@ They do not change with the selected judge model.
## Trace and session formatting
+Online trace and session evaluations run as project workflows. Their event reads include all
+properties regardless of default, role, or member property access rules and AI observability
+product access. Evaluation previews and interactive trace/session queries continue to apply
+the requesting user's permissions.
+
Trace evaluations skip the LLM judge when a trace has no child events and no readable trace-level input or output.
Message role headers, empty OpenTelemetry parts, and whitespace-only message bodies do not count as readable content.
Tool calls embedded in output content count as readable content, even when the adjacent text is whitespace.
@@ -274,6 +279,28 @@ Deploy all evaluation workers before enabling the flag.
## Run history and reports
+Online evaluations and report result queries run as project workflows with complete event properties. Evaluation editors
+are trusted to configure workflows that read properties hidden from ordinary users. Grant Evaluations editor access only
+to people authorized to process the full inputs. AI Observability and Evaluations have separate RBAC settings.
+
+Source-property rules do not automatically apply to evaluation outputs. To keep message content private:
+
+- Keep the source-property rules and separately restrict `$ai_evaluation_reasoning`.
+- Restrict `$ai_clusters` when evaluation cluster titles and descriptions may contain private content.
+- For report events, restrict `$ai_report_content`, `$ai_report_title`, and `$ai_report_citations`.
+- If outcomes are also sensitive, restrict `$ai_evaluation_result`, `$ai_evaluation_numeric_result`, and
+ `$ai_evaluation_categorical_result`. Apply separate rules to sentiment properties when using sentiment evaluations.
+- Restrict Evaluations viewer access to the intended report audience. Stored report API responses follow evaluation RBAC;
+ property rules on report events protect the event copy, not the stored report narrative.
+- Choose report email recipients and Slack channels with the same audience in mind. Deliveries contain report text and do
+ not check each recipient's PostHog property permissions.
+- Evaluation-report Inbox summaries are suppressed when the project has any event-property denial or a denial of access
+ to the evaluation, including default, member, and role rules. Inbox copies cannot enforce those source permissions.
+
+User-facing event queries continue to apply the configured property rules. Background report queries can read evaluation
+results even when the project's default rules hide those result properties from users.
+Report agents also read complete source conversations when inspecting trace or session evaluation results.
+
The evaluation's Runs tab defaults to the last seven days.
Its date filter applies to both the run list and summary statistics, supports custom ranges and All time, and is preserved in the URL.
Opening a specific backfill shows all runs from that backfill, regardless of the date filter.
diff --git a/posthog/hogql/ACCESS_CONTROL.md b/posthog/hogql/ACCESS_CONTROL.md
index a0f93d8a0a91..d22be43db44a 100644
--- a/posthog/hogql/ACCESS_CONTROL.md
+++ b/posthog/hogql/ACCESS_CONTROL.md
@@ -245,12 +245,18 @@ There is the asymmetry with the warehouse access control, which bypasses entirel
### AI previews and summaries
-AI evaluation and tagger previews read event properties with the requesting user's permissions, including properties available to Hog scripts.
-Background trace and session evaluations skip with `property_access_restricted` when project-default rules deny any event property.
-They skip before reading content or running a Hog or LLM judge, because grading masked data can change the verdict.
-Hog scripts can read arbitrary event properties, so this check covers all event-property denials, not only input and output.
-Person/group restrictions and member-only rules do not trigger this skip.
-Generation evaluations keep using the original Kafka event payload.
+Interactive AI reads retain the requesting user's property permissions. Online trace and session evaluations read full events
+as project workflows, using an explicit property-access bypass and separate query cache keys. Generation evaluations keep
+using the original Kafka event payload. Report result queries also run as project workflows.
+
+Evaluation editors are trusted to configure project workflows with full inputs, even when their own property rules hide
+some of those inputs. Evaluation RBAC controls authoring and access to stored reports; it is separate from AI Observability
+RBAC. Source-property restrictions do not propagate to generated outputs. Configure separate property rules for evaluation
+reasoning, report event content, and any sensitive outcomes. Event queries apply those rules to the generated properties,
+while background report result queries explicitly bypass them. Stored report narratives and report delivery do not inherit
+event-property rules, so configure report access, recipients, and channels for the intended audience. Inbox report summaries
+are suppressed when the project has an event-property denial or a denial of access to the evaluation, including member and role rules.
+See [evaluation access configuration](../../docs/internal/ai-observability-judge-inputs.md#run-history-and-reports).
Summary caches use the caller's current property restrictions, including cached titles and summaries read by PostHog AI.
When the caller has event-property restrictions, summaries generated from client-supplied data refetch the source with the caller's permissions.
diff --git a/posthog/hogql_queries/ai/ai_table_resolver.py b/posthog/hogql_queries/ai/ai_table_resolver.py
index 6057586bc290..bda5e7a938cc 100644
--- a/posthog/hogql_queries/ai/ai_table_resolver.py
+++ b/posthog/hogql_queries/ai/ai_table_resolver.py
@@ -85,6 +85,7 @@ def query_ai_events(
settings: HogQLGlobalSettings | None = None,
workload: Workload | None = None,
posthog_tables_only: bool = False,
+ bypass_property_access_control: bool = False,
) -> Any:
"""Execute a query written against the dedicated ai_events table.
@@ -107,6 +108,8 @@ def query_ai_events(
Pass the requesting ``user`` for interactive reads so member and role property rules
apply on every read path. Userless background reads apply the project's default rules.
+ Online evaluations opt into ``bypass_property_access_control`` to grade complete events
+ as a project workflow, independently of user permissions.
`workload` should be specified explicitly for batch / scheduled callers (e.g. usage
reports). Inside a Celery task the `task_prerun` signal sets `Workload.OFFLINE` on
@@ -130,12 +133,13 @@ def query_ai_events(
kwargs["settings"] = settings
if workload is not None:
kwargs["workload"] = workload
- if posthog_tables_only:
+ if posthog_tables_only or bypass_property_access_control:
kwargs["context"] = HogQLContext(
team_id=team.pk,
team=team,
user=user,
- database=Database.create_for_posthog_tables(team, modifiers=modifiers),
+ database=Database.create_for_posthog_tables(team, modifiers=modifiers) if posthog_tables_only else None,
+ restricted_properties=set() if bypass_property_access_control else None,
)
with tags_context(product=Product.LLM_ANALYTICS):
@@ -160,6 +164,7 @@ def query_ai_events(
user=user,
use_new_events_schema=events_schema,
property_type_overrides=EVENTS_FALLBACK_PROPERTY_TYPE_OVERRIDES,
+ restricted_properties=set() if bypass_property_access_control else None,
database=Database.create_for_posthog_tables(team, modifiers=modifiers) if posthog_tables_only else None,
),
}
diff --git a/posthog/hogql_queries/ai/session_query_runner.py b/posthog/hogql_queries/ai/session_query_runner.py
index 410a6f36c14c..275ce2ebdc75 100644
--- a/posthog/hogql_queries/ai/session_query_runner.py
+++ b/posthog/hogql_queries/ai/session_query_runner.py
@@ -65,11 +65,18 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):
cached_response: CachedSessionQueryResponse
paginator: HogQLHasMorePaginator
- def __init__(self, *args: Any, for_evaluation: bool = False, **kwargs: Any) -> None:
+ def __init__(
+ self,
+ *args: Any,
+ for_evaluation: bool = False,
+ bypass_property_access_control: bool = False,
+ **kwargs: Any,
+ ) -> None:
# Online evaluations need a time-bounded primary query and no events fallback: an
# aged-out session must surface as "expired", not as a stripped events scan whose rows
# lack input/output_choices, which would have the judge grade an empty transcript.
self.for_evaluation = for_evaluation
+ self._bypass_property_access_control = bypass_property_access_control
super().__init__(*args, **kwargs)
limit = self.query.limit
if self.limit_context == LimitContext.EXPORT:
@@ -105,7 +112,12 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):
timings=self.timings,
modifiers=self.modifiers,
limit_context=self.limit_context,
- context=HogQLContext(team_id=self.team.pk, user=self.user, database=database),
+ context=HogQLContext(
+ team_id=self.team.pk,
+ user=self.user,
+ database=database,
+ restricted_properties=set() if self._bypass_property_access_control else None,
+ ),
)
if not query_result.results and not self.for_evaluation:
@@ -133,7 +145,11 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):
modifiers=self.modifiers,
limit_context=self.limit_context,
context=HogQLContext(
- team_id=self.team.pk, user=self.user, use_new_events_schema=events_schema, database=database
+ team_id=self.team.pk,
+ user=self.user,
+ use_new_events_schema=events_schema,
+ database=database,
+ restricted_properties=set() if self._bypass_property_access_control else None,
),
)
@@ -289,6 +305,8 @@ class SessionQueryRunner(AnalyticsQueryRunner[SessionQueryResponse]):
# with a plain read. Keyed only when on, so plain reads keep their cache entries.
if self.for_evaluation:
payload["for_evaluation"] = True
+ if self._bypass_property_access_control:
+ payload["bypass_property_access_control"] = True
return payload
def cache_target_age(self, last_refresh: Optional[datetime], lazy: bool = False) -> Optional[datetime]:
diff --git a/posthog/hogql_queries/ai/trace_query_runner.py b/posthog/hogql_queries/ai/trace_query_runner.py
index 2090b5b79592..09e209fdea1f 100644
--- a/posthog/hogql_queries/ai/trace_query_runner.py
+++ b/posthog/hogql_queries/ai/trace_query_runner.py
@@ -90,8 +90,13 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):
cached_response: CachedTraceQueryResponse
def __init__(
- self, *args: Any, bound_events_to_date_range: bool = False, posthog_tables_only: bool = False, **kwargs: Any
- ):
+ self,
+ *args: Any,
+ bound_events_to_date_range: bool = False,
+ posthog_tables_only: bool = False,
+ bypass_property_access_control: bool = False,
+ **kwargs: Any,
+ ) -> None:
super().__init__(*args, **kwargs)
# The trace view wants the whole trace whatever the date picker says, so the default reads
# `ai_events` unbounded. A caller that grades a trace "as of" an instant opts in here, which
@@ -100,6 +105,7 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):
# See query_ai_events. The trace view keeps the full catalog because its property filters
# and modifiers can reference warehouse tables and group types.
self._posthog_tables_only = posthog_tables_only
+ self._bypass_property_access_control = bypass_property_access_control
def _calculate(self):
query_result = query_ai_events(
@@ -114,6 +120,7 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):
modifiers=self.modifiers,
limit_context=self.limit_context,
posthog_tables_only=self._posthog_tables_only,
+ bypass_property_access_control=self._bypass_property_access_control,
)
columns: list[str] = query_result.columns or []
@@ -264,8 +271,8 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):
)
return cast(ast.SelectQuery, query)
- def get_cache_payload(self):
- return {
+ def get_cache_payload(self) -> dict[str, Any]:
+ payload = {
**super().get_cache_payload(),
# When the response schema changes, increment this version to invalidate the cache.
"schema_version": 11,
@@ -273,6 +280,9 @@ class TraceQueryRunner(AnalyticsQueryRunner[TraceQueryResponse]):
# a bounded and an unbounded read of the same trace must not share a cache entry.
"bound_events_to_date_range": self._bound_events_to_date_range,
}
+ if self._bypass_property_access_control:
+ payload["bypass_property_access_control"] = True
+ return payload
@cached_property
def _date_range(self):
diff --git a/posthog/temporal/ai_observability/eval_reports/emit_signal.py b/posthog/temporal/ai_observability/eval_reports/emit_signal.py
index 37a2a77aa052..14318805a484 100644
--- a/posthog/temporal/ai_observability/eval_reports/emit_signal.py
+++ b/posthog/temporal/ai_observability/eval_reports/emit_signal.py
@@ -21,6 +21,8 @@ from dataclasses import dataclass
from datetime import timedelta
from typing import Any
+from django.db.models import Q
+
import structlog
import temporalio
import temporalio.activity
@@ -28,10 +30,13 @@ import temporalio.workflow
from pydantic import BaseModel, Field
from temporalio.common import RetryPolicy
+from posthog.models import PropertyDefinition
from posthog.models.team import Team
from posthog.sync import database_sync_to_async
from posthog.temporal.common.base import PostHogWorkflow
+from products.access_control.backend.models.access_control import AccessControl
+from products.access_control.backend.models.property_access_control import PropertyAccessControl
from products.ai_observability.backend.models.evaluation_reports import EvaluationReportRun
from products.signals.backend.models import SignalSourceConfig
from products.signals.backend.temporal.llm import call_llm
@@ -176,6 +181,22 @@ async def emit_eval_report_signal_activity(inputs: EmitEvalReportSignalInputs) -
SignalSourceConfig.SourceType.EVALUATION_REPORT,
):
return None
+ # Inbox copies cannot enforce the source's property or evaluation permissions.
+ # Include member and role denials, since Inbox readers need not share the same access.
+ if (
+ PropertyAccessControl.objects.filter(
+ team_id=inputs.team_id,
+ property_definition__type=PropertyDefinition.Type.EVENT,
+ access_level="none",
+ ).exists()
+ or AccessControl.objects.filter(
+ Q(resource_id__isnull=True) | Q(resource_id=inputs.evaluation_id),
+ team_id=inputs.team_id,
+ resource="evaluation",
+ access_level="none",
+ ).exists()
+ ):
+ return None
return Team.objects.get(id=inputs.team_id)
team = await database_sync_to_async(_is_eval_report_source_enabled, thread_sensitive=False)()
diff --git a/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py b/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py
index 0fb27b6c7cf3..741c67fd5a27 100644
--- a/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py
+++ b/posthog/temporal/ai_observability/eval_reports/report_agent/tools.py
@@ -23,6 +23,8 @@ from langchain_core.tools import BaseTool, tool
from langgraph.prebuilt import InjectedState
from posthog.hogql import ast
+from posthog.hogql.context import HogQLContext
+from posthog.hogql.database.database import Database
from posthog.dataclasses import frozen
from posthog.errors import CH_TRANSIENT_ERRORS
@@ -341,6 +343,12 @@ def _execute_hogql(team_id: int, query_str: str, placeholders: dict | None = Non
query=query,
placeholders=placeholders or {},
team=team,
+ context=HogQLContext(
+ team_id=team.pk,
+ team=team,
+ database=Database.create_for_posthog_tables(team),
+ restricted_properties=set(),
+ ),
),
query_type="EvalReportAgent",
)
@@ -375,6 +383,8 @@ def _execute_hogql_via_ai_events(team: "Team", query_str: str, placeholders: dic
team=team,
query_type="EvalReportAgent",
fall_back_to_events=True,
+ posthog_tables_only=True,
+ bypass_property_access_control=True,
),
query_type="EvalReportAgent",
)
@@ -1294,6 +1304,7 @@ def _fetch_trace_detail(state: dict, trace_id: object, max_length: int) -> dict:
max_length=max_length,
max_trace_events=MAX_TRACE_EVENTS_LIMIT,
max_raw_trace_size=MAX_TRACE_PROPERTIES_SIZE,
+ bypass_property_access_control=True,
)
except Exception as error:
logger.exception(
diff --git a/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py b/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py
index 558446433650..83ec4ce0695f 100644
--- a/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py
+++ b/posthog/temporal/ai_observability/eval_reports/test/test_emit_signal.py
@@ -6,7 +6,7 @@ from unittest.mock import AsyncMock, patch
import pytest_asyncio
from asgiref.sync import sync_to_async
-from posthog.models import Organization, Team
+from posthog.models import Organization, PropertyDefinition, Team, User
from posthog.temporal.ai_observability.eval_reports.emit_signal import (
EmitEvalReportSignalInputs,
EvalReportSignalSummary,
@@ -15,6 +15,9 @@ from posthog.temporal.ai_observability.eval_reports.emit_signal import (
summarize_report_for_signal,
)
+from products.access_control.backend.models.access_control import AccessControl
+from products.access_control.backend.models.property_access_control import PropertyAccessControl
+from products.access_control.backend.models.role import Role
from products.signals.backend.models import SignalSourceConfig
@@ -25,6 +28,7 @@ async def aorganization():
is_ai_data_processing_approved=True,
)
yield organization
+ await sync_to_async(User.objects.filter(organization_membership__organization=organization).delete)()
await sync_to_async(organization.delete)()
@@ -168,7 +172,63 @@ class TestEmitEvalReportSignalActivity:
mock_summarize.assert_not_called()
mock_emit.assert_not_called()
- async def test_emits_signal_when_gates_pass(self, ateam):
+ @pytest.mark.parametrize(
+ "rule,subject,should_emit",
+ [
+ ("none", "default", True),
+ ("event", "default", False),
+ ("event", "member", False),
+ ("event", "role", False),
+ ("evaluation", "default", False),
+ ("evaluation", "member", False),
+ ("evaluation", "role", False),
+ ("evaluation_project", "default", False),
+ ("other_evaluation", "default", True),
+ ("person", "default", True),
+ ("event_read", "default", True),
+ ],
+ )
+ async def test_emits_only_for_unrestricted_audiences(
+ self, ateam: Team, rule: str, subject: str, should_emit: bool
+ ) -> None:
+ def configure_access() -> None:
+ member = None
+ role = None
+ if subject == "member":
+ user = User.objects.create_and_join(
+ ateam.organization, f"reader-{ateam.pk}@example.com", "test-password"
+ )
+ member = user.organization_memberships.get(organization=ateam.organization)
+ elif subject == "role":
+ role = Role.objects.create(organization=ateam.organization, name="Report readers")
+ if rule in ("event", "person", "event_read"):
+ definition = PropertyDefinition.objects.create(
+ team=ateam,
+ name="$ai_evaluation_reasoning",
+ type=PropertyDefinition.Type.PERSON if rule == "person" else PropertyDefinition.Type.EVENT,
+ )
+ PropertyAccessControl.objects.create(
+ team=ateam,
+ property_definition=definition,
+ access_level="read" if rule == "event_read" else "none",
+ organization_member=member,
+ role=role,
+ )
+ elif rule != "none":
+ AccessControl.objects.create(
+ team=ateam,
+ resource="evaluation",
+ resource_id=None
+ if rule == "evaluation_project"
+ else "other-eval"
+ if rule == "other_evaluation"
+ else "eval-123",
+ access_level="none",
+ organization_member=member,
+ role=role,
+ )
+
+ await sync_to_async(configure_access)()
await sync_to_async(SignalSourceConfig.objects.create)(
team=ateam,
source_product=SignalSourceConfig.SourceProduct.LLM_ANALYTICS,
@@ -186,7 +246,7 @@ class TestEmitEvalReportSignalActivity:
"posthog.temporal.ai_observability.eval_reports.emit_signal.summarize_report_for_signal",
new_callable=AsyncMock,
return_value=summary,
- ),
+ ) as mock_summarize,
patch(
"posthog.temporal.ai_observability.eval_reports.emit_signal.EvaluationReportRun.objects.values_list"
) as mock_values_list,
@@ -195,6 +255,12 @@ class TestEmitEvalReportSignalActivity:
mock_values_list.return_value.get.return_value = _make_content()
await emit_eval_report_signal_activity(inputs)
+ if not should_emit:
+ mock_summarize.assert_not_called()
+ mock_values_list.assert_not_called()
+ mock_emit.assert_not_called()
+ return
+
mock_emit.assert_called_once()
call_kwargs = mock_emit.call_args.kwargs
assert call_kwargs["source_product"] == "llm_analytics"
diff --git a/posthog/temporal/ai_observability/run_session_evaluation.py b/posthog/temporal/ai_observability/run_session_evaluation.py
index 7358c08a632c..565cb4784ab8 100644
--- a/posthog/temporal/ai_observability/run_session_evaluation.py
+++ b/posthog/temporal/ai_observability/run_session_evaluation.py
@@ -46,10 +46,6 @@ from posthog.temporal.ai_observability.message_utils import extract_text_from_me
from posthog.temporal.ai_observability.run_trace_evaluation import TRACE_EVENTS_LOOKBACK
from posthog.temporal.common.utils import close_db_connections
-from products.access_control.backend.facade.api import (
- get_restricted_properties_with_group_type_index_for_team,
- split_restricted_property_names,
-)
from products.ai_observability.backend.models.evaluation_configs import (
EVALUATION_TEST_LOOKBACK_DAYS,
MAX_SESSION_EVAL_EVENTS,
@@ -74,10 +70,6 @@ if TYPE_CHECKING:
_SESSION_SKIP_REASONING = {
- "property_access_restricted": (
- "Default project permissions hide event properties, so this session was not evaluated. "
- "Review the project's property access rules before running this evaluation."
- ),
"session_not_found": "No session events were found within the evaluation window; evaluation skipped.",
"session_has_no_traces": (
"This session's events have no trace id, so there was nothing to evaluate. Set $ai_trace_id "
@@ -175,6 +167,7 @@ def _sum_session_payload_bytes(
team=team,
user=user,
query_type="SessionEvaluationPayloadBytes",
+ bypass_property_access_control=user is None,
posthog_tables_only=True,
fall_back_to_events=False,
workload=Workload.OFFLINE,
@@ -230,6 +223,7 @@ def _count_session_events(
team=team,
user=user,
query_type="SessionEvaluationEventCount",
+ bypass_property_access_control=user is None,
posthog_tables_only=True,
fall_back_to_events=False,
workload=Workload.OFFLINE,
@@ -260,14 +254,6 @@ def fetch_session_for_evaluation(
window is chosen.
"""
team = Team.objects.get(id=team_id)
- # Hog evaluations can read any event property, so masking can change the verdict even outside input/output.
- if (
- user is None
- and split_restricted_property_names(
- get_restricted_properties_with_group_type_index_for_team(user=None, team_id=team_id)
- ).event
- ):
- return SessionFetchOutcome(traces=None, skip_reason="property_access_restricted", event_count=0)
retention_floor = window_start - timedelta(days=AI_EVENTS_RETENTION_DAYS)
date_to = window_end or datetime.now(UTC)
@@ -306,6 +292,7 @@ def fetch_session_for_evaluation(
limit=MAX_SELECT_TRACES_LIMIT_EXPORT,
),
for_evaluation=True,
+ bypass_property_access_control=user is None,
)
response = runner.calculate()
if not response.results:
diff --git a/posthog/temporal/ai_observability/run_trace_evaluation.py b/posthog/temporal/ai_observability/run_trace_evaluation.py
index f645eeae5ae1..fe92da4f6bf8 100644
--- a/posthog/temporal/ai_observability/run_trace_evaluation.py
+++ b/posthog/temporal/ai_observability/run_trace_evaluation.py
@@ -81,10 +81,6 @@ from posthog.temporal.ai_observability.team_capture import (
from posthog.temporal.common.base import PostHogWorkflow
from posthog.temporal.common.utils import close_db_connections
-from products.access_control.backend.facade.api import (
- get_restricted_properties_with_group_type_index_for_team,
- split_restricted_property_names,
-)
from products.ai_observability.backend.models.evaluation_configs import (
EVALUATION_TEST_LOOKBACK_DAYS,
TRACE_EVAL_DEFAULT_WINDOW_SECONDS,
@@ -144,10 +140,6 @@ HAVING event_count > 0
_SKIP_REASONING = {
- "property_access_restricted": (
- "Default project permissions hide event properties, so this trace was not evaluated. "
- "Review the project's property access rules before running this evaluation."
- ),
"trace_not_found": "No trace events were found within the evaluation window; evaluation skipped.",
"trace_too_large": (
f"Trace exceeds {MAX_TRACE_EVAL_EVENTS} events — likely a shared or runaway trace id; evaluation skipped."
@@ -224,6 +216,7 @@ def _count_trace_events(
team=team,
user=user,
query_type="TraceEvaluationEventCount",
+ bypass_property_access_control=user is None,
posthog_tables_only=True,
fall_back_to_events=True,
)
@@ -255,6 +248,7 @@ def _sum_trace_payload_bytes(
team=team,
user=user,
query_type="TraceEvaluationPayloadBytes",
+ bypass_property_access_control=user is None,
posthog_tables_only=True,
fall_back_to_events=False,
)
@@ -322,6 +316,7 @@ def _fetch_trace(
# early events of the trace, which live runs do grade.
bound_events_to_date_range=bound_to_date_to,
posthog_tables_only=True,
+ bypass_property_access_control=user is None,
)
response = runner.calculate()
if not response.results:
@@ -339,11 +334,6 @@ def fetch_trace_for_evaluation(
) -> TraceFetchOutcome:
"""Fetch the full trace for an online evaluation run, looking back from the workflow start."""
team = Team.objects.get(id=team_id)
- # Hog evaluations can read any event property, so masking can change the verdict even outside input/output.
- if split_restricted_property_names(
- get_restricted_properties_with_group_type_index_for_team(user=None, team_id=team_id)
- ).event:
- return TraceFetchOutcome(trace=None, skip_reason="property_access_restricted", event_count=0)
date_from = window_start - TRACE_EVENTS_LOOKBACK
date_to = window_end or datetime.now(UTC)
return _fetch_trace(team, trace_id, date_from, date_to, bound_to_date_to=window_end is not None)
diff --git a/posthog/temporal/ai_observability/test_run_session_evaluation.py b/posthog/temporal/ai_observability/test_run_session_evaluation.py
index 4b7438a329db..e32bd6ec8d0b 100644
--- a/posthog/temporal/ai_observability/test_run_session_evaluation.py
+++ b/posthog/temporal/ai_observability/test_run_session_evaluation.py
@@ -1,5 +1,4 @@
import uuid
-from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
import pytest
@@ -293,14 +292,6 @@ class TestRunHogEvalOverRecentSessions:
class TestFetchSessionForEvaluation:
- @pytest.fixture(autouse=True)
- def unrestricted_project_defaults(self) -> Iterator[None]:
- with patch(
- "posthog.temporal.ai_observability.run_session_evaluation.get_restricted_properties_with_group_type_index_for_team",
- return_value=set(),
- ):
- yield
-
def test_queries_in_evaluation_mode_with_both_date_bounds(self):
user = Mock()
with (
@@ -598,7 +589,7 @@ class TestExecuteSessionActivities:
"session_too_large",
"session_payload_too_large",
"session_truncated",
- "property_access_restricted",
+ "session_has_no_traces",
],
)
@pytest.mark.parametrize("output_type", ["boolean", "numeric"])
@@ -640,7 +631,7 @@ class TestExecuteSessionActivities:
)
@pytest.mark.parametrize("output_type", ["boolean", "numeric"])
- @pytest.mark.parametrize("skip_reason", ["session_truncated", "property_access_restricted"])
+ @pytest.mark.parametrize("skip_reason", ["session_truncated", "session_has_no_traces"])
def test_judge_skips_without_judging(self, output_type: str, skip_reason: str) -> None:
with (
patch(
@@ -668,8 +659,7 @@ class TestExecuteSessionActivities:
assert "score" not in result
if output_type == "numeric":
assert "verdict" not in result
- if skip_reason == "property_access_restricted":
- assert "property access rules" in result["reasoning"]
+ assert result["reasoning"]
# The whole point of the truncation-as-skip choice: never grade a partial transcript.
mock_call_llm_judge.assert_not_called()
diff --git a/posthog/temporal/ai_observability/test_run_trace_evaluation.py b/posthog/temporal/ai_observability/test_run_trace_evaluation.py
index 6d606e1d2546..a78b2f6f9b12 100644
--- a/posthog/temporal/ai_observability/test_run_trace_evaluation.py
+++ b/posthog/temporal/ai_observability/test_run_trace_evaluation.py
@@ -10,25 +10,32 @@ from unittest.mock import MagicMock, patch
from asgiref.sync import async_to_sync
-from posthog.schema import LLMTrace, LLMTraceEvent
+from posthog.schema import DateRange, LLMTrace, LLMTraceEvent, SessionQuery, TraceQuery
from posthog.hogql import ast
from posthog.hogql.database.database import Database
+from posthog.hogql.query import execute_hogql_query
from posthog.api.capture import CaptureInternalError
from posthog.constants import AvailableFeature
+from posthog.hogql_queries.ai.session_query_runner import SessionQueryRunner
+from posthog.hogql_queries.ai.trace_query_runner import TraceQueryRunner
from posthog.models import Organization, PropertyDefinition, Team, User
from posthog.models.ai_events.test_util import bulk_create_ai_events
from posthog.models.event.util import bulk_create_events
from posthog.temporal.ai_observability.run_session_evaluation import fetch_session_for_evaluation
+from posthog.temporal.ai_observability.trace_summarization.fetch_and_format import _fetch_and_format_trace
from products.access_control.backend.facade.contracts import PropertyAccessLevel
+from products.access_control.backend.models.access_control import AccessControl
from products.access_control.backend.models.property_access_control import PropertyAccessControl
+from products.access_control.backend.property_access_control import restriction_cache_scope
from products.ai_observability.backend.models.evaluation_config import EvaluationConfig
from products.ai_observability.backend.models.evaluations import Evaluation
from products.ai_observability.backend.models.provider_keys import LLMProviderKey
from products.cdp.backend.validation import compile_hog
+from .eval_reports.report_agent.tools import _execute_hogql, get_trace_detail
from .evaluation_hog import execute_hog_eval_bytecode
from .evaluation_llm_judge import BooleanEvalResult
from .evaluation_types import EvaluationActivityResult
@@ -338,59 +345,175 @@ class TestBackgroundEvaluationPropertyAccess:
@pytest.mark.django_db
@pytest.mark.parametrize("target", ["trace", "session"])
@pytest.mark.parametrize(
- "property_type,member_only,access_level,should_skip",
+ "property_type,member_only,access_level",
[
- (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.NONE, True),
- (PropertyDefinition.Type.EVENT, True, PropertyAccessLevel.NONE, False),
- (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.READ, False),
- (PropertyDefinition.Type.PERSON, False, PropertyAccessLevel.NONE, False),
- (PropertyDefinition.Type.GROUP, False, PropertyAccessLevel.NONE, False),
+ (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.NONE),
+ (PropertyDefinition.Type.EVENT, True, PropertyAccessLevel.NONE),
+ (PropertyDefinition.Type.EVENT, False, PropertyAccessLevel.READ),
+ (PropertyDefinition.Type.PERSON, False, PropertyAccessLevel.NONE),
+ (PropertyDefinition.Type.GROUP, False, PropertyAccessLevel.NONE),
],
)
- def test_only_default_event_denials_skip_before_reading_content(
+ @time_machine.travel(FROZEN_NOW, tick=False)
+ def test_online_evaluations_read_properties_independently_of_user_permissions(
self,
setup_data: dict[str, object],
target: str,
property_type: int,
member_only: bool,
access_level: PropertyAccessLevel,
- should_skip: bool,
) -> None:
organization = cast(Organization, setup_data["organization"])
team = cast(Team, setup_data["team"])
organization.available_product_features = [
- {"name": AvailableFeature.PROPERTY_ACCESS_CONTROL, "key": AvailableFeature.PROPERTY_ACCESS_CONTROL}
+ {"name": feature, "key": feature}
+ for feature in (
+ AvailableFeature.PROPERTY_ACCESS_CONTROL,
+ AvailableFeature.ACCESS_CONTROL,
+ AvailableFeature.ROLE_BASED_ACCESS,
+ )
]
organization.save()
- member = None
- if member_only:
- user = User.objects.create_and_join(organization, "[email protected]", "test-password")
- member = user.organization_memberships.get(organization=organization)
- definition = PropertyDefinition.objects.create(
- team=team,
- name="$ai_input",
- type=property_type,
- group_type_index=0 if property_type == PropertyDefinition.Type.GROUP else None,
+ user = User.objects.create_and_join(organization, "[email protected]", "test-password")
+ member = user.organization_memberships.get(organization=organization)
+ AccessControl.objects.create(team=team, resource="llm_analytics", access_level="none")
+ properties = {
+ "$ai_input": [{"role": "user", "content": "What is 2 plus 2?"}],
+ "$ai_output": "The answer is four.",
+ "$ai_output_choices": [{"role": "assistant", "content": "The answer is four."}],
+ "evaluation_context": "arithmetic",
+ }
+ for name in properties:
+ definition = PropertyDefinition.objects.create(
+ team=team,
+ name=name,
+ type=property_type,
+ group_type_index=0 if property_type == PropertyDefinition.Type.GROUP else None,
+ )
+ PropertyAccessControl.objects.create(
+ team=team,
+ property_definition=definition,
+ organization_member=member if member_only else None,
+ access_level=access_level.value,
+ )
+ unit_id = str(uuid.uuid4())
+ bulk_create_ai_events(
+ [
+ {
+ "event": "$ai_generation",
+ "team": team,
+ "distinct_id": "test-person",
+ "timestamp": FROZEN_NOW - timedelta(minutes=1),
+ "properties": {"$ai_trace_id": unit_id, "$ai_session_id": unit_id, **properties},
+ }
+ ]
)
- PropertyAccessControl.objects.create(
- team=team,
- property_definition=definition,
- organization_member=member,
- access_level=access_level.value,
+ fetch = fetch_trace_for_evaluation if target == "trace" else fetch_session_for_evaluation
+ outcome = fetch(team.id, unit_id, FROZEN_NOW)
+
+ assert outcome.skip_reason is None
+ assert outcome.event_count == 1
+ if isinstance(outcome, TraceFetchOutcome):
+ assert outcome.trace is not None
+ trace = outcome.trace
+ else:
+ assert outcome.traces is not None
+ trace = outcome.traces[0]
+ assert len(trace.events) == 1
+ for name, value in properties.items():
+ assert trace.events[0].properties[name] == value
+
+ date_range = DateRange(date_from=(FROZEN_NOW - timedelta(hours=1)).isoformat(), date_to=FROZEN_NOW.isoformat())
+ report_detail = json.loads(
+ get_trace_detail.invoke(
+ {"state": {"team_id": team.id, "trace_id_allowlist": [unit_id]}, "trace_id": unit_id}
+ )
)
- with patch(
- f"posthog.temporal.ai_observability.run_{target}_evaluation.query_ai_events",
- return_value=MagicMock(results=[]),
- ) as mock_query:
- fetch = fetch_trace_for_evaluation if target == "trace" else fetch_session_for_evaluation
- outcome = fetch(team.id, "missing-unit", FROZEN_NOW)
-
- assert outcome.skip_reason == ("property_access_restricted" if should_skip else f"{target}_not_found")
- assert outcome.event_count == 0
- if should_skip:
- mock_query.assert_not_called()
+ summary = _fetch_and_format_trace(
+ unit_id, team.id, (FROZEN_NOW - timedelta(hours=1)).isoformat(), FROZEN_NOW.isoformat()
+ )
+ assert report_detail["event_count"] == 1
+ assert summary is not None and summary.text_repr is not None
+ for text in ("What is 2 plus 2?", "The answer is four."):
+ assert text in report_detail["text"]
+ if (
+ property_type == PropertyDefinition.Type.EVENT
+ and access_level == PropertyAccessLevel.NONE
+ and not member_only
+ ):
+ assert text not in summary.text_repr
+ else:
+ assert text in summary.text_repr
+
+ runner = (
+ TraceQueryRunner(team=team, user=user, query=TraceQuery(traceId=unit_id, dateRange=date_range))
+ if target == "trace"
+ else SessionQueryRunner(team=team, user=user, query=SessionQuery(sessionId=unit_id, dateRange=date_range))
+ )
+ system_runner = type(runner)(team=team, query=runner.query, bypass_property_access_control=True)
+ default_runner = type(runner)(team=team, query=runner.query)
+ assert system_runner.get_cache_key() != default_runner.get_cache_key()
+ user_trace = runner.calculate().results[0]
+ for name, value in properties.items():
+ if property_type == PropertyDefinition.Type.EVENT and access_level == PropertyAccessLevel.NONE:
+ assert name not in user_trace.events[0].properties
+ else:
+ assert user_trace.events[0].properties[name] == value
+
+ bytecode = compile_hog("print(evaluation_events[1].properties.evaluation_context); return true", "destination")
+ result = execute_hog_eval_bytecode(bytecode, build_trace_hog_globals(trace, unit_id, bytecode=bytecode), False)
+ assert result["reasoning"] == "arithmetic"
+ bulk_create_events(
+ [
+ {
+ "event": "$ai_evaluation",
+ "team": team,
+ "distinct_id": "test-person",
+ "timestamp": FROZEN_NOW,
+ "properties": {
+ "$ai_trace_id": unit_id,
+ "$ai_evaluation_reasoning": result["reasoning"],
+ "$ai_evaluation_result": result["verdict"],
+ },
+ }
+ ]
+ )
+ query = "SELECT properties.$ai_evaluation_reasoning, properties.$ai_evaluation_result, properties FROM events WHERE event = '$ai_evaluation' AND properties.$ai_trace_id = {trace_id}"
+ response = execute_hogql_query(
+ query, team=team, user=user, placeholders={"trace_id": ast.Constant(value=unit_id)}
+ )
+ assert response.results[0][0] == "arithmetic"
+
+ if property_type == PropertyDefinition.Type.EVENT and access_level == PropertyAccessLevel.NONE:
+ for name in ("$ai_evaluation_reasoning", "$ai_evaluation_result"):
+ definition = PropertyDefinition.objects.create(team=team, name=name, type=PropertyDefinition.Type.EVENT)
+ PropertyAccessControl.objects.create(
+ team=team,
+ property_definition=definition,
+ organization_member=member if member_only else None,
+ access_level=PropertyAccessLevel.NONE.value,
+ )
+
+ response = execute_hogql_query(
+ query, team=team, user=user, placeholders={"trace_id": ast.Constant(value=unit_id)}
+ )
+ reasoning, verdict, blob = response.results[0]
+ assert _execute_hogql(team.id, query, {"trace_id": ast.Constant(value=unit_id)})[0][0] == "arithmetic"
+ if property_type == PropertyDefinition.Type.EVENT and access_level == PropertyAccessLevel.NONE:
+ assert reasoning is None
+ assert verdict is None
+ assert "$ai_evaluation_reasoning" not in json.loads(blob)
+ assert "$ai_evaluation_result" not in json.loads(blob)
else:
- mock_query.assert_called_once()
+ assert reasoning == "arithmetic"
+
+ # A newly granted reader sees the existing result without rerunning the evaluation.
+ PropertyAccessControl.objects.filter(team=team).update(access_level=PropertyAccessLevel.READ.value)
+ with restriction_cache_scope():
+ response = execute_hogql_query(
+ query, team=team, user=user, placeholders={"trace_id": ast.Constant(value=unit_id)}
+ )
+ assert response.results[0][0] == "arithmetic"
class TestFetchTraceForEvaluation:
@@ -498,6 +621,17 @@ class TestFetchTraceForEvaluation:
@pytest.mark.parametrize("insert_events", [bulk_create_ai_events, bulk_create_events])
def test_reads_the_trace_without_building_the_full_catalog(self, setup_data, insert_events):
team = setup_data["team"]
+ organization = setup_data["organization"]
+ organization.available_product_features = [
+ {"name": AvailableFeature.PROPERTY_ACCESS_CONTROL, "key": AvailableFeature.PROPERTY_ACCESS_CONTROL}
+ ]
+ organization.save()
+ definition = PropertyDefinition.objects.create(
+ team=team, name="evaluation_context", type=PropertyDefinition.Type.EVENT
+ )
+ PropertyAccessControl.objects.create(
+ team=team, property_definition=definition, access_level=PropertyAccessLevel.NONE.value
+ )
trace_id = f"trace-{uuid.uuid4()}"
timestamp = FROZEN_NOW - timedelta(minutes=1)
insert_events(
@@ -518,6 +652,7 @@ class TestFetchTraceForEvaluation:
"$ai_trace_id": trace_id,
"$ai_input": [{"role": "user", "content": "What is the weather?"}],
"$ai_output_choices": [{"role": "assistant", "content": "Sunny all day."}],
+ "evaluation_context": "weather",
},
},
]
@@ -532,6 +667,7 @@ class TestFetchTraceForEvaluation:
assert outcome.skip_reason is None
assert outcome.trace is not None
assert [event.event for event in outcome.trace.events] == ["$ai_generation"]
+ assert outcome.trace.events[0].properties["evaluation_context"] == "weather"
class TestRunHogEvalOverRecentTraces:
@@ -788,7 +924,7 @@ class TestExecuteTraceLLMJudgeActivity:
assert result["result_type"] == output_type
mock_client_class.assert_not_called()
- @pytest.mark.parametrize("skip_reason", ["trace_not_found", "property_access_restricted"])
+ @pytest.mark.parametrize("skip_reason", ["trace_not_found", "trace_too_large"])
def test_skips_without_llm_call(self, skip_reason: str) -> None:
with patch(
"posthog.temporal.ai_observability.run_trace_evaluation.fetch_trace_for_evaluation",
@@ -811,8 +947,7 @@ class TestExecuteTraceLLMJudgeActivity:
assert result["skipped"] is True
assert result["skip_reason"] == skip_reason
- if skip_reason == "property_access_restricted":
- assert "property access rules" in result["reasoning"]
+ assert result["reasoning"]
mock_client_class.assert_not_called()
@@ -878,7 +1013,7 @@ class TestExecuteTraceHogEvalActivity:
assert not result.get("skipped")
@pytest.mark.asyncio
- @pytest.mark.parametrize("skip_reason", ["trace_too_large", "property_access_restricted"])
+ @pytest.mark.parametrize("skip_reason", ["trace_too_large", "trace_not_found"])
async def test_skips_without_running_hog(self, skip_reason: str) -> None:
bytecode = compile_hog("return true", "destination")
evaluation = {
diff --git a/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py b/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py
index 2dc4418b61dc..6706b09abcad 100644
--- a/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py
+++ b/posthog/temporal/ai_observability/trace_summarization/fetch_and_format.py
@@ -44,6 +44,7 @@ def _fetch_and_format_trace(
*,
max_trace_events: int | None = None,
max_raw_trace_size: int | None = None,
+ bypass_property_access_control: bool = False,
) -> FetchResult | None:
"""Fetch trace data and format text representation.
@@ -53,7 +54,7 @@ def _fetch_and_format_trace(
raw_size_limit = max_raw_trace_size if max_raw_trace_size is not None else MAX_RAW_TRACE_SIZE
# A multi-GB trace runs the worker out of memory during the load, before the checks below.
- trace_size = fetch_trace_size(team, trace_id)
+ trace_size = fetch_trace_size(team, trace_id, bypass_property_access_control=bypass_property_access_control)
if trace_size.payload_chars > raw_size_limit or (
max_trace_events is not None and trace_size.event_count > max_trace_events
):
@@ -68,7 +69,9 @@ def _fetch_and_format_trace(
)
return FetchResult(text_repr=None, event_count=trace_size.event_count)
- llm_trace = fetch_trace(team, trace_id, window_start, window_end)
+ llm_trace = fetch_trace(
+ team, trace_id, window_start, window_end, bypass_property_access_control=bypass_property_access_control
+ )
if llm_trace is None:
return None
diff --git a/posthog/temporal/ai_observability/trace_summarization/queries.py b/posthog/temporal/ai_observability/trace_summarization/queries.py
index f44a161ea262..85dbb8366c56 100644
--- a/posthog/temporal/ai_observability/trace_summarization/queries.py
+++ b/posthog/temporal/ai_observability/trace_summarization/queries.py
@@ -48,7 +48,7 @@ class TraceSize:
payload_chars: int
-def fetch_trace_size(team: Team, trace_id: str) -> TraceSize:
+def fetch_trace_size(team: Team, trace_id: str, *, bypass_property_access_control: bool = False) -> TraceSize:
"""Measure a trace in ClickHouse without loading its payload.
A trace ID that a client reuses for every run grows without limit, and `fetch_trace` loads all of
@@ -60,12 +60,20 @@ def fetch_trace_size(team: Team, trace_id: str) -> TraceSize:
placeholders={"trace_id": ast.Constant(value=trace_id)},
team=team,
query_type="TraceSummarizationTraceSize",
+ bypass_property_access_control=bypass_property_access_control,
)
event_count, payload_chars = result.results[0]
return TraceSize(event_count=int(event_count or 0), payload_chars=int(payload_chars or 0))
-def fetch_trace(team: Team, trace_id: str, window_start: str, window_end: str) -> LLMTrace | None:
+def fetch_trace(
+ team: Team,
+ trace_id: str,
+ window_start: str,
+ window_end: str,
+ *,
+ bypass_property_access_control: bool = False,
+) -> LLMTrace | None:
"""Fetch a single trace by ID via the migrated `TraceQueryRunner`.
The runner reads `ai_events` with no timestamp bounds, so this loads the whole trace whatever
@@ -81,7 +89,9 @@ def fetch_trace(team: Team, trace_id: str, window_start: str, window_end: str) -
dateRange=DateRange(date_from=window_start, date_to=window_end),
)
with tags_context(product=Product.LLM_ANALYTICS, feature=Feature.QUERY, team_id=team.id):
- runner = TraceQueryRunner(team=team, query=trace_query)
+ runner = TraceQueryRunner(
+ team=team, query=trace_query, bypass_property_access_control=bypass_property_access_control
+ )
response = runner.calculate()
if not response.results:
diff --git a/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py b/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py
index a836b2a14657..1bff55b5e9ce 100644
--- a/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py
+++ b/posthog/temporal/ai_observability/trace_summarization/tests/test_fetch_and_format.py
@@ -10,6 +10,8 @@ from unittest.mock import MagicMock, patch
from parameterized import parameterized
+from posthog.constants import AvailableFeature
+from posthog.models import PropertyDefinition
from posthog.models.ai_events.test_util import bulk_create_ai_events
from posthog.temporal.ai_observability.trace_summarization.fetch_and_format import (
_fetch_and_format_trace,
@@ -19,6 +21,8 @@ from posthog.temporal.ai_observability.trace_summarization.fetch_and_format impo
from posthog.temporal.ai_observability.trace_summarization.models import FetchAndFormatInput
from posthog.temporal.ai_observability.trace_summarization.queries import TraceSize
+from products.access_control.backend.models.property_access_control import PropertyAccessControl
+
@asynccontextmanager
async def _noop_heartbeater(*args, **kwargs):
@@ -141,10 +145,11 @@ class TestFetchAndFormatTrace:
class TestFetchAndFormatTraceSizePreflight(ClickhouseTestMixin, BaseTest):
@parameterized.expand(
[
- ("heavy_content_outside_window", 5_000, None, True),
- ("small_trace", 10, None, False),
- ("root_and_duplicate_rows_not_counted", 10, 2, False),
- ("over_event_limit", 10, 1, True),
+ ("heavy_content_outside_window", 5_000, None, True, False),
+ ("restricted_heavy_content_outside_window", 5_000, None, True, True),
+ ("small_trace", 10, None, False, False),
+ ("root_and_duplicate_rows_not_counted", 10, 2, False, False),
+ ("over_event_limit", 10, 1, True, False),
]
)
@patch("posthog.temporal.ai_observability.trace_summarization.fetch_and_format.fetch_trace", return_value=None)
@@ -154,8 +159,18 @@ class TestFetchAndFormatTraceSizePreflight(ClickhouseTestMixin, BaseTest):
old_input_length: int,
max_trace_events: int | None,
expect_skip: bool,
+ restricted_input: bool,
mock_fetch: MagicMock,
) -> None:
+ if restricted_input:
+ self.organization.available_product_features = [
+ {"name": AvailableFeature.PROPERTY_ACCESS_CONTROL, "key": AvailableFeature.PROPERTY_ACCESS_CONTROL}
+ ]
+ self.organization.save()
+ definition = PropertyDefinition.objects.create(
+ team=self.team, name="$ai_input", type=PropertyDefinition.Type.EVENT
+ )
+ PropertyAccessControl.objects.create(team=self.team, property_definition=definition, access_level="none")
trace_id = str(uuid.uuid4())
window_end = datetime.now(UTC)
window_start = window_end - timedelta(hours=1)
@@ -186,6 +201,7 @@ class TestFetchAndFormatTraceSizePreflight(ClickhouseTestMixin, BaseTest):
window_end=window_end.isoformat(),
max_raw_trace_size=1_000,
max_trace_events=max_trace_events,
+ bypass_property_access_control=restricted_input,
)
if expect_skip: