Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): compare first-report scout models (#122558)

Georgiy Tarasov503b7c13bb55
diff --git a/docs/internal/scout-online-evals-devbox.md b/docs/internal/scout-online-evals-devbox.md
index 0b0c50ef88dd..d4db6b841bd4 100644
--- a/docs/internal/scout-online-evals-devbox.md
+++ b/docs/internal/scout-online-evals-devbox.md
@@ -2,6 +2,8 @@
Trials run scout versions in parallel, then judge them against the same saved rubric. Versions can change the prompt, model or reasoning effort.
+The onboarding model experiment `self-driving-banger-scout` runs ordinary customer-visible scouts and records normal analytics. It does not use these private trial endpoints or capture suppression. Its project assignment, model variants, gateway requirements, and measurement limits are documented in [First report onboarding](../../products/signals/ARCHITECTURE.md#first-report-onboarding-and-email). Production enrollment also requires a successful live data probe; empty or unknown data never launches a scout sandbox. Both variants use internally marked pipeline tasks; authorized customers can still read their progress. Completion notifications target the activating user in the onboarding environment and are best-effort. First-report email delivery is separate. Pi provisioning retries transient token-mint and model-pin persistence failures, while configuration and authorization refusals remain terminal. Keep the experiment in draft until both runtime paths and the scoped gateway route are deployed; use isolated eval fixtures to verify the variants before enrolling customer projects.
+
## Before starting
Follow the existing [devbox setup](../../.agents/skills/setting-up-devbox/SKILL.md) and [local stack](../../.agents/skills/run-posthog/SKILL.md) instructions.
diff --git a/frontend/src/lib/components/NotificationsMenu/NotificationRow.tsx b/frontend/src/lib/components/NotificationsMenu/NotificationRow.tsx
index 776146c4cbdd..27bae17433b5 100644
--- a/frontend/src/lib/components/NotificationsMenu/NotificationRow.tsx
+++ b/frontend/src/lib/components/NotificationsMenu/NotificationRow.tsx
@@ -81,6 +81,10 @@ export const REALTIME_NOTIFICATION_TYPE_META: Record<string, { label: string; de
label: 'Workflow suggestions',
description: 'When PostHog suggests a change to a workflow you set up',
},
+ first_report_completed: {
+ label: 'First Self-Driving investigation',
+ description: 'When your first Self-Driving investigation finishes, including when it has no report.',
+ },
feature_flag_failure: {
label: 'Feature flag failures',
description: 'When a feature flag in your project fails to evaluate',
diff --git a/frontend/src/lib/components/NotificationsMenu/notificationToasts.tsx b/frontend/src/lib/components/NotificationsMenu/notificationToasts.tsx
index 21cdea092681..dc592a320d60 100644
--- a/frontend/src/lib/components/NotificationsMenu/notificationToasts.tsx
+++ b/frontend/src/lib/components/NotificationsMenu/notificationToasts.tsx
@@ -42,6 +42,7 @@ const NOTIFICATION_TYPE_ICONS: Record<string, { Icon: ComponentType<{ className?
data_quality_check_failure: { Icon: IconWarning, color: 'text-danger' },
workflow_suggestion: { Icon: IconSparkles, color: 'text-primary' },
feature_flag_failure: { Icon: IconToggle, color: 'text-danger' },
+ first_report_completed: { Icon: IconSparkles, color: 'text-success' },
}
export function getNotificationIcon(notificationType: string, className: string = 'size-5'): JSX.Element {
diff --git a/products/notifications/backend/facade/enums.py b/products/notifications/backend/facade/enums.py
index 1f6248c983b1..83c281d57ddb 100644
--- a/products/notifications/backend/facade/enums.py
+++ b/products/notifications/backend/facade/enums.py
@@ -24,6 +24,7 @@ class NotificationType(str, Enum):
DATA_QUALITY_CHECK_FAILURE = "data_quality_check_failure"
WORKFLOW_SUGGESTION = "workflow_suggestion"
FEATURE_FLAG_FAILURE = "feature_flag_failure"
+ FIRST_REPORT_COMPLETED = "first_report_completed"
class Priority(str, Enum):
diff --git a/products/signals/ARCHITECTURE.md b/products/signals/ARCHITECTURE.md
index 629445fc5bcd..723885b7aa55 100644
--- a/products/signals/ARCHITECTURE.md
+++ b/products/signals/ARCHITECTURE.md
@@ -242,7 +242,13 @@ The `signals-scout-first-report` engine performs a bounded investigation of exis
This system scout is excluded from scheduled registration and suggestions and accepts only onboarding dispatches. Its template remains visible in the Skills store like other canonical scout skills. Its structured close-out distinguishes insufficient data, no actionable finding, and incomplete execution. The engine exposes persisted state and task IDs for onboarding consumers; activation and customer progress presentation are integrated separately.
-Guided and quick onboarding create a `SignalFirstReportNotification` in the activation transaction, preserving the activating user for that environment. The first report that becomes `ready`, immediately actionable, and not already addressed after activation triggers a transactional email to that user's email address, tracked against their distinct ID. Control onboarding does not identify an activator and does not enroll a recipient.
+The project-group experiment `self-driving-banger-scout` decides whether the first-report scout runs and which model it uses: `control` runs no first-report scout, `test-sol` selects GPT-6.1 Sol on Codex with high reasoning and priority service, and `test-deepseek` selects DeepSeek Flash 1017 Fast on Pi. Both test arms use the same scout skill. The runner refuses a control assignment. Disabled, unknown, and unavailable flag values prevent production first-report runs; explicit eval and private-trial runtime overrides bypass enrollment. Assignment is evaluated once per run, recorded on the run, and attached to lifecycle and report events. Exposure starts before sandbox creation, after dispatch guards, so setup failures remain in the experiment denominator.
+
+`signals_scout_run_finished` records the structured first-report outcome, whether a report became visible, and its latency from task creation. A persisted visible report still counts when the agent subsequently fails. The experiment's exposure-to-report funnel includes the earlier setup interval; its five-minute delivery metric measures delivery, not report quality. Compare grounding and actionability separately, and join `task_run_id` to gateway usage for model cost; missing usage is unknown, never zero. Turning the experiment flag off prevents future first-report sandbox starts; already-running investigations may finish.
+
+Pi scouts require the Tasks worker's `SANDBOX_AI_GATEWAY_URL`, `SANDBOX_AI_GATEWAY_MINT_KEY`, and an entry in `SANDBOX_AI_GATEWAY_PRODUCTS` matching the scout, such as `signals_scout:first-report`. Only internally created scout tasks may mint these tokens. Pi provisioning fails when routing, minting, or recording the model pin fails; it never falls back to the legacy gateway. Before launching the experiment, verify the deployed Go gateway serves `posthog/deepseek-ai/deepseek-v4.1-flash-1017-fast` for `signals_scout`, with the Baseten host configured. The existing internal scout budget and paying-project configuration still apply.
+
+Guided and quick onboarding create a `SignalFirstReportNotification` in the activation transaction, preserving the activating user for that environment. The first report that becomes `ready`, immediately actionable, and not already addressed after activation triggers a transactional email to that user's email address, tracked against their distinct ID. When the parent project's first investigation records this environment as its activation origin, email delivery selects the parent's reports and links while retaining the environment's recipient and notification preferences. The recipient must retain access to both projects. Other environment notifications keep their own report scope. Control onboarding does not identify an activator and does not enroll a recipient.
Sending requires the organization-scoped `signals-first-report-email` feature flag and is cloud-only. A disabled, missing, or unavailable flag leaves notifications pending without claiming or cancelling them, including in DEBUG. Sending also remains disabled until the dedicated `POSTHOG_TRANSACTIONAL_EMAIL_API_KEY` is configured. `SIGNALS_FIRST_REPORT_EMAIL_TEMPLATE_ID` in `backend/constants.py` selects the first-report transactional email template. The shared `POSTHOG_PROJECT_ID` and `POSTHOG_HOST` settings default to `2` and `https://us.posthog.com`, independently of the recipient's region. Both support optional environment overrides to select another sending project or host. The API key must be a project secret API key with `hog_flow:write` for that sending project, and the template must be an active transactional email template with a verified sender. Sending uses the shared transactional email client synchronously inside the notification task so the notification row can record the API's acceptance. Template creation and deployment configuration are separate from this code.
diff --git a/products/signals/backend/agent_runtime.py b/products/signals/backend/agent_runtime.py
index b3e3eca06a5f..80ef15879f21 100644
--- a/products/signals/backend/agent_runtime.py
+++ b/products/signals/backend/agent_runtime.py
@@ -87,6 +87,7 @@ class AgentRuntime:
model: str | None = None
reasoning_effort: str | None = None
service_tier: str | None = None
+ runtime: str = "acp"
DEFAULT_RUNTIME = AgentRuntime()
diff --git a/products/signals/backend/first_report_completion.py b/products/signals/backend/first_report_completion.py
new file mode 100644
index 000000000000..5fc280b2c76a
--- /dev/null
+++ b/products/signals/backend/first_report_completion.py
@@ -0,0 +1,95 @@
+from __future__ import annotations
+
+from django.db.models import Q
+
+import posthoganalytics
+
+from posthog.exceptions_capture import capture_exception
+from posthog.models import Team, User
+
+from products.notifications.backend.facade.api import (
+ NotificationData,
+ NotificationType,
+ TargetType,
+ create_notification,
+)
+from products.signals.backend.first_report_experiment import first_report_outcome_properties
+from products.signals.backend.models import SignalFirstReportNotification, SignalScoutRun, SignalTeamConfig
+from products.signals.backend.report_access import may_read_reports
+
+
+def first_report_completion_notifications_enabled(team: Team, user: User) -> bool:
+ disabled = (user.partial_notification_settings or {}).get("realtime_notifications_disabled") or {}
+ if (disabled.get(NotificationType.FIRST_REPORT_COMPLETED.value) or {}).get(str(team.id)):
+ return False
+ try:
+ return bool(
+ posthoganalytics.feature_enabled(
+ "real-time-notifications",
+ str(team.organization_id),
+ groups={"organization": str(team.organization_id)},
+ only_evaluate_locally=False,
+ send_feature_flag_events=False,
+ )
+ )
+ except Exception as error:
+ capture_exception(error)
+ return False
+
+
+def complete_first_report_investigation(team_id: int, run_id: str) -> dict[str, str | float | bool | None]:
+ properties = first_report_outcome_properties(team_id, run_id)
+ try:
+ run = SignalScoutRun.objects.for_team(team_id).select_related("team").get(id=run_id)
+ onboarding = SignalTeamConfig.objects.filter(team_id=team_id).values_list("onboarding", flat=True).first() or {}
+ activation = (
+ SignalFirstReportNotification.objects.for_team(onboarding.get("first_report_activation_team_id", team_id))
+ .filter(Q(team_id=team_id) | Q(team__parent_team_id=team_id))
+ .select_related("team", "activated_by")
+ .first()
+ )
+ if activation is None:
+ return properties
+ user = activation.activated_by
+ if (
+ user is None
+ or not user.is_active
+ or not may_read_reports(user=user, team=run.team)
+ or not may_read_reports(user=user, team=activation.team)
+ ):
+ return properties
+ outcome = properties.get("first_report_outcome")
+ messages = {
+ "report_created": (
+ "Your first report is ready",
+ "Open Self-Driving to see the finding and its next steps.",
+ ),
+ "insufficient_data": (
+ "Your first investigation needs more data",
+ "Open Self-Driving to see what data is missing.",
+ ),
+ "no_actionable_finding": (
+ "Your first investigation is complete",
+ "We checked your data and didn’t find anything worth reporting yet.",
+ ),
+ "incomplete": (
+ "Your first investigation couldn’t finish",
+ "Open Self-Driving to review the investigation.",
+ ),
+ }
+ title, body = messages.get(str(outcome), messages["incomplete"])
+ create_notification(
+ NotificationData(
+ team_id=activation.team_id,
+ notification_type=NotificationType.FIRST_REPORT_COMPLETED,
+ title=title,
+ body=body,
+ target_type=TargetType.USER,
+ target_id=str(user.pk),
+ source_url=f"/project/{team_id}/inbox",
+ idempotency_key=f"first-report:{run_id}",
+ )
+ )
+ except Exception as error:
+ capture_exception(error)
+ return properties
diff --git a/products/signals/backend/first_report_experiment.py b/products/signals/backend/first_report_experiment.py
new file mode 100644
index 000000000000..52aa4449478e
--- /dev/null
+++ b/products/signals/backend/first_report_experiment.py
@@ -0,0 +1,128 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import posthoganalytics
+
+from posthog.dataclasses import frozen
+from posthog.event_usage import groups
+from posthog.exceptions_capture import capture_exception
+
+from products.signals.backend.agent_runtime import AgentRuntime
+from products.signals.backend.models import SignalReport, SignalScoutRun
+
+if TYPE_CHECKING:
+ from posthog.models.team.team import Team
+
+FIRST_REPORT_EXPERIMENT = "self-driving-banger-scout"
+
+
+@frozen
+class FirstReportExperiment:
+ variant: str
+ # None for the control arm, which runs no first-report scout.
+ agent_runtime: AgentRuntime | None
+
+ @property
+ def runs_scout(self) -> bool:
+ return self.agent_runtime is not None
+
+ def properties(self) -> dict[str, str]:
+ properties = {
+ "experiment_key": FIRST_REPORT_EXPERIMENT,
+ "experiment_variant": self.variant,
+ f"$feature/{FIRST_REPORT_EXPERIMENT}": self.variant,
+ }
+ if self.agent_runtime is not None:
+ properties["runtime"] = self.agent_runtime.runtime
+ return properties
+
+
+def resolve_first_report_experiment(team: Team) -> FirstReportExperiment | None:
+ try:
+ variant = posthoganalytics.get_feature_flag(
+ FIRST_REPORT_EXPERIMENT,
+ str(team.uuid),
+ groups={"project": str(team.uuid)},
+ group_properties={"project": {"id": team.id, "uuid": str(team.uuid)}},
+ send_feature_flag_events=False,
+ )
+ except Exception as error:
+ capture_exception(error)
+ return None
+ if variant == "control":
+ return FirstReportExperiment(variant=variant, agent_runtime=None)
+ if variant == "test-sol":
+ return FirstReportExperiment(
+ variant=variant,
+ agent_runtime=AgentRuntime(
+ runtime_adapter="codex", model="gpt-6.1-sol", reasoning_effort="high", service_tier="priority"
+ ),
+ )
+ if variant == "test-deepseek":
+ return FirstReportExperiment(
+ variant=variant,
+ agent_runtime=AgentRuntime(runtime="pi", model="deepseek-ai/deepseek-v4.1-flash-1017-fast"),
+ )
+ return None
+
+
+def capture_first_report_exposure(team: Team, experiment: FirstReportExperiment, run_id: str) -> None:
+ try:
+ posthoganalytics.capture(
+ distinct_id=str(team.uuid),
+ event="$experiment_exposure",
+ groups=groups(team.organization, team),
+ properties={
+ **experiment.properties(),
+ "$feature_flag": FIRST_REPORT_EXPERIMENT,
+ "$feature_flag_response": experiment.variant,
+ "run_id": run_id,
+ },
+ )
+ except Exception as error:
+ capture_exception(error)
+
+
+def _first_report_outcome_properties(team_id: int, run_id: str) -> dict[str, str | float | bool | None]:
+ run = SignalScoutRun.objects.for_team(team_id).select_related("task_run").filter(id=run_id).first()
+ if run is None:
+ return {
+ "first_report_outcome": "incomplete",
+ "report_delivered": False,
+ "report_delivered_within_five_minutes": False,
+ }
+ report = (
+ SignalReport.objects.filter(
+ team_id=team_id,
+ id__in=run.emitted_report_ids or [],
+ first_visible_at__isnull=False,
+ )
+ .order_by("first_visible_at")
+ .first()
+ )
+ result = (run.metadata or {}).get("first_report_result", {})
+ outcome = result.get("outcome", "incomplete")
+ if report is None:
+ return {
+ "first_report_outcome": "incomplete" if outcome == "report_created" else outcome,
+ "report_delivered": False,
+ "report_delivered_within_five_minutes": False,
+ }
+ assert report.first_visible_at is not None
+ latency = (report.first_visible_at - run.task_run.created_at).total_seconds()
+ return {
+ "first_report_outcome": "report_created",
+ "report_delivered": True,
+ "report_id": str(report.id),
+ "first_report_latency_seconds": latency,
+ "report_delivered_within_five_minutes": 0 <= latency <= 300,
+ }
+
+
+def first_report_outcome_properties(team_id: int, run_id: str) -> dict[str, str | float | bool | None]:
+ try:
+ return _first_report_outcome_properties(team_id, run_id)
+ except Exception as error:
+ capture_exception(error)
+ return {"first_report_measurement_failed": True}
diff --git a/products/signals/backend/first_report_notification.py b/products/signals/backend/first_report_notification.py
index ad24b3c76a1a..3e7c80486cf1 100644
--- a/products/signals/backend/first_report_notification.py
+++ b/products/signals/backend/first_report_notification.py
@@ -24,7 +24,7 @@ from products.access_control.backend.facade.user_access_control import UserAcces
from products.signals.backend.artefact_schemas import ActionabilityChoice
from products.signals.backend.briefing_reports import SUMMARY_LEAD_LIMIT, summary_lead
from products.signals.backend.constants import SIGNALS_FIRST_REPORT_EMAIL_TEMPLATE_ID
-from products.signals.backend.models import SignalFirstReportNotification, SignalReport
+from products.signals.backend.models import SignalFirstReportNotification, SignalReport, SignalTeamConfig
from products.signals.backend.report_access import may_read_reports
from products.signals.backend.report_content_gates import team_first_report_email_enabled
from products.signals.backend.report_priority import latest_report_priorities
@@ -75,18 +75,28 @@ def _claim(team_id: int) -> SignalFirstReportNotification | None:
notification = (
SignalFirstReportNotification.objects.for_team(team_id)
.select_for_update(of=("self",))
- .select_related("team__organization", "activated_by")
+ .select_related("team__organization", "team__parent_team__organization", "activated_by")
.filter(queued_at__isnull=True, cancelled_at__isnull=True, send_started_at__isnull=True)
.filter(Q(next_attempt_at__isnull=True) | Q(next_attempt_at__lte=timezone.now()))
.first()
)
if notification is None:
return None
- if not _eligible_recipient(notification.activated_by, notification.team):
+ report_team = notification.team
+ parent = notification.team.parent_team
+ if parent is not None:
+ onboarding = (
+ SignalTeamConfig.objects.filter(team_id=parent.id).values_list("onboarding", flat=True).first() or {}
+ )
+ if onboarding.get("first_report_activation_team_id") == team_id:
+ report_team = parent
+ if not _eligible_recipient(notification.activated_by, notification.team) or (
+ report_team.id != team_id and not _eligible_recipient(notification.activated_by, report_team)
+ ):
notification.cancelled_at = timezone.now()
notification.save(update_fields=["cancelled_at"])
return None
- reports = _visible_reports(team_id).filter(first_visible_at__gte=notification.activated_at)
+ reports = _visible_reports(report_team.id).filter(first_visible_at__gte=notification.activated_at)
if notification.message_data:
report = reports.filter(id=notification.report_id).first() if notification.report_id else None
else:
@@ -95,7 +105,7 @@ def _claim(team_id: int) -> SignalFirstReportNotification | None:
if notification.message_data and (
notification.report_id is None
or not SignalReport.objects.filter(
- team_id=team_id,
+ team_id=report_team.id,
id=notification.report_id,
status__in=[
SignalReport.Status.CANDIDATE,
@@ -110,15 +120,15 @@ def _claim(team_id: int) -> SignalFirstReportNotification | None:
if notification.report_id is None:
notification.report = report
notification.message_data = {
- "project_id": team_id,
+ "project_id": report_team.id,
"report_id": str(report.id),
"report_title": report.title,
# The email shows plain text, so a chart link keeps only its label.
"report_summary": Truncator(
summary_lead(strip_chart_references(report.summary or ""), SUMMARY_LEAD_LIMIT)
).chars(320),
- "report_priority": latest_report_priorities(team_id, [str(report.id)]).get(str(report.id), ""),
- "report_url": build_report_url(team_id, report.id, ReportLinkSource.FIRST_REPORT_EMAIL),
+ "report_priority": latest_report_priorities(report_team.id, [str(report.id)]).get(str(report.id), ""),
+ "report_url": build_report_url(report_team.id, report.id, ReportLinkSource.FIRST_REPORT_EMAIL),
"activation_id": str(notification.id),
}
# The transactional API has no idempotency key. A crash or uncertain response must leave
diff --git a/products/signals/backend/first_report_scout.py b/products/signals/backend/first_report_scout.py
index 25f5dda0b3bf..b48c31ec6b3f 100644
--- a/products/signals/backend/first_report_scout.py
+++ b/products/signals/backend/first_report_scout.py
@@ -4,6 +4,7 @@ import json
from datetime import datetime, timedelta
from django.db import transaction
+from django.db.models import Q
from django.utils import timezone
from pydantic import BaseModel, Field
@@ -36,28 +37,25 @@ class WarehouseDataSummary(BaseModel):
name: str
recorded_row_count: int | None
timestamp_columns: list[str]
- has_rows: bool | None = None
class FirstReportContext(BaseModel):
computed_at: str
inventory: Inventory
warehouse_tables: list[WarehouseDataSummary]
+ events_ingested: bool = False
product_data: dict[str, ProductDataVolume] = Field(default_factory=dict)
def has_data(self) -> bool:
return (
- any(event.count > 0 for event in self.inventory.top_events or [])
- or any((table.recorded_row_count or 0) > 0 or table.has_rows is True for table in self.warehouse_tables)
+ self.events_ingested
+ or any((table.recorded_row_count or 0) > 0 for table in self.warehouse_tables)
or any((volume.rows or 0) > 0 or volume.has_rows is True for volume in self.product_data.values())
)
+ # A warehouse table without a recorded row count has not finished a sync, so it is no evidence either way.
def data_is_unknown(self) -> bool:
- return (
- self.inventory.top_events is None
- or any(table.recorded_row_count is None and table.has_rows is None for table in self.warehouse_tables)
- or any(volume.has_rows is None and not volume.rows for volume in self.product_data.values())
- )
+ return any(volume.has_rows is None and not volume.rows for volume in self.product_data.values())
def prompt_context(self) -> str:
inventory = self.inventory.model_dump(
@@ -158,35 +156,19 @@ def build_first_report_context(team: Team, user: User) -> FirstReportContext:
)
for table in tables[:50]
],
+ events_ingested=Team.objects.filter(Q(pk=team.id) | Q(parent_team_id=team.id), ingested_event=True).exists(),
)
for name, table_path, timestamp in (
+ ("events", ["events"], "timestamp"),
("session_replay_fragments", ["raw_session_replay_events"], "min_first_timestamp"),
("logs", ["logs"], "timestamp"),
("apm", ["posthog", "trace_spans"], "timestamp"),
):
context.product_data[name] = _product_data_volume(team, user, table_path, timestamp)
- if not context.has_data():
- context.product_data["events"] = _product_data_volume(team, user, ["events"], "timestamp")
- for table in [table for table in context.warehouse_tables if table.recorded_row_count is None][:3]:
- try:
- result = execute_hogql_query(
- query=parse_select(
- "SELECT 1 FROM {table} LIMIT 1", placeholders={"table": ast.Field(chain=[table.name])}
- ),
- team=team,
- user=user,
- workload=Workload.OFFLINE,
- settings=HogQLGlobalSettings(max_execution_time=3, timeout_overflow_mode="throw"),
- )
- table.has_rows = bool(result.results)
- except Exception:
- pass
- if context.has_data():
- break
return context
-def set_first_report_state(team_id: int, status: str) -> None:
+def set_first_report_state(team_id: int, status: str, *, activation_team_id: int | None = None) -> None:
with transaction.atomic():
config, _ = SignalTeamConfig.objects.get_or_create(team_id=team_id)
config = SignalTeamConfig.objects.select_for_update().get(pk=config.pk, team_id=team_id)
@@ -198,6 +180,8 @@ def set_first_report_state(team_id: int, status: str) -> None:
**state,
"first_report": {"status": status, "started_at": timezone.now().isoformat()},
}
+ if activation_team_id is not None:
+ config.onboarding.setdefault("first_report_activation_team_id", activation_team_id)
config.save(update_fields=["onboarding", "updated_at"])
@@ -212,7 +196,7 @@ def first_report_state(team_id: int) -> dict[str, object]:
.first()
)
if run is None:
- if state.get("status") == FirstReportStatus.RUNNING and state.get("started_at"):
+ if state.get("status") in (FirstReportStatus.PENDING, FirstReportStatus.RUNNING) and state.get("started_at"):
started_at = datetime.fromisoformat(state["started_at"])
if timezone.now() - started_at > timedelta(seconds=FIRST_REPORT_SCOUT_TIMEOUT_SECONDS + 300):
return {"status": FirstReportStatus.INCOMPLETE}
diff --git a/products/signals/backend/scout_harness/runner.py b/products/signals/backend/scout_harness/runner.py
index 18e9f48ad5e2..e0df3ef6eff5 100644
--- a/products/signals/backend/scout_harness/runner.py
+++ b/products/signals/backend/scout_harness/runner.py
@@ -34,6 +34,13 @@ from products.data_catalog.backend.facade.api import approved_metric_names_for_t
from products.mcp_store.backend.facade.api import get_sandbox_mcp_server_names
from products.signals.backend.agent_runtime import STEP_SCOUT, AgentRuntime, resolve_agent_runtime
from products.signals.backend.constants import FIRST_REPORT_SCOUT, FIRST_REPORT_SCOUT_TIMEOUT_SECONDS
+from products.signals.backend.first_report_completion import complete_first_report_investigation
+from products.signals.backend.first_report_experiment import (
+ FirstReportExperiment,
+ capture_first_report_exposure,
+ resolve_first_report_experiment,
+)
+from products.signals.backend.first_report_scout import set_first_report_state
from products.signals.backend.first_report_scout_types import FirstReportScoutResult
from products.signals.backend.models import SignalScoutConfig, SignalScoutRun
from products.signals.backend.scout_harness.acting_user import resolve_scout_run_acting_user
@@ -463,6 +470,27 @@ async def _arun_signals_scout(
# runtime/model/effort triple is taken from one source — a Codex runtime never pairs with a
# model it can't serve. Model-only pin entries are still ignored for scout: a pin supplies
# model+runtime as a pair, and overriding one without the other would mis-route.
+ experiment = None
+ if skill.name == FIRST_REPORT_SCOUT and agent_runtime is None and trial is None:
+ experiment = await database_sync_to_async(resolve_first_report_experiment, thread_sensitive=False)(team)
+ if experiment is None or experiment.agent_runtime is None:
+ await database_sync_to_async(set_first_report_state, thread_sensitive=False)(team.id, "incomplete")
+ return RunResult(
+ run_id=None,
+ task_run_id=None,
+ status=None,
+ last_message=None,
+ runtime_s=0.0,
+ skill_name=skill.name,
+ skill_version=skill.version,
+ skip_reason="first-report experiment disabled" if experiment is None else "first-report control arm",
+ )
+ agent_runtime = experiment.agent_runtime
+ runtime = agent_runtime.runtime if agent_runtime is not None else "acp"
+ if experiment is not None:
+ await database_sync_to_async(capture_first_report_exposure, thread_sensitive=False)(
+ team, experiment, str(run_id)
+ )
if agent_runtime is not None:
runtime_adapter: str | None = agent_runtime.runtime_adapter
model: str | None = agent_runtime.model
@@ -535,6 +563,8 @@ async def _arun_signals_scout(
runtime_adapter=runtime_adapter,
reasoning_effort=reasoning_effort,
service_tier=service_tier,
+ runtime=runtime,
+ experiment=experiment,
triggered_by=triggered_by,
run_note=run_note,
trigger_workflow_id=trigger_workflow_id,
@@ -555,6 +585,13 @@ async def _arun_signals_scout(
# after fixing its skill.
if trial is None:
await database_sync_to_async(_clear_failure_streak, thread_sensitive=False)(config.pk)
+ outcome_properties = (
+ await database_sync_to_async(complete_first_report_investigation, thread_sensitive=False)(
+ team.id, str(run_id)
+ )
+ if experiment is not None
+ else {}
+ )
_capture_run_finished(
team=team,
config=config,
@@ -571,6 +608,9 @@ async def _arun_signals_scout(
model=model,
runtime_adapter=runtime_adapter,
service_tier=service_tier,
+ runtime=runtime,
+ experiment=experiment,
+ extra_properties=outcome_properties,
)
return RunResult(
run_id=str(run_id),
@@ -640,6 +680,13 @@ async def _arun_signals_scout(
if triggered_by == TRIGGERED_BY_SCHEDULE
else None
)
+ outcome_properties = (
+ await database_sync_to_async(complete_first_report_investigation, thread_sensitive=False)(
+ team.id, str(run_id)
+ )
+ if experiment is not None
+ else {}
+ )
_capture_run_finished(
team=team,
config=config,
@@ -656,9 +703,11 @@ async def _arun_signals_scout(
model=model,
runtime_adapter=runtime_adapter,
service_tier=service_tier,
+ runtime=runtime,
+ experiment=experiment,
error_type=type(exc).__name__,
error_message=str(exc)[:300],
- extra_properties=_failure_properties(exc),
+ extra_properties={**outcome_properties, **(_failure_properties(exc) or {})},
)
if streak is not None and streak.tripped:
_capture_config_auto_paused(
@@ -701,6 +750,13 @@ async def _arun_signals_scout(
# The failure-streak breaker is deliberately untouched: a cancelled run says nothing
# about whether this lane can succeed, and counting worker shutdowns toward the streak
# would pause healthy scouts after a few deploys.
+ outcome_properties = (
+ await database_sync_to_async(complete_first_report_investigation, thread_sensitive=False)(
+ team.id, str(run_id)
+ )
+ if experiment is not None
+ else {}
+ )
_capture_run_finished(
team=team,
config=config,
@@ -717,6 +773,9 @@ async def _arun_signals_scout(
model=model,
runtime_adapter=runtime_adapter,
service_tier=service_tier,
+ runtime=runtime,
+ experiment=experiment,
+ extra_properties=outcome_properties,
)
raise
finally:
@@ -941,6 +1000,8 @@ async def _spawn_and_run(
runtime_adapter: str | None = None,
reasoning_effort: str | None = None,
service_tier: str | None = None,
+ runtime: str = "acp",
+ experiment: FirstReportExperiment | None = None,
triggered_by: str = TRIGGERED_BY_SCHEDULE,
run_note: str | None = None,
trigger_workflow_id: str | None = None,
@@ -1033,6 +1094,7 @@ async def _spawn_and_run(
reasoning_effort=reasoning_effort,
# Codex-only, and independent of the model pin: which OpenAI queue the run's turns join.
service_tier=service_tier,
+ runtime=runtime,
sandbox_timeout_seconds=TRIAL_ACTIVITY_TIMEOUT_S + 60 if trial is not None else None,
)
project_has_governed_metrics = await database_sync_to_async(_project_has_governed_metrics, thread_sensitive=False)(
@@ -1100,6 +1162,8 @@ async def _spawn_and_run(
runtime_adapter=runtime_adapter,
reasoning_effort=reasoning_effort,
service_tier=service_tier,
+ runtime=runtime,
+ experiment=experiment,
github_guidance=github_guidance,
business_knowledge_maintained=business_knowledge_maintained,
background_backoff=background_backoff,
@@ -1131,6 +1195,8 @@ async def _spawn_and_run(
model=model,
runtime_adapter=runtime_adapter,
service_tier=service_tier,
+ runtime=runtime,
+ experiment=experiment,
)
)
@@ -1142,6 +1208,7 @@ async def _spawn_and_run(
step_name=_step_name(skill),
verbose=verbose,
origin_product=tasks_facade.TaskOriginProduct.SIGNALS_SCOUT,
+ internal=skill.name == FIRST_REPORT_SCOUT,
mcp_builtin_agent_key="scout",
# No credential owner on purpose: a scout is a team resource, so its runs mount only
# connections members shared to the whole team, never anyone's personal grants. That
@@ -1398,6 +1465,8 @@ def _create_run_row(
runtime_adapter: str | None = None,
reasoning_effort: str | None = None,
service_tier: str | None = None,
+ runtime: str = "acp",
+ experiment: FirstReportExperiment | None = None,
github_guidance: bool = False,
business_knowledge_maintained: bool = False,
background_backoff: BackgroundBackoff | None = None,
@@ -1424,6 +1493,9 @@ def _create_run_row(
)
if value is not None
}
+ metadata["runtime"] = runtime
+ if experiment is not None:
+ metadata.update(experiment.properties())
# Stamped only when non-default, like the model triple: absence means the run held the
# trusted-domains posture. Stamped (rather than read off the config later) because the
# config row can be edited after the fact, which would silently rewrite what past runs
@@ -1720,6 +1792,8 @@ def _capture_run_started(
model: str | None = None,
runtime_adapter: str | None = None,
service_tier: str | None = None,
+ runtime: str = "acp",
+ experiment: FirstReportExperiment | None = None,
) -> None:
"""Emit the scout-owned run-started analytics event.
@@ -1739,6 +1813,9 @@ def _capture_run_started(
"run_id": str(run_id),
"task_run_id": task_run_id,
}
+ properties["runtime"] = runtime
+ if experiment is not None:
+ properties.update(experiment.properties())
_attach_run_shape_props(
properties,
config=config,
@@ -1935,6 +2012,8 @@ def _capture_run_finished(
model: str | None = None,
runtime_adapter: str | None = None,
service_tier: str | None = None,
+ runtime: str = "acp",
+ experiment: FirstReportExperiment | None = None,
error_type: str | None = None,
error_message: str | None = None,
extra_properties: dict[str, Any] | None = None,
@@ -1968,6 +2047,9 @@ def _capture_run_finished(
"runtime_seconds": round(runtime_s, 1),
"emitted_count": emitted_count,
}
+ properties["runtime"] = runtime
+ if experiment is not None:
+ properties.update(experiment.properties())
_attach_run_shape_props(
properties,
config=config,
diff --git a/products/signals/backend/scout_harness/tools/report_author.py b/products/signals/backend/scout_harness/tools/report_author.py
index f5168aec6e0d..ff025d21bcce 100644
--- a/products/signals/backend/scout_harness/tools/report_author.py
+++ b/products/signals/backend/scout_harness/tools/report_author.py
@@ -88,7 +88,20 @@ class ScoutRunReportAuthor:
"""Shared dimensions for the report-channel lifecycle events, mirroring the `signals_scout_run_*`
events so the two join on `run_id` / `task_run_id` — a report event sits under the run that authored
it. All fields are plain columns on the bridge row (no FK query)."""
+ metadata = self.run.metadata or {}
return {
+ **{
+ key: metadata[key]
+ for key in (
+ "experiment_key",
+ "experiment_variant",
+ "$feature/self-driving-banger-scout",
+ "runtime",
+ "model",
+ "service_tier",
+ )
+ if key in metadata
+ },
"skill_name": self.run.skill_name,
"skill_version": self.run.skill_version,
"scout_config_id": str(self.run.scout_config_id) if self.run.scout_config_id else None,
diff --git a/products/signals/backend/test/test_first_report_notification.py b/products/signals/backend/test/test_first_report_notification.py
index 74641f48a9d0..28c67a78758d 100644
--- a/products/signals/backend/test/test_first_report_notification.py
+++ b/products/signals/backend/test/test_first_report_notification.py
@@ -32,7 +32,12 @@ from products.signals.backend.first_report_notification_tasks import (
dispatch_first_report_notification,
sweep_first_report_notifications,
)
-from products.signals.backend.models import SignalFirstReportNotification, SignalReport, SignalReportArtefact
+from products.signals.backend.models import (
+ SignalFirstReportNotification,
+ SignalReport,
+ SignalReportArtefact,
+ SignalTeamConfig,
+)
from products.signals.backend.report_access import may_read_reports
MODULE = "products.signals.backend.first_report_notification"
@@ -346,18 +351,35 @@ class TestFirstReportNotification(BaseTest):
self.send.assert_called_once()
assert self.notification().queued_at is not None
- def test_environment_activation_and_reports_keep_their_own_recipient(self) -> None:
+ @parameterized.expand([(False, True), (True, True), (True, False)])
+ def test_environment_activation_and_reports_keep_their_own_recipient(
+ self, first_investigation: bool, parent_access: bool
+ ) -> None:
child = Team.objects.create(organization=self.organization, parent_team=self.team, name="Staging")
other_user = User.objects.create_and_join(self.organization, "[email protected]", None)
register_first_report_notification(team=child, user=other_user)
- report = self.report(team=child)
- deliver_first_report_email(self.team.id)
- self.send.assert_not_called()
- deliver_first_report_email(child.id)
+ report_team = self.team if first_investigation else child
+ if first_investigation:
+ SignalTeamConfig.objects.update_or_create(
+ team=self.team, defaults={"onboarding": {"first_report_activation_team_id": child.id}}
+ )
+ report = self.report(team=report_team)
+ if not first_investigation:
+ deliver_first_report_email(self.team.id)
+ self.send.assert_not_called()
+ with patch(
+ f"{MODULE}.may_read_reports", side_effect=lambda *, user, team: parent_access or team.id == child.id
+ ):
+ deliver_first_report_email(child.id)
+ if not parent_access:
+ self.send.assert_not_called()
+ assert SignalFirstReportNotification.objects.for_team(child.id).get().cancelled_at is not None
+ return
+ self.send.assert_called_once()
sent = self.send.call_args.kwargs
assert sent["to"] == other_user.email
assert sent["distinct_id"] == str(other_user.distinct_id)
- assert sent["message_data"]["project_id"] == child.id
+ assert sent["message_data"]["project_id"] == report_team.id
assert sent["message_data"]["report_id"] == str(report.id)
assert self.notification().queued_at is None
diff --git a/products/signals/backend/test/test_first_report_scout.py b/products/signals/backend/test/test_first_report_scout.py
index 72e44e8823f7..02643e5d5086 100644
--- a/products/signals/backend/test/test_first_report_scout.py
+++ b/products/signals/backend/test/test_first_report_scout.py
@@ -1,16 +1,30 @@
+import json
from datetime import timedelta
from posthog.test.base import BaseTest
from unittest.mock import Mock, patch
+from django.test import SimpleTestCase
from django.utils import timezone
from parameterized import parameterized
from posthog.clickhouse.client.connection import Workload
+from posthog.models import Team, User
+from products.notifications.backend.facade.api import create_notification
from products.signals.backend.constants import FIRST_REPORT_SCOUT, FIRST_REPORT_SCOUT_TIMEOUT_SECONDS
-from products.signals.backend.first_report_scout import build_first_report_context, first_report_state
+from products.signals.backend.first_report_completion import complete_first_report_investigation
+from products.signals.backend.first_report_experiment import (
+ first_report_outcome_properties,
+ resolve_first_report_experiment,
+)
+from products.signals.backend.first_report_notification import register_first_report_notification
+from products.signals.backend.first_report_scout import (
+ build_first_report_context,
+ first_report_state,
+ set_first_report_state,
+)
from products.signals.backend.models import SignalReport, SignalScoutConfig, SignalScoutRun
from products.signals.backend.scout_harness.config_registry import live_scout_skill_names, register_missing_configs
from products.signals.backend.scout_harness.tools.profile import compute_project_profile
@@ -19,6 +33,63 @@ from products.tasks.backend.models import Task, TaskRun
from products.warehouse_sources.backend.facade.models import DataWarehouseTable
+class TestFirstReportExperiment(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("test-sol", "acp", "codex", "gpt-6.1-sol", "high", "priority"),
+ ("test-deepseek", "pi", None, "deepseek-ai/deepseek-v4.1-flash-1017-fast", None, None),
+ ]
+ )
+ def test_variant_selects_the_whole_runtime(
+ self, variant: str, runtime: str, adapter: str | None, model: str, effort: str | None, tier: str | None
+ ) -> None:
+ team = Mock(id=123, uuid="project-uuid")
+ with patch(
+ "products.signals.backend.first_report_experiment.posthoganalytics.get_feature_flag", return_value=variant
+ ) as flag:
+ assignment = resolve_first_report_experiment(team)
+ assert assignment is not None
+ assert assignment.variant == variant
+ agent_runtime = assignment.agent_runtime
+ assert agent_runtime is not None
+ assert (
+ agent_runtime.runtime,
+ agent_runtime.runtime_adapter,
+ agent_runtime.model,
+ agent_runtime.reasoning_effort,
+ agent_runtime.service_tier,
+ ) == (runtime, adapter, model, effort, tier)
+ assert flag.call_args.kwargs["groups"] == {"project": "project-uuid"}
+ assert flag.call_args.kwargs["send_feature_flag_events"] is False
+
+ def test_control_is_enrolled_without_a_scout(self) -> None:
+ with patch(
+ "products.signals.backend.first_report_experiment.posthoganalytics.get_feature_flag", return_value="control"
+ ):
+ assignment = resolve_first_report_experiment(Mock(id=123, uuid="project-uuid"))
+ assert assignment is not None
+ assert assignment.variant == "control"
+ assert not assignment.runs_scout
+ assert "runtime" not in assignment.properties()
+
+ @parameterized.expand([(False,), (None,), (True,), ("unknown",), ("deepseek-fast",)])
+ def test_unenrolled_or_invalid_variant_does_not_enroll(self, variant: object) -> None:
+ with patch(
+ "products.signals.backend.first_report_experiment.posthoganalytics.get_feature_flag", return_value=variant
+ ):
+ assert resolve_first_report_experiment(Mock(id=123, uuid="project-uuid")) is None
+
+ def test_flag_outage_does_not_enroll(self) -> None:
+ with (
+ patch(
+ "products.signals.backend.first_report_experiment.posthoganalytics.get_feature_flag",
+ side_effect=RuntimeError("unavailable"),
+ ),
+ patch("products.signals.backend.first_report_experiment.capture_exception"),
+ ):
+ assert resolve_first_report_experiment(Mock(id=123, uuid="project-uuid")) is None
+
+
class TestFirstReportScout(BaseTest):
def test_system_scout_never_enters_the_scheduled_fleet(self) -> None:
LLMSkill.objects.create(team=self.team, name=FIRST_REPORT_SCOUT, body="# First report", is_latest=True)
@@ -35,27 +106,44 @@ class TestFirstReportScout(BaseTest):
):
compute_project_profile(team=self.team, defer_surfaces=True)
- @parameterized.expand([(None, False, False), (0, False, False), (20, True, False)])
- def test_data_gate_includes_warehouse_rows_without_analytics_events(
- self, row_count: int | None, has_data: bool, unknown: bool
+ @parameterized.expand(
+ [
+ ("no_tables", [], False, False),
+ ("unsynced_table", [None], False, False),
+ ("empty_tables", [0, 0, 0, 0], False, False),
+ ("synced_rows", [0, 0, 0, 20], False, True),
+ ("environment_ingested_events", [], True, True),
+ ]
+ )
+ def test_data_gate_reads_recorded_metadata_without_querying_the_warehouse(
+ self, _name: str, row_counts: list[int | None], environment_ingested: bool, has_data: bool
) -> None:
- if row_count is not None:
+ for index, row_count in enumerate(row_counts):
DataWarehouseTable.objects.create(
team=self.team,
- name="orders",
+ name=f"orders_{index}",
format="Parquet",
- url_pattern="https://example.com/orders",
+ url_pattern=f"https://example.com/orders_{index}",
row_count=row_count,
)
+ Team.objects.create(
+ organization=self.organization,
+ parent_team=self.team,
+ name="Production",
+ ingested_event=environment_ingested,
+ )
with patch(
"products.signals.backend.first_report_scout.execute_hogql_query",
side_effect=lambda **kwargs: Mock(results=[[0, None]] if len(kwargs["query"].select) == 2 else []),
- ):
+ ) as query:
context = build_first_report_context(self.team, self.user)
assert context.has_data() is has_data
- assert context.data_is_unknown() is unknown
- if has_data:
- assert '"recorded_row_count": 20' in context.prompt_context()
+ assert not context.data_is_unknown()
+ assert all(
+ not call.kwargs["query"].select_from.table.chain[0].startswith("orders") for call in query.call_args_list
+ )
+ if row_counts:
+ assert f'"recorded_row_count": {json.dumps(row_counts[-1])}' in context.prompt_context()
def test_older_product_data_still_qualifies_for_an_investigation(self) -> None:
with patch(
@@ -67,6 +155,8 @@ class TestFirstReportScout(BaseTest):
assert context.product_data["session_replay_fragments"].rows == 0
assert context.product_data["session_replay_fragments"].has_rows is True
assert [call.kwargs["workload"] for call in query.call_args_list] == [
+ Workload.OFFLINE,
+ Workload.OFFLINE,
Workload.OFFLINE,
Workload.OFFLINE,
Workload.LOGS,
@@ -92,13 +182,41 @@ class TestFirstReportScout(BaseTest):
assert result["status"] == expected
assert result["task_run_id"] == str(task_run.id)
- def test_query_failure_is_unknown_instead_of_no_data(self) -> None:
+ @parameterized.expand([(True, "report_created"), (False, "insufficient_data")])
+ def test_experiment_measures_persisted_delivery_even_when_run_failed(self, delivered: bool, expected: str) -> None:
+ task = Task.objects.create(team=self.team, title="First investigation")
+ task_run = TaskRun.objects.create(team=self.team, task=task, status="failed")
+ report = (
+ SignalReport.objects.create(
+ team=self.team, status="ready", first_visible_at=task_run.created_at + timedelta(seconds=120)
+ )
+ if delivered
+ else None
+ )
+ run = SignalScoutRun.objects.create(
+ team=self.team,
+ task_run=task_run,
+ skill_name=FIRST_REPORT_SCOUT,
+ skill_version=1,
+ emitted_report_ids=[str(report.id)] if report else [],
+ metadata={"first_report_result": {"outcome": "insufficient_data"}},
+ )
+ properties = first_report_outcome_properties(self.team.id, str(run.id))
+ assert properties["first_report_outcome"] == expected
+ assert properties["report_delivered"] is delivered
+ assert properties["report_delivered_within_five_minutes"] is delivered
+ if delivered:
+ assert properties["first_report_latency_seconds"] == 120
+
+ @parameterized.expand([(False,), (True,)])
+ def test_query_failure_is_unknown_instead_of_no_data(self, ingested_event: bool) -> None:
+ Team.objects.filter(pk=self.team.pk).update(ingested_event=ingested_event)
with patch(
"products.signals.backend.first_report_scout.execute_hogql_query",
side_effect=RuntimeError("query unavailable"),
):
context = build_first_report_context(self.team, self.user)
- assert not context.has_data()
+ assert context.has_data() is ingested_event
assert context.data_is_unknown()
@parameterized.expand(
@@ -109,15 +227,26 @@ class TestFirstReportScout(BaseTest):
("report_created", "suppressed", "incomplete"),
("report_created", "ready", "report_created"),
("report_created", "pending_input", "report_created"),
+ ("report_created", "ready", "report_created", True),
]
)
def test_onboarding_success_requires_a_visible_persisted_report(
- self, outcome: str, report_status: str | None, expected: str
+ self, outcome: str, report_status: str | None, expected: str, child_environment: bool = False
) -> None:
- task = Task.objects.create(team=self.team, title="First investigation")
+ activator = User.objects.create_and_join(self.organization, "[email protected]", None)
+ activation_team = (
+ Team.objects.create(organization=self.organization, parent_team=self.team, name="Child")
+ if child_environment
+ else self.team
+ )
+ register_first_report_notification(team=activation_team, user=activator)
+ set_first_report_state(self.team.id, "pending", activation_team_id=activation_team.id)
+ set_first_report_state(self.team.id, "pending", activation_team_id=self.team.id)
+ set_first_report_state(self.team.id, "running")
+ task = Task.objects.create(team=self.team, title="First investigation", created_by=self.user)
task_run = TaskRun.objects.create(team=self.team, task=task, status="completed")
report = SignalReport.objects.create(team=self.team, status=report_status) if report_status else None
- SignalScoutRun.objects.create(
+ run = SignalScoutRun.objects.create(
team=self.team,
task_run=task_run,
skill_name=FIRST_REPORT_SCOUT,
@@ -138,3 +267,20 @@ class TestFirstReportScout(BaseTest):
if expected == "report_created":
assert report is not None
assert result["report_id"] == str(report.id)
+ notifications: list[object] = []
+ with (
+ patch("posthoganalytics.feature_enabled", return_value=True),
+ patch("products.signals.backend.first_report_completion.may_read_reports", return_value=True),
+ patch(
+ "products.signals.backend.first_report_completion.create_notification",
+ side_effect=lambda data: notifications.append(create_notification(data)),
+ ) as notify,
+ ):
+ complete_first_report_investigation(self.team.id, str(run.id))
+ complete_first_report_investigation(self.team.id, str(run.id))
+ assert notifications[0] is not None
+ assert notifications[0] == notifications[1]
+ notification = notify.call_args.args[0]
+ assert notification.target_id == str(activator.pk)
+ assert notification.team_id == activation_team.id
+ assert notification.source_url == f"/project/{self.team.id}/inbox"
diff --git a/products/signals/backend/test/test_scout_harness.py b/products/signals/backend/test/test_scout_harness.py
index 63ddf2317c1d..df7cbcc7e693 100644
--- a/products/signals/backend/test/test_scout_harness.py
+++ b/products/signals/backend/test/test_scout_harness.py
@@ -2824,6 +2824,77 @@ async def test_run_pins_sandbox_to_resolved_scout_model(
assert props.get("service_tier") == expected_service_tier
+ "variant,runtime,adapter,model,tier",
+ [
+ ("test-sol", "acp", "codex", "gpt-6.1-sol", "priority"),
+ ("test-deepseek", "pi", None, "deepseek-ai/deepseek-v4.1-flash-1017-fast", None),
+ ("control", None, None, None, None),
+ (False, None, None, None, None),
+ ],
+)
+async def test_first_report_experiment_routes_and_records_assignment(ateam, variant, runtime, adapter, model, tier):
+ await database_sync_to_async(LLMSkill.objects.create)(
+ team=ateam, name="signals-scout-first-report", description="First report", body="Investigate"
+ )
+ session, _ = await database_sync_to_async(_make_fake_session)(ateam)
+ result = scout_prompt.FirstReportScoutRunSummary(
+ summary="No actionable finding", first_report_result={"outcome": "no_actionable_finding"}
+ )
+ captured = {}
+
+ async def start(*args, before_task_dispatch=None, **kwargs):
+ captured.update(kwargs)
+ await database_sync_to_async(before_task_dispatch)(session.task_run.id)
+ return session, result
+
+ with (
+ patch("products.signals.backend.scout_harness.runner.MultiTurnSession.start", new=start),
+ patch(
+ "products.signals.backend.first_report_experiment.posthoganalytics.get_feature_flag",
+ side_effect=lambda key, *args, **kwargs: variant if key == "self-driving-banger-scout" else None,
+ ),
+ patch(
+ "products.signals.backend.scout_harness.runner.get_or_create_signals_sandbox_env", return_value="env-id"
+ ) as sandbox,
+ patch("products.signals.backend.temporal.agentic.resolve_acting_user_id_for_team", return_value=42),
+ patch("products.signals.backend.scout_harness.runner.posthoganalytics.capture") as capture,
+ ):
+ outcome = await arun_signals_scout(
+ team_id=ateam.id, skill_name="signals-scout-first-report", triggered_by="onboarding"
+ )
+
+ if runtime is None:
+ assert outcome.skip_reason == (
+ "first-report control arm" if variant == "control" else "first-report experiment disabled"
+ )
+ sandbox.assert_not_called()
+ assert not captured
+ return
+ assert outcome.status == "completed"
+ context = captured["context"]
+ assert captured["internal"] is True
+ assert captured["origin_product"] == "signals_scout"
+ assert (context.runtime, context.runtime_adapter, context.model, context.service_tier) == (
+ runtime,
+ adapter,
+ model,
+ tier,
+ )
+ run = await database_sync_to_async(SignalScoutRun.objects.get)(team=ateam)
+ assert run.metadata is not None
+ assert run.metadata["experiment_variant"] == variant
+ assert run.metadata["runtime"] == runtime
+ events = {call.kwargs["event"]: call.kwargs["properties"] for call in capture.call_args_list}
+ assert events["$experiment_exposure"]["$feature_flag_response"] == variant
+ assert events["signals_scout_run_started"]["experiment_variant"] == variant
+ assert events["signals_scout_run_finished"]["experiment_variant"] == variant
+ assert events["signals_scout_run_finished"]["first_report_outcome"] == "no_actionable_finding"
+ assert events["signals_scout_run_finished"]["report_delivered"] is False
+
+
@pytest.mark.asyncio
@pytest.mark.django_db
async def test_failed_run_returns_failed_outcome_and_skips_bridge_insert(ateam, aerrors_skill):
diff --git a/products/signals/backend/test/test_scout_harness_api.py b/products/signals/backend/test/test_scout_harness_api.py
index 035ac7f6794d..b71d06c7d25b 100644
--- a/products/signals/backend/test/test_scout_harness_api.py
+++ b/products/signals/backend/test/test_scout_harness_api.py
@@ -211,11 +211,19 @@ class TestScoutHarnessRunsAPI(APIBaseTest):
team=self.team, skill_name="signals-scout-first-report", enabled=False
)
run = _make_run(self.team, skill_name=config.skill_name, scout_config=config)
+ run.task_run.task.internal = True
+ run.task_run.task.save(update_fields=["internal"])
ordinary = _make_run(self.team)
response = self.client.get(self._list_url())
assert response.status_code == 200
assert {row["run_id"] for row in response.json()} == {str(run.id), str(ordinary.id)}
assert self.client.get(self._detail_url(str(run.id))).status_code == 200
+ assert (
+ self.client.get(
+ f"/api/projects/{self.team.id}/tasks/{run.task_run.task_id}/runs/{run.task_run_id}/"
+ ).status_code
+ == 200
+ )
response = self.client.get(self._list_url() + "recent-per-scout/")
assert {row["run_id"] for row in response.json()} == {str(run.id), str(ordinary.id)}
response = self.client.post(self._list_url() + "emissions/batch/", {"run_ids": [str(run.id)]}, format="json")
diff --git a/products/tasks/backend/temporal/process_task/ai_gateway_token.py b/products/tasks/backend/temporal/process_task/ai_gateway_token.py
index f5acf831f4f5..a9fc08b8cd70 100644
--- a/products/tasks/backend/temporal/process_task/ai_gateway_token.py
+++ b/products/tasks/backend/temporal/process_task/ai_gateway_token.py
@@ -21,6 +21,7 @@ from prometheus_client import Counter
from posthog.llm.gateway_client import GatewayNotConfiguredError
from posthog.security.outbound_proxy import internal_requests
+from products.tasks.backend.exceptions import GatewayTokenUnavailableError, ProcessTaskTransientError
from products.tasks.backend.logic.services.desktop_gateway_token import (
POSTHOG_CODE_PRODUCT,
PRODUCT_CREDIT_BUCKET,
@@ -249,8 +250,7 @@ def mint_refusal(
refusal = _posthog_code_refusal(team_id, model)
if refusal:
return refusal
- # Pi reaches Go only for posthog_code.
- elif runtime == "pi":
+ elif runtime == "pi" and not (ai_product == "signals_scout" and origin_product == "signals_scout" and internal):
return "pi_runtime"
# The gateway denies an off-pin model with no fallback.
if not model_allowed_by_product_pin(ai_product, model):
@@ -359,6 +359,7 @@ def mint_scoped_token(
capture_mode: Literal["none"] | None = None,
expires_in_seconds: int | None = None,
gateway_config: "AIGatewayConfig | None" = None,
+ raise_on_failure: bool = False,
) -> str | None:
"""Mint a `phe_` scoped token pinned to (ai_product, obo=team_id), or None on failure.
@@ -367,6 +368,7 @@ def mint_scoped_token(
`allowed_models` narrows the product's pin (a free Desktop plan).
Retries mint rate limits (429) and transient upstream errors with jittered
backoff. Private callers require an acknowledged capture pin and fail closed on None.
+ `raise_on_failure` preserves retryability for callers that cannot use a fallback gateway.
"""
gateway_url = gateway_config.url if gateway_config is not None else settings.SANDBOX_AI_GATEWAY_URL
base_url = (gateway_url or "").rstrip("/").removesuffix("/v1")
@@ -393,6 +395,7 @@ def mint_scoped_token(
if capture_mode is not None:
body["capture_mode"] = capture_mode
last_error: str = ""
+ retryable = True
for attempt in range(_MINT_ATTEMPTS):
try:
response = post(
@@ -417,6 +420,7 @@ def mint_scoped_token(
# Older gateways ignore unknown fields; never use their unprotected token.
revoke_scoped_token(token, gateway_config=gateway_config)
last_error = "mint response did not acknowledge capture suppression"
+ retryable = False
break
AI_GATEWAY_TOKEN_MINTS.labels(result="ok").inc()
return token
@@ -424,6 +428,7 @@ def mint_scoped_token(
last_error = f"HTTP {response.status_code}"
else:
# 4xx other than 429 will not improve on retry (bad credential, bad body).
+ retryable = False
last_error = (
f"HTTP {response.status_code}"
if capture_mode is not None
@@ -443,4 +448,14 @@ def mint_scoped_token(
last_error,
extra={"ai_product": ai_product, "team_id": team_id, "error": last_error},
)
+ if raise_on_failure:
+ context = {"team_id": team_id, "reason": "mint_failed"}
+ if retryable:
+ raise ProcessTaskTransientError(
+ "The AI gateway could not mint a scoped token. Try again in a few minutes.",
+ context,
+ RuntimeError("mint_failed"),
+ capture=False,
+ )
+ raise GatewayTokenUnavailableError("The AI gateway rejected the scoped token request.", context)
return None
diff --git a/products/tasks/backend/temporal/process_task/tests/test_ai_gateway_token.py b/products/tasks/backend/temporal/process_task/tests/test_ai_gateway_token.py
index 8de3f542c31f..6da27c3f8fd9 100644
--- a/products/tasks/backend/temporal/process_task/tests/test_ai_gateway_token.py
+++ b/products/tasks/backend/temporal/process_task/tests/test_ai_gateway_token.py
@@ -620,7 +620,7 @@ class TestProvisioningBoundaries:
"task_runtime,tokens,initializes_cost,incomplete",
[
("acp", ("phe_abc",), True, False),
- ("pi", ("phe_abc",), False, True),
+ ("pi", ("phe_abc",), True, False),
("acp", (None,), False, True),
("acp", ("phe_abc", None, "phe_abc"), True, True),
("acp", (None, "phe_abc"), True, True),
@@ -643,9 +643,7 @@ class TestProvisioningBoundaries:
with patch.object(utils, "mint_scoped_token", side_effect=tokens):
for token in tokens:
env = utils.run_gateway_env_vars(ctx, self._task())
- assert env.get("AI_GATEWAY_TOKEN") == (
- token if isinstance(token, str) and task_runtime != "pi" else None
- )
+ assert env.get("AI_GATEWAY_TOKEN") == (token if isinstance(token, str) else None)
test_task_run.refresh_from_db()
assert ("token_cost" in test_task_run.state) is initializes_cost
assert ("unprocessed_request_ids" in test_task_run.state) is initializes_cost
@@ -885,6 +883,56 @@ class TestProvisioningBoundaries:
):
assert utils.run_gateway_env_vars(self._ctx(), self._task()) == {}
+ @pytest.mark.parametrize(
+ "failure,error_type",
+ [
+ (503, ProcessTaskTransientError),
+ (429, ProcessTaskTransientError),
+ (requests.Timeout("unavailable"), ProcessTaskTransientError),
+ ("pin_write", ProcessTaskTransientError),
+ (401, GatewayTokenUnavailableError),
+ ("untrusted", GatewayTokenUnavailableError),
+ ("not_routed", GatewayTokenUnavailableError),
+ ("no_mint_key", GatewayTokenUnavailableError),
+ ],
+ )
+ def test_pi_scout_does_not_fall_back_when_gateway_mint_fails(self, mint_settings, failure, error_type):
+ ctx = self._ctx()
+ ctx.task_runtime = "pi"
+ task = self._task()
+ task.internal = failure != "untrusted"
+ ctx.model = "deepseek-ai/deepseek-v4.1-flash-1017-fast"
+ if failure == "not_routed":
+ mint_settings.SANDBOX_AI_GATEWAY_PRODUCTS = "signals_research"
+ elif failure == "no_mint_key":
+ mint_settings.SANDBOX_AI_GATEWAY_MINT_KEY = ""
+ with (
+ patch("products.tasks.backend.temporal.process_task.ai_gateway_token.requests.post") as post,
+ patch("products.tasks.backend.temporal.process_task.ai_gateway_token.time.sleep"),
+ patch("products.tasks.backend.models.TaskRun.update_state_atomic") as update,
+ patch.dict(utils.PRODUCT_ALLOWED_MODELS, {"signals_scout": [ctx.model]}),
+ patch.object(utils, "record_gateway_routing"),
+ ):
+ post.return_value.status_code = failure if isinstance(failure, int) else 201
+ post.return_value.json.return_value = {"token": "phe_scout"}
+ if isinstance(failure, Exception):
+ post.side_effect = failure
+ if failure == "pin_write":
+ update.side_effect = OperationalError("unavailable")
+ ctx.state["ai_gateway_product"] = "slack_app"
+ with pytest.raises(error_type) as raised:
+ utils.run_gateway_env_vars(ctx, task)
+ assert raised.value.non_retryable is (error_type is GatewayTokenUnavailableError)
+ if error_type is ProcessTaskTransientError:
+ post.side_effect = None
+ post.return_value.status_code = 201
+ update.side_effect = None
+ assert utils.run_gateway_env_vars(ctx, task)["AI_GATEWAY_TOKEN"] == "phe_scout"
+ elif failure == 401:
+ assert post.call_count == 1
+ else:
+ post.assert_not_called()
+
@pytest.mark.parametrize("uses_gateway", [False, True])
def test_routing_is_not_returned_when_accounting_write_fails(self, uses_gateway: bool) -> None:
with (
@@ -1330,6 +1378,24 @@ class TestMintRefusalScope:
def test_pi_runs_of_other_products_never_mint(self, product):
assert mint_refusal(product, team_id=2, state=None, model=None, runtime="pi") == "pi_runtime"
+ @pytest.mark.parametrize(
+ "internal,origin,expected",
+ [(True, "signals_scout", None), (False, "signals_scout", "pi_runtime"), (True, "loop", "pi_runtime")],
+ )
+ def test_pi_scout_requires_server_provenance(self, internal, origin, expected):
+ assert (
+ mint_refusal(
+ "signals_scout",
+ team_id=2,
+ state=None,
+ model="deepseek-ai/deepseek-v4.1-flash-1017-fast",
+ runtime="pi",
+ internal=internal,
+ origin_product=origin,
+ )
+ == expected
+ )
+
@pytest.mark.parametrize("product", ["posthog_ai", "workflows", "review_hog"])
def test_pinned_products_refuse_an_off_pin_model(self, product):
refusal = mint_refusal(product, team_id=2, state=None, model="claude-opus-3", runtime="acp")
diff --git a/products/tasks/backend/temporal/process_task/utils.py b/products/tasks/backend/temporal/process_task/utils.py
index 4efdf1f3906e..9a8c6748dc8f 100644
--- a/products/tasks/backend/temporal/process_task/utils.py
+++ b/products/tasks/backend/temporal/process_task/utils.py
@@ -1545,12 +1545,15 @@ def run_gateway_env_vars(ctx: TaskProcessingContext, task: Task) -> dict[str, st
"AI_GATEWAY_AI_STAGE": (ctx.state or {}).get("ai_stage") or "scout",
}
if "own-subscription" in (ctx.claude_model_access, ctx.codex_model_access):
+ if ctx.task_runtime == "pi" and ctx.origin_product == "signals_scout":
+ raise GatewayNotConfiguredError("Pi scouts require the AI gateway")
record_gateway_routing(run_id=ctx.run_id, team_id=ctx.team_id, uses_gateway=False)
return {}
ai_stage = (ctx.state or {}).get("ai_stage")
requires_go = resolve_sandbox_ai_product(ctx.origin_product, ai_stage, internal=task.internal) == (
POSTHOG_CODE_PRODUCT
)
+ requires_pi_scout = ctx.task_runtime == "pi" and ctx.origin_product == "signals_scout"
try:
env_vars = ai_gateway_env_vars(
team_id=ctx.team_id,
@@ -1567,6 +1570,13 @@ def run_gateway_env_vars(ctx: TaskProcessingContext, task: Task) -> dict[str, st
if not _record_pinned_gateway_product(ctx.run_id, ctx.state, pinned):
if requires_go:
raise posthog_code_token_unavailable("pin_unrecorded", ctx.team_id)
+ if requires_pi_scout:
+ raise ProcessTaskTransientError(
+ "Could not record the Pi scout's gateway model pin.",
+ {"team_id": ctx.team_id, "run_id": ctx.run_id},
+ RuntimeError("pin_unrecorded"),
+ capture=False,
+ )
# The model-change guard reads that stamp; unstamped, a run can move off its pin with no fallback.
for key in _TOKEN_ENV_KEYS:
env_vars.pop(key, None)
@@ -1575,6 +1585,13 @@ def run_gateway_env_vars(ctx: TaskProcessingContext, task: Task) -> dict[str, st
except Exception as e:
if requires_go:
raise posthog_code_token_unavailable("mint_failed", ctx.team_id) from e
+ if requires_pi_scout:
+ raise ProcessTaskTransientError(
+ "The Pi scout's AI gateway is temporarily unavailable.",
+ {"team_id": ctx.team_id, "run_id": ctx.run_id},
+ e,
+ capture=False,
+ ) from e
# Degrading to the Python gateway beats failing the provisioning activity and the run.
AI_GATEWAY_TOKEN_MINTS.labels(result="error").inc()
logger.warning(
@@ -1583,6 +1600,10 @@ def run_gateway_env_vars(ctx: TaskProcessingContext, task: Task) -> dict[str, st
exc_info=True,
)
env_vars = {}
+ if ctx.task_runtime == "pi" and ctx.origin_product == "signals_scout" and not env_vars.get("AI_GATEWAY_TOKEN"):
+ raise GatewayTokenUnavailableError(
+ "Pi scouts require a scoped AI gateway token", {"team_id": ctx.team_id, "run_id": ctx.run_id}
+ )
# Retry provisioning if coverage cannot be recorded; otherwise fallback usage can look fully accounted for.
record_gateway_routing(
run_id=ctx.run_id,
@@ -1668,6 +1689,17 @@ def ai_gateway_env_vars(
resolve_sandbox_ai_product(origin_product, ai_stage, internal=internal) if team_id is not None else None
)
requires_go = ai_product == POSTHOG_CODE_PRODUCT
+ requires_pi_scout = runtime == "pi" and origin_product == "signals_scout"
+ if requires_pi_scout and not (
+ ai_product == "signals_scout"
+ and settings.SANDBOX_AI_GATEWAY_URL
+ and settings.SANDBOX_AI_GATEWAY_MINT_KEY
+ and settings.SANDBOX_AI_GATEWAY_PRODUCTS
+ and sandbox_product_routed(ai_product, ai_stage, settings.SANDBOX_AI_GATEWAY_PRODUCTS)
+ ):
+ raise GatewayTokenUnavailableError(
+ "Pi scouts require configured scoped AI gateway routing.", {"team_id": team_id}
+ )
if not (settings.SANDBOX_AI_GATEWAY_URL and settings.SANDBOX_AI_GATEWAY_PRODUCTS):
if requires_go:
raise posthog_code_token_unavailable("not_configured", team_id)
@@ -1713,6 +1745,10 @@ def ai_gateway_env_vars(
)
if requires_go:
raise posthog_code_token_unavailable(refusal, team_id)
+ if requires_pi_scout:
+ raise GatewayTokenUnavailableError(
+ "The AI gateway refused the Pi scout token.", {"team_id": team_id, "reason": refusal}
+ )
# The legacy gateway refuses an exhausted bucket with a retryable 429. A background
# workflow run has nobody to read the refusal, so the agent retries until the idle
# timeout and the run fails as lost. Fail it here with the real reason.
@@ -1722,6 +1758,8 @@ def ai_gateway_env_vars(
)
return env_vars
mint_kwargs: dict[str, Any] = {}
+ if requires_pi_scout:
+ mint_kwargs["raise_on_failure"] = True
if ai_product == POSTHOG_CODE_PRODUCT:
free_pin = posthog_code_allowed_models(team_id)
if free_pin is not None:
diff --git a/products/tasks/backend/tests/test_api.py b/products/tasks/backend/tests/test_api.py
index 5c80b8bd9451..e60da18054e7 100644
--- a/products/tasks/backend/tests/test_api.py
+++ b/products/tasks/backend/tests/test_api.py
@@ -405,6 +405,22 @@ class TestBuiltInAgentTaskAccess(BaseTaskAPITest):
class TestScoutTrialTaskVisibility(BaseTaskAPITest):
+ def test_customer_can_follow_the_first_report_investigation(self) -> None:
+ task = Task.objects.create(
+ team=self.team,
+ created_by=self.user,
+ title="First investigation",
+ origin_product=Task.OriginProduct.SIGNALS_SCOUT,
+ origin_key="first-report",
+ )
+ run = TaskRun.objects.create(team=self.team, task=task)
+ base = f"/api/projects/{self.team.id}/tasks"
+ assert self.client.get(f"{base}/{task.id}/").status_code == 200
+ assert self.client.get(f"{base}/{task.id}/runs/{run.id}/").status_code == 200
+ response = self.client.get(f"{base}/")
+ assert response.status_code == 200
+ assert str(task.id) in {row["id"] for row in response.json()["results"]}
+
def setUp(self) -> None:
super().setUp()
self.ordinary_task = self.create_task()