:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(tasks): allow assessment runs to report runtime results (#128829)
Mikayla Thompson9b37bb18791a
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.mdindex 82bc50f185b6..1d9f4b5d7fee 100644--- a/docs/internal/signals-pr-lifecycle.md+++ b/docs/internal/signals-pr-lifecycle.md@@ -154,6 +154,8 @@ The Monitoring schema includes both outcome and tracker-delivery metadata. NullaThe check coordinator dispatches one report-level assessment workflow per requested attempt. The worker uses the existing agent task machinery with read-only tools and saved final results. Each technical execution has a unique task origin key; collectors reconnect to its first run and reuse saved structured output. Up to three technical executions are allowed, each bounded to 30 minutes. A valid assessment ends the judgment loop even when it cannot establish success. Collector failures also have bounded retries and never launch a competing judgment while an earlier task may still run.+Assessment MCP sessions remain read-only. Their separate runtime credential carries server-only `internal_run:write` for the registered sandbox's own run: log uploads, run summaries, schema-validated structured output, and restricted lifecycle updates. It cannot edit task configuration, create child tasks, cancel runs, or write inbox state. Summary tags stay on the run instead of changing task tags. The assessment consumer retains the current-attempt and evidence guards when deciding the report outcome.+Assessment dispatch requires organization AI consent and uses at most a quarter of the coordinator's remaining budget, with a two-second connection/RPC cap per dispatch. Assessment tokens grant reads and their own run lifecycle only. The acting user's metric permissions filter the private task prompt. Tasks use that user's personal channel, which restricts commands and traces to that user. Assessment notes containing agent reasoning also require that user and their current source-data permissions, because additional evidence may come from beyond the original checks. Public task descriptions contain no metric evidence.Protected assessment notes are excluded from shared Today figure-mark sources.diff --git a/products/tasks/backend/facade/api.py b/products/tasks/backend/facade/api.pyindex 722f4fac0f2e..6e7746ec1111 100644--- a/products/tasks/backend/facade/api.py+++ b/products/tasks/backend/facade/api.py@@ -3943,6 +3943,7 @@ def set_task_run_summary(*,summary: str,tags: list[str] | None = None,+ sync_task_tags: bool = True,include_agent_state: bool = False,user_id: int | None = None,analytics_context_reader: Callable[[object], bool] | None = None,@@ -3956,7 +3957,7 @@ def set_task_run_summary(updates[TASK_RUN_TAGS_STATE_KEY] = list(dict.fromkeys(tags))with transaction.atomic():run.state = TaskRun.update_state_atomic(run.id, updates=updates)- if tags is not None and not run.task.is_scout_experiment:+ if sync_task_tags and tags is not None and not run.task.is_scout_experiment:replace_tags_on_object(tags, run.task)run.refresh_from_db()run.publish_stream_state_event()diff --git a/products/tasks/backend/presentation/views/api.py b/products/tasks/backend/presentation/views/api.pyindex 39e2a6f802d2..4b634256b471 100644--- a/products/tasks/backend/presentation/views/api.py+++ b/products/tasks/backend/presentation/views/api.py@@ -2014,7 +2014,7 @@ class TaskRunViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):self._authorized_task_id = self._ensure_task_accessible()@staticmethod- def _is_trial_lifecycle_update(payload: object) -> bool:+ def _is_runtime_lifecycle_update(payload: object) -> bool:if (not isinstance(payload, dict)or not payload@@ -2046,6 +2046,7 @@ class TaskRunViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):if self.action not in {"append_log","set_summary",+ "set_output","update","partial_update","artifacts_download",@@ -2053,13 +2054,16 @@ class TaskRunViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):}:return Nonescopes = get_authenticator_scopes(request.successful_authenticator) or []+ runtime_scope = "internal_run:write" if "internal_run:write" in scopes else "scout_experiment_internal:read"if ("task:read" in scopes- and "scout_experiment_internal:read" in scopes+ and runtime_scope in scopesand (self.action in {"append_log", "set_summary"}+ or self.action == "set_output"+ and runtime_scope == "internal_run:write"or self.action in {"update", "partial_update"}- and self._is_trial_lifecycle_update(request.data)+ and self._is_runtime_lifecycle_update(request.data))and run_context.is_sandbox_run_request(request=request,@@ -2068,7 +2072,7 @@ class TaskRunViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):run_id=self.kwargs.get("pk"),)):- return ["task:read", "scout_experiment_internal:read"]+ return ["task:read", runtime_scope]if self.action == "artifacts_download":return ["task:read"]if self.action == "retrieve":@@ -2515,7 +2519,6 @@ class TaskRunViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):detail=True,methods=["patch"],url_path="set_output",- required_scopes=["task:write"],)def set_output(self, request, pk=None, **kwargs):task_id = self._ensure_task_accessible()@@ -2550,12 +2553,14 @@ class TaskRunViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):)def set_summary(self, request, pk=None, **kwargs):task_id = self._ensure_task_accessible()+ scopes = get_authenticator_scopes(request.successful_authenticator) or []run = tasks_facade.set_task_run_summary(pk,task_id,self.team_id,summary=request.validated_data["summary"],tags=request.validated_data.get("tags"),+ sync_task_tags="internal_run:write" not in scopes or "task:write" in scopes,include_agent_state=run_context.is_sandbox_run_request(request=self.request,team_id=self.team_id,diff --git a/products/tasks/backend/temporal/oauth.py b/products/tasks/backend/temporal/oauth.pyindex 8e8257694239..69a008a589be 100644--- a/products/tasks/backend/temporal/oauth.py+++ b/products/tasks/backend/temporal/oauth.py@@ -229,8 +229,12 @@ def create_oauth_access_token(token_options["include_interactive_run_scope"] = Trueif task.origin_product == Task.OriginProduct.SLACK:token_options["include_slack_run_scope"] = True- if run_uses_own_subscription(run_state):- token_options["withhold_scopes"] = sorted(SUBSCRIPTION_RUN_WITHHELD_SCOPES)+ withheld = set(SUBSCRIPTION_RUN_WITHHELD_SCOPES) if run_uses_own_subscription(run_state) else set()+ if "internal_run:write" in resolve_scopes(scopes):+ # Flat scopes regain shared internal grants when resolved again during minting.+ withheld.update(RESEARCH_WITHHELD_SCOPES)+ if withheld:+ token_options["withhold_scopes"] = sorted(withheld)if task.origin_product == Task.OriginProduct.SIGNALS_SCOUT and run_id is not None:from products.signals.backend.facade.monitoring import ( # noqa: PLC0415 -- avoids loading Signals during Tasks startupmonitoring_scout_can_submit,@@ -273,6 +277,9 @@ def create_oauth_access_token_for_run(# `start_agent_server` mints its token with explicit scopes.if scopes == "signals_research":scopes = [*resolve_scopes(scopes), *RESEARCH_WITHHELD_SCOPES]+ elif scopes == "signals_assessment":+ # Runtime uploads are bound to this run; the separately minted MCP token stays read-only.+ scopes = [*resolve_scopes(scopes), "internal_run:write"]with transaction.atomic():locked_task = (Task.objects.select_for_update(of=("self",))diff --git a/products/tasks/backend/temporal/tests/test_oauth.py b/products/tasks/backend/temporal/tests/test_oauth.pyindex 5aca9d253ee5..34dc51e4e8b4 100644--- a/products/tasks/backend/temporal/tests/test_oauth.py+++ b/products/tasks/backend/temporal/tests/test_oauth.py@@ -540,6 +540,33 @@ def test_workflow_run_scopes_never_exceed_request_or_snapshot(assert granted <= set(resolve_scopes(snapshot, include_internal_scopes=True))[email protected]("requested", [None, "signals_assessment"])+def test_assessment_runtime_token_cannot_regain_general_writes(+ test_task: Task, requested: PosthogMcpScopes | None+) -> None:+ test_task.origin_product = Task.OriginProduct.SIGNAL_REPORT+ test_task.internal = True+ test_task.save(update_fields=["origin_product", "internal"])+ run = TaskRun.objects.create(+ task=test_task,+ team=test_task.team,+ state={"pending_dispatch": {"posthog_mcp_scopes": "signals_assessment"}},+ )++ raw = create_oauth_access_token_for_run(test_task, run.state, scopes=requested, run_id=run.id)++ token = OAuthAccessToken.objects.get(token=raw)+ scopes = set(token.scope.split())+ assert not scopes & RESEARCH_WITHHELD_SCOPES+ assert ("internal_run:write" in scopes) is (requested is None)+ assert "task:read" in scopes+ assert "query:read" in scopes+ assert token.sandbox_task_id == test_task.id+ run.refresh_from_db()+ assert str(token.id) in run.state["sandbox_oauth_token_ids"]++@pytest.mark.django_db@pytest.mark.parametrize(("origin_product", "state", "requested", "granted"),@@ -587,6 +614,18 @@ def test_workflow_run_scopes_never_exceed_request_or_snapshot("signals_research","signals_research",),+ (+ Task.OriginProduct.SIGNAL_REPORT,+ {"pending_dispatch": {"posthog_mcp_scopes": "signals_assessment"}},+ None,+ [*resolve_scopes("signals_assessment"), "internal_run:write"],+ ),+ (+ Task.OriginProduct.SIGNAL_REPORT,+ {"pending_dispatch": {"posthog_mcp_scopes": "signals_assessment"}},+ "signals_assessment",+ "signals_assessment",+ ),],)@patch("products.tasks.backend.temporal.oauth._create_oauth_access_token_for_user", return_value="token")diff --git a/products/tasks/backend/tests/test_api.py b/products/tasks/backend/tests/test_api.pyindex 4ab6baddc7ea..5f8bd70d2664 100644--- a/products/tasks/backend/tests/test_api.py+++ b/products/tasks/backend/tests/test_api.py@@ -507,9 +507,14 @@ class TestScoutTrialTaskVisibility(BaseTaskAPITest):scopes=scopes,)- def test_trial_can_append_own_log_without_general_task_write_access(self) -> None:+ @parameterized.expand([("trial", "scout_experiment_internal:read"), ("assessment", "internal_run:write")])+ def test_trial_can_append_own_log_without_general_task_write_access(self, _name: str, runtime_scope: str) -> None:task, run = self.trial_tasks[0], self.trial_runs[0]- client = self._trial_log_client()+ if runtime_scope == "internal_run:write":+ task.origin_product = Task.OriginProduct.SIGNAL_REPORT+ task.origin_key = "monitoring-assessment:synthetic"+ task.save(update_fields=["origin_product", "origin_key"])+ client = self._trial_log_client(scopes=f"task:read internal_run:read {runtime_scope}")base = f"/api/projects/@current/tasks/{task.id}/runs/{run.id}/"entry = {"type": "info", "message": "Synthetic scout inspected recent activity"}log_objects: dict[str, str] = {}@@ -617,6 +622,19 @@ class TestScoutTrialTaskVisibility(BaseTaskAPITest):assert run.state == original_stateassert run.status == original_status+ runtime_client = self._trial_log_client(+ client_id=ARRAY_APP_CLIENT_ID_US, scopes="task:read internal_run:read internal_run:write"+ )+ run.refresh_from_db()+ original_state = run.state+ response = runtime_client.patch(+ f"/api/projects/@current/tasks/{task.id}/runs/{run.id}/", payload, format="json"+ )+ assert response.status_code == status.HTTP_403_FORBIDDEN+ run.refresh_from_db()+ assert run.state == original_state+ assert run.status == original_status+@parameterized.expand([("unbound",),@@ -625,17 +643,28 @@ class TestScoutTrialTaskVisibility(BaseTaskAPITest):("unregistered_token",),("foreign_client",),("gateway_only",),+ ("runtime_unbound",),+ ("runtime_sibling_task",),+ ("runtime_other_run",),+ ("runtime_unregistered_token",),+ ("runtime_foreign_client",),+ ("runtime_gateway_only",),+ ("runtime_mcp_read_only",),])def test_trial_log_append_requires_exact_trusted_run(self, case: str) -> None:+ runtime_scope = "internal_run:write" if case.startswith("runtime_") else "scout_experiment_internal:read"+ case = case.removeprefix("runtime_")task, run = self.trial_tasks[0], self.trial_runs[0]client = self._trial_log_client(bound=case != "unbound",client_id="synthetic-foreign-client" if case == "foreign_client" else ARRAY_APP_CLIENT_ID_DEV,scopes=(- "llm_gateway:read internal_run:read scout_experiment_internal:read"+ "task:read internal_run:read signal_scout_internal:read"+ if case == "mcp_read_only"+ else f"llm_gateway:read internal_run:read {runtime_scope}"if case == "gateway_only"- else "task:read internal_run:read scout_experiment_internal:read"+ else f"task:read internal_run:read {runtime_scope}"),)if case == "sibling_task":@@ -661,6 +690,12 @@ class TestScoutTrialTaskVisibility(BaseTaskAPITest):format="json",)assert response.status_code == status.HTTP_403_FORBIDDEN+ response = client.patch(+ f"/api/projects/@current/tasks/{task.id}/runs/{run.id}/set_output/",+ {"output": {"resolved": True, "reasoning": "Untrusted result", "evidence_refs": []}},+ format="json",+ )+ assert response.status_code == status.HTTP_403_FORBIDDENrun.refresh_from_db()assert run.state == original_stateassert run.status == original_status@@ -8760,6 +8795,26 @@ class TestTaskRunAPI(BaseTaskAPITest):run.refresh_from_db()self.assertEqual(run.output, {"summary": "Final result"})+ runtime_client = self._sandbox_oauth_client(task.id, scopes="task:read internal_run:read internal_run:write")+ mock_signal_workflow_completion.reset_mock()+ response = runtime_client.patch(+ f"/api/projects/@current/tasks/{task.id}/runs/{run.id}/set_output/",+ {"output": {"summary": "Assessed result"}},+ format="json",+ )+ assert response.status_code == status.HTTP_200_OK, response.json()+ assert mock_signal_workflow_completion.call_count == expected_signals+ run.refresh_from_db()+ assert run.output == {"summary": "Assessed result"}+ response = runtime_client.patch(+ f"/api/projects/@current/tasks/{task.id}/runs/{run.id}/set_output/",+ {"output": {"unvalidated": True}},+ format="json",+ )+ assert response.status_code == status.HTTP_400_BAD_REQUEST+ run.refresh_from_db()+ assert run.output == {"summary": "Assessed result"}+@patch("products.tasks.backend.models.TaskRun.publish_stream_state_event")def test_set_output_publishes_stream_state_event(self, mock_publish_stream_state_event):task = self.create_task()