Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(signals): show final agent verification status (#126597)

Mikayla Thompson87bdf7097d1b
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.md
index 66287a8e30d6..82bc50f185b6 100644
--- a/docs/internal/signals-pr-lifecycle.md
+++ b/docs/internal/signals-pr-lifecycle.md
@@ -118,7 +118,7 @@ During Monitoring, checks with an explicit Monitoring definition sample at their
Each completed attempt appends a protected `check_observation` log entry with its exact bounds, soak deadline, display format, unit, effective comparison, value, and match result, or a sanitized query error. The log labels it as an interim measurement, shows whether the goal is on track, and displays time remaining when the measurement was taken. It does not assign a terminal outcome. Observations do not spend `runs_remaining`, change the report status, or emit recurrence signals. The next sample is due one interval after the cutoff; missed intervals do not create catch-up runs. Restoring Monitoring schedules from the latest attempt in that period, preserving a failed attempt's error streak and six-hour retry across suppression. Re-enabling the flag restores the measurement origin, the next attempt from the latest run, and the expiry from the original Monitoring start. This repair precedes the expiry sweep so a fallback expiry cannot shorten or extend the Monitoring horizon. A check that cannot fit its source queries into the remaining coordinator budget stays due without an observation or an error-budget change. Query failures retain the existing six-hour retry and consecutive-error limit, and the 90-day check horizon still applies. Writes reject a changed check, Monitoring period, report state, or flag, and duplicate workers cannot save the same attempt twice. Expiry sweeps and schedule repairs retain their check-row concurrency guards after database lock waits, so they cannot overwrite a concurrent cancellation, expiry extension, or saved observation.
-With Monitoring disabled, observations disappear from the artefact API and counts. No Resolved executor runs. The coordinator also leaves disabled or unavailable Monitoring periods dormant. Observation values require the same query permissions as terminal check results. The inbox follows all artefact pages within the current project before deriving task runs, reviewers, and activity so observations cannot displace older report state. Polling waits for an in-flight paginated load to finish. With the Monitoring flag enabled, scouts see metric checks with an explicit definition as scheduled or due inside their Monitoring window; agent checks and metric checks without a definition remain paused in this presentation. The soak deadline bounds interim execution. Final metric results are saved separately without reopening reports or creating follow-up reports.
+With Monitoring disabled, observations disappear from the artefact API and counts. No Resolved executor runs. The coordinator also leaves disabled or unavailable Monitoring periods dormant. Observation values require the same query permissions as terminal check results. The inbox follows all artefact pages within the current project before deriving task runs, reviewers, and activity so observations cannot displace older report state. Polling waits for an in-flight paginated load to finish. With the Monitoring flag enabled, scouts see metric checks with an explicit definition as scheduled or due inside their Monitoring window; metric checks without a definition remain paused in this presentation. Agent checks in the current frozen attempt show Monitoring before their window ends, Awaiting verification after the deadline, and Verifying while a live final scout runs for the current attempt and check definition. Checks added or revised after the snapshot, and checks in child environments, stay paused because the final executor cannot run them in this attempt. Settled insufficient or unsupported evidence and exhausted technical retries show Inconclusive and require assessment; accepted sufficient evidence shows Verification complete until aggregation publishes the check verdict. Only checks still eligible for another final evaluation show Awaiting verification. The API preserves the existing timing state values: these four final verification phases return `state: "paused"` with additive, nullable `final_verification_state` metadata (`monitoring`, `verifying`, `assessment_required`, or `verification_complete`). Older consumers show the coarse Paused fallback. Updated consumers prefer the metadata when present and otherwise use `state`; legacy `window_ended` remains unchanged. Queued runs, dispatch backoff, stale runs, and submitted evaluations do not imply active verification. The soak deadline bounds interim execution. Final metric results are saved separately without reopening reports or creating follow-up reports.
Disabling Monitoring stops execution without starting a post-resolution schedule or recalculating a recurring deadline. Re-enabling restores the original Monitoring horizon for active checks and leaves terminal checks unchanged. Pending checks arm only during enabled Monitoring, using the original implementation-complete timestamp. Queued legacy check-triggered scout activities skip before starting agent work, and already-running legacy scouts cannot record a result through the check-result tool. Scout check lists mark these checks paused and do not advertise them as waiting on a run.
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index d93c52edefea..7bc747ac5130 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -12375,13 +12375,13 @@ snapshots:
scenes-app-inbox-detail-follow-up-checks--every-state--light:
hash: v1.k794b7964.275081e63f24959c0017f4a3f295444edd91eb89f9cf9876ab4365dc29c8107c.FaarzsWlYt7_qiYo7KGl1dukS7877MEEhxlH1ZiiyzU
scenes-app-inbox-detail-follow-up-checks--monitoring--dark:
- hash: v1.k794b7964.8203910c8643bee68c2895b77db7101544290e636a6cecaf6a7fd27d6e0fe87b.RurNhxFb2VgJ6Y5LHmClwOH4ihVHG6V9Rivg06oEqlE
+ hash: v1.k794b7964.1722a8ff2683badff5f73dffe8d72ca72498b99f30b3ce2c3b96cdfda54c2f62.nOyRFp04viufCvb6BhelyBFTgM52ST-7SE4OJF28FHk
scenes-app-inbox-detail-follow-up-checks--monitoring--light:
- hash: v1.k794b7964.be376e8a1d9201dc66aa9ce7bb3433f6095bf9bf9062558aa9318bd94aff95b4.KMp3_kcYPglF3avwTCjwqU2Fa30Bq-Dku0iqNdgLKDA
+ hash: v1.k794b7964.8343f2f1850a6015001ead05597540dc01c529669bb3f3b5389537999a7d3b8b.jEGDikY_1oEBlO78uaIfvssMDaP-hzdS6VNGcIFmDkM
scenes-app-inbox-detail-follow-up-checks--monitoring-narrow-rail--dark:
- hash: v1.k794b7964.18b179f7ea5b615c1a0bdb2e400033a286a36b4f988b5533f506b07e738851d9.1z0G9ypKvN7BaB_DpHuNAUnoQFF5yz8iij-EcG0wQcs
+ hash: v1.k794b7964.12195f8cfae43825ceda8239e1bf21fc1ec9083bd0e9a4a200e7210fb1ed5493.2s2RfKCd6NBTF6x_qzA-7FfHVj-7GXVFR8v-VgnXgvw
scenes-app-inbox-detail-follow-up-checks--monitoring-narrow-rail--light:
- hash: v1.k794b7964.9baf07cf5b989f8f22c8fadba9d915297b026d3eee1128dbf28478cc88e0b98c.g2-gNbvDMXhe7Wkkt9JfjDNK3N3S3kyqfaLdwM96-98
+ hash: v1.k794b7964.d0e101969add84590b5aa9e08821de3d492f0442f0f29db4cab1b643d9348e8d.ORm3zfwZyTAVLMCkytQYprD_B_3uKAiXvZPDLX1Q62A
scenes-app-inbox-detail-follow-up-checks--narrow-rail--dark:
hash: v1.k794b7964.be9f8ab7b553ea303e03e4b375a4156c486fad10ce4b66936fb4fbe923ad0bdd.m4ThsrMsAizyTDwlW6uvlP5yjhc0ZjL96P66-MNn85M
scenes-app-inbox-detail-follow-up-checks--narrow-rail--light:
diff --git a/products/signals/backend/report_check_monitoring_timing.py b/products/signals/backend/report_check_monitoring_timing.py
index e99f5a5b3596..51aa48155fa2 100644
--- a/products/signals/backend/report_check_monitoring_timing.py
+++ b/products/signals/backend/report_check_monitoring_timing.py
@@ -1,19 +1,32 @@
from __future__ import annotations
-from datetime import datetime
+from collections import defaultdict
+from datetime import datetime, timedelta
from typing import TYPE_CHECKING
+from uuid import UUID
from django.db.models import TextChoices
from posthog.dataclasses import frozen
+from products.signals.backend.models import SignalReportMonitoringResult, SignalScoutRun
+from products.signals.backend.report_check_final_agents import MonitoringScoutCheck, _monitoring_binding_for_run
from products.signals.backend.report_checks import MAX_CHECK_HORIZON, monitoring_window_end
+from products.signals.backend.report_monitoring_aggregation import MonitoringMemberState
+from products.signals.backend.report_monitoring_attempts import check_definition_version
+from products.signals.backend.report_monitoring_outcomes import _member_results, _member_state
+from products.signals.backend.report_monitoring_results import InvalidMonitoringResult
+from products.signals.backend.scout_harness.limits import STALE_RUN_CUTOFF_S, TRIGGERED_BY_MONITORING
if TYPE_CHECKING:
- from products.signals.backend.models import SignalReportCheck
+ from products.signals.backend.models import SignalReport, SignalReportCheck, SignalReportMonitoringAttempt
class MonitoringCheckTimingState(TextChoices):
+ MONITORING = "monitoring", "Monitoring"
+ VERIFYING = "verifying", "Verifying"
+ ASSESSMENT_REQUIRED = "assessment_required", "Assessment required"
+ VERIFICATION_COMPLETE = "verification_complete", "Verification complete"
SCHEDULED = "scheduled", "Scheduled"
DUE = "due", "Due"
WINDOW_ENDED = "window_ended", "Window ended"
@@ -25,6 +38,12 @@ class MonitoringCheckTimingState(TextChoices):
EXPIRY_REACHED = "expiry_reached", "Expiry reached"
+@frozen
+class MonitoringCheckVerification:
+ run: MonitoringScoutCheck | None = None
+ results: tuple[SignalReportMonitoringResult, ...] = ()
+
+
@frozen
class MonitoringCheckTiming:
state: MonitoringCheckTimingState
@@ -33,7 +52,106 @@ class MonitoringCheckTiming:
next_measurement_at: datetime | None = None
@classmethod
- def for_check(cls, check: SignalReportCheck, *, now: datetime) -> MonitoringCheckTiming | None:
+ def final_verifications(
+ cls, report: SignalReport, *, now: datetime, attempt: SignalReportMonitoringAttempt
+ ) -> dict[UUID, MonitoringCheckVerification]:
+ submitted = list(
+ SignalReportMonitoringResult.objects.for_team(report.team_id)
+ .filter(attempt_id=attempt.id)
+ .only(
+ "report_check_id",
+ "submission_id",
+ "definition_version",
+ "window_start",
+ "window_end",
+ "evaluated_at",
+ "evidence_status",
+ "outcome",
+ "expectation_matches",
+ )
+ )
+ submitted_ids = {result.submission_id for result in submitted}
+ results_by_check: defaultdict[UUID, list[SignalReportMonitoringResult]] = defaultdict(list)
+ for result in submitted:
+ results_by_check[result.report_check_id].append(result)
+ runs = (
+ SignalScoutRun.objects.for_team(report.team_id)
+ .only("team_id", "metadata")
+ .filter(
+ metadata__triggered_by=TRIGGERED_BY_MONITORING,
+ metadata__run_note__icontains=str(report.id),
+ task_run__status="in_progress",
+ task_run__team_id=report.team_id,
+ created_at__gt=now - timedelta(seconds=STALE_RUN_CUTOFF_S),
+ )
+ )
+ running: dict[UUID, MonitoringScoutCheck] = {}
+ for run in runs:
+ try:
+ binding = _monitoring_binding_for_run(run)
+ except InvalidMonitoringResult:
+ continue
+ if (
+ binding.report_id == report.id
+ and binding.attempt_id == attempt.id
+ and binding.window_start == attempt.monitoring_started_at
+ and attempt.check_versions.get(str(binding.check_id)) == binding.definition_version
+ and binding.submission_id not in submitted_ids
+ ):
+ running[binding.check_id] = binding
+ return {
+ check_id: MonitoringCheckVerification(run=running.get(check_id), results=tuple(results_by_check[check_id]))
+ for check_id in running.keys() | results_by_check.keys()
+ }
+
+ @staticmethod
+ def _agent_state(
+ check: SignalReportCheck,
+ *,
+ now: datetime,
+ start: datetime,
+ end: datetime,
+ attempt: SignalReportMonitoringAttempt | None,
+ verification: MonitoringCheckVerification | None,
+ ) -> MonitoringCheckTimingState:
+ version = check_definition_version(check)
+ if (
+ attempt is None
+ or attempt.report_id != check.report_id
+ or attempt.monitoring_started_at != start
+ or attempt.check_versions.get(str(check.id)) != version
+ or check.report.team.parent_team_id is not None
+ ):
+ return MonitoringCheckTimingState.PAUSED
+ if now < end:
+ return MonitoringCheckTimingState.MONITORING
+ if verification is None:
+ return MonitoringCheckTimingState.WINDOW_ENDED
+ state = _member_state(_member_results(check, version, attempt, list(verification.results), now))
+ if state in {MonitoringMemberState.INCONCLUSIVE, MonitoringMemberState.EXHAUSTED}:
+ return MonitoringCheckTimingState.ASSESSMENT_REQUIRED
+ if state in {MonitoringMemberState.PASSED, MonitoringMemberState.FAILED}:
+ return MonitoringCheckTimingState.VERIFICATION_COMPLETE
+ run = verification.run
+ if (
+ run is not None
+ and run.check_id == check.id
+ and run.window_start == start
+ and run.window_end == end
+ and run.definition_version == version
+ ):
+ return MonitoringCheckTimingState.VERIFYING
+ return MonitoringCheckTimingState.WINDOW_ENDED
+
+ @classmethod
+ def for_check(
+ cls,
+ check: SignalReportCheck,
+ *,
+ now: datetime,
+ attempt: SignalReportMonitoringAttempt | None = None,
+ verification: MonitoringCheckVerification | None = None,
+ ) -> MonitoringCheckTiming | None:
report = check.report
archived = (
report.status == report.Status.SUPPRESSED and report.status_before_suppression == report.Status.MONITORING
@@ -51,6 +169,8 @@ class MonitoringCheckTiming:
state = MonitoringCheckTimingState.ARCHIVED
elif start is None or end is None:
state = MonitoringCheckTimingState.UNAVAILABLE
+ elif check.kind == check.Kind.AGENT:
+ state = cls._agent_state(check, now=now, start=start, end=end, attempt=attempt, verification=verification)
elif check.expires_at <= now or now >= start + MAX_CHECK_HORIZON:
state = MonitoringCheckTimingState.EXPIRY_REACHED
elif check.kind != check.Kind.METRIC_THRESHOLD or not check.config.get("monitoring"):
diff --git a/products/signals/backend/serializers.py b/products/signals/backend/serializers.py
index 8fa6e057f862..93d853ec5f79 100644
--- a/products/signals/backend/serializers.py
+++ b/products/signals/backend/serializers.py
@@ -48,7 +48,11 @@ from products.signals.backend.enums import (
SignalSourceType,
SuggestedSourceProduct,
)
-from products.signals.backend.report_check_monitoring_timing import MonitoringCheckTiming, MonitoringCheckTimingState
+from products.signals.backend.report_check_monitoring_timing import (
+ MonitoringCheckTiming,
+ MonitoringCheckTimingState as MonitoringCheckTimingPhase,
+ MonitoringCheckVerification,
+)
from products.signals.backend.report_checks import (
CHECK_CONFIG_SCHEMAS,
DEFAULT_CHECK_EXPIRY_AFTER_LAST_RUN,
@@ -64,6 +68,7 @@ from products.signals.backend.report_checks import (
parse_check_config,
)
from products.signals.backend.report_content_gates import team_report_monitoring_enabled
+from products.signals.backend.report_monitoring_attempts import current_monitoring_attempt
from products.signals.backend.reviewer_rules import ReviewerRules
from products.signals.backend.scout_harness.tools.report import is_self_improvement_title
from products.warehouse_sources.backend.facade.models import ExternalDataSchema
@@ -96,6 +101,7 @@ from .models import (
SignalReportArtefact,
SignalReportAssignment,
SignalReportCheck,
+ SignalReportMonitoringAttempt,
SignalReportPullRequest,
SignalReportRefund,
SignalReportSuppressionSource,
@@ -2873,10 +2879,35 @@ def redact_check_config(config: Mapping[str, object], policy: ReportMetricAccess
return redacted
+class MonitoringCheckTimingState(TextChoices):
+ SCHEDULED = "scheduled", "Scheduled"
+ DUE = "due", "Due"
+ WINDOW_ENDED = "window_ended", "Window ended"
+ NO_INTERIM_MEASUREMENTS = "no_interim_measurements", "No interim measurements"
+ WAITING = "waiting", "Waiting for scheduling"
+ PAUSED = "paused", "Paused during Monitoring"
+ ARCHIVED = "archived", "Archived"
+ UNAVAILABLE = "unavailable", "Timing unavailable"
+ EXPIRY_REACHED = "expiry_reached", "Expiry reached"
+
+
+class FinalVerificationState(TextChoices):
+ MONITORING = "monitoring", "Monitoring"
+ VERIFYING = "verifying", "Verifying"
+ ASSESSMENT_REQUIRED = "assessment_required", "Assessment required"
+ VERIFICATION_COMPLETE = "verification_complete", "Verification complete"
+
+
class MonitoringCheckTimingSerializer(serializers.Serializer):
state = serializers.ChoiceField(
choices=MonitoringCheckTimingState.choices,
- help_text="Whether this check has an interim measurement due, is paused, or has finished its observation window. No state implies a final verdict.",
+ help_text="Monitoring timing state. Final verification phases use paused for compatibility; read final_verification_state for their phase.",
+ )
+ final_verification_state = serializers.ChoiceField(
+ choices=FinalVerificationState.choices,
+ allow_null=True,
+ read_only=True,
+ help_text="Final verification phase when state is paused. Null for other timing states. Read the check outcome for its verdict.",
)
started_at = serializers.DateTimeField(
allow_null=True, help_text="Original Monitoring start for this check's report."
@@ -2890,6 +2921,24 @@ class MonitoringCheckTimingSerializer(serializers.Serializer):
help_text="Stored next interim attempt, only when it falls before the soak deadline and expiry. Null when no interim measurement is scheduled.",
)
+ def to_representation(self, instance: MonitoringCheckTiming) -> dict[str, object]:
+ data = super().to_representation(instance)
+ final_state = instance.state if instance.state in FinalVerificationState.values else None
+ data["state"] = MonitoringCheckTimingState.PAUSED if final_state else instance.state
+ data["final_verification_state"] = final_state
+ return data
+
+
+class MonitoringCheckTimingSerializerExtension(OpenApiSerializerExtension):
+ target_class = "products.signals.backend.serializers.MonitoringCheckTimingSerializer"
+
+ def map_serializer(self, auto_schema: "AutoSchema", direction: "Direction") -> dict[str, object]:
+ schema: dict[str, object] = auto_schema._map_serializer(self.target, direction, bypass_extensions=True)
+ required = schema.get("required")
+ if isinstance(required, list):
+ schema["required"] = [name for name in required if name != "final_verification_state"]
+ return schema
+
class SignalReportCheckSerializerExtension(OpenApiSerializerExtension):
target_class = "products.signals.backend.serializers.SignalReportCheckSerializer"
@@ -2919,12 +2968,40 @@ class SignalReportCheckSerializer(serializers.ModelSerializer):
enabled_by_team[instance.team_id] = team_report_monitoring_enabled(instance.team_id)
return enabled_by_team[instance.team_id]
+ def _agent_timing(self, instance: SignalReportCheck, *, now: datetime) -> MonitoringCheckTiming | None:
+ attempts = cast(
+ dict[uuid.UUID, SignalReportMonitoringAttempt | None],
+ self.context.setdefault("report_check_monitoring_attempts", {}),
+ )
+ if instance.report_id not in attempts:
+ attempts[instance.report_id] = current_monitoring_attempt(instance.report)
+ attempt = attempts[instance.report_id]
+ timing = MonitoringCheckTiming.for_check(instance, now=now, attempt=attempt)
+ if timing is None or timing.state != MonitoringCheckTimingPhase.WINDOW_ENDED or attempt is None:
+ return timing
+ verifications_by_report = cast(
+ dict[uuid.UUID, dict[uuid.UUID, MonitoringCheckVerification]],
+ self.context.setdefault("report_check_final_verifications", {}),
+ )
+ if instance.report_id not in verifications_by_report:
+ verifications_by_report[instance.report_id] = MonitoringCheckTiming.final_verifications(
+ instance.report, now=now, attempt=attempt
+ )
+ return MonitoringCheckTiming.for_check(
+ instance,
+ now=now,
+ attempt=attempt,
+ verification=verifications_by_report[instance.report_id].get(instance.id),
+ )
+
@extend_schema_field(MonitoringCheckTimingSerializer(allow_null=True))
def get_monitoring_timing(self, instance: SignalReportCheck) -> dict[str, object] | None:
now = cast(datetime, self.context.setdefault("report_check_timing_now", timezone.now()))
timing = MonitoringCheckTiming.for_check(instance, now=now)
if timing is None or not self._monitoring_enabled(instance):
return None
+ if instance.kind == SignalReportCheck.Kind.AGENT and timing.state == MonitoringCheckTimingPhase.PAUSED:
+ timing = self._agent_timing(instance, now=now)
return dict(MonitoringCheckTimingSerializer(timing).data)
def to_representation(self, instance: SignalReportCheck) -> dict[str, object]:
diff --git a/products/signals/backend/test/test_report_check_final_agents.py b/products/signals/backend/test/test_report_check_final_agents.py
index efa7a7cc08cf..b149ac7099ee 100644
--- a/products/signals/backend/test/test_report_check_final_agents.py
+++ b/products/signals/backend/test/test_report_check_final_agents.py
@@ -47,8 +47,10 @@ from products.signals.backend.report_check_final_agents import (
from products.signals.backend.report_check_final_candidates import DueFinalCheck, collect_due_final_checks
from products.signals.backend.report_metric_refresh import MetricMeasurement
from products.signals.backend.report_monitoring_attempts import check_definition_version, complete_implementation
+from products.signals.backend.report_monitoring_outcomes import aggregate_monitoring_attempt
from products.signals.backend.scout_harness.limits import STALE_RUN_CUTOFF_S, TRIGGERED_BY_MONITORING
from products.signals.backend.scout_harness.prompt import SignalScoutRunSummary
+from products.signals.backend.serializers import SignalReportCheckSerializer
from products.signals.backend.temporal.agentic.scout_scheduler import (
RunSignalsScoutInput,
RunSignalsScoutOutput,
@@ -168,6 +170,112 @@ class FinalAuthoredAgentChecksSetup(PostHogTestCase):
class TestFinalAuthoredAgentChecks(FinalAuthoredAgentChecksSetup, BaseTest):
+ @parameterized.expand(
+ [
+ ("running", "in_progress", "verifying"),
+ ("queued", "queued", "window_ended"),
+ ("finished", "completed", "window_ended"),
+ ("stale_run", "in_progress", "window_ended"),
+ ("old_attempt", "in_progress", "window_ended"),
+ ("missing_attempt", "in_progress", "paused"),
+ ("outside_attempt", "in_progress", "paused"),
+ ("malformed_assignment", "in_progress", "window_ended"),
+ ("changed_definition", "in_progress", "paused"),
+ ("submitted", "in_progress", "assessment_required"),
+ ("retry_backoff", "completed", "window_ended"),
+ ("archived", "in_progress", "archived"),
+ ]
+ )
+ def test_check_timing_only_verifies_a_live_current_assignment(self, case: str, status: str, state: str) -> None:
+ run = self._run("Verifying the assigned check.", status=status)
+ if case == "stale_run":
+ SignalScoutRun.objects.for_team(self.team.id).filter(id=run.id).update(
+ created_at=self.now - timedelta(seconds=STALE_RUN_CUTOFF_S)
+ )
+ elif case == "old_attempt":
+ run.metadata = {
+ "triggered_by": TRIGGERED_BY_MONITORING,
+ "run_note": build_monitoring_run_note(replace(self.binding, attempt_id=uuid4()), self.check),
+ }
+ run.save(update_fields=["metadata"])
+ elif case == "changed_definition":
+ self.check.title = "A revised expectation"
+ self.check.save(update_fields=["title"])
+ elif case == "missing_attempt":
+ self.attempt.delete()
+ elif case == "outside_attempt":
+ self.check.id = uuid4()
+ self.check.save(force_insert=True)
+ elif case == "malformed_assignment":
+ run.metadata = {
+ "triggered_by": TRIGGERED_BY_MONITORING,
+ "run_note": f"Invalid assignment for report {self.report.id}",
+ }
+ run.save(update_fields=["metadata"])
+ elif case == "submitted":
+ submit_monitoring_scout_result(
+ run, MonitoringScoutResult.model_validate_json(self._payload("insufficient"))
+ )
+ elif case == "retry_backoff":
+ self.check.monitoring_dispatch_after = self.now + timedelta(minutes=5)
+ self.check.save(update_fields=["monitoring_dispatch_after"])
+ elif case == "archived":
+ self.report.status = SignalReport.Status.SUPPRESSED
+ self.report.status_before_suppression = SignalReport.Status.MONITORING
+ self.check.report = self.report
+ self.report.team = self.team
+ context = {"report_monitoring_enabled_by_team": {self.team.id: True}, "report_check_timing_now": self.now}
+
+ expected_queries = {"archived": 0, "missing_attempt": 1, "outside_attempt": 1, "changed_definition": 1}.get(
+ case, 4
+ )
+ with self.assertNumQueries(expected_queries):
+ data = SignalReportCheckSerializer([self.check, self.check], many=True, context=context).data
+
+ final_state = state if state in {"verifying", "assessment_required"} else None
+ assert [row["monitoring_timing"]["state"] for row in data] == ["paused" if final_state else state] * 2
+ assert [row["monitoring_timing"]["final_verification_state"] for row in data] == [final_state] * 2
+ assert self.check.dispatched_at is None
+
+ @parameterized.expand(
+ [
+ ("insufficient", None, 1, "assessment_required"),
+ ("unsupported", None, 1, "assessment_required"),
+ ("errored", None, 1, "window_ended"),
+ ("errored", None, 2, "window_ended"),
+ ("errored", None, 3, "assessment_required"),
+ ("sufficient", True, 1, "verification_complete"),
+ ("sufficient", False, 1, "verification_complete"),
+ ]
+ )
+ def test_saved_evaluations_replace_pending_verification(
+ self, evidence: str, matches: bool | None, evaluations: int, state: str
+ ) -> None:
+ for _ in range(evaluations):
+ self.binding = replace(self.binding, submission_id=uuid4())
+ self.note = build_monitoring_run_note(self.binding, self.check)
+ run = self._run("Evaluating the assigned check.", status="in_progress")
+ submit_monitoring_scout_result(
+ run, MonitoringScoutResult.model_validate_json(self._payload(evidence, matches))
+ )
+ if evidence != "sufficient":
+ aggregate_monitoring_attempt(team_id=self.team.id, report_id=self.report.id, attempt_id=self.attempt.id)
+ self.report.refresh_from_db()
+ self.check.refresh_from_db()
+ self.report.team = self.team
+ self.check.report = self.report
+ assert self.report.status == SignalReport.Status.MONITORING
+ assert self.check.status in SignalReportCheck.OPEN_STATUSES
+
+ data = SignalReportCheckSerializer(
+ self.check,
+ context={"report_monitoring_enabled_by_team": {self.team.id: True}, "report_check_timing_now": self.now},
+ ).data
+
+ final_state = state if state in {"assessment_required", "verification_complete"} else None
+ assert data["monitoring_timing"]["state"] == ("paused" if final_state else state)
+ assert data["monitoring_timing"]["final_verification_state"] == final_state
+
def test_due_selection_uses_original_membership_soak_and_version(self) -> None:
assert self.binding.window_start == self.start
assert self.binding.window_end == self.start + timedelta(hours=1)
diff --git a/products/signals/backend/test/test_report_check_monitoring_timing.py b/products/signals/backend/test/test_report_check_monitoring_timing.py
index d2cf61bc4840..005662b5e7f9 100644
--- a/products/signals/backend/test/test_report_check_monitoring_timing.py
+++ b/products/signals/backend/test/test_report_check_monitoring_timing.py
@@ -4,9 +4,12 @@ from django.test import SimpleTestCase
from parameterized import parameterized
-from products.signals.backend.models import SignalReport, SignalReportCheck
+from posthog.models import Team
+
+from products.signals.backend.models import SignalReport, SignalReportCheck, SignalReportMonitoringAttempt
from products.signals.backend.report_check_monitoring_timing import MonitoringCheckTiming
-from products.signals.backend.serializers import SignalReportCheckSerializer
+from products.signals.backend.report_monitoring_attempts import check_definition_version
+from products.signals.backend.serializers import MonitoringCheckTimingSerializer, SignalReportCheckSerializer
class TestMonitoringCheckTiming(SimpleTestCase):
@@ -42,6 +45,7 @@ class TestMonitoringCheckTiming(SimpleTestCase):
if enabled:
assert data["monitoring_timing"] == {
"state": "scheduled",
+ "final_verification_state": None,
"started_at": "2026-10-20T00:00:00Z",
"ends_at": "2026-10-21T00:00:00Z",
"next_measurement_at": "2026-10-20T06:00:00Z",
@@ -79,26 +83,46 @@ class TestMonitoringCheckTiming(SimpleTestCase):
assert timing is not None
assert timing.state == state
+ data = MonitoringCheckTimingSerializer(timing).data
+ assert data["state"] == state
+ assert data["final_verification_state"] is None
assert timing.ends_at == start + timedelta(days=1)
assert timing.next_measurement_at == (check.next_run_at if has_next else None)
@parameterized.expand(
[
- ("agent", "agent", "active", 1440, True, True, "paused"),
+ ("agent", "agent", "active", 1440, True, True, "monitoring"),
("legacy_metric", "metric_threshold", "active", 1440, False, True, "paused"),
("pending", "metric_threshold", "pending", 1440, True, True, "waiting"),
("missing_start", "metric_threshold", "active", 1440, True, False, "unavailable"),
("missing_soak", "metric_threshold", "active", None, True, True, "unavailable"),
("zero_soak", "metric_threshold", "active", 0, True, True, "window_ended"),
+ ("agent_exact_deadline", "agent", "active", 360, False, True, "window_ended"),
+ ("agent_past_deadline", "agent", "active", 60, False, True, "window_ended"),
+ ("agent_zero_soak", "agent", "active", 0, False, True, "window_ended"),
+ ("child_before_deadline", "agent", "active", 1440, True, True, "paused", 2),
+ ("child_after_deadline", "agent", "active", 60, True, True, "paused", 2),
("terminal", "metric_threshold", "errored", 1440, True, True, None),
]
)
def test_only_armed_monitoring_metrics_offer_measurements(
- self, _name: str, kind: str, status: str, soak: int | None, definition: bool, has_start: bool, state: str | None
+ self,
+ _name: str,
+ kind: str,
+ status: str,
+ soak: int | None,
+ definition: bool,
+ has_start: bool,
+ state: str | None,
+ parent_team_id: int | None = None,
) -> None:
now = datetime(2026, 10, 20, tzinfo=UTC)
start = now - timedelta(hours=6)
- report = SignalReport(team_id=1, status="monitoring", monitoring_started_at=start if has_start else None)
+ report = SignalReport(
+ team=Team(id=1, parent_team_id=parent_team_id),
+ status="monitoring",
+ monitoring_started_at=start if has_start else None,
+ )
check = SignalReportCheck(
team_id=1,
report=report,
@@ -111,10 +135,20 @@ class TestMonitoringCheckTiming(SimpleTestCase):
expires_at=now + timedelta(days=1),
)
- timing = MonitoringCheckTiming.for_check(check, now=now)
+ attempt = SignalReportMonitoringAttempt(
+ team_id=1,
+ report=report,
+ monitoring_started_at=start,
+ check_versions={str(check.id): check_definition_version(check)},
+ )
+ timing = MonitoringCheckTiming.for_check(check, now=now, attempt=attempt)
assert (timing.state if timing else None) == state
assert timing is None or timing.next_measurement_at is None
+ if timing is not None:
+ data = MonitoringCheckTimingSerializer(timing).data
+ assert data["state"] == ("paused" if state == "monitoring" else state)
+ assert data["final_verification_state"] == ("monitoring" if state == "monitoring" else None)
@parameterized.expand(
[
diff --git a/products/signals/backend/views.py b/products/signals/backend/views.py
index 1be8a26fa4ab..b6710ce86d20 100644
--- a/products/signals/backend/views.py
+++ b/products/signals/backend/views.py
@@ -6327,7 +6327,7 @@ class SignalReportCheckViewSet(
return report
def safely_get_queryset(self, queryset):
- return queryset.filter(report_id=self._validated_report().id, team=self.team).select_related("report")
+ return queryset.filter(report_id=self._validated_report().id, team=self.team).select_related("report__team")
def destroy(self, request: Request, *args, **kwargs) -> Response:
check = cast(SignalReportCheck, self.get_object())
diff --git a/products/signals/frontend/generated/api.schemas.ts b/products/signals/frontend/generated/api.schemas.ts
index 6b5618d68ccf..994d5721f428 100644
--- a/products/signals/frontend/generated/api.schemas.ts
+++ b/products/signals/frontend/generated/api.schemas.ts
@@ -6144,8 +6144,24 @@ export const MonitoringCheckTimingStateEnumApi = {
ExpiryReached: 'expiry_reached',
} as const
+/**
+ * * `monitoring` - Monitoring
+ * * `verifying` - Verifying
+ * * `assessment_required` - Assessment required
+ * * `verification_complete` - Verification complete
+ */
+export type FinalVerificationStateEnumApi =
+ (typeof FinalVerificationStateEnumApi)[keyof typeof FinalVerificationStateEnumApi]
+
+export const FinalVerificationStateEnumApi = {
+ Monitoring: 'monitoring',
+ Verifying: 'verifying',
+ AssessmentRequired: 'assessment_required',
+ VerificationComplete: 'verification_complete',
+} as const
+
export interface MonitoringCheckTimingApi {
- /** Whether this check has an interim measurement due, is paused, or has finished its observation window. No state implies a final verdict.
+ /** Monitoring timing state. Final verification phases use paused for compatibility; read final_verification_state for their phase.
*
* * `scheduled` - Scheduled
* * `due` - Due
@@ -6157,6 +6173,13 @@ export interface MonitoringCheckTimingApi {
* * `unavailable` - Timing unavailable
* * `expiry_reached` - Expiry reached */
state: MonitoringCheckTimingStateEnumApi
+ /** Final verification phase when state is paused. Null for other timing states. Read the check outcome for its verdict.
+ *
+ * * `monitoring` - Monitoring
+ * * `verifying` - Verifying
+ * * `assessment_required` - Assessment required
+ * * `verification_complete` - Verification complete */
+ readonly final_verification_state?: FinalVerificationStateEnumApi | null
/**
* Original Monitoring start for this check's report.
* @nullable
diff --git a/products/signals/frontend/inbox/components/detail/ReportCheckRow.stories.tsx b/products/signals/frontend/inbox/components/detail/ReportCheckRow.stories.tsx
index a0a7de45ac00..f4b36e8f921c 100644
--- a/products/signals/frontend/inbox/components/detail/ReportCheckRow.stories.tsx
+++ b/products/signals/frontend/inbox/components/detail/ReportCheckRow.stories.tsx
@@ -174,17 +174,31 @@ export const Monitoring: Story = {
},
}),
check({
- id: 'monitoring-paused',
+ id: 'monitoring-agent',
title: 'No new reports of the export timeout',
soak_minutes: 1440,
next_run_at: '2026-09-22T00:00:00Z',
monitoring_timing: {
state: 'paused',
+ final_verification_state: 'monitoring',
started_at: '2026-09-21T00:00:00Z',
ends_at: '2026-09-22T00:00:00Z',
next_measurement_at: null,
},
}),
+ check({
+ id: 'verifying-agent',
+ title: 'Checkout errors stay at zero after the retry fix',
+ soak_minutes: 1440,
+ next_run_at: '2026-09-21T00:00:00Z',
+ monitoring_timing: {
+ state: 'paused',
+ final_verification_state: 'verifying',
+ started_at: '2026-09-20T00:00:00Z',
+ ends_at: '2026-09-21T00:00:00Z',
+ next_measurement_at: null,
+ },
+ }),
],
},
}
diff --git a/products/signals/frontend/inbox/components/detail/ReportCheckTiming.tsx b/products/signals/frontend/inbox/components/detail/ReportCheckTiming.tsx
index a00bee873bdc..bd00ec5a2445 100644
--- a/products/signals/frontend/inbox/components/detail/ReportCheckTiming.tsx
+++ b/products/signals/frontend/inbox/components/detail/ReportCheckTiming.tsx
@@ -4,6 +4,14 @@ import type { SignalReportCheckApi, SignalReportStatusEnumApi } from 'products/s
import { monitoringPeriodLabel } from './reportCheckPresentation'
+function observationWindowEndLabel(state: string): string {
+ return state === 'monitoring'
+ ? 'Final verification scheduled for: '
+ : ['window_ended', 'verifying', 'assessment_required', 'verification_complete'].includes(state)
+ ? 'Window ended: '
+ : 'Window end: '
+}
+
export function ReportCheckTiming({
check,
reportStatus,
@@ -14,6 +22,7 @@ export function ReportCheckTiming({
detail: string
}): JSX.Element {
const timing = reportStatus === 'monitoring' ? check.monitoring_timing : null
+ const state = timing?.final_verification_state ?? timing?.state ?? ''
return (
<span className="flex min-w-0 flex-col gap-1">
<span>{detail}</span>
@@ -22,13 +31,13 @@ export function ReportCheckTiming({
)}
{timing?.next_measurement_at && (
<span>
- <span>{timing.state === 'due' ? 'Measurement due since: ' : 'Next measurement: '}</span>
+ <span>{state === 'due' ? 'Measurement due since: ' : 'Next measurement: '}</span>
<TZLabel time={timing.next_measurement_at} timestampStyle="absolute" formatDate="MMM D" />
</span>
)}
{timing?.ends_at && (
<span>
- <span>{timing.state === 'window_ended' ? 'Window ended: ' : 'Window end: '}</span>
+ <span>{observationWindowEndLabel(state)}</span>
<TZLabel time={timing.ends_at} timestampStyle="absolute" formatDate="MMM D" />
</span>
)}
diff --git a/products/signals/frontend/inbox/components/detail/reportCheckPresentation.test.ts b/products/signals/frontend/inbox/components/detail/reportCheckPresentation.test.ts
index 32514d8fa93e..c13cdf7ee39f 100644
--- a/products/signals/frontend/inbox/components/detail/reportCheckPresentation.test.ts
+++ b/products/signals/frontend/inbox/components/detail/reportCheckPresentation.test.ts
@@ -140,7 +140,64 @@ describe('reportCheckPresentation', () => {
},
},
'Awaiting verification',
- 'The observation window has ended. Interim measurements have stopped. A final verdict has not been recorded.',
+ 'The observation window has ended. Final verification is pending.',
+ ],
+ [
+ 'an agent check awaits its final window without appearing paused',
+ {
+ monitoring_timing: {
+ state: 'paused',
+ final_verification_state: 'monitoring',
+ started_at: '2026-09-20T09:00:00Z',
+ ends_at: '2026-09-27T09:00:00Z',
+ next_measurement_at: null,
+ },
+ },
+ 'Monitoring',
+ 'Final verification is scheduled for the end of this check’s observation window.',
+ ],
+ [
+ 'a final scout runs without a legacy dispatch timestamp',
+ {
+ dispatched_at: null,
+ monitoring_timing: {
+ state: 'paused',
+ final_verification_state: 'verifying',
+ started_at: '2026-09-20T09:00:00Z',
+ ends_at: '2026-09-27T09:00:00Z',
+ next_measurement_at: null,
+ },
+ },
+ 'Verifying',
+ 'The scout is running the final check.',
+ ],
+ [
+ 'a settled inconclusive evaluation needs assessment',
+ {
+ monitoring_timing: {
+ state: 'paused',
+ final_verification_state: 'assessment_required',
+ started_at: '2026-09-20T09:00:00Z',
+ ends_at: '2026-09-27T09:00:00Z',
+ next_measurement_at: null,
+ },
+ },
+ 'Inconclusive',
+ 'Final verification could not settle this check. Further assessment is needed.',
+ ],
+ [
+ 'a recorded verdict does not promise another final run',
+ {
+ monitoring_timing: {
+ state: 'paused',
+ final_verification_state: 'verification_complete',
+ started_at: '2026-09-20T09:00:00Z',
+ ends_at: '2026-09-27T09:00:00Z',
+ next_measurement_at: null,
+ },
+ },
+ 'Verification complete',
+ 'A final verdict has been recorded for this check.',
],
[
'a paused Monitoring check does not promise a scout run',
@@ -155,6 +212,20 @@ describe('reportCheckPresentation', () => {
'Paused',
'This check is paused during Monitoring.',
],
+ [
+ 'null final verification metadata keeps the legacy paused state',
+ {
+ monitoring_timing: {
+ state: 'paused',
+ final_verification_state: null,
+ started_at: null,
+ ends_at: null,
+ next_measurement_at: null,
+ },
+ },
+ 'Paused',
+ 'This check is paused during Monitoring.',
+ ],
[
'a scheduled check leads with its run date and its lane',
{},
diff --git a/products/signals/frontend/inbox/components/detail/reportCheckPresentation.ts b/products/signals/frontend/inbox/components/detail/reportCheckPresentation.ts
index 4d5504deabde..1a65a8d5def8 100644
--- a/products/signals/frontend/inbox/components/detail/reportCheckPresentation.ts
+++ b/products/signals/frontend/inbox/components/detail/reportCheckPresentation.ts
@@ -123,9 +123,26 @@ function joinDetail(parts: (string | null | undefined)[]): string {
}
const MONITORING_CHECK_ROWS: Record<
- NonNullable<SignalReportCheckApi['monitoring_timing']>['state'],
+ | NonNullable<SignalReportCheckApi['monitoring_timing']>['state']
+ | NonNullable<NonNullable<SignalReportCheckApi['monitoring_timing']>['final_verification_state']>,
Pick<ReportCheckRowData, 'tag' | 'detail'>
> = {
+ monitoring: {
+ tag: { label: 'Monitoring', type: 'primary' },
+ detail: 'Final verification is scheduled for the end of this check’s observation window.',
+ },
+ verifying: {
+ tag: { label: 'Verifying', type: 'primary' },
+ detail: 'The scout is running the final check.',
+ },
+ assessment_required: {
+ tag: { label: 'Inconclusive', type: 'warning' },
+ detail: 'Final verification could not settle this check. Further assessment is needed.',
+ },
+ verification_complete: {
+ tag: { label: 'Verification complete', type: 'info' },
+ detail: 'A final verdict has been recorded for this check.',
+ },
scheduled: {
tag: { label: 'Scheduled', type: 'primary' },
detail: 'Interim measurements collect evidence during this check’s observation window.',
@@ -136,7 +153,7 @@ const MONITORING_CHECK_ROWS: Record<
},
window_ended: {
tag: { label: 'Awaiting verification', type: 'warning' },
- detail: 'The observation window has ended. Interim measurements have stopped. A final verdict has not been recorded.',
+ detail: 'The observation window has ended. Final verification is pending.',
},
no_interim_measurements: {
tag: { label: 'Observing', type: 'primary' },
@@ -164,6 +181,21 @@ const MONITORING_CHECK_ROWS: Record<
},
}
+function pendingCheckRow(check: SignalReportCheckApi, lane: string | null): Pick<ReportCheckRowData, 'tag' | 'detail'> {
+ const start = check.soak_minutes
+ ? `${check.kind === 'metric_threshold' ? 'At least' : 'Starts'} ${soakLabel(check.soak_minutes)} after this report is resolved`
+ : 'Starts when this report is resolved'
+ return { tag: { label: 'Waiting', type: 'muted' }, detail: joinDetail([start, lane && `${lane} runs it`]) }
+}
+
+function runningCheckRow(check: SignalReportCheckApi, lane: string | null): Pick<ReportCheckRowData, 'tag' | 'detail'> {
+ const startedAgo = dayjs(check.dispatched_at).fromNow()
+ return {
+ tag: { label: 'Running', type: 'primary' },
+ detail: lane ? `${lane} started ${startedAgo}` : `Started ${startedAgo}`,
+ }
+}
+
function openCheckRow(
check: SignalReportCheckApi,
reportStatus: SignalReportStatusEnumApi
@@ -175,23 +207,16 @@ function openCheckRow(
}
}
if (check.monitoring_timing) {
- return MONITORING_CHECK_ROWS[check.monitoring_timing.state]
+ return MONITORING_CHECK_ROWS[check.monitoring_timing.final_verification_state ?? check.monitoring_timing.state]
}
const lane = laneLabel(check)
if (check.status === 'pending') {
- const start = check.soak_minutes
- ? `${check.kind === 'metric_threshold' ? 'At least' : 'Starts'} ${soakLabel(check.soak_minutes)} after this report is resolved`
- : 'Starts when this report is resolved'
- return { tag: { label: 'Waiting', type: 'muted' }, detail: joinDetail([start, lane && `${lane} runs it`]) }
+ return pendingCheckRow(check, lane)
}
if (isRunning(check)) {
- const startedAgo = dayjs(check.dispatched_at).fromNow()
- return {
- tag: { label: 'Running', type: 'primary' },
- detail: lane ? `${lane} started ${startedAgo}` : `Started ${startedAgo}`,
- }
+ return runningCheckRow(check, lane)
}
// An `awaiting_data` verdict keeps the check open, so the row says why it looks again.
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 595144ac92af..0c6f70ae39b7 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -49160,6 +49160,22 @@ export namespace Schemas {
size_bytes: number;
}
+ /**
+ * * `monitoring` - Monitoring
+ * * `verifying` - Verifying
+ * * `assessment_required` - Assessment required
+ * * `verification_complete` - Verification complete
+ */
+ export type FinalVerificationStateEnum = typeof FinalVerificationStateEnum[keyof typeof FinalVerificationStateEnum];
+
+
+ export const FinalVerificationStateEnum = {
+ Monitoring: 'monitoring',
+ Verifying: 'verifying',
+ AssessmentRequired: 'assessment_required',
+ VerificationComplete: 'verification_complete',
+ } as const;
+
/**
* * `review` - review
* * `observe` - observe
@@ -66674,7 +66690,7 @@ export namespace Schemas {
} as const;
export interface MonitoringCheckTiming {
- /** Whether this check has an interim measurement due, is paused, or has finished its observation window. No state implies a final verdict.
+ /** Monitoring timing state. Final verification phases use paused for compatibility; read final_verification_state for their phase.
*
* * `scheduled` - Scheduled
* * `due` - Due
@@ -66686,6 +66702,13 @@ export namespace Schemas {
* * `unavailable` - Timing unavailable
* * `expiry_reached` - Expiry reached */
state: MonitoringCheckTimingStateEnum;
+ /** Final verification phase when state is paused. Null for other timing states. Read the check outcome for its verdict.
+ *
+ * * `monitoring` - Monitoring
+ * * `verifying` - Verifying
+ * * `assessment_required` - Assessment required
+ * * `verification_complete` - Verification complete */
+ readonly final_verification_state?: FinalVerificationStateEnum | null;
/**
* Original Monitoring start for this check's report.
* @nullable