Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): persist guarded monitoring evaluations (#122873)

Mikayla Thompsonff022be71429
diff --git a/.semgrep/rules/security/idor-team-scoped-models.yaml b/.semgrep/rules/security/idor-team-scoped-models.yaml
index 2ada6507f784..4e7d90a32ebd 100644
--- a/.semgrep/rules/security/idor-team-scoped-models.yaml
+++ b/.semgrep/rules/security/idor-team-scoped-models.yaml
@@ -391,6 +391,7 @@ rules:
|SignalReportArtefact
|SignalReportCheck
|SignalReportMonitoringAttempt
+ |SignalReportMonitoringResult
|SignalReportGithubComment
|SignalReportPullRequest
|SignalReportAssignment
@@ -868,6 +869,7 @@ rules:
|SignalReportArtefact
|SignalReportCheck
|SignalReportMonitoringAttempt
+ |SignalReportMonitoringResult
|SignalReportGithubComment
|SignalReportPullRequest
|SignalReportAssignment
@@ -1216,6 +1218,7 @@ rules:
|SignalReportArtefact
|SignalReportCheck
|SignalReportMonitoringAttempt
+ |SignalReportMonitoringResult
|SignalReportSlackThread
|SignalReportTask
|SignalSourceConfig
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.md
index 6c6a2d8a3022..8f1497c26bd5 100644
--- a/docs/internal/signals-pr-lifecycle.md
+++ b/docs/internal/signals-pr-lifecycle.md
@@ -16,6 +16,10 @@ A Monitoring period begins with an explicit, timezone-aware implementation-compl
The backend `complete_implementation` service records a durable attempt and freezes this report's pending or active, unexpired check IDs and authored-definition versions in the same transaction as Monitoring entry. Kind, approval, missing configuration, and missing or zero soak do not remove a check from this selection. Stale checks receive expiry records before selection. No applicable checks, or a disabled Monitoring flag, resolve directly without a verification attempt. Carried-over checks lose their earlier evidence and dispatch claims while keeping their remaining runs. Duplicate completion and archive/restore retain the snapshot; reopening creates a new attempt when implementation completes again. The single and bulk state APIs share this service. PR-merge handlers still resolve directly.
+`save_monitoring_result(MonitoringEvaluation(...))` stores final evaluation submissions separately from interim observations and activity artefacts. Each submission identifies the environment, report, attempt, check, definition version, fixed evidence window, evaluation time, qualification status and reason, expectation match, explanation, and opaque evidence references. Evaluators supply qualification using their own evidence policy; metric evaluators can use `qualify_metric_evidence`. Only sufficient evidence can produce a pass or failure. Insufficient or unsupported evidence produces an inconclusive outcome; execution errors remain errors.
+
+The save transaction locks the report before the check. Replaying an identical submission ID returns its saved row even after lifecycle changes; changing its content raises `MonitoringResultConflict`. New submissions require enabled Monitoring, the current attempt's membership and definition version, an unchanged open and unexpired check, and an explicit completed soak. The evidence window must equal `[monitoring_started_at, monitoring_started_at + soak_minutes]`, including zero soak. New IDs preserve inconclusive and error history and allow later evaluation. Once a pass or failure exists, another submission cannot replace it. Invalidated new work raises `InvalidMonitoringResult`. Evaluators must retain their submission ID and payload across delivery retries and use a fresh ID for a new evaluation. This interface does not run evaluators, schedule retries, spend check runs, complete checks, or transition reports.
+
Monitoring keeps the researched report available for verification and new matching signals, without starting another implementation. Enabled responses expose the server-owned start time, and implementation work is done while the report itself remains unfinished. A Monitoring child keeps its parent plan open, and child rollups cannot finish a Monitoring parent. The single and bulk state APIs accept the Monitoring target. Start monitoring appears after follow-up checks load and at least one exists; the service decides whether any remain applicable when the request runs. Existing Monitoring reports can be finished manually, reopened, or archived. Manual completion records caller attribution and explicitly states that automatic verification and an evidence amendment were not performed. PR merges continue to resolve reports.
Disabled organizations treat stored Monitoring reports as Resolved across reads, filters, counts, sorting, actions, recurrence handling, work state, plan rollups, and background checks. Responses omit the Monitoring start field and the inbox offers no Monitoring controls. Monitoring entry is refused. Restoring a report archived from Monitoring returns it to Resolved while disabled. Reads preserve the stored status.
diff --git a/products/signals/backend/migrations/0204_report_monitoring_result.py b/products/signals/backend/migrations/0204_report_monitoring_result.py
new file mode 100644
index 000000000000..8d10c764fdda
--- /dev/null
+++ b/products/signals/backend/migrations/0204_report_monitoring_result.py
@@ -0,0 +1,110 @@
+# Generated by Django 5.2.18 on 2026-10-23 19:35
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+import posthog.uuidt
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("posthog", "1412_drop_retired_desktop_file_system_tables"),
+ ("signals", "0203_reviewer_exclusion"),
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name="SignalReportMonitoringResult",
+ fields=[
+ (
+ "id",
+ models.UUIDField(
+ default=posthog.uuidt.uuid7,
+ editable=False,
+ primary_key=True,
+ serialize=False,
+ ),
+ ),
+ ("submission_id", models.UUIDField()),
+ ("definition_version", models.CharField(max_length=64)),
+ ("window_start", models.DateTimeField()),
+ ("window_end", models.DateTimeField()),
+ ("evaluated_at", models.DateTimeField()),
+ (
+ "outcome",
+ models.CharField(
+ choices=[
+ ("passed", "Passed"),
+ ("failed", "Failed"),
+ ("errored", "Errored"),
+ ("inconclusive", "Inconclusive"),
+ ],
+ max_length=20,
+ ),
+ ),
+ (
+ "evidence_status",
+ models.CharField(
+ choices=[
+ ("sufficient", "Sufficient"),
+ ("insufficient", "Insufficient"),
+ ("unsupported", "Unsupported"),
+ ("errored", "Errored"),
+ ],
+ max_length=20,
+ ),
+ ),
+ ("evidence_reason", models.TextField()),
+ ("expectation_matches", models.BooleanField(null=True)),
+ ("explanation", models.TextField()),
+ ("evidence_refs", models.JSONField(default=list)),
+ ("created_at", models.DateTimeField(auto_now_add=True)),
+ (
+ "attempt",
+ models.ForeignKey(
+ on_delete=django.db.models.deletion.CASCADE,
+ related_name="results",
+ to="signals.signalreportmonitoringattempt",
+ ),
+ ),
+ (
+ "report_check",
+ models.ForeignKey(
+ on_delete=django.db.models.deletion.CASCADE,
+ related_name="monitoring_results",
+ to="signals.signalreportcheck",
+ ),
+ ),
+ (
+ "report",
+ models.ForeignKey(
+ on_delete=django.db.models.deletion.CASCADE,
+ related_name="monitoring_results",
+ to="signals.signalreport",
+ ),
+ ),
+ (
+ "team",
+ models.ForeignKey(
+ db_constraint=False,
+ on_delete=django.db.models.deletion.CASCADE,
+ related_name="+",
+ to="posthog.team",
+ ),
+ ),
+ ],
+ options={
+ "constraints": [
+ models.UniqueConstraint(
+ fields=("attempt", "report_check", "submission_id"),
+ name="signals_result_submission_uniq",
+ ),
+ models.UniqueConstraint(
+ condition=models.Q(("outcome__in", ["passed", "failed"])),
+ fields=("attempt", "report_check"),
+ name="signals_result_verdict_uniq",
+ ),
+ ],
+ },
+ ),
+ ]
diff --git a/products/signals/backend/migrations/max_migration.txt b/products/signals/backend/migrations/max_migration.txt
index 9f784deae797..9ad0218fcc9c 100644
--- a/products/signals/backend/migrations/max_migration.txt
+++ b/products/signals/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0203_reviewer_exclusion
+0204_report_monitoring_result
diff --git a/products/signals/backend/models.py b/products/signals/backend/models.py
index 1ff816d539c6..3e91917ce679 100644
--- a/products/signals/backend/models.py
+++ b/products/signals/backend/models.py
@@ -2617,6 +2617,45 @@ class SignalReportMonitoringAttempt(UUIDModel):
]
+class SignalReportMonitoringResult(UUIDModel):
+ class EvidenceStatus(models.TextChoices):
+ SUFFICIENT = "sufficient"
+ INSUFFICIENT = "insufficient"
+ UNSUPPORTED = "unsupported"
+ ERRORED = "errored"
+
+ objects = EnvironmentScopedManager()
+
+ team = models.ForeignKey("posthog.Team", on_delete=models.CASCADE, db_constraint=False, related_name="+")
+ report = models.ForeignKey(SignalReport, on_delete=models.CASCADE, related_name="monitoring_results")
+ attempt = models.ForeignKey(SignalReportMonitoringAttempt, on_delete=models.CASCADE, related_name="results")
+ report_check = models.ForeignKey(SignalReportCheck, on_delete=models.CASCADE, related_name="monitoring_results")
+ submission_id = models.UUIDField()
+ definition_version = models.CharField(max_length=64)
+ window_start = models.DateTimeField()
+ window_end = models.DateTimeField()
+ evaluated_at = models.DateTimeField()
+ outcome = models.CharField(max_length=20, choices=SignalReportCheck.Outcome)
+ evidence_status = models.CharField(max_length=20, choices=EvidenceStatus)
+ evidence_reason = models.TextField()
+ expectation_matches = models.BooleanField(null=True)
+ explanation = models.TextField()
+ evidence_refs = models.JSONField(default=list)
+ created_at = models.DateTimeField(auto_now_add=True)
+
+ class Meta:
+ constraints = [
+ models.UniqueConstraint(
+ fields=["attempt", "report_check", "submission_id"], name="signals_result_submission_uniq"
+ ),
+ models.UniqueConstraint(
+ fields=["attempt", "report_check"],
+ condition=models.Q(outcome__in=["passed", "failed"]),
+ name="signals_result_verdict_uniq",
+ ),
+ ]
+
+
class SignalReviewerExclusion(TeamScopedRootMixin, UUIDModel):
"""A standing instruction that keeps one person off automatic reviewer selection in one area.
diff --git a/products/signals/backend/report_monitoring_results.py b/products/signals/backend/report_monitoring_results.py
new file mode 100644
index 000000000000..98d254b9b839
--- /dev/null
+++ b/products/signals/backend/report_monitoring_results.py
@@ -0,0 +1,143 @@
+from __future__ import annotations
+
+from datetime import datetime, timedelta
+from uuid import UUID
+
+from django.db import transaction
+from django.utils import timezone
+
+from posthog.dataclasses import frozen
+
+from products.signals.backend.models import SignalReport, SignalReportCheck, SignalReportMonitoringResult
+from products.signals.backend.report_check_evidence import EvidenceStatus
+from products.signals.backend.report_content_gates import team_report_monitoring_enabled
+from products.signals.backend.report_monitoring_attempts import check_definition_version, current_monitoring_attempt
+
+
+class InvalidMonitoringResult(ValueError):
+ pass
+
+
+class MonitoringResultConflict(ValueError):
+ pass
+
+
+@frozen
+class MonitoringEvaluation:
+ team_id: int
+ report_id: UUID
+ attempt_id: UUID
+ check_id: UUID
+ submission_id: UUID
+ definition_version: str
+ window_start: datetime
+ window_end: datetime
+ evaluated_at: datetime
+ evidence_status: EvidenceStatus
+ evidence_reason: str
+ expectation_matches: bool | None
+ explanation: str
+ evidence_refs: tuple[str, ...] = ()
+
+ def __post_init__(self) -> None:
+ if any(timezone.is_naive(value) for value in (self.window_start, self.window_end, self.evaluated_at)):
+ raise ValueError("Evaluation timestamps must be timezone-aware.")
+ if self.window_end < self.window_start or self.evaluated_at < self.window_end:
+ raise ValueError("Evaluation must follow an ordered evidence window.")
+ if self.evidence_status not in {"sufficient", "insufficient", "unsupported", "errored"}:
+ raise ValueError("Unknown evidence qualification.")
+ if self.expectation_matches is not None and not isinstance(self.expectation_matches, bool):
+ raise ValueError("An expectation match must be a boolean or absent.")
+ if self.evidence_status == "sufficient" and self.expectation_matches is None:
+ raise ValueError("Sufficient evidence needs an expectation match.")
+ if self.evidence_status in {"unsupported", "errored"} and self.expectation_matches is not None:
+ raise ValueError("Unsupported or errored evaluation cannot establish an expectation match.")
+ if not self.explanation.strip() or not self.evidence_reason.strip():
+ raise ValueError("An evaluation needs an explanation and evidence qualification reason.")
+ if any(not reference.strip() for reference in self.evidence_refs):
+ raise ValueError("Evidence references must not be empty.")
+
+ @property
+ def outcome(self) -> SignalReportCheck.Outcome:
+ if self.evidence_status == "errored":
+ return SignalReportCheck.Outcome.ERRORED
+ if self.evidence_status != "sufficient":
+ return SignalReportCheck.Outcome.INCONCLUSIVE
+ return SignalReportCheck.Outcome.PASSED if self.expectation_matches else SignalReportCheck.Outcome.FAILED
+
+ def matches(self, result: SignalReportMonitoringResult) -> bool:
+ return (
+ result.definition_version == self.definition_version
+ and result.window_start == self.window_start
+ and result.window_end == self.window_end
+ and result.evaluated_at == self.evaluated_at
+ and result.evidence_status == self.evidence_status
+ and result.evidence_reason == self.evidence_reason
+ and result.expectation_matches == self.expectation_matches
+ and result.explanation == self.explanation
+ and result.evidence_refs == list(self.evidence_refs)
+ )
+
+
+def save_monitoring_result(evaluation: MonitoringEvaluation) -> SignalReportMonitoringResult:
+ with transaction.atomic():
+ # Cancellation writes an artefact FK while holding the check row, so permit report key-share locks.
+ report = (
+ SignalReport.objects.select_for_update(no_key=True)
+ .filter(team_id=evaluation.team_id, id=evaluation.report_id)
+ .first()
+ )
+ results = SignalReportMonitoringResult.objects.for_team(evaluation.team_id).filter(
+ report_id=evaluation.report_id, attempt_id=evaluation.attempt_id, report_check_id=evaluation.check_id
+ )
+ previous = results.filter(submission_id=evaluation.submission_id).first()
+ if previous is not None:
+ if not evaluation.matches(previous):
+ raise MonitoringResultConflict("The submission ID already records a different evaluation.")
+ return previous
+ if report is None or report.status != SignalReport.Status.MONITORING:
+ raise InvalidMonitoringResult("The report is not Monitoring.")
+ attempt = current_monitoring_attempt(report)
+ if attempt is None or attempt.id != evaluation.attempt_id:
+ raise InvalidMonitoringResult("The implementation attempt is no longer current.")
+ if attempt.check_versions.get(str(evaluation.check_id)) != evaluation.definition_version:
+ raise InvalidMonitoringResult("The check and definition version must belong to the attempt snapshot.")
+ check = (
+ SignalReportCheck.objects.for_team(evaluation.team_id)
+ .select_for_update()
+ .filter(id=evaluation.check_id, report_id=report.id)
+ .first()
+ )
+ now = timezone.now()
+ if not team_report_monitoring_enabled(evaluation.team_id):
+ raise InvalidMonitoringResult("Monitoring is disabled.")
+ if check is None or check.status not in SignalReportCheck.OPEN_STATUSES or check.expires_at <= now:
+ raise InvalidMonitoringResult("The check is no longer open and unexpired.")
+ if check_definition_version(check) != evaluation.definition_version:
+ raise InvalidMonitoringResult("The authored definition changed.")
+ if check.soak_minutes is None:
+ raise InvalidMonitoringResult("The check needs an explicit soak.")
+ end = attempt.monitoring_started_at + timedelta(minutes=check.soak_minutes)
+ if evaluation.window_start != attempt.monitoring_started_at or evaluation.window_end != end:
+ raise InvalidMonitoringResult("The evidence window must match the fixed Monitoring window.")
+ if now < end or evaluation.evaluated_at > now:
+ raise InvalidMonitoringResult("The soak must finish before evaluation and acceptance.")
+ if results.filter(outcome__in=[SignalReportCheck.Outcome.PASSED, SignalReportCheck.Outcome.FAILED]).exists():
+ raise MonitoringResultConflict("The check already has an accepted verdict for this attempt.")
+ return results.create(
+ team_id=evaluation.team_id,
+ report_id=report.id,
+ attempt_id=attempt.id,
+ report_check_id=check.id,
+ submission_id=evaluation.submission_id,
+ definition_version=evaluation.definition_version,
+ window_start=evaluation.window_start,
+ window_end=evaluation.window_end,
+ evaluated_at=evaluation.evaluated_at,
+ outcome=evaluation.outcome,
+ evidence_status=evaluation.evidence_status,
+ evidence_reason=evaluation.evidence_reason,
+ expectation_matches=evaluation.expectation_matches,
+ explanation=evaluation.explanation,
+ evidence_refs=list(evaluation.evidence_refs),
+ )
diff --git a/products/signals/backend/test/test_report_check_concurrency.py b/products/signals/backend/test/test_report_check_concurrency.py
index baf23a6cbeaa..8483aa9e7114 100644
--- a/products/signals/backend/test/test_report_check_concurrency.py
+++ b/products/signals/backend/test/test_report_check_concurrency.py
@@ -1,8 +1,10 @@
from collections.abc import Callable
from concurrent.futures import Future, ThreadPoolExecutor
+from dataclasses import replace
from datetime import timedelta
from queue import Queue
from time import monotonic
+from uuid import uuid4
import time_machine
from posthog.test.base import NonAtomicBaseTest
@@ -21,6 +23,7 @@ from products.signals.backend.models import (
SignalReportArtefact,
SignalReportCheck,
SignalReportMonitoringAttempt,
+ SignalReportMonitoringResult,
)
from products.signals.backend.report_check_authoring import cancel_check
from products.signals.backend.report_check_evaluation import CheckVerdict
@@ -31,7 +34,13 @@ from products.signals.backend.report_check_execution import (
)
from products.signals.backend.report_check_monitoring import MONITORING_ERROR_RETRY_AFTER, record_monitoring_observation
from products.signals.backend.report_checks import MAX_CHECK_HORIZON
-from products.signals.backend.report_monitoring_attempts import complete_implementation
+from products.signals.backend.report_monitoring_attempts import check_definition_version, complete_implementation
+from products.signals.backend.report_monitoring_results import (
+ InvalidMonitoringResult,
+ MonitoringEvaluation,
+ MonitoringResultConflict,
+ save_monitoring_result,
+)
from products.signals.backend.test.report_metric_test_fixtures import trends_metric_query
@@ -134,6 +143,107 @@ class TestReportCheckConcurrency(NonAtomicBaseTest):
assert self.check.last_outcome is None
assert not SignalReportArtefact.objects.filter(report=self.report, type="check_result").exists()
+ def _final_evaluation(self) -> MonitoringEvaluation:
+ now = timezone.now()
+ start = now - timedelta(hours=1)
+ SignalReport.objects.filter(id=self.report.id, team_id=self.team.id).update(status=SignalReport.Status.READY)
+ SignalReportCheck.objects.for_team(self.team.id).filter(id=self.check.id).update(
+ soak_minutes=60, expires_at=now + timedelta(days=1)
+ )
+ self.check.refresh_from_db()
+ attempt = complete_implementation(team_id=self.team.id, report_id=self.report.id, monitoring_started_at=start)
+ assert attempt is not None
+ return MonitoringEvaluation(
+ team_id=self.team.id,
+ report_id=self.report.id,
+ attempt_id=attempt.id,
+ check_id=self.check.id,
+ submission_id=uuid4(),
+ definition_version=check_definition_version(self.check),
+ window_start=start,
+ window_end=now,
+ evaluated_at=now,
+ evidence_status="sufficient",
+ evidence_reason="qualifying_observations",
+ expectation_matches=True,
+ explanation="The expectation holds.",
+ )
+
+ @parameterized.expand([("cancelled",), ("definition",), ("expired",), ("flag_off",)])
+ def test_final_result_rechecks_check_invalidation_after_a_lock_wait(self, change: str) -> None:
+ with (
+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True) as flag,
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ evaluation = self._final_evaluation()
+ checks = SignalReportCheck.objects.for_team(self.team.id).filter(id=self.check.id)
+ with transaction.atomic():
+ checks.select_for_update().get()
+ future = self._start_write(executor, lambda: save_monitoring_result(evaluation))
+ self._assert_write_waits(future)
+ if change == "cancelled":
+ assert cancel_check(
+ self.check, reason="stopped_by_person", attribution=ArtefactAttribution.system()
+ )
+ elif change == "definition":
+ checks.update(title="A different expectation")
+ elif change == "expired":
+ checks.update(expires_at=timezone.now())
+ else:
+ flag.return_value = False
+ with self.assertRaises(InvalidMonitoringResult):
+ future.result(timeout=10)
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+
+ @parameterized.expand([("resolved",), ("suppressed",), ("reopened",)])
+ def test_final_result_rechecks_report_state_after_a_lock_wait(self, change: str) -> None:
+ with (
+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ evaluation = self._final_evaluation()
+ with transaction.atomic():
+ report = SignalReport.objects.select_for_update(no_key=True).get(
+ id=self.report.id, team_id=self.team.id
+ )
+ future = self._start_write(executor, lambda: save_monitoring_result(evaluation))
+ self._assert_write_waits(future)
+ target = {
+ "resolved": SignalReport.Status.RESOLVED,
+ "suppressed": SignalReport.Status.SUPPRESSED,
+ "reopened": SignalReport.Status.READY,
+ }[change]
+ report.save(update_fields=report.transition_to(target))
+ with self.assertRaises(InvalidMonitoringResult):
+ future.result(timeout=10)
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+
+ @parameterized.expand([("duplicate",), ("conflict",)])
+ def test_concurrent_final_submissions_accept_one_verdict(self, change: str) -> None:
+ with (
+ patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ evaluation = self._final_evaluation()
+ competitor = (
+ replace(evaluation, submission_id=uuid4(), expectation_matches=False)
+ if change == "conflict"
+ else evaluation
+ )
+ with transaction.atomic():
+ SignalReport.objects.select_for_update(no_key=True).get(id=self.report.id, team_id=self.team.id)
+ future = self._start_write(executor, lambda: save_monitoring_result(competitor))
+ self._assert_write_waits(future)
+ accepted = save_monitoring_result(evaluation)
+ if change == "conflict":
+ with self.assertRaises(MonitoringResultConflict):
+ future.result(timeout=10)
+ else:
+ duplicate = future.result(timeout=10)
+ assert isinstance(duplicate, SignalReportMonitoringResult)
+ assert duplicate.id == accepted.id
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 1
+
@parameterized.expand([("cancelled",), ("extended",)])
def test_expiry_sweep_preserves_a_concurrent_change(self, change: str) -> None:
now = timezone.now()
diff --git a/products/signals/backend/test/test_report_monitoring_attempts.py b/products/signals/backend/test/test_report_monitoring_attempts.py
index 8a1961af0fed..22a076a70dc2 100644
--- a/products/signals/backend/test/test_report_monitoring_attempts.py
+++ b/products/signals/backend/test/test_report_monitoring_attempts.py
@@ -1,5 +1,7 @@
import json
+from dataclasses import replace
from datetime import timedelta
+from uuid import uuid4
import time_machine
from posthog.test.base import BaseTest
@@ -19,17 +21,55 @@ from products.signals.backend.models import (
SignalReportArtefact,
SignalReportCheck,
SignalReportMonitoringAttempt,
+ SignalReportMonitoringResult,
)
+from products.signals.backend.report_check_evidence import EvidenceStatus
from products.signals.backend.report_monitoring_attempts import (
check_definition_version,
complete_implementation,
current_monitoring_attempt,
)
+from products.signals.backend.report_monitoring_results import (
+ InvalidMonitoringResult,
+ MonitoringEvaluation,
+ MonitoringResultConflict,
+ save_monitoring_result,
+)
_FLAG_CLIENT = "products.signals.backend.report_content_gates.feature_enabled_or_false"
class TestMonitoringAttemptDefinitionVersion(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("sufficient", None, "Evidence is sufficient.", "qualified"),
+ ("errored", False, "The query failed.", "query_error"),
+ ("unsupported", True, "The source is unsupported.", "unsupported_source"),
+ ("insufficient", False, " ", "no_activity"),
+ ("insufficient", False, "No activity.", " "),
+ ]
+ )
+ def test_invalid_evidence_cannot_claim_an_expectation_verdict(
+ self, evidence_status: EvidenceStatus, matches: bool | None, explanation: str, reason: str
+ ) -> None:
+ now = timezone.now()
+ with self.assertRaises(ValueError):
+ MonitoringEvaluation(
+ team_id=1,
+ report_id=uuid4(),
+ attempt_id=uuid4(),
+ check_id=uuid4(),
+ submission_id=uuid4(),
+ definition_version="example",
+ window_start=now,
+ window_end=now,
+ evaluated_at=now,
+ evidence_status=evidence_status,
+ evidence_reason=reason,
+ expectation_matches=matches,
+ explanation=explanation,
+ )
+
@parameterized.expand(
[
("title", "Verify deployment", True),
@@ -94,6 +134,209 @@ class TestReportMonitoringAttempts(BaseTest):
expires_at=self.now + timedelta(days=30),
)
+ def _evaluation(self, check: SignalReportCheck) -> MonitoringEvaluation:
+ start = self.now - timedelta(minutes=60)
+ attempt = complete_implementation(team_id=check.team_id, report_id=check.report_id, monitoring_started_at=start)
+ assert attempt is not None
+ return MonitoringEvaluation(
+ team_id=check.team_id,
+ report_id=check.report_id,
+ attempt_id=attempt.id,
+ check_id=check.id,
+ submission_id=uuid4(),
+ definition_version=check_definition_version(check),
+ window_start=start,
+ window_end=start + timedelta(minutes=check.soak_minutes or 0),
+ evaluated_at=self.now,
+ evidence_status="sufficient",
+ evidence_reason="qualifying_observations",
+ expectation_matches=True,
+ explanation="The observed behavior meets the expectation.",
+ evidence_refs=("observation:example",),
+ )
+
+ @parameterized.expand([("metric_threshold",), ("agent",)])
+ def test_final_result_is_durable_idempotent_and_does_not_complete_or_spend_runs(self, kind: str) -> None:
+ check = self._check(kind=kind)
+ evaluation = self._evaluation(check)
+
+ result = save_monitoring_result(evaluation)
+ assert save_monitoring_result(evaluation).id == result.id
+ result.refresh_from_db()
+ assert result.team_id == self.team.id
+ assert result.report_id == self.report.id
+ assert result.attempt_id == evaluation.attempt_id
+ assert result.report_check_id == check.id
+ assert result.definition_version == evaluation.definition_version
+ assert result.window_start == evaluation.window_start
+ assert result.window_end == evaluation.window_end
+ assert result.evaluated_at == evaluation.evaluated_at
+ assert result.outcome == "passed"
+ assert result.evidence_status == "sufficient"
+ assert result.evidence_reason == evaluation.evidence_reason
+ assert result.expectation_matches is True
+ assert result.explanation == evaluation.explanation
+ assert result.evidence_refs == list(evaluation.evidence_refs)
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 1
+ self.report.refresh_from_db()
+ check.refresh_from_db()
+ assert self.report.status == SignalReport.Status.MONITORING
+ assert check.status == SignalReportCheck.Status.PENDING
+ assert check.runs_remaining == 1
+ assert check.last_outcome is None
+
+ @parameterized.expand([("expired",), ("resolved",), ("reopened",), ("cancelled",), ("definition",), ("disabled",)])
+ def test_saved_submission_retries_survive_lifecycle_changes(self, change: str) -> None:
+ check = self._check()
+ evaluation = self._evaluation(check)
+ result = save_monitoring_result(evaluation)
+ if change in {"resolved", "reopened"}:
+ self.report.refresh_from_db()
+ target = SignalReport.Status.RESOLVED if change == "resolved" else SignalReport.Status.READY
+ self.report.save(update_fields=self.report.transition_to(target))
+ elif change == "disabled":
+ self.flag.return_value = False
+ else:
+ if change == "expired":
+ check.expires_at = self.now
+ elif change == "cancelled":
+ check.status = SignalReportCheck.Status.CANCELLED
+ else:
+ check.title = "A changed expectation"
+ check.save()
+
+ assert save_monitoring_result(evaluation).id == result.id
+ with self.assertRaises(MonitoringResultConflict):
+ save_monitoring_result(replace(evaluation, explanation="A different evaluation"))
+ with self.assertRaises(InvalidMonitoringResult):
+ save_monitoring_result(replace(evaluation, submission_id=uuid4()))
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 1
+
+ def test_nonconclusive_submissions_allow_later_evaluation_and_do_not_claim_failure(self) -> None:
+ evaluation = self._evaluation(self._check())
+ statuses: tuple[EvidenceStatus, ...] = ("insufficient", "unsupported", "errored")
+ for status in statuses:
+ retry = replace(
+ evaluation,
+ submission_id=uuid4(),
+ evidence_status=status,
+ expectation_matches=False if status == "insufficient" else None,
+ )
+ result = save_monitoring_result(retry)
+ assert save_monitoring_result(retry).id == result.id
+ assert result.outcome == ("errored" if status == "errored" else "inconclusive")
+ verdict = save_monitoring_result(replace(evaluation, expectation_matches=False))
+ assert verdict.outcome == "failed"
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 4
+
+ @parameterized.expand([("same_id",), ("new_id",)])
+ def test_conflicting_submission_cannot_replace_a_verdict(self, change: str) -> None:
+ evaluation = self._evaluation(self._check())
+ result = save_monitoring_result(evaluation)
+ conflict = replace(
+ evaluation,
+ submission_id=uuid4() if change == "new_id" else evaluation.submission_id,
+ expectation_matches=False,
+ )
+ with self.assertRaises(MonitoringResultConflict):
+ save_monitoring_result(conflict)
+ result.refresh_from_db()
+ assert result.outcome == "passed"
+ assert SignalReportMonitoringResult.objects.for_team(self.team.id).count() == 1
+
+ @parameterized.expand(
+ [
+ ("resolved",),
+ ("suppressed",),
+ ("reopened",),
+ ("new_attempt",),
+ ("cancelled",),
+ ("expired",),
+ ("terminal",),
+ ("definition",),
+ ("snapshot",),
+ ("nonmember",),
+ ("wrong_attempt",),
+ ("wrong_team",),
+ ("window_start",),
+ ("window_end",),
+ ("not_due",),
+ ("future_evaluation",),
+ ("missing_soak",),
+ ("disabled",),
+ ("unavailable",),
+ ]
+ )
+ def test_invalidated_final_work_is_rejected(self, change: str) -> None:
+ check = self._check(soak_minutes=None if change == "missing_soak" else 60)
+ evaluation = self._evaluation(check)
+ if change in {"resolved", "suppressed", "reopened", "new_attempt"}:
+ self.report.refresh_from_db()
+ target = {"resolved": SignalReport.Status.RESOLVED, "suppressed": SignalReport.Status.SUPPRESSED}.get(
+ change, SignalReport.Status.READY
+ )
+ self.report.save(update_fields=self.report.transition_to(target))
+ if change == "new_attempt":
+ complete_implementation(team_id=self.team.id, report_id=self.report.id, monitoring_started_at=self.now)
+ elif change in {"cancelled", "expired", "terminal", "definition"}:
+ if change == "cancelled":
+ check.status = SignalReportCheck.Status.CANCELLED
+ elif change == "expired":
+ check.expires_at = self.now
+ elif change == "terminal":
+ check.status = SignalReportCheck.Status.ERRORED
+ else:
+ check.config = {"instructions": "A changed expectation"}
+ check.save()
+ elif change in {"snapshot", "nonmember"}:
+ SignalReportMonitoringAttempt.objects.for_team(self.team.id).filter(id=evaluation.attempt_id).update(
+ check_versions={} if change == "nonmember" else {str(check.id): "different-version"}
+ )
+ elif change == "wrong_attempt":
+ evaluation = replace(evaluation, attempt_id=uuid4())
+ elif change == "wrong_team":
+ other_team = Team.objects.create(organization=self.organization)
+ evaluation = replace(evaluation, team_id=other_team.id)
+ elif change == "window_start":
+ evaluation = replace(evaluation, window_start=evaluation.window_start - timedelta(seconds=1))
+ elif change == "window_end":
+ evaluation = replace(evaluation, window_end=evaluation.window_end - timedelta(seconds=1))
+ elif change == "future_evaluation":
+ evaluation = replace(evaluation, evaluated_at=self.now + timedelta(seconds=1))
+ elif change == "disabled":
+ self.flag.return_value = False
+ elif change == "unavailable":
+ self.flag.side_effect = RuntimeError("flag service unavailable")
+ with time_machine.travel(self.now - timedelta(seconds=1) if change == "not_due" else self.now, tick=False):
+ with self.assertRaises(InvalidMonitoringResult):
+ save_monitoring_result(evaluation)
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+
+ def test_zero_soak_is_immediately_due_but_does_not_imply_qualified_evidence(self) -> None:
+ evaluation = self._evaluation(self._check(soak_minutes=0))
+ evaluation = replace(
+ evaluation,
+ evaluated_at=evaluation.window_start,
+ evidence_status="insufficient",
+ evidence_reason="empty_window",
+ )
+ with time_machine.travel(evaluation.window_start, tick=False):
+ result = save_monitoring_result(evaluation)
+ assert result.window_start == result.window_end
+ assert result.expectation_matches is True
+ assert result.outcome == "inconclusive"
+
+ def test_final_results_preserve_literal_child_environment_scoping(self) -> None:
+ child = Team.objects.create(organization=self.organization, parent_team=self.team)
+ report = SignalReport.objects.create(team=child, status=SignalReport.Status.READY)
+ evaluation = self._evaluation(self._check(report=report))
+ result = save_monitoring_result(evaluation)
+ assert result.team_id == child.id
+ assert not SignalReportMonitoringResult.objects.for_team(self.team.id).exists()
+ assert SignalReportMonitoringResult.objects.for_team(child.id).get().id == result.id
+ with self.assertRaises(InvalidMonitoringResult):
+ save_monitoring_result(replace(evaluation, team_id=self.team.id))
+
@parameterized.expand(
[
("pending_agent_missing_soak", SignalReportCheck.Kind.AGENT, SignalReportCheck.Status.PENDING, None),