Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): enter monitoring after implementation prs merge (#123680)

Mikayla Thompson8dcc426da62b
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.md
index 72f52476e27c..24a6cf5290f3 100644
--- a/docs/internal/signals-pr-lifecycle.md
+++ b/docs/internal/signals-pr-lifecycle.md
@@ -1,7 +1,7 @@
# Report completion and late PR attachments
A report completes after all linked implementation PRs are closed or merged.
-At least one merged PR resolves the report; otherwise all closed PRs suppress it.
+At least one merged PR completes implementation: applicable follow-up checks enter Monitoring when enabled; otherwise the report resolves directly. All closed PRs without a merge suppress it.
PR-driven resolutions emit `signal_report_status_changed` with `resolution_reason=pr_merged`, including when attaching an already-merged PR completes a report. `dismissal_reason` remains reserved for feedback supplied with the transition. Outcome queries should prefer `resolution_reason` and fall back to `dismissal_reason`; a missing reason alone does not imply that no PR merged.
@@ -10,11 +10,11 @@ PR-driven dismissals emit `dismissal_cause=pr_closed_without_merge` when all lin
A person can resolve a report by recording a manual implementation or a useful finding that needs no action. Feedback from the state API, Slack, PR closures, report merges, refunds, and plan rollups saves its verdict so the activity log distinguishes resolutions from dismissals even after later state changes. Feedback predating recorded verdicts uses a neutral activity label.
The `no_action_needed` reason requires a resolved state. It leaves the report resolved, cancels its open follow-up checks, and is excluded from fixed outcomes in inbox ranking. Manual resolution closes an open implementation PR; the resulting GitHub close event preserves the resolved report status.
-Monitoring means implementation is complete and outcome verification is pending. Resolved means the report is finished and runs no follow-up checks. This includes stored Monitoring reports whose effective status becomes Resolved when the flag is disabled or unavailable. The organization-level `signals-report-monitoring` gate requires explicit enablement, including in local development, and treats missing definitions or evaluation failures as disabled.
+Monitoring means implementation is complete and outcome verification is pending. Resolved means the report is finished and runs no follow-up checks. This includes stored Monitoring reports whose effective status becomes Resolved when the flag is disabled or unavailable. The organization-level `signals-report-monitoring` gate requires explicit enablement, including in local development, and treats missing definitions or evaluation failures as disabled. With Monitoring disabled, PR merges retain legacy resolution transitions, including archived reports.
-A Monitoring period begins with an explicit, timezone-aware implementation-complete timestamp supplied by the server. Merge-driven entry can use the linked PR's recorded `merged_at`, even when its webhook arrives later. The lifecycle never substitutes receipt time for a missing implementation timestamp. Repeated entry preserves the original period; archive and restore preserve it too. Reopening implementation clears the start so a later implementation creates a new period. Resolving after verification keeps the implementation timestamp for history.
+A Monitoring period begins with an explicit, timezone-aware implementation-complete timestamp supplied by the server. Merge-driven entry uses the latest recorded `merged_at` among the merged implementation PRs, even when a webhook arrives later. Every merged PR must have a recorded merge time before a new Monitoring period can start; missing timestamps leave implementation open until reconciliation supplies them. For completed stacks waiting on timestamps, merged PRs from task history are persisted and queued for GitHub verification; timestamp repairs retry completion after commit. The lifecycle never substitutes receipt time for a missing implementation timestamp. Repeated entry preserves the original period; archive and restore preserve it too. Reopening implementation clears the start so a later implementation creates a new period. Resolving after verification keeps the implementation timestamp for history.
-The backend `complete_implementation` service records a durable attempt and freezes this report's pending or active, unexpired check IDs and authored-definition versions in the same transaction as Monitoring entry. Kind, approval, missing configuration, and missing or zero soak do not remove a check from this selection. Stale checks receive expiry records before selection. No applicable checks, or a disabled Monitoring flag, resolve directly without a verification attempt. Carried-over checks lose their earlier evidence and dispatch claims while keeping their remaining runs. Duplicate completion and archive/restore retain the snapshot; reopening creates a new attempt when implementation completes again. The single and bulk state APIs share this service. PR-merge handlers still resolve directly.
+The backend `complete_implementation` service records a durable attempt and freezes this report's pending or active, unexpired check IDs and authored-definition versions in the same transaction as Monitoring entry. Kind, approval, missing configuration, and missing or zero soak do not remove a check from this selection. Stale checks receive expiry records before selection. No applicable checks, or a disabled Monitoring flag, resolve directly without a verification attempt. Carried-over checks lose their earlier evidence and dispatch claims while keeping their remaining runs. Duplicate completion and archive/restore retain the snapshot; reopening creates a new attempt when implementation completes again. The single and bulk state APIs, merged-PR webhooks, and reconciliation after attaching an already-merged PR share this service. Background participant sync retries completion after the repaired merge timestamp commits, including old PRs outside the inbox summary window and failed participant lookups. Monitoring entry leaves tracker completion pending; direct PR-driven resolution retains its completion notification and attribution.
`save_monitoring_result(MonitoringEvaluation(...))` stores final evaluation submissions separately from interim observations and activity artefacts. Each submission identifies the environment, report, attempt, check, definition version, fixed evidence window, evaluation time, qualification status and reason, expectation match, explanation, and opaque evidence references. Evaluators supply qualification using their own evidence policy; metric evaluators can use `qualify_metric_evidence`. Only sufficient evidence can produce a pass or failure. Insufficient or unsupported evidence produces an inconclusive outcome; execution errors remain errors.
diff --git a/products/signals/backend/inbox_summary.py b/products/signals/backend/inbox_summary.py
index b496dce9dd24..4190df210e1f 100644
--- a/products/signals/backend/inbox_summary.py
+++ b/products/signals/backend/inbox_summary.py
@@ -1,5 +1,7 @@
from datetime import datetime, timedelta
+from functools import partial
+from django.db import transaction
from django.db.models import Exists, OuterRef, Q, QuerySet, TextField, Value
from django.db.models.functions import Cast, Concat, JSONArray
from django.utils import timezone
@@ -15,6 +17,7 @@ from posthog.models.user_integration import UserGitHubIntegration
from products.signals.backend.artefact_schemas import TASK_RUN_TYPE_IMPLEMENTATION
from products.signals.backend.models import SignalReportPullRequest, SignalReportTask
+from products.signals.backend.pull_requests import reconcile_reports_for_pull_request
INBOX_SUMMARY_PERIOD = timedelta(days=7)
@@ -158,6 +161,7 @@ def sync_pull_request_participants(*, team_id: int, repository: str, pr_number:
raise GitHubIntegrationError("Missing verified merge time")
now = timezone.now()
SignalReportPullRequest.objects.for_team(team_id).filter(id=pr.id).update(merged_at=merged_at, checked_at=now)
+ transaction.on_commit(partial(reconcile_reports_for_pull_request, team_id=team_id, pr_id=str(pr.id)))
# Repair the merge time before applying the window: older imports can have no timestamp.
if merged_at < now - INBOX_SUMMARY_PERIOD:
return
diff --git a/products/signals/backend/pull_requests.py b/products/signals/backend/pull_requests.py
index 831d7756a79d..5daef029ae49 100644
--- a/products/signals/backend/pull_requests.py
+++ b/products/signals/backend/pull_requests.py
@@ -10,12 +10,16 @@ from django.utils import timezone
import structlog
+from posthog.models.github_integration_base import GitHubIntegrationBase
+
from products.signals.backend.artefact_attribution import ArtefactAttribution
from products.signals.backend.artefact_schemas import PullRequestLink
from products.signals.backend.claim_display_name import claim_display_name
from products.signals.backend.models import SignalReport, SignalReportArtefact, SignalReportPullRequest
+from products.signals.backend.report_content_gates import team_report_monitoring_enabled
if TYPE_CHECKING:
+ from products.signals.backend.implementation_pr import ImplementationPr
from products.signals.backend.report_assignments import PullRequestDetails
logger = structlog.get_logger(__name__)
@@ -69,7 +73,11 @@ def reconcile_reports_for_pull_request(*, team_id: int, pr_id: str) -> None:
pr = SignalReportPullRequest.objects.for_team(team_id).get(id=pr_id)
report_ids = report_ids_for_implementation_pr(team_id=team_id, repository=pr.repository, pr_number=pr.number)
with transaction.atomic():
- reports = SignalReport.objects.select_for_update().filter(team_id=team_id, id__in=report_ids).order_by("id")
+ reports = (
+ SignalReport.objects.select_for_update(no_key=True)
+ .filter(team_id=team_id, id__in=report_ids)
+ .order_by("id")
+ )
for report in reports:
apply_report_completion(report)
schedule_report_replacements(team_id, str(report.id))
@@ -107,20 +115,26 @@ def link_pull_request(
defaults={"url": details.url},
)
pr = SignalReportPullRequest.objects.for_team(report.team_id).select_for_update().get(id=pr.id)
+ previous_state = pr.state
+ update_fields: list[str] = []
if pr.state != SignalReportPullRequest.State.MERGED and details.state != SignalReportPullRequest.State.UNKNOWN:
# Imported/task snapshots must not overwrite a state already verified with GitHub.
if state_source == PullRequestStateSource.GITHUB or pr.checked_at is None:
- previous_state = pr.state
pr.state = details.state
if state_source == PullRequestStateSource.GITHUB:
pr.checked_at = timezone.now()
- pr.save(update_fields=["state", "checked_at", "updated_at"])
- # Wait for the complete PR batch and release its locks before locking shared reports.
- transaction.on_commit(partial(reconcile_reports_for_pull_request, team_id=report.team_id, pr_id=str(pr.id)))
- if state_source != PullRequestStateSource.LEGACY_ASSIGNMENT:
- from products.signals.backend.inbox_push_notifications import schedule_pull_request_ready_push
-
- schedule_pull_request_ready_push(pr, previous_state=previous_state)
+ update_fields.extend(["state", "checked_at"])
+ if pr.state == SignalReportPullRequest.State.MERGED and pr.merged_at is None and details.merged_at is not None:
+ pr.merged_at = details.merged_at
+ update_fields.append("merged_at")
+ if update_fields:
+ pr.save(update_fields=[*update_fields, "updated_at"])
+ # Wait for the complete PR batch and release its locks before locking shared reports.
+ transaction.on_commit(partial(reconcile_reports_for_pull_request, team_id=report.team_id, pr_id=str(pr.id)))
+ if state_source != PullRequestStateSource.LEGACY_ASSIGNMENT:
+ from products.signals.backend.inbox_push_notifications import schedule_pull_request_ready_push
+
+ schedule_pull_request_ready_push(pr, previous_state=previous_state)
links = SignalReportArtefact.objects.filter(
team_id=report.team_id,
report_id=report.id,
@@ -175,21 +189,61 @@ def link_pull_request(
return pr
+def _repair_missing_merge_times(report: SignalReport, prs: Sequence["ImplementationPr"]) -> None:
+ from products.signals.backend.report_assignments import MAX_PR_NUMBER, MAX_REPOSITORY_LENGTH, PullRequestDetails
+ from products.signals.backend.tasks import refresh_pull_request_review_decision
+
+ for pr in prs:
+ if pr.state != "merged" or pr.merged_at is not None:
+ continue
+ parsed = GitHubIntegrationBase.parse_pull_request_url(pr.url)
+ if parsed is None or not 0 < parsed.number <= MAX_PR_NUMBER or len(parsed.repository) > MAX_REPOSITORY_LENGTH:
+ continue
+ stored = link_pull_request(
+ report=report,
+ details=PullRequestDetails(
+ url=pr.url, repository=parsed.repository.lower(), number=parsed.number, state="merged", merged=True
+ ),
+ actor=ArtefactAttribution.from_task(pr.task_id) if pr.task_id else ArtefactAttribution.system(),
+ claim_id=pr.claim_id,
+ state_source=PullRequestStateSource.TASK_OUTPUT,
+ notify_reviewers=False,
+ )
+ transaction.on_commit(
+ partial(
+ refresh_pull_request_review_decision.delay,
+ team_id=report.team_id,
+ repository=stored.repository,
+ pr_number=stored.number,
+ )
+ )
+
+
def apply_report_completion(report: SignalReport) -> None:
from products.signals.backend.implementation_pr import fetch_implementation_prs_for_reports
from products.signals.backend.report_assignments import _apply_pr_report_state
from products.signals.backend.supersession import pending_replacement
- if pending_replacement(report.team_id, str(report.id)) is not None:
+ if pending_replacement(report.team_id, str(report.id), using="default") is not None:
return
- states = [
- pr.state
- for pr in fetch_implementation_prs_for_reports([str(report.id)], team_id=report.team_id).get(str(report.id), [])
- ]
- state = completion_state(states)
+ prs = fetch_implementation_prs_for_reports([str(report.id)], team_id=report.team_id, using="default").get(
+ str(report.id), []
+ )
+ state = completion_state([pr.state for pr in prs])
if state is not None:
- _apply_pr_report_state(report, state)
+ merged_prs = [pr for pr in prs if pr.state == "merged"]
+ merge_times = [pr.merged_at for pr in merged_prs if pr.merged_at is not None]
+ merged_at = max(merge_times) if merge_times and len(merge_times) == len(merged_prs) else None
+ _apply_pr_report_state(report, state, merged_at=merged_at)
+ if (
+ state == "merged"
+ and merged_at is None
+ and report.status
+ in {SignalReport.Status.READY, SignalReport.Status.PENDING_INPUT, SignalReport.Status.FAILED}
+ and team_report_monitoring_enabled(report.team_id)
+ ):
+ _repair_missing_merge_times(report, prs)
def import_report_pull_requests(report: SignalReport, *, notify_reviewers: bool = False) -> None:
@@ -279,7 +333,9 @@ def update_pull_request_state(
).values_list("report_id", flat=True)
with transaction.atomic():
reports = list(
- SignalReport.objects.select_for_update().filter(team_id=team_id, id__in=report_ids).order_by("id")
+ SignalReport.objects.select_for_update(no_key=True)
+ .filter(team_id=team_id, id__in=report_ids)
+ .order_by("id")
)
pr = (
SignalReportPullRequest.objects.for_team(team_id)
@@ -296,7 +352,7 @@ def update_pull_request_state(
if pr.state != SignalReportPullRequest.State.MERGED:
pr.state = state
update_fields = ["state", "checked_at", "updated_at"]
- if merged_at is not None:
+ if state == SignalReportPullRequest.State.MERGED and pr.merged_at is None and merged_at is not None:
pr.merged_at = merged_at
update_fields.append("merged_at")
pr.checked_at = timezone.now()
diff --git a/products/signals/backend/report_assignments.py b/products/signals/backend/report_assignments.py
index 4ab593121f4c..515d3793bd73 100644
--- a/products/signals/backend/report_assignments.py
+++ b/products/signals/backend/report_assignments.py
@@ -6,6 +6,8 @@ from datetime import datetime
from typing import TypedDict
from django.db import transaction
+from django.utils import timezone
+from django.utils.dateparse import parse_datetime
import structlog
@@ -40,6 +42,7 @@ from products.signals.backend.report_claims import (
get_active_claim,
is_user_claim_of_task_creator,
)
+from products.signals.backend.report_monitoring_attempts import complete_implementation
from products.tasks.backend.facade import api as tasks_facade
logger = structlog.get_logger(__name__)
@@ -97,6 +100,7 @@ class PullRequestDetails:
number: int
state: str
merged: bool
+ merged_at: datetime | None = None
def assignment_snapshot(report: SignalReport, assignment: ReportClaim | None) -> AssignmentSnapshot:
@@ -260,12 +264,18 @@ def _pull_request_details(team_id: int, pr_url: str) -> PullRequestDetails:
pr_state = SignalReportAssignment.PrState.OPEN
else:
pr_state = SignalReportAssignment.PrState.UNKNOWN
+ raw_merged_at = status.get("merged_at")
+ try:
+ merged_at = parse_datetime(raw_merged_at) if isinstance(raw_merged_at, str) else None
+ except ValueError:
+ merged_at = None
return PullRequestDetails(
url=status.get("url") or pr_url,
repository=details.repository,
number=details.number,
state=pr_state,
merged=merged,
+ merged_at=merged_at if merged_at is not None and timezone.is_aware(merged_at) else None,
)
@@ -301,7 +311,9 @@ def sync_task_pull_request_to_assignments(
actor = ArtefactAttribution.from_task(task_id)
with transaction.atomic():
reports = list(
- SignalReport.objects.select_for_update().filter(team_id=team_id, id__in=report_ids).order_by("id")
+ SignalReport.objects.select_for_update(no_key=True)
+ .filter(team_id=team_id, id__in=report_ids)
+ .order_by("id")
)
for report in reports:
import_report_pull_requests(report, notify_reviewers=True)
@@ -328,7 +340,7 @@ def sync_task_pull_request_to_assignments(
return len(reports)
-def _apply_pr_report_state(report: SignalReport, pr_state: str | None) -> None:
+def _apply_pr_report_state(report: SignalReport, pr_state: str | None, *, merged_at: datetime | None = None) -> None:
target = None
if pr_state == SignalReportAssignment.PrState.MERGED:
target = SignalReport.Status.RESOLVED
@@ -344,6 +356,16 @@ def _apply_pr_report_state(report: SignalReport, pr_state: str | None) -> None:
}:
return
try:
+ if target == SignalReport.Status.RESOLVED:
+ complete_implementation(
+ team_id=report.team_id,
+ report_id=report.id,
+ monitoring_started_at=merged_at,
+ attribution=ArtefactAttribution.system(),
+ from_pr_merge=True,
+ )
+ report.refresh_from_db()
+ return
updated_fields = report.transition_to(target)
except (InvalidStatusTransition, ValueError, TypeError):
logger.info(
@@ -389,7 +411,7 @@ def claim_report(
)
with transaction.atomic():
- locked_report = SignalReport.objects.select_for_update().get(id=report.id, team_id=report.team_id)
+ locked_report = SignalReport.objects.select_for_update(no_key=True).get(id=report.id, team_id=report.team_id)
import_report_pull_requests(locked_report, notify_reviewers=True)
assignment = get_active_claim(team_id=report.team_id, report_id=report.id)
before = assignment_snapshot(locked_report, assignment)
@@ -499,7 +521,11 @@ def update_assignments_for_pull_request(
for team_id in sorted(set(team_ids)):
report_ids = report_ids_for_implementation_pr(team_id=team_id, repository=repository, pr_number=pr_number)
with transaction.atomic():
- reports = SignalReport.objects.select_for_update().filter(team_id=team_id, id__in=report_ids).order_by("id")
+ reports = (
+ SignalReport.objects.select_for_update(no_key=True)
+ .filter(team_id=team_id, id__in=report_ids)
+ .order_by("id")
+ )
for report in reports:
import_report_pull_requests(report, notify_reviewers=True)
for pr in fetch_implementation_prs_for_reports([str(report.id)], team_id=report.team_id).get(
@@ -520,6 +546,7 @@ def update_assignments_for_pull_request(
number=pr_number,
state=pr_state,
merged=pr_state == "merged",
+ merged_at=merged_at,
),
actor=ArtefactAttribution.from_task(pr.task_id),
claim_id=pr.claim_id,
diff --git a/products/signals/backend/report_monitoring_attempts.py b/products/signals/backend/report_monitoring_attempts.py
index b547835bf9e7..f70f2604046b 100644
--- a/products/signals/backend/report_monitoring_attempts.py
+++ b/products/signals/backend/report_monitoring_attempts.py
@@ -49,14 +49,15 @@ def complete_implementation(
*,
team_id: int,
report_id: UUID | str,
- monitoring_started_at: datetime,
+ monitoring_started_at: datetime | None,
attribution: ArtefactAttribution | None = None,
actor_user: User | None = None,
close_pr_on_resolve: bool = False,
+ from_pr_merge: bool = False,
) -> SignalReportMonitoringAttempt | None:
- if timezone.is_naive(monitoring_started_at):
+ if monitoring_started_at is not None and timezone.is_naive(monitoring_started_at):
raise ValueError("An aware implementation-complete timestamp is required.")
- if monitoring_started_at > timezone.now():
+ if monitoring_started_at is not None and monitoring_started_at > timezone.now():
raise ValueError("Implementation cannot complete in the future.")
with transaction.atomic():
@@ -64,6 +65,7 @@ def complete_implementation(
report = SignalReport.objects.select_for_update(no_key=True).get(team_id=team_id, id=report_id)
report._transition_actor = attribution # type: ignore[attr-defined]
report._transition_actor_user = actor_user # type: ignore[attr-defined]
+ report._status_from_pr_state = from_pr_merge # type: ignore[attr-defined]
if report.status == SignalReport.Status.RESOLVED:
return None
enabled = team_report_monitoring_enabled(team_id)
@@ -74,21 +76,20 @@ def complete_implementation(
if attempt is None:
raise ValueError("The Monitoring period has no check snapshot.")
return attempt
+ if not enabled:
+ report._close_pr_on_resolve = close_pr_on_resolve # type: ignore[attr-defined]
+ report.save(update_fields=report.transition_to(SignalReport.Status.RESOLVED))
+ return None
if report.status not in {
SignalReport.Status.READY,
SignalReport.Status.PENDING_INPUT,
SignalReport.Status.FAILED,
}:
raise InvalidStatusTransition(report.status, SignalReport.Status.MONITORING)
- if not enabled:
- report._close_pr_on_resolve = close_pr_on_resolve # type: ignore[attr-defined]
- report.save(update_fields=report.transition_to(SignalReport.Status.RESOLVED))
- return None
- if (
- SignalReportMonitoringAttempt.objects.for_team(team_id)
- .filter(report_id=report.id, monitoring_started_at__gte=monitoring_started_at)
- .exists()
- ):
+ earlier_attempts = SignalReportMonitoringAttempt.objects.for_team(team_id).filter(report_id=report.id)
+ if monitoring_started_at is not None:
+ earlier_attempts = earlier_attempts.filter(monitoring_started_at__gte=monitoring_started_at)
+ if earlier_attempts.exists():
raise ValueError("Implementation must complete after the report's earlier Monitoring periods.")
checks = list(
@@ -111,6 +112,9 @@ def complete_implementation(
report.save(update_fields=report.transition_to(SignalReport.Status.RESOLVED))
return None
+ if monitoring_started_at is None:
+ raise ValueError("An implementation-complete timestamp is required to enter Monitoring.")
+
attempt = SignalReportMonitoringAttempt.objects.for_team(team_id).create(
team_id=team_id,
report=report,
diff --git a/products/signals/backend/tasks.py b/products/signals/backend/tasks.py
index 8940f383b3b6..6217f4278b01 100644
--- a/products/signals/backend/tasks.py
+++ b/products/signals/backend/tasks.py
@@ -40,7 +40,7 @@ from products.signals.backend.implementation_dispatch_tasks import (
)
from products.signals.backend.implementation_pr import PrCloseReason, close_implementation_pr_for_report
from products.signals.backend.inbox_push_notifications import notify_pull_request_ready, notify_report_visible
-from products.signals.backend.inbox_summary import sync_pull_request_participants
+from products.signals.backend.inbox_summary import github_datetime, sync_pull_request_participants
from products.signals.backend.models import (
SignalReport,
SignalReportPullRequest,
@@ -54,7 +54,7 @@ from products.signals.backend.models import (
from products.signals.backend.pr_origin import write_origin_section
from products.signals.backend.pull_request_body import BodyEditOutcome
from products.signals.backend.pull_request_label import apply_pull_request_label
-from products.signals.backend.pull_requests import update_pull_request_review_decision
+from products.signals.backend.pull_requests import update_pull_request_review_decision, update_pull_request_state
from products.signals.backend.report_generation.repo_activity import (
ACTIVITY_KEEP_WARM_WINDOW,
rebuild_repository_activity,
@@ -170,6 +170,14 @@ def refresh_pull_request_review_decision(team_id: int, repository: str, pr_numbe
)
if github is None:
return
+ if pr.state == SignalReportPullRequest.State.MERGED and pr.merged_at is None:
+ details = github.get_pull_request(pr.repository, pr.number)
+ merged_at = github_datetime(details.get("merged_at"))
+ if not details.get("success") or details.get("merged") is not True or merged_at is None:
+ raise GitHubIntegrationError("Could not verify the pull request merge time")
+ update_pull_request_state(
+ team_id=team_id, repository=pr.repository, number=pr.number, state="merged", merged_at=merged_at
+ )
snapshot = github.get_pull_request_snapshot(pr.url)
if not snapshot.get("success"):
logger.warning(
diff --git a/products/signals/backend/test/test_backfill_report_pull_requests.py b/products/signals/backend/test/test_backfill_report_pull_requests.py
index b7504586e342..405cbb0e5cfc 100644
--- a/products/signals/backend/test/test_backfill_report_pull_requests.py
+++ b/products/signals/backend/test/test_backfill_report_pull_requests.py
@@ -1,15 +1,18 @@
-from datetime import UTC, datetime
+from datetime import UTC, datetime, timedelta
from io import StringIO
from posthog.test.base import BaseTest
from unittest.mock import MagicMock, patch
from django.core.management import call_command
+from django.utils import timezone
from products.signals.backend.models import (
SignalReport,
SignalReportArtefact,
SignalReportAssignment,
+ SignalReportCheck,
+ SignalReportMonitoringAttempt,
SignalReportPullRequest,
SignalReportTask,
)
@@ -18,6 +21,74 @@ from products.tasks.backend.models import Task, TaskRun
class TestBackfillReportPullRequests(BaseTest):
+ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True)
+ def test_repairs_task_history_merge_time_before_starting_monitoring(self, _flag: MagicMock) -> None:
+ from products.signals.backend.tasks import refresh_pull_request_review_decision
+
+ report = SignalReport.objects.create(team=self.team, status="ready", title="Report", summary="Summary")
+ task = Task.objects.create(team=self.team, title="Implementation", description="", origin_product="signals")
+ SignalReportTask.objects.create(team=self.team, report=report, task=task, relationship="implementation")
+ run = TaskRun.objects.create(team=self.team, task=task, status=TaskRun.Status.COMPLETED)
+ TaskRun.objects.filter(id=run.id).update(
+ output={"pr_url": "https://github.com/example/app/pull/1", "pr_state": "merged", "pr_merged": True}
+ )
+ SignalReportAssignment.all_teams.create(
+ team=self.team,
+ report=report,
+ pr_url="https://github.com/example/app/pull/2",
+ repository="example/app",
+ pr_number=2,
+ pr_state="open",
+ )
+ check = SignalReportCheck.objects.for_team(self.team.id).create(
+ team=self.team,
+ report=report,
+ title="Verify recovery",
+ kind=SignalReportCheck.Kind.AGENT,
+ config={"instructions": "Verify recovery", "skill_name": "verification"},
+ soak_minutes=60,
+ status=SignalReportCheck.Status.PENDING,
+ next_run_at=timezone.now(),
+ expires_at=timezone.now() + timedelta(days=7),
+ )
+ latest_merge = timezone.now() - timedelta(hours=1)
+ github = MagicMock()
+ github.get_pull_request.return_value = {
+ "success": True,
+ "merged": True,
+ "merged_at": (latest_merge - timedelta(hours=1)).isoformat(),
+ }
+ github.get_pull_request_snapshot.return_value = {"success": True, "review_decision": "approved"}
+ with (
+ patch("products.signals.backend.tasks.refresh_pull_request_review_decision.delay") as refresh,
+ patch("products.signals.backend.tasks.refresh_pull_request_participants.delay"),
+ patch("products.signals.backend.tasks.assign_reviewers_on_implementation_pr.delay"),
+ patch("products.signals.backend.tasks.send_pull_request_ready_push.delay"),
+ patch("posthog.models.integration.GitHubIntegration.first_for_team_repository", return_value=github),
+ ):
+ with self.captureOnCommitCallbacks(execute=True):
+ update_assignments_for_pull_request(
+ team_ids=[self.team.id],
+ repository="example/app",
+ pr_number=2,
+ pr_state="merged",
+ merged_at=latest_merge,
+ )
+ report.refresh_from_db()
+ assert report.status == "ready"
+ refresh.assert_any_call(team_id=self.team.id, repository="example/app", pr_number=1)
+ link = SignalReportArtefact.objects.get(report=report, type="pull_request", task=task)
+ assert link.actor_kind == "task"
+ assert not SignalReportArtefact.objects.filter(report=report, type="work_claim").exists()
+ with self.captureOnCommitCallbacks(execute=True):
+ refresh_pull_request_review_decision(team_id=self.team.id, repository="example/app", pr_number=1)
+
+ completed = SignalReport.objects.get(id=report.id, team_id=self.team.id)
+ assert completed.status == SignalReport.Status.MONITORING
+ attempt = SignalReportMonitoringAttempt.objects.for_team(self.team.id).get(report=report)
+ assert attempt.monitoring_started_at == latest_merge
+ assert set(attempt.check_versions) == {str(check.id)}
+
def test_refreshes_stored_review_decision(self) -> None:
from products.signals.backend.tasks import refresh_pull_request_review_decision
diff --git a/products/signals/backend/test/test_inbox_summary.py b/products/signals/backend/test/test_inbox_summary.py
index 48debd816254..ee1be1e1b7f4 100644
--- a/products/signals/backend/test/test_inbox_summary.py
+++ b/products/signals/backend/test/test_inbox_summary.py
@@ -20,8 +20,17 @@ from posthog.models.integration import GitHubIntegration, Integration
from posthog.models.user_integration import UserIntegration
from products.access_control.backend.models.access_control import AccessControl
+from products.signals.backend.artefact_attribution import ArtefactAttribution
+from products.signals.backend.artefact_schemas import PullRequestLink
from products.signals.backend.inbox_summary import sync_pull_request_participants
-from products.signals.backend.models import SignalReport, SignalReportPullRequest, SignalReportTask
+from products.signals.backend.models import (
+ SignalReport,
+ SignalReportArtefact,
+ SignalReportCheck,
+ SignalReportMonitoringAttempt,
+ SignalReportPullRequest,
+ SignalReportTask,
+)
from products.signals.backend.pull_requests import schedule_participant_refresh, update_pull_request_state
from products.signals.backend.tasks import refresh_pull_request_participants, refresh_pull_request_review_decision
from products.tasks.backend.facade import api as tasks_facade
@@ -410,14 +419,56 @@ class TestInboxSummary(APIBaseTest):
assert pr.participant_logins is None
assert pr.participants_synced_at is None
- def test_an_old_missing_merge_time_is_repaired_without_fetching_reviews(self) -> None:
+ @parameterized.expand([("recent", 1, False), ("old", 30, False), ("reviews_unavailable", 1, True)])
+ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True)
+ def test_merge_time_repair_retries_report_completion_after_commit(
+ self, _name: str, days_ago: int, reviews_fail: bool, _flag: MagicMock
+ ) -> None:
pr = self.make_pr(merged_at=None)
+ report = SignalReportTask.objects.get(team=self.team).report
+ report.status = SignalReport.Status.READY
+ report.save(update_fields=["status"])
+ link = SignalReportArtefact.add_log(
+ team_id=self.team.id,
+ report_id=str(report.id),
+ content=PullRequestLink(url=pr.url),
+ attribution=ArtefactAttribution.system(),
+ )
+ link.pull_request = pr
+ link.save(update_fields=["pull_request"])
+ check = SignalReportCheck.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ report=report,
+ title="Verify the implementation",
+ kind=SignalReportCheck.Kind.AGENT,
+ config={},
+ soak_minutes=60,
+ next_run_at=NOW,
+ expires_at=NOW + timedelta(days=30),
+ )
+ merged_at = NOW - timedelta(days=days_ago)
github = self.github()
- github.get_pull_request.return_value["merged_at"] = (NOW - timedelta(days=30)).isoformat()
- self.sync(github)
- github.get_pull_request_reviews.assert_not_called()
+ github.get_pull_request.return_value["merged_at"] = merged_at.isoformat()
+ if reviews_fail:
+ github.get_pull_request_reviews.side_effect = GitHubIntegrationError("Reviews unavailable")
+ with self.captureOnCommitCallbacks(execute=True):
+ if reviews_fail:
+ with self.assertRaises(GitHubIntegrationError):
+ self.sync(github)
+ else:
+ self.sync(github)
+ report.refresh_from_db()
+ assert report.status == SignalReport.Status.READY
+ if days_ago > 7:
+ github.get_pull_request_reviews.assert_not_called()
pr.refresh_from_db()
- assert pr.merged_at == NOW - timedelta(days=30)
+ completed_report = SignalReport.objects.get(team_id=self.team.id, id=report.id)
+ assert pr.merged_at == merged_at
+ assert completed_report.status == SignalReport.Status.MONITORING
+ assert completed_report.monitoring_started_at == merged_at
+ attempt = SignalReportMonitoringAttempt.objects.for_team(self.team.id).get(report=completed_report)
+ assert attempt.monitoring_started_at == merged_at
+ assert list(attempt.check_versions) == [str(check.id)]
def test_a_failed_refresh_keeps_the_last_complete_snapshot(self) -> None:
pr = self.make_pr(participant_ids=[10, 20])
diff --git a/products/signals/backend/test/test_report_monitoring_attempts.py b/products/signals/backend/test/test_report_monitoring_attempts.py
index 26ee940dfe98..e1d71ea2c9e0 100644
--- a/products/signals/backend/test/test_report_monitoring_attempts.py
+++ b/products/signals/backend/test/test_report_monitoring_attempts.py
@@ -478,11 +478,20 @@ class TestReportMonitoringAttempts(BaseTest):
report=self.report, type=SignalReportArtefact.ArtefactType.CHECK_EXPIRED
).exists()
- @parameterized.expand([("disabled", False), ("unavailable", RuntimeError("flag service unavailable"))])
+ @parameterized.expand(
+ [
+ ("disabled", False),
+ ("unavailable", RuntimeError("flag service unavailable")),
+ ("archived_disabled", False, True),
+ ("archived_unavailable", RuntimeError("flag service unavailable"), True),
+ ]
+ )
def test_disabled_or_unavailable_flag_resolves_without_creating_an_attempt(
- self, _name: str, flag_result: bool | Exception
+ self, _name: str, flag_result: bool | Exception, archived: bool = False
) -> None:
check = self._check()
+ if archived:
+ self.report.save(update_fields=self.report.transition_to(SignalReport.Status.SUPPRESSED))
if isinstance(flag_result, Exception):
self.flag.side_effect = flag_result
else:
diff --git a/products/signals/backend/test/test_signal_report_assignment_api.py b/products/signals/backend/test/test_signal_report_assignment_api.py
index 1d1b4173f41d..787389632855 100644
--- a/products/signals/backend/test/test_signal_report_assignment_api.py
+++ b/products/signals/backend/test/test_signal_report_assignment_api.py
@@ -1,10 +1,12 @@
import json
+from datetime import timedelta
from posthog.test.base import APIBaseTest
from unittest.mock import MagicMock, call, patch
from django.apps import apps
from django.test import SimpleTestCase
+from django.utils import timezone
from parameterized import parameterized
from rest_framework import status
@@ -19,10 +21,13 @@ from products.signals.backend.models import (
SignalReport,
SignalReportArtefact,
SignalReportAssignment,
+ SignalReportCheck,
+ SignalReportMonitoringAttempt,
SignalReportPullRequest,
)
from products.signals.backend.report_assignments import update_assignments_for_pull_request
from products.signals.backend.report_claims import get_active_claim
+from products.signals.backend.report_monitoring_attempts import complete_implementation
class TestPrimaryPullRequest(SimpleTestCase):
@@ -44,6 +49,18 @@ class TestPrimaryPullRequest(SimpleTestCase):
class TestSignalReportAssignmentAPI(APIBaseTest):
+ def _create_check(self, report: SignalReport, *, soak_minutes: int | None = None) -> SignalReportCheck:
+ return SignalReportCheck.objects.for_team(report.team_id).create(
+ team_id=report.team_id,
+ report=report,
+ title="Verify the implementation",
+ kind=SignalReportCheck.Kind.AGENT,
+ config={},
+ soak_minutes=soak_minutes,
+ next_run_at=timezone.now(),
+ expires_at=timezone.now() + timedelta(days=30),
+ )
+
def _create_report(
self,
*,
@@ -261,16 +278,37 @@ class TestSignalReportAssignmentAPI(APIBaseTest):
@parameterized.expand(
[
- ("all_closed", "closed", "closed", SignalReport.Status.SUPPRESSED),
- ("one_merged", "merged", "closed", SignalReport.Status.RESOLVED),
- ("all_merged", "merged", "merged", SignalReport.Status.RESOLVED),
- ("unknown", "merged", "unknown", SignalReport.Status.READY),
- ("draft", "merged", "draft", SignalReport.Status.READY),
+ ("all_closed", "closed", "closed", "suppressed", False, True, False),
+ ("one_merged", "merged", "closed", "resolved", False, True, False),
+ ("all_merged", "merged", "merged", "resolved", False, True, False),
+ ("no_checks_without_merge_time", "merged", "closed", "resolved", False, True, True),
+ ("unknown", "merged", "unknown", "ready", True, True, False),
+ ("draft", "merged", "draft", "ready", True, True, False),
+ ("monitoring", "merged", "closed", "monitoring", True, True, False),
+ ("latest_merge", "merged", "merged", "monitoring", True, True, False),
+ ("disabled", "merged", "merged", "resolved", True, False, False),
+ ("missing_merge_time", "merged", "closed", "ready", True, True, True),
]
)
+ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false")
@patch("products.signals.backend.report_assignments.GitHubIntegration.first_for_team_repository", return_value=None)
- def test_stack_completion_waits_for_all_prs(self, _name, first_state, last_state, expected, _integration):
+ def test_stack_completion_waits_for_all_prs(
+ self,
+ _name: str,
+ first_state: str,
+ last_state: str,
+ expected: str,
+ with_check: bool,
+ enabled: bool,
+ missing_time: bool,
+ _integration: MagicMock,
+ flag: MagicMock,
+ ) -> None:
+ flag.return_value = enabled
+ first_merge = timezone.now() - timedelta(hours=2)
+ last_merge = first_merge + timedelta(hours=1)
report = self._create_report()
+ check = self._create_check(report) if with_check else None
response = self.client.post(
self._claim_url(report),
{
@@ -283,16 +321,45 @@ class TestSignalReportAssignmentAPI(APIBaseTest):
)
assert response.status_code == 200
update_assignments_for_pull_request(
- team_ids=[self.team.id], repository="example/app", pr_number=1, pr_state=first_state
+ team_ids=[self.team.id],
+ repository="example/app",
+ pr_number=1,
+ pr_state=first_state,
+ merged_at=first_merge if first_state == "merged" and not missing_time else None,
)
report.refresh_from_db()
assert report.status == SignalReport.Status.READY
- for _ in range(2):
+ for retry_number in range(2):
update_assignments_for_pull_request(
- team_ids=[self.team.id], repository="example/sdk", pr_number=2, pr_state=last_state
+ team_ids=[self.team.id],
+ repository="example/sdk",
+ pr_number=2,
+ pr_state=last_state,
+ merged_at=last_merge + timedelta(minutes=retry_number) if last_state == "merged" else None,
)
report.refresh_from_db()
assert report.status == expected
+ attempts = SignalReportMonitoringAttempt.objects.for_team(self.team.id).filter(report_id=report.id)
+ if expected == SignalReport.Status.MONITORING:
+ assert check is not None
+ attempt = attempts.get()
+ assert list(attempt.check_versions) == [str(check.id)]
+ assert report.monitoring_started_at == (last_merge if last_state == "merged" else first_merge)
+ assert attempt.monitoring_started_at == report.monitoring_started_at
+ else:
+ assert not attempts.exists()
+ if missing_time and with_check:
+ update_assignments_for_pull_request(
+ team_ids=[self.team.id],
+ repository="example/app",
+ pr_number=1,
+ pr_state="merged",
+ merged_at=first_merge,
+ )
+ report.refresh_from_db()
+ assert report.status == SignalReport.Status.MONITORING
+ assert report.monitoring_started_at == first_merge
+ assert attempts.count() == 1
if expected != SignalReport.Status.READY:
retry = self.client.post(
self._claim_url(report),
@@ -305,6 +372,50 @@ class TestSignalReportAssignmentAPI(APIBaseTest):
assert retry.status_code == 200
assert SignalReportArtefact.objects.filter(report=report, type="pull_request").count() == 2
+ @parameterized.expand(
+ [
+ ("monitoring", "monitoring", False),
+ ("archived", "suppressed", False),
+ ("reopened", "ready", False),
+ ("reopened_missing_time", "ready", True),
+ ]
+ )
+ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True)
+ @patch("products.signals.backend.report_assignments.GitHubIntegration.first_for_team_repository", return_value=None)
+ def test_old_merge_preserves_an_existing_period_and_lifecycle_decisions(
+ self, _name: str, current_status: str, missing_time: bool, _integration: MagicMock, _flag: MagicMock
+ ) -> None:
+ report = self._create_report()
+ check = self._create_check(report)
+ assert (
+ self.client.post(
+ self._claim_url(report), {"pr_url": "https://github.com/example/app/pull/1"}, format="json"
+ ).status_code
+ == 200
+ )
+ started_at = timezone.now() - timedelta(hours=2)
+ attempt = complete_implementation(team_id=self.team.id, report_id=report.id, monitoring_started_at=started_at)
+ assert attempt is not None
+ report.refresh_from_db()
+ if current_status != SignalReport.Status.MONITORING:
+ report.save(update_fields=report.transition_to(SignalReport.Status(current_status)))
+ if missing_time:
+ check.status = SignalReportCheck.Status.CANCELLED
+ check.save(update_fields=["status"])
+
+ update_assignments_for_pull_request(
+ team_ids=[self.team.id],
+ repository="example/app",
+ pr_number=1,
+ pr_state="merged",
+ merged_at=None if missing_time else started_at,
+ )
+
+ report.refresh_from_db()
+ assert report.status == current_status
+ assert report.monitoring_started_at == (None if current_status == "ready" else started_at)
+ assert list(SignalReportMonitoringAttempt.objects.for_team(self.team.id).filter(report=report)) == [attempt]
+
@patch("products.signals.backend.report_assignments.GitHubIntegration.first_for_team_repository")
def test_attaching_a_partly_merged_stack_does_not_finish_early(self, integration):
github = integration.return_value
@@ -568,8 +679,13 @@ class TestSignalReportAssignmentAPI(APIBaseTest):
legacy.refresh_from_db()
assert legacy.actor_user_id == self.user.id
+ @parameterized.expand([("no_checks", False), ("checks", True)])
+ @patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True)
@patch("products.signals.backend.report_assignments.GitHubIntegration.first_for_team_repository")
- def test_merged_pull_request_resolves_report_on_claim(self, mock_first_for_repository):
+ def test_merged_pull_request_completes_implementation_on_claim(
+ self, _name: str, with_check: bool, mock_first_for_repository: MagicMock, _flag: MagicMock
+ ) -> None:
+ merged_at = timezone.now() - timedelta(hours=1)
github = MagicMock()
github.get_pull_request.return_value = {
"success": True,
@@ -577,9 +693,12 @@ class TestSignalReportAssignmentAPI(APIBaseTest):
"state": "closed",
"draft": False,
"merged": True,
+ "merged_at": merged_at.isoformat(),
}
mock_first_for_repository.return_value = github
report = self._create_report()
+ if with_check:
+ self._create_check(report, soak_minutes=60)
response = self.client.post(
self._claim_url(report),
@@ -589,10 +708,12 @@ class TestSignalReportAssignmentAPI(APIBaseTest):
assert response.status_code == status.HTTP_200_OK
report.refresh_from_db()
- assert report.status == SignalReport.Status.RESOLVED
+ assert report.status == (SignalReport.Status.MONITORING if with_check else SignalReport.Status.RESOLVED)
+ assert report.monitoring_started_at == (merged_at if with_check else None)
pr = SignalReportPullRequest.objects.for_team(self.team.id).get(repository="posthog/posthog", number=123)
assert pr.state == SignalReportAssignment.PrState.MERGED
assert pr.state == "merged"
+ assert pr.merged_at == merged_at
assert response.json()["work_state"] == "done"
@patch(