:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(signals): complete verified tracker issues durably (#124560)
Mikayla Thompson00b666d90215
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.mdindex 630a89965ce8..b9854b582991 100644--- a/docs/internal/signals-pr-lifecycle.md+++ b/docs/internal/signals-pr-lifecycle.md@@ -147,6 +147,14 @@ The Monitoring schema includes both outcome and tracker-delivery metadata. NullaThis aggregation path persists requests only. The report-level assessment worker and dispatch/execution of follow-up research remain separate work; a research request does not claim that a follow-up report has been created.+All-pass resolution also records a durable tracker-completion request. A separate one-minute sweep consumes pending requests directly, outside the check coordinator and its time budget. It has a 45-second soft limit and 60-second hard limit. Tracker closure serializes deliveries and verifies the same attempt is still effectively resolved before contacting the provider; archiving preserves that verified outcome, while reopening or replacing the attempt makes its request obsolete. Each claimed request reserves its next attempt five minutes ahead, so failures and interrupted workers cannot monopolize every sweep. Successful tracker closing comments are recorded independently of the provider close, and Redis lock failures use the normal retry path. Completed or obsolete requests are marked processed, and a partial index orders outstanding work by its next retry.++Manual completion preserves its open-PR closure intent on the frozen attempt. After verification passes, the sweep uses the existing best-effort PR closure path with a stable comment marker, then completes the tracker issue.++The request stays pending until every requested PR close and tracker close succeeds or is no longer needed, including reports without a tracker issue. Scheduled GitHub delivery uses the batch lane. Budget denial stops the sweep, and provider rate limits delay the request until at least the indicated retry time.++Single and bulk report-state requests share the delivery lock through their transaction commit. A request that overlaps completion is refused with a retry message. Delivery revalidates the frozen attempt before closing each PR and again after any intervening provider call. A tracker comment is recorded as delivered only when its provider reports success.+With `signals-report-metrics` enabled, research gathers short-window baselines before verification and prefers metric-based follow-up checks. It aims for practical 70-80% confidence in improvement, without claiming a calculated confidence score. Observed volume, recurrence, ingestion delay, and deployment timing justify the shortest useful query and wait. Frequent or high-volume checks can use 4-8 hours, allowing for several hours of deployment delay before the fix reaches users. Around 24 hours or several days require the corresponding activity or rollout evidence. Agent checks remain available when interpretation or unsupported measurements prevent a useful metric check. Missing activity is not success. These prompt preferences do not change execution, schema defaults, or final outcomes.The server refuses new Monitoring definitions when `signals-report-monitoring` is disabled or unavailable and omits the definition from check responses. Stored definitions remain intact. Research carries its definition-visibility decision into reconciliation, so flag recovery during a run does not replace an unchanged check or reset its approval and schedule. Existing definitions, charts, check schedules, and Resolved outcomes otherwise follow their current behavior.diff --git a/posthog/egress/github/README.md b/posthog/egress/github/README.mdindex f4bcb23362fb..d0402a026c90 100644--- a/posthog/egress/github/README.md+++ b/posthog/egress/github/README.md@@ -30,7 +30,7 @@ Use one stable snake_case `source` per subsystem, so the egress metrics show whoDeferrable background callers construct their client on the `BATCH` lane (`GitHubIntegration(integration, source=..., priority=Priority.BATCH)`; `api_request` also takes a per-call override).A shed sweep stops for the cycle and resumes on the next scheduled run.A caller that walks pages, such as the warehouse source, paces with `github_installation_pace_seconds` instead of getting denied.-The warehouse source's page fetches run on `BATCH`, while its repository validation and webhook management run on `NORMAL`, because a person waits on them. Customer Analytics feature-request link and resume lookups also run on `NORMAL` because an editor waits for the current issue state. Signals refreshes the stored pull request review decision from GitHub webhooks on `BATCH`; the inbox can keep the last known value when the lane is shed and update it after a retry.+The warehouse source's page fetches run on `BATCH`, while its repository validation and webhook management run on `NORMAL`, because a person waits on them. Customer Analytics feature-request link and resume lookups also run on `NORMAL` because an editor waits for the current issue state. Signals refreshes the stored pull request review decision from GitHub webhooks on `BATCH`; the inbox can keep the last known value when the lane is shed and update it after a retry. Its scheduled Monitoring completion sweep also closes linked PRs and tracker issues on `BATCH`, retaining completion requests and deferring the sweep when the installation budget or provider rate limit denies delivery.Self-Driving onboarding probes one repository per installation on `NORMAL` during activation and on `BATCH` during background completion checks. Failed probes back off without exhausting the completion polling window.The GitHub-optional Self-driving pilot checks implementation repository access (source `signals_github_optional`) on `NORMAL` for manual PR creation and on `BATCH` for automatic implementation. Unavailable access blocks implementation while research from project data remains available.The ownership file reader (`posthog/ownership/`, source `ownership_github_api`) runs on `NORMAL` where a person waits for a board or a snapshot list, and on `BATCH` for the scheduled digests and report generation. Its GraphQL reads charge `core`, as every GraphQL call does, and so does the one compare call per new head commit that lets it reuse cached files.diff --git a/posthog/tasks/scheduled.py b/posthog/tasks/scheduled.pyindex 4cd608b7e228..733f7f173cf7 100644--- a/posthog/tasks/scheduled.py+++ b/posthog/tasks/scheduled.py@@ -126,6 +126,7 @@ from products.signals.backend.tasks import (refresh_signal_scout_background_bands,sweep_first_report_notifications,sweep_implementation_dispatches,+ sweep_monitoring_tracker_issues,sync_pending_signals_refund_credits,)from products.skills.backend.tasks import sync_community_skills@@ -450,6 +451,14 @@ def setup_periodic_tasks(sender: Celery, **kwargs: Any) -> None:expires_seconds=5 * 60,)+ add_periodic_task_with_expiry(+ sender,+ crontab(minute="*"),+ sweep_monitoring_tracker_issues.s(),+ name="complete verified signals tracker issues",+ expires_seconds=60,+ )+# Re-enqueue signals PR refunds whose billing credit sync hasn't landed - hourly at minute 25sender.add_periodic_task(crontab(hour="*", minute="25"),diff --git a/products/signals/backend/implementation_pr.py b/products/signals/backend/implementation_pr.pyindex 61fbf31244af..11f88f46f278 100644--- a/products/signals/backend/implementation_pr.py+++ b/products/signals/backend/implementation_pr.py@@ -13,6 +13,7 @@ from django.db.models import F, Lookup, Q, QuerySetimport structlogfrom posthog.dataclasses import frozen+from posthog.egress.github.transport import GitHubEgressBudgetExhausted, GitHubRateLimitErrorfrom posthog.egress.limiter.policies import Priorityfrom posthog.models.github_integration_base import GitHubIntegrationBasefrom posthog.models.integration import GitHubIntegration@@ -427,7 +428,6 @@ def _pr_close_comment(if actor_mentionelse f"🔕 Closing this PR because the {report_link} was resolved without it.")- # A resolved report never reopens, so the restore advice below does not apply to it.return f"{opening}\n\nIf that wasn't intended, reopen this PR."opening = (f"🔕 Closing this PR because {actor_mention} {reason} the {report_link}."@@ -448,15 +448,15 @@ def _close_implementation_pr(expected_head_sha: str | None = None,comment_marker: str | None = None,before_close: Callable[[], bool] | None = None,-) -> bool:+ priority: Priority = Priority.CRITICAL,+) -> bool | None:"""Best-effort: comment on and close the GitHub PR attached to this report.Called when a report is suppressed, snoozed, or resolved without its PR — the open PR shouldn't linger. Only acts on a PRthat is still open: an already-closed or merged PR is left untouched (no comment, no close), sowe never leave a confusing "closing this PR" note on a PR that shipped months ago. Leaves an- explanatory comment, then closes the PR. Returns True when the PR was closed, False when there- was nothing to close or the close couldn't be completed. Never raises: the state transition- must succeed regardless.+ explanatory comment, then closes the PR. Returns True when closed, None when no close is+ needed, and False on failure. Batch rate limits propagate so scheduled delivery can back off."""try:if not SignalReport.objects.filter(id=report_id, team_id=team_id).exists():@@ -467,7 +467,7 @@ def _close_implementation_pr(report_id=str(report_id),actor_kind=pr.actor_kind,)- return False+ return Nonepr_url = pr.urlparsed = GitHubIntegrationBase.parse_pull_request_url(pr_url)@@ -483,10 +483,10 @@ def _close_implementation_pr(report_id=str(report_id),pr_url=pr_url,)- return False+ return Nonegithub = GitHubIntegration.first_for_team_repository(- team_id, parsed.repository, source="signals_implementation_pr", priority=Priority.CRITICAL+ team_id, parsed.repository, source="signals_implementation_pr", priority=priority)if github is None:logger.info(@@ -523,7 +523,7 @@ def _close_implementation_pr(state=pr_status.get("state"),merged=pr_status.get("merged"),)- return False+ return None if pr_status.get("state") == "closed" or pr_status.get("merged") else Falseif expected_head_sha is not None and pr_status.get("head_sha") != expected_head_sha:return False@@ -575,6 +575,10 @@ def _close_implementation_pr(pr_state="closed",)return True+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):+ if priority == Priority.BATCH:+ raise+ return Falseexcept Exception:logger.exception("close_implementation_pr_unexpected_error", report_id=str(report_id))return False@@ -588,26 +592,37 @@ def close_implementation_pr_for_report(actor_user_id: int | None = None,pr_url: str | None = None,replacement_pr_url: str | None = None,+ comment_marker: str | None = None,+ before_close: Callable[[], bool] | None = None,+ priority: Priority = Priority.CRITICAL,+ require_all: bool = False,) -> bool:try:if not SignalReport.objects.filter(team_id=team_id, id=report_id).exists():return Falseclosed = False+ failed = Falsefor pr in fetch_implementation_prs_for_reports([str(report_id)], team_id=team_id).get(str(report_id), []):if pr_url is not None and pr.url != pr_url:continue- closed = (- _close_implementation_pr(- team_id,- report_id,- reason=reason,- pr=pr,- actor_user_id=actor_user_id,- replacement_pr_url=replacement_pr_url,- )- or closed+ outcome = _close_implementation_pr(+ team_id,+ report_id,+ reason=reason,+ pr=pr,+ actor_user_id=actor_user_id,+ replacement_pr_url=replacement_pr_url,+ comment_marker=comment_marker,+ before_close=before_close,+ priority=priority,)- return closed+ closed = outcome is True or closed+ failed = outcome is False or failed+ return not failed if require_all else closed+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):+ if priority == Priority.BATCH:+ raise+ return Falseexcept Exception:logger.exception("close_implementation_pr_lookup_failed", report_id=str(report_id))return Falsediff --git a/products/signals/backend/report_monitoring_attempts.py b/products/signals/backend/report_monitoring_attempts.pyindex 05d8ae9bd1ec..f3b718338012 100644--- a/products/signals/backend/report_monitoring_attempts.py+++ b/products/signals/backend/report_monitoring_attempts.py@@ -120,6 +120,7 @@ def complete_implementation(report=report,monitoring_started_at=monitoring_started_at,check_versions={str(check.id): check_definition_version(check) for check in applicable},+ close_pr_on_resolve=close_pr_on_resolve,)SignalReportCheck.objects.for_team(team_id).filter(id__in=[check.id for check in applicable]).update(status=SignalReportCheck.Status.PENDING,diff --git a/products/signals/backend/tasks.py b/products/signals/backend/tasks.pyindex 1822bf429ed0..f4dad19aae21 100644--- a/products/signals/backend/tasks.py+++ b/products/signals/backend/tasks.py@@ -1,3 +1,4 @@+import timefrom contextlib import suppressfrom datetime import timedeltafrom decimal import Decimal@@ -89,7 +90,11 @@ from products.signals.backend.support_writeback import (post_pull_request_to_tickets,task_can_write_support_ticket_update,)-from products.signals.backend.tracker_issues import close_tracker_issue_for_report, link_pull_request_to_tracker_issue+from products.signals.backend.tracker_issues import (+ close_tracker_issue_for_report,+ complete_monitoring_tracker_issues,+ link_pull_request_to_tracker_issue,+)from products.tasks.backend.facade.repo_activity import RepositoryCommitActivityErrorlogger = structlog.get_logger(__name__)@@ -300,6 +305,8 @@ def close_dismissed_report_pr(ignore_result=True,bind=True,max_retries=5,+ soft_time_limit=45,+ time_limit=60,)@with_team_scope()def close_report_tracker_issue(@@ -322,6 +329,12 @@ def close_report_tracker_issue(raise self.retry(countdown=min(60 * (2**self.request.retries), 900))+@shared_task(ignore_result=True, soft_time_limit=45, time_limit=60)+@skip_team_scope_audit+def sweep_monitoring_tracker_issues() -> None:+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 40)++@shared_task(name="products.signals.backend.tasks.link_report_tracker_issues",ignore_result=True,diff --git a/products/signals/backend/test/test_dismissal_pr_close.py b/products/signals/backend/test/test_dismissal_pr_close.pyindex 9120d06bff7e..247110de54f7 100644--- a/products/signals/backend/test/test_dismissal_pr_close.py+++ b/products/signals/backend/test/test_dismissal_pr_close.py@@ -272,7 +272,8 @@ class TestCloseImplementationPrForReport(BaseTest):github.get_pull_request.assert_called_once_with("PostHog/posthog", 456)github.close_pull_request.assert_called_once_with("PostHog/posthog", 456)- def test_task_claim_actor_can_close_pr(self):+ @parameterized.expand([("current", True), ("reopened_during_comment", False)])+ def test_task_claim_actor_can_close_pr(self, _name: str, still_current: bool) -> None:self.assignment.actor_kind = SignalActorKind.TASKself.assignment.actor_task_id = uuid.uuid4()self.assignment.save(update_fields=["actor_kind", "actor_task_id", "updated_at"])@@ -280,12 +281,27 @@ class TestCloseImplementationPrForReport(BaseTest):github.get_pull_request.return_value = {"success": True, "state": "open", "merged": False}github.comment_on_pull_request.return_value = {"success": True}github.close_pull_request.return_value = {"success": True}+ current = True++ def comment(*args: object, **kwargs: object) -> dict[str, bool]:+ nonlocal current+ current = still_current+ return {"success": True}++ github.comment_on_pull_request.side_effect = commentwith patch("products.signals.backend.implementation_pr.GitHubIntegration.first_for_team_repository",return_value=github,):- assert close_implementation_pr_for_report(self.team.id, str(self.report.id)) is True+ assert (+ close_implementation_pr_for_report(self.team.id, str(self.report.id), before_close=lambda: current)+ is still_current+ )+ if still_current:+ github.close_pull_request.assert_called_once()+ else:+ github.close_pull_request.assert_not_called()@parameterized.expand([@@ -582,6 +598,17 @@ class TestSupersededPrClose(BaseTest):github.close_pull_request.return_value = {"success": True, "number": 123, "state": "closed"}return github+ def test_completion_waits_for_every_linked_pr(self) -> None:+ github = self._github()+ github.close_pull_request.side_effect = [{"success": False}, {"success": True}]+ with patch(+ "products.signals.backend.implementation_pr.GitHubIntegration.first_for_team_repository",+ return_value=github,+ ):+ assert close_implementation_pr_for_report(self.team.id, str(self.report.id), require_all=True) is False+ github.close_pull_request.side_effect = None+ assert close_implementation_pr_for_report(self.team.id, str(self.report.id), require_all=True) is True+def test_closes_the_named_pr_and_points_at_its_replacement(self):github = self._github()with patch(diff --git a/products/signals/backend/test/test_report_monitoring_aggregation.py b/products/signals/backend/test/test_report_monitoring_aggregation.pyindex 11a02135b758..1d7f298a5988 100644--- a/products/signals/backend/test/test_report_monitoring_aggregation.py+++ b/products/signals/backend/test/test_report_monitoring_aggregation.py@@ -13,6 +13,7 @@ from django.utils import timezonefrom parameterized import parameterized+from posthog.egress.limiter.policies import Priorityfrom posthog.models import Teamfrom products.signals.backend.artefact_attribution import ArtefactAttribution@@ -22,7 +23,9 @@ from products.signals.backend.models import (SignalReport,SignalReportArtefact,SignalReportCheck,+ SignalReportMonitoringAttempt,SignalReportMonitoringResult,+ SignalReportTrackerIssue,)from products.signals.backend.report_monitoring_aggregation import (MonitoringAggregation,@@ -35,6 +38,7 @@ from products.signals.backend.report_monitoring_outcomes import (aggregate_monitoring_attempt,)from products.signals.backend.report_monitoring_results import MonitoringEvaluation, save_monitoring_result+from products.signals.backend.tracker_issues import complete_monitoring_tracker_issuesclass TestMonitoringAggregation(SimpleTestCase):@@ -152,6 +156,14 @@ class TestReportMonitoringAggregation(BaseTest):historical.refresh_from_db()assert historical.status == SignalReportCheck.Status.EXPIREDself.close_tracker.assert_not_called()+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ self.close_tracker.assert_called_once_with(+ report_id=str(self.report.id),+ team_id=self.team.id,+ completed=True,+ monitoring_attempt_id=str(self.attempt.id),+ priority=Priority.BATCH,+ )def test_failure_closes_immediately_and_records_research_intent_atomically(self) -> None:earlier = SignalReportArtefact.add_log(@@ -228,6 +240,66 @@ class TestReportMonitoringAggregation(BaseTest):assert self.attempt.assessment_requested_at == self.nowassert self.attempt.research_requested_at == self.now+ @parameterized.expand([("provider_failure",), ("missed_sweep",)])+ def test_tracker_completion_recovers_after_report_resolution(self, cause: str) -> None:+ self.close_tracker.return_value = False+ SignalReportTrackerIssue.objects.for_team(self.team.id).create(+ team_id=self.team.id,+ report=self.report,+ provider="github",+ status=SignalReportTrackerIssue.Status.CREATED,+ )+ with self.captureOnCommitCallbacks(execute=True):+ save_monitoring_result(self._evaluation(0))+ save_monitoring_result(self._evaluation(1))+ self._aggregate()+ assert self.attempt is not None+ self.attempt.refresh_from_db()+ self.report.refresh_from_db()+ assert self.report.status == SignalReport.Status.RESOLVED+ assert self.attempt.tracker_issue_close_requested_at == self.now+ assert self.attempt.tracker_issue_close_processed_at is None+ self.close_tracker.assert_not_called()+ aggregate_due_monitoring_attempts(deadline=time.monotonic() + 30, limit=10)+ self.close_tracker.assert_not_called()+ if cause == "provider_failure":+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ self.attempt.refresh_from_db()+ assert self.attempt.tracker_issue_close_processed_at is None+ initial_calls = self.close_tracker.call_count+ self.close_tracker.return_value = True+ retry_at = self.now + timedelta(minutes=5) if cause == "provider_failure" else self.now+ with time_machine.travel(retry_at, tick=False):+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ self.attempt.refresh_from_db()+ assert self.attempt.tracker_issue_close_processed_at == retry_at+ assert self.close_tracker.call_count == initial_calls + 1+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ assert self.close_tracker.call_count == initial_calls + 1++ def test_failed_tracker_batch_does_not_starve_new_requests(self) -> None:+ self.close_tracker.return_value = False+ SignalReportTrackerIssue.objects.for_team(self.team.id).create(+ team_id=self.team.id,+ report=self.report,+ status=SignalReportTrackerIssue.Status.CREATED,+ )+ attempts = [+ SignalReportMonitoringAttempt(+ team_id=self.team.id,+ report=self.report,+ monitoring_started_at=self.now - timedelta(days=200 - index),+ tracker_issue_close_requested_at=self.now - timedelta(hours=1) + timedelta(seconds=index),+ )+ for index in range(101)+ ]+ SignalReportMonitoringAttempt.objects.for_team(self.team.id).bulk_create(attempts)+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ assert self.close_tracker.call_count == 100+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ assert self.close_tracker.call_count == 101+ assert self.close_tracker.call_args.kwargs["monitoring_attempt_id"] == str(attempts[-1].id)+@parameterized.expand([("suppressed",),diff --git a/products/signals/backend/test/test_report_monitoring_attempts.py b/products/signals/backend/test/test_report_monitoring_attempts.pyindex 524b09bb8158..37db03f41414 100644--- a/products/signals/backend/test/test_report_monitoring_attempts.py+++ b/products/signals/backend/test/test_report_monitoring_attempts.py@@ -586,7 +586,7 @@ class TestReportMonitoringAttempts(BaseTest):patch("products.signals.backend.tasks.close_dismissed_report_pr.delay") as close_pr,self.captureOnCommitCallbacks(execute=True),):- complete_implementation(+ attempt = complete_implementation(team_id=self.team.id,report_id=self.report.id,monitoring_started_at=self.now,@@ -594,6 +594,9 @@ class TestReportMonitoringAttempts(BaseTest):actor_user=self.user,close_pr_on_resolve=close_pr_on_resolve,)+ if has_checks:+ assert attempt is not None+ assert attempt.close_pr_on_resolve == close_pr_on_resolveproperties = capture.call_args.kwargs["properties"]assert properties["actor_kind"] == "user"assert properties["actor_user_uuid"] == str(self.user.uuid)diff --git a/products/signals/backend/test/test_tracker_issues.py b/products/signals/backend/test/test_tracker_issues.pyindex 2161a0acf4f3..bebfefe9f9f3 100644--- a/products/signals/backend/test/test_tracker_issues.py+++ b/products/signals/backend/test/test_tracker_issues.py@@ -1,14 +1,20 @@+import timefrom datetime import timedeltafrom typing import Anyimport pytest-from unittest.mock import MagicMock, patch+import time_machine+from unittest.mock import ANY, MagicMock, patchfrom django.utils import timezonefrom celery.exceptions import Retry+from redis.exceptions import ConnectionError as RedisConnectionErrorfrom rest_framework.exceptions import ValidationError as DRFValidationError+from rest_framework.test import APIClient+from posthog.egress.github.transport import GitHubEgressBudgetExhausted, GitHubRateLimitError+from posthog.egress.limiter.policies import Priorityfrom posthog.models import Team, Userfrom posthog.models.integration import (GitHubIntegration,@@ -18,14 +24,24 @@ from posthog.models.integration import (JiraIntegration,LinearIntegration,)--from products.signals.backend.models import SignalReport, SignalReportTrackerIssue, SignalTeamConfig+from posthog.redis import get_client++from products.signals.backend.implementation_pr import fetch_implementation_pr_state_for_reports+from products.signals.backend.models import (+ SignalActorKind,+ SignalReport,+ SignalReportAssignment,+ SignalReportMonitoringAttempt,+ SignalReportTrackerIssue,+ SignalTeamConfig,+)from products.signals.backend.serializers import SignalTeamConfigSerializerfrom products.signals.backend.tasks import close_report_tracker_issuefrom products.signals.backend.tracker_issues import (ABANDONED_CLAIM_REASON,PR_BODY_MARKER,close_tracker_issue_for_report,+ complete_monitoring_tracker_issues,create_tracker_issue_for_report,issue_reference,link_pull_request_to_tracker_issue,@@ -264,10 +280,54 @@ def test_link_pull_request_backs_off_when_the_body_changes_during_the_edit(team)@pytest.mark.django_db-def test_close_tracker_issue_is_recorded_once(team):+ "delivery",+ [+ "sequential",+ "overlapping",+ "monitoring",+ "reopened",+ "replaced_attempt",+ "archived",+ "manual",+ "provider_retry",+ "failed_comment_retry",+ "reopened_during_pr",+ "reopened_during_comment",+ ],+)+def test_close_tracker_issue_is_recorded_once(team: Team, delivery: str) -> None:integration = _connect_tracker(team, "github")report = _make_report(team)- SignalReportTrackerIssue.all_teams.create(+ attempt_id = None+ if delivery in {+ "monitoring",+ "reopened",+ "replaced_attempt",+ "archived",+ "manual",+ "reopened_during_pr",+ "reopened_during_comment",+ }:+ started_at = timezone.now() - timedelta(hours=1)+ attempt = SignalReportMonitoringAttempt.objects.for_team(team.id).create(+ team=team,+ report=report,+ monitoring_started_at=started_at,+ aggregation_state="passed",+ close_pr_on_resolve=delivery in {"manual", "reopened_during_pr"},+ )+ attempt_id = str(attempt.id)+ SignalReport.objects.filter(id=report.id).update(+ status=SignalReport.Status.READY if delivery == "reopened" else SignalReport.Status.RESOLVED,+ monitoring_started_at=timezone.now() if delivery == "replaced_attempt" else started_at,+ )+ if delivery == "archived":+ SignalReport.objects.filter(id=report.id).update(+ status=SignalReport.Status.SUPPRESSED,+ status_before_suppression=SignalReport.Status.RESOLVED,+ )+ tracker = SignalReportTrackerIssue.all_teams.create(team=team,report=report,integration=integration,@@ -276,11 +336,290 @@ def test_close_tracker_issue_is_recorded_once(team):external_context={"repository": "web", "number": 12},)- with patch.object(GitHubIntegration, "close_issue") as close_issue:- assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is True+ def overlapping_close(*args: object, **kwargs: object) -> None:assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is False- close_issue.assert_called_once_with("web", 12, completed=False)+ def reopen_report(*args: object, **kwargs: object) -> dict[str, bool]:+ SignalReport.objects.filter(team_id=team.id, id=report.id).update(status=SignalReport.Status.READY)+ return {"success": True}++ with (+ patch.object(GitHubIntegration, "close_issue") as close_issue,+ patch.object(GitHubIntegration, "comment_on_issue", return_value={"success": True}) as comment,+ patch(+ "products.signals.backend.tracker_issues.close_implementation_pr_for_report", return_value=True+ ) as close_pr,+ ):+ if delivery == "overlapping":+ close_issue.side_effect = overlapping_close+ if delivery == "reopened_during_pr":+ close_pr.side_effect = reopen_report+ if delivery == "reopened_during_comment":+ comment.side_effect = reopen_report+ if delivery in {"provider_retry", "failed_comment_retry"}:+ close_issue.side_effect = [RuntimeError("Provider unavailable"), None]+ if delivery == "failed_comment_retry":+ comment.side_effect = [{"success": False}, {"success": True}]+ assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is False+ tracker.refresh_from_db()+ assert (tracker.close_comment_posted_at is None) == (delivery == "failed_comment_retry")+ assert (+ close_tracker_issue_for_report(+ team_id=team.id,+ report_id=str(report.id),+ monitoring_attempt_id=attempt_id,+ completed=attempt_id is not None,+ )+ is True+ )+ if delivery == "manual":+ close_pr.assert_called_once_with(+ team.id,+ str(report.id),+ reason="resolved",+ actor_user_id=None,+ comment_marker=f"<!-- monitoring-completion:{attempt_id} -->",+ before_close=ANY,+ priority=Priority.CRITICAL,+ require_all=True,+ )+ elif delivery != "reopened_during_pr":+ close_pr.assert_not_called()+ if delivery in {"reopened", "replaced_attempt", "reopened_during_pr"}:+ close_issue.assert_not_called()+ comment.assert_not_called()+ elif delivery == "reopened_during_comment":+ close_issue.assert_not_called()+ comment.assert_called_once()+ else:+ assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is False+ if delivery in {"provider_retry", "failed_comment_retry"}:+ assert close_issue.call_count == 2+ else:+ close_issue.assert_called_once_with("web", 12, completed=attempt_id is not None)+ assert comment.call_count == (2 if delivery == "failed_comment_retry" else 1)+++ ("failure", "with_tracker"),+ [+ ("lock", False),+ ("redis", False),+ ("pr_close", False),+ ("pr_close", True),+ ("already_closed", False),+ ("budget_probe", False),+ ("rate_pr_close", False),+ ("budget_tracker_close", True),+ ("rate_tracker_comment", True),+ ("tracker_close", True),+ ],+)+def test_monitoring_completion_retries_until_all_delivery_finishes(+ team: Team, failure: str, with_tracker: bool+) -> None:+ integration = _connect_tracker(team, "github")+ report = _make_report(team)+ now = timezone.now()+ started_at = now - timedelta(hours=1)+ SignalReport.objects.filter(team_id=team.id, id=report.id).update(+ status=SignalReport.Status.RESOLVED, monitoring_started_at=started_at+ )+ attempt = SignalReportMonitoringAttempt.objects.for_team(team.id).create(+ team=team,+ report=report,+ monitoring_started_at=started_at,+ aggregation_state="passed",+ close_pr_on_resolve=True,+ tracker_issue_close_requested_at=now,+ )+ SignalReportAssignment.objects.for_team(team.id).create(+ team=team,+ report=report,+ actor_kind=SignalActorKind.SYSTEM,+ pr_url="https://github.com/acme/web/pull/50",+ repository="acme/web",+ pr_number=50,+ pr_state=SignalReportAssignment.PrState.OPEN,+ )+ tracker = (+ SignalReportTrackerIssue.objects.for_team(team.id).create(+ team=team,+ report=report,+ integration=integration,+ provider="github",+ status=SignalReportTrackerIssue.Status.CREATED,+ external_context={"repository": "web", "number": 12},+ )+ if with_tracker+ else None+ )+ waiting_attempt = (+ SignalReportMonitoringAttempt.objects.for_team(team.id).create(+ team=team,+ report=_make_report(team),+ monitoring_started_at=started_at,+ aggregation_state="passed",+ tracker_issue_close_requested_at=now + timedelta(seconds=1),+ )+ if failure.startswith(("budget_", "rate_"))+ else None+ )+ pr_closed = failure == "already_closed"+ commented = False+ failed = False+ calls: list[str] = []+ priorities: list[Priority] = []++ def deliver(client: GitHubIntegration, stage: str) -> None:+ nonlocal failed+ priorities.append(client.priority)+ calls.append(stage)+ if not failed and failure.removeprefix("budget_").removeprefix("rate_") == stage:+ failed = True+ if failure.startswith("budget_"):+ raise GitHubEgressBudgetExhausted("Budget unavailable")+ if failure.startswith("rate_"):+ if failure == "rate_tracker_comment":+ raise GitHubRateLimitError("Provider limited", reset_at=int(now.timestamp()) + 900)+ raise GitHubRateLimitError("Provider limited", retry_after=900)+ raise RuntimeError("Provider unavailable")++ def probe(client: GitHubIntegration, repository: str, *, raise_on_transient: bool) -> bool:+ deliver(client, "probe")+ return True++ def status(client: GitHubIntegration, repository: str, number: int) -> dict[str, object]:+ deliver(client, "status")+ return {"success": True, "state": "closed" if pr_closed else "open", "merged": False}++ def find_comment(client: GitHubIntegration, repository: str, number: int, marker: str) -> bool:+ deliver(client, "find_comment")+ return commented++ def comment_pr(client: GitHubIntegration, repository: str, number: int, body: str) -> dict[str, bool]:+ nonlocal commented+ deliver(client, "pr_comment")+ commented = True+ return {"success": True}++ def close_pr(client: GitHubIntegration, repository: str, number: int) -> dict[str, bool]:+ nonlocal pr_closed+ deliver(client, "pr_close")+ pr_closed = True+ return {"success": True}++ def comment_tracker(client: GitHubIntegration, repository: str, number: int, body: str) -> dict[str, bool]:+ deliver(client, "tracker_comment")+ return {"success": True}++ def close_tracker(client: GitHubIntegration, repository: str, number: int, *, completed: bool) -> None:+ deliver(client, "tracker_close")+ assert completed++ with (+ time_machine.travel(now, tick=False),+ patch.object(GitHubIntegration, "installation_can_access_repository", autospec=True, side_effect=probe),+ patch.object(GitHubIntegration, "get_pull_request", autospec=True, side_effect=status),+ patch.object(GitHubIntegration, "has_pull_request_comment", autospec=True, side_effect=find_comment),+ patch.object(GitHubIntegration, "comment_on_pull_request", autospec=True, side_effect=comment_pr),+ patch.object(GitHubIntegration, "close_pull_request", autospec=True, side_effect=close_pr),+ patch.object(GitHubIntegration, "comment_on_issue", autospec=True, side_effect=comment_tracker),+ patch.object(GitHubIntegration, "close_issue", autospec=True, side_effect=close_tracker),+ ):+ if failure == "lock":+ with get_client().lock(f"signals:tracker-close:{team.id}:{report.id}", timeout=120):+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ elif failure == "redis":+ with patch("products.signals.backend.tracker_issues.get_client", side_effect=RedisConnectionError()):+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ else:+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ attempt.refresh_from_db()+ if waiting_attempt is not None:+ waiting_attempt.refresh_from_db()+ assert waiting_attempt.tracker_issue_close_processed_at is None+ if failure != "already_closed":+ assert attempt.tracker_issue_close_processed_at is None+ retry_at = now + timedelta(minutes=15 if failure.startswith("rate_") else 5)+ assert attempt.tracker_issue_close_next_attempt_at == retry_at+ call_count = len(calls)+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ assert len(calls) == call_count+ with time_machine.travel(retry_at, tick=False):+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ attempt.refresh_from_db()+ assert attempt.tracker_issue_close_processed_at is not None+ pr_state = fetch_implementation_pr_state_for_reports([str(report.id)], team_id=team.id)[str(report.id)]+ assert pr_state.state == "closed"+ if tracker is not None:+ tracker.refresh_from_db()+ assert tracker.closed_at is not None+ assert calls.count("pr_comment") <= 1+ assert priorities and all(priority == Priority.BATCH for priority in priorities)+ call_count = len(calls)+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)+ assert len(calls) == call_count++[email protected]("bulk", [False, True])+def test_reopening_waits_for_tracker_delivery(team: Team, bulk: bool) -> None:+ integration = _connect_tracker(team, "github")+ user = User.objects.create_and_join(team.organization, "[email protected]", None)+ client = APIClient()+ client.force_authenticate(user=user)+ report = _make_report(team)+ started_at = timezone.now() - timedelta(hours=1)+ SignalReport.objects.filter(team_id=team.id, id=report.id).update(+ status=SignalReport.Status.RESOLVED, monitoring_started_at=started_at+ )+ attempt = SignalReportMonitoringAttempt.objects.for_team(team.id).create(+ team=team,+ report=report,+ monitoring_started_at=started_at,+ aggregation_state="passed",+ tracker_issue_close_requested_at=timezone.now(),+ )+ SignalReportTrackerIssue.all_teams.create(+ team=team,+ report=report,+ integration=integration,+ provider="github",+ status=SignalReportTrackerIssue.Status.CREATED,+ external_context={"repository": "web", "number": 12},+ )+ url = (+ f"/api/projects/{team.id}/signals/reports/bulk-state/"+ if bulk+ else f"/api/projects/{team.id}/signals/reports/{report.id}/state/"+ )+ body: dict[str, object] = {"state": "potential"}+ if bulk:+ body["ids"] = [str(report.id)]++ def try_reopening(*args: object, **kwargs: object) -> None:+ response = client.post(url, body, format="json")+ assert response.status_code == (200 if bulk else 409), response.json()+ if bulk:+ assert response.json()["results"][0]["outcome"] == "skipped"+ report.refresh_from_db()+ assert report.status == SignalReport.Status.RESOLVED++ with (+ patch.object(GitHubIntegration, "comment_on_issue", return_value={"success": True}),+ patch.object(GitHubIntegration, "close_issue", side_effect=try_reopening) as close_issue,+ ):+ assert close_tracker_issue_for_report(+ team_id=team.id, report_id=str(report.id), completed=True, monitoring_attempt_id=str(attempt.id)+ )+ close_issue.assert_called_once()++ response = client.post(url, body, format="json")+ assert response.status_code == 200, response.json()+ report.refresh_from_db()+ assert report.status == SignalReport.Status.POTENTIAL@pytest.mark.django_db@@ -339,7 +678,8 @@ def test_completed_report_closes_tracker_issue_as_completed(team):@pytest.mark.django_db@pytest.mark.parametrize("status", [SignalReportTrackerIssue.Status.PENDING, SignalReportTrackerIssue.Status.CREATED])-def test_close_tracker_issue_task_retries_pending_or_failed_provider_close(team, status):[email protected]("failure", ["provider", "redis"])+def test_close_tracker_issue_task_retries_pending_or_failed_provider_close(team, status, failure):integration = _connect_tracker(team, "github")report = _make_report(team)SignalReportTrackerIssue.all_teams.create(@@ -351,7 +691,13 @@ def test_close_tracker_issue_task_retries_pending_or_failed_provider_close(team,external_context={"repository": "web", "number": 12},)- with patch("products.signals.backend.tasks.close_tracker_issue_for_report", return_value=False):+ with (+ patch("products.signals.backend.tasks.close_tracker_issue_for_report", return_value=False)+ if failure == "provider"+ else patch(+ "products.signals.backend.tracker_issues.get_client", side_effect=RedisConnectionError("Unavailable")+ )+ ):with pytest.raises(Retry):close_report_tracker_issue.apply(kwargs={"report_id": str(report.id), "team_id": team.id, "completed": False},diff --git a/products/signals/backend/tracker_issues.py b/products/signals/backend/tracker_issues.pyindex 2c2ef2c779cb..f2203adb7839 100644--- a/products/signals/backend/tracker_issues.py+++ b/products/signals/backend/tracker_issues.py@@ -10,15 +10,21 @@ recorded as a failure on the report instead, so the team can see the gap and fixfrom __future__ import annotationsimport re+import time+from collections.abc import Iterator+from contextlib import contextmanagerfrom datetime import timedeltafrom typing import Any+from django.db import modelsfrom django.utils import timezoneimport structlog+from redis.exceptions import RedisErrorfrom rest_framework.exceptions import ValidationError as DRFValidationErrorfrom posthog.dataclasses import frozen+from posthog.egress.github.transport import GitHubEgressBudgetExhausted, GitHubRateLimitErrorfrom posthog.egress.limiter.policies import Priorityfrom posthog.models.integration import (SUPPORTED_EXTERNAL_ISSUE_PROVIDERS,@@ -29,9 +35,16 @@ from posthog.models.integration import (LinearIntegration,external_issue_url,)+from posthog.redis import get_clientfrom products.signals.backend.github_actor import github_mention_for_user-from products.signals.backend.models import SignalReport, SignalReportTrackerIssue, SignalTeamConfig+from products.signals.backend.implementation_pr import close_implementation_pr_for_report+from products.signals.backend.models import (+ SignalReport,+ SignalReportMonitoringAttempt,+ SignalReportTrackerIssue,+ SignalTeamConfig,+)from products.signals.backend.pull_request_body import BodyEditOutcome, edit_pull_request_bodyfrom products.signals.backend.report_urls import ReportLinkSource, build_report_url@@ -350,36 +363,48 @@ def _close_notice(*, report_link: str, actor_mention: str | None) -> str:return f"🔕 The {report_link} was closed in PostHog, so this issue is closed too."-def _comment_before_close(tracker: SignalReportTrackerIssue, *, report_link: str, actor_mention: str | None) -> None:+def _comment_before_close(+ tracker: SignalReportTrackerIssue, *, report_link: str, actor_mention: str | None, priority: Priority+) -> None:"""Best-effort note on the issue naming who closed the report. Only GitHub takes one today.Explain first, close second, as the implementation pull request does. A failed comment mustnot stop the close: an issue left open would grow the backlog this close exists to prevent,while a missing comment costs only the attribution."""- if tracker.provider != Integration.IntegrationKind.GITHUB or tracker.integration is None:+ if (+ tracker.provider != Integration.IntegrationKind.GITHUB+ or tracker.integration is None+ or tracker.close_comment_posted_at is not None+ ):returncontext = tracker.external_context or {}try:- GitHubIntegration(+ result = GitHubIntegration(tracker.integration, source="signals_tracker_issues", priority=Priority.BATCH).comment_on_issue(context["repository"],int(context["number"]),_close_notice(report_link=report_link, actor_mention=actor_mention),)+ if result.get("success"):+ tracker.close_comment_posted_at = timezone.now()+ tracker.save(update_fields=["close_comment_posted_at", "updated_at"])+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):+ if priority == Priority.BATCH:+ raiseexcept Exception:logger.warning("signals.tracker_issue_close_comment_failed", tracker_id=str(tracker.id), exc_info=True)-def _close_provider_issue(tracker: SignalReportTrackerIssue, *, completed: bool) -> None:+def _close_provider_issue(tracker: SignalReportTrackerIssue, *, completed: bool, priority: Priority) -> None:integration = tracker.integrationif integration is None:raise ValueError("Tracker issue has no integration left to close it through")context = tracker.external_context or {}if tracker.provider == Integration.IntegrationKind.GITHUB:- GitHubIntegration(integration, source="signals_tracker_issues", priority=Priority.CRITICAL).close_issue(+ GitHubIntegration(integration, source="signals_tracker_issues", priority=priority).close_issue(context["repository"], int(context["number"]), completed=completed)elif tracker.provider == Integration.IntegrationKind.GITLAB:@@ -392,34 +417,197 @@ def _close_provider_issue(tracker: SignalReportTrackerIssue, *, completed: bool)raise ValueError(f"Unsupported tracker provider {tracker.provider}")-def close_tracker_issue_for_report(- *, team_id: int, report_id: str, completed: bool = False, actor_user_id: int | None = None+def _current_completion_attempt(+ *, team_id: int, report_id: str, monitoring_attempt_id: str+) -> SignalReportMonitoringAttempt | None:+ return (+ SignalReportMonitoringAttempt.objects.for_team(team_id)+ .filter(+ id=monitoring_attempt_id,+ report_id=report_id,+ aggregation_state="passed",+ tracker_issue_close_processed_at__isnull=True,+ monitoring_started_at=models.F("report__monitoring_started_at"),+ )+ .filter(+ models.Q(report__status=SignalReport.Status.RESOLVED)+ | models.Q(+ report__status=SignalReport.Status.SUPPRESSED,+ report__status_before_suppression=SignalReport.Status.RESOLVED,+ )+ )+ .first()+ )+++def _close_tracker_issue_for_report(+ *,+ team_id: int,+ report_id: str,+ completed: bool,+ actor_user_id: int | None,+ monitoring_attempt_id: str | None,+ priority: Priority,) -> bool:- """Close the report's tracker issue once the report is dismissed or completed. Never raises.+ """Close the report's tracker issue once the report is dismissed or completed.A dismissed report will not produce a pull request, so its work item is finished. Leaving it- open would grow a backlog of tracker issues that no pull request will ever answer.+ open would grow a backlog of tracker issues that no pull request will ever answer. Batch rate+ limits propagate so the sweep can honor the provider's retry window."""++ def can_complete() -> bool:+ return (+ monitoring_attempt_id is None+ or _current_completion_attempt(+ team_id=team_id, report_id=report_id, monitoring_attempt_id=monitoring_attempt_id+ )+ is not None+ )+try:+ pr_completed = True+ if monitoring_attempt_id is not None:+ attempt = _current_completion_attempt(+ team_id=team_id, report_id=report_id, monitoring_attempt_id=monitoring_attempt_id+ )+ if attempt is None:+ return True+ if attempt.close_pr_on_resolve:+ pr_completed = close_implementation_pr_for_report(+ team_id,+ report_id,+ reason="resolved",+ actor_user_id=actor_user_id,+ comment_marker=f"<!-- monitoring-completion:{attempt.id} -->",+ before_close=can_complete,+ priority=priority,+ require_all=True,+ )tracker = (SignalReportTrackerIssue.objects.for_team(team_id).select_related("integration")- .filter(report_id=report_id, status=SignalReportTrackerIssue.Status.CREATED)+ .filter(+ report_id=report_id,+ status__in=[SignalReportTrackerIssue.Status.PENDING, SignalReportTrackerIssue.Status.CREATED],+ ).first())if tracker is None or tracker.closed_at is not None:+ return monitoring_attempt_id is not None and pr_completed+ if tracker.status == SignalReportTrackerIssue.Status.PENDING:return False+ if not can_complete():+ return Truereport_url = build_report_url(team_id, report_id, ReportLinkSource.TRACKER)_comment_before_close(tracker,report_link=f"[linked PostHog report]({report_url})",actor_mention=github_mention_for_user(actor_user_id),+ priority=priority,)- _close_provider_issue(tracker, completed=completed)+ if not can_complete():+ return True+ _close_provider_issue(tracker, completed=completed, priority=priority)tracker.closed_at = timezone.now()tracker.save(update_fields=["closed_at", "updated_at"])- return True+ return pr_completed+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):+ if priority == Priority.BATCH:+ raise+ return Falseexcept Exception:logger.warning("signals.tracker_issue_close_failed", report_id=report_id, team_id=team_id, exc_info=True)return False+++@contextmanager+def tracker_issue_completion_lock(*, team_id: int, report_id: str) -> Iterator[bool]:+ lock = None+ acquired = False+ try:+ lock = get_client().lock(f"signals:tracker-close:{team_id}:{report_id}", timeout=120)+ acquired = lock.acquire(blocking=False)+ except RedisError:+ logger.warning("signals.tracker_issue_close_lock_failed", report_id=report_id, team_id=team_id, exc_info=True)+ yield False+ return+ try:+ yield acquired+ finally:+ if acquired and lock is not None:+ try:+ lock.release()+ except RedisError:+ logger.warning("signals.tracker_issue_close_lock_release_failed", report_id=report_id, exc_info=True)+++def close_tracker_issue_for_report(+ *,+ team_id: int,+ report_id: str,+ completed: bool = False,+ actor_user_id: int | None = None,+ monitoring_attempt_id: str | None = None,+ priority: Priority = Priority.CRITICAL,+) -> bool:+ with tracker_issue_completion_lock(team_id=team_id, report_id=report_id) as acquired:+ if not acquired:+ return False+ return _close_tracker_issue_for_report(+ team_id=team_id,+ report_id=report_id,+ completed=completed,+ actor_user_id=actor_user_id,+ monitoring_attempt_id=monitoring_attempt_id,+ priority=priority,+ )+++def complete_monitoring_tracker_issues(*, deadline: float, limit: int = 100) -> None:+ now = timezone.now()+ due = models.Q(tracker_issue_close_next_attempt_at__isnull=True) | models.Q(+ tracker_issue_close_next_attempt_at__lte=now+ )+ attempts = (+ SignalReportMonitoringAttempt.objects.unscoped()+ .filter(tracker_issue_close_requested_at__isnull=False, tracker_issue_close_processed_at__isnull=True)+ .filter(due)+ .order_by(+ models.F("tracker_issue_close_next_attempt_at").asc(nulls_first=True),+ "tracker_issue_close_requested_at",+ "id",+ )[:limit]+ )+ for attempt in attempts:+ if time.monotonic() >= deadline:+ break+ claimed = (+ SignalReportMonitoringAttempt.objects.for_team(attempt.team_id)+ .filter(id=attempt.id, tracker_issue_close_processed_at__isnull=True)+ .filter(due)+ .update(tracker_issue_close_next_attempt_at=now + timedelta(minutes=5))+ )+ if not claimed:+ continue+ try:+ completed = close_tracker_issue_for_report(+ team_id=attempt.team_id,+ report_id=str(attempt.report_id),+ completed=True,+ monitoring_attempt_id=str(attempt.id),+ priority=Priority.BATCH,+ )+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError) as error:+ retry_after = 300+ if isinstance(error, GitHubRateLimitError):+ retry_after = max(retry_after, error.retry_after or 0, (error.reset_at or 0) - int(time.time()))+ SignalReportMonitoringAttempt.objects.for_team(attempt.team_id).filter(+ id=attempt.id, tracker_issue_close_processed_at__isnull=True+ ).update(tracker_issue_close_next_attempt_at=timezone.now() + timedelta(seconds=retry_after))+ return+ if completed:+ SignalReportMonitoringAttempt.objects.for_team(attempt.team_id).filter(id=attempt.id).update(+ tracker_issue_close_processed_at=timezone.now()+ )diff --git a/products/signals/backend/views.py b/products/signals/backend/views.pyindex 8023ea0611ea..c56114552d79 100644--- a/products/signals/backend/views.py+++ b/products/signals/backend/views.py@@ -3,6 +3,7 @@ import jsonimport mathimport uuidfrom collections.abc import Callable, Iterable, Sequence+from contextlib import ExitStackfrom datetime import UTC, timedeltafrom functools import cached_property, partialfrom typing import Any, Literal, cast@@ -148,6 +149,7 @@ from products.signals.backend.models import (SignalReportAction,SignalReportArtefact,SignalReportCheck,+ SignalReportMonitoringAttempt,SignalReportRefund,SignalScoutConfig,SignalSourceConfig,@@ -322,6 +324,7 @@ from products.signals.backend.throttles import (ReportTaskActionRunBurstThrottle,ReportTaskActionRunSustainedThrottle,)+from products.signals.backend.tracker_issues import tracker_issue_completion_lockfrom products.tasks.backend.facade import api as tasks_facadefrom products.tasks.backend.facade.access import code_access_required_response, usage_limit_responsefrom products.tasks.backend.facade.repo_selection_types import RepoSelectionResult@@ -4049,7 +4052,8 @@ class SignalReportViewSet(if target_status == SignalReport.Status.MONITORING and not self._report_monitoring_enabled:return SignalReportBulkStateOutcome.SKIPPED, "Monitoring is unavailable for this organization."- with transaction.atomic():+ # Release the delivery lock after commit so workers cannot read the previous verdict.+ with ExitStack() as completion_locks, transaction.atomic():# Lock the report row for the whole guard + transition so it serializes with the refund# path (which locks the same row before suppressing). Re-read committed state under the# lock: a resolve/restore that raced a refund must observe the refund's SUPPRESSED status@@ -4060,6 +4064,23 @@ class SignalReportViewSet(is not None):report.refresh_from_db()+ if (+ SignalReportMonitoringAttempt.objects.for_team(self.team.id)+ .filter(+ report_id=report.id,+ tracker_issue_close_requested_at__isnull=False,+ tracker_issue_close_processed_at__isnull=True,+ )+ .exists()+ ):+ acquired = completion_locks.enter_context(+ tracker_issue_completion_lock(team_id=self.team.id, report_id=str(report.id))+ )+ if not acquired:+ return (+ SignalReportBulkStateOutcome.SKIPPED,+ "This report is being finalized. Try again in a moment.",+ )if dismissal_reason == "verified_manually" and (not self._report_monitoring_enabledor report.monitoring_started_at is None