Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(signals): complete verified tracker issues durably (#124560)

Mikayla Thompson00b666d90215
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.md
index 630a89965ce8..b9854b582991 100644
--- a/docs/internal/signals-pr-lifecycle.md
+++ b/docs/internal/signals-pr-lifecycle.md
@@ -147,6 +147,14 @@ The Monitoring schema includes both outcome and tracker-delivery metadata. Nulla
This aggregation path persists requests only. The report-level assessment worker and dispatch/execution of follow-up research remain separate work; a research request does not claim that a follow-up report has been created.
+All-pass resolution also records a durable tracker-completion request. A separate one-minute sweep consumes pending requests directly, outside the check coordinator and its time budget. It has a 45-second soft limit and 60-second hard limit. Tracker closure serializes deliveries and verifies the same attempt is still effectively resolved before contacting the provider; archiving preserves that verified outcome, while reopening or replacing the attempt makes its request obsolete. Each claimed request reserves its next attempt five minutes ahead, so failures and interrupted workers cannot monopolize every sweep. Successful tracker closing comments are recorded independently of the provider close, and Redis lock failures use the normal retry path. Completed or obsolete requests are marked processed, and a partial index orders outstanding work by its next retry.
+
+Manual completion preserves its open-PR closure intent on the frozen attempt. After verification passes, the sweep uses the existing best-effort PR closure path with a stable comment marker, then completes the tracker issue.
+
+The request stays pending until every requested PR close and tracker close succeeds or is no longer needed, including reports without a tracker issue. Scheduled GitHub delivery uses the batch lane. Budget denial stops the sweep, and provider rate limits delay the request until at least the indicated retry time.
+
+Single and bulk report-state requests share the delivery lock through their transaction commit. A request that overlaps completion is refused with a retry message. Delivery revalidates the frozen attempt before closing each PR and again after any intervening provider call. A tracker comment is recorded as delivered only when its provider reports success.
+
With `signals-report-metrics` enabled, research gathers short-window baselines before verification and prefers metric-based follow-up checks. It aims for practical 70-80% confidence in improvement, without claiming a calculated confidence score. Observed volume, recurrence, ingestion delay, and deployment timing justify the shortest useful query and wait. Frequent or high-volume checks can use 4-8 hours, allowing for several hours of deployment delay before the fix reaches users. Around 24 hours or several days require the corresponding activity or rollout evidence. Agent checks remain available when interpretation or unsupported measurements prevent a useful metric check. Missing activity is not success. These prompt preferences do not change execution, schema defaults, or final outcomes.
The server refuses new Monitoring definitions when `signals-report-monitoring` is disabled or unavailable and omits the definition from check responses. Stored definitions remain intact. Research carries its definition-visibility decision into reconciliation, so flag recovery during a run does not replace an unchanged check or reset its approval and schedule. Existing definitions, charts, check schedules, and Resolved outcomes otherwise follow their current behavior.
diff --git a/posthog/egress/github/README.md b/posthog/egress/github/README.md
index f4bcb23362fb..d0402a026c90 100644
--- a/posthog/egress/github/README.md
+++ b/posthog/egress/github/README.md
@@ -30,7 +30,7 @@ Use one stable snake_case `source` per subsystem, so the egress metrics show who
Deferrable background callers construct their client on the `BATCH` lane (`GitHubIntegration(integration, source=..., priority=Priority.BATCH)`; `api_request` also takes a per-call override).
A shed sweep stops for the cycle and resumes on the next scheduled run.
A caller that walks pages, such as the warehouse source, paces with `github_installation_pace_seconds` instead of getting denied.
-The warehouse source's page fetches run on `BATCH`, while its repository validation and webhook management run on `NORMAL`, because a person waits on them. Customer Analytics feature-request link and resume lookups also run on `NORMAL` because an editor waits for the current issue state. Signals refreshes the stored pull request review decision from GitHub webhooks on `BATCH`; the inbox can keep the last known value when the lane is shed and update it after a retry.
+The warehouse source's page fetches run on `BATCH`, while its repository validation and webhook management run on `NORMAL`, because a person waits on them. Customer Analytics feature-request link and resume lookups also run on `NORMAL` because an editor waits for the current issue state. Signals refreshes the stored pull request review decision from GitHub webhooks on `BATCH`; the inbox can keep the last known value when the lane is shed and update it after a retry. Its scheduled Monitoring completion sweep also closes linked PRs and tracker issues on `BATCH`, retaining completion requests and deferring the sweep when the installation budget or provider rate limit denies delivery.
Self-Driving onboarding probes one repository per installation on `NORMAL` during activation and on `BATCH` during background completion checks. Failed probes back off without exhausting the completion polling window.
The GitHub-optional Self-driving pilot checks implementation repository access (source `signals_github_optional`) on `NORMAL` for manual PR creation and on `BATCH` for automatic implementation. Unavailable access blocks implementation while research from project data remains available.
The ownership file reader (`posthog/ownership/`, source `ownership_github_api`) runs on `NORMAL` where a person waits for a board or a snapshot list, and on `BATCH` for the scheduled digests and report generation. Its GraphQL reads charge `core`, as every GraphQL call does, and so does the one compare call per new head commit that lets it reuse cached files.
diff --git a/posthog/tasks/scheduled.py b/posthog/tasks/scheduled.py
index 4cd608b7e228..733f7f173cf7 100644
--- a/posthog/tasks/scheduled.py
+++ b/posthog/tasks/scheduled.py
@@ -126,6 +126,7 @@ from products.signals.backend.tasks import (
refresh_signal_scout_background_bands,
sweep_first_report_notifications,
sweep_implementation_dispatches,
+ sweep_monitoring_tracker_issues,
sync_pending_signals_refund_credits,
)
from products.skills.backend.tasks import sync_community_skills
@@ -450,6 +451,14 @@ def setup_periodic_tasks(sender: Celery, **kwargs: Any) -> None:
expires_seconds=5 * 60,
)
+ add_periodic_task_with_expiry(
+ sender,
+ crontab(minute="*"),
+ sweep_monitoring_tracker_issues.s(),
+ name="complete verified signals tracker issues",
+ expires_seconds=60,
+ )
+
# Re-enqueue signals PR refunds whose billing credit sync hasn't landed - hourly at minute 25
sender.add_periodic_task(
crontab(hour="*", minute="25"),
diff --git a/products/signals/backend/implementation_pr.py b/products/signals/backend/implementation_pr.py
index 61fbf31244af..11f88f46f278 100644
--- a/products/signals/backend/implementation_pr.py
+++ b/products/signals/backend/implementation_pr.py
@@ -13,6 +13,7 @@ from django.db.models import F, Lookup, Q, QuerySet
import structlog
from posthog.dataclasses import frozen
+from posthog.egress.github.transport import GitHubEgressBudgetExhausted, GitHubRateLimitError
from posthog.egress.limiter.policies import Priority
from posthog.models.github_integration_base import GitHubIntegrationBase
from posthog.models.integration import GitHubIntegration
@@ -427,7 +428,6 @@ def _pr_close_comment(
if actor_mention
else f"🔕 Closing this PR because the {report_link} was resolved without it."
)
- # A resolved report never reopens, so the restore advice below does not apply to it.
return f"{opening}\n\nIf that wasn't intended, reopen this PR."
opening = (
f"🔕 Closing this PR because {actor_mention} {reason} the {report_link}."
@@ -448,15 +448,15 @@ def _close_implementation_pr(
expected_head_sha: str | None = None,
comment_marker: str | None = None,
before_close: Callable[[], bool] | None = None,
-) -> bool:
+ priority: Priority = Priority.CRITICAL,
+) -> bool | None:
"""Best-effort: comment on and close the GitHub PR attached to this report.
Called when a report is suppressed, snoozed, or resolved without its PR — the open PR shouldn't linger. Only acts on a PR
that is still open: an already-closed or merged PR is left untouched (no comment, no close), so
we never leave a confusing "closing this PR" note on a PR that shipped months ago. Leaves an
- explanatory comment, then closes the PR. Returns True when the PR was closed, False when there
- was nothing to close or the close couldn't be completed. Never raises: the state transition
- must succeed regardless.
+ explanatory comment, then closes the PR. Returns True when closed, None when no close is
+ needed, and False on failure. Batch rate limits propagate so scheduled delivery can back off.
"""
try:
if not SignalReport.objects.filter(id=report_id, team_id=team_id).exists():
@@ -467,7 +467,7 @@ def _close_implementation_pr(
report_id=str(report_id),
actor_kind=pr.actor_kind,
)
- return False
+ return None
pr_url = pr.url
parsed = GitHubIntegrationBase.parse_pull_request_url(pr_url)
@@ -483,10 +483,10 @@ def _close_implementation_pr(
report_id=str(report_id),
pr_url=pr_url,
)
- return False
+ return None
github = GitHubIntegration.first_for_team_repository(
- team_id, parsed.repository, source="signals_implementation_pr", priority=Priority.CRITICAL
+ team_id, parsed.repository, source="signals_implementation_pr", priority=priority
)
if github is None:
logger.info(
@@ -523,7 +523,7 @@ def _close_implementation_pr(
state=pr_status.get("state"),
merged=pr_status.get("merged"),
)
- return False
+ return None if pr_status.get("state") == "closed" or pr_status.get("merged") else False
if expected_head_sha is not None and pr_status.get("head_sha") != expected_head_sha:
return False
@@ -575,6 +575,10 @@ def _close_implementation_pr(
pr_state="closed",
)
return True
+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):
+ if priority == Priority.BATCH:
+ raise
+ return False
except Exception:
logger.exception("close_implementation_pr_unexpected_error", report_id=str(report_id))
return False
@@ -588,26 +592,37 @@ def close_implementation_pr_for_report(
actor_user_id: int | None = None,
pr_url: str | None = None,
replacement_pr_url: str | None = None,
+ comment_marker: str | None = None,
+ before_close: Callable[[], bool] | None = None,
+ priority: Priority = Priority.CRITICAL,
+ require_all: bool = False,
) -> bool:
try:
if not SignalReport.objects.filter(team_id=team_id, id=report_id).exists():
return False
closed = False
+ failed = False
for pr in fetch_implementation_prs_for_reports([str(report_id)], team_id=team_id).get(str(report_id), []):
if pr_url is not None and pr.url != pr_url:
continue
- closed = (
- _close_implementation_pr(
- team_id,
- report_id,
- reason=reason,
- pr=pr,
- actor_user_id=actor_user_id,
- replacement_pr_url=replacement_pr_url,
- )
- or closed
+ outcome = _close_implementation_pr(
+ team_id,
+ report_id,
+ reason=reason,
+ pr=pr,
+ actor_user_id=actor_user_id,
+ replacement_pr_url=replacement_pr_url,
+ comment_marker=comment_marker,
+ before_close=before_close,
+ priority=priority,
)
- return closed
+ closed = outcome is True or closed
+ failed = outcome is False or failed
+ return not failed if require_all else closed
+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):
+ if priority == Priority.BATCH:
+ raise
+ return False
except Exception:
logger.exception("close_implementation_pr_lookup_failed", report_id=str(report_id))
return False
diff --git a/products/signals/backend/report_monitoring_attempts.py b/products/signals/backend/report_monitoring_attempts.py
index 05d8ae9bd1ec..f3b718338012 100644
--- a/products/signals/backend/report_monitoring_attempts.py
+++ b/products/signals/backend/report_monitoring_attempts.py
@@ -120,6 +120,7 @@ def complete_implementation(
report=report,
monitoring_started_at=monitoring_started_at,
check_versions={str(check.id): check_definition_version(check) for check in applicable},
+ close_pr_on_resolve=close_pr_on_resolve,
)
SignalReportCheck.objects.for_team(team_id).filter(id__in=[check.id for check in applicable]).update(
status=SignalReportCheck.Status.PENDING,
diff --git a/products/signals/backend/tasks.py b/products/signals/backend/tasks.py
index 1822bf429ed0..f4dad19aae21 100644
--- a/products/signals/backend/tasks.py
+++ b/products/signals/backend/tasks.py
@@ -1,3 +1,4 @@
+import time
from contextlib import suppress
from datetime import timedelta
from decimal import Decimal
@@ -89,7 +90,11 @@ from products.signals.backend.support_writeback import (
post_pull_request_to_tickets,
task_can_write_support_ticket_update,
)
-from products.signals.backend.tracker_issues import close_tracker_issue_for_report, link_pull_request_to_tracker_issue
+from products.signals.backend.tracker_issues import (
+ close_tracker_issue_for_report,
+ complete_monitoring_tracker_issues,
+ link_pull_request_to_tracker_issue,
+)
from products.tasks.backend.facade.repo_activity import RepositoryCommitActivityError
logger = structlog.get_logger(__name__)
@@ -300,6 +305,8 @@ def close_dismissed_report_pr(
ignore_result=True,
bind=True,
max_retries=5,
+ soft_time_limit=45,
+ time_limit=60,
)
@with_team_scope()
def close_report_tracker_issue(
@@ -322,6 +329,12 @@ def close_report_tracker_issue(
raise self.retry(countdown=min(60 * (2**self.request.retries), 900))
+@shared_task(ignore_result=True, soft_time_limit=45, time_limit=60)
+@skip_team_scope_audit
+def sweep_monitoring_tracker_issues() -> None:
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 40)
+
+
@shared_task(
name="products.signals.backend.tasks.link_report_tracker_issues",
ignore_result=True,
diff --git a/products/signals/backend/test/test_dismissal_pr_close.py b/products/signals/backend/test/test_dismissal_pr_close.py
index 9120d06bff7e..247110de54f7 100644
--- a/products/signals/backend/test/test_dismissal_pr_close.py
+++ b/products/signals/backend/test/test_dismissal_pr_close.py
@@ -272,7 +272,8 @@ class TestCloseImplementationPrForReport(BaseTest):
github.get_pull_request.assert_called_once_with("PostHog/posthog", 456)
github.close_pull_request.assert_called_once_with("PostHog/posthog", 456)
- def test_task_claim_actor_can_close_pr(self):
+ @parameterized.expand([("current", True), ("reopened_during_comment", False)])
+ def test_task_claim_actor_can_close_pr(self, _name: str, still_current: bool) -> None:
self.assignment.actor_kind = SignalActorKind.TASK
self.assignment.actor_task_id = uuid.uuid4()
self.assignment.save(update_fields=["actor_kind", "actor_task_id", "updated_at"])
@@ -280,12 +281,27 @@ class TestCloseImplementationPrForReport(BaseTest):
github.get_pull_request.return_value = {"success": True, "state": "open", "merged": False}
github.comment_on_pull_request.return_value = {"success": True}
github.close_pull_request.return_value = {"success": True}
+ current = True
+
+ def comment(*args: object, **kwargs: object) -> dict[str, bool]:
+ nonlocal current
+ current = still_current
+ return {"success": True}
+
+ github.comment_on_pull_request.side_effect = comment
with patch(
"products.signals.backend.implementation_pr.GitHubIntegration.first_for_team_repository",
return_value=github,
):
- assert close_implementation_pr_for_report(self.team.id, str(self.report.id)) is True
+ assert (
+ close_implementation_pr_for_report(self.team.id, str(self.report.id), before_close=lambda: current)
+ is still_current
+ )
+ if still_current:
+ github.close_pull_request.assert_called_once()
+ else:
+ github.close_pull_request.assert_not_called()
@parameterized.expand(
[
@@ -582,6 +598,17 @@ class TestSupersededPrClose(BaseTest):
github.close_pull_request.return_value = {"success": True, "number": 123, "state": "closed"}
return github
+ def test_completion_waits_for_every_linked_pr(self) -> None:
+ github = self._github()
+ github.close_pull_request.side_effect = [{"success": False}, {"success": True}]
+ with patch(
+ "products.signals.backend.implementation_pr.GitHubIntegration.first_for_team_repository",
+ return_value=github,
+ ):
+ assert close_implementation_pr_for_report(self.team.id, str(self.report.id), require_all=True) is False
+ github.close_pull_request.side_effect = None
+ assert close_implementation_pr_for_report(self.team.id, str(self.report.id), require_all=True) is True
+
def test_closes_the_named_pr_and_points_at_its_replacement(self):
github = self._github()
with patch(
diff --git a/products/signals/backend/test/test_report_monitoring_aggregation.py b/products/signals/backend/test/test_report_monitoring_aggregation.py
index 11a02135b758..1d7f298a5988 100644
--- a/products/signals/backend/test/test_report_monitoring_aggregation.py
+++ b/products/signals/backend/test/test_report_monitoring_aggregation.py
@@ -13,6 +13,7 @@ from django.utils import timezone
from parameterized import parameterized
+from posthog.egress.limiter.policies import Priority
from posthog.models import Team
from products.signals.backend.artefact_attribution import ArtefactAttribution
@@ -22,7 +23,9 @@ from products.signals.backend.models import (
SignalReport,
SignalReportArtefact,
SignalReportCheck,
+ SignalReportMonitoringAttempt,
SignalReportMonitoringResult,
+ SignalReportTrackerIssue,
)
from products.signals.backend.report_monitoring_aggregation import (
MonitoringAggregation,
@@ -35,6 +38,7 @@ from products.signals.backend.report_monitoring_outcomes import (
aggregate_monitoring_attempt,
)
from products.signals.backend.report_monitoring_results import MonitoringEvaluation, save_monitoring_result
+from products.signals.backend.tracker_issues import complete_monitoring_tracker_issues
class TestMonitoringAggregation(SimpleTestCase):
@@ -152,6 +156,14 @@ class TestReportMonitoringAggregation(BaseTest):
historical.refresh_from_db()
assert historical.status == SignalReportCheck.Status.EXPIRED
self.close_tracker.assert_not_called()
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ self.close_tracker.assert_called_once_with(
+ report_id=str(self.report.id),
+ team_id=self.team.id,
+ completed=True,
+ monitoring_attempt_id=str(self.attempt.id),
+ priority=Priority.BATCH,
+ )
def test_failure_closes_immediately_and_records_research_intent_atomically(self) -> None:
earlier = SignalReportArtefact.add_log(
@@ -228,6 +240,66 @@ class TestReportMonitoringAggregation(BaseTest):
assert self.attempt.assessment_requested_at == self.now
assert self.attempt.research_requested_at == self.now
+ @parameterized.expand([("provider_failure",), ("missed_sweep",)])
+ def test_tracker_completion_recovers_after_report_resolution(self, cause: str) -> None:
+ self.close_tracker.return_value = False
+ SignalReportTrackerIssue.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ report=self.report,
+ provider="github",
+ status=SignalReportTrackerIssue.Status.CREATED,
+ )
+ with self.captureOnCommitCallbacks(execute=True):
+ save_monitoring_result(self._evaluation(0))
+ save_monitoring_result(self._evaluation(1))
+ self._aggregate()
+ assert self.attempt is not None
+ self.attempt.refresh_from_db()
+ self.report.refresh_from_db()
+ assert self.report.status == SignalReport.Status.RESOLVED
+ assert self.attempt.tracker_issue_close_requested_at == self.now
+ assert self.attempt.tracker_issue_close_processed_at is None
+ self.close_tracker.assert_not_called()
+ aggregate_due_monitoring_attempts(deadline=time.monotonic() + 30, limit=10)
+ self.close_tracker.assert_not_called()
+ if cause == "provider_failure":
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ self.attempt.refresh_from_db()
+ assert self.attempt.tracker_issue_close_processed_at is None
+ initial_calls = self.close_tracker.call_count
+ self.close_tracker.return_value = True
+ retry_at = self.now + timedelta(minutes=5) if cause == "provider_failure" else self.now
+ with time_machine.travel(retry_at, tick=False):
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ self.attempt.refresh_from_db()
+ assert self.attempt.tracker_issue_close_processed_at == retry_at
+ assert self.close_tracker.call_count == initial_calls + 1
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ assert self.close_tracker.call_count == initial_calls + 1
+
+ def test_failed_tracker_batch_does_not_starve_new_requests(self) -> None:
+ self.close_tracker.return_value = False
+ SignalReportTrackerIssue.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ report=self.report,
+ status=SignalReportTrackerIssue.Status.CREATED,
+ )
+ attempts = [
+ SignalReportMonitoringAttempt(
+ team_id=self.team.id,
+ report=self.report,
+ monitoring_started_at=self.now - timedelta(days=200 - index),
+ tracker_issue_close_requested_at=self.now - timedelta(hours=1) + timedelta(seconds=index),
+ )
+ for index in range(101)
+ ]
+ SignalReportMonitoringAttempt.objects.for_team(self.team.id).bulk_create(attempts)
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ assert self.close_tracker.call_count == 100
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ assert self.close_tracker.call_count == 101
+ assert self.close_tracker.call_args.kwargs["monitoring_attempt_id"] == str(attempts[-1].id)
+
@parameterized.expand(
[
("suppressed",),
diff --git a/products/signals/backend/test/test_report_monitoring_attempts.py b/products/signals/backend/test/test_report_monitoring_attempts.py
index 524b09bb8158..37db03f41414 100644
--- a/products/signals/backend/test/test_report_monitoring_attempts.py
+++ b/products/signals/backend/test/test_report_monitoring_attempts.py
@@ -586,7 +586,7 @@ class TestReportMonitoringAttempts(BaseTest):
patch("products.signals.backend.tasks.close_dismissed_report_pr.delay") as close_pr,
self.captureOnCommitCallbacks(execute=True),
):
- complete_implementation(
+ attempt = complete_implementation(
team_id=self.team.id,
report_id=self.report.id,
monitoring_started_at=self.now,
@@ -594,6 +594,9 @@ class TestReportMonitoringAttempts(BaseTest):
actor_user=self.user,
close_pr_on_resolve=close_pr_on_resolve,
)
+ if has_checks:
+ assert attempt is not None
+ assert attempt.close_pr_on_resolve == close_pr_on_resolve
properties = capture.call_args.kwargs["properties"]
assert properties["actor_kind"] == "user"
assert properties["actor_user_uuid"] == str(self.user.uuid)
diff --git a/products/signals/backend/test/test_tracker_issues.py b/products/signals/backend/test/test_tracker_issues.py
index 2161a0acf4f3..bebfefe9f9f3 100644
--- a/products/signals/backend/test/test_tracker_issues.py
+++ b/products/signals/backend/test/test_tracker_issues.py
@@ -1,14 +1,20 @@
+import time
from datetime import timedelta
from typing import Any
import pytest
-from unittest.mock import MagicMock, patch
+import time_machine
+from unittest.mock import ANY, MagicMock, patch
from django.utils import timezone
from celery.exceptions import Retry
+from redis.exceptions import ConnectionError as RedisConnectionError
from rest_framework.exceptions import ValidationError as DRFValidationError
+from rest_framework.test import APIClient
+from posthog.egress.github.transport import GitHubEgressBudgetExhausted, GitHubRateLimitError
+from posthog.egress.limiter.policies import Priority
from posthog.models import Team, User
from posthog.models.integration import (
GitHubIntegration,
@@ -18,14 +24,24 @@ from posthog.models.integration import (
JiraIntegration,
LinearIntegration,
)
-
-from products.signals.backend.models import SignalReport, SignalReportTrackerIssue, SignalTeamConfig
+from posthog.redis import get_client
+
+from products.signals.backend.implementation_pr import fetch_implementation_pr_state_for_reports
+from products.signals.backend.models import (
+ SignalActorKind,
+ SignalReport,
+ SignalReportAssignment,
+ SignalReportMonitoringAttempt,
+ SignalReportTrackerIssue,
+ SignalTeamConfig,
+)
from products.signals.backend.serializers import SignalTeamConfigSerializer
from products.signals.backend.tasks import close_report_tracker_issue
from products.signals.backend.tracker_issues import (
ABANDONED_CLAIM_REASON,
PR_BODY_MARKER,
close_tracker_issue_for_report,
+ complete_monitoring_tracker_issues,
create_tracker_issue_for_report,
issue_reference,
link_pull_request_to_tracker_issue,
@@ -264,10 +280,54 @@ def test_link_pull_request_backs_off_when_the_body_changes_during_the_edit(team)
@pytest.mark.django_db
-def test_close_tracker_issue_is_recorded_once(team):
+ "delivery",
+ [
+ "sequential",
+ "overlapping",
+ "monitoring",
+ "reopened",
+ "replaced_attempt",
+ "archived",
+ "manual",
+ "provider_retry",
+ "failed_comment_retry",
+ "reopened_during_pr",
+ "reopened_during_comment",
+ ],
+)
+def test_close_tracker_issue_is_recorded_once(team: Team, delivery: str) -> None:
integration = _connect_tracker(team, "github")
report = _make_report(team)
- SignalReportTrackerIssue.all_teams.create(
+ attempt_id = None
+ if delivery in {
+ "monitoring",
+ "reopened",
+ "replaced_attempt",
+ "archived",
+ "manual",
+ "reopened_during_pr",
+ "reopened_during_comment",
+ }:
+ started_at = timezone.now() - timedelta(hours=1)
+ attempt = SignalReportMonitoringAttempt.objects.for_team(team.id).create(
+ team=team,
+ report=report,
+ monitoring_started_at=started_at,
+ aggregation_state="passed",
+ close_pr_on_resolve=delivery in {"manual", "reopened_during_pr"},
+ )
+ attempt_id = str(attempt.id)
+ SignalReport.objects.filter(id=report.id).update(
+ status=SignalReport.Status.READY if delivery == "reopened" else SignalReport.Status.RESOLVED,
+ monitoring_started_at=timezone.now() if delivery == "replaced_attempt" else started_at,
+ )
+ if delivery == "archived":
+ SignalReport.objects.filter(id=report.id).update(
+ status=SignalReport.Status.SUPPRESSED,
+ status_before_suppression=SignalReport.Status.RESOLVED,
+ )
+ tracker = SignalReportTrackerIssue.all_teams.create(
team=team,
report=report,
integration=integration,
@@ -276,11 +336,290 @@ def test_close_tracker_issue_is_recorded_once(team):
external_context={"repository": "web", "number": 12},
)
- with patch.object(GitHubIntegration, "close_issue") as close_issue:
- assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is True
+ def overlapping_close(*args: object, **kwargs: object) -> None:
assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is False
- close_issue.assert_called_once_with("web", 12, completed=False)
+ def reopen_report(*args: object, **kwargs: object) -> dict[str, bool]:
+ SignalReport.objects.filter(team_id=team.id, id=report.id).update(status=SignalReport.Status.READY)
+ return {"success": True}
+
+ with (
+ patch.object(GitHubIntegration, "close_issue") as close_issue,
+ patch.object(GitHubIntegration, "comment_on_issue", return_value={"success": True}) as comment,
+ patch(
+ "products.signals.backend.tracker_issues.close_implementation_pr_for_report", return_value=True
+ ) as close_pr,
+ ):
+ if delivery == "overlapping":
+ close_issue.side_effect = overlapping_close
+ if delivery == "reopened_during_pr":
+ close_pr.side_effect = reopen_report
+ if delivery == "reopened_during_comment":
+ comment.side_effect = reopen_report
+ if delivery in {"provider_retry", "failed_comment_retry"}:
+ close_issue.side_effect = [RuntimeError("Provider unavailable"), None]
+ if delivery == "failed_comment_retry":
+ comment.side_effect = [{"success": False}, {"success": True}]
+ assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is False
+ tracker.refresh_from_db()
+ assert (tracker.close_comment_posted_at is None) == (delivery == "failed_comment_retry")
+ assert (
+ close_tracker_issue_for_report(
+ team_id=team.id,
+ report_id=str(report.id),
+ monitoring_attempt_id=attempt_id,
+ completed=attempt_id is not None,
+ )
+ is True
+ )
+ if delivery == "manual":
+ close_pr.assert_called_once_with(
+ team.id,
+ str(report.id),
+ reason="resolved",
+ actor_user_id=None,
+ comment_marker=f"<!-- monitoring-completion:{attempt_id} -->",
+ before_close=ANY,
+ priority=Priority.CRITICAL,
+ require_all=True,
+ )
+ elif delivery != "reopened_during_pr":
+ close_pr.assert_not_called()
+ if delivery in {"reopened", "replaced_attempt", "reopened_during_pr"}:
+ close_issue.assert_not_called()
+ comment.assert_not_called()
+ elif delivery == "reopened_during_comment":
+ close_issue.assert_not_called()
+ comment.assert_called_once()
+ else:
+ assert close_tracker_issue_for_report(team_id=team.id, report_id=str(report.id)) is False
+ if delivery in {"provider_retry", "failed_comment_retry"}:
+ assert close_issue.call_count == 2
+ else:
+ close_issue.assert_called_once_with("web", 12, completed=attempt_id is not None)
+ assert comment.call_count == (2 if delivery == "failed_comment_retry" else 1)
+
+
+ ("failure", "with_tracker"),
+ [
+ ("lock", False),
+ ("redis", False),
+ ("pr_close", False),
+ ("pr_close", True),
+ ("already_closed", False),
+ ("budget_probe", False),
+ ("rate_pr_close", False),
+ ("budget_tracker_close", True),
+ ("rate_tracker_comment", True),
+ ("tracker_close", True),
+ ],
+)
+def test_monitoring_completion_retries_until_all_delivery_finishes(
+ team: Team, failure: str, with_tracker: bool
+) -> None:
+ integration = _connect_tracker(team, "github")
+ report = _make_report(team)
+ now = timezone.now()
+ started_at = now - timedelta(hours=1)
+ SignalReport.objects.filter(team_id=team.id, id=report.id).update(
+ status=SignalReport.Status.RESOLVED, monitoring_started_at=started_at
+ )
+ attempt = SignalReportMonitoringAttempt.objects.for_team(team.id).create(
+ team=team,
+ report=report,
+ monitoring_started_at=started_at,
+ aggregation_state="passed",
+ close_pr_on_resolve=True,
+ tracker_issue_close_requested_at=now,
+ )
+ SignalReportAssignment.objects.for_team(team.id).create(
+ team=team,
+ report=report,
+ actor_kind=SignalActorKind.SYSTEM,
+ pr_url="https://github.com/acme/web/pull/50",
+ repository="acme/web",
+ pr_number=50,
+ pr_state=SignalReportAssignment.PrState.OPEN,
+ )
+ tracker = (
+ SignalReportTrackerIssue.objects.for_team(team.id).create(
+ team=team,
+ report=report,
+ integration=integration,
+ provider="github",
+ status=SignalReportTrackerIssue.Status.CREATED,
+ external_context={"repository": "web", "number": 12},
+ )
+ if with_tracker
+ else None
+ )
+ waiting_attempt = (
+ SignalReportMonitoringAttempt.objects.for_team(team.id).create(
+ team=team,
+ report=_make_report(team),
+ monitoring_started_at=started_at,
+ aggregation_state="passed",
+ tracker_issue_close_requested_at=now + timedelta(seconds=1),
+ )
+ if failure.startswith(("budget_", "rate_"))
+ else None
+ )
+ pr_closed = failure == "already_closed"
+ commented = False
+ failed = False
+ calls: list[str] = []
+ priorities: list[Priority] = []
+
+ def deliver(client: GitHubIntegration, stage: str) -> None:
+ nonlocal failed
+ priorities.append(client.priority)
+ calls.append(stage)
+ if not failed and failure.removeprefix("budget_").removeprefix("rate_") == stage:
+ failed = True
+ if failure.startswith("budget_"):
+ raise GitHubEgressBudgetExhausted("Budget unavailable")
+ if failure.startswith("rate_"):
+ if failure == "rate_tracker_comment":
+ raise GitHubRateLimitError("Provider limited", reset_at=int(now.timestamp()) + 900)
+ raise GitHubRateLimitError("Provider limited", retry_after=900)
+ raise RuntimeError("Provider unavailable")
+
+ def probe(client: GitHubIntegration, repository: str, *, raise_on_transient: bool) -> bool:
+ deliver(client, "probe")
+ return True
+
+ def status(client: GitHubIntegration, repository: str, number: int) -> dict[str, object]:
+ deliver(client, "status")
+ return {"success": True, "state": "closed" if pr_closed else "open", "merged": False}
+
+ def find_comment(client: GitHubIntegration, repository: str, number: int, marker: str) -> bool:
+ deliver(client, "find_comment")
+ return commented
+
+ def comment_pr(client: GitHubIntegration, repository: str, number: int, body: str) -> dict[str, bool]:
+ nonlocal commented
+ deliver(client, "pr_comment")
+ commented = True
+ return {"success": True}
+
+ def close_pr(client: GitHubIntegration, repository: str, number: int) -> dict[str, bool]:
+ nonlocal pr_closed
+ deliver(client, "pr_close")
+ pr_closed = True
+ return {"success": True}
+
+ def comment_tracker(client: GitHubIntegration, repository: str, number: int, body: str) -> dict[str, bool]:
+ deliver(client, "tracker_comment")
+ return {"success": True}
+
+ def close_tracker(client: GitHubIntegration, repository: str, number: int, *, completed: bool) -> None:
+ deliver(client, "tracker_close")
+ assert completed
+
+ with (
+ time_machine.travel(now, tick=False),
+ patch.object(GitHubIntegration, "installation_can_access_repository", autospec=True, side_effect=probe),
+ patch.object(GitHubIntegration, "get_pull_request", autospec=True, side_effect=status),
+ patch.object(GitHubIntegration, "has_pull_request_comment", autospec=True, side_effect=find_comment),
+ patch.object(GitHubIntegration, "comment_on_pull_request", autospec=True, side_effect=comment_pr),
+ patch.object(GitHubIntegration, "close_pull_request", autospec=True, side_effect=close_pr),
+ patch.object(GitHubIntegration, "comment_on_issue", autospec=True, side_effect=comment_tracker),
+ patch.object(GitHubIntegration, "close_issue", autospec=True, side_effect=close_tracker),
+ ):
+ if failure == "lock":
+ with get_client().lock(f"signals:tracker-close:{team.id}:{report.id}", timeout=120):
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ elif failure == "redis":
+ with patch("products.signals.backend.tracker_issues.get_client", side_effect=RedisConnectionError()):
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ else:
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ attempt.refresh_from_db()
+ if waiting_attempt is not None:
+ waiting_attempt.refresh_from_db()
+ assert waiting_attempt.tracker_issue_close_processed_at is None
+ if failure != "already_closed":
+ assert attempt.tracker_issue_close_processed_at is None
+ retry_at = now + timedelta(minutes=15 if failure.startswith("rate_") else 5)
+ assert attempt.tracker_issue_close_next_attempt_at == retry_at
+ call_count = len(calls)
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ assert len(calls) == call_count
+ with time_machine.travel(retry_at, tick=False):
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ attempt.refresh_from_db()
+ assert attempt.tracker_issue_close_processed_at is not None
+ pr_state = fetch_implementation_pr_state_for_reports([str(report.id)], team_id=team.id)[str(report.id)]
+ assert pr_state.state == "closed"
+ if tracker is not None:
+ tracker.refresh_from_db()
+ assert tracker.closed_at is not None
+ assert calls.count("pr_comment") <= 1
+ assert priorities and all(priority == Priority.BATCH for priority in priorities)
+ call_count = len(calls)
+ complete_monitoring_tracker_issues(deadline=time.monotonic() + 30)
+ assert len(calls) == call_count
+
+
[email protected]("bulk", [False, True])
+def test_reopening_waits_for_tracker_delivery(team: Team, bulk: bool) -> None:
+ integration = _connect_tracker(team, "github")
+ user = User.objects.create_and_join(team.organization, "[email protected]", None)
+ client = APIClient()
+ client.force_authenticate(user=user)
+ report = _make_report(team)
+ started_at = timezone.now() - timedelta(hours=1)
+ SignalReport.objects.filter(team_id=team.id, id=report.id).update(
+ status=SignalReport.Status.RESOLVED, monitoring_started_at=started_at
+ )
+ attempt = SignalReportMonitoringAttempt.objects.for_team(team.id).create(
+ team=team,
+ report=report,
+ monitoring_started_at=started_at,
+ aggregation_state="passed",
+ tracker_issue_close_requested_at=timezone.now(),
+ )
+ SignalReportTrackerIssue.all_teams.create(
+ team=team,
+ report=report,
+ integration=integration,
+ provider="github",
+ status=SignalReportTrackerIssue.Status.CREATED,
+ external_context={"repository": "web", "number": 12},
+ )
+ url = (
+ f"/api/projects/{team.id}/signals/reports/bulk-state/"
+ if bulk
+ else f"/api/projects/{team.id}/signals/reports/{report.id}/state/"
+ )
+ body: dict[str, object] = {"state": "potential"}
+ if bulk:
+ body["ids"] = [str(report.id)]
+
+ def try_reopening(*args: object, **kwargs: object) -> None:
+ response = client.post(url, body, format="json")
+ assert response.status_code == (200 if bulk else 409), response.json()
+ if bulk:
+ assert response.json()["results"][0]["outcome"] == "skipped"
+ report.refresh_from_db()
+ assert report.status == SignalReport.Status.RESOLVED
+
+ with (
+ patch.object(GitHubIntegration, "comment_on_issue", return_value={"success": True}),
+ patch.object(GitHubIntegration, "close_issue", side_effect=try_reopening) as close_issue,
+ ):
+ assert close_tracker_issue_for_report(
+ team_id=team.id, report_id=str(report.id), completed=True, monitoring_attempt_id=str(attempt.id)
+ )
+ close_issue.assert_called_once()
+
+ response = client.post(url, body, format="json")
+ assert response.status_code == 200, response.json()
+ report.refresh_from_db()
+ assert report.status == SignalReport.Status.POTENTIAL
@pytest.mark.django_db
@@ -339,7 +678,8 @@ def test_completed_report_closes_tracker_issue_as_completed(team):
@pytest.mark.django_db
@pytest.mark.parametrize("status", [SignalReportTrackerIssue.Status.PENDING, SignalReportTrackerIssue.Status.CREATED])
-def test_close_tracker_issue_task_retries_pending_or_failed_provider_close(team, status):
[email protected]("failure", ["provider", "redis"])
+def test_close_tracker_issue_task_retries_pending_or_failed_provider_close(team, status, failure):
integration = _connect_tracker(team, "github")
report = _make_report(team)
SignalReportTrackerIssue.all_teams.create(
@@ -351,7 +691,13 @@ def test_close_tracker_issue_task_retries_pending_or_failed_provider_close(team,
external_context={"repository": "web", "number": 12},
)
- with patch("products.signals.backend.tasks.close_tracker_issue_for_report", return_value=False):
+ with (
+ patch("products.signals.backend.tasks.close_tracker_issue_for_report", return_value=False)
+ if failure == "provider"
+ else patch(
+ "products.signals.backend.tracker_issues.get_client", side_effect=RedisConnectionError("Unavailable")
+ )
+ ):
with pytest.raises(Retry):
close_report_tracker_issue.apply(
kwargs={"report_id": str(report.id), "team_id": team.id, "completed": False},
diff --git a/products/signals/backend/tracker_issues.py b/products/signals/backend/tracker_issues.py
index 2c2ef2c779cb..f2203adb7839 100644
--- a/products/signals/backend/tracker_issues.py
+++ b/products/signals/backend/tracker_issues.py
@@ -10,15 +10,21 @@ recorded as a failure on the report instead, so the team can see the gap and fix
from __future__ import annotations
import re
+import time
+from collections.abc import Iterator
+from contextlib import contextmanager
from datetime import timedelta
from typing import Any
+from django.db import models
from django.utils import timezone
import structlog
+from redis.exceptions import RedisError
from rest_framework.exceptions import ValidationError as DRFValidationError
from posthog.dataclasses import frozen
+from posthog.egress.github.transport import GitHubEgressBudgetExhausted, GitHubRateLimitError
from posthog.egress.limiter.policies import Priority
from posthog.models.integration import (
SUPPORTED_EXTERNAL_ISSUE_PROVIDERS,
@@ -29,9 +35,16 @@ from posthog.models.integration import (
LinearIntegration,
external_issue_url,
)
+from posthog.redis import get_client
from products.signals.backend.github_actor import github_mention_for_user
-from products.signals.backend.models import SignalReport, SignalReportTrackerIssue, SignalTeamConfig
+from products.signals.backend.implementation_pr import close_implementation_pr_for_report
+from products.signals.backend.models import (
+ SignalReport,
+ SignalReportMonitoringAttempt,
+ SignalReportTrackerIssue,
+ SignalTeamConfig,
+)
from products.signals.backend.pull_request_body import BodyEditOutcome, edit_pull_request_body
from products.signals.backend.report_urls import ReportLinkSource, build_report_url
@@ -350,36 +363,48 @@ def _close_notice(*, report_link: str, actor_mention: str | None) -> str:
return f"🔕 The {report_link} was closed in PostHog, so this issue is closed too."
-def _comment_before_close(tracker: SignalReportTrackerIssue, *, report_link: str, actor_mention: str | None) -> None:
+def _comment_before_close(
+ tracker: SignalReportTrackerIssue, *, report_link: str, actor_mention: str | None, priority: Priority
+) -> None:
"""Best-effort note on the issue naming who closed the report. Only GitHub takes one today.
Explain first, close second, as the implementation pull request does. A failed comment must
not stop the close: an issue left open would grow the backlog this close exists to prevent,
while a missing comment costs only the attribution.
"""
- if tracker.provider != Integration.IntegrationKind.GITHUB or tracker.integration is None:
+ if (
+ tracker.provider != Integration.IntegrationKind.GITHUB
+ or tracker.integration is None
+ or tracker.close_comment_posted_at is not None
+ ):
return
context = tracker.external_context or {}
try:
- GitHubIntegration(
+ result = GitHubIntegration(
tracker.integration, source="signals_tracker_issues", priority=Priority.BATCH
).comment_on_issue(
context["repository"],
int(context["number"]),
_close_notice(report_link=report_link, actor_mention=actor_mention),
)
+ if result.get("success"):
+ tracker.close_comment_posted_at = timezone.now()
+ tracker.save(update_fields=["close_comment_posted_at", "updated_at"])
+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):
+ if priority == Priority.BATCH:
+ raise
except Exception:
logger.warning("signals.tracker_issue_close_comment_failed", tracker_id=str(tracker.id), exc_info=True)
-def _close_provider_issue(tracker: SignalReportTrackerIssue, *, completed: bool) -> None:
+def _close_provider_issue(tracker: SignalReportTrackerIssue, *, completed: bool, priority: Priority) -> None:
integration = tracker.integration
if integration is None:
raise ValueError("Tracker issue has no integration left to close it through")
context = tracker.external_context or {}
if tracker.provider == Integration.IntegrationKind.GITHUB:
- GitHubIntegration(integration, source="signals_tracker_issues", priority=Priority.CRITICAL).close_issue(
+ GitHubIntegration(integration, source="signals_tracker_issues", priority=priority).close_issue(
context["repository"], int(context["number"]), completed=completed
)
elif tracker.provider == Integration.IntegrationKind.GITLAB:
@@ -392,34 +417,197 @@ def _close_provider_issue(tracker: SignalReportTrackerIssue, *, completed: bool)
raise ValueError(f"Unsupported tracker provider {tracker.provider}")
-def close_tracker_issue_for_report(
- *, team_id: int, report_id: str, completed: bool = False, actor_user_id: int | None = None
+def _current_completion_attempt(
+ *, team_id: int, report_id: str, monitoring_attempt_id: str
+) -> SignalReportMonitoringAttempt | None:
+ return (
+ SignalReportMonitoringAttempt.objects.for_team(team_id)
+ .filter(
+ id=monitoring_attempt_id,
+ report_id=report_id,
+ aggregation_state="passed",
+ tracker_issue_close_processed_at__isnull=True,
+ monitoring_started_at=models.F("report__monitoring_started_at"),
+ )
+ .filter(
+ models.Q(report__status=SignalReport.Status.RESOLVED)
+ | models.Q(
+ report__status=SignalReport.Status.SUPPRESSED,
+ report__status_before_suppression=SignalReport.Status.RESOLVED,
+ )
+ )
+ .first()
+ )
+
+
+def _close_tracker_issue_for_report(
+ *,
+ team_id: int,
+ report_id: str,
+ completed: bool,
+ actor_user_id: int | None,
+ monitoring_attempt_id: str | None,
+ priority: Priority,
) -> bool:
- """Close the report's tracker issue once the report is dismissed or completed. Never raises.
+ """Close the report's tracker issue once the report is dismissed or completed.
A dismissed report will not produce a pull request, so its work item is finished. Leaving it
- open would grow a backlog of tracker issues that no pull request will ever answer.
+ open would grow a backlog of tracker issues that no pull request will ever answer. Batch rate
+ limits propagate so the sweep can honor the provider's retry window.
"""
+
+ def can_complete() -> bool:
+ return (
+ monitoring_attempt_id is None
+ or _current_completion_attempt(
+ team_id=team_id, report_id=report_id, monitoring_attempt_id=monitoring_attempt_id
+ )
+ is not None
+ )
+
try:
+ pr_completed = True
+ if monitoring_attempt_id is not None:
+ attempt = _current_completion_attempt(
+ team_id=team_id, report_id=report_id, monitoring_attempt_id=monitoring_attempt_id
+ )
+ if attempt is None:
+ return True
+ if attempt.close_pr_on_resolve:
+ pr_completed = close_implementation_pr_for_report(
+ team_id,
+ report_id,
+ reason="resolved",
+ actor_user_id=actor_user_id,
+ comment_marker=f"<!-- monitoring-completion:{attempt.id} -->",
+ before_close=can_complete,
+ priority=priority,
+ require_all=True,
+ )
tracker = (
SignalReportTrackerIssue.objects.for_team(team_id)
.select_related("integration")
- .filter(report_id=report_id, status=SignalReportTrackerIssue.Status.CREATED)
+ .filter(
+ report_id=report_id,
+ status__in=[SignalReportTrackerIssue.Status.PENDING, SignalReportTrackerIssue.Status.CREATED],
+ )
.first()
)
if tracker is None or tracker.closed_at is not None:
+ return monitoring_attempt_id is not None and pr_completed
+ if tracker.status == SignalReportTrackerIssue.Status.PENDING:
return False
+ if not can_complete():
+ return True
report_url = build_report_url(team_id, report_id, ReportLinkSource.TRACKER)
_comment_before_close(
tracker,
report_link=f"[linked PostHog report]({report_url})",
actor_mention=github_mention_for_user(actor_user_id),
+ priority=priority,
)
- _close_provider_issue(tracker, completed=completed)
+ if not can_complete():
+ return True
+ _close_provider_issue(tracker, completed=completed, priority=priority)
tracker.closed_at = timezone.now()
tracker.save(update_fields=["closed_at", "updated_at"])
- return True
+ return pr_completed
+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError):
+ if priority == Priority.BATCH:
+ raise
+ return False
except Exception:
logger.warning("signals.tracker_issue_close_failed", report_id=report_id, team_id=team_id, exc_info=True)
return False
+
+
+@contextmanager
+def tracker_issue_completion_lock(*, team_id: int, report_id: str) -> Iterator[bool]:
+ lock = None
+ acquired = False
+ try:
+ lock = get_client().lock(f"signals:tracker-close:{team_id}:{report_id}", timeout=120)
+ acquired = lock.acquire(blocking=False)
+ except RedisError:
+ logger.warning("signals.tracker_issue_close_lock_failed", report_id=report_id, team_id=team_id, exc_info=True)
+ yield False
+ return
+ try:
+ yield acquired
+ finally:
+ if acquired and lock is not None:
+ try:
+ lock.release()
+ except RedisError:
+ logger.warning("signals.tracker_issue_close_lock_release_failed", report_id=report_id, exc_info=True)
+
+
+def close_tracker_issue_for_report(
+ *,
+ team_id: int,
+ report_id: str,
+ completed: bool = False,
+ actor_user_id: int | None = None,
+ monitoring_attempt_id: str | None = None,
+ priority: Priority = Priority.CRITICAL,
+) -> bool:
+ with tracker_issue_completion_lock(team_id=team_id, report_id=report_id) as acquired:
+ if not acquired:
+ return False
+ return _close_tracker_issue_for_report(
+ team_id=team_id,
+ report_id=report_id,
+ completed=completed,
+ actor_user_id=actor_user_id,
+ monitoring_attempt_id=monitoring_attempt_id,
+ priority=priority,
+ )
+
+
+def complete_monitoring_tracker_issues(*, deadline: float, limit: int = 100) -> None:
+ now = timezone.now()
+ due = models.Q(tracker_issue_close_next_attempt_at__isnull=True) | models.Q(
+ tracker_issue_close_next_attempt_at__lte=now
+ )
+ attempts = (
+ SignalReportMonitoringAttempt.objects.unscoped()
+ .filter(tracker_issue_close_requested_at__isnull=False, tracker_issue_close_processed_at__isnull=True)
+ .filter(due)
+ .order_by(
+ models.F("tracker_issue_close_next_attempt_at").asc(nulls_first=True),
+ "tracker_issue_close_requested_at",
+ "id",
+ )[:limit]
+ )
+ for attempt in attempts:
+ if time.monotonic() >= deadline:
+ break
+ claimed = (
+ SignalReportMonitoringAttempt.objects.for_team(attempt.team_id)
+ .filter(id=attempt.id, tracker_issue_close_processed_at__isnull=True)
+ .filter(due)
+ .update(tracker_issue_close_next_attempt_at=now + timedelta(minutes=5))
+ )
+ if not claimed:
+ continue
+ try:
+ completed = close_tracker_issue_for_report(
+ team_id=attempt.team_id,
+ report_id=str(attempt.report_id),
+ completed=True,
+ monitoring_attempt_id=str(attempt.id),
+ priority=Priority.BATCH,
+ )
+ except (GitHubEgressBudgetExhausted, GitHubRateLimitError) as error:
+ retry_after = 300
+ if isinstance(error, GitHubRateLimitError):
+ retry_after = max(retry_after, error.retry_after or 0, (error.reset_at or 0) - int(time.time()))
+ SignalReportMonitoringAttempt.objects.for_team(attempt.team_id).filter(
+ id=attempt.id, tracker_issue_close_processed_at__isnull=True
+ ).update(tracker_issue_close_next_attempt_at=timezone.now() + timedelta(seconds=retry_after))
+ return
+ if completed:
+ SignalReportMonitoringAttempt.objects.for_team(attempt.team_id).filter(id=attempt.id).update(
+ tracker_issue_close_processed_at=timezone.now()
+ )
diff --git a/products/signals/backend/views.py b/products/signals/backend/views.py
index 8023ea0611ea..c56114552d79 100644
--- a/products/signals/backend/views.py
+++ b/products/signals/backend/views.py
@@ -3,6 +3,7 @@ import json
import math
import uuid
from collections.abc import Callable, Iterable, Sequence
+from contextlib import ExitStack
from datetime import UTC, timedelta
from functools import cached_property, partial
from typing import Any, Literal, cast
@@ -148,6 +149,7 @@ from products.signals.backend.models import (
SignalReportAction,
SignalReportArtefact,
SignalReportCheck,
+ SignalReportMonitoringAttempt,
SignalReportRefund,
SignalScoutConfig,
SignalSourceConfig,
@@ -322,6 +324,7 @@ from products.signals.backend.throttles import (
ReportTaskActionRunBurstThrottle,
ReportTaskActionRunSustainedThrottle,
)
+from products.signals.backend.tracker_issues import tracker_issue_completion_lock
from products.tasks.backend.facade import api as tasks_facade
from products.tasks.backend.facade.access import code_access_required_response, usage_limit_response
from products.tasks.backend.facade.repo_selection_types import RepoSelectionResult
@@ -4049,7 +4052,8 @@ class SignalReportViewSet(
if target_status == SignalReport.Status.MONITORING and not self._report_monitoring_enabled:
return SignalReportBulkStateOutcome.SKIPPED, "Monitoring is unavailable for this organization."
- with transaction.atomic():
+ # Release the delivery lock after commit so workers cannot read the previous verdict.
+ with ExitStack() as completion_locks, transaction.atomic():
# Lock the report row for the whole guard + transition so it serializes with the refund
# path (which locks the same row before suppressing). Re-read committed state under the
# lock: a resolve/restore that raced a refund must observe the refund's SUPPRESSED status
@@ -4060,6 +4064,23 @@ class SignalReportViewSet(
is not None
):
report.refresh_from_db()
+ if (
+ SignalReportMonitoringAttempt.objects.for_team(self.team.id)
+ .filter(
+ report_id=report.id,
+ tracker_issue_close_requested_at__isnull=False,
+ tracker_issue_close_processed_at__isnull=True,
+ )
+ .exists()
+ ):
+ acquired = completion_locks.enter_context(
+ tracker_issue_completion_lock(team_id=self.team.id, report_id=str(report.id))
+ )
+ if not acquired:
+ return (
+ SignalReportBulkStateOutcome.SKIPPED,
+ "This report is being finalized. Try again in a moment.",
+ )
if dismissal_reason == "verified_manually" and (
not self._report_monitoring_enabled
or report.monitoring_started_at is None