:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(signals): file linked follow-up reports for failed named-lane checks (#119474)
posthog[bot]8152f018af45
diff --git a/products/signals/ARCHITECTURE.md b/products/signals/ARCHITECTURE.mdindex 66b30bb28355..2312e4a3f77c 100644--- a/products/signals/ARCHITECTURE.md+++ b/products/signals/ARCHITECTURE.md@@ -517,7 +517,7 @@ Execution rides the scout coordinator tick (`run_due_signal_report_checks_activiA check written _before_ the fix exists has no date to carry. The research pipeline's verification turn is that case: it authors the check in the same pass that writes the report. Such a check is stored `pending` with a `soak_minutes` window, and the report's transition to `resolved` is what sets `next_run_at` — a merged pull request's webhook, a manual resolve in the inbox, and an MCP state write all end in the same `save`, so the resolve is the clock for every kind of fix, including the ones that never had a pull request. `receivers.arm_pending_checks_when_report_resolved` is the choke point. A pending check counts against the per-report cap, is never collected as due, and expires at the 90-day horizon if its report never resolves.-A failed verdict on a resolved report is the one outcome nobody would otherwise read, because the inbox no longer lists the report. An `agent` check has a scout in the loop that can author a fresh one, so only the deterministic lane needs help: a `metric_threshold` breach on a `resolved` report emits a `source_product="signals_check"` signal carrying the verdict and the origin report id. The grouping stage then applies the rule it already applies to any signal landing on a resolved report — a fresh report, linked back by `related_to` and by a typed `follow_up_of` link carrying the verdict as its reason. Nothing reopens the terminal one. The two rows say different things: `related_to` (and `recurrence_of`) says the issue came back, `follow_up_of` says a measurement of the fix breached, and the research agent reads the typed one as context.+A failed verdict on a resolved report is the one outcome nobody would otherwise read, because the inbox no longer lists the report. An `agent` check has a scout in the loop: the check brief (`_check_relapse_section` in `scout_harness/prompt.py`) tells a lane that holds `emit_report` to author a fresh report linked `follow_up_of` the original before it records `failed`. Every other failure takes the deterministic path: a `metric_threshold` breach, or a failed `agent` verdict from a run whose stamped `report_channel` lacks `emit_report`, on a `resolved` report emits a `source_product="signals_check"` signal carrying the verdict and the origin report id. The grouping stage then applies the rule it already applies to any signal landing on a resolved report — a fresh report, linked back by `related_to` and by a typed `follow_up_of` link carrying the verdict as its reason. Nothing reopens the terminal one. The two rows say different things: `related_to` (and `recurrence_of`) says the issue came back, `follow_up_of` says a measurement of the fix breached, and the research agent reads the typed one as context.`signals_check` / `check_failed` is self-authorizing in `SignalSourceConfig.is_source_enabled`, alongside the Replay Vision scanner. It is the inbox emitting to itself, so there is no team to configure: `check_failed` is deliberately outside `SignalSourceConfig.SourceType`, the subset that carries a per-team row, so a row-backed gate would refuse the pair forever. The check the team already authored is the opt-in. `CheckFailedSignalInput` is registered in `SIGNAL_INPUT_VARIANTS` for the same reason — without it `validate_signal_input` rejects the pair as an unknown signal type, and the producer logs and swallows the error.diff --git a/products/signals/backend/report_check_agent.py b/products/signals/backend/report_check_agent.pyindex 5ef2c3ec1d7c..0e14500239a5 100644--- a/products/signals/backend/report_check_agent.py+++ b/products/signals/backend/report_check_agent.py@@ -197,6 +197,7 @@ def build_check_run_note(check: SignalReportCheck, config: AgentCheckConfig) ->"`passed` (the evidence meets the bar and the expectation holds), `failed` (it meets the bar ""and the expectation does not hold), `inconclusive` with a `reason` (the evidence cannot settle ""it), or `errored` (a tool or model call failed). That call is the only thing that closes the check.",+ "Before you record `failed`, do what your instructions say about a failed check on a resolved report.",]return "\n".join(lines)diff --git a/products/signals/backend/report_check_execution.py b/products/signals/backend/report_check_execution.pyindex fcb072fd7232..16ed5617aeee 100644--- a/products/signals/backend/report_check_execution.py+++ b/products/signals/backend/report_check_execution.py@@ -322,6 +322,7 @@ def record_check_verdict(run_id: str | None = None,skill_name: str | None = None,refusal_reason: str | None = None,+ lane_can_emit_report: bool | None = None,) -> None:"""The single persistence funnel: append the result artefact and advance or retire the check.@@ -331,6 +332,7 @@ def record_check_verdict(`attribution` and `run_id` name the scout run that decided an `agent` check. A deterministic runhas neither, so it keeps the `system()` attribution the executor writes under. `skill_name` and`refusal_reason` name the scout and the gate when the fleet refused to run an `agent` check.+ `lane_can_emit_report` says whether that run held `emit_report`, and None when nobody knows."""now = now or timezone.now()# The result's context is best-effort. A stored config can stop parsing part-way through a soak,@@ -409,7 +411,7 @@ def record_check_verdict(),attribution=attribution or ArtefactAttribution.system(),)- if _should_resurface(current, verdict):+ if _should_resurface(current, verdict, lane_can_emit_report=lane_can_emit_report):baseline = config.baseline_value if config is not None else Nonethreshold = _describe_comparison(config.comparison) if config is not None else Nonetransaction.on_commit(lambda: resurface_failed_check(current, verdict, baseline, threshold))@@ -430,19 +432,24 @@ def record_check_verdict()-def _should_resurface(check: SignalReportCheck, verdict: CheckVerdict) -> bool:+def _should_resurface(+ check: SignalReportCheck, verdict: CheckVerdict, *, lane_can_emit_report: bool | None = None+) -> bool:"""Whether this verdict has to become a fresh report, because nothing else will carry it.- An `agent` check has a scout in the loop that can author one. A `metric_threshold` check has- nobody, so a breach on a report the inbox no longer lists is an artefact on a closed item that- no reader will open. Only a resolved report qualifies: a breach on a report still being worked- lands where the person working it already looks.+ A `metric_threshold` check has nobody in the loop, so a breach on a report the inbox no longer+ lists is an artefact on a closed item that no reader will open. An `agent` check has a scout in+ the loop, and the check brief tells a lane that holds `emit_report` to author the follow-up+ itself. A lane known to lack `emit_report` cannot, so its verdict takes the metric path. An+ unknown lane stays with the scout, because a second report for one relapse is a duplicate. Only+ a resolved report qualifies: a breach on a report still being worked lands where the person+ working it already looks."""- return (- verdict.outcome == "failed"- and check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD- and check.report.effective_status == SignalReport.Status.RESOLVED- )+ if verdict.outcome != "failed" or check.report.effective_status != SignalReport.Status.RESOLVED:+ return False+ if check.kind == SignalReportCheck.Kind.METRIC_THRESHOLD:+ return True+ return check.kind == SignalReportCheck.Kind.AGENT and lane_can_emit_report is Falsedef resurface_failed_check(diff --git a/products/signals/backend/scout_harness/prompt.py b/products/signals/backend/scout_harness/prompt.pyindex d0d59a361463..f6c2ca08a11c 100644--- a/products/signals/backend/scout_harness/prompt.py+++ b/products/signals/backend/scout_harness/prompt.py@@ -1250,7 +1250,7 @@ _CHECK_NOTE_TEMPLATE = """# The check this run must answerThe coordinator started this run to answer one follow-up check on an inbox report. That is thisrun's job. Do the work the check asks for, reach a verdict, and record it; everything your skill-says about finding new problems is out of scope for this run.+says about finding new problems unrelated to the check is out of scope for this run.<check>{note}@@ -1278,11 +1278,69 @@ actually established:- `errored` only when a tool, a query, or a model call failed and stopped you. An unsettledquestion is `inconclusive`, not `errored`.+{relapse_section}+Nothing else closes the check, so a run that investigates and does not call the toolleaves the report with an unanswered follow-up. Say in your run summary what you recorded."""+# A check only runs on a resolved report, and a resolved report has left the inbox, so a `failed`+# verdict recorded on it alone reaches nobody. A lane that can author reports carries the relapse+# into the inbox itself. A lane that cannot has it re-surfaced as a signal by+# `report_check_execution._should_resurface`, so its brief must not ask it to file anything.+_CHECK_RELAPSE_EMIT = """## Before you record `failed`++The report this check sits on is resolved, so it has left the inbox, and a verdict recorded only on+it reaches nobody. A `failed` verdict means the problem came back, which makes that relapse part of+this run's job. Put it in front of a reader before you record the verdict:++- Author a fresh report with `scout-emit-report` that describes the relapse with the evidence you+ gathered, and link it to the original with+ `links: [{"kind": "follow_up_of", "report_id": "<the report id in the check block>", "reason": "Follow-up check <check_id> failed: <what you found, in one line>"}]`.+ The reader then sees the original findings, its pull request, and the fix that did not hold.+ Every other rule for a report you author applies to it.+- {live_report_bullet}+- Record `failed` after the report exists, and name its id in the `explanation`. `failed` retires+ the check, so a run that stops after the verdict leaves nothing that carries the relapse. If the+ emit comes back skipped, still record `failed`, and say in the `explanation` that no report was+ filed and why.++A `passed`, `inconclusive`, or `errored` verdict files nothing new."""++_CHECK_LIVE_REPORT_EDIT = (+ "Skip the new report only when the relapse is already in the inbox: an open report you authored "+ "about the same fix (append your fresh evidence to it with `scout-edit-report`), or an open report "+ "filed after the fix that already describes the recurrence. Search the inbox for both first, and "+ "cite the report you found instead."+)++_CHECK_LIVE_REPORT_EMIT_ONLY = (+ "Skip the new report only when the relapse is already in the inbox: an open report you authored "+ "about the same fix, or an open report filed after the fix that already describes the recurrence. "+ "Search the inbox for both first, and cite the report you found instead."+)++_CHECK_RELAPSE_NO_EMIT = """## When you record `failed`++This run cannot author reports. A `failed` verdict on the resolved report this check sits on+re-surfaces without you: it becomes a signal, and the pipeline files a fresh report linked to the+original. Your `explanation` is where that report starts, so make it carry the evidence a reader+needs: what you measured, against what the check expected."""+-def _run_note_section(run_note: str | None, triggered_by: str = TRIGGERED_BY_SCHEDULE) -> str:+def _check_relapse_section(*, can_emit_report: bool, can_edit_report: bool) -> str:+ if not can_emit_report:+ return _CHECK_RELAPSE_NO_EMIT+ live_report_bullet = _CHECK_LIVE_REPORT_EDIT if can_edit_report else _CHECK_LIVE_REPORT_EMIT_ONLY+ return _CHECK_RELAPSE_EMIT.replace("{live_report_bullet}", live_report_bullet)+++def _run_note_section(+ run_note: str | None,+ triggered_by: str = TRIGGERED_BY_SCHEDULE,+ *,+ can_emit_report: bool = False,+ can_edit_report: bool = False,+) -> str:"""The note this run was dispatched with, framed by what dispatched it, or empty without one.Rendered outside `_render_tail` on purpose: the tail formats any section holding a@@ -1292,8 +1350,12 @@ def _run_note_section(run_note: str | None, triggered_by: str = TRIGGERED_BY_SCHnote = (run_note or "").strip()if not note:return ""- template = _CHECK_NOTE_TEMPLATE if triggered_by == TRIGGERED_BY_CHECK else _RUN_NOTE_TEMPLATE- return template.format(note=note)+ if triggered_by == TRIGGERED_BY_CHECK:+ return _CHECK_NOTE_TEMPLATE.format(+ note=note,+ relapse_section=_check_relapse_section(can_emit_report=can_emit_report, can_edit_report=can_edit_report),+ )+ return _RUN_NOTE_TEMPLATE.format(note=note)_PRECHECK_RESULT_TEMPLATE = """# What the pre-check found@@ -1482,7 +1544,9 @@ def build_run_prompt(improvement = _CANONICAL_IMPROVEMENTsections = [*sections[:-1], improvement, sections[-1]]tail = _render_tail(sections, schema_json=schema_json)- run_note_section = _run_note_section(run_note, triggered_by)+ run_note_section = _run_note_section(+ run_note, triggered_by, can_emit_report=can_emit_report, can_edit_report=can_edit_report+ )# Report-channel scouts only: the authors line exists to steer `suggested_reviewers`, and a# signal-channel scout has no reviewers field — member names/emails are PII that shouldn't# flow into a prompt with no feature path to use them.diff --git a/products/signals/backend/scout_harness/tools/checks.py b/products/signals/backend/scout_harness/tools/checks.pyindex adca6bd61508..e96b72926d3a 100644--- a/products/signals/backend/scout_harness/tools/checks.py+++ b/products/signals/backend/scout_harness/tools/checks.py@@ -187,6 +187,7 @@ def record_check_result(# report's log names what produced the verdict rather than the system that scheduled it.attribution=_run_attribution(run),run_id=str(run.id),+ lane_can_emit_report=_run_can_emit_report(run),)check.refresh_from_db()return RecordCheckResultResult(@@ -382,6 +383,18 @@ def _run_attribution(run: SignalScoutRun) -> ArtefactAttribution:return ArtefactAttribution.from_task(task_id) if task_id else ArtefactAttribution.system()+def _run_can_emit_report(run: SignalScoutRun) -> bool | None:+ """Whether the run held `emit_report`, read off the channel the runner stamped when it started.++ Read from the stamp rather than the skill, because a skill's `allowed_tools` can be edited mid-run+ and the prompt this run got was built from the old list. None for a run without the stamp.+ """+ channel = (run.metadata or {}).get("report_channel")+ if channel is None:+ return None+ return channel in ("emit", "both")++def cancel_report_check(*, team: Team, run: SignalScoutRun, check_id: str) -> ScoutCheckSummary:"""Stop a check that is no longer worth running. Its recorded results stay on the report."""_assert_run_may_write_checks(team, run)diff --git a/products/signals/backend/test/test_report_checks.py b/products/signals/backend/test/test_report_checks.pyindex 3502a8e534dc..5eafd11807fb 100644--- a/products/signals/backend/test/test_report_checks.py+++ b/products/signals/backend/test/test_report_checks.py@@ -2044,6 +2044,34 @@ class TestCheckResultTool(APIBaseTest):assert check.runs_remaining == 1assert check.next_run_at > timezone.now() + timedelta(minutes=MIN_CHECK_INTERVAL_MINUTES - 5)+ @parameterized.expand(+ [+ ("emit_only", "emit", 0),+ ("emit_and_edit", "both", 0),+ ("edit_only", "edit", 1),+ ("no_report_tools", "none", 1),+ ("unstamped", None, 0),+ ]+ )+ def test_a_failure_resurfaces_only_when_the_lane_cannot_file_the_follow_up(+ self, _name: str, report_channel: str | None, expected_signals: int+ ) -> None:+ check = self._check()+ if report_channel is not None:+ self.scout_run.metadata = {"report_channel": report_channel}+ self.scout_run.save(update_fields=["metadata"])++ with patch(_EMIT_SIGNAL, new=AsyncMock(return_value=None)) as emit:+ with self.captureOnCommitCallbacks(execute=True):+ self._record(check)++ assert emit.call_count == expected_signals+ if expected_signals:+ sent = emit.call_args.kwargs+ assert sent["source_type"] == SignalSourceType.CHECK_FAILED+ assert sent["extra"]["report_id"] == str(self.report.id)+ assert "fired 30 times yesterday" in sent["description"]+@parameterized.expand([("not_due_and_no_run_is_waiting", {"dispatched_at": None}),diff --git a/products/signals/backend/test/test_scout_harness.py b/products/signals/backend/test/test_scout_harness.pyindex 44bacbfc2709..7a45dc5b0e49 100644--- a/products/signals/backend/test/test_scout_harness.py+++ b/products/signals/backend/test/test_scout_harness.py@@ -716,14 +716,16 @@ class TestStructuredOutputPromptSection(SimpleTestCase):class TestRunNotePromptSection(SimpleTestCase):- def _prompt(self, run_note: str | None, triggered_by: str = TRIGGERED_BY_SCHEDULE) -> str:+ def _prompt(+ self, run_note: str | None, triggered_by: str = TRIGGERED_BY_SCHEDULE, allowed_tools: list[str] | None = None+ ) -> str:return build_run_prompt(LoadedSkill(name="signals-scout-errors",version=1,body="watch",description="d",- allowed_tools=[],+ allowed_tools=allowed_tools or [],files=[],skill_id="skill-1",origin="canonical",@@ -759,15 +761,33 @@ class TestRunNotePromptSection(SimpleTestCase):assert "do not record it in the scratchpad as a durable memory" in promptassert "# Notes left for you" in prompt- def test_a_check_dispatch_frames_its_note_as_the_run_assignment(self) -> None:+ @parameterized.expand(+ [+ ("no_report_tools", [], False, False),+ ("emit_only", ["emit_report"], True, False),+ ("emit_and_edit", ["emit_report", "edit_report"], True, True),+ ("edit_only", ["edit_report"], False, False),+ ]+ )+ def test_a_check_dispatch_frames_its_note_as_the_run_assignment(+ self, _name: str, allowed_tools: list[str], files_follow_up: bool, edits_live_report: bool+ ) -> None:# Framed as a person's nudge, a check run reads its assignment as optional steering and is- # never told about the one tool that closes the check.- prompt = self._prompt("Check id: abc. Did the exception stop?", triggered_by=TRIGGERED_BY_CHECK)+ # never told about the one tool that closes the check. A failed verdict lands on a resolved+ # report nobody reads, so a lane that can author must file the linked follow-up first, and+ # a lane that cannot must not be told to, because the harness re-surfaces it instead.+ prompt = self._prompt(+ "Check id: abc. Did the exception stop?", triggered_by=TRIGGERED_BY_CHECK, allowed_tools=allowed_tools+ )assert "# The check this run must answer" in promptassert "<check>\nCheck id: abc. Did the exception stop?\n</check>" in promptassert "scout-check-record-result" in promptassert "# A note for this run" not in prompt+ check_section = prompt[prompt.index("# The check this run must answer") :]+ assert ('"kind": "follow_up_of"' in check_section) is files_follow_up+ assert ("re-surfaces without you" in check_section) is not files_follow_up+ assert ("scout-edit-report" in check_section) is edits_live_reportclass TestPrecheckResultPromptSection(SimpleTestCase):diff --git a/products/signals/skills/authoring-scouts/references/report-checks.md b/products/signals/skills/authoring-scouts/references/report-checks.mdindex d55781830029..929aa0a799ed 100644--- a/products/signals/skills/authoring-scouts/references/report-checks.md+++ b/products/signals/skills/authoring-scouts/references/report-checks.md@@ -90,12 +90,16 @@ The text in `instructions` and `probe_hints` reaches the run as evidence, not as- `unmeasurable`, `needs_manual_verification`, and `no_fix_to_measure` end the check as `inconclusive` at once, because waiting does not help.- **`cancelled`** comes from `scout-report-check-cancel`, for a check no longer worth running. The coordinator also cancels an `agent` check on a project with no scout configs, because no run can ever answer it. Verdicts already recorded stay on the report, and a finished check cannot be cancelled.-A **failed `metric_threshold` check on a resolved report re-surfaces**: the breach is emitted as a signal and the pipeline files a fresh report linked back to the resolved one.-A resolved report is no longer in the inbox, so a verdict left only on its artefact log would reach nobody, and this lane has no scout in it to author anything.-An `agent` check has no such path, because the run that answers it can file the follow-up itself.-The fallback lane, `signals-scout-inbox-validation`, does: when it records `failed` on a resolved report, it first authors a fresh report linked `follow_up_of` the original, the same link the pipeline writes for a metric breach.-A named lane gets only the shared check brief, which asks for the verdict and nothing more.-If its failures must reach the inbox, its own skill has to say to file the linked follow-up, and the lane needs `emit_report`.+A **failed check on a resolved report always reaches the inbox** as a fresh report linked `follow_up_of` the resolved one.+A resolved report is no longer in the inbox, so a verdict left only on its artefact log would reach nobody.+Which path files the fresh report depends on the check:++- **`metric_threshold`**: the breach is emitted as a signal, and the pipeline files the report. This lane has no scout in it to author anything.+- **`agent`, on a lane that holds `emit_report`**: the shared check brief tells the run to author the report itself before it records `failed`, and to name the new report in the verdict's explanation. This covers every lane, the fallback lane and named lanes alike. The run skips the new report only when the relapse is already in the inbox: an open report it authored about the same fix, which it edits when it holds `edit_report`, or an open report filed after the fix that already describes the recurrence, which it cites.+- **`agent`, on a lane without `emit_report`**: the run cannot author, so the verdict is emitted as a signal, the same path a metric breach takes.++A lane's own skill does not need to repeat this rule. It can add to it: the fallback lane, `signals-scout-inbox-validation`, also sets the report's priority and adds a `recurrence_of` link when the same problem is still happening.+`passed`, `inconclusive`, and `errored` verdicts file nothing new.## Reading a report's checks