Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(signals): retain verification authoring failures (#126584)

Mikayla Thompson648849fc278e
diff --git a/docs/internal/signals-pr-lifecycle.md b/docs/internal/signals-pr-lifecycle.md
index 439aca836d98..24cb5412c005 100644
--- a/docs/internal/signals-pr-lifecycle.md
+++ b/docs/internal/signals-pr-lifecycle.md
@@ -610,7 +610,7 @@ Research traces that contain existing metric-check context require the same anal
Metric suggestions are available only when `signals-report-checks-replace` enables the replacement tool.
-The note names proposed follow-up checks without claiming they were scheduled. The Follow-up checks sidebar shows the stored checks. If any optional check spec is malformed, research keeps valid verification prose and skips check reconciliation, preserving existing checks. An explicitly empty, valid check list still retires omitted checks. Legacy activity payloads preserve approved, person-authored, and nonpending checks. Referenced checks inherit their existing wait when the proposal omits it. Verification’s revised summary is saved only after its check reconciliation succeeds, so a skipped or invalid proposal cannot change Expected impact prose independently of the stored checks.
+The note names proposed follow-up checks without claiming they were scheduled. The Follow-up checks sidebar shows the stored checks. Verification permits five submissions: the initial proposal and up to four correction turns on the same research session. Each correction receives the latest failing fields and validation rules. Only verification output is retried; completed research is retained, and existing execution deadlines still apply. Exhausted correction or failed check persistence preserves existing checks and the presentation summary, records a server-owned `check_authoring` failure, and adds a visible work-log note. With Monitoring enabled, implementation completion and plan roll-up read the authoring outcome from the locked report’s writer connection and cannot automatically resolve without checks while that failure is current. Existing applicable checks can still enter Monitoring. A later successfully saved plan clears the blocker, including an explicitly empty plan; manual resolution remains available. An explicitly empty, valid check list still retires omitted checks. Legacy activity payloads preserve approved, person-authored, and nonpending checks. Referenced checks inherit their existing wait when the proposal omits it. Verification’s revised summary is saved only after its check reconciliation succeeds.
The plan separates `Confirm the current state` guidance from `Confirm the outcome` guidance.
Each section says what evidence to collect, which result supports a conclusion, and which result is inconclusive.
diff --git a/products/signals/backend/artefact_schemas.py b/products/signals/backend/artefact_schemas.py
index 7886c25ea279..bc5616a1b90d 100644
--- a/products/signals/backend/artefact_schemas.py
+++ b/products/signals/backend/artefact_schemas.py
@@ -804,6 +804,11 @@ class NoteArtefact(BaseModel):
return v
+class CheckAuthoringStatus(BaseModel):
+ status: Literal["succeeded", "failed"]
+ reason: str = Field(min_length=1)
+
+
class TitleChange(BaseModel):
"""Content schema for a `title_change` artefact: a record of an edit to the report's title.
@@ -1261,7 +1266,8 @@ class ImpactMeasurementPlan(BaseModel):
StatusArtefactContent = (
- SafetyJudgment
+ CheckAuthoringStatus
+ | SafetyJudgment
| ActionabilityAssessment
| PriorityAssessment
| RepoSelectionResult
@@ -1318,6 +1324,7 @@ ARTEFACT_CONTENT_SCHEMAS: Mapping[str, type[BaseModel]] = {
"commit": Commit,
"task_run": TaskRunArtefact,
"note": NoteArtefact,
+ "check_authoring": CheckAuthoringStatus,
"title_change": TitleChange,
"summary_change": SummaryChange,
"code_review": CodeReview,
@@ -1378,6 +1385,7 @@ NON_WRITABLE_ARTEFACT_TYPES: frozenset[str] = frozenset(
"check_result",
"check_observation",
"check_scheduled",
+ "check_authoring",
"check_expired",
"check_cancelled",
"report_link",
diff --git a/products/signals/backend/models.py b/products/signals/backend/models.py
index e768f3458470..f46c1b41cbe4 100644
--- a/products/signals/backend/models.py
+++ b/products/signals/backend/models.py
@@ -1463,6 +1463,7 @@ class SignalReportArtefact(UUIDModel):
CHECK_RESULT = "check_result"
CHECK_OBSERVATION = "check_observation"
CHECK_SCHEDULED = "check_scheduled"
+ CHECK_AUTHORING = "check_authoring"
CHECK_EXPIRED = "check_expired"
CHECK_CANCELLED = "check_cancelled"
IMPLEMENTATION_DECISION = "implementation_decision"
@@ -1498,6 +1499,7 @@ class SignalReportArtefact(UUIDModel):
ArtefactType.IMPLEMENTATION_DISPATCH,
ArtefactType.RANKING_SCORE,
ArtefactType.SOURCE_SUGGESTION,
+ ArtefactType.CHECK_AUTHORING,
}
)
# Rows the scoring sweep writes on every text edit and every new serving manifest. They record
@@ -1505,7 +1507,7 @@ class SignalReportArtefact(UUIDModel):
# them to staff only.
SYSTEM_SCORING_ARTEFACT_TYPES: frozenset[str] = frozenset({ArtefactType.RANKING_SCORE})
INTERNAL_ARTEFACT_TYPES: frozenset[str] = frozenset(
- {ArtefactType.REVIEWER_CORRECTIONS, ArtefactType.PR_OWNER_ASSIGNMENT}
+ {ArtefactType.REVIEWER_CORRECTIONS, ArtefactType.PR_OWNER_ASSIGNMENT, ArtefactType.CHECK_AUTHORING}
)
# A `report_link` graph is written by hand or by an agent, one report at a time, so a real
# chain is a handful of reports deep. The budgets guard the cycle walk on the write path
diff --git a/products/signals/backend/plan_rollup.py b/products/signals/backend/plan_rollup.py
index 7e9d5e611206..c529a7b6cf07 100644
--- a/products/signals/backend/plan_rollup.py
+++ b/products/signals/backend/plan_rollup.py
@@ -25,6 +25,7 @@ from products.signals.backend.models import (
SignalReportArtefact,
SignalReportPullRequest,
)
+from products.signals.backend.report_check_research import check_authoring_failed
from products.signals.backend.report_content_gates import team_report_monitoring_enabled
from products.signals.backend.report_lifecycle import effective_report_status
from products.signals.backend.typed_report_links import incoming_links, outgoing_links
@@ -141,6 +142,16 @@ def _close_parent(
if target == SignalReport.Status.SUPPRESSED and parent_status == SignalReport.Status.RESOLVED:
return False
try:
+ if (
+ target == SignalReport.Status.RESOLVED
+ and check_authoring_failed(parent)
+ and (
+ get_monitoring_enabled()
+ if get_monitoring_enabled is not None
+ else team_report_monitoring_enabled(parent.team_id)
+ )
+ ):
+ return False
updated_fields = parent.transition_to(target)
except (InvalidStatusTransition, ValueError, TypeError):
logger.info(
diff --git a/products/signals/backend/report_check_authoring.py b/products/signals/backend/report_check_authoring.py
index c71406c6cce1..4dddcd2495f2 100644
--- a/products/signals/backend/report_check_authoring.py
+++ b/products/signals/backend/report_check_authoring.py
@@ -348,7 +348,10 @@ def create_checks_from_specs(
replacements start unapproved. If a new spec cannot be stored, the transaction rolls back and
leaves the old checks running.
"""
- specs = _without_unrunnable_agent_specs(report, specs)
+ runnable_specs = _without_unrunnable_agent_specs(report, specs)
+ if specs and not runnable_specs:
+ return CheckReconciliation(applied=False, created=[])
+ specs = runnable_specs
try:
with transaction.atomic():
report = SignalReport.objects.select_for_update().get(id=report.id, team_id=report.team_id)
diff --git a/products/signals/backend/report_check_research.py b/products/signals/backend/report_check_research.py
index dd516a6a717c..34e65d071bc5 100644
--- a/products/signals/backend/report_check_research.py
+++ b/products/signals/backend/report_check_research.py
@@ -3,8 +3,22 @@ from __future__ import annotations
from collections.abc import Iterable, Mapping
from typing import TYPE_CHECKING
+from products.signals.backend.artefact_schemas import CheckAuthoringStatus
+from products.signals.backend.models import SignalReportArtefact
+
if TYPE_CHECKING:
- from products.signals.backend.models import SignalReportCheck
+ from products.signals.backend.models import SignalReport, SignalReportCheck
+
+
+def check_authoring_failed(report: SignalReport) -> bool:
+ """Use the locked report's writer so replica lag cannot hide a new authoring failure."""
+ authoring = (
+ SignalReportArtefact.objects.using(report._state.db or "default")
+ .filter(team_id=report.team_id, report_id=report.id, type=SignalReportArtefact.ArtefactType.CHECK_AUTHORING)
+ .order_by("-created_at", "-id")
+ .first()
+ )
+ return authoring is not None and CheckAuthoringStatus.model_validate_json(authoring.content).status == "failed"
def check_versions(checks: Iterable[SignalReportCheck]) -> dict[str, str]:
diff --git a/products/signals/backend/report_checks.py b/products/signals/backend/report_checks.py
index e2867afc5ad2..2935bd93217d 100644
--- a/products/signals/backend/report_checks.py
+++ b/products/signals/backend/report_checks.py
@@ -520,7 +520,12 @@ class CheckSpec(BaseModel):
@model_validator(mode="after")
def config_must_match_its_kind(self) -> CheckSpec:
- config = parse_check_config(self.kind, self.config)
+ try:
+ config = parse_check_config(self.kind, self.config)
+ except CheckConfigValidationError as error:
+ if isinstance(error.__cause__, ValidationError):
+ raise error.__cause__
+ raise
if self.run_interval_minutes is not None:
if not isinstance(config, MetricThresholdConfig) or config.monitoring is None:
raise ValueError("Research can author an interval only for a Monitoring metric check.")
diff --git a/products/signals/backend/report_generation/AGENTS.md b/products/signals/backend/report_generation/AGENTS.md
index 16909ac740b0..2425db07d23b 100644
--- a/products/signals/backend/report_generation/AGENTS.md
+++ b/products/signals/backend/report_generation/AGENTS.md
@@ -95,8 +95,8 @@ This module is intentionally prompt-orchestration only.
Production persistence is handled outside `run_multi_turn_research()`, in the caller activity, so this module stays isolated from report DB writes.
Monitoring metric definitions are offered only with `signals-report-monitoring` enabled. The shared writer validates the reference period, scaling policy, and baseline period. Disabled research contexts omit stored Monitoring definitions. When metrics are enabled, the research, presentation, and verification prompts prefer supported metric checks and evidence-backed shorter windows. Read [Monitoring metric definitions](../../../../docs/internal/signals-pr-lifecycle.md#monitoring-metric-definitions) when changing this contract; execution and final outcomes ship separately.
-Fix verification is best-effort. Generation failures do not fail completed research, but cancellation still does.
-Presentation receives the current follow-up checks as untrusted evidence. When verification proposes metric checks, it can return a separate `checks_summary` with Expected impact prose aligned with those goals and baselines, preserving other sections and chart markers. The transition applies that summary only if the matching check reconciliation succeeds. Omitted waits inherit the referenced check’s wait before generating the verification note. Invalid optional check specs leave the presentation summary and existing checks intact.
+Fix verification allows five submissions in the same session: the initial proposal and up to four corrections with the latest field-level errors. Exhausted authoring failures preserve completed research and existing checks, but cancellation still fails the run. Settlement records the authoring outcome and a visible failure note. Failed authoring blocks automatic no-check resolution until a valid plan is saved; existing applicable checks can still enter Monitoring.
+Presentation receives the current follow-up checks as untrusted evidence. Verification can return a separate `checks_summary` with Expected impact prose aligned with the proposed goals and baselines, preserving other sections and chart markers. Settlement applies it only after successful check reconciliation. Omitted waits inherit the referenced check’s wait before generating the verification note.
### Charts
diff --git a/products/signals/backend/report_generation/fixtures/analyze_report_funnel_research_output.json b/products/signals/backend/report_generation/fixtures/analyze_report_funnel_research_output.json
index 471bb2c4bb6a..8de3c9d1f2a8 100644
--- a/products/signals/backend/report_generation/fixtures/analyze_report_funnel_research_output.json
+++ b/products/signals/backend/report_generation/fixtures/analyze_report_funnel_research_output.json
@@ -16,6 +16,7 @@
"charts": [],
"metrics": [],
"checks_summary": null,
+ "checks_error": null,
"verification_note": null,
"new_artefacts": [],
"old_artefacts": [
diff --git a/products/signals/backend/report_generation/fixtures/insight_scene_logic_mode_property_bug.json b/products/signals/backend/report_generation/fixtures/insight_scene_logic_mode_property_bug.json
index e2984f3c1dc7..c4678f56d465 100644
--- a/products/signals/backend/report_generation/fixtures/insight_scene_logic_mode_property_bug.json
+++ b/products/signals/backend/report_generation/fixtures/insight_scene_logic_mode_property_bug.json
@@ -11,6 +11,7 @@
"charts": [],
"metrics": [],
"checks_summary": null,
+ "checks_error": null,
"verification_note": null,
"old_artefacts": [],
"new_artefacts": [
diff --git a/products/signals/backend/report_generation/research.py b/products/signals/backend/report_generation/research.py
index 75c54688f596..0d53f52932d1 100644
--- a/products/signals/backend/report_generation/research.py
+++ b/products/signals/backend/report_generation/research.py
@@ -9,15 +9,7 @@ from html import escape
from typing import TYPE_CHECKING, Literal
from uuid import UUID
-from pydantic import (
- BaseModel,
- Field,
- ValidationError,
- ValidationInfo,
- ValidatorFunctionWrapHandler,
- field_validator,
- model_validator,
-)
+from pydantic import BaseModel, Field, ValidationError, ValidationInfo, field_validator, model_validator
from posthog.dataclasses import frozen
@@ -525,11 +517,9 @@ class FixVerificationOutput(BaseModel):
raise ValueError("Only the Expected impact section may change during verification")
return summary
- @field_validator("checks", mode="wrap")
+ @field_validator("checks", mode="before")
@classmethod
- def preserve_checks_on_invalid_proposals(
- cls, value: object, handler: ValidatorFunctionWrapHandler, info: ValidationInfo
- ) -> list[CheckSpec] | None:
+ def inherit_existing_check_waits(cls, value: object, info: ValidationInfo) -> object:
previous = {
str(check["id"]): check["soak_hours"]
for check in (info.context or {}).get("previous_checks", [])
@@ -544,19 +534,7 @@ class FixVerificationOutput(BaseModel):
else check
for check in value
]
- try:
- checks: list[CheckSpec] | None = handler(value)
- return checks
- except ValidationError as error:
- logger.warning(
- "fix verification check specs did not validate",
- extra={
- "report_id": (info.context or {}).get("report_id"),
- "team_id": (info.context or {}).get("team_id"),
- "validation_rules": sorted({item["type"] for item in error.errors(include_input=False)}),
- },
- )
- return None
+ return value
def to_note(self) -> NoteArtefact:
# The note is written before persistence can confirm which checks were scheduled.
@@ -587,6 +565,7 @@ class ReportResearchOutput(BaseModel):
title: str = Field(description="Generated report title.")
summary: str = Field(description="Generated factual report summary.")
checks_summary: str | None = Field(default=None, description="Summary to apply only with reconciled checks.")
+ checks_error: str | None = None
tags: list[str] | None = Field(
default=None,
description="The report's whole tag set, replaced with title and summary. `None` keeps the stored set.",
@@ -1974,6 +1953,7 @@ async def run_multi_turn_research(
verification_note: NoteArtefact | None = None
checks: list[CheckSpec] | None = None
checks_summary: str | None = None
+ checks_error: str | None = None
if actionability_result.actionability != ActionabilityChoice.NOT_ACTIONABLE:
if output_fn:
output_fn("Generating fix verification steps...")
@@ -1990,7 +1970,12 @@ async def run_multi_turn_research(
verification_prompt,
FixVerificationOutput,
label="fix_verification",
- json_retry_prompt=RESEARCH_JSON_RETRY_PROMPT,
+ max_json_retries=4,
+ json_retry_prompt=(
+ "Correct only the verification plan from your previous reply using the validation errors below. "
+ "Preserve the completed research. Do not do more research or change other report sections. "
+ "Reply with only the corrected verification JSON object matching the schema above."
+ ),
validation_context={
"report_id": signal_report_id,
"team_id": context.team_id,
@@ -2006,7 +1991,8 @@ async def run_multi_turn_research(
):
checks = list(verification_result.checks)
checks_summary = verification_result.summary
- except Exception:
+ except Exception as error:
+ checks_error = _rejection_reason(error)
logger.exception(
"multi_turn_research: failed to generate fix verification note",
extra={
@@ -2086,6 +2072,7 @@ async def run_multi_turn_research(
verification_note=verification_note,
checks=checks,
checks_summary=checks_summary,
+ checks_error=checks_error,
old_artefacts=old_artefacts,
new_artefacts=new_artefacts,
)
diff --git a/products/signals/backend/report_monitoring_attempts.py b/products/signals/backend/report_monitoring_attempts.py
index f3b718338012..6d1129e9f972 100644
--- a/products/signals/backend/report_monitoring_attempts.py
+++ b/products/signals/backend/report_monitoring_attempts.py
@@ -17,6 +17,7 @@ from products.signals.backend.models import (
SignalReportMonitoringAttempt,
)
from products.signals.backend.report_check_artefacts import write_check_expired
+from products.signals.backend.report_check_research import check_authoring_failed
from products.signals.backend.report_content_gates import team_report_monitoring_enabled
if TYPE_CHECKING:
@@ -108,6 +109,8 @@ def complete_implementation(
else:
applicable.append(check)
if not applicable:
+ if check_authoring_failed(report):
+ raise ValueError("Follow-up check authoring failed. Configure verification or resolve manually.")
report._close_pr_on_resolve = close_pr_on_resolve # type: ignore[attr-defined]
report.save(update_fields=report.transition_to(SignalReport.Status.RESOLVED))
return None
diff --git a/products/signals/backend/temporal/agentic/report.py b/products/signals/backend/temporal/agentic/report.py
index e78abb406868..ac1e5538726c 100644
--- a/products/signals/backend/temporal/agentic/report.py
+++ b/products/signals/backend/temporal/agentic/report.py
@@ -153,6 +153,7 @@ class RunAgenticReportOutput:
# reconcile existing rows only when this marker came from a new verification turn.
reconcile_checks: bool = False
checks_summary: str | None = None
+ checks_error: str | None = None
# The plan of dependent pull requests, as `ReportLayer` dicts. The ready transition turns each
# one into a child report. `None` predates the field and creates none.
layers: list[dict[str, Any]] | None = None
@@ -1317,6 +1318,7 @@ async def _run_agentic_report(input: RunAgenticReportInput) -> RunAgenticReportO
else None,
reconcile_checks=result.checks is not None,
checks_summary=result.checks_summary,
+ checks_error=result.checks_error,
checks_snapshot=checks_snapshot,
checks_monitoring_enabled=monitoring_enabled,
layers=[layer.model_dump(mode="json") for layer in result.layers],
diff --git a/products/signals/backend/temporal/summary.py b/products/signals/backend/temporal/summary.py
index 261592f90c5a..a68401649002 100644
--- a/products/signals/backend/temporal/summary.py
+++ b/products/signals/backend/temporal/summary.py
@@ -37,7 +37,7 @@ from posthog.temporal.common.scoped import scoped_temporal
from posthog.temporal.common.utils import close_db_connections
from products.signals.backend.artefact_attribution import ArtefactAttribution
-from products.signals.backend.artefact_schemas import NoteArtefact
+from products.signals.backend.artefact_schemas import CheckAuthoringStatus, NoteArtefact
from products.signals.backend.auto_start import (
RequestedImplementation,
maybe_autostart_from_report_artefacts,
@@ -231,6 +231,7 @@ class ReportDecision:
checks: list[dict[str, Any]] | None = None
reconcile_checks: bool = False
checks_summary: str | None = None
+ checks_error: str | None = None
checks_snapshot: dict[str, str] | None = None
checks_monitoring_enabled: bool = False
layers: list[dict[str, Any]] = field(default_factory=list)
@@ -686,6 +687,7 @@ class SignalReportSummaryWorkflow:
checks=agentic_result.checks,
reconcile_checks=agentic_result.reconcile_checks,
checks_summary=agentic_result.checks_summary,
+ checks_error=agentic_result.checks_error,
checks_snapshot=agentic_result.checks_snapshot,
checks_monitoring_enabled=agentic_result.checks_monitoring_enabled,
layers=agentic_result.layers or [],
@@ -735,6 +737,7 @@ class SignalReportSummaryWorkflow:
checks_monitoring_enabled=decision.checks_monitoring_enabled,
reconcile_checks=decision.reconcile_checks,
checks_summary=decision.checks_summary,
+ checks_error=decision.checks_error,
checks_task_id=decision.research_task_id,
suggested_prompts=decision.suggested_prompts,
charts_enabled=decision.charts_enabled,
@@ -766,6 +769,7 @@ class SignalReportSummaryWorkflow:
checks_monitoring_enabled=decision.checks_monitoring_enabled,
reconcile_checks=decision.reconcile_checks,
checks_summary=decision.checks_summary,
+ checks_error=decision.checks_error,
checks_task_id=decision.research_task_id,
layers=decision.layers,
suggested_prompts=decision.suggested_prompts,
@@ -1119,6 +1123,7 @@ class MarkReportReadyInput:
reconcile_checks: bool = False
checks_summary: str | None = None
+ checks_error: str | None = None
# Task the check rows are attributed to: the research sandbox that authored the specs.
checks_task_id: str | None = None
# The research plan of dependent pull requests, as `ReportLayer` dicts. Each becomes a child
@@ -1151,13 +1156,35 @@ def _observation_metrics(report: SignalReport, metrics: list[dict]) -> list[dict
return observations
-def _write_research_checks(report: SignalReport, input: "MarkReportReadyInput | MarkReportPendingInput") -> None:
- """Persist the research run's check specs when its report settles.
+def _write_check_authoring_status(
+ report: SignalReport, input: "MarkReportReadyInput | MarkReportPendingInput", error: str | None
+) -> None:
+ attribution = (
+ ArtefactAttribution.from_task(input.checks_task_id) if input.checks_task_id else ArtefactAttribution.system()
+ )
+ SignalReportArtefact.append_status(
+ team_id=report.team_id,
+ report_id=str(report.id),
+ content=CheckAuthoringStatus(status="failed" if error else "succeeded", reason=error or "checks_saved"),
+ attribution=attribution,
+ )
+ if error:
+ SignalReportArtefact.add_log(
+ team_id=report.team_id,
+ report_id=str(report.id),
+ content=NoteArtefact(
+ note="Could not configure follow-up checks. Existing checks and completed research were preserved. "
+ "Research the report again to configure verification, or verify and resolve it manually. "
+ "Automatic resolution without checks is blocked while Monitoring is enabled."
+ ),
+ attribution=attribution,
+ )
- Best-effort as a whole: the report's prose is what this transition exists to write, so a spec
- the pipeline cannot store is dropped with a log rather than failing the transition and leaving
- the report stuck in progress.
- """
+
+def _write_research_checks(report: SignalReport, input: "MarkReportReadyInput | MarkReportPendingInput") -> None:
+ if input.checks_error is not None:
+ _write_check_authoring_status(report, input, input.checks_error)
+ return
if input.checks is None or (not input.reconcile_checks and not input.checks):
return
# Function-local: the authoring module reaches the alerts facade through the check executor,
@@ -1171,6 +1198,7 @@ def _write_research_checks(report: SignalReport, input: "MarkReportReadyInput |
specs.append(CheckSpec.model_validate(raw))
except Exception:
logger.warning("signals report check spec did not validate", report_id=str(report.id))
+ _write_check_authoring_status(report, input, "invalid_check_spec")
return
result = create_checks_from_specs(
report=report,
@@ -1184,6 +1212,7 @@ def _write_research_checks(report: SignalReport, input: "MarkReportReadyInput |
else ArtefactAttribution.system()
),
)
+ _write_check_authoring_status(report, input, None if result.applied else "check_reconciliation_failed")
if input.reconcile_checks and result.applied and input.checks_summary is not None:
report.summary = input.checks_summary
report.save(update_fields=["summary"])
@@ -1534,6 +1563,7 @@ class MarkReportPendingInput:
checks_monitoring_enabled: bool = False
reconcile_checks: bool = False
checks_summary: str | None = None
+ checks_error: str | None = None
checks_task_id: str | None = None
# See MarkReportReadyInput.suggested_prompts — same transaction, same three states.
suggested_prompts: list[str] | None = None
diff --git a/products/signals/backend/test/test_agentic_report_activity.py b/products/signals/backend/test/test_agentic_report_activity.py
index 9b8fba44d210..a8d60c41ba1b 100644
--- a/products/signals/backend/test/test_agentic_report_activity.py
+++ b/products/signals/backend/test/test_agentic_report_activity.py
@@ -8,7 +8,7 @@ from uuid import UUID, uuid4
import pytest
from unittest.mock import AsyncMock, Mock, patch
-from django.db import OperationalError
+from django.db import OperationalError, transaction
from django.utils import timezone
import pytest_asyncio
@@ -40,12 +40,15 @@ from products.signals.backend.models import (
SignalReport,
SignalReportArtefact,
SignalReportCheck,
+ SignalScoutConfig,
SignalScoutNote,
)
+from products.signals.backend.pull_requests import link_pull_request, reconcile_reports_for_pull_request
from products.signals.backend.repo_corrections import SCOUT_REPOSITORY_REASON
+from products.signals.backend.report_assignments import PullRequestDetails
from products.signals.backend.report_charts import ReportChart
+from products.signals.backend.report_check_agent import FALLBACK_CHECK_SKILL_NAME
from products.signals.backend.report_generation.research import (
- RESEARCH_JSON_RETRY_PROMPT,
ActionabilityAssessment,
ActionabilityChoice,
ActionabilityUpdate,
@@ -89,9 +92,11 @@ from products.signals.backend.temporal.summary import (
mark_report_ready_activity,
)
from products.signals.backend.temporal.types import SignalData
+from products.skills.backend.models.skills import LLMSkill
from products.tasks.backend.facade.agents import (
AgentStartupFailed,
CustomPromptSandboxContext,
+ MultiTurnSession,
RepositoryContextUnavailable,
SandboxRunFailed,
)
@@ -1858,6 +1863,155 @@ async def test_research_returns_reviewer_decision_only_when_enabled(enabled: boo
assert result.reviewer_decision == (decision if enabled else None)
[email protected]("correction_attempt,existing_check", [(1, False), (4, False), (None, False), (None, True)])
+async def test_verification_correction_persists_before_pr_merge(
+ ateam: Team, correction_attempt: int | None, existing_check: bool
+) -> None:
+ corrected = correction_attempt is not None
+ report = await SignalReport.objects.acreate(
+ team=ateam, status=SignalReport.Status.IN_PROGRESS, signal_count=1, total_weight=1
+ )
+ task = await Task.objects.acreate(team=ateam, title="Research verification")
+ await LLMSkill.objects.acreate(team=ateam, name=FALLBACK_CHECK_SKILL_NAME, is_latest=True, deleted=False)
+ await database_sync_to_async(
+ lambda: SignalScoutConfig.objects.for_team(ateam.id).create(team=ateam, skill_name=FALLBACK_CHECK_SKILL_NAME)
+ )()
+ previous = None
+ if existing_check:
+ previous = await SignalReportCheck.objects.for_team(ateam.id).acreate(
+ team_id=ateam.id,
+ report=report,
+ title="Existing verification",
+ kind="agent",
+ config={"instructions": "Verify the outcome."},
+ status="pending",
+ soak_minutes=360,
+ next_run_at=timezone.now() + timedelta(hours=6),
+ expires_at=timezone.now() + timedelta(days=30),
+ )
+ finding = SignalFinding(signal_id="sig-1", relevant_code_paths=[], data_queried="", verified=True)
+ proposal = {
+ "current_state": "Confirm the issue in current evidence.",
+ "outcome": "Confirm recovery after deployment.",
+ "checks": [
+ {
+ "kind": "agent",
+ "title": "Verify recovery",
+ "soak_hours": 6,
+ "config": {
+ "instructions": "Verify recovery with traffic.",
+ "probe_hints": [f"Probe {i}" for i in range(7)],
+ },
+ }
+ ],
+ }
+ correction = json.loads(json.dumps(proposal))
+ if corrected:
+ correction["checks"][0]["config"]["probe_hints"] = ["Inspect traffic", "Confirm recovery"]
+ replies = [
+ json.dumps(
+ {
+ "explanation": "The issue needs a fix.",
+ "actionability": "immediately_actionable",
+ "already_addressed": False,
+ }
+ ),
+ json.dumps({"explanation": "The issue affects users.", "priority": "P2", "dollar_value": 1000}),
+ json.dumps({"title": "fix: restore recovery", "summary": "Users need the recovery fix."}),
+ ]
+ replies.extend([json.dumps(proposal)] * (correction_attempt if correction_attempt is not None else 5))
+ if corrected:
+ replies.append(json.dumps(correction))
+ session = MultiTurnSession(task=task, task_run=Mock(id=uuid4()))
+ with (
+ patch.object(MultiTurnSession, "start", AsyncMock(return_value=(session, finding))),
+ patch.object(MultiTurnSession, "send_followup_raw", AsyncMock(side_effect=replies)) as raw,
+ patch.object(MultiTurnSession, "end", AsyncMock()),
+ patch("products.signals.backend.task_run_artefacts.aappend_task_run_artefact", new_callable=AsyncMock),
+ ):
+ result = await run_multi_turn_research(
+ _build_signals()[:1], Mock(team_id=ateam.id), signal_report_id=str(report.id), agent_checks_enabled=True
+ )
+ assert raw.await_count == 3 + (correction_attempt + 1 if correction_attempt is not None else 5)
+ correction_prompt = raw.await_args_list[-1].args[0]
+ assert "probe_hints" in correction_prompt
+ assert "at most 5 items" in correction_prompt
+ assert "7" in correction_prompt
+ assert "Correct only the verification plan" in correction_prompt
+ assert "Do not do more research" in correction_prompt
+ assert result.effective_findings() == [finding]
+ assert result.summary == "Users need the recovery fix."
+ await mark_report_ready_activity(
+ MarkReportReadyInput(
+ team_id=ateam.id,
+ report_id=str(report.id),
+ title=result.title,
+ summary=result.summary,
+ processed_signal_count=1,
+ checks=[check.model_dump(mode="json") for check in result.checks] if result.checks is not None else None,
+ checks_error=result.checks_error,
+ reconcile_checks=result.checks is not None,
+ checks_task_id=str(task.id),
+ )
+ )
+ authoring = await SignalReportArtefact.objects.filter(report=report, type="check_authoring").alatest("created_at")
+ assert json.loads(authoring.content)["status"] == ("succeeded" if corrected else "failed")
+ if corrected:
+ check = await SignalReportCheck.objects.for_team(ateam.id).aget(report=report)
+ assert check.soak_minutes == 360
+ assert check.config["probe_hints"] == ["Inspect traffic", "Confirm recovery"]
+ else:
+ assert await SignalReportArtefact.objects.filter(
+ report=report, type="note", content__contains="Could not configure"
+ ).aexists()
+ assert await SignalReportCheck.objects.for_team(ateam.id).filter(report=report).acount() == int(existing_check)
+ if previous is not None:
+ await previous.arefresh_from_db()
+ assert previous.status == "pending"
+ assert previous.title == "Existing verification"
+ assert previous.config == {"instructions": "Verify the outcome."}
+ with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True):
+ pr = await database_sync_to_async(transaction.atomic()(link_pull_request))(
+ report=report,
+ details=PullRequestDetails(
+ url="https://github.com/example/app/pull/1",
+ repository="example/app",
+ number=1,
+ state="merged",
+ merged=True,
+ merged_at=timezone.now(),
+ ),
+ actor=ArtefactAttribution.system(),
+ claim_id=None,
+ notify_reviewers=False,
+ )
+ await database_sync_to_async(reconcile_reports_for_pull_request)(team_id=ateam.id, pr_id=str(pr.id))
+ await report.arefresh_from_db()
+ assert report.status == (
+ SignalReport.Status.MONITORING if corrected or existing_check else SignalReport.Status.READY
+ )
+ if not corrected and not existing_check:
+ report.status = SignalReport.Status.IN_PROGRESS
+ await report.asave(update_fields=["status"])
+ await mark_report_ready_activity(
+ MarkReportReadyInput(
+ team_id=ateam.id,
+ report_id=str(report.id),
+ title=result.title,
+ summary=result.summary,
+ processed_signal_count=1,
+ checks=[],
+ reconcile_checks=True,
+ )
+ )
+ with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True):
+ await database_sync_to_async(reconcile_reports_for_pull_request)(team_id=ateam.id, pr_id=str(pr.id))
+ await report.arefresh_from_db()
+ assert report.status == SignalReport.Status.RESOLVED
+
+
@parameterized.expand(
[
("immediately_actionable", ActionabilityChoice.IMMEDIATELY_ACTIONABLE, None),
@@ -1865,7 +2019,6 @@ async def test_research_returns_reviewer_decision_only_when_enabled(enabled: boo
("not_actionable", ActionabilityChoice.NOT_ACTIONABLE, None),
("timeout", ActionabilityChoice.IMMEDIATELY_ACTIONABLE, TimeoutError),
("validation_failure", ActionabilityChoice.IMMEDIATELY_ACTIONABLE, ValidationError),
- ("malformed_optional_checks", ActionabilityChoice.IMMEDIATELY_ACTIONABLE, None),
("revised_metric_summary", ActionabilityChoice.IMMEDIATELY_ACTIONABLE, None),
("cancellation", ActionabilityChoice.IMMEDIATELY_ACTIONABLE, asyncio.CancelledError),
]
@@ -1918,9 +2071,7 @@ async def test_run_multi_turn_research_requests_verification_note_as_the_final_a
"current_state": "Run query-trends for onboarding_completed over the same 14-day window.",
"outcome": "Confirm event volume returns to the pre-regression baseline.",
**(
- {"checks": [{"kind": "metric_threshold", "title": "A malformed check"}]}
- if _name == "malformed_optional_checks"
- else {
+ {
"checks": [
{
"title": "Completion returns to baseline",
@@ -1959,7 +2110,7 @@ async def test_run_multi_turn_research_requests_verification_note_as_the_final_a
_build_signals()[:1],
Mock(team_id=1),
signal_report_id="report-id",
- metrics_enabled=_name in ("malformed_optional_checks", "revised_metric_summary"),
+ metrics_enabled=_name == "revised_metric_summary",
monitoring_enabled=_name == "revised_metric_summary",
)
assert result.effective_findings() == [first_finding]
@@ -1973,6 +2124,7 @@ async def test_run_multi_turn_research_requests_verification_note_as_the_final_a
else None
)
assert result.research_task_id == "research-task-id"
+ assert (result.checks_error is not None) is (failure is not None)
if actionability != ActionabilityChoice.NOT_ACTIONABLE and failure is None:
assert result.verification_note is not None
if _name == "revised_metric_summary":
@@ -1986,7 +2138,7 @@ async def test_run_multi_turn_research_requests_verification_note_as_the_final_a
else:
assert result.verification_note is None
- preferred = _name in ("malformed_optional_checks", "revised_metric_summary")
+ preferred = _name == "revised_metric_summary"
assert ("## Prepare metric-based follow-up checks" in start_session.call_args.kwargs["prompt"]) is preferred
prompts = {call.kwargs["label"]: call.args[0] for call in session.send_followup.await_args_list}
assert ("## Metrics for follow-up checks" in prompts["presentation"]) is preferred
@@ -2065,8 +2217,9 @@ async def test_run_multi_turn_research_survives_a_failed_supersede_turn(supersed
"supersede": supersede_outcome,
}
- async def fake_send_followup(message, model, *, label="", validation_context=None, json_retry_prompt=None):
- assert json_retry_prompt == RESEARCH_JSON_RETRY_PROMPT
+ async def fake_send_followup(
+ message, model, *, label="", validation_context=None, json_retry_prompt=None, max_json_retries=1
+ ):
outcome = by_label[label]
if isinstance(outcome, Exception):
raise outcome
@@ -2160,8 +2313,9 @@ async def test_run_multi_turn_research_only_asks_about_the_pr_when_actionable(ac
}
asked_labels: list[str] = []
- async def fake_send_followup(message, model, *, label="", validation_context=None, json_retry_prompt=None):
- assert json_retry_prompt == RESEARCH_JSON_RETRY_PROMPT
+ async def fake_send_followup(
+ message, model, *, label="", validation_context=None, json_retry_prompt=None, max_json_retries=1
+ ):
asked_labels.append(label)
return by_label[label]
diff --git a/products/signals/backend/test/test_plan_rollup.py b/products/signals/backend/test/test_plan_rollup.py
index 1bb33fe7af68..11e34ce10c46 100644
--- a/products/signals/backend/test/test_plan_rollup.py
+++ b/products/signals/backend/test/test_plan_rollup.py
@@ -1,4 +1,5 @@
from concurrent.futures import ThreadPoolExecutor
+from typing import Literal
import pytest
from posthog.test.base import BaseTest
@@ -9,7 +10,7 @@ from django.utils import timezone
from parameterized import parameterized
-from products.signals.backend.artefact_schemas import Dismissal, PullRequestLink, ReportLink
+from products.signals.backend.artefact_schemas import CheckAuthoringStatus, Dismissal, PullRequestLink, ReportLink
from products.signals.backend.enums import ReportLinkKind, SignalSourceProduct
from products.signals.backend.models import (
ArtefactAttribution,
@@ -88,6 +89,10 @@ class TestPlanRollup(BaseTest):
SignalReport.Status.MONITORING,
SignalReport.Status.MONITORING,
),
+ ("failed_authoring_blocks_resolution", ["resolved"], "ready", "ready", False, "failed", True),
+ ("successful_authoring_allows_resolution", ["resolved"], "resolved", "ready", False, "succeeded", True),
+ ("disabled_authoring_failure_resolves", ["resolved"], "resolved", "ready", False, "failed", False),
+ ("failed_authoring_allows_archive", ["suppressed"], "suppressed", "ready", False, "failed", True),
]
)
def test_a_plan_takes_the_verdict_of_its_steps(
@@ -97,15 +102,26 @@ class TestPlanRollup(BaseTest):
expected: str,
parent_status: str = SignalReport.Status.READY,
archive_parent: bool = False,
+ authoring_status: Literal["failed", "succeeded"] | None = None,
+ monitoring_enabled: bool = True,
) -> None:
parent = self._report("plan", status=parent_status)
+ if authoring_status is not None:
+ SignalReportArtefact.append_status(
+ team_id=self.team.id,
+ report_id=str(parent.id),
+ content=CheckAuthoringStatus(status=authoring_status, reason="Verification proposal validation"),
+ attribution=ArtefactAttribution.system(),
+ )
if archive_parent:
self._close(parent, SignalReport.Status.SUPPRESSED)
children = [self._report(f"step-{index}") for index in range(len(child_statuses))]
for child in children:
self._part_of(child, parent)
- with patch("products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=True):
+ with patch(
+ "products.signals.backend.report_content_gates.feature_enabled_or_false", return_value=monitoring_enabled
+ ):
for child, status in zip(children, child_statuses):
self._close(child, status)
diff --git a/products/signals/backend/test/test_report_checks.py b/products/signals/backend/test/test_report_checks.py
index 203fcbff2164..0d8d0084be0d 100644
--- a/products/signals/backend/test/test_report_checks.py
+++ b/products/signals/backend/test/test_report_checks.py
@@ -2960,22 +2960,38 @@ class TestResearchAuthoredChecks(APIBaseTest):
assert existing.updated_at == version
assert existing.approved_at is not None
- @parameterized.expand([("no_lane", False), ("lane", True)])
- def test_research_drops_an_agent_spec_with_no_lane_and_keeps_the_rest(self, _name: str, has_lane: bool) -> None:
+ @parameterized.expand([("no_lane", False, False), ("lane", True, False), ("only_agent_no_lane", False, True)])
+ def test_research_drops_an_agent_spec_with_no_lane_and_keeps_the_rest(
+ self, _name: str, has_lane: bool, only_agent: bool
+ ) -> None:
if has_lane:
_seed_check_lane(self.team)
agent_spec = self._spec(title="The exception stops", kind="agent", config={"instructions": "Re-read it."})
+ original = None
+ if only_agent:
+ original = create_checks_from_specs(
+ report=self.report, specs=[self._spec()], attribution=ArtefactAttribution.system()
+ ).created[0]
with patch(_CAPTURE) as capture:
- written = create_checks_from_specs(
- report=self.report, specs=[self._spec(), agent_spec], attribution=ArtefactAttribution.system()
- ).created
+ result = create_checks_from_specs(
+ report=self.report,
+ specs=[agent_spec] if only_agent else [self._spec(), agent_spec],
+ attribution=ArtefactAttribution.system(),
+ )
+ written = result.created
kinds = sorted(check.kind for check in written)
skipped = [c for c in capture.call_args_list if c.kwargs["event"] == "signals_report_check_skipped"]
if has_lane:
assert kinds == ["agent", "metric_threshold"]
assert skipped == []
+ elif only_agent:
+ assert not result.applied
+ assert written == []
+ assert original is not None
+ original.refresh_from_db()
+ assert original.status == SignalReportCheck.Status.PENDING
else:
assert kinds == ["metric_threshold"]
assert len(skipped) == 1
diff --git a/products/signals/backend/test/test_report_monitoring_attempts.py b/products/signals/backend/test/test_report_monitoring_attempts.py
index 37db03f41414..bb72dc8adea8 100644
--- a/products/signals/backend/test/test_report_monitoring_attempts.py
+++ b/products/signals/backend/test/test_report_monitoring_attempts.py
@@ -12,9 +12,11 @@ from django.utils import timezone
from parameterized import parameterized
+from posthog.dbrouter import ReplicaRouter
from posthog.models import Team
from products.signals.backend.artefact_attribution import ArtefactAttribution
+from products.signals.backend.artefact_schemas import CheckAuthoringStatus
from products.signals.backend.models import (
InvalidStatusTransition,
SignalReport,
@@ -486,6 +488,22 @@ class TestReportMonitoringAttempts(BaseTest):
== 1
)
+ def test_authoring_failure_blocks_completion_with_replica_routing(self) -> None:
+ SignalReportArtefact.append_status(
+ team_id=self.team.id,
+ report_id=str(self.report.id),
+ content=CheckAuthoringStatus(status="failed", reason="Invalid verification proposal"),
+ attribution=ArtefactAttribution.system(),
+ )
+ with (
+ self.settings(DATABASE_ROUTERS=[ReplicaRouter(["SignalReportArtefact"])]),
+ self.assertRaisesRegex(ValueError, "Follow-up check authoring failed"),
+ ):
+ complete_implementation(team_id=self.team.id, report_id=self.report.id, monitoring_started_at=self.now)
+ self.report.refresh_from_db()
+ assert self.report.status == SignalReport.Status.READY
+ assert not SignalReportMonitoringAttempt.objects.for_team(self.team.id).filter(report=self.report).exists()
+
def test_expiry_log_failure_rolls_back_entry_and_check_changes(self) -> None:
stale = self._check(status=SignalReportCheck.Status.ACTIVE)
stale.expires_at = self.now
diff --git a/products/signals/backend/test/test_research_prompt.py b/products/signals/backend/test/test_research_prompt.py
index 158816211e0b..e6d292b4cda9 100644
--- a/products/signals/backend/test/test_research_prompt.py
+++ b/products/signals/backend/test/test_research_prompt.py
@@ -6,7 +6,6 @@ from typing import Literal
from xml.etree import ElementTree
import pytest
-from unittest.mock import patch
from pydantic import ValidationError
@@ -491,11 +490,9 @@ class TestBuildFixVerificationPrompt:
* (MAX_ACTIVE_CHECKS_PER_REPORT + 1),
],
)
- def test_invalid_checks_preserve_prose_without_requesting_reconciliation(
- self, checks: list[dict[str, object]]
- ) -> None:
- with patch("products.signals.backend.report_generation.research.logger.warning") as warning:
- result = FixVerificationOutput.model_validate(
+ def test_invalid_checks_require_correction(self, checks: list[dict[str, object]]) -> None:
+ with pytest.raises(ValidationError):
+ FixVerificationOutput.model_validate(
{
"current_state": "Check the current issue.",
"outcome": "Check it again after the fix.",
@@ -503,17 +500,6 @@ class TestBuildFixVerificationPrompt:
},
context={"report_id": "test-report"},
)
- assert result.checks is None
- assert result.to_note().note == (
- "## Verification plan\n\n### Confirm the current state\n\nCheck the current issue.\n\n"
- "### Confirm the outcome\n\nCheck it again after the fix."
- )
-
- warning.assert_called_once()
- details = warning.call_args.kwargs["extra"]
- assert details["report_id"] == "test-report"
- assert details["validation_rules"]
- assert "checks" not in details
def _make_chart() -> ReportChart:
diff --git a/products/signals/backend/test/test_signal_report_artefact_api.py b/products/signals/backend/test/test_signal_report_artefact_api.py
index 076f07f79ad0..7378bd7d2332 100644
--- a/products/signals/backend/test/test_signal_report_artefact_api.py
+++ b/products/signals/backend/test/test_signal_report_artefact_api.py
@@ -54,7 +54,7 @@ def _attach_github_login(user: User, login: str, *, uid: str | None = None) -> N
class TestSignalReportArtefactViewSet(APIBaseTest):
- @parameterized.expand([("impact_measurement_plan",), ("reviewer_corrections",)])
+ @parameterized.expand([("impact_measurement_plan",), ("reviewer_corrections",), ("check_authoring",)])
def test_system_decision_type_cannot_be_written(self, artefact_type: str) -> None:
report = self._create_report()
response = self.client.post(
diff --git a/products/signals/backend/test/test_summary_workflow.py b/products/signals/backend/test/test_summary_workflow.py
index 42c9e56606bc..c85a226e7064 100644
--- a/products/signals/backend/test/test_summary_workflow.py
+++ b/products/signals/backend/test/test_summary_workflow.py
@@ -205,6 +205,7 @@ class _Recorder:
self.research_metrics = research_metrics
self.checks_snapshot = checks_snapshot
self.checks_summary: str | None = None
+ self.checks_error: str | None = None
self.checks_monitoring_enabled = False
self.selected_repository = selected_repository
self.research_errors = list(research_errors or [])
@@ -322,6 +323,7 @@ async def _run_summary_workflow(recorder: _Recorder) -> None:
checks_snapshot=recorder.checks_snapshot,
checks_monitoring_enabled=recorder.checks_monitoring_enabled,
checks_summary=recorder.checks_summary,
+ checks_error=recorder.checks_error,
)
@activity.defn(name="mark_report_pending_input_activity")
@@ -526,11 +528,13 @@ async def test_research_startup_retry_policy_and_failure_details_survive_tempora
(ActionabilityChoice.IMMEDIATELY_ACTIONABLE, "ready"),
],
)
-async def test_metric_payload_reaches_the_report_transition(choice, target):
[email protected]("checks_error", [None, "checks.0.probe_hints: too_long"])
+async def test_metric_payload_reaches_the_report_transition(choice, target, checks_error):
metrics = [{"metric_id": "affected-users", "kind": "affected_users", "value": 12}]
snapshot = {"check-1": "2026-10-02T12:00:00+00:00"}
recorder = _Recorder(research_choice=choice, research_metrics=metrics, checks_snapshot=snapshot)
recorder.checks_summary = "Expected impact matches the reconciled checks."
+ recorder.checks_error = checks_error
recorder.checks_monitoring_enabled = True
await _run_summary_workflow(recorder)
@@ -541,6 +545,7 @@ async def test_metric_payload_reaches_the_report_transition(choice, target):
assert inputs[0].checks_snapshot == snapshot
assert inputs[0].checks_monitoring_enabled is True
assert inputs[0].checks_summary == recorder.checks_summary
+ assert inputs[0].checks_error == checks_error
@pytest.mark.asyncio
diff --git a/products/signals/frontend/generated/api.schemas.ts b/products/signals/frontend/generated/api.schemas.ts
index ccf4255e3642..80ec49531b1a 100644
--- a/products/signals/frontend/generated/api.schemas.ts
+++ b/products/signals/frontend/generated/api.schemas.ts
@@ -5280,6 +5280,7 @@ export interface SignalReportStateRequestApi {
* * `check_result` - Check Result
* * `check_observation` - Check Observation
* * `check_scheduled` - Check Scheduled
+ * * `check_authoring` - Check Authoring
* * `check_expired` - Check Expired
* * `check_cancelled` - Check Cancelled
* * `implementation_decision` - Implementation Decision
@@ -5321,6 +5322,7 @@ export const SignalReportArtefactArtefactTypeEnumApi = {
CheckResult: 'check_result',
CheckObservation: 'check_observation',
CheckScheduled: 'check_scheduled',
+ CheckAuthoring: 'check_authoring',
CheckExpired: 'check_expired',
CheckCancelled: 'check_cancelled',
ImplementationDecision: 'implementation_decision',
diff --git a/products/signals/frontend/generated/api.ts b/products/signals/frontend/generated/api.ts
index 1322f4965f82..6e52ea5c76ac 100644
--- a/products/signals/frontend/generated/api.ts
+++ b/products/signals/frontend/generated/api.ts
@@ -1297,7 +1297,7 @@ export const getSignalsReportArtefactsDestroyUrl = (projectId: string, reportId:
}
/**
- * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_cancelled`, `check_expired`, `check_observation`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pr_owner_assignment`, `pull_request`, `ranking_score`, `report_link`, `reviewer_corrections`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.
+ * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_authoring`, `check_cancelled`, `check_expired`, `check_observation`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pr_owner_assignment`, `pull_request`, `ranking_score`, `report_link`, `reviewer_corrections`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.
* @summary Delete an artefact
*/
export const signalsReportArtefactsDestroy = async (
diff --git a/products/tasks/backend/logic/services/custom_prompt_multi_turn_runner.py b/products/tasks/backend/logic/services/custom_prompt_multi_turn_runner.py
index d89dc4a6b489..d7d974584de8 100644
--- a/products/tasks/backend/logic/services/custom_prompt_multi_turn_runner.py
+++ b/products/tasks/backend/logic/services/custom_prompt_multi_turn_runner.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import json
import time
import asyncio
import logging
@@ -349,26 +350,40 @@ class MultiTurnSession:
label: str = "",
validation_context: dict[str, object] | None = None,
json_retry_prompt: str | None = None,
+ max_json_retries: int = 1,
) -> _ModelT:
- """Send a structured follow-up, optionally asking once for corrected JSON on parse or validation failure."""
- last_message = await self.send_followup_raw(message, label=label)
- try:
- return self._parse_and_validate(
- last_message, model, label=label or "followup", validation_context=validation_context
- )
- except ValueError as error:
- if json_retry_prompt is None:
- raise
- logger.warning(
- "multi_turn: followup did not validate against %s for run=%s label=%s, asking for JSON once (%s)",
- model.__name__,
- self.task_run.id,
- label,
- type(error).__name__,
- )
- retry_label = f"{label or 'followup'} json retry"
- retry_message = await self.send_followup_raw(json_retry_prompt, label=retry_label)
- return self._parse_and_validate(retry_message, model, label=retry_label, validation_context=validation_context)
+ """Correct invalid structured output on the same session, within the caller's retry budget."""
+ if max_json_retries < 0:
+ raise ValueError("max_json_retries must not be negative")
+ retry_count = 0
+ turn_message = message
+ turn_label = label
+ while True:
+ last_message = await self.send_followup_raw(turn_message, label=turn_label)
+ try:
+ return self._parse_and_validate(
+ last_message, model, label=turn_label or "followup", validation_context=validation_context
+ )
+ except ValueError as error:
+ if json_retry_prompt is None or retry_count >= max_json_retries:
+ raise
+ retry_count += 1
+ logger.warning(
+ "multi_turn: followup did not validate against %s for run=%s label=%s, "
+ "asking for corrected JSON (retry %d/%d, %s)",
+ model.__name__,
+ self.task_run.id,
+ label,
+ retry_count,
+ max_json_retries,
+ type(error).__name__,
+ )
+ turn_message = json_retry_prompt
+ if isinstance(error, ValidationError):
+ turn_message += "\n\nCorrect these validation errors in your previous output:\n" + json.dumps(
+ error.errors(include_input=False, include_context=False, include_url=False)
+ )
+ turn_label = f"{label or 'followup'} json retry {retry_count}"
async def send_followup_raw(
self,
diff --git a/products/tasks/backend/tests/test_multi_turn_session.py b/products/tasks/backend/tests/test_multi_turn_session.py
index 11fb59c5e845..bb8b658cfc06 100644
--- a/products/tasks/backend/tests/test_multi_turn_session.py
+++ b/products/tasks/backend/tests/test_multi_turn_session.py
@@ -1378,12 +1378,28 @@ class TestMultiTurnSessionRetry:
@pytest.mark.asyncio
@pytest.mark.parametrize("first_reply", ["private-reply-text", '{"value":', '{"value": "private-value"}'])
- @pytest.mark.parametrize("retry_reply", [None, '{"value": "ok"}', '{"value": "still-invalid"}'])
+ @pytest.mark.parametrize(
+ "retry_reply,max_json_retries,invalid_corrections",
+ [
+ (None, 1, 0),
+ ('{"value": "ok"}', 1, 0),
+ ('{"value": "still-invalid"}', 1, 0),
+ ('{"value": "ok"}', 4, 3),
+ ('{"value": "still-invalid"}', 4, 3),
+ ],
+ )
async def test_json_retry_preserves_schema_and_context(
- self, first_reply: str, retry_reply: str | None, caplog: pytest.LogCaptureFixture
+ self,
+ first_reply: str,
+ retry_reply: str | None,
+ max_json_retries: int,
+ invalid_corrections: int,
+ caplog: pytest.LogCaptureFixture,
) -> None:
session = self._make_session()
- replies = [first_reply] + ([retry_reply] if retry_reply is not None else [])
+ replies = [first_reply] + ['{"value": "private-value"}'] * invalid_corrections
+ if retry_reply is not None:
+ replies.append(retry_reply)
poll = AsyncMock(
side_effect=[
TurnPollResult(last_message=reply, full_log=None, total_lines=index * 10, printed_lines=index * 5)
@@ -1400,6 +1416,7 @@ class TestMultiTurnSessionRetry:
"question",
_ContextResp,
json_retry_prompt="Return valid JSON",
+ max_json_retries=max_json_retries,
validation_context=validation_context,
)
assert result.value == "ok"
@@ -1409,14 +1426,19 @@ class TestMultiTurnSessionRetry:
"question",
_ContextResp,
json_retry_prompt="Return valid JSON" if retry_reply is not None else None,
+ max_json_retries=max_json_retries,
validation_context=validation_context,
)
assert poll.await_count == len(replies)
if retry_reply is not None:
assert poll.await_args_list[1].kwargs["skip_lines"] == 10
- assert cast(AsyncMock, session._workflow_handle).signal.await_args_list[1].args[1] == "Return valid JSON"
- assert "asking for JSON once" in caplog.text
+ retry_prompt = cast(AsyncMock, session._workflow_handle).signal.await_args_list[-1].args[1]
+ assert retry_prompt.startswith("Return valid JSON")
+ if first_reply == '{"value": "private-value"}' or invalid_corrections:
+ assert retry_prompt.count("value must match the validation context") == 1
+ assert "private-value" not in retry_prompt
+ assert "asking for corrected JSON" in caplog.text
assert "private-reply-text" not in caplog.text
assert "private-value" not in caplog.text
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 0d1d9dcff065..478ad28217c9 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -80508,6 +80508,7 @@ export namespace Schemas {
* * `check_result` - Check Result
* * `check_observation` - Check Observation
* * `check_scheduled` - Check Scheduled
+ * * `check_authoring` - Check Authoring
* * `check_expired` - Check Expired
* * `check_cancelled` - Check Cancelled
* * `implementation_decision` - Implementation Decision
@@ -80549,6 +80550,7 @@ export namespace Schemas {
CheckResult: 'check_result',
CheckObservation: 'check_observation',
CheckScheduled: 'check_scheduled',
+ CheckAuthoring: 'check_authoring',
CheckExpired: 'check_expired',
CheckCancelled: 'check_cancelled',
ImplementationDecision: 'implementation_decision',
diff --git a/services/mcp/src/generated/signals/api.ts b/services/mcp/src/generated/signals/api.ts
index 037b2cf1ecb0..85061a2f5392 100644
--- a/services/mcp/src/generated/signals/api.ts
+++ b/services/mcp/src/generated/signals/api.ts
@@ -1122,7 +1122,7 @@ export const SignalsReportArtefactsPartialUpdateBody = () => zod
)
/**
- * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_cancelled`, `check_expired`, `check_observation`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pr_owner_assignment`, `pull_request`, `ranking_score`, `report_link`, `reviewer_corrections`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.
+ * Delete an artefact, addressed by id. Deleting the latest row of a status type reverts the report's canonical status to the previous version (latest-wins over what remains). `task_run` artefacts are an append-only work log and cannot be deleted. Neither can the types this API cannot write, which the pipeline owns: `autostart_skip`, `check_authoring`, `check_cancelled`, `check_expired`, `check_observation`, `check_result`, `check_scheduled`, `code_review`, `impact_measurement_plan`, `implementation_decision`, `implementation_dispatch`, `implementation_handover`, `implementation_replacement`, `pr_owner_assignment`, `pull_request`, `ranking_score`, `report_link`, `reviewer_corrections`, `summary_change`, `task_run`, `title_change`, `video_segment`, `work_claim`, `work_release`.
* @summary Delete an artefact
*/
export const SignalsReportArtefactsDestroyParams = () => zod.object({