Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
SKY-11182: verify structured run outputs before terminal success (#6751)
Andrew Neilson98e769593edb
diff --git a/skyvern/client/core/pydantic_utilities.py b/skyvern/client/core/pydantic_utilities.pyindex 8906cdfa4..b92e0b67a 100644--- a/skyvern/client/core/pydantic_utilities.py+++ b/skyvern/client/core/pydantic_utilities.py@@ -206,7 +206,15 @@ def encode_by_type(o: Any) -> Any:def update_forward_refs(model: Type["Model"], **localns: Any) -> None:if IS_PYDANTIC_V2:- model.model_rebuild(raise_errors=False) # type: ignore[attr-defined]+ try:+ model.model_rebuild(raise_errors=False) # type: ignore[attr-defined]+ except KeyError as exc:+ # Pydantic v2 can still raise internal schema-gathering KeyErrors+ # for Fern-generated recursive unions even with raise_errors=False.+ # Match on the "definitions" key rather than the exact args tuple so a+ # Pydantic format change that adds context can't reintroduce the crash.+ if "definitions" not in exc.args:+ raiseelse:model.update_forward_refs(**localns)diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index de9731f8e..b5514cee6 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -160,6 +160,8 @@ IMPORTANT WORKFLOW RULES:* Commit a draft workflow as soon as you have observed every page its blocks act on — not as soon as a reusable block sequence merely looks plausible. Use Skyvern MCP inspection tools before mutation when they are needed to resolve uncertainty about the page, especially URL-known form/search composition. When an existing workflow draft needs a well-scoped additive edit, update the YAML with update_workflow or update_and_run_blocks using a clear condition plus action target instead of answering with a prose-only block outline. Do not ask for confirmation when the workflow can be updated and tested.* Scout each page you will act on with the fast browser tools before you author the blocks that act on it. Drive the live browser through the goal path (navigate_browser, click, type_text, scroll, select_option, press_key) and call `inspect_page_for_composition` on every page you will act on — including pages reached only by clicking and pages behind a login — BEFORE authoring those blocks. `inspect_page_for_composition(target_url=...)` navigates to that URL itself and reports the reached `current_url`, so once you have inspected a URL you are already on it — do NOT then call `navigate_browser` to a URL you just inspected. A control or state that appears only after an interaction (a Delete control after an Add click, a cart after add-to-cart, the secure area after login) must be observed live by performing that interaction first; do not assume the post-interaction page from the page before it. Block-runs validate a finished workflow — they do not discover an unknown route, so do not run blocks just to see what a page looks like; for a login wall, author and run the `login` block (referencing a saved credential by id), then resume scouting from the authenticated page.* Keep `navigation_goal` scoped to one coherent page state change grounded in observed evidence, not one guessed site model and not one block per field. Do not pre-author row/detail expansion from the user's words alone: first run or inspect your way to the actual results page, then use live result-page evidence to decide whether expansion controls exist. If requested details are already visible or no explicit expansion affordance is observed, skip the expansion block and go straight to an `extraction` block. The final stage of visible results -> extract should still use an `extraction` block for data capture, not another `navigation` block. Prefer the most specific block that fits the step — use a more specific block (`action` for a single action, `login` for authentication, `file_download` for downloads, `extraction` for data) over `navigation`, and reach for `navigation` only when the step is best expressed as a goal for the agent to work out rather than a specific named action (see CHOOSING A BLOCK in the knowledge base).+* When extracting table-like or list-like records (locations, affiliations, prices, inventory, line items, statuses), iterate the actual row/item elements (`tr`, `[role="row"]`, or equivalent repeated item containers) and read fields from that same row/item. Do NOT scan broad containers such as `section`, `.card`, `article`, or `li` and then infer fields from mixed text; that can combine several rows into one record. Derive summary fields only from the parsed row objects, not from broad page text.+* When a successful search/open-details block reaches a details or profile page, include the visible details/profile page label in the block's returned `evidence_text`. Keep structured record fields focused on the requested data; use the page label only as supporting evidence.* Save the complete reusable workflow, but run only a small verified frontier when the workflow has several page-changing stages. `update_and_run_blocks` accepts the full YAML plus a subset of `block_labels`; for form/search workflows, first run the entry/form-state frontier, then continue with submit/results, then expansion/extraction. After a verified prefix or a budgeted run that leaves the browser on a later page, preserve the upstream workflow text and run only the missing downstream labels. Do not remove later blocks from the YAML just to shorten a test; do not submit a long chain of all blocks at once when each stage can be verified independently.* Set `complete_criterion` on every decomposed `navigation` block to the observable post-action page state — without it, smaller blocks still return vague "continue" verdicts.* Use descriptive, unique labels for blocks (snake_case format).@@ -226,6 +228,7 @@ RESPONSE-TYPE CLASSIFICATION:- If your `user_response` asks the user to provide, choose, confirm, or correct information required before you can continue, you MUST use ASK_QUESTION (Option 2). This holds even when this turn built or tested a partial workflow. Choosing `goal_reached: false` on a REPLY does NOT substitute for ASK_QUESTION when input is required.- A REPLY that carries `updated_workflow` can surface an Accept/Reject proposal panel for the verified workflow. Pairing that panel with a question awaiting required user input is contradictory UX — the user does not know whether to type an answer or click Accept.- Classify by intent, not punctuation: if the answer would decide what block, value, credential, URL, selector, or correction to use next AND that information is not plainly resolvable from chat_history, the current workflow_yaml, or global_llm_context, choose ASK_QUESTION — even if phrased softly (e.g. "Does that match what you wanted?"). Courtesy questions on a complete verified proposal that the user can ignore without blocking your next turn — you would proceed identically whether they answer or not — are NOT ASK_QUESTION triggers.+- If the user already specified the target record identity, required identifier, grouped rows/items with statuses, summary status, and no-results behavior, do NOT ask them to confirm output fields or schema; author and test the workflow.- ASK_QUESTION may still include `workflow_state` in `global_llm_context` to record the partial draft you built; a populated `workflow_state` does NOT imply REPLY.When you are done using tools and ready to respond to the user, output a JSON object (and nothing else) with ONE of these actions:diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.pyindex 0c9288c1e..52082b2d0 100644--- a/skyvern/forge/sdk/copilot/agent.py+++ b/skyvern/forge/sdk/copilot/agent.py@@ -1284,7 +1284,7 @@ async def _build_goal_satisfied_exit_result(ctx: CopilotContext, global_llm_contverified_workflow, verified_yaml = _verified_workflow_or_none(ctx)clean_test = ctx.last_test_ok is True and ctx.last_full_workflow_test_ok is Trueif clean_test and verified_goal_claim_authorized(ctx):- user_response = "I created and tested the workflow successfully."+ user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLYelif clean_test:user_response = ("I built the workflow and the test run completed, but the goal outcome was not "@@ -1411,6 +1411,7 @@ _MAX_TURNS_REPLY_UNVALIDATED = (_MAX_TURNS_REPLY_TESTED = ("I've reached the maximum number of steps, but I have a tested draft for you. Accept it to save, or discard.")+_VERIFIED_WORKFLOW_SUCCESS_REPLY = "I created and tested the workflow successfully."_UNEXPECTED_ERROR_REPLY_UNVALIDATED = ("I hit an unexpected issue before I could finish testing. I have a draft workflow you can keep — ""accept it to save (note: it hasn't been verified end-to-end), or discard."@@ -1514,9 +1515,7 @@ def _render_typed_run_outcome_reply(if not has_verified_workflow or not verified_goal_claim_authorized(ctx):return Nonereturn _TypedRunOutcomeReply(- user_response=(- "I created and tested the workflow successfully. The latest run demonstrated the requested outcome."- ),+ user_response=f"{_VERIFIED_WORKFLOW_SUCCESS_REPLY} The latest run demonstrated the requested outcome.",demonstrated=True,)@@ -1973,6 +1972,38 @@ def _build_wip_exit_result(terminal_reason=effective_terminal,)+ verified_workflow, verified_yaml = _verified_workflow_or_none(ctx)+ if ctx.verified_terminal_proposal_ready is True and verified_workflow is not None:+ final_text, outcome = _guard(tested_reply)+ return _finalize_result_with_blocker_override(+ ctx,+ _make_agent_result(+ ctx,+ user_response=final_text,+ updated_workflow=verified_workflow,+ global_llm_context=global_llm_context,+ workflow_yaml=verified_yaml,+ workflow_was_persisted=ctx.workflow_persisted,+ has_staged_proposal=ctx.has_staged_proposal,+ staged_workflow_yaml=ctx.staged_workflow_yaml,+ staged_workflow=ctx.staged_workflow,+ canonical_was_persisted_due_to_param_change=ctx.canonical_was_persisted_due_to_param_change,+ total_tokens=ctx.total_tokens_used,+ proposal_disposition="auto_applicable",+ cancelled=cancelled,+ turn_outcome=outcome,+ turn_id=ctx.turn_id,+ narrative_summary=ctx.narrative_summary,+ narrative_payload=_build_narrative_payload(+ ctx,+ terminal="response",+ terminal_message=final_text,+ narrative_summary=ctx.narrative_summary,+ ),+ ),+ exit_site="wip_verified_terminal_proposal",+ )+# When an unverified edit/run has overwritten ``last_workflow``, prefer the# verified shape while still forcing explicit review.if (@@ -2356,10 +2387,21 @@ async def _translate_to_agent_result(elif not salvaged_reply:user_response = _rewrite_failed_test_response(str(user_response), ctx)verified_workflow, verified_yaml = _verified_workflow_or_none(ctx)- # Default-true preserves backwards-compat with stale prompts and missing fields.- agent_admits_incomplete = _is_explicit_false(- action_data.get("goal_reached")- ) and not verified_goal_claim_authorized(ctx)+ verified_terminal_ready = (+ ctx.verified_terminal_proposal_ready is True+ and verified_workflow is not None+ and verified_goal_claim_authorized(ctx)+ and not blocker_active+ )+ if verified_terminal_ready:+ resp_type = "REPLY"+ user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY+ agent_admits_incomplete = False+ else:+ # Default-true preserves backwards-compat with stale prompts and missing fields.+ agent_admits_incomplete = _is_explicit_false(+ action_data.get("goal_reached")+ ) and not verified_goal_claim_authorized(ctx)typed_outcome_reply = _render_typed_run_outcome_reply(ctx,response_type=resp_type,diff --git a/skyvern/forge/sdk/copilot/completion_verification.py b/skyvern/forge/sdk/copilot/completion_verification.pyindex e64affb60..aeda47aca 100644--- a/skyvern/forge/sdk/copilot/completion_verification.py+++ b/skyvern/forge/sdk/copilot/completion_verification.py@@ -13,6 +13,7 @@ from __future__ import annotationsimport asyncioimport jsonimport re+from collections.abc import Iterablefrom dataclasses import dataclass, fieldfrom typing import Any, Literal@@ -529,6 +530,366 @@ def grade_present_value_criteria(return verdicts+def grade_record_semantic_consistency(+ criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot+) -> list[CriterionVerdict]:+ """Deterministically reject internally contradictory structured-record status outputs."""++ criterion = next(iter(_status_consistency_criterion(criteria)), None)+ if criterion is None:+ return []++ for label, payload in snapshot.block_outputs.items():+ contradiction = _structured_record_contradiction(payload)+ if contradiction:+ return [+ CriterionVerdict(+ criterion_id=criterion.id,+ state="unsatisfied",+ reason_code="evidence_contradicts",+ evidence_ref=f"block_outputs:{label}",+ missing_evidence=contradiction,+ )+ ]+ return []+++_STRUCTURED_RECORD_CRITERION_IDS = frozenset(+ {+ "fallback_record_identity",+ "fallback_record_identifier",+ "fallback_record_groups",+ "fallback_record_status",+ }+)+++def grade_structured_record_criteria(+ criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot+) -> list[CriterionVerdict]:+ """Deterministically credit generic structured-record fallback criteria.++ Single-block-wins: a criterion set is credited only when one block satisfies it.+ Verdicts are never merged across blocks because a structured record is a single+ coherent record, and crediting fields drawn from different blocks could certify a+ record the run never produced as a whole.+ """++ criteria_by_id = {+ criterion.id: criterion for criterion in criteria if criterion.id in _STRUCTURED_RECORD_CRITERION_IDS+ }+ if not criteria_by_id:+ return []+ best_verdicts: list[CriterionVerdict] = []+ for label, payload in snapshot.block_outputs.items():+ record = _structured_record_payload(payload)+ if record is None:+ continue+ verdicts: list[CriterionVerdict] = []+ if "fallback_record_identity" in criteria_by_id and structured_record_has_identity(record):+ verdicts.append(_structured_record_satisfied("fallback_record_identity", label))+ if "fallback_record_identifier" in criteria_by_id and _structured_record_has_identifier(record):+ verdicts.append(_structured_record_satisfied("fallback_record_identifier", label))+ if "fallback_record_groups" in criteria_by_id and _structured_record_has_group_entries(record):+ verdicts.append(_structured_record_satisfied("fallback_record_groups", label))+ if "fallback_record_status" in criteria_by_id:+ contradiction = _structured_record_contradiction(record)+ if contradiction:+ verdicts.append(+ CriterionVerdict(+ criterion_id="fallback_record_status",+ state="unsatisfied",+ reason_code="evidence_contradicts",+ evidence_ref=f"block_outputs:{label}",+ missing_evidence=contradiction,+ )+ )+ elif _structured_record_has_status(record):+ verdicts.append(_structured_record_satisfied("fallback_record_status", label))+ if verdicts:+ if len(verdicts) == len(criteria_by_id):+ return verdicts+ if len(verdicts) > len(best_verdicts):+ best_verdicts = verdicts+ return best_verdicts+++def _structured_record_satisfied(criterion_id: str, label: str) -> CriterionVerdict:+ return CriterionVerdict(+ criterion_id=criterion_id,+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref=f"block_outputs:{label}",+ )+++def _structured_record_payload(payload: Any) -> dict[str, Any] | None:+ if not isinstance(payload, dict):+ return None+ for key, value in payload.items():+ if (+ isinstance(key, str)+ and key.endswith("_output")+ and isinstance(value, dict)+ and _looks_like_structured_record(value)+ ):+ return value+ if _looks_like_structured_record(payload):+ return payload+ return None+++def _looks_like_structured_record(value: dict[str, Any]) -> bool:+ return (+ structured_record_has_identity(value)+ or _structured_record_has_identifier(value)+ or _structured_record_has_group_entries(value)+ or _structured_record_has_status(value)+ )+++def structured_record_has_identity(record: dict[str, Any]) -> bool:+ if any(value is True for key, value in record.items() if isinstance(key, str) and key.endswith("_found")):+ return True+ return any(+ isinstance(value, str) and bool(value.strip())+ for key, value in record.items()+ if isinstance(key, str) and _key_has_identity_token(key)+ )+++def _key_has_identity_token(key: str) -> bool:+ normalized_key = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", key).casefold().replace("_", " ")+ return any(+ re.search(rf"(?:^|[^a-z0-9]){re.escape(token)}(?:$|[^a-z0-9])", normalized_key)+ for token in ("name", "title", "entity", "label")+ )+++def structured_record_has_goal_content(record: dict[str, Any]) -> bool:+ """Return True only when a structured record has the full terminal-proof shape."""++ if (+ not structured_record_has_identity(record)+ or not _structured_record_has_identifier(record)+ or not _record_summary_status(record)+ ):+ return False+ return any(+ bool(status.strip())+ and any(+ _is_meaningful_record_value(item_value)+ for item_key, item_value in item.items()+ if not (isinstance(item_key, str) and "status" in item_key.casefold())+ )+ for status, item in _record_row_statuses(record)+ )+++def _key_word_tokens(key: str) -> set[str]:+ """Split a (possibly dotted/camelCase) record key path into whole-word tokens."""+ spaced = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", key).casefold()+ return set(re.findall(r"[a-z0-9]+", spaced))+++def _structured_record_has_identifier(record: dict[str, Any]) -> bool:+ for key, value in _walk_record_scalars(record):+ value_text = str(value)+ digit_runs = "".join(ch if ch.isdigit() else " " for ch in value_text).split()+ if any(len(run) >= 6 for run in digit_runs):+ return True+ if _key_word_tokens(key) & {"identifier", "id", "number"} and value_text.strip():+ return True+ return False+++def _structured_record_has_group_entries(record: dict[str, Any]) -> bool:+ for value in record.values():+ if not isinstance(value, list):+ continue+ for item in value:+ if isinstance(item, dict) and any(_is_meaningful_record_value(nested) for nested in item.values()):+ return True+ return False+++def _structured_record_has_status(record: dict[str, Any]) -> bool:+ summary_status = _record_summary_status(record)+ row_statuses = _record_row_statuses(record)+ return bool(summary_status and row_statuses)+++def _status_consistency_criterion(criteria: list[CompletionCriterion]) -> Iterable[CompletionCriterion]:+ for criterion in criteria:+ normalized = _normalize_present_value(criterion.outcome)+ if "status" in normalized and (+ "consistent" in normalized or "overall" in normalized or "summary" in normalized or "per-" in normalized+ ):+ yield criterion+++def _structured_record_contradiction(payload: Any) -> str | None:+ if not isinstance(payload, dict):+ return None+ if not (_structured_record_has_group_entries(payload) or _structured_record_has_status(payload)):+ return None+ summary_status = _record_summary_status(payload)+ summary_polarity = _status_polarity(summary_status) if summary_status else None+ row_statuses = _record_row_statuses(payload)+ for status, item in row_statuses:+ polarity = _status_polarity(status)+ if polarity is None:+ continue+ is_negative, positive_phrase = polarity+ if not is_negative:+ continue+ non_status_text = " ".join(+ str(value)+ for key, value in item.items()+ if not (isinstance(key, str) and "status" in key.casefold()) and value is not None+ )+ if _contains_positive_status_phrase(non_status_text, positive_phrase):+ return (+ "a parsed row reports a negative status, but non-status fields include the positive status text; "+ "parse name/address/status from cells in the same row"+ )+ if summary_polarity is not None:+ summary_negative, summary_positive_phrase = summary_polarity+ if summary_negative and any(+ (row_polarity := _status_polarity(status)) is not None and row_polarity == (False, summary_positive_phrase)+ for status, _item in row_statuses+ ):+ return "summary status is negative, but a parsed row has the matching positive status"++ evidence_text = payload.get("evidence_text")+ if isinstance(evidence_text, str) and summary_polarity is not None:+ summary_negative, summary_positive_phrase = summary_polarity+ if (+ summary_negative+ and row_statuses+ and not any(_status_polarity(status) == (False, summary_positive_phrase) for status, _item in row_statuses)+ and _contains_positive_status_phrase(evidence_text, summary_positive_phrase)+ ):+ return (+ "evidence_text contains positive status rows, but structured rows and summary report a negative status"+ )+ return None+++def _record_summary_status(record: dict[str, Any]) -> str | None:+ for key, value in record.items():+ if not isinstance(key, str) or not isinstance(value, str):+ continue+ normalized_key = key.casefold()+ if "status" in normalized_key and ("overall" in normalized_key or "summary" in normalized_key):+ return value+ value = record.get("status")+ return value if isinstance(value, str) else None+++def _record_row_statuses(record: dict[str, Any]) -> list[tuple[str, dict[str, Any]]]:+ statuses: list[tuple[str, dict[str, Any]]] = []+ for value in record.values():+ if not isinstance(value, list):+ continue+ for item in value:+ if not isinstance(item, dict):+ continue+ for item_key, item_value in item.items():+ if isinstance(item_key, str) and "status" in item_key.casefold() and isinstance(item_value, str):+ statuses.append((item_value, item))+ return statuses+++_NEGATIVE_STATUS_POSITIVE_COUNTERPARTS = {+ "expired": "active",+ "inactive": "active",+ "lapsed": "active",+ "pending": "active",+ "revoked": "active",+ "suspended": "active",+ "terminated": "active",+}+++def _status_polarity(value: str) -> tuple[bool, str] | None:+ normalized = _normalize_present_value(value)+ if not normalized:+ return None+ if normalized in _NEGATIVE_STATUS_POSITIVE_COUNTERPARTS:+ return True, _NEGATIVE_STATUS_POSITIVE_COUNTERPARTS[normalized]+ for prefix in ("not ", "non-", "non "):+ if normalized.startswith(prefix):+ positive_phrase = normalized[len(prefix) :].strip()+ return (True, positive_phrase) if positive_phrase else None+ return False, normalized+++# Negators, temporal qualifiers, and negative-status words that, near a positive-status+# word, flip it into a negative or adjectival usage ("no longer active", "previously+# active", "the active license expired") rather than a positive status assertion.+_STATUS_NEGATION_QUALIFIERS = frozenset(+ {+ "not",+ "non",+ "no",+ "longer",+ "previously",+ "formerly",+ "once",+ "currently",+ "recently",+ "was",+ "were",+ "until",+ "never",+ *_NEGATIVE_STATUS_POSITIVE_COUNTERPARTS,+ }+)+++def _status_word_tokens(text: str) -> list[str]:+ return "".join(char if char.isalnum() else " " for char in text.casefold()).split()+++def _contains_positive_status_phrase(text: str, positive_phrase: str) -> bool:+ tokens = _status_word_tokens(text)+ phrase_tokens = _status_word_tokens(positive_phrase)+ if not tokens or not phrase_tokens:+ return False+ span = len(phrase_tokens)+ for start in range(len(tokens) - span + 1):+ if tokens[start : start + span] != phrase_tokens:+ continue+ window = set(tokens[max(0, start - 3) : start]) | set(tokens[start + span : start + span + 3])+ if not window & _STATUS_NEGATION_QUALIFIERS:+ return True+ return False+++def _walk_record_scalars(value: Any, prefix: str = "") -> Iterable[tuple[str, Any]]:+ if isinstance(value, dict):+ for key, nested in value.items():+ key_text = str(key)+ nested_prefix = f"{prefix}.{key_text}" if prefix else key_text+ yield from _walk_record_scalars(nested, nested_prefix)+ elif isinstance(value, list):+ for index, nested in enumerate(value):+ yield from _walk_record_scalars(nested, f"{prefix}[{index}]")+ elif value is not None:+ yield prefix, value+++def _is_meaningful_record_value(value: Any) -> bool:+ if value is None:+ return False+ if isinstance(value, str):+ return bool(value.strip())+ if isinstance(value, (list, tuple, set, dict)):+ return bool(value)+ return True++_DEFINITION_REASON_PREFIX = "definition_"diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.pyindex 8a37311c8..de9a91266 100644--- a/skyvern/forge/sdk/copilot/enforcement.py+++ b/skyvern/forge/sdk/copilot/enforcement.py@@ -1270,6 +1270,9 @@ def _check_enforcement(raise_if_turn_halt(ctx)_raise_if_unrecoverable_contract_stop(ctx)+ if getattr(ctx, "verified_terminal_proposal_ready", False) is True and verified_goal_satisfied_context(ctx):+ raise CopilotGoalSatisfied()+if _needs_repair_ceiling_halt(ctx):contract = getattr(ctx, "latest_diagnosis_repair_contract", None)state = _repair_loop_state(ctx)diff --git a/skyvern/forge/sdk/copilot/output_policy.py b/skyvern/forge/sdk/copilot/output_policy.pyindex c10fb6120..7b9c225d2 100644--- a/skyvern/forge/sdk/copilot/output_policy.py+++ b/skyvern/forge/sdk/copilot/output_policy.py@@ -149,6 +149,15 @@ _SELF_PRESCRIPTIVE_CONTINUATION_RE = re.compile(r")\b",re.IGNORECASE,)+_OUTPUT_FIELD_CONFIRMATION_RE = re.compile(+ r"(?:"+ r"\b(?:confirm|verify|approve|check|tell me|let me know)\b"+ r"(?=[\s\S]{0,180}\b(?:output|record|schema|field|fields)\b)"+ r"(?=[\s\S]{0,220}\b(?:field|fields|schema|record)\b)"+ r"|\b(?:which|what)\s+fields\s+should\b"+ r")",+ re.IGNORECASE,+)# Response types whose `user_response` is rendered verbatim as the agent's final# message — REPLY and REPLACE_WORKFLOW. ASK_QUESTION is excluded because legitimate@@ -187,6 +196,7 @@ class OutputPolicyReason(StrEnum):INTERNAL_CLASSIFIER_VOCAB_LEAK = "internal_classifier_vocab_leak"SELF_PRESCRIPTIVE_PHRASE_LEAK = "self_prescriptive_phrase_leak"WORKFLOW_YAML_IN_REPLY = "workflow_yaml_in_reply"+ AVOIDABLE_OUTPUT_FIELD_CONFIRMATION = "avoidable_output_field_confirmation"@dataclass@@ -219,6 +229,7 @@ _FINAL_OUTPUT_HARD_BLOCK_REASONS: frozenset[OutputPolicyReason] = frozenset(OutputPolicyReason.PERSISTENCE_STATE_MISMATCH,OutputPolicyReason.INTERNAL_TOOL_INSTRUCTION_LEAK,OutputPolicyReason.OUTPUT_POLICY_CONTEXT_MISSING,+ OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION,})@@ -422,6 +433,15 @@ def evaluate_output_policy(if isinstance(request_policy, RequestPolicy):if request_policy.user_response_policy == "ask_clarification" and response_type != "ASK_QUESTION":verdict.add(OutputPolicyReason.REQUEST_POLICY_CLARIFICATION_BYPASS)+ if (+ response_type == "ASK_QUESTION"+ and request_policy.user_response_policy != "ask_clarification"+ and _request_policy_has_present_completion_contract(request_policy)+ and not has_workflow_proposal+ and not workflow_attempted+ and _asks_to_confirm_output_fields(user_response)+ ):+ verdict.add(OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION)_apply_credential_policy(verdict, request_policy, values, workflow_yaml)if output_kind == CopilotOutputKind.WORKFLOW_UPDATE_PROPOSAL and not workflow_was_persisted:@@ -432,6 +452,16 @@ def evaluate_output_policy(return verdict+def _request_policy_has_present_completion_contract(request_policy: RequestPolicy) -> bool:+ return request_policy.completion_contract_status == "present" or bool(request_policy.completion_criteria)+++def _asks_to_confirm_output_fields(user_response: str | None) -> bool:+ if not isinstance(user_response, str) or not user_response.strip():+ return False+ return bool(_OUTPUT_FIELD_CONFIRMATION_RE.search(user_response[:500]))++def format_output_policy_tool_error(verdict: OutputPolicyVerdict) -> str:reasons = ", ".join(reason.value for reason in verdict.reason_codes) or "unknown"message = f"Output policy blocked this Copilot output before persistence. Reason codes: {reasons}."diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.pyindex 5acd8a8f9..67ee13660 100644--- a/skyvern/forge/sdk/copilot/request_policy.py+++ b/skyvern/forge/sdk/copilot/request_policy.py@@ -567,6 +567,7 @@ def _classifier_fallback_policy(raw_secret_present: bool,failure_kind: str,retry_count: int = 0,+ user_message: str = "",) -> RequestPolicy:if failure_kind not in _CLASSIFIER_FAILURE_KINDS:failure_kind = "provider_error"@@ -582,16 +583,81 @@ def _classifier_fallback_policy(classifier_retry_count=retry_count,completion_contract_status="unknown",)+ fallback_criteria = _fallback_structured_record_completion_criteria(user_message)+ if fallback_criteria:+ LOG.info(+ "copilot request policy synthesized fallback structured-record criteria",+ classifier_failure_kind=failure_kind,+ completion_criterion_ids=[criterion.id for criterion in fallback_criteria],+ )return RequestPolicy(credential_input_kind="credential_id" if ids else "none",credential_refs=ids,+ completion_criteria=fallback_criteria,classifier_status="fallback",classifier_failure_kind=failure_kind,classifier_retry_count=retry_count,- completion_contract_status="unknown",+ completion_contract_status="present" if fallback_criteria else "unknown",)+def _word_tokens(text: str) -> list[str]:+ return "".join(char if char.isalnum() else " " for char in text.casefold()).split()+++def _fallback_structured_record_completion_criteria(user_message: str) -> list[CompletionCriterion]:+ """Conservative structured-record criteria when the request-policy classifier is unavailable."""++ words = _word_tokens(user_message)+ if not words:+ return []+ haystack = f" {' '.join(words)} "++ def _mentions(*terms: str) -> bool:+ # Whole-word/phrase match so "id" can't fire on "consider" nor "name" on "filename".+ return any(f" {' '.join(_word_tokens(term))} " in haystack for term in terms)++ has_long_number = any(word.isdigit() and len(word) >= 6 for word in words)+ if not _mentions("return", "record", "capture", "read", "extract"):+ return []+ if not _mentions("name", "entity", "person"):+ return []+ if not _mentions("identifier", "id", "number") and not has_long_number:+ return []+ if not _mentions("status"):+ return []+ if not _mentions("locations", "items", "rows", "groups", "entries", "per location", "per item", "per row"):+ return []+ return [+ CompletionCriterion(+ id="fallback_record_identity",+ outcome="The returned record identifies the target entity.",+ implicit=True,+ level="run",+ ),+ CompletionCriterion(+ id="fallback_record_identifier",+ outcome="The returned record includes the requested identifier-like value.",+ implicit=True,+ level="run",+ ),+ CompletionCriterion(+ id="fallback_record_groups",+ outcome="The returned record includes the requested grouped row entries.",+ implicit=True,+ level="run",+ ),+ CompletionCriterion(+ id="fallback_record_status",+ outcome=(+ "The returned record's per-row statuses and summary status are present and internally consistent."+ ),+ implicit=True,+ level="run",+ ),+ ]++def _explicit_code_block_credential_draft_requested(user_message: str) -> bool:normalized = " ".join((user_message or "").lower().split())if not normalized:@@ -656,7 +722,12 @@ async def _classify_request(ids = _credential_ids(user_message)raw_secret_present = _raw_secret_detected(user_message)if raw_secret_present and handler is None:- return _classifier_fallback_policy(ids, raw_secret_present=True, failure_kind="raw_secret_no_handler")+ return _classifier_fallback_policy(+ ids,+ raw_secret_present=True,+ failure_kind="raw_secret_no_handler",+ user_message=user_message,+ )structural_reason = _structural_clarification_reason(user_message)if structural_reason != "none" and not raw_secret_present:return RequestPolicy(@@ -666,7 +737,12 @@ async def _classify_request(clarification_reason=structural_reason,)if handler is None:- return _classifier_fallback_policy(ids, raw_secret_present=False, failure_kind="missing_handler")+ return _classifier_fallback_policy(+ ids,+ raw_secret_present=False,+ failure_kind="missing_handler",+ user_message=user_message,+ )# Raw-secret turns intentionally reach the classifier when available so it# can distinguish unsafe secret use from redacted draft/spec conversion.@@ -693,6 +769,7 @@ async def _classify_request(raw_secret_present=raw_secret_present,failure_kind=failure_kind,retry_count=retry_count,+ user_message=user_message,)raw_payload = _coerce_classifier_payload(raw)@@ -703,6 +780,7 @@ async def _classify_request(raw_secret_present=raw_secret_present,failure_kind="provider_error",retry_count=retry_count,+ user_message=user_message,)policy = _classification_from_raw(raw_payload)diff --git a/skyvern/forge/sdk/copilot/runtime.py b/skyvern/forge/sdk/copilot/runtime.pyindex c4c6ef691..f2c121b31 100644--- a/skyvern/forge/sdk/copilot/runtime.py+++ b/skyvern/forge/sdk/copilot/runtime.py@@ -234,6 +234,7 @@ class AgentContext:last_run_outcome: RecordedRunOutcome | None = Nonelast_run_outcome_block_labels: list[str] = field(default_factory=list)completion_verification_result: CompletionVerificationResult | None = None+ verified_terminal_proposal_ready: bool = Falseoutcome_verification_trace_snapshot: dict[str, Any] = field(default_factory=dict)composition_page_evidence: dict[str, Any] | None = None# Ordered, bounded list of typed page-evidence packets — one per page observeddiff --git a/skyvern/forge/sdk/copilot/tools/_shared.py b/skyvern/forge/sdk/copilot/tools/_shared.pyindex e106bade4..a25f6baa7 100644--- a/skyvern/forge/sdk/copilot/tools/_shared.py+++ b/skyvern/forge/sdk/copilot/tools/_shared.py@@ -182,6 +182,41 @@ _TASK_OUTPUT_PAYLOAD_FIELDS: tuple[str, ...] = ("downloaded_files","downloaded_file_urls",)+_TASK_OUTPUT_PARAMETER_SUFFIX = "_output"+++def _workflow_output_parameter_payloads(extracted_data: Any) -> dict[str, Any]:+ """Return workflow output-parameter values embedded in a block output."""+ if not isinstance(extracted_data, dict):+ return {}+ return {+ key: value+ for key, value in extracted_data.items()+ if isinstance(key, str) and key.endswith(_TASK_OUTPUT_PARAMETER_SUFFIX)+ }+++def _registered_output_parameter_payloads(data: Mapping[str, Any]) -> list[Mapping[str, Any]]:+ """``workflow_run_output_parameters`` is the runtime's authoritative persisted+ value surface. Keep this scoped to the run result carrying it so prior-run+ accumulated outputs cannot satisfy the current run's completion contract.+ """+ run_id = data.get("workflow_run_id")+ registered = data.get("registered_output_parameter_values")+ if not isinstance(registered, list):+ return []+ if not isinstance(run_id, str):+ # Without a current run id the payloads can't be attributed to this run;+ # fail closed so prior-run accumulated outputs can't satisfy its contract.+ return []+ payloads: list[Mapping[str, Any]] = []+ for item in registered:+ if not isinstance(item, Mapping):+ continue+ if item.get("workflow_run_id") != run_id:+ continue+ payloads.append(item)+ return payloadsdef _block_data_payload(extracted_data: Any, block_type: str | None) -> Any:@@ -196,7 +231,9 @@ def _block_data_payload(extracted_data: Any, block_type: str | None) -> Any:json_schema that happens to include an ``extracted_information`` field."""if block_type in _TASK_ENVELOPE_BLOCK_TYPES and isinstance(extracted_data, dict):- return {field: extracted_data.get(field) for field in _TASK_OUTPUT_PAYLOAD_FIELDS}+ payload = {field: extracted_data.get(field) for field in _TASK_OUTPUT_PAYLOAD_FIELDS}+ payload.update(_workflow_output_parameter_payloads(extracted_data))+ return payloadreturn extracted_datadiff --git a/skyvern/forge/sdk/copilot/tools/blockers.py b/skyvern/forge/sdk/copilot/tools/blockers.pyindex 2e232e556..61859d8ee 100644--- a/skyvern/forge/sdk/copilot/tools/blockers.py+++ b/skyvern/forge/sdk/copilot/tools/blockers.py@@ -17,6 +17,9 @@ from skyvern.forge.sdk.copilot.blocker_signal import (loop_blocker_evidence_from_ctx,refresh_held_loop_blocker_evidence,)+from skyvern.forge.sdk.copilot.completion_verification import (+ structured_record_has_goal_content as _structured_record_candidate_has_goal_content,+)from skyvern.forge.sdk.copilot.composition_evidence import interactive_challenge_controlsfrom skyvern.forge.sdk.copilot.enforcement import (TOTAL_TIMEOUT_SECONDS,@@ -52,6 +55,8 @@ from ._shared import (_is_meaningful_extracted_data,_parse_workflow_blocks,_raw_yaml_proxy_location,+ _registered_output_parameter_payloads,+ _workflow_output_parameter_payloads,)LOG = structlog.get_logger()@@ -1641,6 +1646,7 @@ def _analyze_run_blocks(has_data_blocks = Falseany_data_output = Falsemissing_metadata_goal_content = False+ complete_structured_record_output = Falseblocks = data.get("blocks")if isinstance(blocks, list):@@ -1675,6 +1681,13 @@ def _analyze_run_blocks()if structured_blocker:texts_to_scan.append(structured_blocker)+ output_parameter_payloads = _workflow_output_parameter_payloads(extracted)+ if output_parameter_payloads:+ has_data_blocks = True+ if any(_is_meaningful_extracted_data(value) for value in output_parameter_payloads.values()):+ any_data_output = True+ if any(_structured_record_has_goal_content(value) for value in output_parameter_payloads.values()):+ complete_structured_record_output = Trueextraction_schema = _extraction_schema_for_code_block(copilot_ctx, block.get("label"))# Array-typed schemas would coerce the keyed dict return to [] (fill_missing_fields), making the# goal-path check below read a real extraction as empty; the keyed-return floor guarantees a dict.@@ -1687,8 +1700,10 @@ def _analyze_run_blocks(goal_value_paths = _goal_value_paths_for_code_block(copilot_ctx, block.get("label"))if goal_value_paths:has_data_blocks = True- if _code_output_has_registered_download_content(extracted) or _code_output_goal_paths_have_content(- extracted, goal_value_paths+ if (+ _code_output_has_registered_download_content(extracted)+ or _code_output_goal_paths_have_content(extracted, goal_value_paths)+ or _structured_record_has_goal_content(extracted)):any_data_output = Trueelse:@@ -1706,6 +1721,32 @@ def _analyze_run_blocks(if _code_output_has_goal_content(extracted, declared_keys=declared_keys):any_data_output = True+ top_level_output_payloads = _workflow_output_parameter_payloads(data.get("output"))+ if top_level_output_payloads:+ has_data_blocks = True+ if any(_is_meaningful_extracted_data(value) for value in top_level_output_payloads.values()):+ any_data_output = True+ if any(_structured_record_has_goal_content(value) for value in top_level_output_payloads.values()):+ complete_structured_record_output = True++ registered_payloads = _registered_output_parameter_payloads(data)+ if registered_payloads:+ has_data_blocks = True+ for registered in registered_payloads:+ value = registered.get("value")+ if _is_meaningful_extracted_data(value):+ any_data_output = True+ if _structured_record_has_goal_content(value):+ complete_structured_record_output = True+ structured_blocker = _structured_blocker_message(+ value,+ include_flag_keys=True,+ key_terms=_STRICT_BLOCKER_FLAG_TERMS,+ scan_all_values_for_anti_bot=True,+ )+ if structured_blocker:+ texts_to_scan.append(structured_blocker)+combined = "\n".join(texts_to_scan)categories = classify_from_failure_reason(combined)if categories:@@ -1714,5 +1755,25 @@ def _analyze_run_blocks(anti_bot_match = cat.get("reasoning", "anti-bot pattern detected")break+ if complete_structured_record_output:+ if missing_metadata_goal_content:+ LOG.info(+ "copilot run evidence: a complete structured-record output suppressed a "+ "per-block missing-metadata-goal-content signal",+ workflow_run_id=data.get("workflow_run_id"),+ )+ missing_metadata_goal_content = Falseempty_data_blocks = (has_data_blocks and not any_data_output) or missing_metadata_goal_contentreturn anti_bot_match, empty_data_blocks, categories+++def _structured_record_has_goal_content(value: Any) -> bool:+ if not isinstance(value, dict):+ return False+ candidates = [value]+ candidates.extend(+ nested+ for key, nested in value.items()+ if isinstance(key, str) and key.endswith("_output") and isinstance(nested, dict)+ )+ return any(_structured_record_candidate_has_goal_content(candidate) for candidate in candidates)diff --git a/skyvern/forge/sdk/copilot/tools/completion.py b/skyvern/forge/sdk/copilot/tools/completion.pyindex 1f3e24e85..d44016db1 100644--- a/skyvern/forge/sdk/copilot/tools/completion.py+++ b/skyvern/forge/sdk/copilot/tools/completion.py@@ -13,6 +13,8 @@ from skyvern.forge.sdk.copilot.completion_verification import (evaluate_completion_criteria,grade_definition_criteria,grade_present_value_criteria,+ grade_record_semantic_consistency,+ grade_structured_record_criteria,summarize_unsatisfied_outcomes,verdict_missing_evidence,)@@ -31,7 +33,9 @@ from ._shared import (_current_workflow_has_evidence_block,_failed_run_block_labels,_is_meaningful_extracted_data,+ _registered_output_parameter_payloads,_valid_runtime_anchor_url,+ _workflow_output_parameter_payloads,)from .blockers import (_active_run_terminal_evidence_detected,@@ -373,6 +377,29 @@ def _build_run_evidence_snapshot(copilot_ctx: Any, result: dict[str, Any]) -> Ruevidence_output = _completion_evidence_payload(output)if isinstance(label, str) and label in current_labels and _is_meaningful_extracted_data(evidence_output):block_outputs[label] = evidence_output+ for output_key, output_value in _workflow_output_parameter_payloads(output).items():+ if _is_meaningful_extracted_data(output_value):+ block_outputs[output_key] = output_value+ for output_key, output_value in _workflow_output_parameter_payloads(data.get("output")).items():+ if _is_meaningful_extracted_data(output_value):+ block_outputs[output_key] = output_value+ for registered in _registered_output_parameter_payloads(data):+ registered_output_key = registered.get("output_parameter_key")+ registered_output_value = registered.get("value")+ registered_block_label = registered.get("block_label")+ if not _is_meaningful_extracted_data(registered_output_value):+ continue+ if isinstance(registered_output_key, str) and registered_output_key:+ block_outputs[registered_output_key] = registered_output_value+ if isinstance(registered_block_label, str) and registered_block_label in current_labels:+ if isinstance(registered_output_key, str) and registered_output_key:+ existing = block_outputs.get(registered_block_label)+ if isinstance(existing, dict):+ existing.setdefault(registered_output_key, registered_output_value)+ else:+ block_outputs[registered_block_label] = {registered_output_key: registered_output_value}+ else:+ block_outputs[registered_block_label] = registered_output_valueexecuted = data.get("executed_block_labels")executed_block_labels = [str(label) for label in executed] if isinstance(executed, list) else []page_title = data.get("page_title")@@ -416,19 +443,85 @@ def _apply_present_value_upgrades(if run_result.status != "evaluated":return run_resultupgrades = {verdict.criterion_id: verdict for verdict in grade_present_value_criteria(run_criteria, snapshot)}- if not upgrades:+ upgrades.update(+ {verdict.criterion_id: verdict for verdict in grade_structured_record_criteria(run_criteria, snapshot)}+ )+ semantic_verdicts = {+ verdict.criterion_id: verdict for verdict in grade_record_semantic_consistency(run_criteria, snapshot)+ }+ if not upgrades and not semantic_verdicts:return run_result- verdicts = [- upgrades[verdict.criterion_id]- if verdict.criterion_id in upgrades and not verdict.satisfied and verdict.reason_code != "evidence_contradicts"- else verdict- for verdict in run_result.verdicts- ]++ def _merged(verdict: CriterionVerdict) -> CriterionVerdict:+ # Semantic contradictions are stronger than earlier satisfied verdicts:+ # a row-mixing/status contradiction means the record evidence is invalid.+ if verdict.criterion_id in semantic_verdicts:+ return semantic_verdicts[verdict.criterion_id]+ if verdict.criterion_id in upgrades and not verdict.satisfied and verdict.reason_code != "evidence_contradicts":+ return upgrades[verdict.criterion_id]+ return verdict++ verdicts = [_merged(verdict) for verdict in run_result.verdicts]return CompletionVerificationResult(status="evaluated", criterion_ids=list(run_result.criterion_ids), verdicts=verdicts)+def _deterministic_run_verification_result(+ run_criteria: list[CompletionCriterion],+ snapshot: RunEvidenceSnapshot,+) -> tuple[CompletionVerificationResult | None, list[CompletionCriterion]]:+ """Return a deterministic run-plane verdict when the typed graders cover it.++ Provider-record and present-value graders are exact enough to bypass the LLM+ judge only when they satisfy every run-plane criterion. Contradictions also+ bypass the judge, because the output is already semantically invalid. Any+ remaining criterion stays fail-closed through the normal judge path.+ """+ criterion_ids = [criterion.id for criterion in run_criteria]+ semantic_verdicts = grade_record_semantic_consistency(run_criteria, snapshot)+ if any(not verdict.satisfied for verdict in semantic_verdicts):+ return (+ CompletionVerificationResult(+ status="evaluated",+ criterion_ids=criterion_ids,+ verdicts=list(semantic_verdicts),+ ),+ [],+ )++ deterministic_by_id: dict[str, CriterionVerdict] = {}+ for verdict in grade_present_value_criteria(run_criteria, snapshot):+ deterministic_by_id[verdict.criterion_id] = verdict+ for verdict in grade_structured_record_criteria(run_criteria, snapshot):+ deterministic_by_id[verdict.criterion_id] = verdict+ for verdict in semantic_verdicts:+ deterministic_by_id[verdict.criterion_id] = verdict++ deterministic_verdicts = [deterministic_by_id.get(criterion_id) for criterion_id in criterion_ids]+ if criterion_ids and all(verdict is not None and verdict.satisfied for verdict in deterministic_verdicts):+ return (+ CompletionVerificationResult(+ status="evaluated",+ criterion_ids=criterion_ids,+ verdicts=[deterministic_by_id[criterion_id] for criterion_id in criterion_ids],+ ),+ [],+ )++ remaining_criteria = [criterion for criterion in run_criteria if criterion.id not in deterministic_by_id]+ if not deterministic_by_id:+ return None, remaining_criteria+ return (+ CompletionVerificationResult(+ status="evaluated",+ criterion_ids=criterion_ids,+ verdicts=list(deterministic_by_id.values()),+ ),+ remaining_criteria,+ )++async def _maybe_run_completion_verification(copilot_ctx: Any, result: dict[str, Any], handler_start: float) -> CompletionVerificationResult | None:@@ -451,16 +544,6 @@ async def _maybe_run_completion_verification()if not run_criteria:return combine_verification_results(criterion_ids, None, definition_verdicts)- # A missing judge handler is an infra/config state, not a transient failure:- # fall back to the prior gate rather than fail closed on every run.- handler = await _completion_verification_handler(copilot_ctx)- if handler is None:- return None- # Too little budget to verify a candidate run: fail closed (unavailable) rather- # than let the run-status proxy claim an unverified outcome as success.- remaining = RUN_BLOCKS_SAFETY_CEILING_SECONDS - (time.monotonic() - handler_start)- if remaining <= settings.COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS + _COMPLETION_VERIFICATION_BUDGET_MARGIN_SECONDS:- return CompletionVerificationResult(status="unavailable")snapshot = _build_run_evidence_snapshot(copilot_ctx, result)if not snapshot.has_evidence():run_result = CompletionVerificationResult(@@ -472,8 +555,35 @@ async def _maybe_run_completion_verification(],)else:- run_result = await evaluate_completion_criteria(run_criteria, snapshot, handler)- run_result = _apply_present_value_upgrades(run_result, run_criteria, snapshot)+ handler = await _completion_verification_handler(copilot_ctx)+ if handler is None:+ return CompletionVerificationResult(status="unavailable")+ # Too little budget to verify a candidate run: fail closed (unavailable)+ # rather than let the run-status proxy claim an unverified outcome as success.+ remaining = RUN_BLOCKS_SAFETY_CEILING_SECONDS - (time.monotonic() - handler_start)+ if (+ remaining+ <= settings.COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS + _COMPLETION_VERIFICATION_BUDGET_MARGIN_SECONDS+ ):+ return CompletionVerificationResult(status="unavailable")+ deterministic_result, remaining_criteria = _deterministic_run_verification_result(run_criteria, snapshot)+ if deterministic_result is not None and not remaining_criteria:+ run_result = deterministic_result+ else:+ judged_result = await evaluate_completion_criteria(remaining_criteria, snapshot, handler)+ if judged_result.status != "evaluated":+ run_result = judged_result+ else:+ verdicts = []+ if deterministic_result is not None:+ verdicts.extend(deterministic_result.verdicts)+ verdicts.extend(judged_result.verdicts)+ run_result = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=[criterion.id for criterion in run_criteria],+ verdicts=verdicts,+ )+ run_result = _apply_present_value_upgrades(run_result, run_criteria, snapshot)return combine_verification_results(criterion_ids, run_result, definition_verdicts)diff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.pyindex 3124c89bf..7460b05c9 100644--- a/skyvern/forge/sdk/copilot/tools/run_execution.py+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py@@ -820,6 +820,159 @@ def _typed_child_blocks(block: Any) -> list[Any]:return children+def _workflow_output_parameter_indexes(+ workflow: Workflow | None,+) -> tuple[dict[str, dict[str, Any]], dict[str, dict[str, Any]]]:+ if workflow is None:+ return {}, {}+ workflow_definition = getattr(workflow, "workflow_definition", None)+ blocks = getattr(workflow_definition, "blocks", None)+ if not isinstance(blocks, list):+ return {}, {}++ by_id: dict[str, dict[str, Any]] = {}+ by_key: dict[str, dict[str, Any]] = {}++ def visit(block: Any) -> None:+ output_parameter = (+ block.get("output_parameter") if isinstance(block, Mapping) else getattr(block, "output_parameter", None)+ )+ output_parameter_id = (+ output_parameter.get("output_parameter_id")+ if isinstance(output_parameter, Mapping)+ else getattr(output_parameter, "output_parameter_id", None)+ )+ output_parameter_key = (+ output_parameter.get("key")+ if isinstance(output_parameter, Mapping)+ else getattr(output_parameter, "key", None)+ )+ label = block.get("label") if isinstance(block, Mapping) else getattr(block, "label", None)+ block_type = block.get("block_type") if isinstance(block, Mapping) else getattr(block, "block_type", None)+ block_type_name = getattr(block_type, "value", getattr(block_type, "name", block_type))+ entry = {+ "block_label": label if isinstance(label, str) and label else None,+ "block_type": str(block_type_name) if block_type_name is not None else None,+ "output_parameter_id": output_parameter_id if isinstance(output_parameter_id, str) else None,+ "output_parameter_key": output_parameter_key if isinstance(output_parameter_key, str) else None,+ }+ if entry["output_parameter_id"]:+ by_id[entry["output_parameter_id"]] = entry+ if entry["output_parameter_key"]:+ by_key[entry["output_parameter_key"]] = entry+ for child in _mapping_child_blocks(block) if isinstance(block, Mapping) else _typed_child_blocks(block):+ visit(child)++ for block in blocks:+ visit(block)+ return by_id, by_key+++def _merge_registered_output_parameter_values_into_blocks(data: dict[str, Any]) -> None:+ """Mutate ``data["blocks"]`` so registered output parameters share the block-output evidence path."""++ registered = data.get("registered_output_parameter_values")+ if not isinstance(registered, list) or not registered:+ return+ blocks = data.get("blocks")+ if not isinstance(blocks, list):+ blocks = []+ data["blocks"] = blocks+ by_label: dict[str, dict[str, Any]] = {+ block["label"]: block+ for block in blocks+ if isinstance(block, dict) and isinstance(block.get("label"), str) and block.get("label")+ }+ for item in registered:+ if not isinstance(item, Mapping):+ continue+ label = item.get("block_label")+ key = item.get("output_parameter_key")+ if not isinstance(label, str) or not label or not isinstance(key, str) or not key:+ continue+ value = item.get("value")+ block = by_label.get(label)+ if block is None:+ block = {+ "label": label,+ "block_type": item.get("block_type") or "CODE",+ "status": WorkflowRunStatus.completed.value,+ }+ blocks.append(block)+ by_label[label] = block+ extracted = block.get("extracted_data")+ if isinstance(extracted, dict):+ extracted.setdefault(key, value)+ elif extracted is None:+ block["extracted_data"] = {key: value}+++async def _attach_registered_output_parameter_values(+ *,+ workflow_run_id: str,+ workflow: Workflow | None,+ data: dict[str, Any],+ persisted_output_parameters: list[Any] | None = None,+) -> dict[str, Any]:+ try:+ registered_rows = await app.DATABASE.workflow_runs.get_workflow_run_output_parameters(+ workflow_run_id=workflow_run_id+ )+ except Exception:+ LOG.warning(+ "Failed to read workflow run output parameters for copilot run evidence; "+ "deterministic graders lose authoritative output-parameter evidence",+ workflow_run_id=workflow_run_id,+ organization_id=workflow.organization_id if workflow is not None else None,+ exc_info=True,+ )+ return {}+ if not registered_rows:+ return {}++ index_by_id, index_by_key = _workflow_output_parameter_indexes(workflow)+ persisted_key_by_id = {+ output_parameter_id: key+ for parameter in persisted_output_parameters or []+ if isinstance((output_parameter_id := getattr(parameter, "output_parameter_id", None)), str)+ and isinstance((key := getattr(parameter, "key", None)), str)+ }+ normalized: list[dict[str, Any]] = []+ values_by_label: dict[str, Any] = {}+ for row in registered_rows:+ output_parameter_id = getattr(row, "output_parameter_id", None)+ if not isinstance(output_parameter_id, str) or not output_parameter_id:+ continue+ block_info = dict(index_by_id.get(output_parameter_id, {}))+ output_parameter_key = block_info.get("output_parameter_key")+ if not isinstance(output_parameter_key, str) or not output_parameter_key:+ output_parameter_key = persisted_key_by_id.get(output_parameter_id)+ if isinstance(output_parameter_key, str):+ block_info["output_parameter_key"] = output_parameter_key+ if output_parameter_key and not block_info.get("block_label"):+ block_info.update(index_by_key.get(output_parameter_key, {}))+ value = getattr(row, "value", None)+ item = {+ "workflow_run_id": workflow_run_id,+ "output_parameter_id": output_parameter_id,+ "output_parameter_key": block_info.get("output_parameter_key"),+ "block_label": block_info.get("block_label"),+ "block_type": block_info.get("block_type"),+ "value": value,+ }+ normalized.append(item)+ label = item.get("block_label")+ key = item.get("output_parameter_key")+ if isinstance(label, str) and label and isinstance(key, str) and key:+ values_by_label.setdefault(label, {})[key] = value++ if not normalized:+ return {}+ data["registered_output_parameter_values"] = normalized+ _merge_registered_output_parameter_values_into_blocks(data)+ return values_by_label++async def _run_blocks_and_collect_debug(params: dict[str, Any],ctx: CopilotContext,@@ -1412,6 +1565,16 @@ async def _run_blocks_and_collect_debug(if not run_ok and run and getattr(run, "failure_reason", None):result_data["failure_reason"] = run.failure_reason+ registered_outputs_by_label = await _attach_registered_output_parameter_values(+ workflow_run_id=workflow_run.workflow_run_id,+ workflow=runtime_workflow,+ data=result_data,+ persisted_output_parameters=all_output_params,+ )+ for label, output in registered_outputs_by_label.items():+ if isinstance(output, dict) and output:+ block_outputs_by_label[label] = output+# Update verified prefix state ONLY on a fully-successful run. A failed# suffix run leaves the browser in post-failure state, so we must not# trust blocks that individually succeeded inside it.@@ -1790,6 +1953,9 @@ def _record_run_blocks_result(timeout_latched = bool(copilot_ctx.copilot_total_timeout_exceeded)copilot_ctx.last_test_ok = None if (cancelled_by_watchdog and timeout_latched) else run_okcopilot_ctx.last_full_workflow_test_ok = False+ # Re-affirmed per run below only when this run satisfies completion; never let a+ # prior run's terminal-ready latch leak into a run that did not verify.+ copilot_ctx.verified_terminal_proposal_ready = Falsecopilot_ctx.last_unverified_block_labels = _unverified_current_workflow_labels(copilot_ctx)copilot_ctx.last_test_failure_reason = Nonecopilot_ctx.last_artifact_health_blocker_reason = None@@ -1912,6 +2078,11 @@ def _record_run_blocks_result(if run_ok:_mark_page_inspected(copilot_ctx)+ completion_fully_satisfied = (+ completion_verification is not None+ and completion_verification.status == "evaluated"+ and completion_verification.is_fully_satisfied()+ )if structured_blocker:# Terminal anti-bot blockers are handled before run_ok; this branch# remains for non-challenge structured blockers that still make a@@ -1936,7 +2107,17 @@ def _record_run_blocks_result(display_reason=run_outcome_display_reason(structured_blocker),),)- if empty_data_blocks:+ if completion_fully_satisfied:+ # Terminal proposal promotion is authoritative only after the same+ # run has satisfied completion verification; clear stale+ # suspicious-success state before agent/enforcement exits consume+ # ``verified_terminal_proposal_ready``.+ copilot_ctx.verified_terminal_proposal_ready = True+ copilot_ctx.last_test_suspicious_success = False+ copilot_ctx.last_test_failure_reason = None+ copilot_ctx.null_data_streak_count = 0+ copilot_ctx.suspicious_success_nudge_count = 0+ if empty_data_blocks and not completion_fully_satisfied:copilot_ctx.last_test_ok = Nonecopilot_ctx.last_test_suspicious_success = Truecopilot_ctx.null_data_streak_count = getattr(copilot_ctx, "null_data_streak_count", 0) + 1@@ -1983,8 +2164,15 @@ def _record_run_blocks_result(# Real success: clear the signature latch so a subsequent bad URL in# the same session can re-fire the stop nudge.copilot_ctx.non_retriable_nav_error_last_emitted_signature = None- if outcome_unverified_reason is None and not unverified:+ if outcome_unverified_reason is None and completion_fully_satisfied:+ copilot_ctx.last_full_workflow_test_ok = True+ copilot_ctx.last_unverified_block_labels = []+ copilot_ctx.last_good_workflow = copilot_ctx.last_workflow+ copilot_ctx.last_good_workflow_yaml = copilot_ctx.last_workflow_yaml+ copilot_ctx.last_test_failure_reason = None+ elif outcome_unverified_reason is None and not unverified:copilot_ctx.last_full_workflow_test_ok = True+ copilot_ctx.last_unverified_block_labels = []copilot_ctx.last_good_workflow = copilot_ctx.last_workflowcopilot_ctx.last_good_workflow_yaml = copilot_ctx.last_workflow_yamlelif outcome_unverified_reason is None:diff --git a/skyvern/forge/sdk/copilot/tools/workflow_update.py b/skyvern/forge/sdk/copilot/tools/workflow_update.pyindex b1068291f..6b6844250 100644--- a/skyvern/forge/sdk/copilot/tools/workflow_update.py+++ b/skyvern/forge/sdk/copilot/tools/workflow_update.py@@ -3444,10 +3444,17 @@ async def _update_workflow("_workflow": workflow,}except (yaml.YAMLError, ValidationError, BaseWorkflowHTTPException) as e:- return {+ result: dict[str, Any] = {"ok": False,"error": f"Workflow validation failed: {e}",}+ if _copilot_block_authoring_policy(ctx) == BlockAuthoringPolicy.CODE_ONLY_BROWSER:+ result["data"] = _code_repair_progress_data()+ result["user_facing_summary"] = _code_seam_rejection_user_summary(+ metadata_rejected=False,+ code_rejected=True,+ )+ return resultdef _record_workflow_proxy_location_span(workflow_yaml: str, workflow: Workflow) -> None:diff --git a/tests/unit/test_client_pydantic_utilities.py b/tests/unit/test_client_pydantic_utilities.pynew file mode 100644index 000000000..3b2136dde--- /dev/null+++ b/tests/unit/test_client_pydantic_utilities.py@@ -0,0 +1,39 @@+from __future__ import annotations++import pytest++from skyvern.client.core.pydantic_utilities import IS_PYDANTIC_V2, update_forward_refs++[email protected](not IS_PYDANTIC_V2, reason="Pydantic v2-specific generated-client rebuild behavior")+def test_update_forward_refs_suppresses_pydantic_v2_definitions_key_error() -> None:+ class GeneratedModel:+ @classmethod+ def model_rebuild(cls, *, raise_errors: bool) -> None:+ assert raise_errors is False+ raise KeyError("definitions")++ update_forward_refs(GeneratedModel) # type: ignore[type-var]++[email protected](not IS_PYDANTIC_V2, reason="Pydantic v2-specific generated-client rebuild behavior")+def test_update_forward_refs_suppresses_definitions_key_error_with_extra_context() -> None:+ class GeneratedModel:+ @classmethod+ def model_rebuild(cls, *, raise_errors: bool) -> None:+ assert raise_errors is False+ raise KeyError("definitions", "extra-context")++ update_forward_refs(GeneratedModel) # type: ignore[type-var]++[email protected](not IS_PYDANTIC_V2, reason="Pydantic v2-specific generated-client rebuild behavior")+def test_update_forward_refs_preserves_unexpected_key_error() -> None:+ class GeneratedModel:+ @classmethod+ def model_rebuild(cls, *, raise_errors: bool) -> None:+ assert raise_errors is False+ raise KeyError("other")++ with pytest.raises(KeyError, match="other"):+ update_forward_refs(GeneratedModel) # type: ignore[type-var]diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.pyindex 18d9d152f..14473ce9c 100644--- a/tests/unit/test_copilot_agent_helpers.py+++ b/tests/unit/test_copilot_agent_helpers.py@@ -14,6 +14,7 @@ import yamlfrom skyvern.forge.sdk.api.llm.exceptions import LLMProviderErrorfrom skyvern.forge.sdk.copilot import agent as agent_modulefrom skyvern.forge.sdk.copilot.build_phase import BuildPhase+from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdictfrom skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy, CopilotConfigfrom skyvern.forge.sdk.copilot.diagnosis_repair_contract import (DiagnosisInput,@@ -33,6 +34,7 @@ from skyvern.forge.sdk.copilot.recoverable_failure import build_recoverable_failfrom skyvern.forge.sdk.copilot.request_policy import (_REDACTED_REFUSED_SECRET_TURN,TRANSCRIPT_ANCHOR_CHAR_CAP,+ CompletionCriterion,RequestPolicy,_classify_request,build_transcript_context,@@ -1244,6 +1246,7 @@ def _chat_request() -> SimpleNamespace:workflow_id="wf-1",workflow_permanent_id="wfp-1",workflow_copilot_chat_id="chat-1",+ workflow_yaml="",)@@ -1413,6 +1416,93 @@ class TestTranslateToAgentResultGating:assert agent_result.response_type == "ASK_QUESTION"assert agent_result.user_response == "Which account should I use?"+ def test_verified_terminal_state_surfaces_workflow_despite_weak_final_reply(self) -> None:+ workflow = SimpleNamespace(workflow_definition=SimpleNamespace(blocks=[]))+ ctx = _ctx(+ last_workflow=workflow,+ last_workflow_yaml="title: Verified Draft",+ last_test_ok=True,+ last_full_workflow_test_ok=True,+ )+ ctx.verified_terminal_proposal_ready = True+ ctx.completion_verification_result = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=["c0"],+ verdicts=[CriterionVerdict(criterion_id="c0", state="satisfied", reason_code="evidence_confirms")],+ )+ result = _fake_run_result({"type": "ASK_QUESTION", "user_response": "Do you want me to keep repairing?"})++ agent_result = asyncio.run(+ agent_module._translate_to_agent_result(+ result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1"+ )+ )++ assert agent_result.response_type == "REPLY"+ assert agent_result.updated_workflow is workflow+ assert agent_result.workflow_yaml == "title: Verified Draft"+ assert agent_result.clear_proposed_workflow is False+ assert agent_result.proposal_disposition == "auto_applicable"++ def test_output_field_confirmation_question_is_blocked_when_contract_present(self) -> None:+ ctx = _ctx(+ request_policy=RequestPolicy(+ user_response_policy="proceed",+ completion_contract_status="present",+ completion_criteria=[+ CompletionCriterion(id="provider", outcome="The returned record identifies the provider."),+ ],+ )+ )+ result = _fake_run_result(+ {+ "type": "ASK_QUESTION",+ "user_response": "Please confirm the output fields before I build and test this workflow.",+ }+ )++ agent_result = asyncio.run(+ agent_module._translate_to_agent_result(+ result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1"+ )+ )++ assert agent_result.response_type == "ASK_QUESTION"+ assert agent_result.updated_workflow is None+ assert agent_result.clear_proposed_workflow is False+ assert agent_result.proposal_disposition == "no_proposal"+ assert agent_result.output_policy_diagnostics is not None+ assert agent_result.output_policy_diagnostics["final_output_policy_allowed"] is False+ assert agent_result.output_policy_diagnostics["hard_block_reason_codes"] == [+ "avoidable_output_field_confirmation"+ ]++ def test_credential_clarification_question_remains_allowed_with_request_policy(self) -> None:+ ctx = _ctx(+ request_policy=RequestPolicy(+ user_response_policy="ask_clarification",+ clarification_question="Which saved credential should I use?",+ clarification_reason="credential_name_unresolved",+ completion_contract_status="present",+ completion_criteria=[+ CompletionCriterion(id="provider", outcome="The returned record identifies the provider."),+ ],+ )+ )+ result = _fake_run_result({"type": "ASK_QUESTION", "user_response": "Which saved credential should I use?"})++ agent_result = asyncio.run(+ agent_module._translate_to_agent_result(+ result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1"+ )+ )++ assert agent_result.response_type == "ASK_QUESTION"+ assert agent_result.clear_proposed_workflow is True+ assert agent_result.output_policy_diagnostics is not None+ assert agent_result.output_policy_diagnostics["final_output_policy_allowed"] is True+ assert "avoidable_output_field_confirmation" not in agent_result.output_policy_diagnostics["raw_reason_codes"]+def test_inline_replace_workflow_resets_test_ok_after_prior_pass(self, monkeypatch) -> None:# A prior run_blocks test passed for the old workflow (ctx.last_test_ok=True,# ctx.last_workflow=old_wf). The agent then emits inline REPLACE_WORKFLOWdiff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.pyindex 5f9134eb6..b10bab8ae 100644--- a/tests/unit/test_copilot_completion_verification.py+++ b/tests/unit/test_copilot_completion_verification.py@@ -4,6 +4,7 @@ import asyncioimport timefrom collections.abc import Awaitable, Callablefrom types import SimpleNamespace+from typing import Anyimport pytest@@ -18,7 +19,11 @@ from skyvern.forge.sdk.copilot.completion_verification import (CriterionVerdict,RunEvidenceSnapshot,_coerce_result,+ _structured_record_has_identifier,evaluate_completion_criteria,+ grade_record_semantic_consistency,+ grade_structured_record_criteria,+ structured_record_has_identity,summarize_unsatisfied_outcomes,)from skyvern.forge.sdk.copilot.context import CopilotContext@@ -64,6 +69,87 @@ def _criterion(cid: str, outcome: str, *, method_mandated: bool = False) -> Compreturn CompletionCriterion(id=cid, outcome=outcome, method_mandated=method_mandated)+_STRUCTURED_RECORD_CRITERIA = (+ ("fallback_record_identity", "The returned record identifies the target record."),+ ("fallback_record_identifier", "The returned record includes the record identifier."),+ ("fallback_record_groups", "The returned record includes record items."),+ (+ "fallback_record_status",+ "The returned record's per-location statuses and overall status are present and consistent.",+ ),+)+_STRUCTURED_RECORD_CRITERION_IDS = {cid for cid, _ in _STRUCTURED_RECORD_CRITERIA}+++def _structured_record_criteria() -> list[CompletionCriterion]:+ return [CompletionCriterion(id=cid, outcome=outcome) for cid, outcome in _STRUCTURED_RECORD_CRITERIA]+++def _status_consistency_criteria() -> list[CompletionCriterion]:+ return [+ CompletionCriterion(+ id="fallback_record_status",+ outcome="The returned record's per-location statuses and overall status are present and consistent.",+ )+ ]+++def test_structured_record_identity_ignores_substring_only_keys() -> None:+ assert structured_record_has_identity({"provider_name": "Alex Example"}) is True+ assert structured_record_has_identity({"providerName": "Alex Example"}) is True+ assert structured_record_has_identity({"title": "Permit A"}) is True+ assert structured_record_has_identity({"filename": "report.pdf"}) is False+ assert structured_record_has_identity({"tablename": "providers"}) is False+++def test_structured_record_identifier_ignores_substring_only_keys() -> None:+ assert _structured_record_has_identifier({"providerId": "abc"}) is True+ assert _structured_record_has_identifier({"record_number": "x"}) is True+ assert _structured_record_has_identifier({"idea": "some text"}) is False+ assert _structured_record_has_identifier({"_identical": "yes"}) is False+ assert structured_record_has_identity({"subtitle": "detail"}) is False+ assert structured_record_has_identity({"mislabeled": "detail"}) is False+++def _satisfied_criterion_ids(verdicts: list[CriterionVerdict]) -> set[str]:+ return {verdict.criterion_id for verdict in verdicts if verdict.satisfied}+++def _record_payload(**overrides: Any) -> dict[str, Any]:+ payload = {+ "entity_found": True,+ "entity_name": "Jordan Example",+ "record_number": "1234567890",+ "items": [+ {"item_name": "Sample Practice", "address": "100 Main St, Example City, ST 12345", "status": "Active"},+ {+ "item_name": "Secondary Practice",+ "address": "300 Market St, Example City, ST 12345",+ "status": "Inactive",+ },+ ],+ "overall_status": "Active",+ "evidence_text": "Opened Details page; read Overview/Affiliations items and More Details identifier.",+ }+ payload.update(overrides)+ return payload+++def _status_snapshot(+ status: str,+ *,+ item_name: str = "Sample Practice",+ evidence_text: str | None = None,+) -> RunEvidenceSnapshot:+ payload = _record_payload(+ items=[{"item_name": item_name, "address": "100 Main St, Example City, ST 12345", "status": status}],+ overall_status=status,+ )+ if evidence_text is not None:+ payload["evidence_text"] = evidence_text+ return RunEvidenceSnapshot(block_outputs={"lookup_record_status": payload})++def _evaluated(*satisfied_by_id: tuple[str, bool]) -> CompletionVerificationResult:ids = [cid for cid, _ in satisfied_by_id]verdicts = [@@ -84,6 +170,13 @@ def _completion_handler_lookup(handler: object) -> Callable[[object], Awaitable[return _lookup+def _patch_completion_handler(monkeypatch: pytest.MonkeyPatch, handler: object) -> None:+ monkeypatch.setattr(+ "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",+ _completion_handler_lookup(handler),+ )++def test_is_fully_satisfied_requires_every_criterion() -> None:assert _evaluated(("c0", True), ("c1", True)).is_fully_satisfied() is Trueassert _evaluated(("c0", True), ("c1", False)).is_fully_satisfied() is False@@ -223,6 +316,119 @@ def test_coerce_accepts_bytes_and_rejects_malformed() -> None:assert _coerce_result({"no_verdicts_key": 1}, ["c0"]).status == "unavailable"+ ("negative_status", "item_name", "missing_text"),+ [+ ("Inactive", "Sample Practice 200 Oak Ave, Example City, ST 12345 Active", "non-status fields include"),+ *[+ (status, "Sample Practice Active", None)+ for status in ["Expired", "Suspended", "Terminated", "Revoked", "Lapsed", "Pending"]+ ],+ ],+)+def test_record_semantic_consistency_flags_status_contradictions(+ negative_status: str, item_name: str, missing_text: str | None+) -> None:+ verdicts = grade_record_semantic_consistency(+ _status_consistency_criteria(),+ _status_snapshot(negative_status, item_name=item_name),+ )++ assert len(verdicts) == 1+ assert verdicts[0].criterion_id == "fallback_record_status"+ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "evidence_contradicts"+ if missing_text:+ assert missing_text in (verdicts[0].missing_evidence or "")+++ ("status", "item_name", "evidence_text"),+ [+ ("Non-active", "Sample Practice non-active listing", "Sample Practice non-active listing 100 Main St"),+ ("Active", "Sample Practice", "Sample Practice 100 Main St, Example City, ST 12345 Active"),+ *[+ ("Expired", "Sample Practice", text)+ for text in [+ "License is no longer active",+ "Provider was previously active",+ "Status note: not currently active",+ "The active license expired",+ ]+ ],+ ],+)+def test_record_semantic_consistency_accepts_non_contradictory_status_text(+ status: str, item_name: str, evidence_text: str+) -> None:+ assert (+ grade_record_semantic_consistency(+ _status_consistency_criteria(), _status_snapshot(status, item_name=item_name, evidence_text=evidence_text)+ )+ == []+ )+++def test_structured_record_identifier_requires_consecutive_digit_run() -> None:+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "lookup_record": _record_payload(+ phone="555-1234",+ record_number=None,+ items=[{"item_name": "Sample Practice", "address": "1234 Main St, Apt 56", "status": "Active"}],+ )+ }+ )++ satisfied = _satisfied_criterion_ids(grade_structured_record_criteria(_structured_record_criteria(), snapshot))++ assert "fallback_record_identifier" not in satisfied+ assert "fallback_record_identity" in satisfied+++ "block_outputs",+ [+ {+ "extract_record_status_info": {+ "extract_record_status_info_output": _record_payload(),+ "extracted_information": [],+ }+ },+ {"extract_record_status_record_output": _record_payload(found=True, entity_found=None)},+ ],+)+def test_structured_record_criteria_satisfy_structured_record_outputs(block_outputs: dict[str, Any]) -> None:+ snapshot = RunEvidenceSnapshot(block_outputs=block_outputs)++ verdicts = grade_structured_record_criteria(_structured_record_criteria(), snapshot)++ assert _satisfied_criterion_ids(verdicts) == _STRUCTURED_RECORD_CRITERION_IDS+++def test_structured_record_partial_matches_do_not_combine_across_blocks() -> None:+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "identity_block": _record_payload(items=[], overall_status=None),+ "status_block": {+ "items": [{"item_name": "Sample Practice", "address": "100 Main St", "status": "Active"}],+ "overall_status": "Active",+ },+ }+ )++ criteria = _structured_record_criteria()+ verdicts = grade_structured_record_criteria(criteria, snapshot)+ result = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=[criterion.id for criterion in criteria],+ verdicts=verdicts,+ )++ assert _satisfied_criterion_ids(verdicts) < _STRUCTURED_RECORD_CRITERION_IDS+ assert result.is_fully_satisfied() is False++@pytest.mark.asyncioasync def test_evaluate_no_handler_or_no_criteria_is_unavailable() -> None:snapshot = RunEvidenceSnapshot(current_url="https://example.com")@@ -569,6 +775,40 @@ def _clean_success_result() -> dict:}+def _structured_record_top_level_output_result() -> dict:+ return {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_structured_record",+ "overall_status": "completed",+ "executed_block_labels": ["extract_record_status_record"],+ "current_url": "https://structured_record.test/entity-details",+ "blocks": [+ {+ "label": "extract_record_status_record",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {"extracted_information": []},+ }+ ],+ "output": {+ "open_search_search_output": {+ "page_state": "search_search_open",+ "evidence_text": "Opened search search page with search-by-doctor typeahead #searchInput.",+ },+ "search_and_open_record_details_output": {+ "found": True,+ "entity_name": "Jordan Example",+ "opened_record_details": True,+ "evidence_text": "Opened Details page for the selected record.",+ },+ "extract_record_status_record_output": _record_payload(found=True, entity_found=None),+ "extracted_information": [],+ },+ },+ }++def _run_ctx() -> CopilotContext:ctx = CopilotContext(organization_id="o",@@ -591,6 +831,12 @@ def _ctx_with_blocks(*block_types: str) -> CopilotContext:return ctx+def _set_workflow_labels(ctx: CopilotContext, *labels: str) -> None:+ ctx.last_workflow = SimpleNamespace(+ workflow_definition=SimpleNamespace(blocks=[SimpleNamespace(label=label) for label in labels])+ )++def _contradicted(cid: str) -> CompletionVerificationResult:verdict = CriterionVerdict(criterion_id=cid, state="unsatisfied", reason_code="evidence_contradicts")return CompletionVerificationResult(status="evaluated", criterion_ids=[cid], verdicts=[verdict])@@ -688,10 +934,7 @@ async def test_goto_only_run_still_fails_extraction_verification(monkeypatch: py]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _ctx_with_blocks("goto_url")ctx.request_policy = RequestPolicy(completion_criteria=[_criterion("c0", "heading and paragraph are extracted")])@@ -711,10 +954,7 @@ async def test_structured_blocker_run_skips_completion_verification(monkeypatch:async def handler(**_: object) -> dict:raise AssertionError("structured blocker runs must not be sent to the completion judge")- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _ctx_with_blocks("code")result = {"ok": True,@@ -1001,10 +1241,7 @@ async def test_maybe_run_completion_verification_runs_on_canceled_run(monkeypatcasync def handler(**_: object) -> dict:return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()result = await _maybe_run_completion_verification(ctx, _canceled_budget_result(), time.monotonic())assert result is not None@@ -1254,10 +1491,7 @@ async def test_page_observation_verification_recognizes_budgeted_outcome(seen_prompt["prompt"] = str(kwargs.get("prompt") or "")return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()ctx.last_test_ok = Falsectx.last_run_blocks_workflow_run_id = "wr_cancel"@@ -1300,10 +1534,7 @@ async def test_page_observation_verification_does_not_overwrite_satisfied_verdicasync def handler(**_: object) -> dict:raise AssertionError("handler should not be called once the outcome is verified")- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()ctx.last_test_ok = Falsectx.last_run_blocks_workflow_run_id = "wr_cancel"@@ -1337,10 +1568,7 @@ async def test_page_observation_verification_preserves_existing_unsatisfied_verdhandler_calls += 1return {"verdicts": [{"criterion_id": "c0", "satisfied": False, "reason_code": "no_evidence"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()ctx.last_test_ok = Falsectx.last_run_blocks_workflow_run_id = "wr_cancel"@@ -1371,10 +1599,7 @@ async def test_page_observation_verification_can_upgrade_unsatisfied_verdict(async def handler(**_: object) -> dict:return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()ctx.last_test_ok = Falsectx.last_run_blocks_workflow_run_id = "wr_cancel"@@ -1477,10 +1702,7 @@ async def test_maybe_run_completion_verification_runs_on_unverified_prefix(monkeasync def handler(**_: object) -> dict:return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _ctx_unverified_prefix()result = await _maybe_run_completion_verification(ctx, _clean_success_result(), time.monotonic())assert result is not None@@ -1507,10 +1729,7 @@ async def test_method_mandated_criteria_excluded_from_verification(monkeypatch:async def handler(**_: object) -> dict:return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()ctx.request_policy = RequestPolicy(completion_criteria=[@@ -1561,11 +1780,238 @@ def test_snapshot_uses_current_run_blocks_not_stale_outputs() -> None:assert snap2.block_outputs.get("b3") == {"extracted_information": {"price": "9.99"}}-def test_snapshot_summarizes_registered_download_outputs() -> None:+def test_snapshot_indexes_workflow_output_parameter_records() -> None:ctx = _run_ctx()- ctx.last_workflow = SimpleNamespace(- workflow_definition=SimpleNamespace(blocks=[SimpleNamespace(label="download_statement")])+ _set_workflow_labels(+ ctx,+ "open_search_search_page",+ "search_and_open_record_details",+ "extract_record_status_record",)+ run = {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_structured_record",+ "blocks": [+ {+ "label": "open_search_search_page",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {+ "open_search_search_page_output": {"evidence_text": "Opened search search page"}+ },+ },+ {+ "label": "search_and_open_record_details",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {+ "search_and_open_record_details_output": {+ "entity_found": True,+ "evidence_text": "Opened Details page",+ }+ },+ },+ {+ "label": "extract_record_status_record",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {"extract_record_status_record_output": _record_payload(evidence_text=None)},+ },+ ],+ },+ }++ snap = _build_run_evidence_snapshot(ctx, run)++ assert "open_search_search_page_output" in snap.block_outputs+ assert snap.block_outputs["search_and_open_record_details_output"]["evidence_text"] == ("Opened Details page")+ assert snap.block_outputs["extract_record_status_record_output"]["record_number"] == "1234567890"+++def test_snapshot_uses_current_run_registered_output_parameters() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_record_status_details")+ run = {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_structured_record",+ "overall_status": "completed",+ "blocks": [],+ "registered_output_parameter_values": [+ {+ "workflow_run_id": "wr_structured_record",+ "output_parameter_id": "op_record",+ "output_parameter_key": "extract_record_status_details_output",+ "block_label": "extract_record_status_details",+ "block_type": "CODE",+ "value": _record_payload(evidence_text="Opened Details page"),+ }+ ],+ },+ }++ snap = _build_run_evidence_snapshot(ctx, run)+ verdicts = grade_structured_record_criteria(_structured_record_criteria(), snap)++ assert snap.block_outputs["extract_record_status_details_output"]["record_number"] == "1234567890"+ assert (+ snap.block_outputs["extract_record_status_details"]["extract_record_status_details_output"]["record_number"]+ == "1234567890"+ )+ assert _satisfied_criterion_ids(verdicts) == _STRUCTURED_RECORD_CRITERION_IDS+++def test_snapshot_uses_structured_record_top_level_output_parameters() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_record_status_record")++ snap = _build_run_evidence_snapshot(ctx, _structured_record_top_level_output_result())+ verdicts = grade_structured_record_criteria(_structured_record_criteria(), snap)++ assert snap.block_outputs["extract_record_status_record_output"]["record_number"] == "1234567890"+ assert _satisfied_criterion_ids(verdicts) == _STRUCTURED_RECORD_CRITERION_IDS+++async def test_maybe_run_completion_verification_deterministically_satisfies_structured_record_without_judge_call(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ handler_lookup_calls = 0++ async def fail_handler(**_: object) -> object:+ raise AssertionError("deterministically covered structured-record criteria must not call the judge")++ async def handler_lookup(_ctx: object) -> object:+ nonlocal handler_lookup_calls+ handler_lookup_calls += 1+ return fail_handler++ monkeypatch.setattr(+ "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",+ handler_lookup,+ )+ ctx = _run_ctx()+ ctx.request_policy = RequestPolicy(completion_criteria=_structured_record_criteria())++ verification = await _maybe_run_completion_verification(+ ctx,+ _structured_record_top_level_output_result(),+ time.monotonic(),+ )++ assert verification is not None+ assert verification.status == "evaluated"+ assert verification.is_fully_satisfied() is True+ assert handler_lookup_calls == 1+ assert _satisfied_criterion_ids(verification.verdicts) == _STRUCTURED_RECORD_CRITERION_IDS+++async def test_maybe_run_completion_verification_fails_closed_on_no_judge_for_deterministic_success(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ _patch_completion_handler(monkeypatch, None)+ ctx = _run_ctx()+ ctx.request_policy = RequestPolicy(completion_criteria=_structured_record_criteria())++ verification = await _maybe_run_completion_verification(+ ctx,+ _structured_record_top_level_output_result(),+ time.monotonic(),+ )++ assert verification is not None+ assert verification.status == "unavailable"+ assert verification.is_fully_satisfied() is False+++async def test_maybe_run_completion_verification_fails_closed_on_low_budget_for_deterministic_success(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def fail_handler(**_: object) -> object:+ raise AssertionError("low-budget deterministic verification must not call the judge")++ _patch_completion_handler(monkeypatch, fail_handler)+ ctx = _run_ctx()+ ctx.request_policy = RequestPolicy(completion_criteria=_structured_record_criteria())+ starved = time.monotonic() - 100_000++ verification = await _maybe_run_completion_verification(+ ctx,+ _structured_record_top_level_output_result(),+ starved,+ )++ assert verification is not None+ assert verification.status == "unavailable"+ assert verification.is_fully_satisfied() is False+++async def test_maybe_run_completion_verification_mixed_criteria_still_fail_closed_on_judge_timeout(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ calls = 0++ async def slow_handler(**_: object) -> dict:+ nonlocal calls+ calls += 1+ await asyncio.sleep(1)+ return {"verdicts": []}++ monkeypatch.setattr(settings, "COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS", 0.01)+ _patch_completion_handler(monkeypatch, slow_handler)+ ctx = _run_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ *_structured_record_criteria(),+ CompletionCriterion(+ id="source_timestamp_visible", outcome="The source page shows the latest update timestamp."+ ),+ ]+ )++ verification = await _maybe_run_completion_verification(+ ctx,+ _structured_record_top_level_output_result(),+ time.monotonic(),+ )++ assert calls == 1+ assert verification is not None+ assert verification.status == "unavailable"+ assert verification.is_fully_satisfied() is False+++def test_snapshot_ignores_registered_output_parameters_from_prior_run() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_record_status_details")+ run = {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_current",+ "blocks": [],+ "registered_output_parameter_values": [+ {+ "workflow_run_id": "wr_prior",+ "output_parameter_key": "extract_record_status_details_output",+ "block_label": "extract_record_status_details",+ "value": {"entity_name": "Jordan Example", "record_number": "1234567890"},+ }+ ],+ },+ }++ snap = _build_run_evidence_snapshot(ctx, run)++ assert snap.block_outputs == {}+++def test_snapshot_summarizes_registered_download_outputs() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "download_statement")run = {"ok": True,"data": {@@ -1681,10 +2127,7 @@ async def test_completion_verification_receives_verified_context_labels(monkeypa]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()ctx.request_policy = RequestPolicy(completion_criteria=[@@ -1750,10 +2193,7 @@ async def test_maybe_run_completion_verification_unavailable_on_low_budget(monkeasync def handler(**_: object) -> dict:return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(handler),- )+ _patch_completion_handler(monkeypatch, handler)ctx = _run_ctx()starved = time.monotonic() - 100_000 # no budget left to verify this candidate runresult = await _maybe_run_completion_verification(ctx, _clean_success_result(), starved)@@ -1763,12 +2203,13 @@ async def test_maybe_run_completion_verification_unavailable_on_low_budget(monkeassert result.status == "unavailable"assert result.is_fully_satisfied() is False- # A missing judge handler stays a soft fallback (None), not a hard block.- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",- _completion_handler_lookup(None),- )- assert await _maybe_run_completion_verification(ctx, _clean_success_result(), time.monotonic()) is None+ # A missing judge handler means the required completion contract could not be+ # verified, so the run must not pass through on status alone.+ _patch_completion_handler(monkeypatch, None)+ no_handler_result = await _maybe_run_completion_verification(ctx, _clean_success_result(), time.monotonic())+ assert no_handler_result is not None+ assert no_handler_result.status == "unavailable"+ assert no_handler_result.is_fully_satisfied() is Falsedef test_completion_contract_not_violated_unavailable_blocks_surfacing() -> None:diff --git a/tests/unit/test_copilot_enforcement_pruning.py b/tests/unit/test_copilot_enforcement_pruning.pyindex 6f9bae7de..236de30da 100644--- a/tests/unit/test_copilot_enforcement_pruning.py+++ b/tests/unit/test_copilot_enforcement_pruning.py@@ -15,11 +15,13 @@ from unittest.mock import AsyncMock, MagicMockimport pytest+from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdictfrom skyvern.forge.sdk.copilot.enforcement import (KEEP_RECENT_TOOL_OUTPUTS,NULL_DATA_STREAK_ESCALATE_AT,POST_REPEATED_NULL_DATA_NUDGE,POST_SUSPICIOUS_SUCCESS_NUDGE,+ CopilotGoalSatisfied,_check_enforcement,_needs_repeated_null_data_nudge,_needs_suspicious_success_nudge,@@ -64,6 +66,10 @@ class _Ctx:self.null_data_streak_count = 0self.repeated_failure_streak_count = 0self.repeated_failure_nudge_emitted_at_streak = 0+ self.verified_terminal_proposal_ready = False+ self.completion_verification_result = None+ self.last_artifact_health_blocker_reason = None+ self.latest_diagnosis_repair_contract = None# ---------------------------------------------------------------------------@@ -243,6 +249,26 @@ def _text_prompt_block(extracted_data: Any) -> dict[str, Any]:_EXTRACTION_ENVELOPE_CASES: list[tuple[str, dict[str, Any], bool]] = [("empty_payload_trace_repro", {}, True),("real_extraction", {"extracted_information": [{"price": "260.48"}]}, False),+ (+ "nested_code_output_record",+ {+ "extracted_information": [],+ "extract_record_status_info_output": {+ "entity_found": True,+ "entity_name": "Jordan Example",+ "record_number": "1234567890",+ "items": [+ {+ "item_name": "Sample Practice",+ "address": "100 Main St, Example City, ST 12345",+ "status": "Active",+ }+ ],+ "overall_status": "Active",+ },+ },+ False,+ ),("download_only_files",{"downloaded_files": [{"url": "https://example.com/a.pdf", "checksum": "abc123"}]},@@ -384,6 +410,7 @@ def test_record_run_blocks_result_promotes_when_verified_prefix_covers_workflow()ctx.last_workflow_yaml = "workflow: yaml"ctx.verified_prefix_labels = ["open", "extract"]+ ctx.last_unverified_block_labels = ["stale_extract"]result = {"ok": True,@@ -404,6 +431,121 @@ def test_record_run_blocks_result_promotes_when_verified_prefix_covers_workflow(assert ctx.last_good_workflow_yaml == ctx.last_workflow_yaml+def test_record_run_blocks_result_promotes_structured_record_top_level_output_to_terminal_proposal() -> None:+ ctx = _fresh_ctx_for_record()+ ctx.last_workflow = SimpleNamespace(+ workflow_definition=SimpleNamespace(+ blocks=[+ SimpleNamespace(label="open_search_search"),+ SimpleNamespace(label="search_and_open_record_details"),+ SimpleNamespace(label="extract_record_status_record"),+ ]+ )+ )+ ctx.last_workflow_yaml = "title: Record lookup"+ ctx.verified_prefix_labels = ["open_search_search"]+ result = {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_structured_record",+ "overall_status": "completed",+ "executed_block_labels": ["extract_record_status_record"],+ "blocks": [+ {+ "label": "extract_record_status_record",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {"extracted_information": []},+ }+ ],+ "output": {+ "search_and_open_record_details_output": {+ "found": True,+ "entity_name": "Jordan Example",+ "opened_record_details": True,+ "evidence_text": "Opened Details page for the selected record.",+ },+ "extract_record_status_record_output": {+ "found": True,+ "entity_name": "Jordan Example",+ "record_number": "1234567890",+ "items": [+ {+ "item_label": "Sample Practice",+ "address": "100 Main St, Example City, ST 12345",+ "status": "Active",+ }+ ],+ "overall_status": "Active",+ "evidence_text": "Opened Details page; read Overview/Affiliations items and More Details identifier.",+ },+ "extracted_information": [],+ },+ },+ }+ verification = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=[+ "fallback_record_identity",+ "fallback_record_identifier",+ "fallback_record_groups",+ "fallback_record_status",+ ],+ verdicts=[+ CriterionVerdict(criterion_id=cid, state="satisfied", reason_code="evidence_confirms")+ for cid in (+ "fallback_record_identity",+ "fallback_record_identifier",+ "fallback_record_groups",+ "fallback_record_status",+ )+ ],+ )++ _record_run_blocks_result(ctx, result, completion_verification=verification)++ assert ctx.verified_terminal_proposal_ready is True+ assert ctx.last_test_ok is True+ assert ctx.last_full_workflow_test_ok is True+ assert ctx.last_test_suspicious_success is False+ assert ctx.last_test_failure_reason is None+++def test_record_run_blocks_result_resets_stale_verified_terminal_proposal_latch() -> None:+ ctx = _fresh_ctx_for_record()+ ctx.verified_terminal_proposal_ready = True+ result = {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_unverified",+ "overall_status": "completed",+ "executed_block_labels": [],+ "blocks": [],+ "output": {},+ },+ }++ _record_run_blocks_result(ctx, result, completion_verification=None)++ assert ctx.verified_terminal_proposal_ready is False+++def test_enforcement_stops_after_verified_terminal_proposal() -> None:+ ctx = _Ctx()+ ctx.verified_terminal_proposal_ready = True+ ctx.last_test_ok = True+ ctx.last_full_workflow_test_ok = True+ ctx.completion_verification_result = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=["c0"],+ verdicts=[CriterionVerdict(criterion_id="c0", state="satisfied", reason_code="evidence_confirms")],+ )+ ctx.last_test_suspicious_success = True++ with pytest.raises(CopilotGoalSatisfied):+ _check_enforcement(ctx)++def test_record_run_blocks_result_keeps_failure_when_watchdog_cancel_without_timeout() -> None:"""Stagnation/ceiling cancels mid-session must still set last_test_ok=Falseso the failed-test nudge can fire — only a coincident total timeout softensdiff --git a/tests/unit/test_copilot_loop_detection_blocker.py b/tests/unit/test_copilot_loop_detection_blocker.pyindex 7ad15ac09..e589d7fd9 100644--- a/tests/unit/test_copilot_loop_detection_blocker.py+++ b/tests/unit/test_copilot_loop_detection_blocker.py@@ -175,11 +175,8 @@ def test_consecutive_non_evaluate_loop_ignores_loaded_result_steer() -> None:assert dict(ctx.blocker_signal.extra) == {}-def test_consecutive_evaluate_loop_ignores_stale_composition_page_results_without_latest_steer() -> None:+def test_consecutive_evaluate_loop_without_composition_steer_uses_generic_copy() -> None:ctx = _ctx(consecutive_tool_tracker=["evaluate", "evaluate"])- ctx.composition_page_evidence = {- "result_containers": [{"tag": "table", "selector": "#old-results", "row_count": 3}],- }payload = _tool_loop_error(ctx, "evaluate", None)assert payload is not Nonediff --git a/tests/unit/test_copilot_output_policy.py b/tests/unit/test_copilot_output_policy.pyindex 4bbb09d58..23261fafb 100644--- a/tests/unit/test_copilot_output_policy.py+++ b/tests/unit/test_copilot_output_policy.py@@ -547,6 +547,65 @@ def test_rejects_reply_when_request_policy_required_clarification() -> None:assert OutputPolicyReason.REQUEST_POLICY_CLARIFICATION_BYPASS in verdict.reason_codes+def test_avoidable_output_field_confirmation_is_hard_blocked() -> None:+ policy = _policy(+ user_response_policy="proceed",+ completion_contract_status="present",+ completion_criteria=[+ SimpleNamespace(id="record_identity", outcome="The returned record identifies the target record."),+ ],+ )++ verdict = evaluate_output_policy(+ request_policy=policy,+ response_type="ASK_QUESTION",+ user_response="Please confirm the output fields before I build this record status workflow.",+ has_workflow_proposal=False,+ workflow_attempted=False,+ )+ hard = hard_block_output_policy_verdict(verdict)++ assert OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION in verdict.reason_codes+ assert hard.reason_codes == [OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION]+++def test_output_field_confirmation_allowed_when_request_policy_requires_clarification() -> None:+ policy = _policy(+ user_response_policy="ask_clarification",+ clarification_question="Which saved credential should I use?",+ clarification_reason="credential_name_unresolved",+ completion_contract_status="present",+ completion_criteria=[+ SimpleNamespace(id="record_identity", outcome="The returned record identifies the target record."),+ ],+ )++ verdict = evaluate_output_policy(+ request_policy=policy,+ response_type="ASK_QUESTION",+ user_response="Which saved credential should I use?",+ has_workflow_proposal=False,+ workflow_attempted=False,+ )++ assert OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION not in verdict.reason_codes+ assert verdict.allowed is True+++def test_url_clarification_does_not_trigger_output_field_confirmation_guard() -> None:+ policy = _policy(user_response_policy="proceed", completion_contract_status="present")++ verdict = evaluate_output_policy(+ request_policy=policy,+ response_type="ASK_QUESTION",+ user_response="Which URL should I use for this workflow?",+ has_workflow_proposal=False,+ workflow_attempted=False,+ )++ assert OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION not in verdict.reason_codes++def test_classifies_unbacked_workflow_delivery_claim() -> None:verdict = evaluate_output_policy(request_policy=_policy(),diff --git a/tests/unit/test_copilot_page_evidence_policy.py b/tests/unit/test_copilot_page_evidence_policy.pyindex aae163f39..bc16b9b53 100644--- a/tests/unit/test_copilot_page_evidence_policy.py+++ b/tests/unit/test_copilot_page_evidence_policy.py@@ -84,6 +84,14 @@ def test_agent_prompt_requires_result_page_evidence_before_expansion_blocks() ->assert "omit the expansion block and proceed to extraction from the visible results" in rendered+def test_agent_prompt_requires_details_evidence_label() -> None:+ rendered = _render_agent_prompt()++ assert "details or profile page" in rendered+ assert "visible details/profile page label" in rendered+ assert "page label only as supporting evidence" in rendered++def test_tool_descriptions_ground_composition_without_prescribing_extra_workflow_blocks() -> None:for tool in (update_workflow_tool, run_blocks_tool, update_and_run_blocks_tool):desc = tool.description # type: ignore[attr-defined]diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.pyindex 93691e5dd..4a59df3ab 100644--- a/tests/unit/test_copilot_request_policy_prompt.py+++ b/tests/unit/test_copilot_request_policy_prompt.py@@ -6,10 +6,12 @@ from pathlib import Pathimport pytestfrom skyvern.forge.prompts import prompt_engine+from skyvern.forge.sdk.copilot import request_policy as request_policy_modulefrom skyvern.forge.sdk.copilot.request_policy import (PROMPT_NAME,RAW_SECRET_REFUSAL_SENTINEL,CompletionCriterion,+ _classifier_fallback_policy,_classify_request,_credential_ids,_raw_secret_detected,@@ -364,3 +366,99 @@ class TestActiveCriteriaPromptAnchor:handler=_capture_handler(captured),)assert "ACTIVE COMPLETION CRITERIA (canonical phrasing for the current goal):" not in captured["prompt"]+++class TestClassifierFallbackCompletionCriteria:+ @pytest.mark.parametrize(+ ("user_message", "expect_criteria"),+ [+ (+ (+ "I want to build a reusable workflow that checks record status. "+ "Capture the identifier and the list of practice items with each location's status, "+ "and the result should come out as a record with the entity name, identifier, items, "+ "and an overall status."+ ),+ True,+ ),+ (+ (+ "Build a reusable fixture directory lookup workflow for Jordan Example. Return a record with "+ "the entity name, identifier 1234567890, items, per-location status, overall status, and "+ "no-results behavior."+ ),+ True,+ ),+ ("Extract the user's name, id, address, and status from the portal.", False),+ ("Open https://example.com and click the pricing link.", False),+ # Substring matches ("read" in "already", "name" in "filename", "id" in "decided")+ # must not satisfy the whole-word gate.+ ("I already decided the filename and reviewed the status of locations.", False),+ # Generic structural group term ("entries") satisfies the group gate.+ ("Return a record with the entity name, identifier 1234567890, status, and the grouped entries.", True),+ ],+ )+ @pytest.mark.asyncio+ async def test_classifier_fallback_structured_record_criteria(+ self, user_message: str, expect_criteria: bool+ ) -> None:+ policy = await _classify_request(+ user_message=user_message,+ workflow_yaml="",+ chat_history=[],+ global_llm_context="",+ handler=None,+ )++ assert policy.classifier_status == "fallback"+ assert policy.completion_contract_status == ("present" if expect_criteria else "unknown")+ if not expect_criteria:+ assert policy.completion_criteria == []+ return++ assert [criterion.id for criterion in policy.completion_criteria] == [+ "fallback_record_identity",+ "fallback_record_identifier",+ "fallback_record_groups",+ "fallback_record_status",+ ]+ assert policy.requires_user_clarification is False+ assert policy.user_response_policy == "proceed"+ assert all(criterion.implicit for criterion in policy.completion_criteria)+ assert all(criterion.level == "run" for criterion in policy.completion_criteria)++ def test_classifier_fallback_logs_synthesized_structured_record_criteria(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[tuple[str, dict[str, object]]] = []++ class FakeLogger:+ def info(self, event: str, **kwargs: object) -> None:+ calls.append((event, kwargs))++ monkeypatch.setattr(request_policy_module, "LOG", FakeLogger())++ policy = _classifier_fallback_policy(+ [],+ raw_secret_present=False,+ failure_kind="missing_handler",+ user_message=(+ "Return a record with the entity name, identifier, items, per-location status, and overall status."+ ),+ )++ assert policy.completion_contract_status == "present"+ assert calls == [+ (+ "copilot request policy synthesized fallback structured-record criteria",+ {+ "classifier_failure_kind": "missing_handler",+ "completion_criterion_ids": [+ "fallback_record_identity",+ "fallback_record_identifier",+ "fallback_record_groups",+ "fallback_record_status",+ ],+ },+ )+ ]diff --git a/tests/unit/test_copilot_run_outcome_event.py b/tests/unit/test_copilot_run_outcome_event.pyindex af5c16881..c5fbc90bf 100644--- a/tests/unit/test_copilot_run_outcome_event.py+++ b/tests/unit/test_copilot_run_outcome_event.py@@ -325,7 +325,9 @@ async def test_satisfied_adjudication_emits_demonstrated_with_unverified_prefix(frames = _run_outcome_frames(ctx.stream) # type: ignore[arg-type]assert [frame.verdict for frame in frames] == ["evaluating", "demonstrated"]assert frames[-1].reason_code is None- assert ctx.last_full_workflow_test_ok is False+ # A fully-verified completion promotes the full-workflow test result; this is the+ # deterministic terminal path that lets a verified run finalize success.+ assert ctx.last_full_workflow_test_ok is Trueassert ctx.last_run_outcome == RecordedRunOutcome(verdict="demonstrated", workflow_run_id="wr_test")diff --git a/tests/unit/test_copilot_run_output_content_gate.py b/tests/unit/test_copilot_run_output_content_gate.pyindex f6b5b539c..6b4b895ad 100644--- a/tests/unit/test_copilot_run_output_content_gate.py+++ b/tests/unit/test_copilot_run_output_content_gate.py@@ -24,9 +24,41 @@ from skyvern.forge.sdk.copilot.tools import (_record_run_blocks_result,_run_blocks_structured_blocker_message,)+from skyvern.forge.sdk.copilot.tools._shared import _registered_output_parameter_payloads+from skyvern.forge.sdk.copilot.tools.run_execution import _attach_registered_output_parameter_valuesfrom skyvern.forge.sdk.copilot.turn_halt import TurnHaltKind+ ("data", "expected"),+ [+ (+ {+ "registered_output_parameter_values": [+ {"workflow_run_id": "wr_prior", "value": {"x": 1}},+ {"value": {"y": 2}},+ ]+ },+ [],+ ),+ (+ {+ "workflow_run_id": "wr_now",+ "registered_output_parameter_values": [+ {"workflow_run_id": "wr_now", "value": {"x": 1}},+ {"workflow_run_id": "wr_prior", "value": {"y": 2}},+ ],+ },+ [{"x": 1}],+ ),+ ],+)+def test_registered_output_parameter_payloads_scope_to_current_run(+ data: dict[str, Any], expected: list[dict[str, int]]+) -> None:+ assert [dict(p["value"]) for p in _registered_output_parameter_payloads(data)] == expected++def _code_block(label: str, extracted: Any, *, block_type: str = "CODE") -> dict[str, Any]:return {"label": label, "block_type": block_type, "status": "completed", "extracted_data": extracted}@@ -43,6 +75,21 @@ def _run_result(blocks: list[dict[str, Any]], *, ok: bool = True) -> dict[str, A}+def _structured_record_payload(**overrides: Any) -> dict[str, Any]:+ payload = {+ "entity_found": True,+ "entity_name": "Jordan Example",+ "record_number": "1234567890",+ "items": [+ {"item_label": "Sample Practice", "address": "100 Main St, Example City, ST 12345", "status": "Active"}+ ],+ "overall_status": "Active",+ "evidence_text": "Opened Details page",+ }+ payload.update(overrides)+ return payload++def _ctx(blocks: list[dict[str, Any]] | None = None) -> CopilotContext:ctx = CopilotContext(organization_id="o",@@ -69,6 +116,17 @@ def _no_evidence(cid: str) -> CompletionVerificationResult:return CompletionVerificationResult(status="evaluated", criterion_ids=[cid], verdicts=[verdict])+def _satisfied(*criterion_ids: str) -> CompletionVerificationResult:+ return CompletionVerificationResult(+ status="evaluated",+ criterion_ids=list(criterion_ids),+ verdicts=[+ CriterionVerdict(criterion_id=criterion_id, state="satisfied", reason_code="evidence_confirms")+ for criterion_id in criterion_ids+ ],+ )++def _blocked_flag_run_result() -> dict[str, Any]:return _run_result([@@ -168,6 +226,80 @@ def _goal_field_success_run_result() -> dict[str, Any]:)+def _nested_code_output_extraction_run_result() -> dict[str, Any]:+ return _run_result(+ [+ _code_block(+ "extract_record_status_info",+ {+ "extract_record_status_info_output": _structured_record_payload(),+ "extracted_information": [],+ },+ block_type="EXTRACTION",+ )+ ]+ )+++def _empty_extraction_run_result() -> dict[str, Any]:+ return _run_result(+ [+ _code_block(+ "extract_empty_results",+ {"extracted_information": [], "downloaded_files": [], "downloaded_file_urls": None},+ block_type="EXTRACTION",+ )+ ]+ )+++def _registered_code_output_parameter_run_result(*, workflow_run_id: str = "wr_test") -> dict[str, Any]:+ return {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_test",+ "overall_status": "completed",+ "blocks": [],+ "registered_output_parameter_values": [+ {+ "workflow_run_id": workflow_run_id,+ "output_parameter_id": "op_details",+ "output_parameter_key": "extract_record_status_details_output",+ "block_label": "extract_record_status_details",+ "block_type": "CODE",+ "value": _structured_record_payload(),+ }+ ],+ },+ }+++def _structured_record_qa_top_level_output_run_result() -> dict[str, Any]:+ result = _run_result(+ [+ _code_block(+ "extract_record_status_record",+ {"extracted_information": []},+ )+ ]+ )+ result["data"]["output"] = {+ "open_search_search_output": {+ "page_state": "search_search_open",+ "evidence_text": "Opened search search page with search-by-doctor typeahead #searchInput.",+ },+ "search_and_open_record_details_output": {+ "found": True,+ "entity_name": "Jordan Example",+ "opened_record_details": True,+ "evidence_text": "Opened Details page for the selected record.",+ },+ "extract_record_status_record_output": _structured_record_payload(found=True, entity_found=None),+ "extracted_information": [],+ }+ return result++def _partial_goal_field_run_result() -> dict[str, Any]:return _run_result([@@ -335,6 +467,133 @@ def test_genuine_success_run_keeps_clean_path() -> None:assert ctx.last_full_workflow_test_ok is True+def test_nested_code_output_record_is_meaningful_and_not_suspicious_when_verified() -> None:+ result = _nested_code_output_extraction_run_result()+ ctx = _ctx(result["data"]["blocks"])++ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)+ assert empty_data_blocks is False+ assert _is_outcome_evidence_candidate(ctx, result) is True++ _record_run_blocks_result(+ ctx,+ result,+ completion_verification=_satisfied("fallback_record_identity", "fallback_record_identifier"),+ )++ assert ctx.last_test_ok is True+ assert ctx.last_test_suspicious_success is False+ assert ctx.last_full_workflow_test_ok is True+ assert verified_goal_satisfied_context(ctx) is True+++def test_registered_code_output_parameter_record_is_meaningful() -> None:+ result = _registered_code_output_parameter_run_result()+ ctx = _ctx([])++ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)++ assert empty_data_blocks is False+++def test_registered_code_output_parameter_record_is_current_run_scoped() -> None:+ result = _empty_extraction_run_result()+ result["data"]["registered_output_parameter_values"] = _registered_code_output_parameter_run_result(+ workflow_run_id="wr_prior"+ )["data"]["registered_output_parameter_values"]+ ctx = _ctx(result["data"]["blocks"])++ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)++ assert empty_data_blocks is True+++def test_structured_record_top_level_output_record_is_meaningful() -> None:+ result = _structured_record_qa_top_level_output_run_result()+ ctx = _ctx(result["data"]["blocks"])++ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)++ assert empty_data_blocks is False+ assert _is_outcome_evidence_candidate(ctx, result) is True+++async def test_registered_output_adapter_fetches_db_values_and_synthesizes_block(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ from skyvern.forge.sdk.copilot.tools import run_execution++ async def fake_get_workflow_run_output_parameters(*, workflow_run_id: str) -> list[SimpleNamespace]:+ assert workflow_run_id == "wr_test"+ return [+ SimpleNamespace(+ workflow_run_id="wr_test",+ output_parameter_id="op_details",+ value={+ "entity_name": "Jordan Example",+ "record_number": "1234567890",+ "evidence_text": "Opened Details page",+ },+ )+ ]++ monkeypatch.setattr(+ run_execution.app.DATABASE,+ "workflow_runs",+ SimpleNamespace(get_workflow_run_output_parameters=fake_get_workflow_run_output_parameters),+ )+ workflow = SimpleNamespace(+ workflow_definition=SimpleNamespace(+ blocks=[+ SimpleNamespace(+ label="extract_record_status_details",+ block_type="CODE",+ output_parameter=SimpleNamespace(+ output_parameter_id="op_details",+ key="extract_record_status_details_output",+ ),+ )+ ]+ )+ )+ data: dict[str, Any] = {"workflow_run_id": "wr_test", "blocks": []}++ by_label = await _attach_registered_output_parameter_values(+ workflow_run_id="wr_test",+ workflow=workflow, # type: ignore[arg-type]+ data=data,+ )++ assert by_label == {+ "extract_record_status_details": {+ "extract_record_status_details_output": {+ "entity_name": "Jordan Example",+ "record_number": "1234567890",+ "evidence_text": "Opened Details page",+ }+ }+ }+ assert data["registered_output_parameter_values"][0]["value"]["record_number"] == "1234567890"+ assert data["blocks"][0]["label"] == "extract_record_status_details"+ assert data["blocks"][0]["extracted_data"]["extract_record_status_details_output"]["record_number"] == "1234567890"+++def test_satisfied_completion_prevents_empty_output_suspicious_success() -> None:+ result = _empty_extraction_run_result()+ ctx = _ctx(result["data"]["blocks"])++ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)+ assert empty_data_blocks is True++ _record_run_blocks_result(ctx, result, completion_verification=_satisfied("c0"))++ assert ctx.last_test_ok is True+ assert ctx.last_test_suspicious_success is False+ assert ctx.last_test_failure_reason is None+ assert verified_goal_satisfied_context(ctx) is True++def test_all_null_metadata_goal_fields_are_flagged_as_no_goal_content() -> None:result = _all_null_goal_fields_run_result()ctx = _ctx(result["data"]["blocks"])diff --git a/tests/unit/test_copilot_timeout_wip.py b/tests/unit/test_copilot_timeout_wip.pyindex a19100e72..8645079b1 100644--- a/tests/unit/test_copilot_timeout_wip.py+++ b/tests/unit/test_copilot_timeout_wip.py@@ -22,6 +22,7 @@ from skyvern.forge.sdk.copilot.agent import (_build_timeout_exit_result,_build_unexpected_error_exit_result,)+from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdictfrom skyvern.forge.sdk.copilot.diagnosis_repair_contract import (DiagnosisInput,DiagnosisRepairContract,@@ -152,6 +153,48 @@ class TestBuildTimeoutExitResult:assert result.proposal_disposition == "auto_applicable"assert result.user_response == _TIMEOUT_REPLY_DEFAULT+ def test_verified_terminal_state_preserves_tested_proposal_on_timeout(self) -> None:+ wf = MagicMock(name="wf")+ ctx = _ctx(+ last_workflow=wf,+ last_workflow_yaml="version: '1.0'",+ last_test_ok=None,+ last_test_suspicious_success=True,+ )+ ctx.verified_terminal_proposal_ready = True+ ctx.completion_verification_result = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=["c0"],+ verdicts=[CriterionVerdict(criterion_id="c0", state="satisfied", reason_code="evidence_confirms")],+ )++ result = _build_timeout_exit_result(ctx, global_llm_context=None)++ assert result.updated_workflow is wf+ assert result.workflow_yaml == "version: '1.0'"+ assert result.proposal_disposition == "auto_applicable"+ assert result.user_response == _TIMEOUT_REPLY_TESTED+ assert result.clear_proposed_workflow is False++ def test_suspicious_current_run_drops_last_good_workflow_without_verified_terminal_state(self) -> None:+ wf = MagicMock(name="wf")+ last_good = MagicMock(name="last_good")+ ctx = _ctx(+ last_workflow=wf,+ last_workflow_yaml="version: 'broken'",+ last_test_ok=None,+ last_test_suspicious_success=True,+ last_good_workflow=last_good,+ last_good_workflow_yaml="version: 'tested'",+ )++ result = _build_timeout_exit_result(ctx, global_llm_context=None)++ assert result.updated_workflow is None+ assert result.workflow_yaml is None+ assert result.proposal_disposition == "auto_applicable"+ assert result.user_response == _TIMEOUT_REPLY_DEFAULT+class TestBuildMaxTurnsExitResult:def test_no_workflow_falls_back_to_default_reply(self) -> None:diff --git a/tests/unit/test_workflow_copilot_cancel.py b/tests/unit/test_workflow_copilot_cancel.pyindex 43a166c6e..14fb9542c 100644--- a/tests/unit/test_workflow_copilot_cancel.py+++ b/tests/unit/test_workflow_copilot_cancel.py@@ -37,6 +37,7 @@ from skyvern.forge.sdk.routes.workflow_copilot import (COPILOT_CANCEL_TTL,_copilot_cancel_key,_persist_cancel_turn,+ _persist_proposed_workflow_state,_watch_for_cancel,workflow_copilot_cancel,workflow_copilot_chat_post,@@ -261,6 +262,42 @@ def _make_original_workflow() -> SimpleNamespace:)+async def test_timeout_wip_persists_proposed_workflow_with_yaml_when_canonical_restored() -> None:+ workflow_params = SimpleNamespace(update_workflow_copilot_chat=AsyncMock())+ app.DATABASE.workflow_params = workflow_params+ chat = _make_chat(auto_accept=False)+ updated_workflow = MagicMock()+ updated_workflow.model_dump.return_value = {+ "workflow_id": "wf-canonical",+ "workflow_definition": {+ "parameters": [{"key": "full_name", "parameter_type": "workflow"}],+ "blocks": [{"block_type": "code", "label": "extract_record_status_info"}],+ },+ }+ agent_result = AgentResult(+ user_response="Timed out, but I have a tested draft.",+ updated_workflow=updated_workflow,+ global_llm_context=None,+ workflow_yaml=(+ "title: Record lookup\n"+ "workflow_definition:\n"+ " blocks:\n"+ " - block_type: code\n"+ " label: extract_record_status_info\n"+ ),+ proposal_disposition="review_tested",+ )++ await _persist_proposed_workflow_state(chat, agent_result, restored=True)++ workflow_params.update_workflow_copilot_chat.assert_awaited_once()+ proposed_workflow = workflow_params.update_workflow_copilot_chat.await_args.kwargs["proposed_workflow"]+ assert proposed_workflow["workflow_definition"]["blocks"][0]["label"] == "extract_record_status_info"+ assert proposed_workflow["_copilot_yaml"].startswith("title: Record lookup")+ assert "_copilot_unvalidated" not in proposed_workflow++async def _drive_cancel_route(monkeypatch: pytest.MonkeyPatch,chat: SimpleNamespace,@@ -665,6 +702,57 @@ async def test_timeout_wip_result_streams_normal_response_frame(assert "Cancelled by user." not in contents+async def test_verified_terminal_timeout_result_persists_proposed_workflow_with_blocks_and_params() -> None:+ workflow_params = SimpleNamespace(update_workflow_copilot_chat=AsyncMock())+ app.DATABASE.workflow_params = workflow_params+ chat = _make_chat(proposed_workflow=None, auto_accept=False)+ blocks = [+ {"block_type": "code", "label": "open_search_search_page"},+ {"block_type": "code", "label": "search_and_open_record_details"},+ {"block_type": "code", "label": "extract_record_status_record"},+ ]+ params = [{"key": f"param_{index}", "parameter_type": "workflow"} for index in range(6)]+ updated_workflow = MagicMock()+ updated_workflow.model_dump.side_effect = lambda mode="json": {+ "workflow_id": "wf-draft",+ "title": "Record Status Draft",+ "workflow_definition": {"blocks": blocks, "parameters": params},+ }+ agent_result = AgentResult(+ user_response="I created and tested the workflow successfully.",+ updated_workflow=updated_workflow,+ global_llm_context=None,+ workflow_yaml=(+ "title: Record Status Draft\n"+ "workflow_definition:\n"+ " parameters:\n"+ " - key: param_0\n"+ " blocks:\n"+ " - block_type: code\n"+ " label: open_search_search_page\n"+ " - block_type: code\n"+ " label: search_and_open_record_details\n"+ " - block_type: code\n"+ " label: extract_record_status_record\n"+ ),+ workflow_was_persisted=True,+ proposal_disposition="auto_applicable",+ )++ await _persist_proposed_workflow_state(chat, agent_result, restored=False)++ proposal = workflow_params.update_workflow_copilot_chat.await_args.kwargs["proposed_workflow"]+ assert proposal["_copilot_yaml"].startswith("title: Record Status Draft")+ assert [block["label"] for block in proposal["workflow_definition"]["blocks"]] == [+ "open_search_search_page",+ "search_and_open_record_details",+ "extract_record_status_record",+ ]+ assert len(proposal["workflow_definition"]["parameters"]) == 6+ assert "_copilot_unvalidated" not in proposal++@pytest.mark.asyncioasync def test_timeout_wip_review_tested_propagates_to_response_frame(monkeypatch: pytest.MonkeyPatch,