Skip to content

Automate browser based workflows with AI

SKY-11182: verify structured run outputs before terminal success (#6751)

Andrew Neilson98e769593edb
diff --git a/skyvern/client/core/pydantic_utilities.py b/skyvern/client/core/pydantic_utilities.py
index 8906cdfa4..b92e0b67a 100644
--- a/skyvern/client/core/pydantic_utilities.py
+++ b/skyvern/client/core/pydantic_utilities.py
@@ -206,7 +206,15 @@ def encode_by_type(o: Any) -> Any:
def update_forward_refs(model: Type["Model"], **localns: Any) -> None:
if IS_PYDANTIC_V2:
- model.model_rebuild(raise_errors=False) # type: ignore[attr-defined]
+ try:
+ model.model_rebuild(raise_errors=False) # type: ignore[attr-defined]
+ except KeyError as exc:
+ # Pydantic v2 can still raise internal schema-gathering KeyErrors
+ # for Fern-generated recursive unions even with raise_errors=False.
+ # Match on the "definitions" key rather than the exact args tuple so a
+ # Pydantic format change that adds context can't reintroduce the crash.
+ if "definitions" not in exc.args:
+ raise
else:
model.update_forward_refs(**localns)
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index de9731f8e..b5514cee6 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -160,6 +160,8 @@ IMPORTANT WORKFLOW RULES:
* Commit a draft workflow as soon as you have observed every page its blocks act on — not as soon as a reusable block sequence merely looks plausible. Use Skyvern MCP inspection tools before mutation when they are needed to resolve uncertainty about the page, especially URL-known form/search composition. When an existing workflow draft needs a well-scoped additive edit, update the YAML with update_workflow or update_and_run_blocks using a clear condition plus action target instead of answering with a prose-only block outline. Do not ask for confirmation when the workflow can be updated and tested.
* Scout each page you will act on with the fast browser tools before you author the blocks that act on it. Drive the live browser through the goal path (navigate_browser, click, type_text, scroll, select_option, press_key) and call `inspect_page_for_composition` on every page you will act on — including pages reached only by clicking and pages behind a login — BEFORE authoring those blocks. `inspect_page_for_composition(target_url=...)` navigates to that URL itself and reports the reached `current_url`, so once you have inspected a URL you are already on it — do NOT then call `navigate_browser` to a URL you just inspected. A control or state that appears only after an interaction (a Delete control after an Add click, a cart after add-to-cart, the secure area after login) must be observed live by performing that interaction first; do not assume the post-interaction page from the page before it. Block-runs validate a finished workflow — they do not discover an unknown route, so do not run blocks just to see what a page looks like; for a login wall, author and run the `login` block (referencing a saved credential by id), then resume scouting from the authenticated page.
* Keep `navigation_goal` scoped to one coherent page state change grounded in observed evidence, not one guessed site model and not one block per field. Do not pre-author row/detail expansion from the user's words alone: first run or inspect your way to the actual results page, then use live result-page evidence to decide whether expansion controls exist. If requested details are already visible or no explicit expansion affordance is observed, skip the expansion block and go straight to an `extraction` block. The final stage of visible results -> extract should still use an `extraction` block for data capture, not another `navigation` block. Prefer the most specific block that fits the step — use a more specific block (`action` for a single action, `login` for authentication, `file_download` for downloads, `extraction` for data) over `navigation`, and reach for `navigation` only when the step is best expressed as a goal for the agent to work out rather than a specific named action (see CHOOSING A BLOCK in the knowledge base).
+* When extracting table-like or list-like records (locations, affiliations, prices, inventory, line items, statuses), iterate the actual row/item elements (`tr`, `[role="row"]`, or equivalent repeated item containers) and read fields from that same row/item. Do NOT scan broad containers such as `section`, `.card`, `article`, or `li` and then infer fields from mixed text; that can combine several rows into one record. Derive summary fields only from the parsed row objects, not from broad page text.
+* When a successful search/open-details block reaches a details or profile page, include the visible details/profile page label in the block's returned `evidence_text`. Keep structured record fields focused on the requested data; use the page label only as supporting evidence.
* Save the complete reusable workflow, but run only a small verified frontier when the workflow has several page-changing stages. `update_and_run_blocks` accepts the full YAML plus a subset of `block_labels`; for form/search workflows, first run the entry/form-state frontier, then continue with submit/results, then expansion/extraction. After a verified prefix or a budgeted run that leaves the browser on a later page, preserve the upstream workflow text and run only the missing downstream labels. Do not remove later blocks from the YAML just to shorten a test; do not submit a long chain of all blocks at once when each stage can be verified independently.
* Set `complete_criterion` on every decomposed `navigation` block to the observable post-action page state — without it, smaller blocks still return vague "continue" verdicts.
* Use descriptive, unique labels for blocks (snake_case format).
@@ -226,6 +228,7 @@ RESPONSE-TYPE CLASSIFICATION:
- If your `user_response` asks the user to provide, choose, confirm, or correct information required before you can continue, you MUST use ASK_QUESTION (Option 2). This holds even when this turn built or tested a partial workflow. Choosing `goal_reached: false` on a REPLY does NOT substitute for ASK_QUESTION when input is required.
- A REPLY that carries `updated_workflow` can surface an Accept/Reject proposal panel for the verified workflow. Pairing that panel with a question awaiting required user input is contradictory UX — the user does not know whether to type an answer or click Accept.
- Classify by intent, not punctuation: if the answer would decide what block, value, credential, URL, selector, or correction to use next AND that information is not plainly resolvable from chat_history, the current workflow_yaml, or global_llm_context, choose ASK_QUESTION — even if phrased softly (e.g. "Does that match what you wanted?"). Courtesy questions on a complete verified proposal that the user can ignore without blocking your next turn — you would proceed identically whether they answer or not — are NOT ASK_QUESTION triggers.
+- If the user already specified the target record identity, required identifier, grouped rows/items with statuses, summary status, and no-results behavior, do NOT ask them to confirm output fields or schema; author and test the workflow.
- ASK_QUESTION may still include `workflow_state` in `global_llm_context` to record the partial draft you built; a populated `workflow_state` does NOT imply REPLY.
When you are done using tools and ready to respond to the user, output a JSON object (and nothing else) with ONE of these actions:
diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.py
index 0c9288c1e..52082b2d0 100644
--- a/skyvern/forge/sdk/copilot/agent.py
+++ b/skyvern/forge/sdk/copilot/agent.py
@@ -1284,7 +1284,7 @@ async def _build_goal_satisfied_exit_result(ctx: CopilotContext, global_llm_cont
verified_workflow, verified_yaml = _verified_workflow_or_none(ctx)
clean_test = ctx.last_test_ok is True and ctx.last_full_workflow_test_ok is True
if clean_test and verified_goal_claim_authorized(ctx):
- user_response = "I created and tested the workflow successfully."
+ user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY
elif clean_test:
user_response = (
"I built the workflow and the test run completed, but the goal outcome was not "
@@ -1411,6 +1411,7 @@ _MAX_TURNS_REPLY_UNVALIDATED = (
_MAX_TURNS_REPLY_TESTED = (
"I've reached the maximum number of steps, but I have a tested draft for you. Accept it to save, or discard."
)
+_VERIFIED_WORKFLOW_SUCCESS_REPLY = "I created and tested the workflow successfully."
_UNEXPECTED_ERROR_REPLY_UNVALIDATED = (
"I hit an unexpected issue before I could finish testing. I have a draft workflow you can keep — "
"accept it to save (note: it hasn't been verified end-to-end), or discard."
@@ -1514,9 +1515,7 @@ def _render_typed_run_outcome_reply(
if not has_verified_workflow or not verified_goal_claim_authorized(ctx):
return None
return _TypedRunOutcomeReply(
- user_response=(
- "I created and tested the workflow successfully. The latest run demonstrated the requested outcome."
- ),
+ user_response=f"{_VERIFIED_WORKFLOW_SUCCESS_REPLY} The latest run demonstrated the requested outcome.",
demonstrated=True,
)
@@ -1973,6 +1972,38 @@ def _build_wip_exit_result(
terminal_reason=effective_terminal,
)
+ verified_workflow, verified_yaml = _verified_workflow_or_none(ctx)
+ if ctx.verified_terminal_proposal_ready is True and verified_workflow is not None:
+ final_text, outcome = _guard(tested_reply)
+ return _finalize_result_with_blocker_override(
+ ctx,
+ _make_agent_result(
+ ctx,
+ user_response=final_text,
+ updated_workflow=verified_workflow,
+ global_llm_context=global_llm_context,
+ workflow_yaml=verified_yaml,
+ workflow_was_persisted=ctx.workflow_persisted,
+ has_staged_proposal=ctx.has_staged_proposal,
+ staged_workflow_yaml=ctx.staged_workflow_yaml,
+ staged_workflow=ctx.staged_workflow,
+ canonical_was_persisted_due_to_param_change=ctx.canonical_was_persisted_due_to_param_change,
+ total_tokens=ctx.total_tokens_used,
+ proposal_disposition="auto_applicable",
+ cancelled=cancelled,
+ turn_outcome=outcome,
+ turn_id=ctx.turn_id,
+ narrative_summary=ctx.narrative_summary,
+ narrative_payload=_build_narrative_payload(
+ ctx,
+ terminal="response",
+ terminal_message=final_text,
+ narrative_summary=ctx.narrative_summary,
+ ),
+ ),
+ exit_site="wip_verified_terminal_proposal",
+ )
+
# When an unverified edit/run has overwritten ``last_workflow``, prefer the
# verified shape while still forcing explicit review.
if (
@@ -2356,10 +2387,21 @@ async def _translate_to_agent_result(
elif not salvaged_reply:
user_response = _rewrite_failed_test_response(str(user_response), ctx)
verified_workflow, verified_yaml = _verified_workflow_or_none(ctx)
- # Default-true preserves backwards-compat with stale prompts and missing fields.
- agent_admits_incomplete = _is_explicit_false(
- action_data.get("goal_reached")
- ) and not verified_goal_claim_authorized(ctx)
+ verified_terminal_ready = (
+ ctx.verified_terminal_proposal_ready is True
+ and verified_workflow is not None
+ and verified_goal_claim_authorized(ctx)
+ and not blocker_active
+ )
+ if verified_terminal_ready:
+ resp_type = "REPLY"
+ user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY
+ agent_admits_incomplete = False
+ else:
+ # Default-true preserves backwards-compat with stale prompts and missing fields.
+ agent_admits_incomplete = _is_explicit_false(
+ action_data.get("goal_reached")
+ ) and not verified_goal_claim_authorized(ctx)
typed_outcome_reply = _render_typed_run_outcome_reply(
ctx,
response_type=resp_type,
diff --git a/skyvern/forge/sdk/copilot/completion_verification.py b/skyvern/forge/sdk/copilot/completion_verification.py
index e64affb60..aeda47aca 100644
--- a/skyvern/forge/sdk/copilot/completion_verification.py
+++ b/skyvern/forge/sdk/copilot/completion_verification.py
@@ -13,6 +13,7 @@ from __future__ import annotations
import asyncio
import json
import re
+from collections.abc import Iterable
from dataclasses import dataclass, field
from typing import Any, Literal
@@ -529,6 +530,366 @@ def grade_present_value_criteria(
return verdicts
+def grade_record_semantic_consistency(
+ criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot
+) -> list[CriterionVerdict]:
+ """Deterministically reject internally contradictory structured-record status outputs."""
+
+ criterion = next(iter(_status_consistency_criterion(criteria)), None)
+ if criterion is None:
+ return []
+
+ for label, payload in snapshot.block_outputs.items():
+ contradiction = _structured_record_contradiction(payload)
+ if contradiction:
+ return [
+ CriterionVerdict(
+ criterion_id=criterion.id,
+ state="unsatisfied",
+ reason_code="evidence_contradicts",
+ evidence_ref=f"block_outputs:{label}",
+ missing_evidence=contradiction,
+ )
+ ]
+ return []
+
+
+_STRUCTURED_RECORD_CRITERION_IDS = frozenset(
+ {
+ "fallback_record_identity",
+ "fallback_record_identifier",
+ "fallback_record_groups",
+ "fallback_record_status",
+ }
+)
+
+
+def grade_structured_record_criteria(
+ criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot
+) -> list[CriterionVerdict]:
+ """Deterministically credit generic structured-record fallback criteria.
+
+ Single-block-wins: a criterion set is credited only when one block satisfies it.
+ Verdicts are never merged across blocks because a structured record is a single
+ coherent record, and crediting fields drawn from different blocks could certify a
+ record the run never produced as a whole.
+ """
+
+ criteria_by_id = {
+ criterion.id: criterion for criterion in criteria if criterion.id in _STRUCTURED_RECORD_CRITERION_IDS
+ }
+ if not criteria_by_id:
+ return []
+ best_verdicts: list[CriterionVerdict] = []
+ for label, payload in snapshot.block_outputs.items():
+ record = _structured_record_payload(payload)
+ if record is None:
+ continue
+ verdicts: list[CriterionVerdict] = []
+ if "fallback_record_identity" in criteria_by_id and structured_record_has_identity(record):
+ verdicts.append(_structured_record_satisfied("fallback_record_identity", label))
+ if "fallback_record_identifier" in criteria_by_id and _structured_record_has_identifier(record):
+ verdicts.append(_structured_record_satisfied("fallback_record_identifier", label))
+ if "fallback_record_groups" in criteria_by_id and _structured_record_has_group_entries(record):
+ verdicts.append(_structured_record_satisfied("fallback_record_groups", label))
+ if "fallback_record_status" in criteria_by_id:
+ contradiction = _structured_record_contradiction(record)
+ if contradiction:
+ verdicts.append(
+ CriterionVerdict(
+ criterion_id="fallback_record_status",
+ state="unsatisfied",
+ reason_code="evidence_contradicts",
+ evidence_ref=f"block_outputs:{label}",
+ missing_evidence=contradiction,
+ )
+ )
+ elif _structured_record_has_status(record):
+ verdicts.append(_structured_record_satisfied("fallback_record_status", label))
+ if verdicts:
+ if len(verdicts) == len(criteria_by_id):
+ return verdicts
+ if len(verdicts) > len(best_verdicts):
+ best_verdicts = verdicts
+ return best_verdicts
+
+
+def _structured_record_satisfied(criterion_id: str, label: str) -> CriterionVerdict:
+ return CriterionVerdict(
+ criterion_id=criterion_id,
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref=f"block_outputs:{label}",
+ )
+
+
+def _structured_record_payload(payload: Any) -> dict[str, Any] | None:
+ if not isinstance(payload, dict):
+ return None
+ for key, value in payload.items():
+ if (
+ isinstance(key, str)
+ and key.endswith("_output")
+ and isinstance(value, dict)
+ and _looks_like_structured_record(value)
+ ):
+ return value
+ if _looks_like_structured_record(payload):
+ return payload
+ return None
+
+
+def _looks_like_structured_record(value: dict[str, Any]) -> bool:
+ return (
+ structured_record_has_identity(value)
+ or _structured_record_has_identifier(value)
+ or _structured_record_has_group_entries(value)
+ or _structured_record_has_status(value)
+ )
+
+
+def structured_record_has_identity(record: dict[str, Any]) -> bool:
+ if any(value is True for key, value in record.items() if isinstance(key, str) and key.endswith("_found")):
+ return True
+ return any(
+ isinstance(value, str) and bool(value.strip())
+ for key, value in record.items()
+ if isinstance(key, str) and _key_has_identity_token(key)
+ )
+
+
+def _key_has_identity_token(key: str) -> bool:
+ normalized_key = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", key).casefold().replace("_", " ")
+ return any(
+ re.search(rf"(?:^|[^a-z0-9]){re.escape(token)}(?:$|[^a-z0-9])", normalized_key)
+ for token in ("name", "title", "entity", "label")
+ )
+
+
+def structured_record_has_goal_content(record: dict[str, Any]) -> bool:
+ """Return True only when a structured record has the full terminal-proof shape."""
+
+ if (
+ not structured_record_has_identity(record)
+ or not _structured_record_has_identifier(record)
+ or not _record_summary_status(record)
+ ):
+ return False
+ return any(
+ bool(status.strip())
+ and any(
+ _is_meaningful_record_value(item_value)
+ for item_key, item_value in item.items()
+ if not (isinstance(item_key, str) and "status" in item_key.casefold())
+ )
+ for status, item in _record_row_statuses(record)
+ )
+
+
+def _key_word_tokens(key: str) -> set[str]:
+ """Split a (possibly dotted/camelCase) record key path into whole-word tokens."""
+ spaced = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", key).casefold()
+ return set(re.findall(r"[a-z0-9]+", spaced))
+
+
+def _structured_record_has_identifier(record: dict[str, Any]) -> bool:
+ for key, value in _walk_record_scalars(record):
+ value_text = str(value)
+ digit_runs = "".join(ch if ch.isdigit() else " " for ch in value_text).split()
+ if any(len(run) >= 6 for run in digit_runs):
+ return True
+ if _key_word_tokens(key) & {"identifier", "id", "number"} and value_text.strip():
+ return True
+ return False
+
+
+def _structured_record_has_group_entries(record: dict[str, Any]) -> bool:
+ for value in record.values():
+ if not isinstance(value, list):
+ continue
+ for item in value:
+ if isinstance(item, dict) and any(_is_meaningful_record_value(nested) for nested in item.values()):
+ return True
+ return False
+
+
+def _structured_record_has_status(record: dict[str, Any]) -> bool:
+ summary_status = _record_summary_status(record)
+ row_statuses = _record_row_statuses(record)
+ return bool(summary_status and row_statuses)
+
+
+def _status_consistency_criterion(criteria: list[CompletionCriterion]) -> Iterable[CompletionCriterion]:
+ for criterion in criteria:
+ normalized = _normalize_present_value(criterion.outcome)
+ if "status" in normalized and (
+ "consistent" in normalized or "overall" in normalized or "summary" in normalized or "per-" in normalized
+ ):
+ yield criterion
+
+
+def _structured_record_contradiction(payload: Any) -> str | None:
+ if not isinstance(payload, dict):
+ return None
+ if not (_structured_record_has_group_entries(payload) or _structured_record_has_status(payload)):
+ return None
+ summary_status = _record_summary_status(payload)
+ summary_polarity = _status_polarity(summary_status) if summary_status else None
+ row_statuses = _record_row_statuses(payload)
+ for status, item in row_statuses:
+ polarity = _status_polarity(status)
+ if polarity is None:
+ continue
+ is_negative, positive_phrase = polarity
+ if not is_negative:
+ continue
+ non_status_text = " ".join(
+ str(value)
+ for key, value in item.items()
+ if not (isinstance(key, str) and "status" in key.casefold()) and value is not None
+ )
+ if _contains_positive_status_phrase(non_status_text, positive_phrase):
+ return (
+ "a parsed row reports a negative status, but non-status fields include the positive status text; "
+ "parse name/address/status from cells in the same row"
+ )
+ if summary_polarity is not None:
+ summary_negative, summary_positive_phrase = summary_polarity
+ if summary_negative and any(
+ (row_polarity := _status_polarity(status)) is not None and row_polarity == (False, summary_positive_phrase)
+ for status, _item in row_statuses
+ ):
+ return "summary status is negative, but a parsed row has the matching positive status"
+
+ evidence_text = payload.get("evidence_text")
+ if isinstance(evidence_text, str) and summary_polarity is not None:
+ summary_negative, summary_positive_phrase = summary_polarity
+ if (
+ summary_negative
+ and row_statuses
+ and not any(_status_polarity(status) == (False, summary_positive_phrase) for status, _item in row_statuses)
+ and _contains_positive_status_phrase(evidence_text, summary_positive_phrase)
+ ):
+ return (
+ "evidence_text contains positive status rows, but structured rows and summary report a negative status"
+ )
+ return None
+
+
+def _record_summary_status(record: dict[str, Any]) -> str | None:
+ for key, value in record.items():
+ if not isinstance(key, str) or not isinstance(value, str):
+ continue
+ normalized_key = key.casefold()
+ if "status" in normalized_key and ("overall" in normalized_key or "summary" in normalized_key):
+ return value
+ value = record.get("status")
+ return value if isinstance(value, str) else None
+
+
+def _record_row_statuses(record: dict[str, Any]) -> list[tuple[str, dict[str, Any]]]:
+ statuses: list[tuple[str, dict[str, Any]]] = []
+ for value in record.values():
+ if not isinstance(value, list):
+ continue
+ for item in value:
+ if not isinstance(item, dict):
+ continue
+ for item_key, item_value in item.items():
+ if isinstance(item_key, str) and "status" in item_key.casefold() and isinstance(item_value, str):
+ statuses.append((item_value, item))
+ return statuses
+
+
+_NEGATIVE_STATUS_POSITIVE_COUNTERPARTS = {
+ "expired": "active",
+ "inactive": "active",
+ "lapsed": "active",
+ "pending": "active",
+ "revoked": "active",
+ "suspended": "active",
+ "terminated": "active",
+}
+
+
+def _status_polarity(value: str) -> tuple[bool, str] | None:
+ normalized = _normalize_present_value(value)
+ if not normalized:
+ return None
+ if normalized in _NEGATIVE_STATUS_POSITIVE_COUNTERPARTS:
+ return True, _NEGATIVE_STATUS_POSITIVE_COUNTERPARTS[normalized]
+ for prefix in ("not ", "non-", "non "):
+ if normalized.startswith(prefix):
+ positive_phrase = normalized[len(prefix) :].strip()
+ return (True, positive_phrase) if positive_phrase else None
+ return False, normalized
+
+
+# Negators, temporal qualifiers, and negative-status words that, near a positive-status
+# word, flip it into a negative or adjectival usage ("no longer active", "previously
+# active", "the active license expired") rather than a positive status assertion.
+_STATUS_NEGATION_QUALIFIERS = frozenset(
+ {
+ "not",
+ "non",
+ "no",
+ "longer",
+ "previously",
+ "formerly",
+ "once",
+ "currently",
+ "recently",
+ "was",
+ "were",
+ "until",
+ "never",
+ *_NEGATIVE_STATUS_POSITIVE_COUNTERPARTS,
+ }
+)
+
+
+def _status_word_tokens(text: str) -> list[str]:
+ return "".join(char if char.isalnum() else " " for char in text.casefold()).split()
+
+
+def _contains_positive_status_phrase(text: str, positive_phrase: str) -> bool:
+ tokens = _status_word_tokens(text)
+ phrase_tokens = _status_word_tokens(positive_phrase)
+ if not tokens or not phrase_tokens:
+ return False
+ span = len(phrase_tokens)
+ for start in range(len(tokens) - span + 1):
+ if tokens[start : start + span] != phrase_tokens:
+ continue
+ window = set(tokens[max(0, start - 3) : start]) | set(tokens[start + span : start + span + 3])
+ if not window & _STATUS_NEGATION_QUALIFIERS:
+ return True
+ return False
+
+
+def _walk_record_scalars(value: Any, prefix: str = "") -> Iterable[tuple[str, Any]]:
+ if isinstance(value, dict):
+ for key, nested in value.items():
+ key_text = str(key)
+ nested_prefix = f"{prefix}.{key_text}" if prefix else key_text
+ yield from _walk_record_scalars(nested, nested_prefix)
+ elif isinstance(value, list):
+ for index, nested in enumerate(value):
+ yield from _walk_record_scalars(nested, f"{prefix}[{index}]")
+ elif value is not None:
+ yield prefix, value
+
+
+def _is_meaningful_record_value(value: Any) -> bool:
+ if value is None:
+ return False
+ if isinstance(value, str):
+ return bool(value.strip())
+ if isinstance(value, (list, tuple, set, dict)):
+ return bool(value)
+ return True
+
+
_DEFINITION_REASON_PREFIX = "definition_"
diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.py
index 8a37311c8..de9a91266 100644
--- a/skyvern/forge/sdk/copilot/enforcement.py
+++ b/skyvern/forge/sdk/copilot/enforcement.py
@@ -1270,6 +1270,9 @@ def _check_enforcement(
raise_if_turn_halt(ctx)
_raise_if_unrecoverable_contract_stop(ctx)
+ if getattr(ctx, "verified_terminal_proposal_ready", False) is True and verified_goal_satisfied_context(ctx):
+ raise CopilotGoalSatisfied()
+
if _needs_repair_ceiling_halt(ctx):
contract = getattr(ctx, "latest_diagnosis_repair_contract", None)
state = _repair_loop_state(ctx)
diff --git a/skyvern/forge/sdk/copilot/output_policy.py b/skyvern/forge/sdk/copilot/output_policy.py
index c10fb6120..7b9c225d2 100644
--- a/skyvern/forge/sdk/copilot/output_policy.py
+++ b/skyvern/forge/sdk/copilot/output_policy.py
@@ -149,6 +149,15 @@ _SELF_PRESCRIPTIVE_CONTINUATION_RE = re.compile(
r")\b",
re.IGNORECASE,
)
+_OUTPUT_FIELD_CONFIRMATION_RE = re.compile(
+ r"(?:"
+ r"\b(?:confirm|verify|approve|check|tell me|let me know)\b"
+ r"(?=[\s\S]{0,180}\b(?:output|record|schema|field|fields)\b)"
+ r"(?=[\s\S]{0,220}\b(?:field|fields|schema|record)\b)"
+ r"|\b(?:which|what)\s+fields\s+should\b"
+ r")",
+ re.IGNORECASE,
+)
# Response types whose `user_response` is rendered verbatim as the agent's final
# message — REPLY and REPLACE_WORKFLOW. ASK_QUESTION is excluded because legitimate
@@ -187,6 +196,7 @@ class OutputPolicyReason(StrEnum):
INTERNAL_CLASSIFIER_VOCAB_LEAK = "internal_classifier_vocab_leak"
SELF_PRESCRIPTIVE_PHRASE_LEAK = "self_prescriptive_phrase_leak"
WORKFLOW_YAML_IN_REPLY = "workflow_yaml_in_reply"
+ AVOIDABLE_OUTPUT_FIELD_CONFIRMATION = "avoidable_output_field_confirmation"
@dataclass
@@ -219,6 +229,7 @@ _FINAL_OUTPUT_HARD_BLOCK_REASONS: frozenset[OutputPolicyReason] = frozenset(
OutputPolicyReason.PERSISTENCE_STATE_MISMATCH,
OutputPolicyReason.INTERNAL_TOOL_INSTRUCTION_LEAK,
OutputPolicyReason.OUTPUT_POLICY_CONTEXT_MISSING,
+ OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION,
}
)
@@ -422,6 +433,15 @@ def evaluate_output_policy(
if isinstance(request_policy, RequestPolicy):
if request_policy.user_response_policy == "ask_clarification" and response_type != "ASK_QUESTION":
verdict.add(OutputPolicyReason.REQUEST_POLICY_CLARIFICATION_BYPASS)
+ if (
+ response_type == "ASK_QUESTION"
+ and request_policy.user_response_policy != "ask_clarification"
+ and _request_policy_has_present_completion_contract(request_policy)
+ and not has_workflow_proposal
+ and not workflow_attempted
+ and _asks_to_confirm_output_fields(user_response)
+ ):
+ verdict.add(OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION)
_apply_credential_policy(verdict, request_policy, values, workflow_yaml)
if output_kind == CopilotOutputKind.WORKFLOW_UPDATE_PROPOSAL and not workflow_was_persisted:
@@ -432,6 +452,16 @@ def evaluate_output_policy(
return verdict
+def _request_policy_has_present_completion_contract(request_policy: RequestPolicy) -> bool:
+ return request_policy.completion_contract_status == "present" or bool(request_policy.completion_criteria)
+
+
+def _asks_to_confirm_output_fields(user_response: str | None) -> bool:
+ if not isinstance(user_response, str) or not user_response.strip():
+ return False
+ return bool(_OUTPUT_FIELD_CONFIRMATION_RE.search(user_response[:500]))
+
+
def format_output_policy_tool_error(verdict: OutputPolicyVerdict) -> str:
reasons = ", ".join(reason.value for reason in verdict.reason_codes) or "unknown"
message = f"Output policy blocked this Copilot output before persistence. Reason codes: {reasons}."
diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.py
index 5acd8a8f9..67ee13660 100644
--- a/skyvern/forge/sdk/copilot/request_policy.py
+++ b/skyvern/forge/sdk/copilot/request_policy.py
@@ -567,6 +567,7 @@ def _classifier_fallback_policy(
raw_secret_present: bool,
failure_kind: str,
retry_count: int = 0,
+ user_message: str = "",
) -> RequestPolicy:
if failure_kind not in _CLASSIFIER_FAILURE_KINDS:
failure_kind = "provider_error"
@@ -582,16 +583,81 @@ def _classifier_fallback_policy(
classifier_retry_count=retry_count,
completion_contract_status="unknown",
)
+ fallback_criteria = _fallback_structured_record_completion_criteria(user_message)
+ if fallback_criteria:
+ LOG.info(
+ "copilot request policy synthesized fallback structured-record criteria",
+ classifier_failure_kind=failure_kind,
+ completion_criterion_ids=[criterion.id for criterion in fallback_criteria],
+ )
return RequestPolicy(
credential_input_kind="credential_id" if ids else "none",
credential_refs=ids,
+ completion_criteria=fallback_criteria,
classifier_status="fallback",
classifier_failure_kind=failure_kind,
classifier_retry_count=retry_count,
- completion_contract_status="unknown",
+ completion_contract_status="present" if fallback_criteria else "unknown",
)
+def _word_tokens(text: str) -> list[str]:
+ return "".join(char if char.isalnum() else " " for char in text.casefold()).split()
+
+
+def _fallback_structured_record_completion_criteria(user_message: str) -> list[CompletionCriterion]:
+ """Conservative structured-record criteria when the request-policy classifier is unavailable."""
+
+ words = _word_tokens(user_message)
+ if not words:
+ return []
+ haystack = f" {' '.join(words)} "
+
+ def _mentions(*terms: str) -> bool:
+ # Whole-word/phrase match so "id" can't fire on "consider" nor "name" on "filename".
+ return any(f" {' '.join(_word_tokens(term))} " in haystack for term in terms)
+
+ has_long_number = any(word.isdigit() and len(word) >= 6 for word in words)
+ if not _mentions("return", "record", "capture", "read", "extract"):
+ return []
+ if not _mentions("name", "entity", "person"):
+ return []
+ if not _mentions("identifier", "id", "number") and not has_long_number:
+ return []
+ if not _mentions("status"):
+ return []
+ if not _mentions("locations", "items", "rows", "groups", "entries", "per location", "per item", "per row"):
+ return []
+ return [
+ CompletionCriterion(
+ id="fallback_record_identity",
+ outcome="The returned record identifies the target entity.",
+ implicit=True,
+ level="run",
+ ),
+ CompletionCriterion(
+ id="fallback_record_identifier",
+ outcome="The returned record includes the requested identifier-like value.",
+ implicit=True,
+ level="run",
+ ),
+ CompletionCriterion(
+ id="fallback_record_groups",
+ outcome="The returned record includes the requested grouped row entries.",
+ implicit=True,
+ level="run",
+ ),
+ CompletionCriterion(
+ id="fallback_record_status",
+ outcome=(
+ "The returned record's per-row statuses and summary status are present and internally consistent."
+ ),
+ implicit=True,
+ level="run",
+ ),
+ ]
+
+
def _explicit_code_block_credential_draft_requested(user_message: str) -> bool:
normalized = " ".join((user_message or "").lower().split())
if not normalized:
@@ -656,7 +722,12 @@ async def _classify_request(
ids = _credential_ids(user_message)
raw_secret_present = _raw_secret_detected(user_message)
if raw_secret_present and handler is None:
- return _classifier_fallback_policy(ids, raw_secret_present=True, failure_kind="raw_secret_no_handler")
+ return _classifier_fallback_policy(
+ ids,
+ raw_secret_present=True,
+ failure_kind="raw_secret_no_handler",
+ user_message=user_message,
+ )
structural_reason = _structural_clarification_reason(user_message)
if structural_reason != "none" and not raw_secret_present:
return RequestPolicy(
@@ -666,7 +737,12 @@ async def _classify_request(
clarification_reason=structural_reason,
)
if handler is None:
- return _classifier_fallback_policy(ids, raw_secret_present=False, failure_kind="missing_handler")
+ return _classifier_fallback_policy(
+ ids,
+ raw_secret_present=False,
+ failure_kind="missing_handler",
+ user_message=user_message,
+ )
# Raw-secret turns intentionally reach the classifier when available so it
# can distinguish unsafe secret use from redacted draft/spec conversion.
@@ -693,6 +769,7 @@ async def _classify_request(
raw_secret_present=raw_secret_present,
failure_kind=failure_kind,
retry_count=retry_count,
+ user_message=user_message,
)
raw_payload = _coerce_classifier_payload(raw)
@@ -703,6 +780,7 @@ async def _classify_request(
raw_secret_present=raw_secret_present,
failure_kind="provider_error",
retry_count=retry_count,
+ user_message=user_message,
)
policy = _classification_from_raw(raw_payload)
diff --git a/skyvern/forge/sdk/copilot/runtime.py b/skyvern/forge/sdk/copilot/runtime.py
index c4c6ef691..f2c121b31 100644
--- a/skyvern/forge/sdk/copilot/runtime.py
+++ b/skyvern/forge/sdk/copilot/runtime.py
@@ -234,6 +234,7 @@ class AgentContext:
last_run_outcome: RecordedRunOutcome | None = None
last_run_outcome_block_labels: list[str] = field(default_factory=list)
completion_verification_result: CompletionVerificationResult | None = None
+ verified_terminal_proposal_ready: bool = False
outcome_verification_trace_snapshot: dict[str, Any] = field(default_factory=dict)
composition_page_evidence: dict[str, Any] | None = None
# Ordered, bounded list of typed page-evidence packets — one per page observed
diff --git a/skyvern/forge/sdk/copilot/tools/_shared.py b/skyvern/forge/sdk/copilot/tools/_shared.py
index e106bade4..a25f6baa7 100644
--- a/skyvern/forge/sdk/copilot/tools/_shared.py
+++ b/skyvern/forge/sdk/copilot/tools/_shared.py
@@ -182,6 +182,41 @@ _TASK_OUTPUT_PAYLOAD_FIELDS: tuple[str, ...] = (
"downloaded_files",
"downloaded_file_urls",
)
+_TASK_OUTPUT_PARAMETER_SUFFIX = "_output"
+
+
+def _workflow_output_parameter_payloads(extracted_data: Any) -> dict[str, Any]:
+ """Return workflow output-parameter values embedded in a block output."""
+ if not isinstance(extracted_data, dict):
+ return {}
+ return {
+ key: value
+ for key, value in extracted_data.items()
+ if isinstance(key, str) and key.endswith(_TASK_OUTPUT_PARAMETER_SUFFIX)
+ }
+
+
+def _registered_output_parameter_payloads(data: Mapping[str, Any]) -> list[Mapping[str, Any]]:
+ """``workflow_run_output_parameters`` is the runtime's authoritative persisted
+ value surface. Keep this scoped to the run result carrying it so prior-run
+ accumulated outputs cannot satisfy the current run's completion contract.
+ """
+ run_id = data.get("workflow_run_id")
+ registered = data.get("registered_output_parameter_values")
+ if not isinstance(registered, list):
+ return []
+ if not isinstance(run_id, str):
+ # Without a current run id the payloads can't be attributed to this run;
+ # fail closed so prior-run accumulated outputs can't satisfy its contract.
+ return []
+ payloads: list[Mapping[str, Any]] = []
+ for item in registered:
+ if not isinstance(item, Mapping):
+ continue
+ if item.get("workflow_run_id") != run_id:
+ continue
+ payloads.append(item)
+ return payloads
def _block_data_payload(extracted_data: Any, block_type: str | None) -> Any:
@@ -196,7 +231,9 @@ def _block_data_payload(extracted_data: Any, block_type: str | None) -> Any:
json_schema that happens to include an ``extracted_information`` field.
"""
if block_type in _TASK_ENVELOPE_BLOCK_TYPES and isinstance(extracted_data, dict):
- return {field: extracted_data.get(field) for field in _TASK_OUTPUT_PAYLOAD_FIELDS}
+ payload = {field: extracted_data.get(field) for field in _TASK_OUTPUT_PAYLOAD_FIELDS}
+ payload.update(_workflow_output_parameter_payloads(extracted_data))
+ return payload
return extracted_data
diff --git a/skyvern/forge/sdk/copilot/tools/blockers.py b/skyvern/forge/sdk/copilot/tools/blockers.py
index 2e232e556..61859d8ee 100644
--- a/skyvern/forge/sdk/copilot/tools/blockers.py
+++ b/skyvern/forge/sdk/copilot/tools/blockers.py
@@ -17,6 +17,9 @@ from skyvern.forge.sdk.copilot.blocker_signal import (
loop_blocker_evidence_from_ctx,
refresh_held_loop_blocker_evidence,
)
+from skyvern.forge.sdk.copilot.completion_verification import (
+ structured_record_has_goal_content as _structured_record_candidate_has_goal_content,
+)
from skyvern.forge.sdk.copilot.composition_evidence import interactive_challenge_controls
from skyvern.forge.sdk.copilot.enforcement import (
TOTAL_TIMEOUT_SECONDS,
@@ -52,6 +55,8 @@ from ._shared import (
_is_meaningful_extracted_data,
_parse_workflow_blocks,
_raw_yaml_proxy_location,
+ _registered_output_parameter_payloads,
+ _workflow_output_parameter_payloads,
)
LOG = structlog.get_logger()
@@ -1641,6 +1646,7 @@ def _analyze_run_blocks(
has_data_blocks = False
any_data_output = False
missing_metadata_goal_content = False
+ complete_structured_record_output = False
blocks = data.get("blocks")
if isinstance(blocks, list):
@@ -1675,6 +1681,13 @@ def _analyze_run_blocks(
)
if structured_blocker:
texts_to_scan.append(structured_blocker)
+ output_parameter_payloads = _workflow_output_parameter_payloads(extracted)
+ if output_parameter_payloads:
+ has_data_blocks = True
+ if any(_is_meaningful_extracted_data(value) for value in output_parameter_payloads.values()):
+ any_data_output = True
+ if any(_structured_record_has_goal_content(value) for value in output_parameter_payloads.values()):
+ complete_structured_record_output = True
extraction_schema = _extraction_schema_for_code_block(copilot_ctx, block.get("label"))
# Array-typed schemas would coerce the keyed dict return to [] (fill_missing_fields), making the
# goal-path check below read a real extraction as empty; the keyed-return floor guarantees a dict.
@@ -1687,8 +1700,10 @@ def _analyze_run_blocks(
goal_value_paths = _goal_value_paths_for_code_block(copilot_ctx, block.get("label"))
if goal_value_paths:
has_data_blocks = True
- if _code_output_has_registered_download_content(extracted) or _code_output_goal_paths_have_content(
- extracted, goal_value_paths
+ if (
+ _code_output_has_registered_download_content(extracted)
+ or _code_output_goal_paths_have_content(extracted, goal_value_paths)
+ or _structured_record_has_goal_content(extracted)
):
any_data_output = True
else:
@@ -1706,6 +1721,32 @@ def _analyze_run_blocks(
if _code_output_has_goal_content(extracted, declared_keys=declared_keys):
any_data_output = True
+ top_level_output_payloads = _workflow_output_parameter_payloads(data.get("output"))
+ if top_level_output_payloads:
+ has_data_blocks = True
+ if any(_is_meaningful_extracted_data(value) for value in top_level_output_payloads.values()):
+ any_data_output = True
+ if any(_structured_record_has_goal_content(value) for value in top_level_output_payloads.values()):
+ complete_structured_record_output = True
+
+ registered_payloads = _registered_output_parameter_payloads(data)
+ if registered_payloads:
+ has_data_blocks = True
+ for registered in registered_payloads:
+ value = registered.get("value")
+ if _is_meaningful_extracted_data(value):
+ any_data_output = True
+ if _structured_record_has_goal_content(value):
+ complete_structured_record_output = True
+ structured_blocker = _structured_blocker_message(
+ value,
+ include_flag_keys=True,
+ key_terms=_STRICT_BLOCKER_FLAG_TERMS,
+ scan_all_values_for_anti_bot=True,
+ )
+ if structured_blocker:
+ texts_to_scan.append(structured_blocker)
+
combined = "\n".join(texts_to_scan)
categories = classify_from_failure_reason(combined)
if categories:
@@ -1714,5 +1755,25 @@ def _analyze_run_blocks(
anti_bot_match = cat.get("reasoning", "anti-bot pattern detected")
break
+ if complete_structured_record_output:
+ if missing_metadata_goal_content:
+ LOG.info(
+ "copilot run evidence: a complete structured-record output suppressed a "
+ "per-block missing-metadata-goal-content signal",
+ workflow_run_id=data.get("workflow_run_id"),
+ )
+ missing_metadata_goal_content = False
empty_data_blocks = (has_data_blocks and not any_data_output) or missing_metadata_goal_content
return anti_bot_match, empty_data_blocks, categories
+
+
+def _structured_record_has_goal_content(value: Any) -> bool:
+ if not isinstance(value, dict):
+ return False
+ candidates = [value]
+ candidates.extend(
+ nested
+ for key, nested in value.items()
+ if isinstance(key, str) and key.endswith("_output") and isinstance(nested, dict)
+ )
+ return any(_structured_record_candidate_has_goal_content(candidate) for candidate in candidates)
diff --git a/skyvern/forge/sdk/copilot/tools/completion.py b/skyvern/forge/sdk/copilot/tools/completion.py
index 1f3e24e85..d44016db1 100644
--- a/skyvern/forge/sdk/copilot/tools/completion.py
+++ b/skyvern/forge/sdk/copilot/tools/completion.py
@@ -13,6 +13,8 @@ from skyvern.forge.sdk.copilot.completion_verification import (
evaluate_completion_criteria,
grade_definition_criteria,
grade_present_value_criteria,
+ grade_record_semantic_consistency,
+ grade_structured_record_criteria,
summarize_unsatisfied_outcomes,
verdict_missing_evidence,
)
@@ -31,7 +33,9 @@ from ._shared import (
_current_workflow_has_evidence_block,
_failed_run_block_labels,
_is_meaningful_extracted_data,
+ _registered_output_parameter_payloads,
_valid_runtime_anchor_url,
+ _workflow_output_parameter_payloads,
)
from .blockers import (
_active_run_terminal_evidence_detected,
@@ -373,6 +377,29 @@ def _build_run_evidence_snapshot(copilot_ctx: Any, result: dict[str, Any]) -> Ru
evidence_output = _completion_evidence_payload(output)
if isinstance(label, str) and label in current_labels and _is_meaningful_extracted_data(evidence_output):
block_outputs[label] = evidence_output
+ for output_key, output_value in _workflow_output_parameter_payloads(output).items():
+ if _is_meaningful_extracted_data(output_value):
+ block_outputs[output_key] = output_value
+ for output_key, output_value in _workflow_output_parameter_payloads(data.get("output")).items():
+ if _is_meaningful_extracted_data(output_value):
+ block_outputs[output_key] = output_value
+ for registered in _registered_output_parameter_payloads(data):
+ registered_output_key = registered.get("output_parameter_key")
+ registered_output_value = registered.get("value")
+ registered_block_label = registered.get("block_label")
+ if not _is_meaningful_extracted_data(registered_output_value):
+ continue
+ if isinstance(registered_output_key, str) and registered_output_key:
+ block_outputs[registered_output_key] = registered_output_value
+ if isinstance(registered_block_label, str) and registered_block_label in current_labels:
+ if isinstance(registered_output_key, str) and registered_output_key:
+ existing = block_outputs.get(registered_block_label)
+ if isinstance(existing, dict):
+ existing.setdefault(registered_output_key, registered_output_value)
+ else:
+ block_outputs[registered_block_label] = {registered_output_key: registered_output_value}
+ else:
+ block_outputs[registered_block_label] = registered_output_value
executed = data.get("executed_block_labels")
executed_block_labels = [str(label) for label in executed] if isinstance(executed, list) else []
page_title = data.get("page_title")
@@ -416,19 +443,85 @@ def _apply_present_value_upgrades(
if run_result.status != "evaluated":
return run_result
upgrades = {verdict.criterion_id: verdict for verdict in grade_present_value_criteria(run_criteria, snapshot)}
- if not upgrades:
+ upgrades.update(
+ {verdict.criterion_id: verdict for verdict in grade_structured_record_criteria(run_criteria, snapshot)}
+ )
+ semantic_verdicts = {
+ verdict.criterion_id: verdict for verdict in grade_record_semantic_consistency(run_criteria, snapshot)
+ }
+ if not upgrades and not semantic_verdicts:
return run_result
- verdicts = [
- upgrades[verdict.criterion_id]
- if verdict.criterion_id in upgrades and not verdict.satisfied and verdict.reason_code != "evidence_contradicts"
- else verdict
- for verdict in run_result.verdicts
- ]
+
+ def _merged(verdict: CriterionVerdict) -> CriterionVerdict:
+ # Semantic contradictions are stronger than earlier satisfied verdicts:
+ # a row-mixing/status contradiction means the record evidence is invalid.
+ if verdict.criterion_id in semantic_verdicts:
+ return semantic_verdicts[verdict.criterion_id]
+ if verdict.criterion_id in upgrades and not verdict.satisfied and verdict.reason_code != "evidence_contradicts":
+ return upgrades[verdict.criterion_id]
+ return verdict
+
+ verdicts = [_merged(verdict) for verdict in run_result.verdicts]
return CompletionVerificationResult(
status="evaluated", criterion_ids=list(run_result.criterion_ids), verdicts=verdicts
)
+def _deterministic_run_verification_result(
+ run_criteria: list[CompletionCriterion],
+ snapshot: RunEvidenceSnapshot,
+) -> tuple[CompletionVerificationResult | None, list[CompletionCriterion]]:
+ """Return a deterministic run-plane verdict when the typed graders cover it.
+
+ Provider-record and present-value graders are exact enough to bypass the LLM
+ judge only when they satisfy every run-plane criterion. Contradictions also
+ bypass the judge, because the output is already semantically invalid. Any
+ remaining criterion stays fail-closed through the normal judge path.
+ """
+ criterion_ids = [criterion.id for criterion in run_criteria]
+ semantic_verdicts = grade_record_semantic_consistency(run_criteria, snapshot)
+ if any(not verdict.satisfied for verdict in semantic_verdicts):
+ return (
+ CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=criterion_ids,
+ verdicts=list(semantic_verdicts),
+ ),
+ [],
+ )
+
+ deterministic_by_id: dict[str, CriterionVerdict] = {}
+ for verdict in grade_present_value_criteria(run_criteria, snapshot):
+ deterministic_by_id[verdict.criterion_id] = verdict
+ for verdict in grade_structured_record_criteria(run_criteria, snapshot):
+ deterministic_by_id[verdict.criterion_id] = verdict
+ for verdict in semantic_verdicts:
+ deterministic_by_id[verdict.criterion_id] = verdict
+
+ deterministic_verdicts = [deterministic_by_id.get(criterion_id) for criterion_id in criterion_ids]
+ if criterion_ids and all(verdict is not None and verdict.satisfied for verdict in deterministic_verdicts):
+ return (
+ CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=criterion_ids,
+ verdicts=[deterministic_by_id[criterion_id] for criterion_id in criterion_ids],
+ ),
+ [],
+ )
+
+ remaining_criteria = [criterion for criterion in run_criteria if criterion.id not in deterministic_by_id]
+ if not deterministic_by_id:
+ return None, remaining_criteria
+ return (
+ CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=criterion_ids,
+ verdicts=list(deterministic_by_id.values()),
+ ),
+ remaining_criteria,
+ )
+
+
async def _maybe_run_completion_verification(
copilot_ctx: Any, result: dict[str, Any], handler_start: float
) -> CompletionVerificationResult | None:
@@ -451,16 +544,6 @@ async def _maybe_run_completion_verification(
)
if not run_criteria:
return combine_verification_results(criterion_ids, None, definition_verdicts)
- # A missing judge handler is an infra/config state, not a transient failure:
- # fall back to the prior gate rather than fail closed on every run.
- handler = await _completion_verification_handler(copilot_ctx)
- if handler is None:
- return None
- # Too little budget to verify a candidate run: fail closed (unavailable) rather
- # than let the run-status proxy claim an unverified outcome as success.
- remaining = RUN_BLOCKS_SAFETY_CEILING_SECONDS - (time.monotonic() - handler_start)
- if remaining <= settings.COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS + _COMPLETION_VERIFICATION_BUDGET_MARGIN_SECONDS:
- return CompletionVerificationResult(status="unavailable")
snapshot = _build_run_evidence_snapshot(copilot_ctx, result)
if not snapshot.has_evidence():
run_result = CompletionVerificationResult(
@@ -472,8 +555,35 @@ async def _maybe_run_completion_verification(
],
)
else:
- run_result = await evaluate_completion_criteria(run_criteria, snapshot, handler)
- run_result = _apply_present_value_upgrades(run_result, run_criteria, snapshot)
+ handler = await _completion_verification_handler(copilot_ctx)
+ if handler is None:
+ return CompletionVerificationResult(status="unavailable")
+ # Too little budget to verify a candidate run: fail closed (unavailable)
+ # rather than let the run-status proxy claim an unverified outcome as success.
+ remaining = RUN_BLOCKS_SAFETY_CEILING_SECONDS - (time.monotonic() - handler_start)
+ if (
+ remaining
+ <= settings.COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS + _COMPLETION_VERIFICATION_BUDGET_MARGIN_SECONDS
+ ):
+ return CompletionVerificationResult(status="unavailable")
+ deterministic_result, remaining_criteria = _deterministic_run_verification_result(run_criteria, snapshot)
+ if deterministic_result is not None and not remaining_criteria:
+ run_result = deterministic_result
+ else:
+ judged_result = await evaluate_completion_criteria(remaining_criteria, snapshot, handler)
+ if judged_result.status != "evaluated":
+ run_result = judged_result
+ else:
+ verdicts = []
+ if deterministic_result is not None:
+ verdicts.extend(deterministic_result.verdicts)
+ verdicts.extend(judged_result.verdicts)
+ run_result = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=[criterion.id for criterion in run_criteria],
+ verdicts=verdicts,
+ )
+ run_result = _apply_present_value_upgrades(run_result, run_criteria, snapshot)
return combine_verification_results(criterion_ids, run_result, definition_verdicts)
diff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.py
index 3124c89bf..7460b05c9 100644
--- a/skyvern/forge/sdk/copilot/tools/run_execution.py
+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py
@@ -820,6 +820,159 @@ def _typed_child_blocks(block: Any) -> list[Any]:
return children
+def _workflow_output_parameter_indexes(
+ workflow: Workflow | None,
+) -> tuple[dict[str, dict[str, Any]], dict[str, dict[str, Any]]]:
+ if workflow is None:
+ return {}, {}
+ workflow_definition = getattr(workflow, "workflow_definition", None)
+ blocks = getattr(workflow_definition, "blocks", None)
+ if not isinstance(blocks, list):
+ return {}, {}
+
+ by_id: dict[str, dict[str, Any]] = {}
+ by_key: dict[str, dict[str, Any]] = {}
+
+ def visit(block: Any) -> None:
+ output_parameter = (
+ block.get("output_parameter") if isinstance(block, Mapping) else getattr(block, "output_parameter", None)
+ )
+ output_parameter_id = (
+ output_parameter.get("output_parameter_id")
+ if isinstance(output_parameter, Mapping)
+ else getattr(output_parameter, "output_parameter_id", None)
+ )
+ output_parameter_key = (
+ output_parameter.get("key")
+ if isinstance(output_parameter, Mapping)
+ else getattr(output_parameter, "key", None)
+ )
+ label = block.get("label") if isinstance(block, Mapping) else getattr(block, "label", None)
+ block_type = block.get("block_type") if isinstance(block, Mapping) else getattr(block, "block_type", None)
+ block_type_name = getattr(block_type, "value", getattr(block_type, "name", block_type))
+ entry = {
+ "block_label": label if isinstance(label, str) and label else None,
+ "block_type": str(block_type_name) if block_type_name is not None else None,
+ "output_parameter_id": output_parameter_id if isinstance(output_parameter_id, str) else None,
+ "output_parameter_key": output_parameter_key if isinstance(output_parameter_key, str) else None,
+ }
+ if entry["output_parameter_id"]:
+ by_id[entry["output_parameter_id"]] = entry
+ if entry["output_parameter_key"]:
+ by_key[entry["output_parameter_key"]] = entry
+ for child in _mapping_child_blocks(block) if isinstance(block, Mapping) else _typed_child_blocks(block):
+ visit(child)
+
+ for block in blocks:
+ visit(block)
+ return by_id, by_key
+
+
+def _merge_registered_output_parameter_values_into_blocks(data: dict[str, Any]) -> None:
+ """Mutate ``data["blocks"]`` so registered output parameters share the block-output evidence path."""
+
+ registered = data.get("registered_output_parameter_values")
+ if not isinstance(registered, list) or not registered:
+ return
+ blocks = data.get("blocks")
+ if not isinstance(blocks, list):
+ blocks = []
+ data["blocks"] = blocks
+ by_label: dict[str, dict[str, Any]] = {
+ block["label"]: block
+ for block in blocks
+ if isinstance(block, dict) and isinstance(block.get("label"), str) and block.get("label")
+ }
+ for item in registered:
+ if not isinstance(item, Mapping):
+ continue
+ label = item.get("block_label")
+ key = item.get("output_parameter_key")
+ if not isinstance(label, str) or not label or not isinstance(key, str) or not key:
+ continue
+ value = item.get("value")
+ block = by_label.get(label)
+ if block is None:
+ block = {
+ "label": label,
+ "block_type": item.get("block_type") or "CODE",
+ "status": WorkflowRunStatus.completed.value,
+ }
+ blocks.append(block)
+ by_label[label] = block
+ extracted = block.get("extracted_data")
+ if isinstance(extracted, dict):
+ extracted.setdefault(key, value)
+ elif extracted is None:
+ block["extracted_data"] = {key: value}
+
+
+async def _attach_registered_output_parameter_values(
+ *,
+ workflow_run_id: str,
+ workflow: Workflow | None,
+ data: dict[str, Any],
+ persisted_output_parameters: list[Any] | None = None,
+) -> dict[str, Any]:
+ try:
+ registered_rows = await app.DATABASE.workflow_runs.get_workflow_run_output_parameters(
+ workflow_run_id=workflow_run_id
+ )
+ except Exception:
+ LOG.warning(
+ "Failed to read workflow run output parameters for copilot run evidence; "
+ "deterministic graders lose authoritative output-parameter evidence",
+ workflow_run_id=workflow_run_id,
+ organization_id=workflow.organization_id if workflow is not None else None,
+ exc_info=True,
+ )
+ return {}
+ if not registered_rows:
+ return {}
+
+ index_by_id, index_by_key = _workflow_output_parameter_indexes(workflow)
+ persisted_key_by_id = {
+ output_parameter_id: key
+ for parameter in persisted_output_parameters or []
+ if isinstance((output_parameter_id := getattr(parameter, "output_parameter_id", None)), str)
+ and isinstance((key := getattr(parameter, "key", None)), str)
+ }
+ normalized: list[dict[str, Any]] = []
+ values_by_label: dict[str, Any] = {}
+ for row in registered_rows:
+ output_parameter_id = getattr(row, "output_parameter_id", None)
+ if not isinstance(output_parameter_id, str) or not output_parameter_id:
+ continue
+ block_info = dict(index_by_id.get(output_parameter_id, {}))
+ output_parameter_key = block_info.get("output_parameter_key")
+ if not isinstance(output_parameter_key, str) or not output_parameter_key:
+ output_parameter_key = persisted_key_by_id.get(output_parameter_id)
+ if isinstance(output_parameter_key, str):
+ block_info["output_parameter_key"] = output_parameter_key
+ if output_parameter_key and not block_info.get("block_label"):
+ block_info.update(index_by_key.get(output_parameter_key, {}))
+ value = getattr(row, "value", None)
+ item = {
+ "workflow_run_id": workflow_run_id,
+ "output_parameter_id": output_parameter_id,
+ "output_parameter_key": block_info.get("output_parameter_key"),
+ "block_label": block_info.get("block_label"),
+ "block_type": block_info.get("block_type"),
+ "value": value,
+ }
+ normalized.append(item)
+ label = item.get("block_label")
+ key = item.get("output_parameter_key")
+ if isinstance(label, str) and label and isinstance(key, str) and key:
+ values_by_label.setdefault(label, {})[key] = value
+
+ if not normalized:
+ return {}
+ data["registered_output_parameter_values"] = normalized
+ _merge_registered_output_parameter_values_into_blocks(data)
+ return values_by_label
+
+
async def _run_blocks_and_collect_debug(
params: dict[str, Any],
ctx: CopilotContext,
@@ -1412,6 +1565,16 @@ async def _run_blocks_and_collect_debug(
if not run_ok and run and getattr(run, "failure_reason", None):
result_data["failure_reason"] = run.failure_reason
+ registered_outputs_by_label = await _attach_registered_output_parameter_values(
+ workflow_run_id=workflow_run.workflow_run_id,
+ workflow=runtime_workflow,
+ data=result_data,
+ persisted_output_parameters=all_output_params,
+ )
+ for label, output in registered_outputs_by_label.items():
+ if isinstance(output, dict) and output:
+ block_outputs_by_label[label] = output
+
# Update verified prefix state ONLY on a fully-successful run. A failed
# suffix run leaves the browser in post-failure state, so we must not
# trust blocks that individually succeeded inside it.
@@ -1790,6 +1953,9 @@ def _record_run_blocks_result(
timeout_latched = bool(copilot_ctx.copilot_total_timeout_exceeded)
copilot_ctx.last_test_ok = None if (cancelled_by_watchdog and timeout_latched) else run_ok
copilot_ctx.last_full_workflow_test_ok = False
+ # Re-affirmed per run below only when this run satisfies completion; never let a
+ # prior run's terminal-ready latch leak into a run that did not verify.
+ copilot_ctx.verified_terminal_proposal_ready = False
copilot_ctx.last_unverified_block_labels = _unverified_current_workflow_labels(copilot_ctx)
copilot_ctx.last_test_failure_reason = None
copilot_ctx.last_artifact_health_blocker_reason = None
@@ -1912,6 +2078,11 @@ def _record_run_blocks_result(
if run_ok:
_mark_page_inspected(copilot_ctx)
+ completion_fully_satisfied = (
+ completion_verification is not None
+ and completion_verification.status == "evaluated"
+ and completion_verification.is_fully_satisfied()
+ )
if structured_blocker:
# Terminal anti-bot blockers are handled before run_ok; this branch
# remains for non-challenge structured blockers that still make a
@@ -1936,7 +2107,17 @@ def _record_run_blocks_result(
display_reason=run_outcome_display_reason(structured_blocker),
),
)
- if empty_data_blocks:
+ if completion_fully_satisfied:
+ # Terminal proposal promotion is authoritative only after the same
+ # run has satisfied completion verification; clear stale
+ # suspicious-success state before agent/enforcement exits consume
+ # ``verified_terminal_proposal_ready``.
+ copilot_ctx.verified_terminal_proposal_ready = True
+ copilot_ctx.last_test_suspicious_success = False
+ copilot_ctx.last_test_failure_reason = None
+ copilot_ctx.null_data_streak_count = 0
+ copilot_ctx.suspicious_success_nudge_count = 0
+ if empty_data_blocks and not completion_fully_satisfied:
copilot_ctx.last_test_ok = None
copilot_ctx.last_test_suspicious_success = True
copilot_ctx.null_data_streak_count = getattr(copilot_ctx, "null_data_streak_count", 0) + 1
@@ -1983,8 +2164,15 @@ def _record_run_blocks_result(
# Real success: clear the signature latch so a subsequent bad URL in
# the same session can re-fire the stop nudge.
copilot_ctx.non_retriable_nav_error_last_emitted_signature = None
- if outcome_unverified_reason is None and not unverified:
+ if outcome_unverified_reason is None and completion_fully_satisfied:
+ copilot_ctx.last_full_workflow_test_ok = True
+ copilot_ctx.last_unverified_block_labels = []
+ copilot_ctx.last_good_workflow = copilot_ctx.last_workflow
+ copilot_ctx.last_good_workflow_yaml = copilot_ctx.last_workflow_yaml
+ copilot_ctx.last_test_failure_reason = None
+ elif outcome_unverified_reason is None and not unverified:
copilot_ctx.last_full_workflow_test_ok = True
+ copilot_ctx.last_unverified_block_labels = []
copilot_ctx.last_good_workflow = copilot_ctx.last_workflow
copilot_ctx.last_good_workflow_yaml = copilot_ctx.last_workflow_yaml
elif outcome_unverified_reason is None:
diff --git a/skyvern/forge/sdk/copilot/tools/workflow_update.py b/skyvern/forge/sdk/copilot/tools/workflow_update.py
index b1068291f..6b6844250 100644
--- a/skyvern/forge/sdk/copilot/tools/workflow_update.py
+++ b/skyvern/forge/sdk/copilot/tools/workflow_update.py
@@ -3444,10 +3444,17 @@ async def _update_workflow(
"_workflow": workflow,
}
except (yaml.YAMLError, ValidationError, BaseWorkflowHTTPException) as e:
- return {
+ result: dict[str, Any] = {
"ok": False,
"error": f"Workflow validation failed: {e}",
}
+ if _copilot_block_authoring_policy(ctx) == BlockAuthoringPolicy.CODE_ONLY_BROWSER:
+ result["data"] = _code_repair_progress_data()
+ result["user_facing_summary"] = _code_seam_rejection_user_summary(
+ metadata_rejected=False,
+ code_rejected=True,
+ )
+ return result
def _record_workflow_proxy_location_span(workflow_yaml: str, workflow: Workflow) -> None:
diff --git a/tests/unit/test_client_pydantic_utilities.py b/tests/unit/test_client_pydantic_utilities.py
new file mode 100644
index 000000000..3b2136dde
--- /dev/null
+++ b/tests/unit/test_client_pydantic_utilities.py
@@ -0,0 +1,39 @@
+from __future__ import annotations
+
+import pytest
+
+from skyvern.client.core.pydantic_utilities import IS_PYDANTIC_V2, update_forward_refs
+
+
[email protected](not IS_PYDANTIC_V2, reason="Pydantic v2-specific generated-client rebuild behavior")
+def test_update_forward_refs_suppresses_pydantic_v2_definitions_key_error() -> None:
+ class GeneratedModel:
+ @classmethod
+ def model_rebuild(cls, *, raise_errors: bool) -> None:
+ assert raise_errors is False
+ raise KeyError("definitions")
+
+ update_forward_refs(GeneratedModel) # type: ignore[type-var]
+
+
[email protected](not IS_PYDANTIC_V2, reason="Pydantic v2-specific generated-client rebuild behavior")
+def test_update_forward_refs_suppresses_definitions_key_error_with_extra_context() -> None:
+ class GeneratedModel:
+ @classmethod
+ def model_rebuild(cls, *, raise_errors: bool) -> None:
+ assert raise_errors is False
+ raise KeyError("definitions", "extra-context")
+
+ update_forward_refs(GeneratedModel) # type: ignore[type-var]
+
+
[email protected](not IS_PYDANTIC_V2, reason="Pydantic v2-specific generated-client rebuild behavior")
+def test_update_forward_refs_preserves_unexpected_key_error() -> None:
+ class GeneratedModel:
+ @classmethod
+ def model_rebuild(cls, *, raise_errors: bool) -> None:
+ assert raise_errors is False
+ raise KeyError("other")
+
+ with pytest.raises(KeyError, match="other"):
+ update_forward_refs(GeneratedModel) # type: ignore[type-var]
diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.py
index 18d9d152f..14473ce9c 100644
--- a/tests/unit/test_copilot_agent_helpers.py
+++ b/tests/unit/test_copilot_agent_helpers.py
@@ -14,6 +14,7 @@ import yaml
from skyvern.forge.sdk.api.llm.exceptions import LLMProviderError
from skyvern.forge.sdk.copilot import agent as agent_module
from skyvern.forge.sdk.copilot.build_phase import BuildPhase
+from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdict
from skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy, CopilotConfig
from skyvern.forge.sdk.copilot.diagnosis_repair_contract import (
DiagnosisInput,
@@ -33,6 +34,7 @@ from skyvern.forge.sdk.copilot.recoverable_failure import build_recoverable_fail
from skyvern.forge.sdk.copilot.request_policy import (
_REDACTED_REFUSED_SECRET_TURN,
TRANSCRIPT_ANCHOR_CHAR_CAP,
+ CompletionCriterion,
RequestPolicy,
_classify_request,
build_transcript_context,
@@ -1244,6 +1246,7 @@ def _chat_request() -> SimpleNamespace:
workflow_id="wf-1",
workflow_permanent_id="wfp-1",
workflow_copilot_chat_id="chat-1",
+ workflow_yaml="",
)
@@ -1413,6 +1416,93 @@ class TestTranslateToAgentResultGating:
assert agent_result.response_type == "ASK_QUESTION"
assert agent_result.user_response == "Which account should I use?"
+ def test_verified_terminal_state_surfaces_workflow_despite_weak_final_reply(self) -> None:
+ workflow = SimpleNamespace(workflow_definition=SimpleNamespace(blocks=[]))
+ ctx = _ctx(
+ last_workflow=workflow,
+ last_workflow_yaml="title: Verified Draft",
+ last_test_ok=True,
+ last_full_workflow_test_ok=True,
+ )
+ ctx.verified_terminal_proposal_ready = True
+ ctx.completion_verification_result = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=["c0"],
+ verdicts=[CriterionVerdict(criterion_id="c0", state="satisfied", reason_code="evidence_confirms")],
+ )
+ result = _fake_run_result({"type": "ASK_QUESTION", "user_response": "Do you want me to keep repairing?"})
+
+ agent_result = asyncio.run(
+ agent_module._translate_to_agent_result(
+ result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1"
+ )
+ )
+
+ assert agent_result.response_type == "REPLY"
+ assert agent_result.updated_workflow is workflow
+ assert agent_result.workflow_yaml == "title: Verified Draft"
+ assert agent_result.clear_proposed_workflow is False
+ assert agent_result.proposal_disposition == "auto_applicable"
+
+ def test_output_field_confirmation_question_is_blocked_when_contract_present(self) -> None:
+ ctx = _ctx(
+ request_policy=RequestPolicy(
+ user_response_policy="proceed",
+ completion_contract_status="present",
+ completion_criteria=[
+ CompletionCriterion(id="provider", outcome="The returned record identifies the provider."),
+ ],
+ )
+ )
+ result = _fake_run_result(
+ {
+ "type": "ASK_QUESTION",
+ "user_response": "Please confirm the output fields before I build and test this workflow.",
+ }
+ )
+
+ agent_result = asyncio.run(
+ agent_module._translate_to_agent_result(
+ result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1"
+ )
+ )
+
+ assert agent_result.response_type == "ASK_QUESTION"
+ assert agent_result.updated_workflow is None
+ assert agent_result.clear_proposed_workflow is False
+ assert agent_result.proposal_disposition == "no_proposal"
+ assert agent_result.output_policy_diagnostics is not None
+ assert agent_result.output_policy_diagnostics["final_output_policy_allowed"] is False
+ assert agent_result.output_policy_diagnostics["hard_block_reason_codes"] == [
+ "avoidable_output_field_confirmation"
+ ]
+
+ def test_credential_clarification_question_remains_allowed_with_request_policy(self) -> None:
+ ctx = _ctx(
+ request_policy=RequestPolicy(
+ user_response_policy="ask_clarification",
+ clarification_question="Which saved credential should I use?",
+ clarification_reason="credential_name_unresolved",
+ completion_contract_status="present",
+ completion_criteria=[
+ CompletionCriterion(id="provider", outcome="The returned record identifies the provider."),
+ ],
+ )
+ )
+ result = _fake_run_result({"type": "ASK_QUESTION", "user_response": "Which saved credential should I use?"})
+
+ agent_result = asyncio.run(
+ agent_module._translate_to_agent_result(
+ result, ctx, global_llm_context=None, chat_request=_chat_request(), organization_id="org-1"
+ )
+ )
+
+ assert agent_result.response_type == "ASK_QUESTION"
+ assert agent_result.clear_proposed_workflow is True
+ assert agent_result.output_policy_diagnostics is not None
+ assert agent_result.output_policy_diagnostics["final_output_policy_allowed"] is True
+ assert "avoidable_output_field_confirmation" not in agent_result.output_policy_diagnostics["raw_reason_codes"]
+
def test_inline_replace_workflow_resets_test_ok_after_prior_pass(self, monkeypatch) -> None:
# A prior run_blocks test passed for the old workflow (ctx.last_test_ok=True,
# ctx.last_workflow=old_wf). The agent then emits inline REPLACE_WORKFLOW
diff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.py
index 5f9134eb6..b10bab8ae 100644
--- a/tests/unit/test_copilot_completion_verification.py
+++ b/tests/unit/test_copilot_completion_verification.py
@@ -4,6 +4,7 @@ import asyncio
import time
from collections.abc import Awaitable, Callable
from types import SimpleNamespace
+from typing import Any
import pytest
@@ -18,7 +19,11 @@ from skyvern.forge.sdk.copilot.completion_verification import (
CriterionVerdict,
RunEvidenceSnapshot,
_coerce_result,
+ _structured_record_has_identifier,
evaluate_completion_criteria,
+ grade_record_semantic_consistency,
+ grade_structured_record_criteria,
+ structured_record_has_identity,
summarize_unsatisfied_outcomes,
)
from skyvern.forge.sdk.copilot.context import CopilotContext
@@ -64,6 +69,87 @@ def _criterion(cid: str, outcome: str, *, method_mandated: bool = False) -> Comp
return CompletionCriterion(id=cid, outcome=outcome, method_mandated=method_mandated)
+_STRUCTURED_RECORD_CRITERIA = (
+ ("fallback_record_identity", "The returned record identifies the target record."),
+ ("fallback_record_identifier", "The returned record includes the record identifier."),
+ ("fallback_record_groups", "The returned record includes record items."),
+ (
+ "fallback_record_status",
+ "The returned record's per-location statuses and overall status are present and consistent.",
+ ),
+)
+_STRUCTURED_RECORD_CRITERION_IDS = {cid for cid, _ in _STRUCTURED_RECORD_CRITERIA}
+
+
+def _structured_record_criteria() -> list[CompletionCriterion]:
+ return [CompletionCriterion(id=cid, outcome=outcome) for cid, outcome in _STRUCTURED_RECORD_CRITERIA]
+
+
+def _status_consistency_criteria() -> list[CompletionCriterion]:
+ return [
+ CompletionCriterion(
+ id="fallback_record_status",
+ outcome="The returned record's per-location statuses and overall status are present and consistent.",
+ )
+ ]
+
+
+def test_structured_record_identity_ignores_substring_only_keys() -> None:
+ assert structured_record_has_identity({"provider_name": "Alex Example"}) is True
+ assert structured_record_has_identity({"providerName": "Alex Example"}) is True
+ assert structured_record_has_identity({"title": "Permit A"}) is True
+ assert structured_record_has_identity({"filename": "report.pdf"}) is False
+ assert structured_record_has_identity({"tablename": "providers"}) is False
+
+
+def test_structured_record_identifier_ignores_substring_only_keys() -> None:
+ assert _structured_record_has_identifier({"providerId": "abc"}) is True
+ assert _structured_record_has_identifier({"record_number": "x"}) is True
+ assert _structured_record_has_identifier({"idea": "some text"}) is False
+ assert _structured_record_has_identifier({"_identical": "yes"}) is False
+ assert structured_record_has_identity({"subtitle": "detail"}) is False
+ assert structured_record_has_identity({"mislabeled": "detail"}) is False
+
+
+def _satisfied_criterion_ids(verdicts: list[CriterionVerdict]) -> set[str]:
+ return {verdict.criterion_id for verdict in verdicts if verdict.satisfied}
+
+
+def _record_payload(**overrides: Any) -> dict[str, Any]:
+ payload = {
+ "entity_found": True,
+ "entity_name": "Jordan Example",
+ "record_number": "1234567890",
+ "items": [
+ {"item_name": "Sample Practice", "address": "100 Main St, Example City, ST 12345", "status": "Active"},
+ {
+ "item_name": "Secondary Practice",
+ "address": "300 Market St, Example City, ST 12345",
+ "status": "Inactive",
+ },
+ ],
+ "overall_status": "Active",
+ "evidence_text": "Opened Details page; read Overview/Affiliations items and More Details identifier.",
+ }
+ payload.update(overrides)
+ return payload
+
+
+def _status_snapshot(
+ status: str,
+ *,
+ item_name: str = "Sample Practice",
+ evidence_text: str | None = None,
+) -> RunEvidenceSnapshot:
+ payload = _record_payload(
+ items=[{"item_name": item_name, "address": "100 Main St, Example City, ST 12345", "status": status}],
+ overall_status=status,
+ )
+ if evidence_text is not None:
+ payload["evidence_text"] = evidence_text
+ return RunEvidenceSnapshot(block_outputs={"lookup_record_status": payload})
+
+
def _evaluated(*satisfied_by_id: tuple[str, bool]) -> CompletionVerificationResult:
ids = [cid for cid, _ in satisfied_by_id]
verdicts = [
@@ -84,6 +170,13 @@ def _completion_handler_lookup(handler: object) -> Callable[[object], Awaitable[
return _lookup
+def _patch_completion_handler(monkeypatch: pytest.MonkeyPatch, handler: object) -> None:
+ monkeypatch.setattr(
+ "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
+ _completion_handler_lookup(handler),
+ )
+
+
def test_is_fully_satisfied_requires_every_criterion() -> None:
assert _evaluated(("c0", True), ("c1", True)).is_fully_satisfied() is True
assert _evaluated(("c0", True), ("c1", False)).is_fully_satisfied() is False
@@ -223,6 +316,119 @@ def test_coerce_accepts_bytes_and_rejects_malformed() -> None:
assert _coerce_result({"no_verdicts_key": 1}, ["c0"]).status == "unavailable"
+ ("negative_status", "item_name", "missing_text"),
+ [
+ ("Inactive", "Sample Practice 200 Oak Ave, Example City, ST 12345 Active", "non-status fields include"),
+ *[
+ (status, "Sample Practice Active", None)
+ for status in ["Expired", "Suspended", "Terminated", "Revoked", "Lapsed", "Pending"]
+ ],
+ ],
+)
+def test_record_semantic_consistency_flags_status_contradictions(
+ negative_status: str, item_name: str, missing_text: str | None
+) -> None:
+ verdicts = grade_record_semantic_consistency(
+ _status_consistency_criteria(),
+ _status_snapshot(negative_status, item_name=item_name),
+ )
+
+ assert len(verdicts) == 1
+ assert verdicts[0].criterion_id == "fallback_record_status"
+ assert verdicts[0].state == "unsatisfied"
+ assert verdicts[0].reason_code == "evidence_contradicts"
+ if missing_text:
+ assert missing_text in (verdicts[0].missing_evidence or "")
+
+
+ ("status", "item_name", "evidence_text"),
+ [
+ ("Non-active", "Sample Practice non-active listing", "Sample Practice non-active listing 100 Main St"),
+ ("Active", "Sample Practice", "Sample Practice 100 Main St, Example City, ST 12345 Active"),
+ *[
+ ("Expired", "Sample Practice", text)
+ for text in [
+ "License is no longer active",
+ "Provider was previously active",
+ "Status note: not currently active",
+ "The active license expired",
+ ]
+ ],
+ ],
+)
+def test_record_semantic_consistency_accepts_non_contradictory_status_text(
+ status: str, item_name: str, evidence_text: str
+) -> None:
+ assert (
+ grade_record_semantic_consistency(
+ _status_consistency_criteria(), _status_snapshot(status, item_name=item_name, evidence_text=evidence_text)
+ )
+ == []
+ )
+
+
+def test_structured_record_identifier_requires_consecutive_digit_run() -> None:
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "lookup_record": _record_payload(
+ phone="555-1234",
+ record_number=None,
+ items=[{"item_name": "Sample Practice", "address": "1234 Main St, Apt 56", "status": "Active"}],
+ )
+ }
+ )
+
+ satisfied = _satisfied_criterion_ids(grade_structured_record_criteria(_structured_record_criteria(), snapshot))
+
+ assert "fallback_record_identifier" not in satisfied
+ assert "fallback_record_identity" in satisfied
+
+
+ "block_outputs",
+ [
+ {
+ "extract_record_status_info": {
+ "extract_record_status_info_output": _record_payload(),
+ "extracted_information": [],
+ }
+ },
+ {"extract_record_status_record_output": _record_payload(found=True, entity_found=None)},
+ ],
+)
+def test_structured_record_criteria_satisfy_structured_record_outputs(block_outputs: dict[str, Any]) -> None:
+ snapshot = RunEvidenceSnapshot(block_outputs=block_outputs)
+
+ verdicts = grade_structured_record_criteria(_structured_record_criteria(), snapshot)
+
+ assert _satisfied_criterion_ids(verdicts) == _STRUCTURED_RECORD_CRITERION_IDS
+
+
+def test_structured_record_partial_matches_do_not_combine_across_blocks() -> None:
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "identity_block": _record_payload(items=[], overall_status=None),
+ "status_block": {
+ "items": [{"item_name": "Sample Practice", "address": "100 Main St", "status": "Active"}],
+ "overall_status": "Active",
+ },
+ }
+ )
+
+ criteria = _structured_record_criteria()
+ verdicts = grade_structured_record_criteria(criteria, snapshot)
+ result = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=[criterion.id for criterion in criteria],
+ verdicts=verdicts,
+ )
+
+ assert _satisfied_criterion_ids(verdicts) < _STRUCTURED_RECORD_CRITERION_IDS
+ assert result.is_fully_satisfied() is False
+
+
@pytest.mark.asyncio
async def test_evaluate_no_handler_or_no_criteria_is_unavailable() -> None:
snapshot = RunEvidenceSnapshot(current_url="https://example.com")
@@ -569,6 +775,40 @@ def _clean_success_result() -> dict:
}
+def _structured_record_top_level_output_result() -> dict:
+ return {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_structured_record",
+ "overall_status": "completed",
+ "executed_block_labels": ["extract_record_status_record"],
+ "current_url": "https://structured_record.test/entity-details",
+ "blocks": [
+ {
+ "label": "extract_record_status_record",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": {"extracted_information": []},
+ }
+ ],
+ "output": {
+ "open_search_search_output": {
+ "page_state": "search_search_open",
+ "evidence_text": "Opened search search page with search-by-doctor typeahead #searchInput.",
+ },
+ "search_and_open_record_details_output": {
+ "found": True,
+ "entity_name": "Jordan Example",
+ "opened_record_details": True,
+ "evidence_text": "Opened Details page for the selected record.",
+ },
+ "extract_record_status_record_output": _record_payload(found=True, entity_found=None),
+ "extracted_information": [],
+ },
+ },
+ }
+
+
def _run_ctx() -> CopilotContext:
ctx = CopilotContext(
organization_id="o",
@@ -591,6 +831,12 @@ def _ctx_with_blocks(*block_types: str) -> CopilotContext:
return ctx
+def _set_workflow_labels(ctx: CopilotContext, *labels: str) -> None:
+ ctx.last_workflow = SimpleNamespace(
+ workflow_definition=SimpleNamespace(blocks=[SimpleNamespace(label=label) for label in labels])
+ )
+
+
def _contradicted(cid: str) -> CompletionVerificationResult:
verdict = CriterionVerdict(criterion_id=cid, state="unsatisfied", reason_code="evidence_contradicts")
return CompletionVerificationResult(status="evaluated", criterion_ids=[cid], verdicts=[verdict])
@@ -688,10 +934,7 @@ async def test_goto_only_run_still_fails_extraction_verification(monkeypatch: py
]
}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _ctx_with_blocks("goto_url")
ctx.request_policy = RequestPolicy(completion_criteria=[_criterion("c0", "heading and paragraph are extracted")])
@@ -711,10 +954,7 @@ async def test_structured_blocker_run_skips_completion_verification(monkeypatch:
async def handler(**_: object) -> dict:
raise AssertionError("structured blocker runs must not be sent to the completion judge")
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _ctx_with_blocks("code")
result = {
"ok": True,
@@ -1001,10 +1241,7 @@ async def test_maybe_run_completion_verification_runs_on_canceled_run(monkeypatc
async def handler(**_: object) -> dict:
return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
result = await _maybe_run_completion_verification(ctx, _canceled_budget_result(), time.monotonic())
assert result is not None
@@ -1254,10 +1491,7 @@ async def test_page_observation_verification_recognizes_budgeted_outcome(
seen_prompt["prompt"] = str(kwargs.get("prompt") or "")
return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
ctx.last_test_ok = False
ctx.last_run_blocks_workflow_run_id = "wr_cancel"
@@ -1300,10 +1534,7 @@ async def test_page_observation_verification_does_not_overwrite_satisfied_verdic
async def handler(**_: object) -> dict:
raise AssertionError("handler should not be called once the outcome is verified")
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
ctx.last_test_ok = False
ctx.last_run_blocks_workflow_run_id = "wr_cancel"
@@ -1337,10 +1568,7 @@ async def test_page_observation_verification_preserves_existing_unsatisfied_verd
handler_calls += 1
return {"verdicts": [{"criterion_id": "c0", "satisfied": False, "reason_code": "no_evidence"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
ctx.last_test_ok = False
ctx.last_run_blocks_workflow_run_id = "wr_cancel"
@@ -1371,10 +1599,7 @@ async def test_page_observation_verification_can_upgrade_unsatisfied_verdict(
async def handler(**_: object) -> dict:
return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
ctx.last_test_ok = False
ctx.last_run_blocks_workflow_run_id = "wr_cancel"
@@ -1477,10 +1702,7 @@ async def test_maybe_run_completion_verification_runs_on_unverified_prefix(monke
async def handler(**_: object) -> dict:
return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _ctx_unverified_prefix()
result = await _maybe_run_completion_verification(ctx, _clean_success_result(), time.monotonic())
assert result is not None
@@ -1507,10 +1729,7 @@ async def test_method_mandated_criteria_excluded_from_verification(monkeypatch:
async def handler(**_: object) -> dict:
return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
ctx.request_policy = RequestPolicy(
completion_criteria=[
@@ -1561,11 +1780,238 @@ def test_snapshot_uses_current_run_blocks_not_stale_outputs() -> None:
assert snap2.block_outputs.get("b3") == {"extracted_information": {"price": "9.99"}}
-def test_snapshot_summarizes_registered_download_outputs() -> None:
+def test_snapshot_indexes_workflow_output_parameter_records() -> None:
ctx = _run_ctx()
- ctx.last_workflow = SimpleNamespace(
- workflow_definition=SimpleNamespace(blocks=[SimpleNamespace(label="download_statement")])
+ _set_workflow_labels(
+ ctx,
+ "open_search_search_page",
+ "search_and_open_record_details",
+ "extract_record_status_record",
)
+ run = {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_structured_record",
+ "blocks": [
+ {
+ "label": "open_search_search_page",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": {
+ "open_search_search_page_output": {"evidence_text": "Opened search search page"}
+ },
+ },
+ {
+ "label": "search_and_open_record_details",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": {
+ "search_and_open_record_details_output": {
+ "entity_found": True,
+ "evidence_text": "Opened Details page",
+ }
+ },
+ },
+ {
+ "label": "extract_record_status_record",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": {"extract_record_status_record_output": _record_payload(evidence_text=None)},
+ },
+ ],
+ },
+ }
+
+ snap = _build_run_evidence_snapshot(ctx, run)
+
+ assert "open_search_search_page_output" in snap.block_outputs
+ assert snap.block_outputs["search_and_open_record_details_output"]["evidence_text"] == ("Opened Details page")
+ assert snap.block_outputs["extract_record_status_record_output"]["record_number"] == "1234567890"
+
+
+def test_snapshot_uses_current_run_registered_output_parameters() -> None:
+ ctx = _run_ctx()
+ _set_workflow_labels(ctx, "extract_record_status_details")
+ run = {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_structured_record",
+ "overall_status": "completed",
+ "blocks": [],
+ "registered_output_parameter_values": [
+ {
+ "workflow_run_id": "wr_structured_record",
+ "output_parameter_id": "op_record",
+ "output_parameter_key": "extract_record_status_details_output",
+ "block_label": "extract_record_status_details",
+ "block_type": "CODE",
+ "value": _record_payload(evidence_text="Opened Details page"),
+ }
+ ],
+ },
+ }
+
+ snap = _build_run_evidence_snapshot(ctx, run)
+ verdicts = grade_structured_record_criteria(_structured_record_criteria(), snap)
+
+ assert snap.block_outputs["extract_record_status_details_output"]["record_number"] == "1234567890"
+ assert (
+ snap.block_outputs["extract_record_status_details"]["extract_record_status_details_output"]["record_number"]
+ == "1234567890"
+ )
+ assert _satisfied_criterion_ids(verdicts) == _STRUCTURED_RECORD_CRITERION_IDS
+
+
+def test_snapshot_uses_structured_record_top_level_output_parameters() -> None:
+ ctx = _run_ctx()
+ _set_workflow_labels(ctx, "extract_record_status_record")
+
+ snap = _build_run_evidence_snapshot(ctx, _structured_record_top_level_output_result())
+ verdicts = grade_structured_record_criteria(_structured_record_criteria(), snap)
+
+ assert snap.block_outputs["extract_record_status_record_output"]["record_number"] == "1234567890"
+ assert _satisfied_criterion_ids(verdicts) == _STRUCTURED_RECORD_CRITERION_IDS
+
+
+async def test_maybe_run_completion_verification_deterministically_satisfies_structured_record_without_judge_call(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ handler_lookup_calls = 0
+
+ async def fail_handler(**_: object) -> object:
+ raise AssertionError("deterministically covered structured-record criteria must not call the judge")
+
+ async def handler_lookup(_ctx: object) -> object:
+ nonlocal handler_lookup_calls
+ handler_lookup_calls += 1
+ return fail_handler
+
+ monkeypatch.setattr(
+ "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
+ handler_lookup,
+ )
+ ctx = _run_ctx()
+ ctx.request_policy = RequestPolicy(completion_criteria=_structured_record_criteria())
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _structured_record_top_level_output_result(),
+ time.monotonic(),
+ )
+
+ assert verification is not None
+ assert verification.status == "evaluated"
+ assert verification.is_fully_satisfied() is True
+ assert handler_lookup_calls == 1
+ assert _satisfied_criterion_ids(verification.verdicts) == _STRUCTURED_RECORD_CRITERION_IDS
+
+
+async def test_maybe_run_completion_verification_fails_closed_on_no_judge_for_deterministic_success(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ _patch_completion_handler(monkeypatch, None)
+ ctx = _run_ctx()
+ ctx.request_policy = RequestPolicy(completion_criteria=_structured_record_criteria())
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _structured_record_top_level_output_result(),
+ time.monotonic(),
+ )
+
+ assert verification is not None
+ assert verification.status == "unavailable"
+ assert verification.is_fully_satisfied() is False
+
+
+async def test_maybe_run_completion_verification_fails_closed_on_low_budget_for_deterministic_success(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def fail_handler(**_: object) -> object:
+ raise AssertionError("low-budget deterministic verification must not call the judge")
+
+ _patch_completion_handler(monkeypatch, fail_handler)
+ ctx = _run_ctx()
+ ctx.request_policy = RequestPolicy(completion_criteria=_structured_record_criteria())
+ starved = time.monotonic() - 100_000
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _structured_record_top_level_output_result(),
+ starved,
+ )
+
+ assert verification is not None
+ assert verification.status == "unavailable"
+ assert verification.is_fully_satisfied() is False
+
+
+async def test_maybe_run_completion_verification_mixed_criteria_still_fail_closed_on_judge_timeout(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ calls = 0
+
+ async def slow_handler(**_: object) -> dict:
+ nonlocal calls
+ calls += 1
+ await asyncio.sleep(1)
+ return {"verdicts": []}
+
+ monkeypatch.setattr(settings, "COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS", 0.01)
+ _patch_completion_handler(monkeypatch, slow_handler)
+ ctx = _run_ctx()
+ ctx.request_policy = RequestPolicy(
+ completion_criteria=[
+ *_structured_record_criteria(),
+ CompletionCriterion(
+ id="source_timestamp_visible", outcome="The source page shows the latest update timestamp."
+ ),
+ ]
+ )
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _structured_record_top_level_output_result(),
+ time.monotonic(),
+ )
+
+ assert calls == 1
+ assert verification is not None
+ assert verification.status == "unavailable"
+ assert verification.is_fully_satisfied() is False
+
+
+def test_snapshot_ignores_registered_output_parameters_from_prior_run() -> None:
+ ctx = _run_ctx()
+ _set_workflow_labels(ctx, "extract_record_status_details")
+ run = {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_current",
+ "blocks": [],
+ "registered_output_parameter_values": [
+ {
+ "workflow_run_id": "wr_prior",
+ "output_parameter_key": "extract_record_status_details_output",
+ "block_label": "extract_record_status_details",
+ "value": {"entity_name": "Jordan Example", "record_number": "1234567890"},
+ }
+ ],
+ },
+ }
+
+ snap = _build_run_evidence_snapshot(ctx, run)
+
+ assert snap.block_outputs == {}
+
+
+def test_snapshot_summarizes_registered_download_outputs() -> None:
+ ctx = _run_ctx()
+ _set_workflow_labels(ctx, "download_statement")
run = {
"ok": True,
"data": {
@@ -1681,10 +2127,7 @@ async def test_completion_verification_receives_verified_context_labels(monkeypa
]
}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
ctx.request_policy = RequestPolicy(
completion_criteria=[
@@ -1750,10 +2193,7 @@ async def test_maybe_run_completion_verification_unavailable_on_low_budget(monke
async def handler(**_: object) -> dict:
return {"verdicts": [{"criterion_id": "c0", "satisfied": True, "reason_code": "evidence_confirms"}]}
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(handler),
- )
+ _patch_completion_handler(monkeypatch, handler)
ctx = _run_ctx()
starved = time.monotonic() - 100_000 # no budget left to verify this candidate run
result = await _maybe_run_completion_verification(ctx, _clean_success_result(), starved)
@@ -1763,12 +2203,13 @@ async def test_maybe_run_completion_verification_unavailable_on_low_budget(monke
assert result.status == "unavailable"
assert result.is_fully_satisfied() is False
- # A missing judge handler stays a soft fallback (None), not a hard block.
- monkeypatch.setattr(
- "skyvern.forge.sdk.copilot.tools.completion._completion_verification_handler",
- _completion_handler_lookup(None),
- )
- assert await _maybe_run_completion_verification(ctx, _clean_success_result(), time.monotonic()) is None
+ # A missing judge handler means the required completion contract could not be
+ # verified, so the run must not pass through on status alone.
+ _patch_completion_handler(monkeypatch, None)
+ no_handler_result = await _maybe_run_completion_verification(ctx, _clean_success_result(), time.monotonic())
+ assert no_handler_result is not None
+ assert no_handler_result.status == "unavailable"
+ assert no_handler_result.is_fully_satisfied() is False
def test_completion_contract_not_violated_unavailable_blocks_surfacing() -> None:
diff --git a/tests/unit/test_copilot_enforcement_pruning.py b/tests/unit/test_copilot_enforcement_pruning.py
index 6f9bae7de..236de30da 100644
--- a/tests/unit/test_copilot_enforcement_pruning.py
+++ b/tests/unit/test_copilot_enforcement_pruning.py
@@ -15,11 +15,13 @@ from unittest.mock import AsyncMock, MagicMock
import pytest
+from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdict
from skyvern.forge.sdk.copilot.enforcement import (
KEEP_RECENT_TOOL_OUTPUTS,
NULL_DATA_STREAK_ESCALATE_AT,
POST_REPEATED_NULL_DATA_NUDGE,
POST_SUSPICIOUS_SUCCESS_NUDGE,
+ CopilotGoalSatisfied,
_check_enforcement,
_needs_repeated_null_data_nudge,
_needs_suspicious_success_nudge,
@@ -64,6 +66,10 @@ class _Ctx:
self.null_data_streak_count = 0
self.repeated_failure_streak_count = 0
self.repeated_failure_nudge_emitted_at_streak = 0
+ self.verified_terminal_proposal_ready = False
+ self.completion_verification_result = None
+ self.last_artifact_health_blocker_reason = None
+ self.latest_diagnosis_repair_contract = None
# ---------------------------------------------------------------------------
@@ -243,6 +249,26 @@ def _text_prompt_block(extracted_data: Any) -> dict[str, Any]:
_EXTRACTION_ENVELOPE_CASES: list[tuple[str, dict[str, Any], bool]] = [
("empty_payload_trace_repro", {}, True),
("real_extraction", {"extracted_information": [{"price": "260.48"}]}, False),
+ (
+ "nested_code_output_record",
+ {
+ "extracted_information": [],
+ "extract_record_status_info_output": {
+ "entity_found": True,
+ "entity_name": "Jordan Example",
+ "record_number": "1234567890",
+ "items": [
+ {
+ "item_name": "Sample Practice",
+ "address": "100 Main St, Example City, ST 12345",
+ "status": "Active",
+ }
+ ],
+ "overall_status": "Active",
+ },
+ },
+ False,
+ ),
(
"download_only_files",
{"downloaded_files": [{"url": "https://example.com/a.pdf", "checksum": "abc123"}]},
@@ -384,6 +410,7 @@ def test_record_run_blocks_result_promotes_when_verified_prefix_covers_workflow(
)
ctx.last_workflow_yaml = "workflow: yaml"
ctx.verified_prefix_labels = ["open", "extract"]
+ ctx.last_unverified_block_labels = ["stale_extract"]
result = {
"ok": True,
@@ -404,6 +431,121 @@ def test_record_run_blocks_result_promotes_when_verified_prefix_covers_workflow(
assert ctx.last_good_workflow_yaml == ctx.last_workflow_yaml
+def test_record_run_blocks_result_promotes_structured_record_top_level_output_to_terminal_proposal() -> None:
+ ctx = _fresh_ctx_for_record()
+ ctx.last_workflow = SimpleNamespace(
+ workflow_definition=SimpleNamespace(
+ blocks=[
+ SimpleNamespace(label="open_search_search"),
+ SimpleNamespace(label="search_and_open_record_details"),
+ SimpleNamespace(label="extract_record_status_record"),
+ ]
+ )
+ )
+ ctx.last_workflow_yaml = "title: Record lookup"
+ ctx.verified_prefix_labels = ["open_search_search"]
+ result = {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_structured_record",
+ "overall_status": "completed",
+ "executed_block_labels": ["extract_record_status_record"],
+ "blocks": [
+ {
+ "label": "extract_record_status_record",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": {"extracted_information": []},
+ }
+ ],
+ "output": {
+ "search_and_open_record_details_output": {
+ "found": True,
+ "entity_name": "Jordan Example",
+ "opened_record_details": True,
+ "evidence_text": "Opened Details page for the selected record.",
+ },
+ "extract_record_status_record_output": {
+ "found": True,
+ "entity_name": "Jordan Example",
+ "record_number": "1234567890",
+ "items": [
+ {
+ "item_label": "Sample Practice",
+ "address": "100 Main St, Example City, ST 12345",
+ "status": "Active",
+ }
+ ],
+ "overall_status": "Active",
+ "evidence_text": "Opened Details page; read Overview/Affiliations items and More Details identifier.",
+ },
+ "extracted_information": [],
+ },
+ },
+ }
+ verification = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=[
+ "fallback_record_identity",
+ "fallback_record_identifier",
+ "fallback_record_groups",
+ "fallback_record_status",
+ ],
+ verdicts=[
+ CriterionVerdict(criterion_id=cid, state="satisfied", reason_code="evidence_confirms")
+ for cid in (
+ "fallback_record_identity",
+ "fallback_record_identifier",
+ "fallback_record_groups",
+ "fallback_record_status",
+ )
+ ],
+ )
+
+ _record_run_blocks_result(ctx, result, completion_verification=verification)
+
+ assert ctx.verified_terminal_proposal_ready is True
+ assert ctx.last_test_ok is True
+ assert ctx.last_full_workflow_test_ok is True
+ assert ctx.last_test_suspicious_success is False
+ assert ctx.last_test_failure_reason is None
+
+
+def test_record_run_blocks_result_resets_stale_verified_terminal_proposal_latch() -> None:
+ ctx = _fresh_ctx_for_record()
+ ctx.verified_terminal_proposal_ready = True
+ result = {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_unverified",
+ "overall_status": "completed",
+ "executed_block_labels": [],
+ "blocks": [],
+ "output": {},
+ },
+ }
+
+ _record_run_blocks_result(ctx, result, completion_verification=None)
+
+ assert ctx.verified_terminal_proposal_ready is False
+
+
+def test_enforcement_stops_after_verified_terminal_proposal() -> None:
+ ctx = _Ctx()
+ ctx.verified_terminal_proposal_ready = True
+ ctx.last_test_ok = True
+ ctx.last_full_workflow_test_ok = True
+ ctx.completion_verification_result = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=["c0"],
+ verdicts=[CriterionVerdict(criterion_id="c0", state="satisfied", reason_code="evidence_confirms")],
+ )
+ ctx.last_test_suspicious_success = True
+
+ with pytest.raises(CopilotGoalSatisfied):
+ _check_enforcement(ctx)
+
+
def test_record_run_blocks_result_keeps_failure_when_watchdog_cancel_without_timeout() -> None:
"""Stagnation/ceiling cancels mid-session must still set last_test_ok=False
so the failed-test nudge can fire — only a coincident total timeout softens
diff --git a/tests/unit/test_copilot_loop_detection_blocker.py b/tests/unit/test_copilot_loop_detection_blocker.py
index 7ad15ac09..e589d7fd9 100644
--- a/tests/unit/test_copilot_loop_detection_blocker.py
+++ b/tests/unit/test_copilot_loop_detection_blocker.py
@@ -175,11 +175,8 @@ def test_consecutive_non_evaluate_loop_ignores_loaded_result_steer() -> None:
assert dict(ctx.blocker_signal.extra) == {}
-def test_consecutive_evaluate_loop_ignores_stale_composition_page_results_without_latest_steer() -> None:
+def test_consecutive_evaluate_loop_without_composition_steer_uses_generic_copy() -> None:
ctx = _ctx(consecutive_tool_tracker=["evaluate", "evaluate"])
- ctx.composition_page_evidence = {
- "result_containers": [{"tag": "table", "selector": "#old-results", "row_count": 3}],
- }
payload = _tool_loop_error(ctx, "evaluate", None)
assert payload is not None
diff --git a/tests/unit/test_copilot_output_policy.py b/tests/unit/test_copilot_output_policy.py
index 4bbb09d58..23261fafb 100644
--- a/tests/unit/test_copilot_output_policy.py
+++ b/tests/unit/test_copilot_output_policy.py
@@ -547,6 +547,65 @@ def test_rejects_reply_when_request_policy_required_clarification() -> None:
assert OutputPolicyReason.REQUEST_POLICY_CLARIFICATION_BYPASS in verdict.reason_codes
+def test_avoidable_output_field_confirmation_is_hard_blocked() -> None:
+ policy = _policy(
+ user_response_policy="proceed",
+ completion_contract_status="present",
+ completion_criteria=[
+ SimpleNamespace(id="record_identity", outcome="The returned record identifies the target record."),
+ ],
+ )
+
+ verdict = evaluate_output_policy(
+ request_policy=policy,
+ response_type="ASK_QUESTION",
+ user_response="Please confirm the output fields before I build this record status workflow.",
+ has_workflow_proposal=False,
+ workflow_attempted=False,
+ )
+ hard = hard_block_output_policy_verdict(verdict)
+
+ assert OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION in verdict.reason_codes
+ assert hard.reason_codes == [OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION]
+
+
+def test_output_field_confirmation_allowed_when_request_policy_requires_clarification() -> None:
+ policy = _policy(
+ user_response_policy="ask_clarification",
+ clarification_question="Which saved credential should I use?",
+ clarification_reason="credential_name_unresolved",
+ completion_contract_status="present",
+ completion_criteria=[
+ SimpleNamespace(id="record_identity", outcome="The returned record identifies the target record."),
+ ],
+ )
+
+ verdict = evaluate_output_policy(
+ request_policy=policy,
+ response_type="ASK_QUESTION",
+ user_response="Which saved credential should I use?",
+ has_workflow_proposal=False,
+ workflow_attempted=False,
+ )
+
+ assert OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION not in verdict.reason_codes
+ assert verdict.allowed is True
+
+
+def test_url_clarification_does_not_trigger_output_field_confirmation_guard() -> None:
+ policy = _policy(user_response_policy="proceed", completion_contract_status="present")
+
+ verdict = evaluate_output_policy(
+ request_policy=policy,
+ response_type="ASK_QUESTION",
+ user_response="Which URL should I use for this workflow?",
+ has_workflow_proposal=False,
+ workflow_attempted=False,
+ )
+
+ assert OutputPolicyReason.AVOIDABLE_OUTPUT_FIELD_CONFIRMATION not in verdict.reason_codes
+
+
def test_classifies_unbacked_workflow_delivery_claim() -> None:
verdict = evaluate_output_policy(
request_policy=_policy(),
diff --git a/tests/unit/test_copilot_page_evidence_policy.py b/tests/unit/test_copilot_page_evidence_policy.py
index aae163f39..bc16b9b53 100644
--- a/tests/unit/test_copilot_page_evidence_policy.py
+++ b/tests/unit/test_copilot_page_evidence_policy.py
@@ -84,6 +84,14 @@ def test_agent_prompt_requires_result_page_evidence_before_expansion_blocks() ->
assert "omit the expansion block and proceed to extraction from the visible results" in rendered
+def test_agent_prompt_requires_details_evidence_label() -> None:
+ rendered = _render_agent_prompt()
+
+ assert "details or profile page" in rendered
+ assert "visible details/profile page label" in rendered
+ assert "page label only as supporting evidence" in rendered
+
+
def test_tool_descriptions_ground_composition_without_prescribing_extra_workflow_blocks() -> None:
for tool in (update_workflow_tool, run_blocks_tool, update_and_run_blocks_tool):
desc = tool.description # type: ignore[attr-defined]
diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.py
index 93691e5dd..4a59df3ab 100644
--- a/tests/unit/test_copilot_request_policy_prompt.py
+++ b/tests/unit/test_copilot_request_policy_prompt.py
@@ -6,10 +6,12 @@ from pathlib import Path
import pytest
from skyvern.forge.prompts import prompt_engine
+from skyvern.forge.sdk.copilot import request_policy as request_policy_module
from skyvern.forge.sdk.copilot.request_policy import (
PROMPT_NAME,
RAW_SECRET_REFUSAL_SENTINEL,
CompletionCriterion,
+ _classifier_fallback_policy,
_classify_request,
_credential_ids,
_raw_secret_detected,
@@ -364,3 +366,99 @@ class TestActiveCriteriaPromptAnchor:
handler=_capture_handler(captured),
)
assert "ACTIVE COMPLETION CRITERIA (canonical phrasing for the current goal):" not in captured["prompt"]
+
+
+class TestClassifierFallbackCompletionCriteria:
+ @pytest.mark.parametrize(
+ ("user_message", "expect_criteria"),
+ [
+ (
+ (
+ "I want to build a reusable workflow that checks record status. "
+ "Capture the identifier and the list of practice items with each location's status, "
+ "and the result should come out as a record with the entity name, identifier, items, "
+ "and an overall status."
+ ),
+ True,
+ ),
+ (
+ (
+ "Build a reusable fixture directory lookup workflow for Jordan Example. Return a record with "
+ "the entity name, identifier 1234567890, items, per-location status, overall status, and "
+ "no-results behavior."
+ ),
+ True,
+ ),
+ ("Extract the user's name, id, address, and status from the portal.", False),
+ ("Open https://example.com and click the pricing link.", False),
+ # Substring matches ("read" in "already", "name" in "filename", "id" in "decided")
+ # must not satisfy the whole-word gate.
+ ("I already decided the filename and reviewed the status of locations.", False),
+ # Generic structural group term ("entries") satisfies the group gate.
+ ("Return a record with the entity name, identifier 1234567890, status, and the grouped entries.", True),
+ ],
+ )
+ @pytest.mark.asyncio
+ async def test_classifier_fallback_structured_record_criteria(
+ self, user_message: str, expect_criteria: bool
+ ) -> None:
+ policy = await _classify_request(
+ user_message=user_message,
+ workflow_yaml="",
+ chat_history=[],
+ global_llm_context="",
+ handler=None,
+ )
+
+ assert policy.classifier_status == "fallback"
+ assert policy.completion_contract_status == ("present" if expect_criteria else "unknown")
+ if not expect_criteria:
+ assert policy.completion_criteria == []
+ return
+
+ assert [criterion.id for criterion in policy.completion_criteria] == [
+ "fallback_record_identity",
+ "fallback_record_identifier",
+ "fallback_record_groups",
+ "fallback_record_status",
+ ]
+ assert policy.requires_user_clarification is False
+ assert policy.user_response_policy == "proceed"
+ assert all(criterion.implicit for criterion in policy.completion_criteria)
+ assert all(criterion.level == "run" for criterion in policy.completion_criteria)
+
+ def test_classifier_fallback_logs_synthesized_structured_record_criteria(
+ self, monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ calls: list[tuple[str, dict[str, object]]] = []
+
+ class FakeLogger:
+ def info(self, event: str, **kwargs: object) -> None:
+ calls.append((event, kwargs))
+
+ monkeypatch.setattr(request_policy_module, "LOG", FakeLogger())
+
+ policy = _classifier_fallback_policy(
+ [],
+ raw_secret_present=False,
+ failure_kind="missing_handler",
+ user_message=(
+ "Return a record with the entity name, identifier, items, per-location status, and overall status."
+ ),
+ )
+
+ assert policy.completion_contract_status == "present"
+ assert calls == [
+ (
+ "copilot request policy synthesized fallback structured-record criteria",
+ {
+ "classifier_failure_kind": "missing_handler",
+ "completion_criterion_ids": [
+ "fallback_record_identity",
+ "fallback_record_identifier",
+ "fallback_record_groups",
+ "fallback_record_status",
+ ],
+ },
+ )
+ ]
diff --git a/tests/unit/test_copilot_run_outcome_event.py b/tests/unit/test_copilot_run_outcome_event.py
index af5c16881..c5fbc90bf 100644
--- a/tests/unit/test_copilot_run_outcome_event.py
+++ b/tests/unit/test_copilot_run_outcome_event.py
@@ -325,7 +325,9 @@ async def test_satisfied_adjudication_emits_demonstrated_with_unverified_prefix(
frames = _run_outcome_frames(ctx.stream) # type: ignore[arg-type]
assert [frame.verdict for frame in frames] == ["evaluating", "demonstrated"]
assert frames[-1].reason_code is None
- assert ctx.last_full_workflow_test_ok is False
+ # A fully-verified completion promotes the full-workflow test result; this is the
+ # deterministic terminal path that lets a verified run finalize success.
+ assert ctx.last_full_workflow_test_ok is True
assert ctx.last_run_outcome == RecordedRunOutcome(verdict="demonstrated", workflow_run_id="wr_test")
diff --git a/tests/unit/test_copilot_run_output_content_gate.py b/tests/unit/test_copilot_run_output_content_gate.py
index f6b5b539c..6b4b895ad 100644
--- a/tests/unit/test_copilot_run_output_content_gate.py
+++ b/tests/unit/test_copilot_run_output_content_gate.py
@@ -24,9 +24,41 @@ from skyvern.forge.sdk.copilot.tools import (
_record_run_blocks_result,
_run_blocks_structured_blocker_message,
)
+from skyvern.forge.sdk.copilot.tools._shared import _registered_output_parameter_payloads
+from skyvern.forge.sdk.copilot.tools.run_execution import _attach_registered_output_parameter_values
from skyvern.forge.sdk.copilot.turn_halt import TurnHaltKind
+ ("data", "expected"),
+ [
+ (
+ {
+ "registered_output_parameter_values": [
+ {"workflow_run_id": "wr_prior", "value": {"x": 1}},
+ {"value": {"y": 2}},
+ ]
+ },
+ [],
+ ),
+ (
+ {
+ "workflow_run_id": "wr_now",
+ "registered_output_parameter_values": [
+ {"workflow_run_id": "wr_now", "value": {"x": 1}},
+ {"workflow_run_id": "wr_prior", "value": {"y": 2}},
+ ],
+ },
+ [{"x": 1}],
+ ),
+ ],
+)
+def test_registered_output_parameter_payloads_scope_to_current_run(
+ data: dict[str, Any], expected: list[dict[str, int]]
+) -> None:
+ assert [dict(p["value"]) for p in _registered_output_parameter_payloads(data)] == expected
+
+
def _code_block(label: str, extracted: Any, *, block_type: str = "CODE") -> dict[str, Any]:
return {"label": label, "block_type": block_type, "status": "completed", "extracted_data": extracted}
@@ -43,6 +75,21 @@ def _run_result(blocks: list[dict[str, Any]], *, ok: bool = True) -> dict[str, A
}
+def _structured_record_payload(**overrides: Any) -> dict[str, Any]:
+ payload = {
+ "entity_found": True,
+ "entity_name": "Jordan Example",
+ "record_number": "1234567890",
+ "items": [
+ {"item_label": "Sample Practice", "address": "100 Main St, Example City, ST 12345", "status": "Active"}
+ ],
+ "overall_status": "Active",
+ "evidence_text": "Opened Details page",
+ }
+ payload.update(overrides)
+ return payload
+
+
def _ctx(blocks: list[dict[str, Any]] | None = None) -> CopilotContext:
ctx = CopilotContext(
organization_id="o",
@@ -69,6 +116,17 @@ def _no_evidence(cid: str) -> CompletionVerificationResult:
return CompletionVerificationResult(status="evaluated", criterion_ids=[cid], verdicts=[verdict])
+def _satisfied(*criterion_ids: str) -> CompletionVerificationResult:
+ return CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=list(criterion_ids),
+ verdicts=[
+ CriterionVerdict(criterion_id=criterion_id, state="satisfied", reason_code="evidence_confirms")
+ for criterion_id in criterion_ids
+ ],
+ )
+
+
def _blocked_flag_run_result() -> dict[str, Any]:
return _run_result(
[
@@ -168,6 +226,80 @@ def _goal_field_success_run_result() -> dict[str, Any]:
)
+def _nested_code_output_extraction_run_result() -> dict[str, Any]:
+ return _run_result(
+ [
+ _code_block(
+ "extract_record_status_info",
+ {
+ "extract_record_status_info_output": _structured_record_payload(),
+ "extracted_information": [],
+ },
+ block_type="EXTRACTION",
+ )
+ ]
+ )
+
+
+def _empty_extraction_run_result() -> dict[str, Any]:
+ return _run_result(
+ [
+ _code_block(
+ "extract_empty_results",
+ {"extracted_information": [], "downloaded_files": [], "downloaded_file_urls": None},
+ block_type="EXTRACTION",
+ )
+ ]
+ )
+
+
+def _registered_code_output_parameter_run_result(*, workflow_run_id: str = "wr_test") -> dict[str, Any]:
+ return {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_test",
+ "overall_status": "completed",
+ "blocks": [],
+ "registered_output_parameter_values": [
+ {
+ "workflow_run_id": workflow_run_id,
+ "output_parameter_id": "op_details",
+ "output_parameter_key": "extract_record_status_details_output",
+ "block_label": "extract_record_status_details",
+ "block_type": "CODE",
+ "value": _structured_record_payload(),
+ }
+ ],
+ },
+ }
+
+
+def _structured_record_qa_top_level_output_run_result() -> dict[str, Any]:
+ result = _run_result(
+ [
+ _code_block(
+ "extract_record_status_record",
+ {"extracted_information": []},
+ )
+ ]
+ )
+ result["data"]["output"] = {
+ "open_search_search_output": {
+ "page_state": "search_search_open",
+ "evidence_text": "Opened search search page with search-by-doctor typeahead #searchInput.",
+ },
+ "search_and_open_record_details_output": {
+ "found": True,
+ "entity_name": "Jordan Example",
+ "opened_record_details": True,
+ "evidence_text": "Opened Details page for the selected record.",
+ },
+ "extract_record_status_record_output": _structured_record_payload(found=True, entity_found=None),
+ "extracted_information": [],
+ }
+ return result
+
+
def _partial_goal_field_run_result() -> dict[str, Any]:
return _run_result(
[
@@ -335,6 +467,133 @@ def test_genuine_success_run_keeps_clean_path() -> None:
assert ctx.last_full_workflow_test_ok is True
+def test_nested_code_output_record_is_meaningful_and_not_suspicious_when_verified() -> None:
+ result = _nested_code_output_extraction_run_result()
+ ctx = _ctx(result["data"]["blocks"])
+
+ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)
+ assert empty_data_blocks is False
+ assert _is_outcome_evidence_candidate(ctx, result) is True
+
+ _record_run_blocks_result(
+ ctx,
+ result,
+ completion_verification=_satisfied("fallback_record_identity", "fallback_record_identifier"),
+ )
+
+ assert ctx.last_test_ok is True
+ assert ctx.last_test_suspicious_success is False
+ assert ctx.last_full_workflow_test_ok is True
+ assert verified_goal_satisfied_context(ctx) is True
+
+
+def test_registered_code_output_parameter_record_is_meaningful() -> None:
+ result = _registered_code_output_parameter_run_result()
+ ctx = _ctx([])
+
+ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)
+
+ assert empty_data_blocks is False
+
+
+def test_registered_code_output_parameter_record_is_current_run_scoped() -> None:
+ result = _empty_extraction_run_result()
+ result["data"]["registered_output_parameter_values"] = _registered_code_output_parameter_run_result(
+ workflow_run_id="wr_prior"
+ )["data"]["registered_output_parameter_values"]
+ ctx = _ctx(result["data"]["blocks"])
+
+ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)
+
+ assert empty_data_blocks is True
+
+
+def test_structured_record_top_level_output_record_is_meaningful() -> None:
+ result = _structured_record_qa_top_level_output_run_result()
+ ctx = _ctx(result["data"]["blocks"])
+
+ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)
+
+ assert empty_data_blocks is False
+ assert _is_outcome_evidence_candidate(ctx, result) is True
+
+
+async def test_registered_output_adapter_fetches_db_values_and_synthesizes_block(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ from skyvern.forge.sdk.copilot.tools import run_execution
+
+ async def fake_get_workflow_run_output_parameters(*, workflow_run_id: str) -> list[SimpleNamespace]:
+ assert workflow_run_id == "wr_test"
+ return [
+ SimpleNamespace(
+ workflow_run_id="wr_test",
+ output_parameter_id="op_details",
+ value={
+ "entity_name": "Jordan Example",
+ "record_number": "1234567890",
+ "evidence_text": "Opened Details page",
+ },
+ )
+ ]
+
+ monkeypatch.setattr(
+ run_execution.app.DATABASE,
+ "workflow_runs",
+ SimpleNamespace(get_workflow_run_output_parameters=fake_get_workflow_run_output_parameters),
+ )
+ workflow = SimpleNamespace(
+ workflow_definition=SimpleNamespace(
+ blocks=[
+ SimpleNamespace(
+ label="extract_record_status_details",
+ block_type="CODE",
+ output_parameter=SimpleNamespace(
+ output_parameter_id="op_details",
+ key="extract_record_status_details_output",
+ ),
+ )
+ ]
+ )
+ )
+ data: dict[str, Any] = {"workflow_run_id": "wr_test", "blocks": []}
+
+ by_label = await _attach_registered_output_parameter_values(
+ workflow_run_id="wr_test",
+ workflow=workflow, # type: ignore[arg-type]
+ data=data,
+ )
+
+ assert by_label == {
+ "extract_record_status_details": {
+ "extract_record_status_details_output": {
+ "entity_name": "Jordan Example",
+ "record_number": "1234567890",
+ "evidence_text": "Opened Details page",
+ }
+ }
+ }
+ assert data["registered_output_parameter_values"][0]["value"]["record_number"] == "1234567890"
+ assert data["blocks"][0]["label"] == "extract_record_status_details"
+ assert data["blocks"][0]["extracted_data"]["extract_record_status_details_output"]["record_number"] == "1234567890"
+
+
+def test_satisfied_completion_prevents_empty_output_suspicious_success() -> None:
+ result = _empty_extraction_run_result()
+ ctx = _ctx(result["data"]["blocks"])
+
+ _, empty_data_blocks, _ = _analyze_run_blocks(result, ctx)
+ assert empty_data_blocks is True
+
+ _record_run_blocks_result(ctx, result, completion_verification=_satisfied("c0"))
+
+ assert ctx.last_test_ok is True
+ assert ctx.last_test_suspicious_success is False
+ assert ctx.last_test_failure_reason is None
+ assert verified_goal_satisfied_context(ctx) is True
+
+
def test_all_null_metadata_goal_fields_are_flagged_as_no_goal_content() -> None:
result = _all_null_goal_fields_run_result()
ctx = _ctx(result["data"]["blocks"])
diff --git a/tests/unit/test_copilot_timeout_wip.py b/tests/unit/test_copilot_timeout_wip.py
index a19100e72..8645079b1 100644
--- a/tests/unit/test_copilot_timeout_wip.py
+++ b/tests/unit/test_copilot_timeout_wip.py
@@ -22,6 +22,7 @@ from skyvern.forge.sdk.copilot.agent import (
_build_timeout_exit_result,
_build_unexpected_error_exit_result,
)
+from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdict
from skyvern.forge.sdk.copilot.diagnosis_repair_contract import (
DiagnosisInput,
DiagnosisRepairContract,
@@ -152,6 +153,48 @@ class TestBuildTimeoutExitResult:
assert result.proposal_disposition == "auto_applicable"
assert result.user_response == _TIMEOUT_REPLY_DEFAULT
+ def test_verified_terminal_state_preserves_tested_proposal_on_timeout(self) -> None:
+ wf = MagicMock(name="wf")
+ ctx = _ctx(
+ last_workflow=wf,
+ last_workflow_yaml="version: '1.0'",
+ last_test_ok=None,
+ last_test_suspicious_success=True,
+ )
+ ctx.verified_terminal_proposal_ready = True
+ ctx.completion_verification_result = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=["c0"],
+ verdicts=[CriterionVerdict(criterion_id="c0", state="satisfied", reason_code="evidence_confirms")],
+ )
+
+ result = _build_timeout_exit_result(ctx, global_llm_context=None)
+
+ assert result.updated_workflow is wf
+ assert result.workflow_yaml == "version: '1.0'"
+ assert result.proposal_disposition == "auto_applicable"
+ assert result.user_response == _TIMEOUT_REPLY_TESTED
+ assert result.clear_proposed_workflow is False
+
+ def test_suspicious_current_run_drops_last_good_workflow_without_verified_terminal_state(self) -> None:
+ wf = MagicMock(name="wf")
+ last_good = MagicMock(name="last_good")
+ ctx = _ctx(
+ last_workflow=wf,
+ last_workflow_yaml="version: 'broken'",
+ last_test_ok=None,
+ last_test_suspicious_success=True,
+ last_good_workflow=last_good,
+ last_good_workflow_yaml="version: 'tested'",
+ )
+
+ result = _build_timeout_exit_result(ctx, global_llm_context=None)
+
+ assert result.updated_workflow is None
+ assert result.workflow_yaml is None
+ assert result.proposal_disposition == "auto_applicable"
+ assert result.user_response == _TIMEOUT_REPLY_DEFAULT
+
class TestBuildMaxTurnsExitResult:
def test_no_workflow_falls_back_to_default_reply(self) -> None:
diff --git a/tests/unit/test_workflow_copilot_cancel.py b/tests/unit/test_workflow_copilot_cancel.py
index 43a166c6e..14fb9542c 100644
--- a/tests/unit/test_workflow_copilot_cancel.py
+++ b/tests/unit/test_workflow_copilot_cancel.py
@@ -37,6 +37,7 @@ from skyvern.forge.sdk.routes.workflow_copilot import (
COPILOT_CANCEL_TTL,
_copilot_cancel_key,
_persist_cancel_turn,
+ _persist_proposed_workflow_state,
_watch_for_cancel,
workflow_copilot_cancel,
workflow_copilot_chat_post,
@@ -261,6 +262,42 @@ def _make_original_workflow() -> SimpleNamespace:
)
+async def test_timeout_wip_persists_proposed_workflow_with_yaml_when_canonical_restored() -> None:
+ workflow_params = SimpleNamespace(update_workflow_copilot_chat=AsyncMock())
+ app.DATABASE.workflow_params = workflow_params
+ chat = _make_chat(auto_accept=False)
+ updated_workflow = MagicMock()
+ updated_workflow.model_dump.return_value = {
+ "workflow_id": "wf-canonical",
+ "workflow_definition": {
+ "parameters": [{"key": "full_name", "parameter_type": "workflow"}],
+ "blocks": [{"block_type": "code", "label": "extract_record_status_info"}],
+ },
+ }
+ agent_result = AgentResult(
+ user_response="Timed out, but I have a tested draft.",
+ updated_workflow=updated_workflow,
+ global_llm_context=None,
+ workflow_yaml=(
+ "title: Record lookup\n"
+ "workflow_definition:\n"
+ " blocks:\n"
+ " - block_type: code\n"
+ " label: extract_record_status_info\n"
+ ),
+ proposal_disposition="review_tested",
+ )
+
+ await _persist_proposed_workflow_state(chat, agent_result, restored=True)
+
+ workflow_params.update_workflow_copilot_chat.assert_awaited_once()
+ proposed_workflow = workflow_params.update_workflow_copilot_chat.await_args.kwargs["proposed_workflow"]
+ assert proposed_workflow["workflow_definition"]["blocks"][0]["label"] == "extract_record_status_info"
+ assert proposed_workflow["_copilot_yaml"].startswith("title: Record lookup")
+ assert "_copilot_unvalidated" not in proposed_workflow
+
+
async def _drive_cancel_route(
monkeypatch: pytest.MonkeyPatch,
chat: SimpleNamespace,
@@ -665,6 +702,57 @@ async def test_timeout_wip_result_streams_normal_response_frame(
assert "Cancelled by user." not in contents
+async def test_verified_terminal_timeout_result_persists_proposed_workflow_with_blocks_and_params() -> None:
+ workflow_params = SimpleNamespace(update_workflow_copilot_chat=AsyncMock())
+ app.DATABASE.workflow_params = workflow_params
+ chat = _make_chat(proposed_workflow=None, auto_accept=False)
+ blocks = [
+ {"block_type": "code", "label": "open_search_search_page"},
+ {"block_type": "code", "label": "search_and_open_record_details"},
+ {"block_type": "code", "label": "extract_record_status_record"},
+ ]
+ params = [{"key": f"param_{index}", "parameter_type": "workflow"} for index in range(6)]
+ updated_workflow = MagicMock()
+ updated_workflow.model_dump.side_effect = lambda mode="json": {
+ "workflow_id": "wf-draft",
+ "title": "Record Status Draft",
+ "workflow_definition": {"blocks": blocks, "parameters": params},
+ }
+ agent_result = AgentResult(
+ user_response="I created and tested the workflow successfully.",
+ updated_workflow=updated_workflow,
+ global_llm_context=None,
+ workflow_yaml=(
+ "title: Record Status Draft\n"
+ "workflow_definition:\n"
+ " parameters:\n"
+ " - key: param_0\n"
+ " blocks:\n"
+ " - block_type: code\n"
+ " label: open_search_search_page\n"
+ " - block_type: code\n"
+ " label: search_and_open_record_details\n"
+ " - block_type: code\n"
+ " label: extract_record_status_record\n"
+ ),
+ workflow_was_persisted=True,
+ proposal_disposition="auto_applicable",
+ )
+
+ await _persist_proposed_workflow_state(chat, agent_result, restored=False)
+
+ proposal = workflow_params.update_workflow_copilot_chat.await_args.kwargs["proposed_workflow"]
+ assert proposal["_copilot_yaml"].startswith("title: Record Status Draft")
+ assert [block["label"] for block in proposal["workflow_definition"]["blocks"]] == [
+ "open_search_search_page",
+ "search_and_open_record_details",
+ "extract_record_status_record",
+ ]
+ assert len(proposal["workflow_definition"]["parameters"]) == 6
+ assert "_copilot_unvalidated" not in proposal
+
+
@pytest.mark.asyncio
async def test_timeout_wip_review_tested_propagates_to_response_frame(
monkeypatch: pytest.MonkeyPatch,