Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
SKY-10346: Replace goto-url heuristics with page evidence policy (PR 5/12) (#6256)
Andrew Neilsonb827c457a2a0
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index 40d259582..880edc79e 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -55,18 +55,12 @@ Prefer workflow-run evidence first. Use navigate_browser, get_browser_screenshot2. The latest workflow run evidence is missing, ambiguous, or contradictory.3. You need to validate a specific UI hypothesis before changing the workflow.-GOTO_URL STATE SHORTCUT POLICY:-Treat `goto_url` as a fast bootstrap, not as the source of truth for dynamic page state.-- URL guessing is allowed for the first pass when it gets you to a plausible results/search page quickly.-- URL refinement for dynamic page state is allowed ONLY when grounded in live site evidence: active chips, checked controls, state-changing links, form values, DOM attributes, or URL deltas observed after UI actions.-- Ungrounded URL parameter tweaking is not allowed. If the page does not verify the intended state, inspect it with a `code` or `validation` block, or apply missing state with a `navigation` block.-- For stateful search tasks with multiple stable inputs before results appear, do not spend one click-driven `navigation` block per field after the entry page loads. Before adding a second stable search-form/date/query-input block, use observed same-site evidence (current URL, form action, field names/values, result links, or URL deltas) to bootstrap stable search state with `goto_url`, or add a short `code` block that inspects the DOM and navigates/verifies the stable results page. Reserve click-driven `navigation` blocks for missing dynamic state or cases where the URL/form evidence is insufficient.-- Do not bypass the previous rule by putting the entry URL and stable search setup in the same `navigation` block. Split pure page loading into `goto_url` first, then inspect/bootstrap/apply state in later blocks.-- Before extraction on stateful search/result tasks, verify every requested constraint independently and simultaneously. Use live page evidence, not the presence of guessed query params.-- Represent broad constraints as intent, not as a single fragile URL shape. Verify the visible state that satisfies the intent instead of assuming one query parameter means the whole constraint is active.-- If a shortcut partially works, keep the useful state and apply/verify the missing constraints through UI/DOM evidence instead of looping on speculative `goto_url` variants.-- If a state-application block hits a per-tool budget, do not rerun the same navigation label unchanged. Split/replace it with a smaller one-action UI block, or use `code`/`validation` to inspect the visible DOM state before deciding what is missing.-- Put the verification evidence in the workflow itself (prefer `code` for DOM checks, `validation` for semantic/visual checks, and `navigation` only when state needs to be changed) and in `global_llm_context.decisions_made` as a per-constraint summary such as `constraint / expected / observed / source`.+PAGE EVIDENCE POLICY:+Use the Skyvern MCP browser tools to fill knowledge gaps while building, editing, or debugging a workflow. The purpose is to make workflow composition evidence-based, not to add defensive verification blocks for every website shape.+- Do not invent website-specific paths, query parameters, form fields, result affordances, or page structure from memory. Ground them in user-supplied information, run evidence, or live MCP observations.+- Treat `goto_url` as a page-loading block. A current results URL can be useful evidence, but do not assume it is a reusable entrypoint when it may encode session-only state.+- For form/search/result pages, inspect when you do not know the real controls or result layout. Use `inspect_page_for_composition`, `evaluate`, `get_browser_screenshot`, or targeted interaction tools as needed, then compose normal workflow blocks from the observed page.+- Add `validation` blocks only when the reusable workflow's task actually includes a durable check. Do not add validation or extra navigation blocks just to double-check the shape of a site before each action.CRITICAL RULES WHEN USING DIRECT BROWSER TOOLS:- If you call navigate_browser, you MUST call at least one observation tool (evaluate, get_browser_screenshot, click, type_text, scroll, select_option, press_key, or console_messages) before responding.diff --git a/skyvern/forge/sdk/copilot/tools.py b/skyvern/forge/sdk/copilot/tools.pyindex c6358fdb9..30b25ec74 100644--- a/skyvern/forge/sdk/copilot/tools.py+++ b/skyvern/forge/sdk/copilot/tools.py@@ -10,7 +10,7 @@ import timefrom collections import Counter, defaultdictfrom datetime import datetimefrom typing import Any, Literal-from urllib.parse import parse_qsl, urljoin, urlparse+from urllib.parse import urljoin, urlparseimport structlogimport yaml@@ -1544,21 +1544,6 @@ async def _update_workflow(_record_banned_block_reject_span("_update_workflow", banned_items)return {"ok": False, "error": _banned_block_reject_message(banned_items)}- goto_url_state_shortcuts = _detect_unverified_goto_url_state_shortcuts(workflow_yaml)- if goto_url_state_shortcuts:- _record_goto_url_state_shortcut_reject_span("_update_workflow", goto_url_state_shortcuts)- return {"ok": False, "error": _goto_url_state_shortcut_message(goto_url_state_shortcuts)}-- navigation_url_search_setups = _detect_navigation_url_stable_search_setups(workflow_yaml)- if navigation_url_search_setups:- _record_navigation_url_stable_search_reject_span("_update_workflow", navigation_url_search_setups)- return {"ok": False, "error": _navigation_url_stable_search_message(navigation_url_search_setups)}-- stable_search_drips = _detect_click_driven_stable_search_drips(workflow_yaml)- if stable_search_drips:- _record_goto_url_stable_search_drip_reject_span("_update_workflow", stable_search_drips)- return {"ok": False, "error": _goto_url_stable_search_drip_message(stable_search_drips)}-composition_evidence_error = composition_page_evidence_error(ctx, workflow_yaml)if composition_evidence_error:LOG.info(@@ -3008,142 +2993,6 @@ _COPILOT_BANNED_BLOCK_ALTERNATIVES = ("`login` for authentication, or `goto_url` for pure URL navigation.")-_GOTO_URL_DYNAMIC_STATE_KEY_TOKENS = frozenset(- {- "attr",- "attribute",- "category",- "constraint",- "criteria",- "criterion",- "facet",- "filter",- "option",- "refine",- "refinement",- "selected",- "selection",- "toggle",- }-)-_GOTO_URL_DYNAMIC_STATE_VALUE_TOKENS = frozenset(- {- "facet",- "filter",- "refine",- "refinement",- "selected",- "selection",- }-)-_GOTO_URL_VERIFICATION_BLOCK_TYPES = frozenset({"code", "navigation", "validation"})-_GOTO_URL_DATA_PRODUCING_BLOCK_TYPES = frozenset({"extraction", "text_prompt"})-_GOTO_URL_STATE_ACTION_TOKENS = frozenset(- {- "apply",- "check",- "choose",- "click",- "confirm",- "confirmed",- "ensure",- "inspect",- "observe",- "select",- "set",- "verify",- "verified",- "verifies",- "verifying",- }-)-_GOTO_URL_STATE_EVIDENCE_TOKENS = frozenset(- {- "active",- "aria",- "checked",- "checkbox",- "chip",- "constraint",- "control",- "criteria",- "criterion",- "dom",- "facet",- "filter",- "form",- "input",- "option",- "selected",- "state",- "url",- "value",- }-)-_GOTO_URL_CODE_EVIDENCE_TOKENS = frozenset(- {- "all_text_contents",- "attribute",- "contents",- "evaluate",- "inner_text",- "locator",- "query",- "query_selector",- "selector",- "text",- "text_content",- }-)-_GOTO_URL_DISMISSAL_TOKENS = frozenset({"banner", "cookie", "cookies", "dismiss", "modal", "popup", "popups"})-_GOTO_URL_SEARCH_SETUP_ACTION_TOKENS = frozenset(- {- "choose",- "enter",- "fill",- "input",- "pick",- "select",- "set",- "type",- }-)-_GOTO_URL_SEARCH_SETUP_CONTEXT_TOKENS = frozenset(- {- "calendar",- "checkout",- "checkin",- "date",- "dates",- "find",- "lookup",- "query",- "search",- }-)-_GOTO_URL_SEARCH_SETUP_TERMINAL_TOKENS = frozenset(- {- "active",- "checked",- "checkbox",- "chip",- "constraint",- "control",- "dom",- "extract",- "extraction",- "facet",- "filter",- "result",- "results",- "verified",- "verify",- }-)-_GOTO_URL_STRUCTURAL_STATE_SPLIT_RE = re.compile(r"[;|]")-_GOTO_URL_STRUCTURAL_ASSIGNMENT_LHS_RE = re.compile(r"[A-Za-z0-9_.:-]{1,64}")-_GOTO_URL_VALUE_SCHEME_RE = re.compile(r"^\s*[a-z][a-z0-9+.-]*:", re.IGNORECASE)-def _banned_block_reject_message(items: list[tuple[str, str]]) -> str:"""Uniform error text for the post-emission reject, sharing the@@ -3207,44 +3056,6 @@ def _record_workflow_proxy_location_span(workflow_yaml: str, workflow: Workflow)pass-def _record_goto_url_state_shortcut_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:- with copilot_span(- "goto_url_state_shortcut_reject",- data={- "source_tool": source_tool,- "items": items,- "labels": [item.get("label") for item in items],- "dynamic_state_keys": sorted({key for item in items for key in item.get("dynamic_state_keys", [])}),- },- ):- pass---def _record_goto_url_stable_search_drip_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:- with copilot_span(- "goto_url_stable_search_drip_reject",- data={- "source_tool": source_tool,- "items": items,- "entry_labels": [item.get("entry_label") for item in items],- "navigation_labels": [item.get("navigation_labels", []) for item in items],- },- ):- pass---def _record_navigation_url_stable_search_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:- with copilot_span(- "navigation_url_stable_search_reject",- data={- "source_tool": source_tool,- "items": items,- "labels": [item.get("label") for item in items],- },- ):- pass--def _collect_banned_block_items(blocks: list[Any]) -> list[tuple[str, str]]:"""Recursively walk ``blocks`` (mirroring:func:`skyvern.forge.sdk.copilot.block_goal_wrapping._wrap_blocks_in_place`)@@ -3289,384 +3100,11 @@ def _parse_workflow_blocks(yaml_str: str | None) -> list[Any] | None:return blocks if isinstance(blocks, list) else None-def _normalized_query_token(value: str) -> str:- return re.sub(r"[^a-z0-9]+", "_", value.lower()).strip("_")---def _query_token_parts(value: str) -> set[str]:- parts: set[str] = set()- for part in value.split("_"):- if not part:- continue- parts.add(part)- if len(part) > 3 and part.endswith("s"):- parts.add(part[:-1])- return parts---def _is_dynamic_state_query_key(key: str) -> bool:- return bool(_query_token_parts(key) & _GOTO_URL_DYNAMIC_STATE_KEY_TOKENS)---def _is_dynamic_state_query_value(value: str) -> bool:- return bool(_query_token_parts(value) & _GOTO_URL_DYNAMIC_STATE_VALUE_TOKENS)---def _is_url_like_query_value(value: str) -> bool:- stripped = value.strip()- if not stripped:- return False- if stripped.startswith("/") or _GOTO_URL_VALUE_SCHEME_RE.match(stripped):- return True- first_equals = stripped.find("=")- first_query = stripped.find("?")- return first_query != -1 and (first_equals == -1 or first_query < first_equals)---def _assignment_lhs(clause: str) -> str | None:- if "=" not in clause:- return None- lhs, rhs = clause.split("=", 1)- lhs = lhs.strip()- if not lhs or not rhs.strip():- return None- if not _GOTO_URL_STRUCTURAL_ASSIGNMENT_LHS_RE.fullmatch(lhs):- return None- return _normalized_query_token(lhs)---def _is_structural_dynamic_state_query_value(value: str, outer_key: str) -> bool:- if not value or _is_url_like_query_value(value):- return False-- clauses = [clause.strip() for clause in _GOTO_URL_STRUCTURAL_STATE_SPLIT_RE.split(value) if clause.strip()]- if not clauses:- return False-- assignment_lhses = [lhs for lhs in (_assignment_lhs(clause) for clause in clauses) if lhs is not None]- if len(assignment_lhses) >= 2:- return True- if len(assignment_lhses) == 1 and assignment_lhses[0] != outer_key:- return True- return False---def _dynamic_state_query_keys(url: Any) -> list[str]:- if not isinstance(url, str) or not url:- return []- try:- pairs = parse_qsl(urlparse(url).query, keep_blank_values=True)- except ValueError:- return []- if not pairs:- return []-- normalized_pairs = [(_normalized_query_token(key), _normalized_query_token(value), value) for key, value in pairs]- key_counts = Counter(key for key, _, _ in normalized_pairs if key)- dynamic_keys: set[str] = set()- for key, value, raw_value in normalized_pairs:- if not key:- continue- if _is_structural_dynamic_state_query_value(raw_value, key):- dynamic_keys.add(key)- elif _is_dynamic_state_query_key(key):- dynamic_keys.add(key)- elif key_counts[key] > 1:- dynamic_keys.add(key)- elif _is_dynamic_state_query_value(value):- dynamic_keys.add(key)- return sorted(dynamic_keys)--def _block_label_from_yaml(block: dict[str, Any]) -> str | None:label = block.get("label")return label if isinstance(label, str) and label else None-def _next_top_level_block(- *,- blocks: list[dict[str, Any]],- by_label: dict[str, dict[str, Any]],- index_by_label: dict[str, int],- current: dict[str, Any],-) -> dict[str, Any] | None:- next_label = current.get("next_block_label")- if isinstance(next_label, str) and next_label:- return by_label.get(next_label)-- label = _block_label_from_yaml(current)- if label is None:- return None- index = index_by_label.get(label)- if index is None:- return None- if index + 1 >= len(blocks):- return None- return blocks[index + 1]---def _next_data_block_without_verification(- *,- blocks: list[dict[str, Any]],- by_label: dict[str, dict[str, Any]],- index_by_label: dict[str, int],- start: dict[str, Any],-) -> dict[str, Any] | None:- seen: set[str] = set()- current = start- while True:- next_block = _next_top_level_block(- blocks=blocks,- by_label=by_label,- index_by_label=index_by_label,- current=current,- )- if next_block is None:- return None- current = next_block-- label = _block_label_from_yaml(current)- if label is not None:- if label in seen:- return None- seen.add(label)-- block_type = str(current.get("block_type", "")).strip().lower()- if _block_verifies_dynamic_goto_url_state(current):- return None- if block_type in _GOTO_URL_DATA_PRODUCING_BLOCK_TYPES:- return current---def _block_state_text(block: dict[str, Any]) -> str:- values = []- for key in (- "navigation_goal",- "complete_criterion",- "terminate_criterion",- "validation_goal",- "code",- "description",- "title",- ):- value = block.get(key)- if isinstance(value, str):- values.append(value)- return _normalized_query_token(" ".join(values))---def _block_primary_goal_text(block: dict[str, Any]) -> str:- for key in ("navigation_goal", "validation_goal", "code", "description", "title"):- value = block.get(key)- if not isinstance(value, str) or not value:- continue- first_fence = value.find("```")- if first_fence == -1:- return _normalized_query_token(value)- second_fence = value.find("```", first_fence + 3)- if second_fence != -1:- fenced = value[first_fence + 3 : second_fence].strip()- if fenced:- return _normalized_query_token(fenced)- return _normalized_query_token(value)- return ""---def _block_verifies_dynamic_goto_url_state(block: dict[str, Any]) -> bool:- block_type = str(block.get("block_type", "")).strip().lower()- if block_type not in _GOTO_URL_VERIFICATION_BLOCK_TYPES:- return False-- parts = _query_token_parts(_block_state_text(block))- if not parts:- return False-- has_state_evidence = bool(parts & _GOTO_URL_STATE_EVIDENCE_TOKENS)- if block_type == "code":- return has_state_evidence and bool(parts & _GOTO_URL_CODE_EVIDENCE_TOKENS)- if block_type == "validation":- return has_state_evidence- if parts & _GOTO_URL_DISMISSAL_TOKENS and not parts & {"active", "checked", "selected"}:- return False- return has_state_evidence and bool(parts & _GOTO_URL_STATE_ACTION_TOKENS)---def _is_stable_search_setup_navigation(block: dict[str, Any]) -> bool:- if str(block.get("block_type", "")).strip().lower() != "navigation":- return False-- parts = _query_token_parts(_block_primary_goal_text(block))- if not parts:- return False- if parts & _GOTO_URL_SEARCH_SETUP_TERMINAL_TOKENS:- return False- return bool(parts & _GOTO_URL_SEARCH_SETUP_ACTION_TOKENS) and bool(parts & _GOTO_URL_SEARCH_SETUP_CONTEXT_TOKENS)---def _detect_navigation_url_stable_search_setups(workflow_yaml: str | None) -> list[dict[str, Any]]:- blocks = _parse_workflow_blocks(workflow_yaml)- if not blocks:- return []-- findings: list[dict[str, Any]] = []- for block in blocks:- if not isinstance(block, dict):- continue- if not isinstance(block.get("url"), str) or not block["url"].strip():- continue- if not _is_stable_search_setup_navigation(block):- continue- findings.append(- {- "label": _block_label_from_yaml(block) or "<missing label>",- "url": block["url"],- }- )- return findings---def _detect_click_driven_stable_search_drips(workflow_yaml: str | None) -> list[dict[str, Any]]:- blocks = _parse_workflow_blocks(workflow_yaml)- if not blocks:- return []-- top_level_blocks = [block for block in blocks if isinstance(block, dict)]- findings: list[dict[str, Any]] = []- entry_label: str | None = None- stable_nav_labels: list[str] = []-- for block in top_level_blocks:- block_type = str(block.get("block_type", "")).strip().lower()- label = _block_label_from_yaml(block) or "<missing label>"-- if block_type == "goto_url":- entry_label = label- stable_nav_labels = []- continue-- if entry_label is None:- continue-- if block_type in {"code", "validation"}:- stable_nav_labels = []- continue-- if block_type in _GOTO_URL_DATA_PRODUCING_BLOCK_TYPES:- entry_label = None- stable_nav_labels = []- continue-- if _is_stable_search_setup_navigation(block):- stable_nav_labels.append(label)- if len(stable_nav_labels) >= 2:- findings.append(- {- "entry_label": entry_label,- "navigation_labels": stable_nav_labels.copy(),- }- )- entry_label = None- stable_nav_labels = []- continue-- return findings---def _detect_unverified_goto_url_state_shortcuts(workflow_yaml: str | None) -> list[dict[str, Any]]:- blocks = _parse_workflow_blocks(workflow_yaml)- if not blocks:- return []-- top_level_blocks = [block for block in blocks if isinstance(block, dict)]- by_label: dict[str, dict[str, Any]] = {}- index_by_label: dict[str, int] = {}- for index, block in enumerate(top_level_blocks):- label = _block_label_from_yaml(block)- if label is None:- continue- by_label[label] = block- index_by_label[label] = index-- findings: list[dict[str, Any]] = []- for block in top_level_blocks:- if str(block.get("block_type", "")).strip().lower() != "goto_url":- continue- dynamic_state_keys = _dynamic_state_query_keys(block.get("url"))- if not dynamic_state_keys:- continue- next_data_block = _next_data_block_without_verification(- blocks=top_level_blocks,- by_label=by_label,- index_by_label=index_by_label,- start=block,- )- if next_data_block is None:- continue- findings.append(- {- "label": _block_label_from_yaml(block) or "<missing label>",- "dynamic_state_keys": dynamic_state_keys,- "next_data_block_label": _block_label_from_yaml(next_data_block) or "<missing label>",- }- )- return findings---_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT = 5---def _goto_url_state_shortcut_message(items: list[dict[str, Any]]) -> str:- details = []- for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:- label = item.get("label", "?")- keys = ", ".join(str(key) for key in item.get("dynamic_state_keys", []))- next_label = item.get("next_data_block_label", "?")- details.append(f"{label}: query keys [{keys}] feed directly into {next_label}")- if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:- details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")- return (- "Workflow validation failed: goto_url may bootstrap stable page state, but URL-encoded dynamic page "- "state must be verified from the live page before data extraction. Add a code, validation, or navigation block "- "between the URL shortcut and extraction that inspects active chips, checked controls, search fields, "- "or DOM/link/form evidence; applies any missing state through UI actions when needed; and verifies all "- "requested constraints are active together. Offending shortcuts: "- f"{'; '.join(details)}"- )---def _goto_url_stable_search_drip_message(items: list[dict[str, Any]]) -> str:- details = []- for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:- entry_label = item.get("entry_label", "?")- labels = ", ".join(str(label) for label in item.get("navigation_labels", []))- details.append(f"{entry_label}: click-driven stable search setup blocks [{labels}]")- if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:- details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")- return (- "Workflow validation failed: after a goto_url entry page, do not build stateful search setup as one "- "click-driven navigation block per stable field. Use observed same-site evidence from the live page "- "(current URL, form action, field names/values, result links, or URL deltas) to bootstrap stable search "- "state with goto_url, or insert a short code/validation block that inspects the DOM and verifies/navigates "- "the stable results page. Reserve click-driven navigation for missing dynamic state or when URL/form "- f"evidence is insufficient. Offending sequence: {'; '.join(details)}"- )---def _navigation_url_stable_search_message(items: list[dict[str, Any]]) -> str:- details = []- for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:- details.append(f"{item.get('label', '?')}: navigation url={item.get('url', '?')}")- if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:- details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")- return (- "Workflow validation failed: a navigation block with a URL is combining entry-page navigation with "- "stable search setup. Split pure page loading into a goto_url block first. Then use observed same-site "- "evidence (current URL, form action, field names/values, result links, or URL deltas) to bootstrap "- "stable search state with goto_url, or add a short code/validation block to inspect the DOM before "- f"click-driven setup. Offending blocks: {'; '.join(details)}"- )--def _detect_new_banned_blocks(submitted_yaml: str,prior_workflow_yaml: str | None,@@ -4391,18 +3829,11 @@ async def update_workflow_tool(reusable domain value the user supplies, not the page widget or action usedto enter it.- If a `goto_url` URL encodes dynamic page state, do not wire it- directly to extraction. Add a `code`, `validation`, or `navigation` block- first that verifies the live page state from active chips, checked controls,- fields, or DOM/link/form evidence.-- After a `goto_url` entry page for a stateful search, do not add one- click-driven `navigation` block per stable field (query terms, dates,- search-form inputs). Use observed current URL / form action / field- names / result links / URL deltas to bootstrap stable search state with- `goto_url`, or add a short `code`/`validation` block to inspect the DOM.- Do not bypass this by combining a URL and stable search setup in a- `navigation` block; split pure loading into `goto_url` first.+ Use browser inspection while building to fill knowledge gaps about page+ controls, result layout, and current URLs. Only refine URL params when they+ are grounded in observed DOM/link/form state or observed URL deltas.+ Browser inspection is build-time context; add durable workflow blocks only+ for the reusable actions/checks the workflow actually needs."""copilot_ctx = ctx.contextarguments = {"workflow_yaml": workflow_yaml}@@ -4650,9 +4081,10 @@ async def update_and_run_blocks_tool(building, editing, or debugging the workflow. Do not invent URL params,form fields, result affordances, or page structure from memory; groundworkflow blocks in observed MCP evidence or information the user supplied.- If `goto_url` is used with URL params for dynamic page state, ground the- state in observed evidence and use `code`, `validation`, or `navigation`- only for durable checks/actions the reusable workflow actually needs.+ Only refine URL params when they are grounded in observed DOM/link/form+ state or observed URL deltas.+ Browser inspection is build-time context; add durable workflow blocks only+ for the reusable actions/checks the workflow actually needs.When inspected evidence shows an anti-bot challenge gating a disabledsubmit/search control, account for challenge resolution before submit;do not compose a click against a control observed as disabled.diff --git a/tests/unit/test_copilot_goto_url_state_policy.py b/tests/unit/test_copilot_goto_url_state_policy.pydeleted file mode 100644index fb6d196b1..000000000--- a/tests/unit/test_copilot_goto_url_state_policy.py+++ /dev/null@@ -1,420 +0,0 @@-"""Tests for Copilot's goto_url dynamic-state shortcut policy."""--from __future__ import annotations--from unittest.mock import AsyncMock, MagicMock, patch--import pytest-import yaml--from skyvern.forge.prompts import prompt_engine-from skyvern.forge.sdk.copilot.tools import (- _detect_click_driven_stable_search_drips,- _detect_navigation_url_stable_search_setups,- _detect_unverified_goto_url_state_shortcuts,- _update_workflow,- run_blocks_tool,- update_and_run_blocks_tool,- update_workflow_tool,-)--_AGENT_TEMPLATE_DEFAULTS = dict(- workflow_knowledge_base="test kb",- current_datetime="2026-01-01T00:00:00Z",- tool_usage_guide="",- security_rules="",-)---def _yaml(*blocks: dict) -> str:- return yaml.safe_dump(- {"title": "wf", "workflow_definition": {"blocks": list(blocks)}},- sort_keys=False,- )---def _render_agent_prompt() -> str:- return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)---class TestGotoUrlStateShortcutDetection:- def test_plain_goto_url_to_extraction_is_allowed(self) -> None:- submitted = _yaml(- {"block_type": "goto_url", "label": "open_home", "url": "https://example.com/news"},- {"block_type": "extraction", "label": "extract_items", "data_extraction_goal": "Extract items"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_stable_search_params_to_extraction_are_allowed(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_results",- "url": "https://example.com/search?q=budget+headphones&start=2026-08-20&end=2026-08-23",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_short_state_tokens_do_not_match_unrelated_substrings(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_embedded_report",- "url": "https://example.com/report?embedded=true&classification=public",- },- {"block_type": "extraction", "label": "extract_report", "data_extraction_goal": "Extract report"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_dynamic_state_goto_url_directly_to_extraction_is_detected(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [- {- "label": "open_dynamic_results",- "dynamic_state_keys": ["facet", "filter"],- "next_data_block_label": "extract_results",- }- ]-- def test_repeated_opaque_query_key_directly_to_extraction_is_detected(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": "https://example.com/search?q=budget+headphones&choice=wireless&choice=noise-canceling",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [- {- "label": "open_dynamic_results",- "dynamic_state_keys": ["choice"],- "next_data_block_label": "extract_results",- }- ]-- def test_dynamic_state_goto_url_with_validation_before_extraction_is_allowed(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),- },- {- "block_type": "validation",- "label": "verify_state",- "complete_criterion": (- "Verify query, dates, and each requested result constraint are all active from visible page state."- ),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_dynamic_state_goto_url_with_unrelated_navigation_before_extraction_is_detected(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": (- "https://example.com/search?q=budget+headphones&"- "state=category%3Delectronics%3Bavailability%3Din_stock"- ),- },- {- "block_type": "navigation",- "label": "dismiss_cookie_popup",- "navigation_goal": "If a cookie popup is visible, close it so the main search form is usable.",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [- {- "label": "open_dynamic_results",- "dynamic_state_keys": ["state"],- "next_data_block_label": "extract_results",- }- ]-- def test_dynamic_state_goto_url_with_filter_navigation_before_extraction_is_allowed(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": (- "https://example.com/search?q=budget+headphones&"- "state=category%3Delectronics%3Bavailability%3Din_stock"- ),- },- {- "block_type": "navigation",- "label": "apply_missing_filters",- "navigation_goal": (- "Inspect the active constraint chips and checked controls, then select any missing "- "requested options."- ),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_dynamic_state_goto_url_with_code_before_extraction_is_allowed(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),- },- {- "block_type": "code",- "label": "verify_state_from_dom",- "code": (- "checked_labels = await page.locator('[aria-checked=\"true\"], input:checked').all_text_contents()\n"- "if not checked_labels:\n"- " raise Exception('No active dynamic page state found')"- ),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_dynamic_state_goto_url_without_downstream_extraction_is_allowed(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": "https://example.com/search?q=budget+headphones&facet=color&filter=available",- }- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []-- def test_structural_state_value_directly_to_extraction_is_detected(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": (- "https://example.com/search?q=budget+headphones&"- "state=category%3Delectronics%3Bavailability%3Din_stock"- ),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [- {- "label": "open_dynamic_results",- "dynamic_state_keys": ["state"],- "next_data_block_label": "extract_results",- }- ]-- def test_single_embedded_assignment_directly_to_extraction_is_detected(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": "https://example.com/search?q=budget+headphones&f=shipping%3Dfree",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [- {- "label": "open_dynamic_results",- "dynamic_state_keys": ["f"],- "next_data_block_label": "extract_results",- }- ]-- def test_url_like_query_values_are_not_treated_as_embedded_state_assignments(self) -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_redirect",- "url": "https://example.com/start?redirect=https%3A%2F%2Fexample.org%2Fsearch%3Fx%3Dy",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []---class TestGotoUrlStableSearchDripDetection:- def test_navigation_url_with_stable_search_setup_is_detected(self) -> None:- submitted = _yaml(- {- "block_type": "navigation",- "label": "enter_query",- "url": "https://example.com/",- "navigation_goal": (- "```Dismiss any pop-up if it appears, then enter budget headphones in the product "- "search field without submitting.```"- ),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_navigation_url_stable_search_setups(submitted) == [- {"label": "enter_query", "url": "https://example.com/"}- ]-- def test_navigation_url_without_stable_search_setup_is_allowed(self) -> None:- submitted = _yaml(- {- "block_type": "navigation",- "label": "open_article_and_accept_cookies",- "url": "https://example.com/article",- "navigation_goal": "```Accept the cookie banner if present.```",- },- {"block_type": "extraction", "label": "extract_article", "data_extraction_goal": "Extract article"},- )-- assert _detect_navigation_url_stable_search_setups(submitted) == []-- def test_multiple_click_driven_stable_search_navigation_blocks_are_detected(self) -> None:- submitted = _yaml(- {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},- {- "block_type": "navigation",- "label": "enter_query",- "navigation_goal": (- "Achieve the following mini goal and once it's achieved, complete: "- "```Enter budget headphones in the product search field. Do not submit the search yet.```"- ),- },- {- "block_type": "navigation",- "label": "set_dates",- "navigation_goal": (- "Achieve the following mini goal and once it's achieved, complete: "- "```Select July 10, 2026 as the start date and July 12, 2026 as the end date.```"- ),- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_click_driven_stable_search_drips(submitted) == [- {- "entry_label": "open_search_site",- "navigation_labels": ["enter_query", "set_dates"],- }- ]-- def test_stable_search_navigation_after_dom_inspection_is_allowed(self) -> None:- submitted = _yaml(- {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},- {- "block_type": "navigation",- "label": "enter_query",- "navigation_goal": "```Enter budget headphones in the product search field.```",- },- {- "block_type": "code",- "label": "inspect_search_form",- "code": "form_action = await page.locator('form').first.get_attribute('action')",- },- {- "block_type": "navigation",- "label": "set_dates",- "navigation_goal": "```Select July 10, 2026 as the start date and July 12, 2026 as the end date.```",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_click_driven_stable_search_drips(submitted) == []-- def test_single_stable_search_navigation_before_submit_is_allowed(self) -> None:- submitted = _yaml(- {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},- {- "block_type": "navigation",- "label": "enter_query",- "navigation_goal": "```Enter budget headphones in the product search field.```",- },- {- "block_type": "navigation",- "label": "submit_search",- "navigation_goal": "```Click the Search button to submit and show results.```",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )-- assert _detect_click_driven_stable_search_drips(submitted) == []---def _ctx(prior_yaml: str | None = None) -> MagicMock:- ctx = MagicMock()- ctx.workflow_yaml = prior_yaml- ctx.workflow_id = "w_test"- ctx.workflow_permanent_id = "wpid_test"- ctx.organization_id = "o_test"- return ctx---async def test_update_workflow_rejects_unverified_goto_url_state_shortcut_and_emits_span() -> None:- submitted = _yaml(- {- "block_type": "goto_url",- "label": "open_dynamic_results",- "url": "https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available",- },- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},- )- ctx = _ctx()-- with (- patch("skyvern.forge.sdk.copilot.tools._record_goto_url_state_shortcut_reject_span") as mock_span,- patch("skyvern.forge.sdk.copilot.tools.app") as mock_app,- ):- mock_app.WORKFLOW_SERVICE.update_workflow_definition = AsyncMock()- result = await _update_workflow({"workflow_yaml": submitted}, ctx)-- assert result["ok"] is False- assert "goto_url may bootstrap stable page state" in result["error"]- assert "URL-encoded dynamic page state must be verified" in result["error"]- assert "open_dynamic_results" in result["error"]- assert "extract_results" in result["error"]- mock_app.WORKFLOW_SERVICE.update_workflow_definition.assert_not_called()- mock_span.assert_called_once()---class TestPromptAndToolDescriptions:- def test_agent_prompt_contains_goto_url_state_policy(self) -> None:- rendered = _render_agent_prompt()-- assert "GOTO_URL STATE SHORTCUT POLICY" in rendered- assert "URL refinement for dynamic page state is allowed ONLY when grounded in live site evidence" in rendered- assert "Before extraction on stateful search/result tasks" in rendered- assert "do not spend one click-driven `navigation` block per field after the entry page loads" in rendered- assert "current URL, form action, field names/values, result links, or URL deltas" in rendered- assert "use `code` for DOM checks" in rendered- assert "per-tool budget" in rendered- assert "constraint / expected / observed / source" in rendered-- def test_tool_descriptions_warn_against_direct_state_shortcut_extraction(self) -> None:- for tool in (update_workflow_tool, run_blocks_tool, update_and_run_blocks_tool):- desc = tool.description # type: ignore[attr-defined]- assert "goto_url" in desc- assert "dynamic page state" in desc- assert "code" in desc- assert "validation" in descdiff --git a/tests/unit/test_copilot_page_evidence_policy.py b/tests/unit/test_copilot_page_evidence_policy.pynew file mode 100644index 000000000..35914ed08--- /dev/null+++ b/tests/unit/test_copilot_page_evidence_policy.py@@ -0,0 +1,37 @@+"""Tests for Copilot's build-time page evidence guidance."""++from __future__ import annotations++from skyvern.forge.prompts import prompt_engine+from skyvern.forge.sdk.copilot.tools import update_and_run_blocks_tool, update_workflow_tool++_AGENT_TEMPLATE_DEFAULTS = dict(+ workflow_knowledge_base="test kb",+ current_datetime="2026-01-01T00:00:00Z",+ tool_usage_guide="",+ security_rules="",+)+++def _render_agent_prompt() -> str:+ return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)+++def test_agent_prompt_frames_inspection_as_build_time_context_not_workflow_shape_policy() -> None:+ rendered = _render_agent_prompt()++ assert "PAGE EVIDENCE POLICY" in rendered+ assert "fill knowledge gaps while building, editing, or debugging a workflow" in rendered+ assert "not to add defensive verification blocks for every website shape" in rendered+ assert "Do not invent website-specific paths, query parameters, form fields" in rendered+ assert "GOTO_URL STATE SHORTCUT POLICY" not in rendered+ assert "Before extraction on stateful search/result tasks" not in rendered+++def test_tool_descriptions_ground_composition_without_prescribing_extra_workflow_blocks() -> None:+ for tool in (update_workflow_tool, update_and_run_blocks_tool):+ desc = tool.description # type: ignore[attr-defined]+ assert "browser inspection" in desc+ assert "build-time context" in desc+ assert "validation` or `navigation` block" not in desc+ assert "verify every requested constraint" not in desc