Skip to content

Automate browser based workflows with AI

SKY-10346: Replace goto-url heuristics with page evidence policy (PR 5/12) (#6256)

Andrew Neilsonb827c457a2a0
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index 40d259582..880edc79e 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -55,18 +55,12 @@ Prefer workflow-run evidence first. Use navigate_browser, get_browser_screenshot
2. The latest workflow run evidence is missing, ambiguous, or contradictory.
3. You need to validate a specific UI hypothesis before changing the workflow.
-GOTO_URL STATE SHORTCUT POLICY:
-Treat `goto_url` as a fast bootstrap, not as the source of truth for dynamic page state.
-- URL guessing is allowed for the first pass when it gets you to a plausible results/search page quickly.
-- URL refinement for dynamic page state is allowed ONLY when grounded in live site evidence: active chips, checked controls, state-changing links, form values, DOM attributes, or URL deltas observed after UI actions.
-- Ungrounded URL parameter tweaking is not allowed. If the page does not verify the intended state, inspect it with a `code` or `validation` block, or apply missing state with a `navigation` block.
-- For stateful search tasks with multiple stable inputs before results appear, do not spend one click-driven `navigation` block per field after the entry page loads. Before adding a second stable search-form/date/query-input block, use observed same-site evidence (current URL, form action, field names/values, result links, or URL deltas) to bootstrap stable search state with `goto_url`, or add a short `code` block that inspects the DOM and navigates/verifies the stable results page. Reserve click-driven `navigation` blocks for missing dynamic state or cases where the URL/form evidence is insufficient.
-- Do not bypass the previous rule by putting the entry URL and stable search setup in the same `navigation` block. Split pure page loading into `goto_url` first, then inspect/bootstrap/apply state in later blocks.
-- Before extraction on stateful search/result tasks, verify every requested constraint independently and simultaneously. Use live page evidence, not the presence of guessed query params.
-- Represent broad constraints as intent, not as a single fragile URL shape. Verify the visible state that satisfies the intent instead of assuming one query parameter means the whole constraint is active.
-- If a shortcut partially works, keep the useful state and apply/verify the missing constraints through UI/DOM evidence instead of looping on speculative `goto_url` variants.
-- If a state-application block hits a per-tool budget, do not rerun the same navigation label unchanged. Split/replace it with a smaller one-action UI block, or use `code`/`validation` to inspect the visible DOM state before deciding what is missing.
-- Put the verification evidence in the workflow itself (prefer `code` for DOM checks, `validation` for semantic/visual checks, and `navigation` only when state needs to be changed) and in `global_llm_context.decisions_made` as a per-constraint summary such as `constraint / expected / observed / source`.
+PAGE EVIDENCE POLICY:
+Use the Skyvern MCP browser tools to fill knowledge gaps while building, editing, or debugging a workflow. The purpose is to make workflow composition evidence-based, not to add defensive verification blocks for every website shape.
+- Do not invent website-specific paths, query parameters, form fields, result affordances, or page structure from memory. Ground them in user-supplied information, run evidence, or live MCP observations.
+- Treat `goto_url` as a page-loading block. A current results URL can be useful evidence, but do not assume it is a reusable entrypoint when it may encode session-only state.
+- For form/search/result pages, inspect when you do not know the real controls or result layout. Use `inspect_page_for_composition`, `evaluate`, `get_browser_screenshot`, or targeted interaction tools as needed, then compose normal workflow blocks from the observed page.
+- Add `validation` blocks only when the reusable workflow's task actually includes a durable check. Do not add validation or extra navigation blocks just to double-check the shape of a site before each action.
CRITICAL RULES WHEN USING DIRECT BROWSER TOOLS:
- If you call navigate_browser, you MUST call at least one observation tool (evaluate, get_browser_screenshot, click, type_text, scroll, select_option, press_key, or console_messages) before responding.
diff --git a/skyvern/forge/sdk/copilot/tools.py b/skyvern/forge/sdk/copilot/tools.py
index c6358fdb9..30b25ec74 100644
--- a/skyvern/forge/sdk/copilot/tools.py
+++ b/skyvern/forge/sdk/copilot/tools.py
@@ -10,7 +10,7 @@ import time
from collections import Counter, defaultdict
from datetime import datetime
from typing import Any, Literal
-from urllib.parse import parse_qsl, urljoin, urlparse
+from urllib.parse import urljoin, urlparse
import structlog
import yaml
@@ -1544,21 +1544,6 @@ async def _update_workflow(
_record_banned_block_reject_span("_update_workflow", banned_items)
return {"ok": False, "error": _banned_block_reject_message(banned_items)}
- goto_url_state_shortcuts = _detect_unverified_goto_url_state_shortcuts(workflow_yaml)
- if goto_url_state_shortcuts:
- _record_goto_url_state_shortcut_reject_span("_update_workflow", goto_url_state_shortcuts)
- return {"ok": False, "error": _goto_url_state_shortcut_message(goto_url_state_shortcuts)}
-
- navigation_url_search_setups = _detect_navigation_url_stable_search_setups(workflow_yaml)
- if navigation_url_search_setups:
- _record_navigation_url_stable_search_reject_span("_update_workflow", navigation_url_search_setups)
- return {"ok": False, "error": _navigation_url_stable_search_message(navigation_url_search_setups)}
-
- stable_search_drips = _detect_click_driven_stable_search_drips(workflow_yaml)
- if stable_search_drips:
- _record_goto_url_stable_search_drip_reject_span("_update_workflow", stable_search_drips)
- return {"ok": False, "error": _goto_url_stable_search_drip_message(stable_search_drips)}
-
composition_evidence_error = composition_page_evidence_error(ctx, workflow_yaml)
if composition_evidence_error:
LOG.info(
@@ -3008,142 +2993,6 @@ _COPILOT_BANNED_BLOCK_ALTERNATIVES = (
"`login` for authentication, or `goto_url` for pure URL navigation."
)
-_GOTO_URL_DYNAMIC_STATE_KEY_TOKENS = frozenset(
- {
- "attr",
- "attribute",
- "category",
- "constraint",
- "criteria",
- "criterion",
- "facet",
- "filter",
- "option",
- "refine",
- "refinement",
- "selected",
- "selection",
- "toggle",
- }
-)
-_GOTO_URL_DYNAMIC_STATE_VALUE_TOKENS = frozenset(
- {
- "facet",
- "filter",
- "refine",
- "refinement",
- "selected",
- "selection",
- }
-)
-_GOTO_URL_VERIFICATION_BLOCK_TYPES = frozenset({"code", "navigation", "validation"})
-_GOTO_URL_DATA_PRODUCING_BLOCK_TYPES = frozenset({"extraction", "text_prompt"})
-_GOTO_URL_STATE_ACTION_TOKENS = frozenset(
- {
- "apply",
- "check",
- "choose",
- "click",
- "confirm",
- "confirmed",
- "ensure",
- "inspect",
- "observe",
- "select",
- "set",
- "verify",
- "verified",
- "verifies",
- "verifying",
- }
-)
-_GOTO_URL_STATE_EVIDENCE_TOKENS = frozenset(
- {
- "active",
- "aria",
- "checked",
- "checkbox",
- "chip",
- "constraint",
- "control",
- "criteria",
- "criterion",
- "dom",
- "facet",
- "filter",
- "form",
- "input",
- "option",
- "selected",
- "state",
- "url",
- "value",
- }
-)
-_GOTO_URL_CODE_EVIDENCE_TOKENS = frozenset(
- {
- "all_text_contents",
- "attribute",
- "contents",
- "evaluate",
- "inner_text",
- "locator",
- "query",
- "query_selector",
- "selector",
- "text",
- "text_content",
- }
-)
-_GOTO_URL_DISMISSAL_TOKENS = frozenset({"banner", "cookie", "cookies", "dismiss", "modal", "popup", "popups"})
-_GOTO_URL_SEARCH_SETUP_ACTION_TOKENS = frozenset(
- {
- "choose",
- "enter",
- "fill",
- "input",
- "pick",
- "select",
- "set",
- "type",
- }
-)
-_GOTO_URL_SEARCH_SETUP_CONTEXT_TOKENS = frozenset(
- {
- "calendar",
- "checkout",
- "checkin",
- "date",
- "dates",
- "find",
- "lookup",
- "query",
- "search",
- }
-)
-_GOTO_URL_SEARCH_SETUP_TERMINAL_TOKENS = frozenset(
- {
- "active",
- "checked",
- "checkbox",
- "chip",
- "constraint",
- "control",
- "dom",
- "extract",
- "extraction",
- "facet",
- "filter",
- "result",
- "results",
- "verified",
- "verify",
- }
-)
-_GOTO_URL_STRUCTURAL_STATE_SPLIT_RE = re.compile(r"[;|]")
-_GOTO_URL_STRUCTURAL_ASSIGNMENT_LHS_RE = re.compile(r"[A-Za-z0-9_.:-]{1,64}")
-_GOTO_URL_VALUE_SCHEME_RE = re.compile(r"^\s*[a-z][a-z0-9+.-]*:", re.IGNORECASE)
-
def _banned_block_reject_message(items: list[tuple[str, str]]) -> str:
"""Uniform error text for the post-emission reject, sharing the
@@ -3207,44 +3056,6 @@ def _record_workflow_proxy_location_span(workflow_yaml: str, workflow: Workflow)
pass
-def _record_goto_url_state_shortcut_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:
- with copilot_span(
- "goto_url_state_shortcut_reject",
- data={
- "source_tool": source_tool,
- "items": items,
- "labels": [item.get("label") for item in items],
- "dynamic_state_keys": sorted({key for item in items for key in item.get("dynamic_state_keys", [])}),
- },
- ):
- pass
-
-
-def _record_goto_url_stable_search_drip_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:
- with copilot_span(
- "goto_url_stable_search_drip_reject",
- data={
- "source_tool": source_tool,
- "items": items,
- "entry_labels": [item.get("entry_label") for item in items],
- "navigation_labels": [item.get("navigation_labels", []) for item in items],
- },
- ):
- pass
-
-
-def _record_navigation_url_stable_search_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:
- with copilot_span(
- "navigation_url_stable_search_reject",
- data={
- "source_tool": source_tool,
- "items": items,
- "labels": [item.get("label") for item in items],
- },
- ):
- pass
-
-
def _collect_banned_block_items(blocks: list[Any]) -> list[tuple[str, str]]:
"""Recursively walk ``blocks`` (mirroring
:func:`skyvern.forge.sdk.copilot.block_goal_wrapping._wrap_blocks_in_place`)
@@ -3289,384 +3100,11 @@ def _parse_workflow_blocks(yaml_str: str | None) -> list[Any] | None:
return blocks if isinstance(blocks, list) else None
-def _normalized_query_token(value: str) -> str:
- return re.sub(r"[^a-z0-9]+", "_", value.lower()).strip("_")
-
-
-def _query_token_parts(value: str) -> set[str]:
- parts: set[str] = set()
- for part in value.split("_"):
- if not part:
- continue
- parts.add(part)
- if len(part) > 3 and part.endswith("s"):
- parts.add(part[:-1])
- return parts
-
-
-def _is_dynamic_state_query_key(key: str) -> bool:
- return bool(_query_token_parts(key) & _GOTO_URL_DYNAMIC_STATE_KEY_TOKENS)
-
-
-def _is_dynamic_state_query_value(value: str) -> bool:
- return bool(_query_token_parts(value) & _GOTO_URL_DYNAMIC_STATE_VALUE_TOKENS)
-
-
-def _is_url_like_query_value(value: str) -> bool:
- stripped = value.strip()
- if not stripped:
- return False
- if stripped.startswith("/") or _GOTO_URL_VALUE_SCHEME_RE.match(stripped):
- return True
- first_equals = stripped.find("=")
- first_query = stripped.find("?")
- return first_query != -1 and (first_equals == -1 or first_query < first_equals)
-
-
-def _assignment_lhs(clause: str) -> str | None:
- if "=" not in clause:
- return None
- lhs, rhs = clause.split("=", 1)
- lhs = lhs.strip()
- if not lhs or not rhs.strip():
- return None
- if not _GOTO_URL_STRUCTURAL_ASSIGNMENT_LHS_RE.fullmatch(lhs):
- return None
- return _normalized_query_token(lhs)
-
-
-def _is_structural_dynamic_state_query_value(value: str, outer_key: str) -> bool:
- if not value or _is_url_like_query_value(value):
- return False
-
- clauses = [clause.strip() for clause in _GOTO_URL_STRUCTURAL_STATE_SPLIT_RE.split(value) if clause.strip()]
- if not clauses:
- return False
-
- assignment_lhses = [lhs for lhs in (_assignment_lhs(clause) for clause in clauses) if lhs is not None]
- if len(assignment_lhses) >= 2:
- return True
- if len(assignment_lhses) == 1 and assignment_lhses[0] != outer_key:
- return True
- return False
-
-
-def _dynamic_state_query_keys(url: Any) -> list[str]:
- if not isinstance(url, str) or not url:
- return []
- try:
- pairs = parse_qsl(urlparse(url).query, keep_blank_values=True)
- except ValueError:
- return []
- if not pairs:
- return []
-
- normalized_pairs = [(_normalized_query_token(key), _normalized_query_token(value), value) for key, value in pairs]
- key_counts = Counter(key for key, _, _ in normalized_pairs if key)
- dynamic_keys: set[str] = set()
- for key, value, raw_value in normalized_pairs:
- if not key:
- continue
- if _is_structural_dynamic_state_query_value(raw_value, key):
- dynamic_keys.add(key)
- elif _is_dynamic_state_query_key(key):
- dynamic_keys.add(key)
- elif key_counts[key] > 1:
- dynamic_keys.add(key)
- elif _is_dynamic_state_query_value(value):
- dynamic_keys.add(key)
- return sorted(dynamic_keys)
-
-
def _block_label_from_yaml(block: dict[str, Any]) -> str | None:
label = block.get("label")
return label if isinstance(label, str) and label else None
-def _next_top_level_block(
- *,
- blocks: list[dict[str, Any]],
- by_label: dict[str, dict[str, Any]],
- index_by_label: dict[str, int],
- current: dict[str, Any],
-) -> dict[str, Any] | None:
- next_label = current.get("next_block_label")
- if isinstance(next_label, str) and next_label:
- return by_label.get(next_label)
-
- label = _block_label_from_yaml(current)
- if label is None:
- return None
- index = index_by_label.get(label)
- if index is None:
- return None
- if index + 1 >= len(blocks):
- return None
- return blocks[index + 1]
-
-
-def _next_data_block_without_verification(
- *,
- blocks: list[dict[str, Any]],
- by_label: dict[str, dict[str, Any]],
- index_by_label: dict[str, int],
- start: dict[str, Any],
-) -> dict[str, Any] | None:
- seen: set[str] = set()
- current = start
- while True:
- next_block = _next_top_level_block(
- blocks=blocks,
- by_label=by_label,
- index_by_label=index_by_label,
- current=current,
- )
- if next_block is None:
- return None
- current = next_block
-
- label = _block_label_from_yaml(current)
- if label is not None:
- if label in seen:
- return None
- seen.add(label)
-
- block_type = str(current.get("block_type", "")).strip().lower()
- if _block_verifies_dynamic_goto_url_state(current):
- return None
- if block_type in _GOTO_URL_DATA_PRODUCING_BLOCK_TYPES:
- return current
-
-
-def _block_state_text(block: dict[str, Any]) -> str:
- values = []
- for key in (
- "navigation_goal",
- "complete_criterion",
- "terminate_criterion",
- "validation_goal",
- "code",
- "description",
- "title",
- ):
- value = block.get(key)
- if isinstance(value, str):
- values.append(value)
- return _normalized_query_token(" ".join(values))
-
-
-def _block_primary_goal_text(block: dict[str, Any]) -> str:
- for key in ("navigation_goal", "validation_goal", "code", "description", "title"):
- value = block.get(key)
- if not isinstance(value, str) or not value:
- continue
- first_fence = value.find("```")
- if first_fence == -1:
- return _normalized_query_token(value)
- second_fence = value.find("```", first_fence + 3)
- if second_fence != -1:
- fenced = value[first_fence + 3 : second_fence].strip()
- if fenced:
- return _normalized_query_token(fenced)
- return _normalized_query_token(value)
- return ""
-
-
-def _block_verifies_dynamic_goto_url_state(block: dict[str, Any]) -> bool:
- block_type = str(block.get("block_type", "")).strip().lower()
- if block_type not in _GOTO_URL_VERIFICATION_BLOCK_TYPES:
- return False
-
- parts = _query_token_parts(_block_state_text(block))
- if not parts:
- return False
-
- has_state_evidence = bool(parts & _GOTO_URL_STATE_EVIDENCE_TOKENS)
- if block_type == "code":
- return has_state_evidence and bool(parts & _GOTO_URL_CODE_EVIDENCE_TOKENS)
- if block_type == "validation":
- return has_state_evidence
- if parts & _GOTO_URL_DISMISSAL_TOKENS and not parts & {"active", "checked", "selected"}:
- return False
- return has_state_evidence and bool(parts & _GOTO_URL_STATE_ACTION_TOKENS)
-
-
-def _is_stable_search_setup_navigation(block: dict[str, Any]) -> bool:
- if str(block.get("block_type", "")).strip().lower() != "navigation":
- return False
-
- parts = _query_token_parts(_block_primary_goal_text(block))
- if not parts:
- return False
- if parts & _GOTO_URL_SEARCH_SETUP_TERMINAL_TOKENS:
- return False
- return bool(parts & _GOTO_URL_SEARCH_SETUP_ACTION_TOKENS) and bool(parts & _GOTO_URL_SEARCH_SETUP_CONTEXT_TOKENS)
-
-
-def _detect_navigation_url_stable_search_setups(workflow_yaml: str | None) -> list[dict[str, Any]]:
- blocks = _parse_workflow_blocks(workflow_yaml)
- if not blocks:
- return []
-
- findings: list[dict[str, Any]] = []
- for block in blocks:
- if not isinstance(block, dict):
- continue
- if not isinstance(block.get("url"), str) or not block["url"].strip():
- continue
- if not _is_stable_search_setup_navigation(block):
- continue
- findings.append(
- {
- "label": _block_label_from_yaml(block) or "<missing label>",
- "url": block["url"],
- }
- )
- return findings
-
-
-def _detect_click_driven_stable_search_drips(workflow_yaml: str | None) -> list[dict[str, Any]]:
- blocks = _parse_workflow_blocks(workflow_yaml)
- if not blocks:
- return []
-
- top_level_blocks = [block for block in blocks if isinstance(block, dict)]
- findings: list[dict[str, Any]] = []
- entry_label: str | None = None
- stable_nav_labels: list[str] = []
-
- for block in top_level_blocks:
- block_type = str(block.get("block_type", "")).strip().lower()
- label = _block_label_from_yaml(block) or "<missing label>"
-
- if block_type == "goto_url":
- entry_label = label
- stable_nav_labels = []
- continue
-
- if entry_label is None:
- continue
-
- if block_type in {"code", "validation"}:
- stable_nav_labels = []
- continue
-
- if block_type in _GOTO_URL_DATA_PRODUCING_BLOCK_TYPES:
- entry_label = None
- stable_nav_labels = []
- continue
-
- if _is_stable_search_setup_navigation(block):
- stable_nav_labels.append(label)
- if len(stable_nav_labels) >= 2:
- findings.append(
- {
- "entry_label": entry_label,
- "navigation_labels": stable_nav_labels.copy(),
- }
- )
- entry_label = None
- stable_nav_labels = []
- continue
-
- return findings
-
-
-def _detect_unverified_goto_url_state_shortcuts(workflow_yaml: str | None) -> list[dict[str, Any]]:
- blocks = _parse_workflow_blocks(workflow_yaml)
- if not blocks:
- return []
-
- top_level_blocks = [block for block in blocks if isinstance(block, dict)]
- by_label: dict[str, dict[str, Any]] = {}
- index_by_label: dict[str, int] = {}
- for index, block in enumerate(top_level_blocks):
- label = _block_label_from_yaml(block)
- if label is None:
- continue
- by_label[label] = block
- index_by_label[label] = index
-
- findings: list[dict[str, Any]] = []
- for block in top_level_blocks:
- if str(block.get("block_type", "")).strip().lower() != "goto_url":
- continue
- dynamic_state_keys = _dynamic_state_query_keys(block.get("url"))
- if not dynamic_state_keys:
- continue
- next_data_block = _next_data_block_without_verification(
- blocks=top_level_blocks,
- by_label=by_label,
- index_by_label=index_by_label,
- start=block,
- )
- if next_data_block is None:
- continue
- findings.append(
- {
- "label": _block_label_from_yaml(block) or "<missing label>",
- "dynamic_state_keys": dynamic_state_keys,
- "next_data_block_label": _block_label_from_yaml(next_data_block) or "<missing label>",
- }
- )
- return findings
-
-
-_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT = 5
-
-
-def _goto_url_state_shortcut_message(items: list[dict[str, Any]]) -> str:
- details = []
- for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:
- label = item.get("label", "?")
- keys = ", ".join(str(key) for key in item.get("dynamic_state_keys", []))
- next_label = item.get("next_data_block_label", "?")
- details.append(f"{label}: query keys [{keys}] feed directly into {next_label}")
- if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:
- details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")
- return (
- "Workflow validation failed: goto_url may bootstrap stable page state, but URL-encoded dynamic page "
- "state must be verified from the live page before data extraction. Add a code, validation, or navigation block "
- "between the URL shortcut and extraction that inspects active chips, checked controls, search fields, "
- "or DOM/link/form evidence; applies any missing state through UI actions when needed; and verifies all "
- "requested constraints are active together. Offending shortcuts: "
- f"{'; '.join(details)}"
- )
-
-
-def _goto_url_stable_search_drip_message(items: list[dict[str, Any]]) -> str:
- details = []
- for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:
- entry_label = item.get("entry_label", "?")
- labels = ", ".join(str(label) for label in item.get("navigation_labels", []))
- details.append(f"{entry_label}: click-driven stable search setup blocks [{labels}]")
- if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:
- details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")
- return (
- "Workflow validation failed: after a goto_url entry page, do not build stateful search setup as one "
- "click-driven navigation block per stable field. Use observed same-site evidence from the live page "
- "(current URL, form action, field names/values, result links, or URL deltas) to bootstrap stable search "
- "state with goto_url, or insert a short code/validation block that inspects the DOM and verifies/navigates "
- "the stable results page. Reserve click-driven navigation for missing dynamic state or when URL/form "
- f"evidence is insufficient. Offending sequence: {'; '.join(details)}"
- )
-
-
-def _navigation_url_stable_search_message(items: list[dict[str, Any]]) -> str:
- details = []
- for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:
- details.append(f"{item.get('label', '?')}: navigation url={item.get('url', '?')}")
- if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:
- details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")
- return (
- "Workflow validation failed: a navigation block with a URL is combining entry-page navigation with "
- "stable search setup. Split pure page loading into a goto_url block first. Then use observed same-site "
- "evidence (current URL, form action, field names/values, result links, or URL deltas) to bootstrap "
- "stable search state with goto_url, or add a short code/validation block to inspect the DOM before "
- f"click-driven setup. Offending blocks: {'; '.join(details)}"
- )
-
-
def _detect_new_banned_blocks(
submitted_yaml: str,
prior_workflow_yaml: str | None,
@@ -4391,18 +3829,11 @@ async def update_workflow_tool(
reusable domain value the user supplies, not the page widget or action used
to enter it.
- If a `goto_url` URL encodes dynamic page state, do not wire it
- directly to extraction. Add a `code`, `validation`, or `navigation` block
- first that verifies the live page state from active chips, checked controls,
- fields, or DOM/link/form evidence.
-
- After a `goto_url` entry page for a stateful search, do not add one
- click-driven `navigation` block per stable field (query terms, dates,
- search-form inputs). Use observed current URL / form action / field
- names / result links / URL deltas to bootstrap stable search state with
- `goto_url`, or add a short `code`/`validation` block to inspect the DOM.
- Do not bypass this by combining a URL and stable search setup in a
- `navigation` block; split pure loading into `goto_url` first.
+ Use browser inspection while building to fill knowledge gaps about page
+ controls, result layout, and current URLs. Only refine URL params when they
+ are grounded in observed DOM/link/form state or observed URL deltas.
+ Browser inspection is build-time context; add durable workflow blocks only
+ for the reusable actions/checks the workflow actually needs.
"""
copilot_ctx = ctx.context
arguments = {"workflow_yaml": workflow_yaml}
@@ -4650,9 +4081,10 @@ async def update_and_run_blocks_tool(
building, editing, or debugging the workflow. Do not invent URL params,
form fields, result affordances, or page structure from memory; ground
workflow blocks in observed MCP evidence or information the user supplied.
- If `goto_url` is used with URL params for dynamic page state, ground the
- state in observed evidence and use `code`, `validation`, or `navigation`
- only for durable checks/actions the reusable workflow actually needs.
+ Only refine URL params when they are grounded in observed DOM/link/form
+ state or observed URL deltas.
+ Browser inspection is build-time context; add durable workflow blocks only
+ for the reusable actions/checks the workflow actually needs.
When inspected evidence shows an anti-bot challenge gating a disabled
submit/search control, account for challenge resolution before submit;
do not compose a click against a control observed as disabled.
diff --git a/tests/unit/test_copilot_goto_url_state_policy.py b/tests/unit/test_copilot_goto_url_state_policy.py
deleted file mode 100644
index fb6d196b1..000000000
--- a/tests/unit/test_copilot_goto_url_state_policy.py
+++ /dev/null
@@ -1,420 +0,0 @@
-"""Tests for Copilot's goto_url dynamic-state shortcut policy."""
-
-from __future__ import annotations
-
-from unittest.mock import AsyncMock, MagicMock, patch
-
-import pytest
-import yaml
-
-from skyvern.forge.prompts import prompt_engine
-from skyvern.forge.sdk.copilot.tools import (
- _detect_click_driven_stable_search_drips,
- _detect_navigation_url_stable_search_setups,
- _detect_unverified_goto_url_state_shortcuts,
- _update_workflow,
- run_blocks_tool,
- update_and_run_blocks_tool,
- update_workflow_tool,
-)
-
-_AGENT_TEMPLATE_DEFAULTS = dict(
- workflow_knowledge_base="test kb",
- current_datetime="2026-01-01T00:00:00Z",
- tool_usage_guide="",
- security_rules="",
-)
-
-
-def _yaml(*blocks: dict) -> str:
- return yaml.safe_dump(
- {"title": "wf", "workflow_definition": {"blocks": list(blocks)}},
- sort_keys=False,
- )
-
-
-def _render_agent_prompt() -> str:
- return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)
-
-
-class TestGotoUrlStateShortcutDetection:
- def test_plain_goto_url_to_extraction_is_allowed(self) -> None:
- submitted = _yaml(
- {"block_type": "goto_url", "label": "open_home", "url": "https://example.com/news"},
- {"block_type": "extraction", "label": "extract_items", "data_extraction_goal": "Extract items"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_stable_search_params_to_extraction_are_allowed(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_results",
- "url": "https://example.com/search?q=budget+headphones&start=2026-08-20&end=2026-08-23",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_short_state_tokens_do_not_match_unrelated_substrings(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_embedded_report",
- "url": "https://example.com/report?embedded=true&classification=public",
- },
- {"block_type": "extraction", "label": "extract_report", "data_extraction_goal": "Extract report"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_dynamic_state_goto_url_directly_to_extraction_is_detected(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
- {
- "label": "open_dynamic_results",
- "dynamic_state_keys": ["facet", "filter"],
- "next_data_block_label": "extract_results",
- }
- ]
-
- def test_repeated_opaque_query_key_directly_to_extraction_is_detected(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": "https://example.com/search?q=budget+headphones&choice=wireless&choice=noise-canceling",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
- {
- "label": "open_dynamic_results",
- "dynamic_state_keys": ["choice"],
- "next_data_block_label": "extract_results",
- }
- ]
-
- def test_dynamic_state_goto_url_with_validation_before_extraction_is_allowed(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),
- },
- {
- "block_type": "validation",
- "label": "verify_state",
- "complete_criterion": (
- "Verify query, dates, and each requested result constraint are all active from visible page state."
- ),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_dynamic_state_goto_url_with_unrelated_navigation_before_extraction_is_detected(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": (
- "https://example.com/search?q=budget+headphones&"
- "state=category%3Delectronics%3Bavailability%3Din_stock"
- ),
- },
- {
- "block_type": "navigation",
- "label": "dismiss_cookie_popup",
- "navigation_goal": "If a cookie popup is visible, close it so the main search form is usable.",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
- {
- "label": "open_dynamic_results",
- "dynamic_state_keys": ["state"],
- "next_data_block_label": "extract_results",
- }
- ]
-
- def test_dynamic_state_goto_url_with_filter_navigation_before_extraction_is_allowed(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": (
- "https://example.com/search?q=budget+headphones&"
- "state=category%3Delectronics%3Bavailability%3Din_stock"
- ),
- },
- {
- "block_type": "navigation",
- "label": "apply_missing_filters",
- "navigation_goal": (
- "Inspect the active constraint chips and checked controls, then select any missing "
- "requested options."
- ),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_dynamic_state_goto_url_with_code_before_extraction_is_allowed(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),
- },
- {
- "block_type": "code",
- "label": "verify_state_from_dom",
- "code": (
- "checked_labels = await page.locator('[aria-checked=\"true\"], input:checked').all_text_contents()\n"
- "if not checked_labels:\n"
- " raise Exception('No active dynamic page state found')"
- ),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_dynamic_state_goto_url_without_downstream_extraction_is_allowed(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": "https://example.com/search?q=budget+headphones&facet=color&filter=available",
- }
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
- def test_structural_state_value_directly_to_extraction_is_detected(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": (
- "https://example.com/search?q=budget+headphones&"
- "state=category%3Delectronics%3Bavailability%3Din_stock"
- ),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
- {
- "label": "open_dynamic_results",
- "dynamic_state_keys": ["state"],
- "next_data_block_label": "extract_results",
- }
- ]
-
- def test_single_embedded_assignment_directly_to_extraction_is_detected(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": "https://example.com/search?q=budget+headphones&f=shipping%3Dfree",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
- {
- "label": "open_dynamic_results",
- "dynamic_state_keys": ["f"],
- "next_data_block_label": "extract_results",
- }
- ]
-
- def test_url_like_query_values_are_not_treated_as_embedded_state_assignments(self) -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_redirect",
- "url": "https://example.com/start?redirect=https%3A%2F%2Fexample.org%2Fsearch%3Fx%3Dy",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
-
-
-class TestGotoUrlStableSearchDripDetection:
- def test_navigation_url_with_stable_search_setup_is_detected(self) -> None:
- submitted = _yaml(
- {
- "block_type": "navigation",
- "label": "enter_query",
- "url": "https://example.com/",
- "navigation_goal": (
- "```Dismiss any pop-up if it appears, then enter budget headphones in the product "
- "search field without submitting.```"
- ),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_navigation_url_stable_search_setups(submitted) == [
- {"label": "enter_query", "url": "https://example.com/"}
- ]
-
- def test_navigation_url_without_stable_search_setup_is_allowed(self) -> None:
- submitted = _yaml(
- {
- "block_type": "navigation",
- "label": "open_article_and_accept_cookies",
- "url": "https://example.com/article",
- "navigation_goal": "```Accept the cookie banner if present.```",
- },
- {"block_type": "extraction", "label": "extract_article", "data_extraction_goal": "Extract article"},
- )
-
- assert _detect_navigation_url_stable_search_setups(submitted) == []
-
- def test_multiple_click_driven_stable_search_navigation_blocks_are_detected(self) -> None:
- submitted = _yaml(
- {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},
- {
- "block_type": "navigation",
- "label": "enter_query",
- "navigation_goal": (
- "Achieve the following mini goal and once it's achieved, complete: "
- "```Enter budget headphones in the product search field. Do not submit the search yet.```"
- ),
- },
- {
- "block_type": "navigation",
- "label": "set_dates",
- "navigation_goal": (
- "Achieve the following mini goal and once it's achieved, complete: "
- "```Select July 10, 2026 as the start date and July 12, 2026 as the end date.```"
- ),
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_click_driven_stable_search_drips(submitted) == [
- {
- "entry_label": "open_search_site",
- "navigation_labels": ["enter_query", "set_dates"],
- }
- ]
-
- def test_stable_search_navigation_after_dom_inspection_is_allowed(self) -> None:
- submitted = _yaml(
- {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},
- {
- "block_type": "navigation",
- "label": "enter_query",
- "navigation_goal": "```Enter budget headphones in the product search field.```",
- },
- {
- "block_type": "code",
- "label": "inspect_search_form",
- "code": "form_action = await page.locator('form').first.get_attribute('action')",
- },
- {
- "block_type": "navigation",
- "label": "set_dates",
- "navigation_goal": "```Select July 10, 2026 as the start date and July 12, 2026 as the end date.```",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_click_driven_stable_search_drips(submitted) == []
-
- def test_single_stable_search_navigation_before_submit_is_allowed(self) -> None:
- submitted = _yaml(
- {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},
- {
- "block_type": "navigation",
- "label": "enter_query",
- "navigation_goal": "```Enter budget headphones in the product search field.```",
- },
- {
- "block_type": "navigation",
- "label": "submit_search",
- "navigation_goal": "```Click the Search button to submit and show results.```",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
-
- assert _detect_click_driven_stable_search_drips(submitted) == []
-
-
-def _ctx(prior_yaml: str | None = None) -> MagicMock:
- ctx = MagicMock()
- ctx.workflow_yaml = prior_yaml
- ctx.workflow_id = "w_test"
- ctx.workflow_permanent_id = "wpid_test"
- ctx.organization_id = "o_test"
- return ctx
-
-
-async def test_update_workflow_rejects_unverified_goto_url_state_shortcut_and_emits_span() -> None:
- submitted = _yaml(
- {
- "block_type": "goto_url",
- "label": "open_dynamic_results",
- "url": "https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available",
- },
- {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
- )
- ctx = _ctx()
-
- with (
- patch("skyvern.forge.sdk.copilot.tools._record_goto_url_state_shortcut_reject_span") as mock_span,
- patch("skyvern.forge.sdk.copilot.tools.app") as mock_app,
- ):
- mock_app.WORKFLOW_SERVICE.update_workflow_definition = AsyncMock()
- result = await _update_workflow({"workflow_yaml": submitted}, ctx)
-
- assert result["ok"] is False
- assert "goto_url may bootstrap stable page state" in result["error"]
- assert "URL-encoded dynamic page state must be verified" in result["error"]
- assert "open_dynamic_results" in result["error"]
- assert "extract_results" in result["error"]
- mock_app.WORKFLOW_SERVICE.update_workflow_definition.assert_not_called()
- mock_span.assert_called_once()
-
-
-class TestPromptAndToolDescriptions:
- def test_agent_prompt_contains_goto_url_state_policy(self) -> None:
- rendered = _render_agent_prompt()
-
- assert "GOTO_URL STATE SHORTCUT POLICY" in rendered
- assert "URL refinement for dynamic page state is allowed ONLY when grounded in live site evidence" in rendered
- assert "Before extraction on stateful search/result tasks" in rendered
- assert "do not spend one click-driven `navigation` block per field after the entry page loads" in rendered
- assert "current URL, form action, field names/values, result links, or URL deltas" in rendered
- assert "use `code` for DOM checks" in rendered
- assert "per-tool budget" in rendered
- assert "constraint / expected / observed / source" in rendered
-
- def test_tool_descriptions_warn_against_direct_state_shortcut_extraction(self) -> None:
- for tool in (update_workflow_tool, run_blocks_tool, update_and_run_blocks_tool):
- desc = tool.description # type: ignore[attr-defined]
- assert "goto_url" in desc
- assert "dynamic page state" in desc
- assert "code" in desc
- assert "validation" in desc
diff --git a/tests/unit/test_copilot_page_evidence_policy.py b/tests/unit/test_copilot_page_evidence_policy.py
new file mode 100644
index 000000000..35914ed08
--- /dev/null
+++ b/tests/unit/test_copilot_page_evidence_policy.py
@@ -0,0 +1,37 @@
+"""Tests for Copilot's build-time page evidence guidance."""
+
+from __future__ import annotations
+
+from skyvern.forge.prompts import prompt_engine
+from skyvern.forge.sdk.copilot.tools import update_and_run_blocks_tool, update_workflow_tool
+
+_AGENT_TEMPLATE_DEFAULTS = dict(
+ workflow_knowledge_base="test kb",
+ current_datetime="2026-01-01T00:00:00Z",
+ tool_usage_guide="",
+ security_rules="",
+)
+
+
+def _render_agent_prompt() -> str:
+ return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)
+
+
+def test_agent_prompt_frames_inspection_as_build_time_context_not_workflow_shape_policy() -> None:
+ rendered = _render_agent_prompt()
+
+ assert "PAGE EVIDENCE POLICY" in rendered
+ assert "fill knowledge gaps while building, editing, or debugging a workflow" in rendered
+ assert "not to add defensive verification blocks for every website shape" in rendered
+ assert "Do not invent website-specific paths, query parameters, form fields" in rendered
+ assert "GOTO_URL STATE SHORTCUT POLICY" not in rendered
+ assert "Before extraction on stateful search/result tasks" not in rendered
+
+
+def test_tool_descriptions_ground_composition_without_prescribing_extra_workflow_blocks() -> None:
+ for tool in (update_workflow_tool, update_and_run_blocks_tool):
+ desc = tool.description # type: ignore[attr-defined]
+ assert "browser inspection" in desc
+ assert "build-time context" in desc
+ assert "validation` or `navigation` block" not in desc
+ assert "verify every requested constraint" not in desc