Skip to content

Automate browser based workflows with AI

SKY-9637: verify dynamic goto_url state before extraction (#5892)

Andrew Neilson104b8a4e249c
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index 2afbb613e..6590be319 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -46,6 +46,19 @@ Prefer workflow-run evidence first. Use navigate_browser, get_browser_screenshot
2. The latest workflow run evidence is missing, ambiguous, or contradictory.
3. You need to validate a specific UI hypothesis before changing the workflow.
+GOTO_URL STATE SHORTCUT POLICY:
+Treat `goto_url` as a fast bootstrap, not as the source of truth for dynamic page state.
+- URL guessing is allowed for the first pass when it gets you to a plausible results/search page quickly.
+- URL refinement for dynamic page state is allowed ONLY when grounded in live site evidence: active chips, checked controls, state-changing links, form values, DOM attributes, or URL deltas observed after UI actions.
+- Ungrounded URL parameter tweaking is not allowed. If the page does not verify the intended state, inspect it with a `code` or `validation` block, or apply missing state with a `navigation` block.
+- For stateful search tasks with multiple stable inputs before results appear, do not spend one click-driven `navigation` block per field after the entry page loads. Before adding a second stable search-form/date/query-input block, use observed same-site evidence (current URL, form action, field names/values, result links, or URL deltas) to bootstrap stable search state with `goto_url`, or add a short `code` block that inspects the DOM and navigates/verifies the stable results page. Reserve click-driven `navigation` blocks for missing dynamic state or cases where the URL/form evidence is insufficient.
+- Do not bypass the previous rule by putting the entry URL and stable search setup in the same `navigation` block. Split pure page loading into `goto_url` first, then inspect/bootstrap/apply state in later blocks.
+- Before extraction on stateful search/result tasks, verify every requested constraint independently and simultaneously. Use live page evidence, not the presence of guessed query params.
+- Represent broad constraints as intent, not as a single fragile URL shape. Verify the visible state that satisfies the intent instead of assuming one query parameter means the whole constraint is active.
+- If a shortcut partially works, keep the useful state and apply/verify the missing constraints through UI/DOM evidence instead of looping on speculative `goto_url` variants.
+- If a state-application block hits a per-tool budget, do not rerun the same navigation label unchanged. Split/replace it with a smaller one-action UI block, or use `code`/`validation` to inspect the visible DOM state before deciding what is missing.
+- Put the verification evidence in the workflow itself (prefer `code` for DOM checks, `validation` for semantic/visual checks, and `navigation` only when state needs to be changed) and in `global_llm_context.decisions_made` as a per-constraint summary such as `constraint / expected / observed / source`.
+
CRITICAL RULES WHEN USING DIRECT BROWSER TOOLS:
- If you call navigate_browser, you MUST call at least one observation tool (evaluate, get_browser_screenshot, click, type_text, scroll, select_option, press_key, or console_messages) before responding.
- NEVER answer questions about page content, link counts, link targets, or page structure from memory or training data. Use evaluate or get_browser_screenshot.
@@ -125,7 +138,7 @@ IMPORTANT WORKFLOW RULES:
- "login" for authentication flows.
- "wait" for explicit pauses between blocks.
- "goto_url" for direct URL navigation and grounded URL-state shortcuts.
-* Prefer a `goto_url` URL-parameter shortcut over a click-driven `navigation` block when requested state (dates, destination, filters, search terms) maps to a URL that is supplied by the user, observed on the site, present in a link/form/action URL, or follows a same-site pattern visible in context. Do NOT invent website-specific paths or query parameters from memory — when the URL shape is uncertain, use a narrow `navigation` block instead. Put any follow-up actions in subsequent blocks, and follow URL-state shortcuts with extraction or validation that checks the visible page state.
+* Prefer a `goto_url` URL-parameter shortcut over a click-driven `navigation` block when requested state (dates, destination, search terms, stable search params) maps to a URL that is supplied by the user, observed on the site, present in a link/form/action URL, or follows a same-site pattern visible in context. Do NOT invent website-specific paths or query parameters from memory. For URL-encoded dynamic page state, `goto_url` may bootstrap the page but must be followed by a persisted verification block before extraction: use `code` for DOM checks, `validation` for semantic/visual checks, or `navigation` when missing state must be changed through the UI. Put any follow-up actions in subsequent blocks, and verify dynamic URL-state shortcuts from the live page before extraction.
* Commit a draft workflow as soon as you have a plausible block sequence and a target URL/selector. Do NOT walk the search UI step-by-step or fetch page contents through `evaluate`/`fetch` before the first `update_and_run_blocks` call when a `goto_url` to a likely page can be tested directly. Page introspection through direct browser tools is a fallback after a block run fails or returns ambiguous evidence, not preparation for the first run.
* Keep `navigation_goal` to ONE atomic action per block. You MUST split into separate blocks if your `navigation_goal` would cover more than one of: (a) a search-form input, (b) a date or filter selection, (c) a results-page click into a detail view, (d) a final extraction. Conjunctions like "and"/"then"/commas separating distinct UI actions inside a single `navigation_goal` are a strong signal you are bundling — decompose. The verifier checks every sub-condition after each step, so bundled goals return "continue" until the budget exhausts. When one request maps to several sub-tasks and no `goto_url` shortcut applies, decompose into one block per sub-task chained via `next_block_label`. The final stage of search → click → extract should be an `extraction` block, not another `navigation`.
* Set `complete_criterion` on every decomposed `navigation` block to the observable post-action page state — without it, smaller blocks still return vague "continue" verdicts.
diff --git a/skyvern/forge/sdk/copilot/context.py b/skyvern/forge/sdk/copilot/context.py
index d50e30a1a..7f77a18c8 100644
--- a/skyvern/forge/sdk/copilot/context.py
+++ b/skyvern/forge/sdk/copilot/context.py
@@ -210,6 +210,12 @@ class CopilotContext(AgentContext):
probable_site_block_stop_nudge_count: int = 0
per_tool_budget_nudge_count: int = 0
effective_workflow_proxy_location: Any | None = None
+ # Labels of navigation blocks that were canceled/failed inside a
+ # PER_TOOL_BUDGET run. Armed after get_run_results inspects the budgeted
+ # run, and cleared only when a workflow update removes or changes the
+ # label away from navigation. Prevents rerunning the same oversized
+ # navigation block unchanged.
+ per_tool_budget_problem_block_labels: list[str] = field(default_factory=list)
# Per-request frontier state. `verified_block_outputs` and
# `verified_prefix_labels` are populated ONLY from fully-successful runs —
diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.py
index bb306ea57..f5e9b8db9 100644
--- a/skyvern/forge/sdk/copilot/enforcement.py
+++ b/skyvern/forge/sdk/copilot/enforcement.py
@@ -251,15 +251,23 @@ POST_PER_TOOL_BUDGET_NUDGE = (
"single tool call.\n"
"Do NOT retry the same chain. Do NOT change navigation_goal wording or "
"selectors hoping it will run faster.\n"
- "1. Shrink the requested block_labels list to the first 1-2 unverified "
+ "1. Call get_run_results for the budgeted run. If it shows a navigation "
+ "block was canceled or failed, do NOT run that same label again unchanged; "
+ "split or replace it first.\n"
+ "2. Shrink the requested block_labels list to the first 1-2 unverified "
"blocks. The verified-prefix optimization will replay any earlier blocks "
"from cached state without re-running the browser, so passing a smaller "
"frontier is cheap.\n"
- "2. Test that smaller frontier with update_and_run_blocks. If it succeeds, "
- "extend by one block at a time on subsequent calls.\n"
- "3. If your workflow only has 1-2 blocks and one block is still hitting "
- "the budget, the single block is too ambitious — either narrow its "
- "navigation_goal scope, or reply with a blocker explanation."
+ "3. For page-state work, prefer a code or validation block to verify DOM "
+ "state, active chips, checked controls, field values, or URL deltas. Use a "
+ "navigation block only when missing state must be changed through the UI, "
+ "and keep it to one atomic action.\n"
+ "4. Test the smaller frontier. If it succeeds, extend by one block at a "
+ "time on subsequent calls.\n"
+ "5. If your workflow only has 1-2 blocks and one block is still hitting "
+ "the budget, the single block is too ambitious — either narrow its scope, "
+ "replace it with DOM/state verification plus smaller actions, or reply "
+ "with a blocker explanation."
)
POST_NO_WORKFLOW_DELIVERY_NUDGE = (
diff --git a/skyvern/forge/sdk/copilot/mcp_adapter.py b/skyvern/forge/sdk/copilot/mcp_adapter.py
index a5ee9ca03..50a1daec0 100644
--- a/skyvern/forge/sdk/copilot/mcp_adapter.py
+++ b/skyvern/forge/sdk/copilot/mcp_adapter.py
@@ -54,6 +54,7 @@ class SchemaOverlay:
LOG = structlog.get_logger()
+_INTERNAL_TOOL_ARG_KEYS = frozenset({"_summarized"})
def _apply_schema_overlay(
@@ -89,7 +90,7 @@ def _transform_args(
arguments: dict[str, Any],
overlay: SchemaOverlay,
) -> dict[str, Any]:
- mcp_args = {k: v for k, v in arguments.items() if k not in overlay.hide_params}
+ mcp_args = {k: v for k, v in arguments.items() if k not in overlay.hide_params | _INTERNAL_TOOL_ARG_KEYS}
for copilot_param, mcp_param in overlay.arg_transforms.items():
if copilot_param in mcp_args:
@@ -195,6 +196,7 @@ class SkyvernOverlayMCPServer(MCPServer):
raise RuntimeError("Not connected — call connect() first")
arguments = arguments or {}
+ arguments = {k: v for k, v in arguments.items() if k not in _INTERNAL_TOOL_ARG_KEYS}
copilot_ctx = self._context_provider()
overlay = self._overlays.get(tool_name, SchemaOverlay())
diff --git a/skyvern/forge/sdk/copilot/tools.py b/skyvern/forge/sdk/copilot/tools.py
index 6aaf4fc60..340c3d8fe 100644
--- a/skyvern/forge/sdk/copilot/tools.py
+++ b/skyvern/forge/sdk/copilot/tools.py
@@ -10,6 +10,7 @@ import time
from collections import Counter, defaultdict
from datetime import datetime
from typing import Any, Literal
+from urllib.parse import parse_qsl, urlparse
import structlog
import yaml
@@ -376,6 +377,102 @@ def _maybe_clear_reconciliation_flag(copilot_ctx: Any, result: Any) -> None:
copilot_ctx.pending_reconciliation_run_id = None
+def _enum_or_string_name(value: Any) -> str:
+ raw = getattr(value, "value", value)
+ if not isinstance(raw, str):
+ raw = getattr(value, "name", raw)
+ return str(raw).strip().lower()
+
+
+def _per_tool_budget_problem_label_set(ctx: Any) -> set[str]:
+ labels = getattr(ctx, "per_tool_budget_problem_block_labels", None)
+ if not isinstance(labels, list):
+ return set()
+ return {label for label in labels if isinstance(label, str) and label}
+
+
+def _record_per_tool_budget_problem_blocks_from_results(copilot_ctx: Any, result: dict[str, Any]) -> None:
+ if getattr(copilot_ctx, "last_failure_category_top", None) != PER_TOOL_BUDGET_FAILURE_CATEGORY:
+ return
+ data = result.get("data")
+ if not isinstance(data, dict):
+ return
+ pending_run_id = getattr(copilot_ctx, "pending_reconciliation_run_id", None)
+ resolved_run_id = data.get("workflow_run_id")
+ if isinstance(pending_run_id, str) and pending_run_id and resolved_run_id != pending_run_id:
+ return
+ blocks = data.get("blocks")
+ if not isinstance(blocks, list):
+ return
+
+ labels = _per_tool_budget_problem_label_set(copilot_ctx)
+ for block in blocks:
+ if not isinstance(block, dict):
+ continue
+ if _enum_or_string_name(block.get("block_type")) != BlockType.NAVIGATION.value:
+ continue
+ if _enum_or_string_name(block.get("status")) not in _FAILED_BLOCK_STATUSES:
+ continue
+ label = block.get("label")
+ if isinstance(label, str) and label:
+ labels.add(label)
+ setattr(copilot_ctx, "per_tool_budget_problem_block_labels", sorted(labels))
+
+
+def _navigation_labels_in_workflow(workflow: Any) -> set[str]:
+ definition = getattr(workflow, "workflow_definition", None)
+ blocks = getattr(definition, "blocks", None)
+ if not isinstance(blocks, list):
+ return set()
+
+ labels: set[str] = set()
+ for block in blocks:
+ if _enum_or_string_name(getattr(block, "block_type", None)) != BlockType.NAVIGATION.value:
+ continue
+ label = getattr(block, "label", None)
+ if isinstance(label, str) and label:
+ labels.add(label)
+ return labels
+
+
+def _clear_resolved_per_tool_budget_problem_labels(copilot_ctx: Any, workflow: Any) -> None:
+ problem_labels = _per_tool_budget_problem_label_set(copilot_ctx)
+ if not problem_labels:
+ return
+ remaining = sorted(problem_labels & _navigation_labels_in_workflow(workflow))
+ setattr(copilot_ctx, "per_tool_budget_problem_block_labels", remaining)
+
+
+def _requested_block_label_set(arguments: dict[str, Any] | None) -> set[str] | None:
+ if not isinstance(arguments, dict):
+ return None
+ block_labels = arguments.get("block_labels")
+ if not isinstance(block_labels, list):
+ return None
+ return {label for label in block_labels if isinstance(label, str) and label}
+
+
+def _per_tool_budget_problem_rerun_error(ctx: Any, arguments: dict[str, Any] | None) -> str | None:
+ problem_labels = _per_tool_budget_problem_label_set(ctx)
+ if not problem_labels:
+ return None
+
+ requested_labels = _requested_block_label_set(arguments)
+ blocked_labels = problem_labels if not requested_labels else problem_labels & requested_labels
+ if not blocked_labels:
+ return None
+
+ labels = ", ".join(sorted(blocked_labels))
+ return (
+ "The prior PER_TOOL_BUDGET run's get_run_results showed navigation "
+ f"block(s) [{labels}] were canceled or failed while applying page state. "
+ "Do NOT rerun those block label(s) unchanged with run_blocks_and_collect_debug. "
+ "Update the workflow to split or replace the oversized navigation block first, "
+ "prefer a code or validation block for DOM/page-state verification, or run only "
+ "newly-created smaller block labels that apply one missing constraint at a time."
+ )
+
+
# Streak threshold at which the copilot hard-aborts a tool call because the
# same action sequence has repeated run-over-run with no intervening success.
# The streak counter is incremented in ``update_repeated_failure_state`` AFTER
@@ -605,6 +702,11 @@ def _tool_loop_error(ctx: AgentContext, tool_name: str, arguments: dict[str, Any
f"side effects on live sites."
)
+ if tool_name == "run_blocks_and_collect_debug":
+ budget_rerun_error = _per_tool_budget_problem_rerun_error(ctx, arguments)
+ if budget_rerun_error is not None:
+ return budget_rerun_error
+
streak_raw = getattr(ctx, "repeated_action_fingerprint_streak_count", 0)
streak = streak_raw if isinstance(streak_raw, int) else 0
if streak >= REPEATED_ACTION_STREAK_ABORT_AT:
@@ -750,6 +852,21 @@ async def _update_workflow(
if banned_items:
_record_banned_block_reject_span("_update_workflow", banned_items)
return {"ok": False, "error": _banned_block_reject_message(banned_items)}
+
+ goto_url_state_shortcuts = _detect_unverified_goto_url_state_shortcuts(workflow_yaml)
+ if goto_url_state_shortcuts:
+ _record_goto_url_state_shortcut_reject_span("_update_workflow", goto_url_state_shortcuts)
+ return {"ok": False, "error": _goto_url_state_shortcut_message(goto_url_state_shortcuts)}
+
+ navigation_url_search_setups = _detect_navigation_url_stable_search_setups(workflow_yaml)
+ if navigation_url_search_setups:
+ _record_navigation_url_stable_search_reject_span("_update_workflow", navigation_url_search_setups)
+ return {"ok": False, "error": _navigation_url_stable_search_message(navigation_url_search_setups)}
+
+ stable_search_drips = _detect_click_driven_stable_search_drips(workflow_yaml)
+ if stable_search_drips:
+ _record_goto_url_stable_search_drip_reject_span("_update_workflow", stable_search_drips)
+ return {"ok": False, "error": _goto_url_stable_search_drip_message(stable_search_drips)}
try:
workflow = _process_workflow_yaml(
workflow_id=ctx.workflow_id,
@@ -2117,6 +2234,142 @@ _COPILOT_BANNED_BLOCK_ALTERNATIVES = (
"`login` for authentication, or `goto_url` for pure URL navigation."
)
+_GOTO_URL_DYNAMIC_STATE_KEY_TOKENS = frozenset(
+ {
+ "attr",
+ "attribute",
+ "category",
+ "constraint",
+ "criteria",
+ "criterion",
+ "facet",
+ "filter",
+ "option",
+ "refine",
+ "refinement",
+ "selected",
+ "selection",
+ "toggle",
+ }
+)
+_GOTO_URL_DYNAMIC_STATE_VALUE_TOKENS = frozenset(
+ {
+ "facet",
+ "filter",
+ "refine",
+ "refinement",
+ "selected",
+ "selection",
+ }
+)
+_GOTO_URL_VERIFICATION_BLOCK_TYPES = frozenset({"code", "navigation", "validation"})
+_GOTO_URL_DATA_PRODUCING_BLOCK_TYPES = frozenset({"extraction", "text_prompt"})
+_GOTO_URL_STATE_ACTION_TOKENS = frozenset(
+ {
+ "apply",
+ "check",
+ "choose",
+ "click",
+ "confirm",
+ "confirmed",
+ "ensure",
+ "inspect",
+ "observe",
+ "select",
+ "set",
+ "verify",
+ "verified",
+ "verifies",
+ "verifying",
+ }
+)
+_GOTO_URL_STATE_EVIDENCE_TOKENS = frozenset(
+ {
+ "active",
+ "aria",
+ "checked",
+ "checkbox",
+ "chip",
+ "constraint",
+ "control",
+ "criteria",
+ "criterion",
+ "dom",
+ "facet",
+ "filter",
+ "form",
+ "input",
+ "option",
+ "selected",
+ "state",
+ "url",
+ "value",
+ }
+)
+_GOTO_URL_CODE_EVIDENCE_TOKENS = frozenset(
+ {
+ "all_text_contents",
+ "attribute",
+ "contents",
+ "evaluate",
+ "inner_text",
+ "locator",
+ "query",
+ "query_selector",
+ "selector",
+ "text",
+ "text_content",
+ }
+)
+_GOTO_URL_DISMISSAL_TOKENS = frozenset({"banner", "cookie", "cookies", "dismiss", "modal", "popup", "popups"})
+_GOTO_URL_SEARCH_SETUP_ACTION_TOKENS = frozenset(
+ {
+ "choose",
+ "enter",
+ "fill",
+ "input",
+ "pick",
+ "select",
+ "set",
+ "type",
+ }
+)
+_GOTO_URL_SEARCH_SETUP_CONTEXT_TOKENS = frozenset(
+ {
+ "calendar",
+ "checkout",
+ "checkin",
+ "date",
+ "dates",
+ "find",
+ "lookup",
+ "query",
+ "search",
+ }
+)
+_GOTO_URL_SEARCH_SETUP_TERMINAL_TOKENS = frozenset(
+ {
+ "active",
+ "checked",
+ "checkbox",
+ "chip",
+ "constraint",
+ "control",
+ "dom",
+ "extract",
+ "extraction",
+ "facet",
+ "filter",
+ "result",
+ "results",
+ "verified",
+ "verify",
+ }
+)
+_GOTO_URL_STRUCTURAL_STATE_SPLIT_RE = re.compile(r"[;|]")
+_GOTO_URL_STRUCTURAL_ASSIGNMENT_LHS_RE = re.compile(r"[A-Za-z0-9_.:-]{1,64}")
+_GOTO_URL_VALUE_SCHEME_RE = re.compile(r"^\s*[a-z][a-z0-9+.-]*:", re.IGNORECASE)
+
def _banned_block_reject_message(items: list[tuple[str, str]]) -> str:
"""Uniform error text for the post-emission reject, sharing the
@@ -2180,6 +2433,44 @@ def _record_workflow_proxy_location_span(workflow_yaml: str, workflow: Workflow)
pass
+def _record_goto_url_state_shortcut_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:
+ with copilot_span(
+ "goto_url_state_shortcut_reject",
+ data={
+ "source_tool": source_tool,
+ "items": items,
+ "labels": [item.get("label") for item in items],
+ "dynamic_state_keys": sorted({key for item in items for key in item.get("dynamic_state_keys", [])}),
+ },
+ ):
+ pass
+
+
+def _record_goto_url_stable_search_drip_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:
+ with copilot_span(
+ "goto_url_stable_search_drip_reject",
+ data={
+ "source_tool": source_tool,
+ "items": items,
+ "entry_labels": [item.get("entry_label") for item in items],
+ "navigation_labels": [item.get("navigation_labels", []) for item in items],
+ },
+ ):
+ pass
+
+
+def _record_navigation_url_stable_search_reject_span(source_tool: str, items: list[dict[str, Any]]) -> None:
+ with copilot_span(
+ "navigation_url_stable_search_reject",
+ data={
+ "source_tool": source_tool,
+ "items": items,
+ "labels": [item.get("label") for item in items],
+ },
+ ):
+ pass
+
+
def _collect_banned_block_items(blocks: list[Any]) -> list[tuple[str, str]]:
"""Recursively walk ``blocks`` (mirroring
:func:`skyvern.forge.sdk.copilot.block_goal_wrapping._wrap_blocks_in_place`)
@@ -2224,6 +2515,384 @@ def _parse_workflow_blocks(yaml_str: str | None) -> list[Any] | None:
return blocks if isinstance(blocks, list) else None
+def _normalized_query_token(value: str) -> str:
+ return re.sub(r"[^a-z0-9]+", "_", value.lower()).strip("_")
+
+
+def _query_token_parts(value: str) -> set[str]:
+ parts: set[str] = set()
+ for part in value.split("_"):
+ if not part:
+ continue
+ parts.add(part)
+ if len(part) > 3 and part.endswith("s"):
+ parts.add(part[:-1])
+ return parts
+
+
+def _is_dynamic_state_query_key(key: str) -> bool:
+ return bool(_query_token_parts(key) & _GOTO_URL_DYNAMIC_STATE_KEY_TOKENS)
+
+
+def _is_dynamic_state_query_value(value: str) -> bool:
+ return bool(_query_token_parts(value) & _GOTO_URL_DYNAMIC_STATE_VALUE_TOKENS)
+
+
+def _is_url_like_query_value(value: str) -> bool:
+ stripped = value.strip()
+ if not stripped:
+ return False
+ if stripped.startswith("/") or _GOTO_URL_VALUE_SCHEME_RE.match(stripped):
+ return True
+ first_equals = stripped.find("=")
+ first_query = stripped.find("?")
+ return first_query != -1 and (first_equals == -1 or first_query < first_equals)
+
+
+def _assignment_lhs(clause: str) -> str | None:
+ if "=" not in clause:
+ return None
+ lhs, rhs = clause.split("=", 1)
+ lhs = lhs.strip()
+ if not lhs or not rhs.strip():
+ return None
+ if not _GOTO_URL_STRUCTURAL_ASSIGNMENT_LHS_RE.fullmatch(lhs):
+ return None
+ return _normalized_query_token(lhs)
+
+
+def _is_structural_dynamic_state_query_value(value: str, outer_key: str) -> bool:
+ if not value or _is_url_like_query_value(value):
+ return False
+
+ clauses = [clause.strip() for clause in _GOTO_URL_STRUCTURAL_STATE_SPLIT_RE.split(value) if clause.strip()]
+ if not clauses:
+ return False
+
+ assignment_lhses = [lhs for lhs in (_assignment_lhs(clause) for clause in clauses) if lhs is not None]
+ if len(assignment_lhses) >= 2:
+ return True
+ if len(assignment_lhses) == 1 and assignment_lhses[0] != outer_key:
+ return True
+ return False
+
+
+def _dynamic_state_query_keys(url: Any) -> list[str]:
+ if not isinstance(url, str) or not url:
+ return []
+ try:
+ pairs = parse_qsl(urlparse(url).query, keep_blank_values=True)
+ except ValueError:
+ return []
+ if not pairs:
+ return []
+
+ normalized_pairs = [(_normalized_query_token(key), _normalized_query_token(value), value) for key, value in pairs]
+ key_counts = Counter(key for key, _, _ in normalized_pairs if key)
+ dynamic_keys: set[str] = set()
+ for key, value, raw_value in normalized_pairs:
+ if not key:
+ continue
+ if _is_structural_dynamic_state_query_value(raw_value, key):
+ dynamic_keys.add(key)
+ elif _is_dynamic_state_query_key(key):
+ dynamic_keys.add(key)
+ elif key_counts[key] > 1:
+ dynamic_keys.add(key)
+ elif _is_dynamic_state_query_value(value):
+ dynamic_keys.add(key)
+ return sorted(dynamic_keys)
+
+
+def _block_label_from_yaml(block: dict[str, Any]) -> str | None:
+ label = block.get("label")
+ return label if isinstance(label, str) and label else None
+
+
+def _next_top_level_block(
+ *,
+ blocks: list[dict[str, Any]],
+ by_label: dict[str, dict[str, Any]],
+ index_by_label: dict[str, int],
+ current: dict[str, Any],
+) -> dict[str, Any] | None:
+ next_label = current.get("next_block_label")
+ if isinstance(next_label, str) and next_label:
+ return by_label.get(next_label)
+
+ label = _block_label_from_yaml(current)
+ if label is None:
+ return None
+ index = index_by_label.get(label)
+ if index is None:
+ return None
+ if index + 1 >= len(blocks):
+ return None
+ return blocks[index + 1]
+
+
+def _next_data_block_without_verification(
+ *,
+ blocks: list[dict[str, Any]],
+ by_label: dict[str, dict[str, Any]],
+ index_by_label: dict[str, int],
+ start: dict[str, Any],
+) -> dict[str, Any] | None:
+ seen: set[str] = set()
+ current = start
+ while True:
+ next_block = _next_top_level_block(
+ blocks=blocks,
+ by_label=by_label,
+ index_by_label=index_by_label,
+ current=current,
+ )
+ if next_block is None:
+ return None
+ current = next_block
+
+ label = _block_label_from_yaml(current)
+ if label is not None:
+ if label in seen:
+ return None
+ seen.add(label)
+
+ block_type = str(current.get("block_type", "")).strip().lower()
+ if _block_verifies_dynamic_goto_url_state(current):
+ return None
+ if block_type in _GOTO_URL_DATA_PRODUCING_BLOCK_TYPES:
+ return current
+
+
+def _block_state_text(block: dict[str, Any]) -> str:
+ values = []
+ for key in (
+ "navigation_goal",
+ "complete_criterion",
+ "terminate_criterion",
+ "validation_goal",
+ "code",
+ "description",
+ "title",
+ ):
+ value = block.get(key)
+ if isinstance(value, str):
+ values.append(value)
+ return _normalized_query_token(" ".join(values))
+
+
+def _block_primary_goal_text(block: dict[str, Any]) -> str:
+ for key in ("navigation_goal", "validation_goal", "code", "description", "title"):
+ value = block.get(key)
+ if not isinstance(value, str) or not value:
+ continue
+ first_fence = value.find("```")
+ if first_fence == -1:
+ return _normalized_query_token(value)
+ second_fence = value.find("```", first_fence + 3)
+ if second_fence != -1:
+ fenced = value[first_fence + 3 : second_fence].strip()
+ if fenced:
+ return _normalized_query_token(fenced)
+ return _normalized_query_token(value)
+ return ""
+
+
+def _block_verifies_dynamic_goto_url_state(block: dict[str, Any]) -> bool:
+ block_type = str(block.get("block_type", "")).strip().lower()
+ if block_type not in _GOTO_URL_VERIFICATION_BLOCK_TYPES:
+ return False
+
+ parts = _query_token_parts(_block_state_text(block))
+ if not parts:
+ return False
+
+ has_state_evidence = bool(parts & _GOTO_URL_STATE_EVIDENCE_TOKENS)
+ if block_type == "code":
+ return has_state_evidence and bool(parts & _GOTO_URL_CODE_EVIDENCE_TOKENS)
+ if block_type == "validation":
+ return has_state_evidence
+ if parts & _GOTO_URL_DISMISSAL_TOKENS and not parts & {"active", "checked", "selected"}:
+ return False
+ return has_state_evidence and bool(parts & _GOTO_URL_STATE_ACTION_TOKENS)
+
+
+def _is_stable_search_setup_navigation(block: dict[str, Any]) -> bool:
+ if str(block.get("block_type", "")).strip().lower() != "navigation":
+ return False
+
+ parts = _query_token_parts(_block_primary_goal_text(block))
+ if not parts:
+ return False
+ if parts & _GOTO_URL_SEARCH_SETUP_TERMINAL_TOKENS:
+ return False
+ return bool(parts & _GOTO_URL_SEARCH_SETUP_ACTION_TOKENS) and bool(parts & _GOTO_URL_SEARCH_SETUP_CONTEXT_TOKENS)
+
+
+def _detect_navigation_url_stable_search_setups(workflow_yaml: str | None) -> list[dict[str, Any]]:
+ blocks = _parse_workflow_blocks(workflow_yaml)
+ if not blocks:
+ return []
+
+ findings: list[dict[str, Any]] = []
+ for block in blocks:
+ if not isinstance(block, dict):
+ continue
+ if not isinstance(block.get("url"), str) or not block["url"].strip():
+ continue
+ if not _is_stable_search_setup_navigation(block):
+ continue
+ findings.append(
+ {
+ "label": _block_label_from_yaml(block) or "<missing label>",
+ "url": block["url"],
+ }
+ )
+ return findings
+
+
+def _detect_click_driven_stable_search_drips(workflow_yaml: str | None) -> list[dict[str, Any]]:
+ blocks = _parse_workflow_blocks(workflow_yaml)
+ if not blocks:
+ return []
+
+ top_level_blocks = [block for block in blocks if isinstance(block, dict)]
+ findings: list[dict[str, Any]] = []
+ entry_label: str | None = None
+ stable_nav_labels: list[str] = []
+
+ for block in top_level_blocks:
+ block_type = str(block.get("block_type", "")).strip().lower()
+ label = _block_label_from_yaml(block) or "<missing label>"
+
+ if block_type == "goto_url":
+ entry_label = label
+ stable_nav_labels = []
+ continue
+
+ if entry_label is None:
+ continue
+
+ if block_type in {"code", "validation"}:
+ stable_nav_labels = []
+ continue
+
+ if block_type in _GOTO_URL_DATA_PRODUCING_BLOCK_TYPES:
+ entry_label = None
+ stable_nav_labels = []
+ continue
+
+ if _is_stable_search_setup_navigation(block):
+ stable_nav_labels.append(label)
+ if len(stable_nav_labels) >= 2:
+ findings.append(
+ {
+ "entry_label": entry_label,
+ "navigation_labels": stable_nav_labels.copy(),
+ }
+ )
+ entry_label = None
+ stable_nav_labels = []
+ continue
+
+ return findings
+
+
+def _detect_unverified_goto_url_state_shortcuts(workflow_yaml: str | None) -> list[dict[str, Any]]:
+ blocks = _parse_workflow_blocks(workflow_yaml)
+ if not blocks:
+ return []
+
+ top_level_blocks = [block for block in blocks if isinstance(block, dict)]
+ by_label: dict[str, dict[str, Any]] = {}
+ index_by_label: dict[str, int] = {}
+ for index, block in enumerate(top_level_blocks):
+ label = _block_label_from_yaml(block)
+ if label is None:
+ continue
+ by_label[label] = block
+ index_by_label[label] = index
+
+ findings: list[dict[str, Any]] = []
+ for block in top_level_blocks:
+ if str(block.get("block_type", "")).strip().lower() != "goto_url":
+ continue
+ dynamic_state_keys = _dynamic_state_query_keys(block.get("url"))
+ if not dynamic_state_keys:
+ continue
+ next_data_block = _next_data_block_without_verification(
+ blocks=top_level_blocks,
+ by_label=by_label,
+ index_by_label=index_by_label,
+ start=block,
+ )
+ if next_data_block is None:
+ continue
+ findings.append(
+ {
+ "label": _block_label_from_yaml(block) or "<missing label>",
+ "dynamic_state_keys": dynamic_state_keys,
+ "next_data_block_label": _block_label_from_yaml(next_data_block) or "<missing label>",
+ }
+ )
+ return findings
+
+
+_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT = 5
+
+
+def _goto_url_state_shortcut_message(items: list[dict[str, Any]]) -> str:
+ details = []
+ for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:
+ label = item.get("label", "?")
+ keys = ", ".join(str(key) for key in item.get("dynamic_state_keys", []))
+ next_label = item.get("next_data_block_label", "?")
+ details.append(f"{label}: query keys [{keys}] feed directly into {next_label}")
+ if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:
+ details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")
+ return (
+ "Workflow validation failed: goto_url may bootstrap stable page state, but URL-encoded dynamic page "
+ "state must be verified from the live page before data extraction. Add a code, validation, or navigation block "
+ "between the URL shortcut and extraction that inspects active chips, checked controls, search fields, "
+ "or DOM/link/form evidence; applies any missing state through UI actions when needed; and verifies all "
+ "requested constraints are active together. Offending shortcuts: "
+ f"{'; '.join(details)}"
+ )
+
+
+def _goto_url_stable_search_drip_message(items: list[dict[str, Any]]) -> str:
+ details = []
+ for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:
+ entry_label = item.get("entry_label", "?")
+ labels = ", ".join(str(label) for label in item.get("navigation_labels", []))
+ details.append(f"{entry_label}: click-driven stable search setup blocks [{labels}]")
+ if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:
+ details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")
+ return (
+ "Workflow validation failed: after a goto_url entry page, do not build stateful search setup as one "
+ "click-driven navigation block per stable field. Use observed same-site evidence from the live page "
+ "(current URL, form action, field names/values, result links, or URL deltas) to bootstrap stable search "
+ "state with goto_url, or insert a short code/validation block that inspects the DOM and verifies/navigates "
+ "the stable results page. Reserve click-driven navigation for missing dynamic state or when URL/form "
+ f"evidence is insufficient. Offending sequence: {'; '.join(details)}"
+ )
+
+
+def _navigation_url_stable_search_message(items: list[dict[str, Any]]) -> str:
+ details = []
+ for item in items[:_GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT]:
+ details.append(f"{item.get('label', '?')}: navigation url={item.get('url', '?')}")
+ if len(items) > _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT:
+ details.append(f"(and {len(items) - _GOTO_URL_STATE_SHORTCUT_MESSAGE_LIMIT} more)")
+ return (
+ "Workflow validation failed: a navigation block with a URL is combining entry-page navigation with "
+ "stable search setup. Split pure page loading into a goto_url block first. Then use observed same-site "
+ "evidence (current URL, form action, field names/values, result links, or URL deltas) to bootstrap "
+ "stable search state with goto_url, or add a short code/validation block to inspect the DOM before "
+ f"click-driven setup. Offending blocks: {'; '.join(details)}"
+ )
+
+
def _detect_new_banned_blocks(
submitted_yaml: str,
prior_workflow_yaml: str | None,
@@ -2653,6 +3322,7 @@ def _record_workflow_update_result(copilot_ctx: Any, result: dict[str, Any]) ->
wf = result["_workflow"]
copilot_ctx.last_workflow = wf
+ _clear_resolved_per_tool_budget_problem_labels(copilot_ctx, wf)
copilot_ctx.last_workflow_yaml = copilot_ctx.workflow_yaml or None
copilot_ctx.effective_workflow_proxy_location = getattr(wf, "proxy_location", None) or ProxyLocation.RESIDENTIAL
data = result.get("data")
@@ -2877,6 +3547,19 @@ async def update_workflow_tool(
"""Validate and update the workflow YAML definition.
Provide the complete workflow YAML as a string.
Returns the validated workflow or validation errors.
+
+ If a `goto_url` URL encodes dynamic page state, do not wire it
+ directly to extraction. Add a `code`, `validation`, or `navigation` block
+ first that verifies the live page state from active chips, checked controls,
+ fields, or DOM/link/form evidence.
+
+ After a `goto_url` entry page for a stateful search, do not add one
+ click-driven `navigation` block per stable field (query terms, dates,
+ search-form inputs). Use observed current URL / form action / field
+ names / result links / URL deltas to bootstrap stable search state with
+ `goto_url`, or add a short `code`/`validation` block to inspect the DOM.
+ Do not bypass this by combining a URL and stable search setup in a
+ `navigation` block; split pure loading into `goto_url` first.
"""
copilot_ctx = ctx.context
arguments = {"workflow_yaml": workflow_yaml}
@@ -2972,6 +3655,12 @@ async def run_blocks_tool(
the stored `credential_id`. If no stored credential matches, do NOT pass
the inline value via `parameters`; stop and follow the CREDENTIAL
HANDLING refusal rule in the system prompt.
+
+ When a prior `goto_url` used URL params for dynamic page state, the
+ blocks you run must include live-page verification before extraction. If
+ the visible state is uncertain or only partially applied, inspect it
+ through a `code` or `validation` block, or apply missing state through a
+ `navigation` block, instead of retrying guessed URL params.
"""
copilot_ctx = ctx.context
arguments = {"block_labels": block_labels, "parameters": parameters or {}}
@@ -3029,6 +3718,7 @@ async def get_run_results_tool(
return json.dumps({"ok": False, "error": loop_error})
result = await _get_run_results(params, copilot_ctx)
+ _record_per_tool_budget_problem_blocks_from_results(copilot_ctx, result)
_maybe_clear_reconciliation_flag(copilot_ctx, result)
record_tool_step_result_for_ctx(copilot_ctx, "get_run_results", params, result)
@@ -3061,6 +3751,21 @@ async def update_and_run_blocks_tool(
the stored `credential_id`. If no stored credential matches, do NOT pass
the inline value via `parameters`; stop and follow the CREDENTIAL
HANDLING refusal rule in the system prompt.
+
+ Treat `goto_url` as a bootstrap for stateful search/result pages. If URL query
+ params encode dynamic page state, include a `code`, `validation`, or
+ `navigation` block before extraction that verifies every requested
+ constraint is active simultaneously from live page state, and only refine
+ URL params when they are grounded in observed DOM/link/form state or
+ observed URL deltas.
+
+ After a `goto_url` entry page for a stateful search, do not add one
+ click-driven `navigation` block per stable field (query terms, dates,
+ search-form inputs). Use observed current URL / form action / field
+ names / result links / URL deltas to bootstrap stable search state with
+ `goto_url`, or add a short `code`/`validation` block to inspect the DOM.
+ Do not bypass this by combining a URL and stable search setup in a
+ `navigation` block; split pure loading into `goto_url` first.
"""
copilot_ctx = ctx.context
arguments = {"workflow_yaml": workflow_yaml, "block_labels": block_labels, "parameters": parameters or {}}
diff --git a/tests/unit/test_copilot_goto_url_state_policy.py b/tests/unit/test_copilot_goto_url_state_policy.py
new file mode 100644
index 000000000..fb6d196b1
--- /dev/null
+++ b/tests/unit/test_copilot_goto_url_state_policy.py
@@ -0,0 +1,420 @@
+"""Tests for Copilot's goto_url dynamic-state shortcut policy."""
+
+from __future__ import annotations
+
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+import yaml
+
+from skyvern.forge.prompts import prompt_engine
+from skyvern.forge.sdk.copilot.tools import (
+ _detect_click_driven_stable_search_drips,
+ _detect_navigation_url_stable_search_setups,
+ _detect_unverified_goto_url_state_shortcuts,
+ _update_workflow,
+ run_blocks_tool,
+ update_and_run_blocks_tool,
+ update_workflow_tool,
+)
+
+_AGENT_TEMPLATE_DEFAULTS = dict(
+ workflow_knowledge_base="test kb",
+ current_datetime="2026-01-01T00:00:00Z",
+ tool_usage_guide="",
+ security_rules="",
+)
+
+
+def _yaml(*blocks: dict) -> str:
+ return yaml.safe_dump(
+ {"title": "wf", "workflow_definition": {"blocks": list(blocks)}},
+ sort_keys=False,
+ )
+
+
+def _render_agent_prompt() -> str:
+ return prompt_engine.load_prompt("workflow-copilot-agent", **_AGENT_TEMPLATE_DEFAULTS)
+
+
+class TestGotoUrlStateShortcutDetection:
+ def test_plain_goto_url_to_extraction_is_allowed(self) -> None:
+ submitted = _yaml(
+ {"block_type": "goto_url", "label": "open_home", "url": "https://example.com/news"},
+ {"block_type": "extraction", "label": "extract_items", "data_extraction_goal": "Extract items"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_stable_search_params_to_extraction_are_allowed(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_results",
+ "url": "https://example.com/search?q=budget+headphones&start=2026-08-20&end=2026-08-23",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_short_state_tokens_do_not_match_unrelated_substrings(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_embedded_report",
+ "url": "https://example.com/report?embedded=true&classification=public",
+ },
+ {"block_type": "extraction", "label": "extract_report", "data_extraction_goal": "Extract report"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_dynamic_state_goto_url_directly_to_extraction_is_detected(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
+ {
+ "label": "open_dynamic_results",
+ "dynamic_state_keys": ["facet", "filter"],
+ "next_data_block_label": "extract_results",
+ }
+ ]
+
+ def test_repeated_opaque_query_key_directly_to_extraction_is_detected(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": "https://example.com/search?q=budget+headphones&choice=wireless&choice=noise-canceling",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
+ {
+ "label": "open_dynamic_results",
+ "dynamic_state_keys": ["choice"],
+ "next_data_block_label": "extract_results",
+ }
+ ]
+
+ def test_dynamic_state_goto_url_with_validation_before_extraction_is_allowed(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),
+ },
+ {
+ "block_type": "validation",
+ "label": "verify_state",
+ "complete_criterion": (
+ "Verify query, dates, and each requested result constraint are all active from visible page state."
+ ),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_dynamic_state_goto_url_with_unrelated_navigation_before_extraction_is_detected(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": (
+ "https://example.com/search?q=budget+headphones&"
+ "state=category%3Delectronics%3Bavailability%3Din_stock"
+ ),
+ },
+ {
+ "block_type": "navigation",
+ "label": "dismiss_cookie_popup",
+ "navigation_goal": "If a cookie popup is visible, close it so the main search form is usable.",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
+ {
+ "label": "open_dynamic_results",
+ "dynamic_state_keys": ["state"],
+ "next_data_block_label": "extract_results",
+ }
+ ]
+
+ def test_dynamic_state_goto_url_with_filter_navigation_before_extraction_is_allowed(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": (
+ "https://example.com/search?q=budget+headphones&"
+ "state=category%3Delectronics%3Bavailability%3Din_stock"
+ ),
+ },
+ {
+ "block_type": "navigation",
+ "label": "apply_missing_filters",
+ "navigation_goal": (
+ "Inspect the active constraint chips and checked controls, then select any missing "
+ "requested options."
+ ),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_dynamic_state_goto_url_with_code_before_extraction_is_allowed(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": ("https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available"),
+ },
+ {
+ "block_type": "code",
+ "label": "verify_state_from_dom",
+ "code": (
+ "checked_labels = await page.locator('[aria-checked=\"true\"], input:checked').all_text_contents()\n"
+ "if not checked_labels:\n"
+ " raise Exception('No active dynamic page state found')"
+ ),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_dynamic_state_goto_url_without_downstream_extraction_is_allowed(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": "https://example.com/search?q=budget+headphones&facet=color&filter=available",
+ }
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+ def test_structural_state_value_directly_to_extraction_is_detected(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": (
+ "https://example.com/search?q=budget+headphones&"
+ "state=category%3Delectronics%3Bavailability%3Din_stock"
+ ),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
+ {
+ "label": "open_dynamic_results",
+ "dynamic_state_keys": ["state"],
+ "next_data_block_label": "extract_results",
+ }
+ ]
+
+ def test_single_embedded_assignment_directly_to_extraction_is_detected(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": "https://example.com/search?q=budget+headphones&f=shipping%3Dfree",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == [
+ {
+ "label": "open_dynamic_results",
+ "dynamic_state_keys": ["f"],
+ "next_data_block_label": "extract_results",
+ }
+ ]
+
+ def test_url_like_query_values_are_not_treated_as_embedded_state_assignments(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_redirect",
+ "url": "https://example.com/start?redirect=https%3A%2F%2Fexample.org%2Fsearch%3Fx%3Dy",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_unverified_goto_url_state_shortcuts(submitted) == []
+
+
+class TestGotoUrlStableSearchDripDetection:
+ def test_navigation_url_with_stable_search_setup_is_detected(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "navigation",
+ "label": "enter_query",
+ "url": "https://example.com/",
+ "navigation_goal": (
+ "```Dismiss any pop-up if it appears, then enter budget headphones in the product "
+ "search field without submitting.```"
+ ),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_navigation_url_stable_search_setups(submitted) == [
+ {"label": "enter_query", "url": "https://example.com/"}
+ ]
+
+ def test_navigation_url_without_stable_search_setup_is_allowed(self) -> None:
+ submitted = _yaml(
+ {
+ "block_type": "navigation",
+ "label": "open_article_and_accept_cookies",
+ "url": "https://example.com/article",
+ "navigation_goal": "```Accept the cookie banner if present.```",
+ },
+ {"block_type": "extraction", "label": "extract_article", "data_extraction_goal": "Extract article"},
+ )
+
+ assert _detect_navigation_url_stable_search_setups(submitted) == []
+
+ def test_multiple_click_driven_stable_search_navigation_blocks_are_detected(self) -> None:
+ submitted = _yaml(
+ {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},
+ {
+ "block_type": "navigation",
+ "label": "enter_query",
+ "navigation_goal": (
+ "Achieve the following mini goal and once it's achieved, complete: "
+ "```Enter budget headphones in the product search field. Do not submit the search yet.```"
+ ),
+ },
+ {
+ "block_type": "navigation",
+ "label": "set_dates",
+ "navigation_goal": (
+ "Achieve the following mini goal and once it's achieved, complete: "
+ "```Select July 10, 2026 as the start date and July 12, 2026 as the end date.```"
+ ),
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_click_driven_stable_search_drips(submitted) == [
+ {
+ "entry_label": "open_search_site",
+ "navigation_labels": ["enter_query", "set_dates"],
+ }
+ ]
+
+ def test_stable_search_navigation_after_dom_inspection_is_allowed(self) -> None:
+ submitted = _yaml(
+ {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},
+ {
+ "block_type": "navigation",
+ "label": "enter_query",
+ "navigation_goal": "```Enter budget headphones in the product search field.```",
+ },
+ {
+ "block_type": "code",
+ "label": "inspect_search_form",
+ "code": "form_action = await page.locator('form').first.get_attribute('action')",
+ },
+ {
+ "block_type": "navigation",
+ "label": "set_dates",
+ "navigation_goal": "```Select July 10, 2026 as the start date and July 12, 2026 as the end date.```",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_click_driven_stable_search_drips(submitted) == []
+
+ def test_single_stable_search_navigation_before_submit_is_allowed(self) -> None:
+ submitted = _yaml(
+ {"block_type": "goto_url", "label": "open_search_site", "url": "https://example.com/"},
+ {
+ "block_type": "navigation",
+ "label": "enter_query",
+ "navigation_goal": "```Enter budget headphones in the product search field.```",
+ },
+ {
+ "block_type": "navigation",
+ "label": "submit_search",
+ "navigation_goal": "```Click the Search button to submit and show results.```",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+
+ assert _detect_click_driven_stable_search_drips(submitted) == []
+
+
+def _ctx(prior_yaml: str | None = None) -> MagicMock:
+ ctx = MagicMock()
+ ctx.workflow_yaml = prior_yaml
+ ctx.workflow_id = "w_test"
+ ctx.workflow_permanent_id = "wpid_test"
+ ctx.organization_id = "o_test"
+ return ctx
+
+
+async def test_update_workflow_rejects_unverified_goto_url_state_shortcut_and_emits_span() -> None:
+ submitted = _yaml(
+ {
+ "block_type": "goto_url",
+ "label": "open_dynamic_results",
+ "url": "https://example.com/search?q=budget+headphones&facet=color&facet=size&filter=available",
+ },
+ {"block_type": "extraction", "label": "extract_results", "data_extraction_goal": "Extract results"},
+ )
+ ctx = _ctx()
+
+ with (
+ patch("skyvern.forge.sdk.copilot.tools._record_goto_url_state_shortcut_reject_span") as mock_span,
+ patch("skyvern.forge.sdk.copilot.tools.app") as mock_app,
+ ):
+ mock_app.WORKFLOW_SERVICE.update_workflow_definition = AsyncMock()
+ result = await _update_workflow({"workflow_yaml": submitted}, ctx)
+
+ assert result["ok"] is False
+ assert "goto_url may bootstrap stable page state" in result["error"]
+ assert "URL-encoded dynamic page state must be verified" in result["error"]
+ assert "open_dynamic_results" in result["error"]
+ assert "extract_results" in result["error"]
+ mock_app.WORKFLOW_SERVICE.update_workflow_definition.assert_not_called()
+ mock_span.assert_called_once()
+
+
+class TestPromptAndToolDescriptions:
+ def test_agent_prompt_contains_goto_url_state_policy(self) -> None:
+ rendered = _render_agent_prompt()
+
+ assert "GOTO_URL STATE SHORTCUT POLICY" in rendered
+ assert "URL refinement for dynamic page state is allowed ONLY when grounded in live site evidence" in rendered
+ assert "Before extraction on stateful search/result tasks" in rendered
+ assert "do not spend one click-driven `navigation` block per field after the entry page loads" in rendered
+ assert "current URL, form action, field names/values, result links, or URL deltas" in rendered
+ assert "use `code` for DOM checks" in rendered
+ assert "per-tool budget" in rendered
+ assert "constraint / expected / observed / source" in rendered
+
+ def test_tool_descriptions_warn_against_direct_state_shortcut_extraction(self) -> None:
+ for tool in (update_workflow_tool, run_blocks_tool, update_and_run_blocks_tool):
+ desc = tool.description # type: ignore[attr-defined]
+ assert "goto_url" in desc
+ assert "dynamic page state" in desc
+ assert "code" in desc
+ assert "validation" in desc
diff --git a/tests/unit/test_copilot_hooks.py b/tests/unit/test_copilot_hooks.py
index 5466b1209..e99c81d28 100644
--- a/tests/unit/test_copilot_hooks.py
+++ b/tests/unit/test_copilot_hooks.py
@@ -219,7 +219,7 @@ class TestSchemaOverlay:
arg_transforms={"clear_first": "clear"},
forced_args={"inline": True},
)
- args = {"clear_first": True, "text": "hello"}
+ args = {"clear_first": True, "text": "hello", "_summarized": "older tool call"}
result = _transform_args(args, overlay)
assert result == {"clear": True, "text": "hello", "inline": True}
assert "clear_first" not in result
diff --git a/tests/unit/test_copilot_per_tool_budget.py b/tests/unit/test_copilot_per_tool_budget.py
index 1691389e4..16fcafa9f 100644
--- a/tests/unit/test_copilot_per_tool_budget.py
+++ b/tests/unit/test_copilot_per_tool_budget.py
@@ -33,7 +33,13 @@ from skyvern.forge.sdk.copilot.failure_tracking import (
PER_TOOL_BUDGET_FAILURE_CATEGORY,
compute_failure_signature,
)
-from skyvern.forge.sdk.copilot.tools import _maybe_clear_reconciliation_flag, _record_run_blocks_result
+from skyvern.forge.sdk.copilot.tools import (
+ _maybe_clear_reconciliation_flag,
+ _record_per_tool_budget_problem_blocks_from_results,
+ _record_run_blocks_result,
+ _record_workflow_update_result,
+ _tool_loop_error,
+)
def _fresh_context() -> CopilotContext:
@@ -283,6 +289,126 @@ def test_reconciliation_does_not_clear_when_run_id_mismatches() -> None:
assert ctx.pending_reconciliation_run_id == "wr_1"
+def test_get_run_results_arms_problem_navigation_label_for_budget_run() -> None:
+ ctx = _fresh_context()
+ ctx.pending_reconciliation_run_id = "wr_1"
+ ctx.last_failure_category_top = PER_TOOL_BUDGET_FAILURE_CATEGORY
+
+ _record_per_tool_budget_problem_blocks_from_results(
+ ctx,
+ {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_1",
+ "overall_status": "canceled",
+ "blocks": [
+ {"label": "open_results", "block_type": "GOTO_URL", "status": "completed"},
+ {"label": "apply_filters", "block_type": "NAVIGATION", "status": "canceled"},
+ {"label": "extract_results", "block_type": "EXTRACTION", "status": "created"},
+ ],
+ },
+ },
+ )
+
+ assert ctx.per_tool_budget_problem_block_labels == ["apply_filters"]
+
+
+def test_get_run_results_does_not_arm_problem_label_for_different_run() -> None:
+ ctx = _fresh_context()
+ ctx.pending_reconciliation_run_id = "wr_1"
+ ctx.last_failure_category_top = PER_TOOL_BUDGET_FAILURE_CATEGORY
+
+ _record_per_tool_budget_problem_blocks_from_results(
+ ctx,
+ {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_other",
+ "overall_status": "canceled",
+ "blocks": [{"label": "apply_filters", "block_type": "NAVIGATION", "status": "canceled"}],
+ },
+ },
+ )
+
+ assert ctx.per_tool_budget_problem_block_labels == []
+
+
+def test_tool_loop_error_blocks_rerun_of_problem_navigation_label() -> None:
+ ctx = _fresh_context()
+ ctx.per_tool_budget_problem_block_labels = ["apply_filters"]
+
+ msg = _tool_loop_error(
+ ctx,
+ "run_blocks_and_collect_debug",
+ {"block_labels": ["apply_filters"], "parameters": {}},
+ )
+
+ assert msg is not None
+ assert "apply_filters" in msg
+ assert "Do NOT rerun" in msg
+ assert "code or validation" in msg
+
+
+def test_tool_loop_error_treats_missing_labels_as_rerun_all() -> None:
+ ctx = _fresh_context()
+ ctx.per_tool_budget_problem_block_labels = ["apply_filters"]
+
+ msg = _tool_loop_error(ctx, "run_blocks_and_collect_debug", {"parameters": {}})
+
+ assert msg is not None
+ assert "apply_filters" in msg
+
+
+def test_tool_loop_error_allows_new_smaller_label_after_budget_problem() -> None:
+ ctx = _fresh_context()
+ ctx.per_tool_budget_problem_block_labels = ["apply_filters"]
+
+ assert (
+ _tool_loop_error(
+ ctx,
+ "run_blocks_and_collect_debug",
+ {"block_labels": ["click_parking"], "parameters": {}},
+ )
+ is None
+ )
+
+
+def test_successful_workflow_update_keeps_problem_label_while_still_navigation() -> None:
+ ctx = _fresh_context()
+ ctx.workflow_yaml = "updated yaml"
+ ctx.per_tool_budget_problem_block_labels = ["apply_filters"]
+ workflow = SimpleNamespace(
+ workflow_definition=SimpleNamespace(
+ blocks=[
+ SimpleNamespace(block_type="navigation", label="apply_filters"),
+ SimpleNamespace(block_type="extraction", label="extract_results"),
+ ]
+ )
+ )
+
+ _record_workflow_update_result(ctx, {"ok": True, "_workflow": workflow, "data": {"block_count": 2}})
+
+ assert ctx.per_tool_budget_problem_block_labels == ["apply_filters"]
+
+
+def test_successful_workflow_update_clears_problem_label_changed_away_from_navigation() -> None:
+ ctx = _fresh_context()
+ ctx.workflow_yaml = "updated yaml"
+ ctx.per_tool_budget_problem_block_labels = ["apply_filters"]
+ workflow = SimpleNamespace(
+ workflow_definition=SimpleNamespace(
+ blocks=[
+ SimpleNamespace(block_type="code", label="apply_filters"),
+ SimpleNamespace(block_type="navigation", label="click_parking"),
+ ]
+ )
+ )
+
+ _record_workflow_update_result(ctx, {"ok": True, "_workflow": workflow, "data": {"block_count": 2}})
+
+ assert ctx.per_tool_budget_problem_block_labels == []
+
+
def test_nudge_counter_resets_on_success() -> None:
"""The counter must reset so a budget trip on a different chain in the
same turn gets the dedicated split nudge again."""