Skip to content

Automate browser based workflows with AI

SKY-13777 Stop telling the copilot that a challenge on the page closes the submit path (#8210)

Andrew Neilson05d4b9764063
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index 76c56802a..e8203f29d 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -88,8 +88,7 @@ Use the Skyvern MCP browser tools to gather ground-truth evidence in ANY phase o
- For form/search/result pages, inspect when you do not know the real controls or result layout. Use `inspect_page_for_composition(target_url="current_page")`, `evaluate`, `get_browser_screenshot`, or targeted interaction tools as needed, then compose normal workflow blocks from the observed page.
- Treat each successful run or browser observation as evidence for the current page state only. Do not author downstream row expansion or extraction that depends on a future page state until you have run the next page-changing block and inspected the reached page. If the visible results already contain the requested values, add extraction from that observed page; if an expansion is required first, add and run the expansion block before adding extraction.
- Add `validation` blocks only when the reusable workflow's task actually includes a durable check. Do not add validation or extra navigation blocks just to double-check the shape of a site before each action.
-- If page evidence shows anti-bot or challenge controls, account for that in the normal action block that reaches/submits the challenged state. Use `get_browser_screenshot` whenever seeing the rendered page answers the question faster than reading the DOM — a challenge whose affordance the DOM does not expose is one such case, but so is confirming what an action changed on screen. Prefer explicit challenge handling available to Skyvern over blind waits; if the latest run shows the challenge cannot be solved, stop and report the observed blocker instead of retrying the same flow.
-- If bounded page evidence says `challenge_state.gates_submit_controls=true`, the submit/search control is unavailable until the challenge is resolved. If a tested workflow already attempted challenge resolution and current-page evidence still shows that submit/search control disabled with no visible result or no-results state, stop and report the anti-bot blocker rather than repeating the same solve/wait/submit sequence.
+- If page evidence shows anti-bot or challenge controls, account for that in the normal action block that reaches/submits the challenged state. Use `get_browser_screenshot` whenever seeing the rendered page answers the question faster than reading the DOM — a challenge whose affordance the DOM does not expose is one such case, but so is confirming what an action changed on screen. Prefer explicit challenge handling available to Skyvern over blind waits. Page evidence about a challenge is an observation, not a verdict: it does not tell you a run will fail, and a run is what settles that.
- If a state-application block hits a per-tool budget, do not rerun the same navigation label unchanged. First read the budgeted run with `get_run_results`. If it reports a current URL or the browser is still on an advanced page, inspect `current_page`; answer from visible result/no-result evidence when possible, and only split/replace/run a smaller UI block after inspection shows a real missing state change.
- When you already have a verified prefix or a live browser page that advanced beyond the workflow's entrypoint, continue from that evidence. Do not turn the observed current page into a new workflow entrypoint and rerun the whole workflow from the start. Preserve unchanged upstream labels and text, and pass `block_labels` for only the missing downstream frontier.
diff --git a/skyvern/forge/sdk/copilot/tools/__init__.py b/skyvern/forge/sdk/copilot/tools/__init__.py
index fc172c960..289108e5d 100644
--- a/skyvern/forge/sdk/copilot/tools/__init__.py
+++ b/skyvern/forge/sdk/copilot/tools/__init__.py
@@ -958,11 +958,8 @@ async def inspect_page_for_composition_tool(
evidence shows required fields or controls that the user did not supply
enough information for, ASK_QUESTION with that observed missing input. If
evidence is sufficient, compose and run workflow blocks from the observed fields.
- If challenge_state.gates_submit_controls is true, treat challenge resolution
- as a prerequisite for submit/search; do not click a submit control while the
- latest inspected evidence says it is disabled. If a later test still leaves
- that submit/search control disabled after a challenge-resolution attempt,
- report the observed anti-bot blocker rather than retrying the same flow.
+ `challenge_state` reports what the page looks like, which is not what a run will do:
+ it does not establish that a submit/search path is closed, and a run settles that.
When the page visibly shows a requested output but its markup is unclear, pass
`requested_output_reads` with the `output_path` your block will return, the exact
diff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.py
index 7bf168b47..8dc92370b 100644
--- a/skyvern/forge/sdk/copilot/tools/run_execution.py
+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py
@@ -702,9 +702,7 @@ async def _watchdog_error_message(
f"satisfy the bounded page-evidence contract for workflow mutations. Use the "
f"bounded evidence to decide whether the answer is already visible, whether "
f"a challenge-gated submit/search control is still disabled, or which page-state "
- f"change is still missing. If challenge_state.gates_submit_controls=true and "
- f"the requested answer is not visible, stop and report the observed anti-bot "
- f"blocker instead of retrying the same solve/wait/submit chain. Only then call "
+ f"change is still missing. Then call "
f"update_and_run_blocks with a smaller chain — the first 1-2 unverified blocks. "
f"Verified-prefix state is preserved, so the next call only re-runs from the new frontier. "
f"Do NOT retry the same chain unchanged — a longer "
diff --git a/tests/unit/test_copilot_challenge_evidence_is_not_a_verdict.py b/tests/unit/test_copilot_challenge_evidence_is_not_a_verdict.py
new file mode 100644
index 000000000..ed05142f8
--- /dev/null
+++ b/tests/unit/test_copilot_challenge_evidence_is_not_a_verdict.py
@@ -0,0 +1,51 @@
+"""A challenge in page evidence is an observation the model reads, never an instruction to stop.
+
+The veto that refused a block-run tool on this evidence is gone. These pin the quieter form of the
+same rule: no prompt, tool description, or tool result may condition "stop and report" on
+`challenge_state`, because a page that looks challenged has not established that a run will fail —
+in the production session this ticket came from, the belief was wrong three times while the solver
+succeeded 42 times on the same site.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+# The surfaces the model actually reads: its standing prompt, the description on the tool that
+# produces the evidence, and the run tool's own budget-exit result.
+MODEL_FACING_SOURCES = (
+ REPO_ROOT / "skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2",
+ REPO_ROOT / "skyvern/forge/sdk/copilot/tools/__init__.py",
+ REPO_ROOT / "skyvern/forge/sdk/copilot/tools/run_execution.py",
+)
+
+STOP_PHRASES = (
+ "stop and report the anti-bot blocker",
+ "stop and report the observed anti-bot",
+ "report the observed anti-bot blocker rather than retrying",
+ "treat challenge resolution",
+)
+
+
[email protected]("source", MODEL_FACING_SOURCES, ids=lambda p: p.name)
+def test_no_model_facing_surface_orders_a_stop_from_challenge_evidence(source: Path) -> None:
+ text = source.read_text()
+ found = [phrase for phrase in STOP_PHRASES if phrase in text]
+ assert not found, f"{source.name} still tells the model to stop on challenge evidence: {found}"
+
+
[email protected]("source", MODEL_FACING_SOURCES, ids=lambda p: p.name)
+def test_challenge_state_is_never_the_condition_for_abandoning_a_run(source: Path) -> None:
+ # Reading challenge_state is fine and expected; conditioning a retreat on it is not.
+ for line_number, line in enumerate(source.read_text().splitlines(), start=1):
+ if "gates_submit_controls" not in line:
+ continue
+ lowered = line.lower()
+ assert "stop and report" not in lowered, f"{source.name}:{line_number} conditions a stop on challenge state"
+ assert "rather than retrying" not in lowered, (
+ f"{source.name}:{line_number} conditions a retreat on challenge state"
+ )