Skip to content

Automate browser based workflows with AI

Let Copilot read image-text CAPTCHAs while scouting and in saved code blocks (SKY-16361) (#8683)

Andrew Neilson9d6cf0d4f8b4
diff --git a/skyvern-frontend/src/components/BrowserStream.test.tsx b/skyvern-frontend/src/components/BrowserStream.test.tsx
index 66a0bc27e..915b59b1f 100644
--- a/skyvern-frontend/src/components/BrowserStream.test.tsx
+++ b/skyvern-frontend/src/components/BrowserStream.test.tsx
@@ -1,7 +1,6 @@
import { StrictMode, type ComponentProps } from "react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import {
- act,
cleanup,
fireEvent,
render,
@@ -716,9 +715,7 @@ describe("BrowserStream", () => {
try {
const onStreamStateChange = vi.fn();
renderBrowserStream({ onStreamStateChange });
- await act(async () => {
- await vi.advanceTimersByTimeAsync(1000);
- });
+ await vi.advanceTimersByTimeAsync(1000);
expect(mocks.rfbInstances).toHaveLength(1);
expect(onStreamStateChange).toHaveBeenLastCalledWith("live", "pbs_test");
mocks.autoConnect.value = false;
diff --git a/skyvern/forge/prompts/skyvern/workflow_knowledge_base.txt b/skyvern/forge/prompts/skyvern/workflow_knowledge_base.txt
index ab2624515..84d9372d4 100644
--- a/skyvern/forge/prompts/skyvern/workflow_knowledge_base.txt
+++ b/skyvern/forge/prompts/skyvern/workflow_knowledge_base.txt
@@ -740,7 +740,7 @@ Purpose: Pause the workflow and wait for a person. This is the "Human Interactio
Use Cases:
- A person approves before the workflow continues (high-value action, compliance review, a check of extracted data)
-- A person does a step by hand in the browser that has no automated path: solving a CAPTCHA, biometric or voice verification, or a sign-in that credentialed login cannot complete
+- A person does a step by hand in the browser that has no automated path: a CAPTCHA the solver and the image-text form of `solve_captcha` cannot clear, biometric or voice verification, or a sign-in that credentialed login cannot complete
- Escalation for a case the agent is not allowed to decide on its own
Structure:
@@ -776,8 +776,11 @@ Purpose: The managed captcha solver is a product capability, enabled by default
How a run reaches it:
- Task-shaped blocks (navigation, login, action) invoke the solver automatically when a challenge is detected.
- A code block invokes it only when its code calls the builtin `await solve_captcha(page)`; nothing runs the solver for a code block otherwise. A failed solve raises inside the code block and the run's action trace records a `solve_captcha` action with status `failed`.
+- A distorted-text image CAPTCHA (characters drawn in an image, typed into a field) is read by the image form, `await solve_captcha(page, image=<selector>, input=<selector>)`. In Workflow Copilot, while scouting, `solve_page_challenge` with `image` and `input`, or the same call inside `run_browser_code`, runs the same OCR, so the selectors are proven on the live page before they are saved. `image` selects the observed <img>, <svg> or <canvas> holding the text and `input` the observed answer field; both come from the inspected page. It reads the text and types it, and does not submit. It requires image OCR enabled for the organization; otherwise it raises. It raises with nothing typed when the image yields no text, and the runtime enforces a per-block call limit, raising on every call past it. So the block runs a short bounded loop: solve; if the call raised, click the observed refresh control and retry; otherwise submit, and if the form is still shown, click the refresh control and retry. With no refresh control, raise instead of retrying; raise after the last try. An answer typed while scouting fits only that one image and never belongs in the saved code.
-Reading the run: the build-test packet's `challenge` record says whether the solver is available for this organization (`solver_available`), whether this run invoked it (`solver_attempted`, absent when unknown), and what happened (`solver_result`: failed, attempted, not_attempted, or unresolved). "not_attempted" on a code block means the code never called the builtin, not that the product declined. "attempted" means the solve step ran and returned without an error, which is not the same as the challenge clearing: when no solver handles the page the call still reports success. "unresolved" means the run's action history was unavailable, so whether the solver ran is unknown and should not be stated either way. `solver_available` is resolved per page, and is absent when it was only resolved for a different URL.
+Reading the run: the build-test packet's `challenge` record says whether the solver is available for this organization (`solver_available`), whether this run invoked it (`solver_attempted`, absent when unknown), and what happened (`solver_result`: failed, attempted, not_attempted, or unresolved). "not_attempted" on a code block means the code never called the builtin, not that the product declined. "attempted" means the solve step ran and returned without an error, which is not the same as the challenge clearing: when no solver handles the page the call still reports success. "unresolved" means the run's action history was unavailable or cut short, so whether the solver ran is unknown and should not be stated either way. `solver_available` is resolved per page, and is absent when it was only resolved for a different URL.
+
+A run that did not pass and met no wall can still carry a `challenge` record with the solver keys and no levers: it reports the last `solve_captcha` call in a code block, not that a challenge blocked the run. "attempted" there means that call returned without an error, for either form: the image form typed an answer and the widget fallback also returns success, so neither says the site accepted anything. Whether the form moved on is read from the final page and the block status.
** PROXY LOCATION AND BROWSER PROFILE (proxy_location) **
diff --git a/skyvern/forge/sdk/copilot/build_test_outcome.py b/skyvern/forge/sdk/copilot/build_test_outcome.py
index 1d070b9df..c53029485 100644
--- a/skyvern/forge/sdk/copilot/build_test_outcome.py
+++ b/skyvern/forge/sdk/copilot/build_test_outcome.py
@@ -7,7 +7,7 @@ import re
import textwrap
from collections.abc import Iterable, Mapping, Sequence
from itertools import islice
-from typing import Any, Literal, Protocol, TypeVar, get_args
+from typing import Any, Literal, Protocol, TypedDict, TypeVar, get_args
from urllib.parse import urlsplit
import structlog
@@ -383,11 +383,22 @@ class BuildTestPacketUnfinishedItem(BaseModel):
SOLVER_ATTEMPT_KEY = "solver_attempt"
+ACTION_TRACE_PER_TASK_LIMIT = 15
SolverResult = Literal["failed", "not_solved", "attempted", "not_attempted", "unresolved"]
SOLVER_RESULTS: frozenset[str] = frozenset(get_args(SolverResult))
+class SolverFacts(TypedDict):
+ attempted: bool
+ result: SolverResult
+ failure: str | None
+
+
+class SolverAttempt(SolverFacts, total=False):
+ code_block: SolverFacts
+
+
class ChallengeEffects(BaseModel):
"""What the product observed and did about an anti-bot wall on this run; facts, not a verdict."""
@@ -396,7 +407,7 @@ class ChallengeEffects(BaseModel):
kind: str | None = None
# ``page_frames`` means only a vendor frame the page mounted or declared built this record; it is
# the default because the dump excludes the field, so a revalidated record cannot mint a wall.
- basis: Literal["run_wall", "page_frames"] = Field(default="page_frames", exclude=True, repr=False)
+ basis: Literal["run_wall", "page_frames", "solver_call"] = Field(default="page_frames", exclude=True, repr=False)
solver_available: bool | None = None
solver_attempted: bool | None = None
solver_result: SolverResult | None = None
@@ -413,6 +424,20 @@ def challenge_notices(challenge: ChallengeEffects | None, levers: list[Lever]) -
"challenge frames: the final page mounted or declared a frame served by a challenge vendor; that is "
f"what the page holds, not a finding that this run was blocked. Frame hosts: {hosts}."
)
+ elif challenge is not None and challenge.basis == "solver_call":
+ if challenge.solver_result == "failed":
+ outcome = "raised (see `solver_failure`)"
+ elif challenge.solver_result == "not_solved":
+ outcome = "returned `false`, the solver's own report that it did nothing"
+ else:
+ outcome = "returned without error, which for either form only means it ran"
+ frames = ""
+ if challenge.frame_hosts:
+ frames = f" Vendor frames on the final page: {', '.join(challenge.frame_hosts)}."
+ notices.append(
+ f"solver call: the last `solve_captcha` call in this run's code {outcome}; a call is an attempt, not "
+ f"page advancement, so the final page and block status say whether the form moved on.{frames}"
+ )
elif challenge is not None:
kind = challenge.kind or "unclassified"
if challenge.solver_available is True:
@@ -426,26 +451,26 @@ def challenge_notices(challenge: ChallengeEffects | None, levers: list[Lever]) -
availability = "the managed captcha solver's availability was not resolved"
if challenge.solver_result == "attempted":
outcome = (
- "this run called `solve_captcha(page)` and the call returned without an error, which does not by "
+ "this run called `solve_captcha` and the call returned without an error, which does not by "
"itself mean the challenge cleared: the no-solver fallback also returns success"
)
elif challenge.solver_result == "not_solved":
outcome = (
- "this run called `solve_captcha(page)` and the call returned `false`, the solver's own report that "
+ "this run called `solve_captcha` and the call returned `false`, the solver's own report that "
"it cleared nothing; separately, a challenge was recorded for this run, and the call may have run "
"before that challenge appeared"
)
elif challenge.solver_result == "failed":
- outcome = "this run called `solve_captcha(page)` and the solver did not clear it"
+ outcome = "this run called `solve_captcha` and the solver did not clear it"
if challenge.solver_failure:
outcome += f" ({challenge.solver_failure})"
elif challenge.solver_result == "not_attempted":
outcome = (
- "no `solve_captcha(page)` call appears in this run's recorded actions; a code block reaches the "
+ "no `solve_captcha` call appears in this run's recorded actions; a code block reaches the "
"solver only when its code calls that builtin"
)
else:
- outcome = "whether this run called `solve_captcha(page)` is unresolved, so do not state either way"
+ outcome = "whether this run called `solve_captcha` is unresolved, so do not state either way"
notices.append(f"challenge: {kind}; {availability}; {outcome}.")
if levers:
names = ", ".join(lever.mechanism for lever in levers)
diff --git a/skyvern/forge/sdk/copilot/diagnosis_repair_contract.py b/skyvern/forge/sdk/copilot/diagnosis_repair_contract.py
index d1c331282..5eb2bac0f 100644
--- a/skyvern/forge/sdk/copilot/diagnosis_repair_contract.py
+++ b/skyvern/forge/sdk/copilot/diagnosis_repair_contract.py
@@ -3,17 +3,19 @@ from __future__ import annotations
import hashlib
import json
import re
-from collections.abc import Mapping
+from collections.abc import Mapping, Sequence
from enum import StrEnum
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel, ConfigDict, Field, ValidationError
from skyvern.forge.sdk.copilot.build_test_outcome import (
+ ACTION_TRACE_PER_TASK_LIMIT,
SOLVER_ATTEMPT_KEY,
SOLVER_RESULTS,
ChallengeEffects,
Lever,
+ SolverFacts,
failed_block_bound_credential_ids,
)
from skyvern.forge.sdk.copilot.challenge_evidence import (
@@ -41,7 +43,9 @@ from skyvern.forge.sdk.copilot.runtime_authoring_repair import (
)
from skyvern.forge.sdk.copilot.workflow_credential_utils import URL_CANDIDATE_RE
from skyvern.schemas.proxy_location import GeoTarget, ProxyLocationInput
+from skyvern.schemas.workflows import BlockType
from skyvern.webeye.actions.action_types import ActionType
+from skyvern.webeye.actions.actions import ActionStatus
if TYPE_CHECKING:
from skyvern.forge.sdk.copilot.context import CopilotContext
@@ -212,7 +216,7 @@ def build_diagnosis_repair_contract(
challenge_runtime_clearable=run_challenge_is_runtime_clearable(ctx, run_id_from_result_data(data)),
)
next_action = _next_action(failure_type, ctx, data, repair_context)
- challenge = _challenge_effects(ctx, blocks, categories, data)
+ challenge = _challenge_effects(ctx, blocks, categories, data, run_passed=run_ok and not suspicious)
frontier = _safe_str(data.get("frontier_start_label"))
target_blocks = failed_blocks or ([frontier] if frontier else []) if next_action == RepairNextAction.REPAIR else []
if next_action == RepairNextAction.REPAIR and not target_blocks and repair_context is not None:
@@ -321,47 +325,59 @@ def build_diagnosis_repair_contract(
)
-def _solver_facts(blocks: list[Any], data: dict[str, Any]) -> tuple[str, str | None]:
- """Returns (result, failure_text) over failed / not_solved / attempted / not_attempted / unresolved.
-
- A completed row carrying no boolean means the solve step ran, not that the challenge cleared: the
- runtime's no-solver fallback also reports success. An absent action history means unresolved, not a
- non-attempt: the optional lookup swallows its own failures, and a later tool result in the
- same turn carries no run history at all."""
- """Prefer the record captured before the traces were stripped; fall back to any trace still present."""
- carried = data.get(SOLVER_ATTEMPT_KEY)
- if isinstance(carried, dict):
- result = str(carried.get("result") or "unresolved")
- if result not in SOLVER_RESULTS:
- result = "unresolved"
- return result, _safe_text(_safe_str(carried.get("failure")), _SOLVER_FAILURE_MAX)
- attempted = False
- failed = False
- not_solved = False
+def solver_facts_from_traces(blocks: Sequence[object], *, code_block_only: bool = False) -> SolverFacts:
+ """A completed row with no boolean means the step ran, not that the challenge cleared, since the no-solver
+ fallback also reports success. An absent history is unresolved, not a non-attempt, because the lookup
+ swallows its own failures."""
saw_history = False
- failure: str | None = None
+ truncated = False
+ rows: list[Mapping[str, Any]] = []
for block in blocks:
- trace = block.get("action_trace") if isinstance(block, dict) else None
+ if not isinstance(block, Mapping):
+ continue
+ if code_block_only and str(block.get("block_type") or "").upper() != BlockType.CODE.name:
+ continue
+ trace = block.get("action_trace")
if not isinstance(trace, list):
continue
saw_history = True
- for entry in trace:
- if not isinstance(entry, dict) or entry.get("action") != ActionType.SOLVE_CAPTCHA.value:
- continue
- attempted = True
- if entry.get("status") == "failed":
- failed = True
- if failure is None:
- failure = _safe_text(_safe_str(entry.get("response")), _SOLVER_FAILURE_MAX)
- elif entry.get("solver_cleared") is False:
- not_solved = True
- if failed:
- return "failed", failure
- if not_solved:
- return "not_solved", failure
- if attempted:
- return "attempted", failure
- return ("not_attempted" if saw_history else "unresolved"), failure
+ block_rows = [
+ entry
+ for entry in trace
+ if isinstance(entry, Mapping) and entry.get("action") == ActionType.SOLVE_CAPTCHA.value
+ ]
+ if code_block_only and block_rows:
+ # Blocks are chronological and each trace is newest-first, so this is the newest call; a raise
+ # the code's own retry loop recovered from must not outrank it.
+ rows = block_rows[:1]
+ truncated = False
+ elif code_block_only and len(trace) >= ACTION_TRACE_PER_TASK_LIMIT:
+ # A full trace may have cut this block's own call, so an earlier block's row cannot stand in for it.
+ truncated = True
+ else:
+ rows.extend(block_rows)
+ if truncated:
+ return {"attempted": False, "result": "unresolved", "failure": None}
+ failed_row = next((row for row in rows if row.get("status") == ActionStatus.failed.value), None)
+ if failed_row is not None:
+ failure = _safe_text(_safe_str(failed_row.get("response")), _SOLVER_FAILURE_MAX) or None
+ return {"attempted": True, "result": "failed", "failure": failure}
+ if any(row.get("solver_cleared") is False for row in rows):
+ return {"attempted": True, "result": "not_solved", "failure": None}
+ if rows:
+ return {"attempted": True, "result": "attempted", "failure": None}
+ return {"attempted": False, "result": "not_attempted" if saw_history else "unresolved", "failure": None}
+
+
+def _solver_facts(blocks: list[Any], data: dict[str, Any], *, code_block_only: bool = False) -> tuple[str, str | None]:
+ record = data.get(SOLVER_ATTEMPT_KEY)
+ if not isinstance(record, dict):
+ facts = solver_facts_from_traces(blocks, code_block_only=code_block_only)
+ return facts["result"], facts["failure"]
+ carried = record.get("code_block") if code_block_only else record
+ if not isinstance(carried, dict) or str(carried.get("result")) not in SOLVER_RESULTS:
+ return "unresolved", None
+ return str(carried["result"]), _safe_text(_safe_str(carried.get("failure")), _SOLVER_FAILURE_MAX) or None
def _same_run_challenge_frame_hosts(ctx: CopilotContext, data: dict[str, Any]) -> list[str]:
@@ -374,10 +390,11 @@ def _same_run_challenge_frame_hosts(ctx: CopilotContext, data: dict[str, Any]) -
def _challenge_effects(
- ctx: CopilotContext, blocks: list[Any], categories: list[str], data: dict[str, Any]
+ ctx: CopilotContext, blocks: list[Any], categories: list[str], data: dict[str, Any], *, run_passed: bool
) -> ChallengeEffects | None:
- """A run_wall record with solver facts when the run met a wall, a page_frames record carrying only
- frame hosts when the final page merely mounted a vendor frame, and None otherwise."""
+ """A run_wall record with solver facts when the run met a wall, a solver_call record when a code block's
+ solver call ran on a run that did not pass with no wall, a page_frames record when the final page merely
+ mounted a vendor frame, and None otherwise."""
frame_hosts = _same_run_challenge_frame_hosts(ctx, data)
result, failure = _solver_facts(blocks, data)
walled = any(category in ANTI_BOT_CHALLENGE_ALIAS_CATEGORIES for category in categories) or bool(
@@ -389,6 +406,16 @@ def _challenge_effects(
walled = bool(frame_hosts) or (
same_run_typed_challenge_kind(ctx.composition_page_evidence, run_id_from_result_data(data)) is not None
)
+ if not walled and not run_passed:
+ code_result, code_failure = _solver_facts(blocks, data, code_block_only=True)
+ if code_result in {"failed", "not_solved", "attempted"}:
+ return ChallengeEffects(
+ basis="solver_call",
+ solver_attempted=True,
+ solver_result=code_result,
+ solver_failure=code_failure,
+ frame_hosts=frame_hosts or None,
+ )
if not walled:
return ChallengeEffects(basis="page_frames", frame_hosts=frame_hosts) if frame_hosts else None
kind = typed_challenge_kind(ctx.composition_page_evidence)
diff --git a/skyvern/forge/sdk/copilot/output_utils.py b/skyvern/forge/sdk/copilot/output_utils.py
index 62027e411..0a8c03de2 100644
--- a/skyvern/forge/sdk/copilot/output_utils.py
+++ b/skyvern/forge/sdk/copilot/output_utils.py
@@ -1946,6 +1946,7 @@ def _workflow_write_phrase(data: dict[str, Any]) -> str:
_PAGE_CHALLENGE_OUTCOME_SUMMARIES = {
"solved": "Challenge solver reported the challenge solved",
+ "typed": "Challenge solver typed the CAPTCHA image's text",
"none": "Challenge solver found no challenge on the page",
"unsupported": "Challenge solver found no challenge it can operate",
"unsolved": "Challenge solver could not clear the challenge in this browser",
diff --git a/skyvern/forge/sdk/copilot/runtime.py b/skyvern/forge/sdk/copilot/runtime.py
index 116f0f446..a752e0dd0 100644
--- a/skyvern/forge/sdk/copilot/runtime.py
+++ b/skyvern/forge/sdk/copilot/runtime.py
@@ -476,6 +476,7 @@ class AgentContext:
browser_session_continuity_deadline_expired: bool = False
# Each unsolved challenge solve can bill an external solver, so it is counted per browser session.
unsolved_page_challenges_by_session_id: dict[str, int] = field(default_factory=dict)
+ image_captcha_reads: int = 0
supports_vision: bool = True
pending_screenshots: list[ScreenshotEntry] = field(default_factory=list)
pending_frame_lease: PendingFrameLease | None = None
diff --git a/skyvern/forge/sdk/copilot/tools/__init__.py b/skyvern/forge/sdk/copilot/tools/__init__.py
index 75c4c29e6..c4f4a8b25 100644
--- a/skyvern/forge/sdk/copilot/tools/__init__.py
+++ b/skyvern/forge/sdk/copilot/tools/__init__.py
@@ -1640,28 +1640,36 @@ async def search_web_tool(ctx: RunContextWrapper, query: str, max_results: int =
@function_tool(failure_error_function=copilot_tool_failure, name_override=SOLVE_TOOL_NAME)
-async def solve_page_challenge_tool(ctx: RunContextWrapper) -> str:
+async def solve_page_challenge_tool(ctx: RunContextWrapper, image: str | None = None, input: str | None = None) -> str:
"""Run the platform captcha solver on the current page of this chat's browser.
Use it when the page shows a human-verification or anti-bot challenge: a navigate result's
- `challenge_vendor`, or a challenge you see in a screenshot. It detects reCAPTCHA, hCaptcha and
- Cloudflare Turnstile widgets, including ones inside frames, and can take up to 120 seconds.
-
- `outcome` is one of: `solved`; `none` (no challenge detected, nothing ran); `unsupported` (a
- challenge frame is on screen but the solver found nothing it can operate); `unsolved` (with
- `timed_out` or `solver_failed` when that is why); or `unavailable` (solving is off for this
- organization or page). `solved` is the solver's report, not proof the page moved on: look at the
- page again before continuing. Each attempt can bill an external solver.
-
- `unsolved` and `unsupported` describe this browser session only. Many sites decide per browser
- whether to challenge, from its cookies and history, so a new session from `start_fresh_browser`
+ `challenge_vendor`, or a challenge you see in a screenshot. With no arguments it detects reCAPTCHA,
+ hCaptcha and Cloudflare Turnstile widgets, including ones inside frames, and can take up to 120 seconds.
+
+ For a distorted-text image CAPTCHA, pass `image`, the selector of its <img>, <svg> or <canvas> (or a
+ container holding exactly one), and `input`, the selector of its answer field. The OCR a saved code
+ block's `solve_captcha(page, image=..., input=...)` uses reads that image and types the text into that
+ field, so selectors that work here are the ones to save. It requires image OCR enabled for the
+ organization.
+
+ `outcome` is one of: `solved`; `typed` (image form: the text was typed, unconfirmed until the page
+ accepts it); `none` (no challenge detected, nothing ran); `unsupported` (a challenge frame is on screen
+ but the solver found nothing it can operate); `unsolved` (with `timed_out`, `solver_failed` or, for the
+ image form, `read_limit_reached` when that is why); or `unavailable` (solving or image OCR is off for
+ this organization or page). `solved` and `typed` are the solver's report, not proof the page moved on:
+ look at the page again before continuing. Each attempt can bill an external solver.
+
+ For a widget, `unsolved` and `unsupported` describe this browser session only. Many sites decide per
+ browser whether to challenge, from its cookies and history, so a new session from `start_fresh_browser`
is a separate attempt; the result says whether this request has made it yet.
"""
authority_error = _authority_tool_error(ctx.context, SOLVE_TOOL_NAME)
if authority_error:
return _diagnosis_repair_tool_error(ctx.context, SOLVE_TOOL_NAME, authority_error)
- result = await solve_page_challenge(ctx.context)
- record_tool_step_result_for_ctx(ctx.context, SOLVE_TOOL_NAME, {}, result)
+ result = await solve_page_challenge(ctx.context, image=image, input=input)
+ arguments = {} if image is None and input is None else {"image": image, "input": input}
+ record_tool_step_result_for_ctx(ctx.context, SOLVE_TOOL_NAME, arguments, result)
return json.dumps(scrub_secrets_from_structure(ctx.context, result))
diff --git a/skyvern/forge/sdk/copilot/tools/banned_blocks.py b/skyvern/forge/sdk/copilot/tools/banned_blocks.py
index 45cc7d7ad..a154df682 100644
--- a/skyvern/forge/sdk/copilot/tools/banned_blocks.py
+++ b/skyvern/forge/sdk/copilot/tools/banned_blocks.py
@@ -24,6 +24,7 @@ from skyvern.forge.sdk.copilot.tracing_setup import copilot_span
from skyvern.forge.sdk.copilot.workflow_yaml import dump_workflow_yaml
from skyvern.forge.sdk.schemas.credentials import CredentialType, TotpType
from skyvern.utils.yaml_loader import safe_load_no_dates
+from skyvern.webeye.utils.captcha_solver import MAX_IMAGE_CAPTCHA_READS
from ._shared import _parse_workflow_blocks
@@ -349,7 +350,20 @@ def _secret_credential_guidance() -> str:
return f"A `secret` credential is read with {secret_value}; it carries no username or password."
-def _code_only_browser_schema_guidance(*, agent_blocks: bool = False) -> list[str]:
+_IMAGE_CAPTCHA_GUIDANCE = (
+ "For a distorted-text image CAPTCHA, the Code runtime provides await solve_captcha(page, image=<observed image "
+ "selector>, input=<observed answer field selector>): it reads the text in that <img>, <svg> or <canvas> (or a "
+ "container holding exactly one) and types it into the answer field. It requires image OCR enabled for the "
+ "organization; otherwise it raises. It raises with nothing typed when the image yields no text, and raises on "
+ f"any call after the block's {MAX_IMAGE_CAPTCHA_READS}th call. A typed answer is unconfirmed until the page "
+ f"accepts it. Author a bounded loop of at most {MAX_IMAGE_CAPTCHA_READS} tries before the final step: solve; if "
+ "the call raised, click the observed refresh control and retry; otherwise submit, and if the form is still shown, "
+ "click the refresh control and retry. With no refresh control, raise instead of retrying; raise after the last "
+ "try. An answer typed while scouting fits that one image only, so never persist it."
+)
+
+
+def _code_only_browser_schema_guidance(*, agent_blocks: bool = False, image_ocr: bool = False) -> list[str]:
"""The `code` schema response. Two entries answer "what else may this turn author", so they read
as a closed list and have to change when the agent family is authorable too."""
availability = (
@@ -372,6 +386,7 @@ def _code_only_browser_schema_guidance(*, agent_blocks: bool = False) -> list[st
_saved_credential_guidance(),
_secret_credential_guidance(),
"The Code runtime provides await solve_captcha(page) for a platform-managed verification challenge observed while scouting; this is an available capability, not a required step for every login.",
+ *([_IMAGE_CAPTCHA_GUIDANCE] if image_ocr else []),
"The Code runtime provides await clear_browser_data(page) when a site needs a clean session before it will sign in: it drops every cookie in the run's browser and all stored data for every origin it has a page or frame open on, and returns nothing. Read page.url first and navigate back to it afterwards. Browser settings pages (chrome://...) cannot be navigated to; this helper is the way to clear state. A workflow parameter named clear_browser_data shadows the helper in both executors. Before calling the helper in that case, rename the parameter to an unused name, preserve its value/default, and update its block bindings, code/template references, and caller-supplied run input keys.",
"For file attachment: bind the file as a workflow parameter with workflow_parameter_type file_url, then call await attach_authorized_file(page, <file_parameter>, <observed_selector>). The parameter is a handle, not a path: pass it only to that helper. Attaching puts the file's contents in the page, where page scripts and page.evaluate can read them, so attach it only to the page that should receive it. It accepts only that run's materialized file, uploads at most 10 MB, and returns filename and size. To upload a file this block downloads, claim it with async with page.expect_download() as info: and pass await info.value to the same helper, never its path.",
]
diff --git a/skyvern/forge/sdk/copilot/tools/browser_code.py b/skyvern/forge/sdk/copilot/tools/browser_code.py
index 01f25b614..5f9eb7fc5 100644
--- a/skyvern/forge/sdk/copilot/tools/browser_code.py
+++ b/skyvern/forge/sdk/copilot/tools/browser_code.py
@@ -119,8 +119,11 @@ Browser API (async, Playwright-shaped):
with the same `target`; every call starts on the tab that tool acts on.
- `await solve_captcha(page)` runs the platform CAPTCHA solver on the current `page`, as in a saved block,
and raises when the CAPTCHA stays unsolved. A solve can take up to about two minutes, longer than the
- 60-second default `timeout_seconds`, and a call that times out mid-solve stops the interpreter. The
- `image=`/`input=` form works only in a saved block.
+ 60-second default `timeout_seconds`, and a call that times out mid-solve stops the interpreter. For a
+ distorted-text image CAPTCHA, `await solve_captcha(page, image=<selector>, input=<selector>)` reads the
+ image with the same OCR a saved block uses and types the text into the answer field, raising with
+ nothing typed when no text is read. It requires image OCR enabled for the organization; otherwise it
+ raises.
- workbench-only, not valid in a saved block: `tabs`, `switch_tab`, `click_and_wait_for_popup`,
`click_and_download`, and `files`.
diff --git a/skyvern/forge/sdk/copilot/tools/mcp_hooks.py b/skyvern/forge/sdk/copilot/tools/mcp_hooks.py
index 9b3c4e6a7..c31515766 100644
--- a/skyvern/forge/sdk/copilot/tools/mcp_hooks.py
+++ b/skyvern/forge/sdk/copilot/tools/mcp_hooks.py
@@ -584,7 +584,10 @@ async def _get_block_schema_post_hook(
]
if not capability.agent_blocks:
data["code_only_note"] = _code_only_browser_unavailable_summary()
- data["code_only_guidance"] = _code_only_browser_schema_guidance(agent_blocks=capability.agent_blocks)
+ data["code_only_guidance"] = _code_only_browser_schema_guidance(
+ agent_blocks=capability.agent_blocks,
+ image_ocr=app.AGENT_FUNCTION.supports_image_captcha_ocr(),
+ )
data["download_claim_helper_contract"] = download_claim_helper_contract()
data["web_search_helper_contract"] = WEB_SEARCH_HELPER_CONTRACT
data["clear_browser_data_helper_contract"] = CLEAR_BROWSER_DATA_HELPER_CONTRACT
diff --git a/skyvern/forge/sdk/copilot/tools/page_challenge.py b/skyvern/forge/sdk/copilot/tools/page_challenge.py
index cac140d34..f47502641 100644
--- a/skyvern/forge/sdk/copilot/tools/page_challenge.py
+++ b/skyvern/forge/sdk/copilot/tools/page_challenge.py
@@ -26,7 +26,12 @@ from skyvern.forge.sdk.copilot.runtime import (
sensitive_origin_page_has_active_run,
sensitive_origin_page_is_tainted,
)
-from skyvern.webeye.utils.captcha_solver import CaptchaChallengeUnsolvedError, solve_challenge_ladder
+from skyvern.forge.sdk.workflow.models.block import CodeBlockCaptchaError, _code_block_solve_captcha_builtin
+from skyvern.webeye.utils.captcha_solver import (
+ MAX_IMAGE_CAPTCHA_READS,
+ CaptchaChallengeUnsolvedError,
+ solve_challenge_ladder,
+)
from .scouting import rendered_challenge_vendor
@@ -55,8 +60,30 @@ _OUTCOME_TEXT = {
"unavailable": "Challenge solving is not available for this organization or page, so nothing was attempted.",
}
+_IMAGE_OUTCOME_TEXT = {
+ "typed": (
+ "The image's text was read and typed into the answer field. It is unconfirmed until the page accepts it: "
+ "submit, then look at the page again. If the page rejects it, load a new image before reading again."
+ ),
+ "unsolved": (
+ "Nothing was typed: the image yielded no text, the image selector does not name one <img>, <svg> or "
+ "<canvas> (or a container holding exactly one), or the input selector matched no fillable field."
+ ),
+ "unavailable": "Image OCR is not enabled for this organization, so the image was not read.",
+ "read_limit_reached": (
+ f"This request has already made {MAX_IMAGE_CAPTCHA_READS} image read attempts, so this one was not read."
+ ),
+}
+_IMAGE_FORM_ARGUMENTS = (
+ "Pass both image and input as non-empty selectors for an image CAPTCHA, or neither for a widget."
+)
+
-async def solve_page_challenge(ctx: AgentContext) -> dict[str, Any]:
+async def solve_page_challenge(
+ ctx: AgentContext, *, image: str | None = None, input: str | None = None
+) -> dict[str, Any]:
+ if (image is None) != (input is None) or any(value is not None and not value.strip() for value in (image, input)):
+ return {"ok": False, "error": _IMAGE_FORM_ARGUMENTS}
session_id = ctx.browser_session_id
if not session_id:
return {"ok": False, "error": _NO_BROWSER}
@@ -70,7 +97,10 @@ async def solve_page_challenge(ctx: AgentContext) -> dict[str, Any]:
page = await live_working_page(ctx)
if page is None:
return {"ok": False, "error": _NO_PAGE}
- result = await _run_ladder(ctx, page, session_id)
+ if image is not None and input is not None:
+ result = await _read_image(ctx, page, session_id, image, input)
+ else:
+ result = await _run_ladder(ctx, page, session_id)
if result["outcome"] == "unsolved" and (
page.is_closed()
or _browser_context_attachability(page.context) is BrowserProbeOutcome.positively_unreachable
@@ -83,6 +113,9 @@ async def solve_page_challenge(ctx: AgentContext) -> dict[str, Any]:
return _browser_session_loss_result(
{}, disposition=disposition, deadline_expired=ctx.browser_session_continuity_deadline_expired
)
+ # A fresh browser changes whether a site shows a challenge, not whether its image can be read.
+ if image is not None:
+ return result
if result["outcome"] == "unsolved":
ctx.unsolved_page_challenges_by_session_id[session_id] = (
ctx.unsolved_page_challenges_by_session_id.get(session_id, 0) + 1
@@ -130,6 +163,29 @@ async def _run_ladder(ctx: AgentContext, page: Page, session_id: str) -> dict[st
return {"ok": True, "outcome": "none", "detail": _OUTCOME_TEXT["none"]}
+async def _read_image(ctx: AgentContext, page: Page, session_id: str, image: str, input: str) -> dict[str, Any]:
+ if not await app.AGENT_FUNCTION.image_captcha_ocr_enabled(organization_id=ctx.organization_id, url=page.url):
+ return {"ok": True, "outcome": "unavailable", "detail": _IMAGE_OUTCOME_TEXT["unavailable"]}
+ unsolved: dict[str, Any] = {"ok": True, "outcome": "unsolved", "detail": _IMAGE_OUTCOME_TEXT["unsolved"]}
+ # Each read ships a screenshot to a paid OCR vendor, capped per request like a saved block's reads.
+ if ctx.image_captcha_reads >= MAX_IMAGE_CAPTCHA_READS:
+ return {**unsolved, "read_limit_reached": True, "detail": _IMAGE_OUTCOME_TEXT["read_limit_reached"]}
+ ctx.image_captcha_reads += 1
+ try:
+ async with asyncio.timeout(SOLVE_CEILING_SECONDS):
+ await _code_block_solve_captcha_builtin(
+ page, organization_id=ctx.organization_id, browser_session_id=session_id, image=image, input=input
+ )
+ except CodeBlockCaptchaError:
+ return unsolved
+ except TimeoutError:
+ return {**unsolved, "timed_out": True, "detail": f"The image was not read in {SOLVE_CEILING_SECONDS}s."}
+ except Exception:
+ LOG.warning("copilot solve_page_challenge image read failed", exc_info=True)
+ return {**unsolved, "solver_failed": True, "detail": "Reading the image failed with an internal error."}
+ return {"ok": True, "outcome": "typed", "detail": _IMAGE_OUTCOME_TEXT["typed"]}
+
+
async def start_fresh_browser(ctx: AgentContext) -> dict[str, Any]:
if sensitive_origin_page_has_active_run(ctx):
return {"ok": False, "error": SENSITIVE_ORIGIN_ACTIVE_RUN_PAGE_ERROR}
diff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.py
index aeabea84f..93a015de8 100644
--- a/skyvern/forge/sdk/copilot/tools/run_execution.py
+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py
@@ -44,6 +44,7 @@ from skyvern.forge.sdk.copilot.build_test_connect_failure import (
)
from skyvern.forge.sdk.copilot.build_test_outcome import (
ACTION_OBSERVATIONS_EMPTY,
+ ACTION_TRACE_PER_TASK_LIMIT,
INFRASTRUCTURE_RUNNER_ERROR_CODES,
OBSERVED_BLOCK_END_URLS_UNREPORTABLE,
OBSERVED_BLOCK_END_URLS_WITHHELD,
@@ -62,6 +63,7 @@ from skyvern.forge.sdk.copilot.build_test_outcome import (
BuildTestPacketScreenshot,
BuildTestPacketUnfinishedItem,
RecordedBuildTestOutcome,
+ SolverAttempt,
append_omission_notice,
authored_block_parameter_keys_from_workflow,
authored_structure_signature_from_workflow,
@@ -111,6 +113,7 @@ from skyvern.forge.sdk.copilot.context import (
from skyvern.forge.sdk.copilot.diagnosis_repair_contract import (
DiagnosisRepairContract,
build_diagnosis_repair_contract,
+ solver_facts_from_traces,
)
from skyvern.forge.sdk.copilot.enforcement import (
proxy_hop_failure_reason,
@@ -583,6 +586,7 @@ async def _attach_action_traces(
rows = await app.DATABASE.tasks.get_recent_actions_for_tasks(
task_ids=task_ids,
organization_id=organization_id,
+ per_task_limit=ACTION_TRACE_PER_TASK_LIMIT,
)
except Exception:
if not include_completed:
@@ -975,7 +979,7 @@ def _summarize_action_trace(action_trace: list[dict[str, Any]] | None) -> list[s
return summary
-def _capture_solver_facts_and_strip_traces(results: list[dict[str, Any]]) -> dict[str, Any]:
+def _capture_solver_facts_and_strip_traces(results: list[dict[str, Any]]) -> SolverAttempt:
"""Read the solver rows, then drop per-block action_trace from the compact packet.
The order is the point and is why these two steps share a function: after the pop nothing
@@ -983,7 +987,10 @@ def _capture_solver_facts_and_strip_traces(results: list[dict[str, Any]]) -> dic
placed after it would silently report every run as unresolved. ``get_run_results`` remains the
heavier inspection path for the traces themselves.
"""
- solver_attempt = _solve_captcha_attempt(results)
+ solver_attempt: SolverAttempt = {
+ **solver_facts_from_traces(results),
+ "code_block": solver_facts_from_traces(results, code_block_only=True),
+ }
for entry in results:
entry.pop("action_trace", None)
return solver_attempt
@@ -996,44 +1003,6 @@ def _result_current_url(result: Mapping[str, Any] | None) -> str | None:
return str(url) if isinstance(url, str) and url.strip() else None
-def _solve_captcha_attempt(results: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
- """What the managed solver did on this run, read from the full traces before they are stripped."""
- attempted = False
- failed = False
- not_solved = False
- saw_history = False
- failure: str | None = None
- for block_result in results:
- trace = block_result.get("action_trace")
- if not isinstance(trace, list):
- continue
- saw_history = True
- for entry in trace:
- if not isinstance(entry, dict) or entry.get("action") != ActionType.SOLVE_CAPTCHA.value:
- continue
- attempted = True
- if entry.get("status") == ActionStatus.failed.value:
- failed = True
- if failure is None:
- failure = str(entry.get("response") or "").strip() or None
- elif entry.get("solver_cleared") is False:
- not_solved = True
- # A completed row carrying no boolean is not a cleared challenge: the terminal no-solver fallback
- # returns ActionSuccess (cloud/actions.py), so success there means the step ran, nothing more. And
- # an absent history is not a non-attempt: the optional action lookup swallows its failures.
- if failed:
- result = "failed"
- elif not_solved:
- result = "not_solved"
- elif attempted:
- result = "attempted"
- elif saw_history:
- result = "not_attempted"
- else:
- result = "unresolved"
- return {"attempted": attempted, "result": result, "failure": failure}
-
-
def _action_observation(entry: Mapping[str, Any]) -> str | None:
raw_action = entry.get("action")
raw_status = entry.get("status")
diff --git a/tests/unit/force_stub_app.py b/tests/unit/force_stub_app.py
index 74fffe21d..5df81abcc 100644
--- a/tests/unit/force_stub_app.py
+++ b/tests/unit/force_stub_app.py
@@ -93,6 +93,7 @@ def create_forge_stub_app() -> ForgeApp:
# Sync method returning a key or None — _LazyNamespace would auto-mock it as a truthy
# AsyncMock and hijack the TextPromptBlock llm_key. Match the OSS no-op.
fake_app_module.AGENT_FUNCTION.get_fallback_llm_key = MagicMock(return_value=None)
+ fake_app_module.AGENT_FUNCTION.supports_image_captcha_ocr = base_agent_function.supports_image_captcha_ocr
# Credential write-lock gating — _LazyNamespace would auto-mock these as truthy AsyncMocks,
# forcing the update/delete credential routes down the lock path and handing `async with` a
# coroutine instead of a context manager. Match the real OSS base no-ops (unlocked path).
diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.py
index 471651f8c..1f1ee7fa5 100644
--- a/tests/unit/test_copilot_agent_helpers.py
+++ b/tests/unit/test_copilot_agent_helpers.py
@@ -70,6 +70,7 @@ from skyvern.forge.sdk.copilot.blocker_signal import (
CopilotToolBlockerSignal,
)
from skyvern.forge.sdk.copilot.build_test_outcome import (
+ ACTION_TRACE_PER_TASK_LIMIT,
BuildTestConnectFailure,
ChallengeEffects,
Lever,
@@ -103,6 +104,7 @@ from skyvern.forge.sdk.copilot.diagnosis_repair_contract import (
RepairDecision,
RepairNextAction,
VerificationResult,
+ solver_facts_from_traces,
)
from skyvern.forge.sdk.copilot.enforcement import (
NUDGE_SENTINEL,
@@ -3427,7 +3429,9 @@ workflow_definition:
await run_execution_module._attach_action_traces([block], [result], "org-1", include_completed=True)
assert run_execution_module._retained_action_observations([result]) == ["click completed"]
- get_actions.assert_awaited_once_with(task_ids=["task-completed"], organization_id="org-1")
+ get_actions.assert_awaited_once_with(
+ task_ids=["task-completed"], organization_id="org-1", per_task_limit=ACTION_TRACE_PER_TASK_LIMIT
+ )
@pytest.mark.asyncio
@pytest.mark.parametrize(
@@ -3477,7 +3481,7 @@ workflow_definition:
await run_execution_module._attach_action_traces([block], [block_result], "org-1", include_completed=True)
assert block_result["action_trace"][0].get("solver_cleared") is expected_cleared
- assert run_execution_module._solve_captcha_attempt([block_result])["result"] == expected_result
+ assert solver_facts_from_traces([block_result])["result"] == expected_result
@pytest.mark.asyncio
async def test_a_non_solver_row_response_never_reaches_the_trace_as_a_solver_boolean(
@@ -3617,7 +3621,9 @@ workflow_definition:
}
assert packet["action_observations"] == []
assert any("action_observations empty" in notice for notice in packet["omission_notices"])
- get_actions.assert_awaited_once_with(task_ids=["task-completed"], organization_id="org-1")
+ get_actions.assert_awaited_once_with(
+ task_ids=["task-completed"], organization_id="org-1", per_task_limit=ACTION_TRACE_PER_TASK_LIMIT
+ )
class TestTranslateToAgentResultGating:
diff --git a/tests/unit/test_copilot_banned_block_policy.py b/tests/unit/test_copilot_banned_block_policy.py
index d3c9f8a9b..532ec6cba 100644
--- a/tests/unit/test_copilot_banned_block_policy.py
+++ b/tests/unit/test_copilot_banned_block_policy.py
@@ -26,6 +26,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
import yaml
+from skyvern.forge import app
from skyvern.forge.sdk.copilot.config import (
AGENT_BLOCKS_ONLY,
ALL_BLOCK_FAMILIES,
@@ -49,6 +50,7 @@ from skyvern.forge.sdk.copilot.tools import (
from skyvern.forge.sdk.copilot.tools.banned_blocks import (
_AGENT_FAMILY_BLOCK_TYPES,
_CODE_BLOCKS_ONLY_BANNED_BLOCK_TYPES,
+ _IMAGE_CAPTCHA_GUIDANCE,
AUTHORING_FAMILY_GUIDANCE,
CODE_BLOCK_SUMMARY,
CREDENTIAL_CODE_ACCESSORS,
@@ -632,6 +634,18 @@ async def test_code_schema_guidance_is_policy_rendered_and_allows_helper_validat
assert "Do not persist navigation/action/login" not in " ".join(out["data"]["code_only_guidance"])
[email protected]("image_ocr", [True, False])
+async def test_code_schema_guidance_advertises_the_image_captcha_form_only_with_ocr(
+ code_only_ctx: MagicMock, monkeypatch: pytest.MonkeyPatch, image_ocr: bool
+) -> None:
+ monkeypatch.setattr(app.AGENT_FUNCTION, "supports_image_captcha_ocr", lambda: image_ocr)
+
+ out = await _get_block_schema_post_hook({"ok": True, "data": {"block_type": "code"}}, raw={}, ctx=code_only_ctx)
+
+ assert (_IMAGE_CAPTCHA_GUIDANCE in out["data"]["code_only_guidance"]) is image_ocr
+
+
def test_code_schema_guidance_advertises_only_the_authorized_file_attachment_helper() -> None:
guidance = " ".join(_code_only_browser_schema_guidance())
diff --git a/tests/unit/test_copilot_code_block_synthesis.py b/tests/unit/test_copilot_code_block_synthesis.py
index f10a04fc3..664de21f7 100644
--- a/tests/unit/test_copilot_code_block_synthesis.py
+++ b/tests/unit/test_copilot_code_block_synthesis.py
@@ -74,6 +74,7 @@ from skyvern.forge.sdk.copilot.tools.scouting import _fill_carry_to_interaction
from skyvern.forge.sdk.copilot.tools.workflow_update import (
_code_block_safety_errors,
)
+from skyvern.forge.sdk.copilot.workflow_yaml import _process_workflow_yaml
from skyvern.forge.sdk.workflow.models.block import CodeBlock, CodeBlockStep
from tests.unit.copilot_test_helpers import carried_interaction
@@ -4032,6 +4033,7 @@ def test_login_submit_emits_solve_captcha_after_navigation_commit() -> None:
captcha_position = result.code.index("await solve_captcha(page)", navigation_position)
assert submit_position < navigation_position < captcha_position
assert result.code.count("await solve_captcha(page)") == 1
+ assert "image=" not in result.code
def test_non_login_trajectory_does_not_emit_solve_captcha() -> None:
@@ -4054,6 +4056,63 @@ def test_unstamped_click_emits_no_solve_captcha() -> None:
assert "solve_captcha" not in result.code
+def test_image_captcha_trajectory_never_synthesizes_the_image_form() -> None:
+ form = "http://localhost:8888/image_captcha_form.html"
+ trajectory = [
+ _interaction("type_text", selector="#firstName", source_url=form, typed_value="Ada"),
+ _interaction("click", selector="#captchaImage", source_url=form),
+ _interaction("type_text", selector="#captchaAnswer", source_url=form, typed_value="K7QPX"),
+ _interaction("click", selector="button[type=submit]", source_url=form),
+ ]
+
+ result = synthesize_code_block(trajectory, strict_selectors=True)
+
+ assert result is not None
+ assert "solve_captcha" not in result.code
+ assert "image=" not in result.code
+
+
[email protected]("no_saved_workflow")
+async def test_authored_image_captcha_loop_is_saved_with_its_image_call() -> None:
+ image_call = 'await solve_captcha(page, image="#captchaImage", input="#captchaAnswer")'
+ code = (
+ "for attempt in range(8):\n"
+ " try:\n"
+ f" {image_call}\n"
+ " except Exception:\n"
+ ' await page.locator("#refreshCaptcha").click()\n'
+ " continue\n"
+ ' await page.locator("button[type=submit]").click()\n'
+ ' if await page.locator("#captchaAnswer").count() == 0:\n'
+ " break\n"
+ ' await page.locator("#refreshCaptcha").click()\n'
+ "else:\n"
+ ' raise Exception("captcha not accepted")\n'
+ )
+ workflow_yaml = (
+ "title: Image captcha form\n"
+ "workflow_definition:\n"
+ " parameters: []\n"
+ " blocks:\n"
+ " - block_type: code\n"
+ " label: submit_request\n"
+ " code: |\n" + textwrap.indent(code, " ")
+ )
+
+ assert _code_block_safety_errors(workflow_yaml, None) == []
+ workflow = await _process_workflow_yaml(
+ workflow_id="w_1",
+ workflow_permanent_id="wpid_1",
+ organization_id="o_1",
+ workflow_yaml=workflow_yaml,
+ settings_fallback_yaml="enable_self_healing: false",
+ )
+ saved_code = workflow.workflow_definition.blocks[0].code
+ assert image_call in saved_code
+ assert "solve_captcha(page)" not in saved_code
+
+
class TestScoutedReadSynthesis:
"""A value the scout proved with `evaluate` is authored from that proven expression, the way a
login is authored from proven fills — instead of being rediscovered by a guessed selector."""
diff --git a/tests/unit/test_copilot_diagnosis_repair_contract.py b/tests/unit/test_copilot_diagnosis_repair_contract.py
index 6ac05068f..506928c84 100644
--- a/tests/unit/test_copilot_diagnosis_repair_contract.py
+++ b/tests/unit/test_copilot_diagnosis_repair_contract.py
@@ -23,6 +23,7 @@ from skyvern.forge.sdk.copilot.agent import (
_prior_run_debug_text,
)
from skyvern.forge.sdk.copilot.build_test_outcome import (
+ ACTION_TRACE_PER_TASK_LIMIT,
SOLVER_ATTEMPT_KEY,
BuildTestEvidencePacket,
ChallengeEffects,
@@ -31,6 +32,7 @@ from skyvern.forge.sdk.copilot.build_test_outcome import (
recorded_outcome_from_run_blocks_result,
)
from skyvern.forge.sdk.copilot.challenge_evidence import (
+ _RECORDABLE_CHALLENGE_FRAME_HOSTS,
CHALLENGE_KIND_KEY,
ChallengeKind,
composition_challenge_carrier,
@@ -60,7 +62,7 @@ from skyvern.forge.sdk.copilot.diagnosis_repair_contract import (
author_time_levers,
build_diagnosis_repair_contract,
)
-from skyvern.forge.sdk.copilot.enforcement import latest_diagnosis_contract_satisfies_goal
+from skyvern.forge.sdk.copilot.enforcement import _PACKET_NOTICE_CAP, latest_diagnosis_contract_satisfies_goal
from skyvern.forge.sdk.copilot.output_utils import BUILD_TEST_PACKET_KEY, project_direct_test_handoff_packet_for_llm
from skyvern.forge.sdk.copilot.request_policy import RequestPolicy
from skyvern.forge.sdk.copilot.run_outcome import (
@@ -5123,8 +5125,6 @@ def test_finalized_packet_carries_the_contract_challenge_and_levers() -> None:
def test_the_production_capture_reads_the_traces_it_then_strips() -> None:
"""Drives the production unit: reordering the capture after the pop fails this."""
- from skyvern.forge.sdk.copilot.tools import run_execution
-
results = [
{
"label": "search",
@@ -5134,28 +5134,25 @@ def test_the_production_capture_reads_the_traces_it_then_strips() -> None:
}
]
- captured = run_execution._capture_solver_facts_and_strip_traces(results)
+ captured = run_execution_module._capture_solver_facts_and_strip_traces(results)
- assert captured == {"attempted": True, "result": "failed", "failure": "not solved in time"}
+ facts = {"attempted": True, "result": "failed", "failure": "not solved in time"}
+ assert captured == {**facts, "code_block": facts}
assert "action_trace" not in results[0]
@pytest.mark.asyncio
async def test_a_failed_availability_lookup_leaves_the_solver_unresolved(monkeypatch: pytest.MonkeyPatch) -> None:
"""Restoring `available = False` on the exception path must fail this."""
- from unittest.mock import AsyncMock
-
- from skyvern.forge.sdk.copilot.tools import run_execution
-
ctx = _ctx()
ctx.composition_page_evidence = {"current_url": "https://records.example.com/search"}
monkeypatch.setattr(
- run_execution.app.AGENT_FUNCTION,
+ run_execution_module.app.AGENT_FUNCTION,
"captcha_solving_available",
AsyncMock(side_effect=RuntimeError("provider down")),
)
- await run_execution._resolve_captcha_solver_availability(ctx)
+ await run_execution_module._resolve_captcha_solver_availability(ctx)
assert ctx.captcha_solver_available is None
@@ -5210,7 +5207,7 @@ def test_solver_facts_survive_the_packet_dropping_per_block_action_traces() -> N
result["data"][SOLVER_ATTEMPT_KEY] = {
"attempted": True,
"result": "failed",
- "failure": "Captcha not solved in time",
+ "failure": "Captcha not solved in time at https://captcha.example.test/solve?token=abc " + "x" * 400,
}
contract = build_diagnosis_repair_contract(
@@ -5223,7 +5220,10 @@ def test_solver_facts_survive_the_packet_dropping_per_block_action_traces() -> N
assert contract.challenge is not None
assert contract.challenge.solver_attempted is True
assert contract.challenge.solver_result == "failed"
- assert contract.challenge.solver_failure == "Captcha not solved in time"
+ failure = contract.challenge.solver_failure or ""
+ assert failure.startswith("Captcha not solved in time at https://captcha.example.test")
+ assert "token=abc" not in failure
+ assert len(failure) <= 200
def _solver_contract(ctx: CopilotContext, result: dict[str, Any]) -> DiagnosisRepairContract:
@@ -5413,10 +5413,213 @@ def test_a_not_solved_solver_with_no_challenge_evidence_stays_unwalled() -> None
contract = _solver_contract(ctx, result)
+ assert contract.challenge is not None
+ assert contract.challenge.basis == "solver_call"
+ assert contract.levers == []
+
+
[email protected]("shape", ["failed", "completed-bare", "completed-false"])
+def test_a_failed_unwalled_captcha_call_is_reported_as_an_attempt_without_levers(shape: str) -> None:
+ """An image-text CAPTCHA form mounts no vendor frame, so without this record a failed or typed-but-rejected
+ solve never reaches the model at all."""
+ ctx = _ctx()
+ result = _unwalled_run_result()
+ result["data"]["blocks"] = [_traced_block(shape)]
+
+ finalize_build_test_result(ctx, source_tool="update_and_run_blocks", result=result, workflow_updated=True)
+
+ contract = ctx.latest_diagnosis_repair_contract
+ assert contract is not None and contract.challenge is not None
+ assert contract.challenge.basis == "solver_call"
+ assert contract.challenge.solver_attempted is True
+ assert contract.challenge.solver_available is None
+ assert contract.levers == []
+ packet = result["data"][BUILD_TEST_PACKET_KEY]
+ assert not packet.get("levers")
+ notice = " ".join(packet["challenge_notices"]).replace(contract.challenge.solver_failure or "", "")
+ assert "solved" not in notice.lower()
+ assert "cleared" not in notice.lower()
+
+
+ ("run_ok", "shape"),
+ [(True, "failed"), (True, "completed-bare"), (False, "no-solve-row"), (False, "no-history")],
+)
+def test_a_passing_run_or_a_run_without_a_captcha_call_gets_no_solver_record(run_ok: bool, shape: str) -> None:
+ result = _unwalled_run_result()
+ result["ok"] = run_ok
+ result["data"]["blocks"] = [_traced_block(shape)]
+
+ contract = _solver_contract(_ctx(), result)
+
assert contract.challenge is None
assert contract.levers == []
+def test_a_suspicious_success_with_a_captcha_call_gets_a_solver_record() -> None:
+ ctx = _ctx()
+ ctx.last_test_suspicious_success = True
+ result = _unwalled_run_result()
+ result["ok"] = True
+ result["data"]["blocks"] = [_traced_block("completed-bare")]
+
+ contract = _solver_contract(ctx, result)
+
+ assert contract.challenge is not None
+ assert contract.challenge.basis == "solver_call"
+ assert contract.levers == []
+
+
[email protected]("source", ["trace", "carried"])
+ ("block_type", "expected_basis"), [("CODE", "solver_call"), ("NAVIGATION", None), ("TASK", None)]
+)
+def test_only_a_code_block_captcha_row_gets_a_solver_record(
+ source: str, block_type: str, expected_basis: str | None
+) -> None:
+ result = _unwalled_run_result()
+ block = _traced_block("failed")
+ block["block_type"] = block_type
+ for entry in block["action_trace"]:
+ entry.pop("code_line")
+ result["data"]["blocks"] = [block]
+ if source == "carried":
+ result["data"][SOLVER_ATTEMPT_KEY] = run_execution_module._capture_solver_facts_and_strip_traces([block])
+
+ contract = _solver_contract(_ctx(), result)
+
+ assert (contract.challenge.basis if contract.challenge else None) == expected_basis
+ assert contract.levers == []
+
+
+def _recovered_raise_then_unrelated_failure() -> list[dict[str, Any]]:
+ return [
+ {
+ "label": "solve",
+ "block_type": "CODE",
+ "status": "completed",
+ "action_trace": [
+ {"action": "solve_captcha", "status": "completed"},
+ {"action": "click", "status": "completed"},
+ {"action": "solve_captcha", "status": "failed", "code_line": 3, "response": "no text in image"},
+ ],
+ },
+ {
+ "label": "submit",
+ "block_type": "CODE",
+ "status": "failed",
+ "action_trace": [{"action": "click", "status": "failed", "code_line": 2, "response": "Timeout"}],
+ },
+ ]
+
+
+def _code_solve_then_navigation_raise() -> list[dict[str, Any]]:
+ return [
+ {
+ "label": "solve",
+ "block_type": "CODE",
+ "status": "completed",
+ "action_trace": [{"action": "solve_captcha", "status": "completed"}],
+ },
+ {
+ "label": "next",
+ "block_type": "NAVIGATION",
+ "status": "failed",
+ "action_trace": [{"action": "solve_captcha", "status": "failed", "response": "Captcha not solved in time"}],
+ },
+ ]
+
+
[email protected]("source", ["trace", "carried"])
+ "build_blocks",
+ [_recovered_raise_then_unrelated_failure, _code_solve_then_navigation_raise],
+ ids=["recovered_raise", "navigation_raise"],
+)
+def test_the_newest_code_block_solver_row_decides_a_solver_call_record(
+ source: str, build_blocks: Callable[[], list[dict[str, Any]]]
+) -> None:
+ """Traces are newest-first: a raise the retry loop recovered from, or one outside the code blocks, is not
+ the code's failed call."""
+ blocks = build_blocks()
+ result = _unwalled_run_result()
+ result["data"]["blocks"] = blocks
+ if source == "carried":
+ result["data"][SOLVER_ATTEMPT_KEY] = run_execution_module._capture_solver_facts_and_strip_traces(blocks)
+
+ contract = _solver_contract(_ctx(), result)
+
+ assert contract.challenge is not None
+ assert contract.challenge.basis == "solver_call"
+ assert contract.challenge.solver_result == "attempted"
+
+
[email protected]("source", ["trace", "carried"])
+ ("later_trace_length", "expected_basis"),
+ [(ACTION_TRACE_PER_TASK_LIMIT, None), (ACTION_TRACE_PER_TASK_LIMIT - 1, "solver_call")],
+)
+def test_a_later_code_block_with_a_full_trace_does_not_inherit_an_earlier_solver_row(
+ source: str, later_trace_length: int, expected_basis: str | None
+) -> None:
+ """A trace at the per-task cap may have dropped the later block's own call, so the earlier failure is stale."""
+ blocks = [
+ {
+ "label": "solve",
+ "block_type": "CODE",
+ "status": "completed",
+ "action_trace": [{"action": "solve_captcha", "status": "failed", "response": "no text in image"}],
+ },
+ {
+ "label": "retry",
+ "block_type": "CODE",
+ "status": "failed",
+ "action_trace": [{"action": "click", "status": "completed"}] * later_trace_length,
+ },
+ ]
+ result = _unwalled_run_result()
+ result["data"]["blocks"] = blocks
+ if source == "carried":
+ result["data"][SOLVER_ATTEMPT_KEY] = run_execution_module._capture_solver_facts_and_strip_traces(blocks)
+
+ contract = _solver_contract(_ctx(), result)
+
+ assert (contract.challenge.basis if contract.challenge else None) == expected_basis
+ assert contract.levers == []
+
+
[email protected]("solver_result", ["failed", "not_solved", "attempted"])
+def test_a_solver_call_notice_with_every_frame_host_fits_the_retained_notice_cap(solver_result: str) -> None:
+ """The retained packet cuts each notice at the cap, which would drop the frame hosts at its tail."""
+ challenge = ChallengeEffects(
+ basis="solver_call",
+ solver_attempted=True,
+ solver_result=solver_result,
+ solver_failure="x" * 200,
+ frame_hosts=sorted(_RECORDABLE_CHALLENGE_FRAME_HOSTS),
+ )
+
+ (notice,) = challenge_notices(challenge, [])
+
+ assert len(notice) <= _PACKET_NOTICE_CAP
+
+
+def test_a_solver_call_notice_names_the_frame_hosts_and_never_claims_a_solve() -> None:
+ ctx, result = _not_solved_result_and_ctx(
+ _same_run_challenge_evidence(frame_urls=["https://challenges.cloudflare.com/turnstile/v0/api.html"])
+ )
+ result["data"]["blocks"] = [_traced_block("failed")]
+
+ contract = _solver_contract(ctx, result)
+
+ assert contract.challenge is not None and contract.challenge.basis == "solver_call"
+ notice = " ".join(challenge_notices(contract.challenge, contract.levers))
+ assert contract.challenge.frame_hosts
+ assert all(host in notice for host in contract.challenge.frame_hosts)
+ stripped = notice.replace(contract.challenge.solver_failure or "", "").lower()
+ assert "solved" not in stripped and "cleared" not in stripped
+
+
@pytest.mark.parametrize(
"evidence_kwargs",
[
@@ -5431,26 +5634,37 @@ def test_a_not_solved_solver_does_not_wall_on_another_runs_challenge_evidence(
contract = _solver_contract(ctx, result)
- assert contract.challenge is None
+ assert contract.challenge is not None
+ assert contract.challenge.basis == "solver_call"
assert contract.levers == []
[email protected]("carried", ["failed", "attempted", "not_attempted", "unresolved"])
+_KIND_EVIDENCE: dict[str, Any] = {"challenge_kind": "captcha"}
+_FRAME_EVIDENCE: dict[str, Any] = {"frame_urls": ["https://challenges.cloudflare.com/turnstile/v0/api.html"]}
+
+
@pytest.mark.parametrize(
- ("evidence_kwargs", "expected_basis"),
+ ("carried", "evidence_kwargs", "expected_basis"),
[
- ({"challenge_kind": "captcha"}, None),
- ({"frame_urls": ["https://challenges.cloudflare.com/turnstile/v0/api.html"]}, "page_frames"),
+ ("failed", _KIND_EVIDENCE, "solver_call"),
+ ("failed", _FRAME_EVIDENCE, "solver_call"),
+ ("attempted", _KIND_EVIDENCE, "solver_call"),
+ ("attempted", _FRAME_EVIDENCE, "solver_call"),
+ ("not_attempted", _KIND_EVIDENCE, None),
+ ("not_attempted", _FRAME_EVIDENCE, "page_frames"),
+ ("unresolved", _KIND_EVIDENCE, None),
+ ("unresolved", _FRAME_EVIDENCE, "page_frames"),
],
)
def test_only_not_solved_widens_the_wall(
carried: str, evidence_kwargs: dict[str, Any], expected_basis: str | None
) -> None:
- """``failed`` is another ticket's recourse and the rest never observed a refusal, so the same
- challenge evidence must not mint a wall for them."""
+ """The same challenge evidence mints a wall only next to the solver's own refusal: a code block's
+ raised or returned call is a solver_call record, and a run with no call keeps only its page frames."""
ctx, result = _not_solved_result_and_ctx(_same_run_challenge_evidence(**evidence_kwargs))
result["data"]["blocks"][0].pop("action_trace")
- result["data"][SOLVER_ATTEMPT_KEY] = {"attempted": True, "result": carried, "failure": None}
+ facts = {"attempted": True, "result": carried, "failure": None}
+ result["data"][SOLVER_ATTEMPT_KEY] = {**facts, "code_block": facts}
contract = _solver_contract(ctx, result)
diff --git a/tests/unit/test_copilot_page_challenge.py b/tests/unit/test_copilot_page_challenge.py
index ac3a8d452..ea931a2af 100644
--- a/tests/unit/test_copilot_page_challenge.py
+++ b/tests/unit/test_copilot_page_challenge.py
@@ -7,7 +7,10 @@ from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
+from playwright.async_api import Page, async_playwright
+from skyvern.forge import app
+from skyvern.forge.agent_functions import AgentFunction
from skyvern.forge.sdk.copilot.runtime import (
SENSITIVE_ORIGIN_ACTIVE_RUN_PAGE_ERROR,
SENSITIVE_ORIGIN_PAGE_ERROR,
@@ -15,7 +18,9 @@ from skyvern.forge.sdk.copilot.runtime import (
CopilotBrowserSessionUnavailable,
)
from skyvern.forge.sdk.copilot.tools import page_challenge
-from skyvern.webeye.utils.captcha_solver import CaptchaChallengeUnsolvedError
+from skyvern.webeye.utils.captcha_solver import MAX_IMAGE_CAPTCHA_READS, CaptchaChallengeUnsolvedError
+from tests.unit.conftest import OcrRecordingAgentFunction
+from tests.unit.test_code_block_captcha import _skip_no_browser
from tests.unit.test_copilot_hooks import _ListenerPage
from tests.unit.test_copilot_runtime import _FakeBrowserContext, _make_ctx
@@ -34,14 +39,24 @@ class _Page(_ListenerPage):
return self.closed
-def _chat(monkeypatch: pytest.MonkeyPatch, page: _Page, ladder: AsyncMock, *, available: bool = True) -> AgentContext:
+def _chat(
+ monkeypatch: pytest.MonkeyPatch,
+ page: _Page | Page,
+ ladder: AsyncMock,
+ *,
+ available: bool = True,
+ agent_function: AgentFunction | None = None,
+) -> AgentContext:
@asynccontextmanager
async def _admitted(_ctx: AgentContext) -> AsyncIterator[None]:
yield
- agent_function = MagicMock()
- agent_function.captcha_solving_available = AsyncMock(return_value=available)
- monkeypatch.setattr(page_challenge, "app", MagicMock(AGENT_FUNCTION=agent_function))
+ if agent_function is None:
+ stub = MagicMock()
+ stub.captcha_solving_available = AsyncMock(return_value=available)
+ monkeypatch.setattr(page_challenge, "app", MagicMock(AGENT_FUNCTION=stub))
+ else:
+ monkeypatch.setattr(app, "AGENT_FUNCTION", agent_function)
monkeypatch.setattr(page_challenge, "mcp_browser_context", _admitted)
monkeypatch.setattr(page_challenge, "live_working_page", AsyncMock(return_value=page))
monkeypatch.setattr(page_challenge, "solve_challenge_ladder", ladder)
@@ -161,3 +176,49 @@ async def test_a_sensitive_page_is_never_sent_to_the_solver(
assert result == {"ok": False, "error": expected_error}
ladder.assert_not_awaited()
+
+
+_IMAGE_FORM = '<svg id="captcha" width="120" height="40"><text x="10" y="28">K7QPX</text></svg><input id="answer">'
+
+
+@_skip_no_browser
+ ("ocr_text", "enabled", "reads_before", "expected", "typed", "reads"),
+ [
+ ("K7QPX", True, 0, {"outcome": "typed"}, "K7QPX", 1),
+ (None, True, 0, {"outcome": "unsolved"}, "", 1),
+ ("K7QPX", False, 0, {"outcome": "unavailable"}, "", 0),
+ ("K7QPX", True, MAX_IMAGE_CAPTCHA_READS, {"outcome": "unsolved", "read_limit_reached": True}, "", 0),
+ ],
+ ids=["typed", "unreadable", "ocr_off", "read_limit"],
+)
+async def test_the_image_form_types_what_the_saved_block_ocr_reads_into_the_named_field(
+ monkeypatch: pytest.MonkeyPatch,
+ ocr_text: str | None,
+ enabled: bool,
+ reads_before: int,
+ expected: dict[str, Any],
+ typed: str,
+ reads: int,
+) -> None:
+ agent_function = OcrRecordingAgentFunction(ocr_text, enabled=enabled)
+ ladder = AsyncMock()
+ async with async_playwright() as playwright:
+ browser = await playwright.chromium.launch()
+ try:
+ page = await browser.new_page()
+ await page.set_content(_IMAGE_FORM)
+ ctx = _chat(monkeypatch, page, ladder, agent_function=agent_function)
+ ctx.image_captcha_reads = reads_before
+
+ result = await page_challenge.solve_page_challenge(ctx, image="#captcha", input="#answer")
+
+ assert await page.locator("#answer").input_value() == typed
+ finally:
+ await browser.close()
+
+ assert {key: result.get(key) for key in expected} == expected
+ assert len(agent_function.images) == reads
+ assert "untried_in_this_request" not in result
+ ladder.assert_not_awaited()
diff --git a/tests/unit/test_mcp_block_tools.py b/tests/unit/test_mcp_block_tools.py
index 2da9a9e6d..f42fd7d1b 100644
--- a/tests/unit/test_mcp_block_tools.py
+++ b/tests/unit/test_mcp_block_tools.py
@@ -52,6 +52,16 @@ async def test_workflow_knowledge_renders_the_code_block_runtime_names_from_the_
assert "solve_captcha" in content
+async def test_workflow_knowledge_captcha_solver_topic_carries_the_image_form() -> None:
+ result = await skyvern_workflow_knowledge(topics=["captcha_solver"])
+
+ assert result["ok"] is True
+ content = result["data"]["sections"]["captcha_solver"]["content"]
+ assert "solve_captcha(page, image=" in content
+ assert "input=" in content
+
+
@pytest.mark.asyncio
async def test_workflow_knowledge_returns_only_the_requested_authoritative_sections() -> None:
result = await skyvern_workflow_knowledge(topics=["workflow_parameters", "error_handling_and_retries"])