Skip to content

Automate browser based workflows with AI

fix(SKY-12735): preserve open ARIA popups from screenshot-scroll dismissal (#7837)

LawyZheng5e8200f61603
diff --git a/docs/developers/features/browser-sessions.mdx b/docs/developers/features/browser-sessions.mdx
index bc7cb5536..316fd89cd 100644
--- a/docs/developers/features/browser-sessions.mdx
+++ b/docs/developers/features/browser-sessions.mdx
@@ -28,7 +28,7 @@ A Browser Session is a live browser instance that Skyvern keeps running between
## Configuration
-Sessions support timeouts from 5 minutes to 24 hours (defaulting to 60 minutes), proxy locations across any of Skyvern's [21 supported countries](/developers/going-to-production/proxy-geolocation), Chrome or Edge browser types, and built-in extensions for ad-blocking and CAPTCHA solving. Every session exposes a live VNC stream.
+Sessions support timeouts from 5 minutes to 24 hours (defaulting to 60 minutes), proxy locations across any of Skyvern's [21 supported countries](/developers/going-to-production/proxy-geolocation), Chrome or Edge browser types, and built-in extensions for ad-blocking and CAPTCHA solving. The CAPTCHA solver isn't available on the Chrome browser type — use Edge or the managed stealth default (see [Choose the right browser type](/developers/optimization/browser-sessions#choose-the-right-browser-type)). Every session exposes a live VNC stream.
## Sessions vs. Profiles
diff --git a/docs/developers/optimization/browser-sessions.mdx b/docs/developers/optimization/browser-sessions.mdx
index 19c8540f3..72a56d16e 100644
--- a/docs/developers/optimization/browser-sessions.mdx
+++ b/docs/developers/optimization/browser-sessions.mdx
@@ -84,7 +84,7 @@ curl -X POST "https://api.skyvern.com/v1/browser_sessions" \
|-----------|------|-------------|
| `timeout` | integer | Session lifetime in minutes. Min: 5, Max: 1440 (24 hours). Default: `60` |
| `proxy_location` | string | Geographic proxy location (Cloud only). See [Proxy & Geolocation](/developers/going-to-production/proxy-geolocation) for available options |
-| `browser_type` | string | Browser type: `chrome` or `msedge` |
+| `browser_type` | string | Browser type: `chrome` or `msedge`. If omitted, Skyvern picks a managed default that may be a hardened stealth Chromium build rather than Chrome. Set it explicitly to select the engine when no `browser_profile_id` is used — a supplied profile's source browser type takes precedence. See [Choose the right browser type](#choose-the-right-browser-type) |
| `extensions` | array | Extensions to install: `ad-blocker`, `captcha-solver` |
| `generate_browser_profile` | boolean | REST API only for now — not yet accepted by the SDK methods. Save the session's browser profile (cookies, localStorage, session files) when the session ends, so it can be turned into a reusable [browser profile](/developers/optimization/browser-profiles). Default: `false`. See [Save a session's profile](#save-a-sessions-profile) |
@@ -647,33 +647,40 @@ await client.runWorkflow({ body: { workflow_id: "wpid_abc" }, waitForCompletion:
### Choose the right browser type
-Chrome has the widest compatibility. Use Edge only when a site requires or detects it specifically.
+If you omit `browser_type`, Skyvern selects a managed default for your account — this may be a hardened stealth Chromium build rather than Google Chrome, so don't rely on the default being Chrome. Set `browser_type` explicitly when you need a specific engine. Note that if you load a session from a `browser_profile_id`, the profile's source browser type takes precedence over `browser_type`.
+
+Each engine has a trade-off:
+
+- **`chrome`** — Google Chrome. Can work better on some fingerprint-sensitive sites where the managed stealth default gets challenged, but does **not** support the CAPTCHA-solver extension. This varies by site — test for yours.
+- **`msedge`** — Microsoft Edge. Supports the CAPTCHA-solver extension, so use it (or the managed stealth default) when a site serves hCaptcha/reCAPTCHA and you need automatic solving.
+
+Site behavior varies: if a site both fingerprints aggressively and serves CAPTCHAs, test both engines to find the best fit.
<CodeGroup>
```python Python
-# Chrome (default) - widest compatibility
+# Chrome - can help on some fingerprint-sensitive sites; no CAPTCHA-solver extension
session = await client.create_browser_session(browser_type="chrome")
-# Edge - for sites that require or fingerprint Edge
+# Edge - supports the CAPTCHA-solver extension for hCaptcha/reCAPTCHA
session = await client.create_browser_session(browser_type="msedge")
```
```typescript TypeScript
-// Chrome (default) - widest compatibility
+// Chrome - can help on some fingerprint-sensitive sites; no CAPTCHA-solver extension
const chromeSession = await client.createBrowserSession({ browser_type: "chrome" });
-// Edge - for sites that require or fingerprint Edge
+// Edge - supports the CAPTCHA-solver extension for hCaptcha/reCAPTCHA
const edgeSession = await client.createBrowserSession({ browser_type: "msedge" });
```
```bash cURL
-# Chrome (default) - widest compatibility
+# Chrome - can help on some fingerprint-sensitive sites; no CAPTCHA-solver extension
curl -s -X POST "https://api.skyvern.com/v1/browser_sessions" \
-H "x-api-key: $SKYVERN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"browser_type": "chrome"}'
-# Edge - for sites that require or fingerprint Edge
+# Edge - supports the CAPTCHA-solver extension for hCaptcha/reCAPTCHA
curl -s -X POST "https://api.skyvern.com/v1/browser_sessions" \
-H "x-api-key: $SKYVERN_API_KEY" \
-H "Content-Type: application/json" \
@@ -683,7 +690,7 @@ curl -s -X POST "https://api.skyvern.com/v1/browser_sessions" \
### Use extensions strategically
-Extensions add startup time, so only enable them when needed. The ad-blocker removes overlay ads that can interfere with automation. The captcha-solver handles CAPTCHAs automatically but is only available on Cloud.
+Extensions add startup time, so only enable them when needed. The ad-blocker removes overlay ads that can interfere with automation. The captcha-solver handles CAPTCHAs automatically but is only available on Cloud, and it requires the `msedge` browser type (or the managed stealth default) — the `chrome` browser type does not support it.
<CodeGroup>
```python Python
diff --git a/skyvern/forge/agent.py b/skyvern/forge/agent.py
index dc3d3722d..e329d909c 100644
--- a/skyvern/forge/agent.py
+++ b/skyvern/forge/agent.py
@@ -114,6 +114,12 @@ from skyvern.forge.sdk.db.enums import TaskType
from skyvern.forge.sdk.experimentation.enrich_tree import resolve_enrich_tree_for_context
from skyvern.forge.sdk.experimentation.llm_prompt_config import resolve_check_user_goal_handler
from skyvern.forge.sdk.experimentation.slim_llm_output import get_slim_output_template_value
+from skyvern.forge.sdk.experimentation.transient_ui_capture import (
+ decide_transient_ui_suppression,
+ emit_transient_ui_popup_telemetry,
+ resolve_transient_ui_capture_arm,
+ transient_ui_capture_arm,
+)
from skyvern.forge.sdk.fail_fast.shadow import record_fail_fast_shadow
from skyvern.forge.sdk.log_artifacts import save_step_logs, save_task_logs
from skyvern.forge.sdk.models import SpeculativeLLMMetadata, Step, StepStatus
@@ -3446,6 +3452,24 @@ class ForgeAgent:
if engine in CUA_ENGINES:
scrolling_number = 0
+ elif scrolling_number > 0 and skyvern_frame is not None:
+ arm = transient_ui_capture_arm(context)
+ if arm != "off":
+ # Shadow-detect an open transient popup in control; only treatment suppresses the
+ # scroll that would dismiss it (e.g. a portal date-picker popup) before the next step.
+ # This post-action screenshot shares the per-run consecutive-suppression cap with the
+ # agent-step scrape so a stale expanded trigger cannot pin captures at one viewport.
+ popup_trigger = await skyvern_frame.get_open_aria_popup_trigger()
+ _span.set_attribute("transient_ui_arm", arm)
+ if popup_trigger is not None:
+ _span.set_attribute("transient_ui_detected", True)
+ emit_transient_ui_popup_telemetry(_span, popup_trigger)
+ decision = decide_transient_ui_suppression(context, arm, detected=popup_trigger is not None)
+ if decision.suppress:
+ scrolling_number = 0
+ _span.set_attribute("transient_ui_scroll_suppressed", True)
+ elif decision.capped:
+ _span.set_attribute("transient_ui_suppression_capped", True)
artifacts: list[BulkArtifactCreationRequest | None] = []
screenshot_artifact_id: str | None = None
@@ -3661,6 +3685,9 @@ class ForgeAgent:
max_screenshot_number=max_screenshot_number,
draw_boxes=draw_boxes,
scroll=scroll,
+ # Only this agent-step scrape opts into transient-UI scroll suppression (SKY-12735);
+ # verification / extraction / error-detection scrapes keep legacy scrolling.
+ allow_transient_ui_suppression=True,
)
@traced(name="skyvern.agent.scrape_and_prompt", role="wrapper")
@@ -3752,6 +3779,8 @@ class ForgeAgent:
)
context.use_artifact_bundling = False
+ await resolve_transient_ui_capture_arm(context)
+
# start the async tasks while running scrape_website
if engine not in CUA_ENGINES:
self.async_operation_pool.run_operation(task.task_id, AgentPhase.scrape)
diff --git a/skyvern/forge/sdk/core/skyvern_context.py b/skyvern/forge/sdk/core/skyvern_context.py
index 54c95653a..a49965c2b 100644
--- a/skyvern/forge/sdk/core/skyvern_context.py
+++ b/skyvern/forge/sdk/core/skyvern_context.py
@@ -131,6 +131,25 @@ class SkyvernContext:
# PostHog flag ENABLE_LEAN_ELEMENT_TREE, evaluated once per run at scrape time
# and read sync from prompt-build sites.
enable_lean_element_tree: bool = False
+ # PRESERVE_TRANSIENT_UI_CAPTURE experiment arm, resolved per run. Tri-state: True=treatment
+ # (suppress a scroll that would dismiss an open transient popup), False=control (shadow-detect
+ # only), None=off (undefined/no-provider/error -> current scrolling behavior).
+ preserve_transient_ui_capture: bool | None = None
+ # Pinned once resolve_transient_ui_capture_arm resolves the arm (including off/None), so a TTL
+ # expiry or mid-run flag ramp cannot flip the arm later in the same run.
+ preserve_transient_ui_capture_resolved: bool = False
+ # Single-flight the first-use provider resolution when parallel blocks/branches share one
+ # context, so it is queried at most once per run (mirrors slim_output_variant_lock).
+ preserve_transient_ui_capture_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
+ # Count of CONSECUTIVE agent-step captures the treatment arm has suppressed scrolling on. Co-owned
+ # by the two agent-step capture sites — the agent-step scrape (scrape_web_unsafe with
+ # allow_transient_ui_suppression=True) and the post-action screenshot
+ # (record_artifacts_after_action) — via decide_transient_ui_suppression: incremented when a
+ # capture suppresses, reset to 0 when a qualifying popup is not detected, and frozen at the cap
+ # while a stale expanded trigger keeps matching so later captures fall back to legacy scrolling.
+ # Both sites for a run run sequentially, so the read-modify-write needs no lock; verification /
+ # extraction / error-detection scrapes never touch it.
+ transient_ui_consecutive_suppressions: int = 0
enrich_tree_mode: EnrichTreeMode = EnrichTreeMode.CONTROL
step_retry_index: int = 0
diff --git a/skyvern/forge/sdk/experimentation/transient_ui_capture.py b/skyvern/forge/sdk/experimentation/transient_ui_capture.py
new file mode 100644
index 000000000..a9f7f5118
--- /dev/null
+++ b/skyvern/forge/sdk/experimentation/transient_ui_capture.py
@@ -0,0 +1,139 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+import structlog
+
+from skyvern.forge import app
+from skyvern.forge.sdk.core import skyvern_context
+from skyvern.forge.sdk.experimentation.providers import NoOpExperimentationProvider
+
+LOG = structlog.get_logger()
+
+PRESERVE_TRANSIENT_UI_CAPTURE_FLAG = "PRESERVE_TRANSIENT_UI_CAPTURE"
+
+ARM_TREATMENT = "treatment"
+ARM_CONTROL = "control"
+ARM_OFF = "off"
+
+# A trigger that stays aria-expanded=true (e.g. a stale/undismissed combobox) would otherwise hold
+# every agent-step capture at one viewport for the rest of the run. Allow at most this many
+# CONSECUTIVE suppressing captures — enough for the post-action screenshot plus the next agent-step
+# scrape — then fall back to legacy scrolling until a scrape sees no qualifying popup (which resets
+# the run's consecutive counter).
+MAX_CONSECUTIVE_TRANSIENT_UI_SUPPRESSIONS = 2
+
+# role / aria-haspopup are page-controlled attribute values. domUtils.js already returns only the
+# matched token, but the capture site re-validates against these allowlists so arbitrary
+# attacker-authored text can never reach telemetry even if the detector return shape changes.
+TRANSIENT_UI_ROLE_TELEMETRY_ALLOWLIST = frozenset({"combobox"})
+TRANSIENT_UI_HASPOPUP_TELEMETRY_ALLOWLIST = frozenset({"true", "listbox", "menu", "dialog", "grid", "tree"})
+
+
+async def resolve_transient_ui_capture_arm(
+ context: skyvern_context.SkyvernContext,
+ *,
+ distinct_id: str | None = None,
+ organization_id: str | None = None,
+ workflow_permanent_id: str | None = None,
+) -> None:
+ """Resolve the PRESERVE_TRANSIENT_UI_CAPTURE experiment once at execution start and pin the
+ tri-state arm on the context. Idempotent: after the first resolution — True, False, or None
+ (undefined flag, no provider, missing run identity, or resolver error) — later calls return
+ immediately, so a TTL expiry or a mid-run flag ramp cannot flip the arm and the provider is
+ queried at most once per run; the capture hot paths only read the cached value. Off is the
+ safe default.
+
+ An execution-boundary caller whose context does not yet carry full run identity — notably an
+ inline child workflow, whose scoped context has no workflow_run_id/organization_id — may pass
+ distinct_id/organization_id/workflow_permanent_id explicitly so the child is attributed to its
+ own run rather than the parent's; otherwise they are read from the context.
+ """
+ if context.preserve_transient_ui_capture_resolved:
+ return
+ provider = app.EXPERIMENTATION_PROVIDER
+ if isinstance(provider, NoOpExperimentationProvider):
+ # No experimentation configured (OSS default) -> off, and never query the provider.
+ context.preserve_transient_ui_capture_resolved = True
+ return
+ resolved_distinct_id = (
+ distinct_id or context.workflow_run_id or context.task_id or context.task_v2_id or context.run_id
+ )
+ if not resolved_distinct_id:
+ context.preserve_transient_ui_capture_resolved = True
+ return
+ # Single-flight the provider query: concurrent first-resolvers sharing this context must not
+ # both call the provider. Re-check inside the lock in case another coroutine pinned it while we
+ # waited (the fast-path check above is unlocked to keep already-resolved contexts lock-free).
+ async with context.preserve_transient_ui_capture_lock:
+ if context.preserve_transient_ui_capture_resolved:
+ return
+ try:
+ context.preserve_transient_ui_capture = await provider.resolve_feature_flag_cached(
+ PRESERVE_TRANSIENT_UI_CAPTURE_FLAG,
+ resolved_distinct_id,
+ properties={
+ "organization_id": organization_id or context.organization_id,
+ "workflow_permanent_id": workflow_permanent_id or context.workflow_permanent_id or "not_workflow",
+ },
+ )
+ except Exception:
+ LOG.warning("Failed to resolve PRESERVE_TRANSIENT_UI_CAPTURE; defaulting to off/unenrolled", exc_info=True)
+ context.preserve_transient_ui_capture = None
+ context.preserve_transient_ui_capture_resolved = True
+
+
+@dataclass(frozen=True)
+class TransientUiSuppressionDecision:
+ suppress: bool
+ capped: bool
+
+
+def decide_transient_ui_suppression(
+ context: skyvern_context.SkyvernContext | None, arm: str, *, detected: bool
+) -> TransientUiSuppressionDecision:
+ """Apply the per-run consecutive-suppression cap for one agent-step capture and update the run
+ counter. ``detected`` is whether a qualifying popup was found on THIS capture. Only the treatment
+ arm suppresses or touches the counter (control/off shadow-detect but never suppress). The agent-
+ step scrape and the post-action screenshot both call this so their consecutive suppressions count
+ against ONE budget: suppress until the cap, then fall back to legacy scrolling, and reset the
+ counter as soon as a capture sees no qualifying popup."""
+ if arm != ARM_TREATMENT:
+ return TransientUiSuppressionDecision(suppress=False, capped=False)
+ if not detected:
+ if context is not None:
+ context.transient_ui_consecutive_suppressions = 0
+ return TransientUiSuppressionDecision(suppress=False, capped=False)
+ if context is None or context.transient_ui_consecutive_suppressions < MAX_CONSECUTIVE_TRANSIENT_UI_SUPPRESSIONS:
+ if context is not None:
+ context.transient_ui_consecutive_suppressions += 1
+ return TransientUiSuppressionDecision(suppress=True, capped=False)
+ return TransientUiSuppressionDecision(suppress=False, capped=True)
+
+
+def emit_transient_ui_popup_telemetry(span: Any, popup_trigger: dict) -> None:
+ """Emit bounded popup attributes to ``span``. role / aria-haspopup are page-controlled, so only
+ an allowlisted matched token is emitted (a non-allowlisted value is omitted); controls-resolved
+ is a bounded boolean so the no-target portal fallback (the main false-positive source) is
+ measurable. Shared by both capture sites so telemetry stays consistent."""
+ role_val = popup_trigger.get("role")
+ if isinstance(role_val, str) and role_val in TRANSIENT_UI_ROLE_TELEMETRY_ALLOWLIST:
+ span.set_attribute("transient_ui_role", role_val)
+ haspopup_val = popup_trigger.get("hasPopup")
+ if isinstance(haspopup_val, str) and haspopup_val in TRANSIENT_UI_HASPOPUP_TELEMETRY_ALLOWLIST:
+ span.set_attribute("transient_ui_haspopup", haspopup_val)
+ span.set_attribute("transient_ui_controls_resolved", bool(popup_trigger.get("controlsResolved")))
+
+
+def transient_ui_capture_arm(context: skyvern_context.SkyvernContext | None) -> str:
+ """Map the cached tri-state assignment to an arm: treatment (suppress scroll on detection),
+ control (shadow-detect only), or off (undefined/no-provider/error/missing -> current behavior)."""
+ if context is None:
+ return ARM_OFF
+ value = context.preserve_transient_ui_capture
+ if value is True:
+ return ARM_TREATMENT
+ if value is False:
+ return ARM_CONTROL
+ return ARM_OFF
diff --git a/skyvern/forge/sdk/workflow/service.py b/skyvern/forge/sdk/workflow/service.py
index 53d298f68..b89417db2 100644
--- a/skyvern/forge/sdk/workflow/service.py
+++ b/skyvern/forge/sdk/workflow/service.py
@@ -74,6 +74,7 @@ from skyvern.forge.sdk.db.enums import BrowserSeedSource, OrganizationAuthTokenT
from skyvern.forge.sdk.db.id import generate_output_parameter_id, generate_workflow_parameter_id
from skyvern.forge.sdk.enterprise_features import collect_enterprise_gated_run_features
from skyvern.forge.sdk.experimentation.enrich_tree import resolve_enrich_tree_for_context
+from skyvern.forge.sdk.experimentation.transient_ui_capture import resolve_transient_ui_capture_arm
from skyvern.forge.sdk.models import Step, StepStatus
from skyvern.forge.sdk.schemas.browser_profiles import BrowserProfile
from skyvern.forge.sdk.schemas.files import FileInfo
@@ -3286,6 +3287,16 @@ class WorkflowService:
current_context.generate_script = False
if workflow_run.code_gen:
current_context.generate_script = True
+ # Assign the transient-UI capture arm at execution start — before any block
+ # capture, and for cached/code runs that never reach agent_step — using the run
+ # being executed, so an inline child workflow whose scoped context lacks full
+ # identity is attributed to its own run rather than the parent's.
+ await resolve_transient_ui_capture_arm(
+ current_context,
+ distinct_id=workflow_run.workflow_run_id,
+ organization_id=organization.organization_id,
+ workflow_permanent_id=workflow_run.workflow_permanent_id,
+ )
workflow_run, blocks_to_update = await self._execute_workflow_blocks(
workflow=workflow,
workflow_run=workflow_run,
diff --git a/skyvern/services/task_v2_service.py b/skyvern/services/task_v2_service.py
index 01554c19c..723961272 100644
--- a/skyvern/services/task_v2_service.py
+++ b/skyvern/services/task_v2_service.py
@@ -552,9 +552,11 @@ async def run_task_v2(
workflow, workflow_run = None, None
parent_context = skyvern_context.current()
current_run_id = parent_context.run_id if parent_context and parent_context.run_id else task_v2_id
- # Carry trigger_type + use_flex_llm_routing forward from the parent context (set in
- # scripts/run_task_v2.py at worker boot) so flex routing eligibility persists into the
- # scoped TaskV2 execution context.
+ # Carry trigger_type, use_flex_llm_routing, and the execution-start transient-UI capture arm
+ # forward from the parent context (set in scripts/run_task_v2.py at worker boot). scoped() takes
+ # a wholesale replacement context, so any field not copied here is dropped for the scope; without
+ # the arm fields the boot resolution would be lost and TaskV2 would re-resolve (or run off) before
+ # its first capture.
context = SkyvernContext(
organization_id=organization_id,
organization_name=organization.organization_name,
@@ -567,6 +569,10 @@ async def run_task_v2(
trigger_type=parent_context.trigger_type if parent_context else None,
use_flex_llm_routing=parent_context.use_flex_llm_routing if parent_context else False,
consecutive_captcha_timeouts=parent_context.consecutive_captcha_timeouts if parent_context else 0,
+ preserve_transient_ui_capture=parent_context.preserve_transient_ui_capture if parent_context else None,
+ preserve_transient_ui_capture_resolved=(
+ parent_context.preserve_transient_ui_capture_resolved if parent_context else False
+ ),
)
# SKY-7005: scoped() restores the parent context on exit, preserving
# loop_internal_state so per-iteration download filtering continues to
diff --git a/skyvern/webeye/actions/handler.py b/skyvern/webeye/actions/handler.py
index 194906d2c..2e630958c 100644
--- a/skyvern/webeye/actions/handler.py
+++ b/skyvern/webeye/actions/handler.py
@@ -208,6 +208,22 @@ def resolve_engine_selection_for_task(task: Task) -> BrowserEngineSelection | No
return browser_state.engine_selection if browser_state is not None else None
+def _is_selected_engine_timeout(exc: BaseException, engine_selection: BrowserEngineSelection | None) -> bool:
+ """A driver-native timeout under THIS run's selected engine; the stock Playwright ``TimeoutError``
+ identity when no engine is pinned (unchanged default)."""
+ if engine_selection is not None:
+ return engine_selection.is_engine_timeout_error(exc)
+ return isinstance(exc, TimeoutError)
+
+
+def _is_selected_engine_error(exc: BaseException, engine_selection: BrowserEngineSelection | None) -> bool:
+ """A driver-native error under THIS run's selected engine; the stock Playwright ``Error`` identity
+ when no engine is pinned (unchanged default)."""
+ if engine_selection is not None:
+ return engine_selection.is_engine_error(exc)
+ return isinstance(exc, PlaywrightError)
+
+
class _CollapseGateResult(NamedTuple):
family_enabled: bool
assigned: bool | None
@@ -4058,13 +4074,14 @@ async def handle_input_text_action(
skyvern_element = retargeted_element
can_input_text = True
+ engine_selection = resolve_engine_selection_for_task(task)
skyvern_frame = await SkyvernFrame.create_instance(skyvern_element.get_frame())
- incremental_scraped = IncrementalScrapePage(
- skyvern_frame=skyvern_frame, engine_selection=resolve_engine_selection_for_task(task)
- )
+ incremental_scraped = IncrementalScrapePage(skyvern_frame=skyvern_frame, engine_selection=engine_selection)
timeout = settings.BROWSER_ACTION_TIMEOUT_MS
- current_text = await get_input_value(skyvern_element.get_tag_name(), skyvern_element.get_locator())
+ current_text = await get_input_value(
+ skyvern_element.get_tag_name(), skyvern_element.get_locator(), engine_selection=engine_selection
+ )
if not is_totp_value and current_text == current_text_target:
return [ActionSuccess()]
@@ -4196,7 +4213,9 @@ async def handle_input_text_action(
if not prefilter_typeahead:
try:
await skyvern_element.press_key("ArrowDown")
- except TimeoutError:
+ except Exception as exc:
+ if not _is_selected_engine_timeout(exc, engine_selection):
+ raise
# sometimes we notice `press_key()` raise a timeout but actually the dropdown is opened.
LOG.info(
"Timeout to press ArrowDown to open dropdown, ignore the timeout and continue to execute the action",
@@ -4389,17 +4408,11 @@ async def handle_input_text_action(
is_date_related = input_or_select_context is not None and input_or_select_context.is_date_related is True
try:
await skyvern_element.input_clear()
- except TimeoutError:
- LOG.info("None input tag clear timeout", action=action)
- return [
- ActionFailure(
- InvalidElementForTextInput(
- element_id=action.element_id, tag_name=tag_name, is_date_related=is_date_related
- )
- )
- ]
- except Exception:
- LOG.warning("Failed to clear the input field", action=action, exc_info=True)
+ except Exception as exc:
+ if _is_selected_engine_timeout(exc, engine_selection):
+ LOG.info("None input tag clear timeout", action=action)
+ else:
+ LOG.warning("Failed to clear the input field", action=action, exc_info=True)
return [
ActionFailure(
InvalidElementForTextInput(
@@ -4700,35 +4713,50 @@ async def handle_input_text_action(
error_message=str(inc_error),
)
raise inc_error
- except PlaywrightError as inc_error:
- # Handle Playwright-specific errors during incremental element processing
- # (e.g., TOTP form auto-submit, or search-dropdown selection triggering navigation)
- error_message = str(inc_error).lower()
- if (
- "execution context was destroyed" in error_message
- or "navigation" in error_message
- or "target closed" in error_message
- ):
- # These are expected during page navigation/auto-submit, silently continue
- LOG.debug(
- "Playwright error during incremental element processing (likely page navigation)",
+ except Exception as inc_error:
+ # Driver-native errors during incremental processing (e.g. TOTP form auto-submit, or a
+ # search-dropdown selection triggering navigation) are classified against THIS run's
+ # selected engine, so a non-stock driver's navigation errors are tolerated identically;
+ # missing selection keeps the stock Playwright identity. Non-engine errors stay swallowed.
+ if _is_selected_engine_error(inc_error, engine_selection):
+ error_message = str(inc_error).lower()
+ if (
+ "execution context was destroyed" in error_message
+ or "navigation" in error_message
+ or "target closed" in error_message
+ ):
+ # These are expected during page navigation/auto-submit, silently continue
+ LOG.debug(
+ "Engine error during incremental element processing (likely page navigation)",
+ error_type=type(inc_error).__name__,
+ error_message=error_message,
+ )
+ else:
+ LOG.warning(
+ "Unexpected engine error during incremental element processing",
+ error_type=type(inc_error).__name__,
+ error_message=str(inc_error),
+ )
+ raise inc_error
+ elif isinstance(inc_error, PlaywrightError):
+ # A foreign stock-Playwright driver error under a pinned non-stock engine. Under the
+ # stock default this arm is unreachable (a PlaywrightError satisfies the engine
+ # predicate above), so this is a no-op there. It is not one of the selected engine's
+ # tolerances, so propagate it exactly as the pre-PR ``except PlaywrightError`` did
+ # instead of falling through to a false ActionSuccess.
+ LOG.warning(
+ "Foreign driver error during incremental element processing under a pinned engine",
error_type=type(inc_error).__name__,
- error_message=error_message,
+ error_message=str(inc_error),
)
+ raise inc_error
else:
+ # Handle any other unexpected errors during incremental element processing
LOG.warning(
- "Unexpected Playwright error during incremental element processing",
+ "Unexpected error during incremental element processing",
error_type=type(inc_error).__name__,
error_message=str(inc_error),
)
- raise inc_error
- except Exception as inc_error:
- # Handle any other unexpected errors during incremental element processing
- LOG.warning(
- "Unexpected error during incremental element processing",
- error_type=type(inc_error).__name__,
- error_message=str(inc_error),
- )
finally:
# Always stop listening
await incremental_scraped.stop_listen_dom_increment()
@@ -4794,7 +4822,7 @@ def _find_similar_url_in_text(candidate_url: str, text: str) -> str | None:
return matched
-async def _wait_for_upload_processing(page: Page) -> None:
+async def _wait_for_upload_processing(page: Page, engine_selection: BrowserEngineSelection | None = None) -> None:
"""Wait for page readiness signals after a file upload.
Covers upload-processing UI (spinners, progress bars, DOM updates) beyond
@@ -4812,10 +4840,20 @@ async def _wait_for_upload_processing(page: Page) -> None:
dom_stable_ms=300,
dom_stability_timeout_ms=2000,
)
- except (TimeoutError, asyncio.TimeoutError, SkyvernPageAnalysisTimeout):
+ except (asyncio.TimeoutError, SkyvernPageAnalysisTimeout):
LOG.info("Upload processing page-ready wait timed out, continuing")
- except PlaywrightError:
- LOG.warning("Upload processing page-ready wait interrupted by Playwright error, continuing", exc_info=True)
+ except Exception as exc:
+ # Classify against THIS run's selected engine so a non-stock driver's timeout/error is
+ # tolerated identically; missing selection keeps the stock Playwright identity. Anything
+ # that is not an engine error propagates, as before.
+ if _is_selected_engine_timeout(exc, engine_selection):
+ LOG.info("Upload processing page-ready wait timed out, continuing")
+ elif _is_selected_engine_error(exc, engine_selection):
+ LOG.warning(
+ "Upload processing page-ready wait interrupted by browser engine error, continuing", exc_info=True
+ )
+ else:
+ raise
@traced(name="skyvern.agent.action.upload_file")
@@ -4893,7 +4931,7 @@ async def handle_upload_file_action(
timeout=settings.BROWSER_ACTION_TIMEOUT_MS,
)
- await _wait_for_upload_processing(page)
+ await _wait_for_upload_processing(page, engine_selection=resolve_engine_selection_for_task(task))
return [ActionSuccess()]
else:
@@ -6498,7 +6536,7 @@ async def chain_click(
# File chooser opened during this click — upload completed normally
LOG.info("File chooser triggered during this click", action=action)
if file:
- await _wait_for_upload_processing(page)
+ await _wait_for_upload_processing(page, engine_selection=engine_selection)
if not has_pending:
page.remove_listener("filechooser", fc_func)
if context is not None and context.pending_file_chooser is not None:
@@ -6534,7 +6572,7 @@ async def chain_click(
):
# A previous UPLOAD_FILE's deferred listener was consumed by this click
LOG.info("Pending file chooser from previous UPLOAD_FILE was consumed by this click", action=action)
- await _wait_for_upload_processing(page)
+ await _wait_for_upload_processing(page, engine_selection=engine_selection)
context.cleanup_pending_file_chooser()
else:
@@ -7155,7 +7193,9 @@ async def discover_and_select_from_full_dropdown(
)
try:
await skyvern_element.press_key("ArrowDown")
- except TimeoutError:
+ except Exception as exc:
+ if not _is_selected_engine_timeout(exc, resolve_engine_selection_for_task(task)):
+ raise
LOG.info(
"Timeout pressing ArrowDown in discover fallback, continuing",
element_id=skyvern_element.get_id(),
@@ -8254,7 +8294,7 @@ async def _reset_custom_select_combobox_input(
locator = element.get_locator()
await locator.fill("")
await element.click(page=page, engine_selection=engine_selection)
- return await get_input_value(element.get_tag_name(), locator) == ""
+ return await get_input_value(element.get_tag_name(), locator, engine_selection=engine_selection) == ""
except Exception:
LOG.info(
"Failed to reset custom-select combobox input before LLM fallback",
@@ -8474,7 +8514,11 @@ async def select_from_emerging_elements(
)
input_element = await dom_after_open.get_skyvern_element_by_id(element_id)
await input_element.scroll_into_view()
- current_text = await get_input_value(input_element.get_tag_name(), input_element.get_locator())
+ current_text = await get_input_value(
+ input_element.get_tag_name(),
+ input_element.get_locator(),
+ engine_selection=resolve_engine_selection_for_task(task),
+ )
if current_text == actual_value:
return ActionSuccess()
@@ -8666,7 +8710,11 @@ async def select_from_dropdown(
actual_value = get_actual_value_of_parameter_if_secret_with_task(task, value)
input_element = await SkyvernElement.create_from_incremental(incremental_scraped, element_id)
await input_element.scroll_into_view()
- current_text = await get_input_value(input_element.get_tag_name(), input_element.get_locator())
+ current_text = await get_input_value(
+ input_element.get_tag_name(),
+ input_element.get_locator(),
+ engine_selection=resolve_engine_selection_for_task(task),
+ )
if current_text == actual_value:
single_select_result.action_result = ActionSuccess()
return single_select_result
@@ -9776,18 +9824,24 @@ async def click_listbox_option(
return False
-async def get_input_value(tag_name: str, locator: Locator) -> str | None:
+async def get_input_value(
+ tag_name: str, locator: Locator, engine_selection: BrowserEngineSelection | None = None
+) -> str | None:
# input_value() rejects non-<input>/<textarea>/<select> nodes and inner_text() rejects
# non-HTMLElement nodes; the live node can disagree with the scraped tag_name after a
# re-render. Treat an incompatible read as "value unknown" so the caller's own
- # element-type classification runs instead of a raw Playwright exception escaping here.
+ # element-type classification runs instead of a raw driver exception escaping here. The
+ # incompatible-node identity is matched against THIS run's selected engine; missing selection
+ # keeps the stock Playwright identity (unchanged default).
try:
if tag_name in COMMON_INPUT_TAGS:
return await locator.input_value()
# for span, div, p or other tags:
# we need to trim the unicode space for these tags
return (await locator.inner_text()).replace("\xa0", " ").strip()
- except PlaywrightError as exc:
+ except Exception as exc:
+ if not _is_selected_engine_error(exc, engine_selection):
+ raise
if is_incompatible_text_input_error(exc):
LOG.info("Skipping value read on an incompatible element", tag_name=tag_name, error=str(exc))
return None
diff --git a/skyvern/webeye/browser_state.py b/skyvern/webeye/browser_state.py
index 5613fff90..9c30404c3 100644
--- a/skyvern/webeye/browser_state.py
+++ b/skyvern/webeye/browser_state.py
@@ -121,4 +121,5 @@ class BrowserState(Protocol):
support_empty_page: bool = False,
wait_seconds: float = 0,
must_included_tags: list[str] | None = None,
+ allow_transient_ui_suppression: bool = False,
) -> ScrapedPage: ...
diff --git a/skyvern/webeye/real_browser_state.py b/skyvern/webeye/real_browser_state.py
index f9e128032..c1e00fb8c 100644
--- a/skyvern/webeye/real_browser_state.py
+++ b/skyvern/webeye/real_browser_state.py
@@ -599,6 +599,7 @@ class RealBrowserState(BrowserState):
support_empty_page: bool = False,
wait_seconds: float = 0,
must_included_tags: list[str] | None = None,
+ allow_transient_ui_suppression: bool = False,
) -> ScrapedPage:
page = await self.get_working_page()
if page is not None:
@@ -618,6 +619,7 @@ class RealBrowserState(BrowserState):
support_empty_page=support_empty_page,
wait_seconds=wait_seconds,
must_included_tags=must_included_tags,
+ allow_transient_ui_suppression=allow_transient_ui_suppression,
)
async def close(self, close_browser_on_completion: bool = True, release_driver: bool | None = None) -> None:
diff --git a/skyvern/webeye/scraper/domUtils.js b/skyvern/webeye/scraper/domUtils.js
index 198c44e5a..39288066e 100644
--- a/skyvern/webeye/scraper/domUtils.js
+++ b/skyvern/webeye/scraper/domUtils.js
@@ -765,6 +765,204 @@ function isHiddenOrDisabled(element) {
return isHidden(element) || element.disabled;
}
+// Report an open ARIA popup trigger so one scrolling screenshot can skip scrolling while it is
+// open. Combobox / menu / listbox / dialog / grid / tree popups (portal-rendered date/list
+// pickers) usually render in a portal that dismisses on scroll, which the post-action
+// and next-scrape screenshots would otherwise trigger before the agent can act on the popup.
+const ARIA_POPUP_HASPOPUP_VALUES = new Set([
+ "true",
+ "listbox",
+ "menu",
+ "dialog",
+ "grid",
+ "tree",
+]);
+
+function isAriaPopupElementVisible(element) {
+ if (isHidden(element)) return false;
+ const style = getElementComputedStyle(element);
+ if (style) {
+ if (style.visibility === "hidden" || style.visibility === "collapse") {
+ return false;
+ }
+ const opacity = parseFloat(style.opacity);
+ if (!Number.isNaN(opacity) && opacity === 0) return false;
+ if (style.contentVisibility === "hidden") return false;
+ }
+ // Native flat-tree check: Element.checkVisibility (Chromium) rejects an element hidden by an
+ // ancestor's display / content-visibility, or a UA shadow root (e.g. a closed <details>), up the
+ // flat tree — which the light-DOM parentElement walk below cannot see. WebKit (checkVisibility
+ // bug) and unit mocks fall through to the manual walk.
+ if (!isElementStyleVisibilityVisible(element, style)) return false;
+ // WebKit-safe, mockable fallback: display / opacity / content-visibility do not propagate to a
+ // descendant's own computed style, so walk the FULL ancestor chain to reject a hidden portal (a
+ // hidden ancestor can sit at any depth — no arbitrary cap). A visited Set (identity) guards a
+ // malformed/cyclic parentElement chain: each ancestor is inspected at most once, so traversal
+ // terminates in O(distinct ancestors) even on a pathological DOM.
+ let ancestor = element.parentElement;
+ const seenAncestors = new Set();
+ while (ancestor && !seenAncestors.has(ancestor)) {
+ seenAncestors.add(ancestor);
+ if (ancestor.hidden) return false;
+ const ancestorStyle = getElementComputedStyle(ancestor);
+ if (ancestorStyle) {
+ if (ancestorStyle.display === "none") return false;
+ if (
+ ancestorStyle.visibility === "hidden" ||
+ ancestorStyle.visibility === "collapse"
+ ) {
+ return false;
+ }
+ const ancestorOpacity = parseFloat(ancestorStyle.opacity);
+ if (!Number.isNaN(ancestorOpacity) && ancestorOpacity === 0) return false;
+ if (ancestorStyle.contentVisibility === "hidden") return false;
+ }
+ ancestor = ancestor.parentElement;
+ }
+ return true;
+}
+
+// aria-controls / aria-owns are whitespace-separated IDREF lists; resolve them against the
+// light DOM only (IDREF resolution does not cross shadow boundaries). Returns the resolved
+// targets plus hadIdref: whether the trigger named at least one non-empty IDREF. A trigger that
+// named an IDREF but resolved nothing is a popup unmounted on close (dangling), which the caller
+// must distinguish from a trigger with no aria-controls at all (a portal wired without one).
+function resolveAriaPopupTargets(trigger) {
+ const targets = [];
+ let hadIdref = false;
+ for (const attr of ["aria-controls", "aria-owns"]) {
+ const raw = trigger.getAttribute(attr);
+ if (!raw) continue;
+ for (const id of raw.split(/\s+/)) {
+ if (!id) continue;
+ hadIdref = true;
+ const target = document.getElementById(id);
+ if (target) targets.push(target);
+ }
+ }
+ return { targets, hadIdref };
+}
+
+// A box paints an area only if its border box has positive width/height AND intersects the viewport
+// (shared by the trigger gate and the target rendered-area gate below). checkVisibility() and style
+// visibility pass for zero-area / transform:scale(0) / clipped elements, so geometry must be checked
+// separately or a target that renders nothing would false-positive as an open popup.
+function hasPositiveViewportArea(element) {
+ const rect = element.getBoundingClientRect();
+ return (
+ rect.width > 0 &&
+ rect.height > 0 &&
+ rect.bottom > 0 &&
+ rect.right > 0 &&
+ rect.top < window.innerHeight &&
+ rect.left < window.innerWidth
+ );
+}
+
+const CLIPPING_OVERFLOW_VALUES = new Set(["hidden", "clip", "scroll", "auto"]);
+
+function clipsOverflow(style) {
+ if (!style) return false;
+ return (
+ CLIPPING_OVERFLOW_VALUES.has(style.overflow) ||
+ CLIPPING_OVERFLOW_VALUES.has(style.overflowX) ||
+ CLIPPING_OVERFLOW_VALUES.has(style.overflowY)
+ );
+}
+
+// A descendant is clipped to nothing if a clipping-overflow ancestor between it and `boundary`
+// (inclusive) has zero on-screen area, so a zero-size scroll/hidden container swallows it. Identity
+// Set guards a malformed/cyclic chain; the walk stops at `boundary` (the resolved target).
+function isClippedByZeroAreaAncestor(descendant, boundary) {
+ let ancestor = descendant.parentElement;
+ const seen = new Set();
+ while (ancestor && !seen.has(ancestor)) {
+ seen.add(ancestor);
+ if (
+ clipsOverflow(getElementComputedStyle(ancestor)) &&
+ !hasPositiveViewportArea(ancestor)
+ ) {
+ return true;
+ }
+ if (ancestor === boundary) break;
+ ancestor = ancestor.parentElement;
+ }
+ return false;
+}
+
+// A resolved aria-controls target counts as an open popup only if it actually paints an area: its
+// own border box paints, or — when the target is a zero-area container that does NOT clip its
+// overflow (an overflow:visible portal mount point) — a genuinely rendered, visible descendant
+// escapes it. A zero-area target that clips its overflow (or has no rendered descendant) paints
+// nothing, so a stale aria-expanded=true trigger pointing at it is treated as closed. The subtree
+// walk needs no depth/count cap: a DOM subtree is finite and an identity Set visits each node once.
+function ariaPopupTargetHasRenderedArea(target) {
+ if (hasPositiveViewportArea(target)) return true;
+ if (clipsOverflow(getElementComputedStyle(target))) return false;
+ const stack = Array.from(target.children || []);
+ const seen = new Set();
+ while (stack.length) {
+ const descendant = stack.pop();
+ if (!descendant || seen.has(descendant)) continue;
+ seen.add(descendant);
+ if (
+ isAriaPopupElementVisible(descendant) &&
+ hasPositiveViewportArea(descendant) &&
+ !isClippedByZeroAreaAncestor(descendant, target)
+ ) {
+ return true;
+ }
+ for (const child of descendant.children || []) stack.push(child);
+ }
+ return false;
+}
+
+// Returns structural (non-PII) details of the first open ARIA popup trigger, or null. On a trigger
+// with no aria-controls/aria-owns at all, none-resolving => open (a sibling portal not wired via
+// aria-controls). On a trigger that named an IDREF: any resolved target visible+painting => open,
+// otherwise closed — a present-but-dangling IDREF is a popup unmounted on close, not an open one.
+// Scans the light DOM only, so triggers inside shadow roots are not detected. role / aria-haspopup
+// are page-controlled: only the matched allowlisted token is returned so arbitrary attacker-authored
+// attribute text never reaches telemetry.
+function getOpenAriaPopupTrigger() {
+ const candidates = document.querySelectorAll('[aria-expanded="true"]');
+ for (const trigger of candidates) {
+ const role = (trigger.getAttribute("role") || "").toLowerCase();
+ const hasPopup = (
+ trigger.getAttribute("aria-haspopup") || ""
+ ).toLowerCase();
+ const roleMatched = role === "combobox";
+ const hasPopupMatched = ARIA_POPUP_HASPOPUP_VALUES.has(hasPopup);
+ if (!roleMatched && !hasPopupMatched) {
+ continue;
+ }
+ if (!isAriaPopupElementVisible(trigger)) continue;
+ if (!hasPositiveViewportArea(trigger)) continue;
+
+ const { targets, hadIdref } = resolveAriaPopupTargets(trigger);
+ // No aria-controls/aria-owns at all => portal fallback (open). A named IDREF that resolves
+ // nothing => dangling => closed (falls through to targets.some over an empty list). A resolved
+ // target counts only if it is visible AND paints an area, so a stale trigger pointing at a
+ // zero-area/clipped target is treated as closed.
+ const open =
+ !hadIdref ||
+ targets.some(
+ (target) =>
+ isAriaPopupElementVisible(target) &&
+ ariaPopupTargetHasRenderedArea(target),
+ );
+ if (!open) continue;
+
+ return {
+ role: roleMatched ? "combobox" : null,
+ hasPopup: hasPopupMatched ? hasPopup : null,
+ tag: trigger.tagName ? trigger.tagName.toLowerCase() : null,
+ controlsResolved: targets.length,
+ };
+ }
+ return null;
+}
+
function isScriptOrStyle(element) {
const tagName = element.tagName.toLowerCase();
return tagName === "script" || tagName === "style";
diff --git a/skyvern/webeye/scraper/scraper.py b/skyvern/webeye/scraper/scraper.py
index de748d4f6..10a120483 100644
--- a/skyvern/webeye/scraper/scraper.py
+++ b/skyvern/webeye/scraper/scraper.py
@@ -24,6 +24,11 @@ from skyvern.experimentation.wait_utils import empty_page_retry_wait
from skyvern.forge.sdk.api.crypto import calculate_sha256
from skyvern.forge.sdk.browser_action_preflight import advance_observation_epoch
from skyvern.forge.sdk.core import skyvern_context
+from skyvern.forge.sdk.experimentation.transient_ui_capture import (
+ decide_transient_ui_suppression,
+ emit_transient_ui_popup_telemetry,
+ transient_ui_capture_arm,
+)
from skyvern.forge.sdk.settings_manager import SettingsManager
from skyvern.forge.sdk.trace import apply_context_attrs, traced, traced_span
from skyvern.utils.image_resizer import Resolution
@@ -229,6 +234,7 @@ async def scrape_website(
support_empty_page: bool = False,
wait_seconds: float = 0,
must_included_tags: list[str] | None = None,
+ allow_transient_ui_suppression: bool = False,
) -> ScrapedPage:
"""
************************************************************************************************
@@ -265,6 +271,7 @@ async def scrape_website(
support_empty_page=support_empty_page,
wait_seconds=wait_seconds,
must_included_tags=must_included_tags,
+ allow_transient_ui_suppression=allow_transient_ui_suppression,
)
except ScrapingFailedBlankPage:
raise
@@ -306,6 +313,7 @@ async def scrape_website(
max_screenshot_number=max_screenshot_number,
scroll=scroll,
must_included_tags=must_included_tags,
+ allow_transient_ui_suppression=allow_transient_ui_suppression,
)
@@ -433,6 +441,7 @@ async def scrape_web_unsafe(
support_empty_page: bool = False,
wait_seconds: float = 0,
must_included_tags: list[str] | None = None,
+ allow_transient_ui_suppression: bool = False,
) -> ScrapedPage:
"""
Asynchronous function that performs web scraping without any built-in error handling. This function is intended
@@ -496,6 +505,23 @@ async def scrape_web_unsafe(
if token_count > DEFAULT_MAX_TOKENS:
max_screenshot_number = min(max_screenshot_number, 1)
+ # Shadow-detect an open transient popup only on the agent-step scrape (opt-in via
+ # allow_transient_ui_suppression); goal-verification / extraction / error-detection scrapes
+ # keep legacy scrolling. Only the treatment arm suppresses the scroll so the popup survives
+ # into the just-built tree's next action, and only up to a bounded number of consecutive
+ # captures so a stale expanded trigger cannot pin the run at one viewport.
+ popup_trigger: dict | None = None
+ transient_ui_ctx = skyvern_context.current()
+ arm = transient_ui_capture_arm(transient_ui_ctx)
+ suppress_scroll = False
+ suppression_capped = False
+ if allow_transient_ui_suppression and scroll and arm != "off":
+ popup_trigger = await skyvern_frame.get_open_aria_popup_trigger()
+ decision = decide_transient_ui_suppression(transient_ui_ctx, arm, detected=popup_trigger is not None)
+ suppress_scroll = decision.suppress
+ suppression_capped = decision.capped
+ effective_scroll = scroll and not suppress_scroll
+
# get current x, y position of the page
x: int | None = None
y: int | None = None
@@ -515,6 +541,16 @@ async def scrape_web_unsafe(
_ss_span.set_attribute("max_screenshot_number", max_screenshot_number)
_ss_span.set_attribute("draw_boxes", draw_boxes)
_ss_span.set_attribute("scroll", scroll)
+ _ss_span.set_attribute("effective_scroll", effective_scroll)
+ if arm != "off":
+ _ss_span.set_attribute("transient_ui_arm", arm)
+ if popup_trigger is not None:
+ _ss_span.set_attribute("transient_ui_detected", True)
+ emit_transient_ui_popup_telemetry(_ss_span, popup_trigger)
+ if suppress_scroll:
+ _ss_span.set_attribute("transient_ui_scroll_suppressed", True)
+ if suppression_capped:
+ _ss_span.set_attribute("transient_ui_suppression_capped", True)
_scrape_ctx = skyvern_context.current()
if _scrape_ctx:
if _scrape_ctx.scrape_trigger:
@@ -526,7 +562,7 @@ async def scrape_web_unsafe(
url=url,
draw_boxes=draw_boxes,
max_number=max_screenshot_number,
- scroll=scroll,
+ scroll=effective_scroll,
)
_ss_span.set_attribute("screenshot_count", len(screenshots))
_ss_span.set_attribute("screenshot_bytes", sum(len(s) for s in screenshots))
diff --git a/skyvern/webeye/utils/page.py b/skyvern/webeye/utils/page.py
index cfa03bd05..dab46cf6f 100644
--- a/skyvern/webeye/utils/page.py
+++ b/skyvern/webeye/utils/page.py
@@ -1076,6 +1076,23 @@ class SkyvernFrame:
js_script = "() => getScrollXY()"
return await self.evaluate(frame=self.frame, engine_selection=self.engine_selection, expression=js_script)
+ async def get_open_aria_popup_trigger(self) -> dict | None:
+ """Return structural details of an open ARIA popup trigger on this frame, or None.
+
+ Fails open (returns None) on any evaluation error so screenshot-scroll policy keeps the
+ current scrolling behavior rather than breaking the scrape/action loop. Detection
+ semantics live in getOpenAriaPopupTrigger in domUtils.js.
+ """
+ try:
+ result = await self.evaluate(frame=self.frame, expression="() => getOpenAriaPopupTrigger()")
+ except Exception:
+ LOG.warning(
+ "Failed to detect open ARIA popup trigger; using default scrolling behavior",
+ exc_info=True,
+ )
+ return None
+ return result if isinstance(result, dict) else None
+
async def get_scroll_width_and_height(self) -> tuple[int, int]:
js_script = "() => getScrollWidthAndHeight()"
return await self.evaluate(frame=self.frame, engine_selection=self.engine_selection, expression=js_script)
diff --git a/tests/unit/test_aria_popup_trigger_domutils.js b/tests/unit/test_aria_popup_trigger_domutils.js
new file mode 100644
index 000000000..72fcc4539
--- /dev/null
+++ b/tests/unit/test_aria_popup_trigger_domutils.js
@@ -0,0 +1,830 @@
+/**
+ * Behavioral tests for getOpenAriaPopupTrigger() in domUtils.js.
+ *
+ * The predicate reports an open ARIA popup trigger (combobox / haspopup) so the
+ * screenshot-scroll policy can skip scrolling for one capture and avoid dismissing
+ * a just-opened portal popup (e.g. a date-picker overlay or ARIA listbox).
+ *
+ * Exit 0 = pass, exit 1 = failures on stderr.
+ */
+
+const fs = require("fs");
+const path = require("path");
+const vm = require("vm");
+
+const src = fs.readFileSync(
+ path.join(__dirname, "../../skyvern/webeye/scraper/domUtils.js"),
+ "utf8",
+);
+
+// Node-scope DOM registries the mocked document closures read.
+let allElements = [];
+const byId = new Map();
+
+function reset() {
+ allElements = [];
+ byId.clear();
+}
+
+function getComputedStyle(element) {
+ return {
+ display: "block",
+ visibility: "visible",
+ opacity: "1",
+ cursor: "auto",
+ pointerEvents: "auto",
+ ...(element._style || {}),
+ };
+}
+
+class TestElement {}
+
+const context = {
+ console,
+ window: {
+ scrollX: 0,
+ scrollY: 0,
+ innerHeight: 800,
+ innerWidth: 1200,
+ getComputedStyle,
+ },
+ document: {
+ documentElement: {},
+ querySelector() {
+ return null;
+ },
+ querySelectorAll(selector) {
+ if (selector === '[aria-expanded="true"]') {
+ return allElements.filter(
+ (el) => el.getAttribute("aria-expanded") === "true",
+ );
+ }
+ return [];
+ },
+ getElementById(id) {
+ return byId.get(id) || null;
+ },
+ },
+ MutationObserver: class {
+ observe() {}
+ disconnect() {}
+ takeRecords() {
+ return [];
+ }
+ },
+ Element: TestElement,
+ ShadowRoot: class {},
+ Node: { ELEMENT_NODE: 1, TEXT_NODE: 3 },
+ Number,
+ Set,
+};
+context.global = context;
+vm.runInNewContext(
+ `${src}
+this.__exports = { getOpenAriaPopupTrigger };`,
+ context,
+);
+
+function el(opts) {
+ const element = new TestElement();
+ element.tagName = (opts.tagName || "DIV").toUpperCase();
+ element.nodeName = element.tagName;
+ element.hidden = Boolean(opts.hidden);
+ element.disabled = Boolean(opts.disabled);
+ element._style = opts.style || {};
+ const rect = opts.rect || { width: 120, height: 32, left: 40, top: 120 };
+ element._rect = {
+ width: rect.width,
+ height: rect.height,
+ left: rect.left,
+ top: rect.top,
+ right: rect.right !== undefined ? rect.right : rect.left + rect.width,
+ bottom: rect.bottom !== undefined ? rect.bottom : rect.top + rect.height,
+ };
+ element._attributes = { ...(opts.attributes || {}) };
+ if (opts.id) {
+ element._attributes.id = opts.id;
+ }
+ element.getAttribute = function (name) {
+ return name in this._attributes ? this._attributes[name] : null;
+ };
+ element.hasAttribute = function (name) {
+ return name in this._attributes;
+ };
+ element.getBoundingClientRect = function () {
+ return this._rect;
+ };
+ element.ownerDocument = { defaultView: { getComputedStyle } };
+ element.getRootNode = function () {
+ return { host: null };
+ };
+ element.parentElement = opts.parent || null;
+ element.children = opts.children || [];
+ for (const child of element.children) {
+ child.parentElement = element;
+ }
+ element.closest = function () {
+ return null;
+ };
+ allElements.push(element);
+ if (opts.id) {
+ byId.set(opts.id, element);
+ }
+ return element;
+}
+
+// Build a chain of nested ancestors: a top ancestor styled by `topStyle`, then `levels` visible
+// descendants below it; returns the bottom-most node (use it as a deep element's parent).
+function deepChain(topStyle, levels) {
+ let node = el({ tagName: "DIV", style: topStyle });
+ for (let i = 0; i < levels; i++) {
+ node = el({ tagName: "DIV", parent: node });
+ }
+ return node;
+}
+
+let passed = 0;
+let failed = 0;
+
+function test(name, fn) {
+ reset();
+ try {
+ fn();
+ passed++;
+ process.stdout.write(` PASS: ${name}\n`);
+ } catch (e) {
+ failed++;
+ process.stderr.write(` FAIL: ${name}\n ${e.message}\n`);
+ }
+}
+
+function assert(condition, message) {
+ if (!condition) {
+ throw new Error(message);
+ }
+}
+
+const trigger = () => context.__exports.getOpenAriaPopupTrigger();
+
+test("open combobox with no aria-controls is reported (unresolved => open)", () => {
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "true" },
+ });
+ const result = trigger();
+ assert(
+ result !== null,
+ "an open combobox with no controls should be detected",
+ );
+ assert(
+ result.role === "combobox",
+ `role should be combobox, got ${result && result.role}`,
+ );
+});
+
+test("closed combobox (aria-expanded=false) is not reported", () => {
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "false" },
+ });
+ assert(trigger() === null, "a closed combobox must not be reported");
+});
+
+test("aria-haspopup=listbox with a visible controlled target is reported", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "lb1",
+ },
+ });
+ el({ id: "lb1", tagName: "UL", attributes: { role: "listbox" } });
+ const result = trigger();
+ assert(
+ result !== null,
+ "haspopup=listbox with a visible target should be detected",
+ );
+ assert(
+ result.hasPopup === "listbox",
+ `hasPopup should be listbox, got ${result && result.hasPopup}`,
+ );
+ assert(
+ result.controlsResolved === 1,
+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,
+ );
+});
+
+test("menu trigger whose only controlled target is display:none is closed", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "menu",
+ "aria-expanded": "true",
+ "aria-controls": "m1",
+ },
+ });
+ el({ id: "m1", tagName: "DIV", style: { display: "none" } });
+ assert(
+ trigger() === null,
+ "all resolved targets hidden => closed => not reported",
+ );
+});
+
+test("listbox trigger with one hidden and one visible target is open (any visible => open)", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "hidden1 visible1",
+ },
+ });
+ el({ id: "hidden1", tagName: "DIV", style: { display: "none" } });
+ el({ id: "visible1", tagName: "UL" });
+ const result = trigger();
+ assert(result !== null, "any visible resolved target => open");
+ assert(
+ result.controlsResolved === 2,
+ `controlsResolved should be 2, got ${result && result.controlsResolved}`,
+ );
+});
+
+test("aria-owns is parsed like aria-controls", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ role: "combobox",
+ "aria-expanded": "true",
+ "aria-owns": "own1",
+ },
+ });
+ el({ id: "own1", tagName: "DIV", style: { display: "none" } });
+ assert(trigger() === null, "aria-owns target hidden => closed");
+});
+
+test("expanded element with no combobox role and no popup role does not qualify", () => {
+ el({ tagName: "BUTTON", attributes: { "aria-expanded": "true" } });
+ assert(
+ trigger() === null,
+ "a bare aria-expanded element (accordion) must not qualify",
+ );
+});
+
+test("open combobox scrolled fully off-screen is not reported", () => {
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "true" },
+ rect: { width: 120, height: 32, left: 40, top: 2000 },
+ });
+ assert(
+ trigger() === null,
+ "a trigger below the viewport must not be reported",
+ );
+});
+
+test("open combobox with display:none is not reported", () => {
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "true" },
+ style: { display: "none" },
+ });
+ assert(trigger() === null, "a display:none trigger must not be reported");
+});
+
+test("aria-haspopup=true (string) qualifies", () => {
+ el({
+ tagName: "DIV",
+ attributes: { "aria-haspopup": "true", "aria-expanded": "true" },
+ });
+ assert(trigger() !== null, "aria-haspopup='true' should qualify");
+});
+
+test("aria-haspopup=dialog with grid/tree values all qualify", () => {
+ for (const value of ["dialog", "grid", "tree"]) {
+ reset();
+ el({
+ tagName: "DIV",
+ attributes: { "aria-haspopup": value, "aria-expanded": "true" },
+ });
+ assert(trigger() !== null, `aria-haspopup='${value}' should qualify`);
+ }
+});
+
+test("multiple whitespace-separated IDREFs none of which resolve => closed (all dangling)", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ role: "combobox",
+ "aria-expanded": "true",
+ "aria-controls": "gone1 gone2\tgone3",
+ },
+ });
+ // The trigger named IDREFs but none resolve: the popup was unmounted on close, so it is closed —
+ // distinct from a trigger with no aria-controls at all (which keeps the portal fallback => open).
+ assert(
+ trigger() === null,
+ "present-but-all-dangling IDREFs must be treated as closed, not open",
+ );
+});
+
+test("target inside a display:none ancestor is closed (ancestor-aware)", () => {
+ const hiddenAncestor = el({ tagName: "DIV", style: { display: "none" } });
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "menu",
+ "aria-expanded": "true",
+ "aria-controls": "t1",
+ },
+ });
+ el({ id: "t1", tagName: "DIV", parent: hiddenAncestor });
+ assert(
+ trigger() === null,
+ "a target under a display:none ancestor must not count as open",
+ );
+});
+
+test("target inside a visibility:hidden ancestor is closed (ancestor-aware)", () => {
+ const hiddenAncestor = el({
+ tagName: "DIV",
+ style: { visibility: "hidden" },
+ });
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "t2",
+ },
+ });
+ el({ id: "t2", tagName: "UL", parent: hiddenAncestor });
+ assert(
+ trigger() === null,
+ "a target under a visibility:hidden ancestor must not count as open",
+ );
+});
+
+test("target inside an opacity:0 ancestor is closed (ancestor-aware)", () => {
+ const hiddenAncestor = el({ tagName: "DIV", style: { opacity: "0" } });
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "dialog",
+ "aria-expanded": "true",
+ "aria-controls": "t3",
+ },
+ });
+ el({ id: "t3", tagName: "DIV", parent: hiddenAncestor });
+ assert(
+ trigger() === null,
+ "a target under an opacity:0 ancestor must not count as open",
+ );
+});
+
+test("trigger inside a display:none ancestor is not detected (ancestor-aware)", () => {
+ const hiddenAncestor = el({ tagName: "DIV", style: { display: "none" } });
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "true" },
+ parent: hiddenAncestor,
+ });
+ assert(
+ trigger() === null,
+ "a trigger under a display:none ancestor must not be reported",
+ );
+});
+
+test("faithful visible portal under a visible ancestor is still open (control)", () => {
+ const visibleAncestor = el({
+ tagName: "DIV",
+ style: { display: "block", opacity: "1" },
+ });
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "t4",
+ },
+ });
+ el({ id: "t4", tagName: "UL", parent: visibleAncestor });
+ const result = trigger();
+ assert(
+ result !== null,
+ "a target under a visible ancestor must still count as open",
+ );
+ assert(
+ result.controlsResolved === 1,
+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,
+ );
+});
+
+test("target nested >50 levels under a display:none ancestor is closed (full walk to root)", () => {
+ const bottom = deepChain({ display: "none" }, 55);
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "menu",
+ "aria-expanded": "true",
+ "aria-controls": "deepT",
+ },
+ });
+ el({ id: "deepT", tagName: "DIV", parent: bottom });
+ assert(
+ trigger() === null,
+ "a target under a display:none ancestor >50 levels up must not count as open",
+ );
+});
+
+test("trigger nested >50 levels under a display:none ancestor is not reported (full walk to root)", () => {
+ const bottom = deepChain({ display: "none" }, 55);
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "true" },
+ parent: bottom,
+ });
+ assert(
+ trigger() === null,
+ "a trigger under a display:none ancestor >50 levels up must not be reported",
+ );
+});
+
+test("valid deep portal (>50 levels, all visible) is still open (control — walk must not over-reject)", () => {
+ const bottom = deepChain({ display: "block", opacity: "1" }, 55);
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "deepV",
+ },
+ });
+ el({ id: "deepV", tagName: "UL", parent: bottom });
+ const result = trigger();
+ assert(
+ result !== null,
+ "a deep portal with all-visible ancestors must still count as open",
+ );
+ assert(
+ result.controlsResolved === 1,
+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,
+ );
+});
+
+test("target under a content-visibility:hidden ancestor is closed", () => {
+ const hiddenAncestor = el({
+ tagName: "DIV",
+ style: { contentVisibility: "hidden" },
+ });
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "menu",
+ "aria-expanded": "true",
+ "aria-controls": "cv1",
+ },
+ });
+ el({ id: "cv1", tagName: "DIV", parent: hiddenAncestor });
+ assert(
+ trigger() === null,
+ "a target under a content-visibility:hidden ancestor must not count as open",
+ );
+});
+
+test("trigger under a content-visibility:hidden ancestor is not reported", () => {
+ const hiddenAncestor = el({
+ tagName: "DIV",
+ style: { contentVisibility: "hidden" },
+ });
+ el({
+ tagName: "DIV",
+ attributes: { role: "combobox", "aria-expanded": "true" },
+ parent: hiddenAncestor,
+ });
+ assert(
+ trigger() === null,
+ "a trigger under a content-visibility:hidden ancestor must not be reported",
+ );
+});
+
+test("target with its own content-visibility:hidden is closed", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "cv2",
+ },
+ });
+ el({ id: "cv2", tagName: "UL", style: { contentVisibility: "hidden" } });
+ assert(
+ trigger() === null,
+ "a target with its own content-visibility:hidden must not count as open",
+ );
+});
+
+test("content-visibility:visible ancestor does not reject a valid portal (control)", () => {
+ const visibleAncestor = el({
+ tagName: "DIV",
+ style: { contentVisibility: "visible" },
+ });
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "cv3",
+ },
+ });
+ el({ id: "cv3", tagName: "UL", parent: visibleAncestor });
+ assert(
+ trigger() !== null,
+ "a content-visibility:visible ancestor must not reject a valid portal",
+ );
+});
+
+// --- Rendered-area gate for the controlled target ----------------------------------------------
+// checkVisibility() / style visibility pass for a zero-area or transform:scale(0) target, so a
+// stale aria-expanded=true trigger pointing at a target that paints nothing would false-positive as
+// open and suppress a needed scroll under treatment. The target must actually render an area.
+
+test("zero-area controlled target is closed (rendered-area gate)", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "za1",
+ },
+ });
+ el({
+ id: "za1",
+ tagName: "UL",
+ rect: { width: 0, height: 0, left: 40, top: 120 },
+ });
+ assert(
+ trigger() === null,
+ "a zero-area controlled target must not count as open",
+ );
+});
+
+test("transform:scale(0) controlled target is closed (rendered-area gate)", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "sc1",
+ },
+ });
+ // scale(0) collapses the box AND its subtree to zero on-screen area.
+ const scaledChild = el({
+ tagName: "LI",
+ rect: { width: 0, height: 0, left: 40, top: 120 },
+ });
+ el({
+ id: "sc1",
+ tagName: "UL",
+ style: { transform: "scale(0)" },
+ rect: { width: 0, height: 0, left: 40, top: 120 },
+ children: [scaledChild],
+ });
+ assert(
+ trigger() === null,
+ "a transform:scale(0) controlled target must not count as open",
+ );
+});
+
+test("zero-size overflow:hidden target with a positive child is closed (clipped)", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "oh1",
+ },
+ });
+ // The child has layout area but is clipped to the 0x0 content box, so nothing paints.
+ const clippedChild = el({
+ tagName: "LI",
+ rect: { width: 240, height: 120, left: 40, top: 120 },
+ });
+ el({
+ id: "oh1",
+ tagName: "UL",
+ style: { overflow: "hidden", overflowX: "hidden", overflowY: "hidden" },
+ rect: { width: 0, height: 0, left: 40, top: 120 },
+ children: [clippedChild],
+ });
+ assert(
+ trigger() === null,
+ "a zero-size overflow:hidden container clips its child => closed",
+ );
+});
+
+test("zero-size overflow:visible portal root with a rendered visible descendant is open", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "ov1",
+ },
+ });
+ // A portal mount point that itself has no box, but a genuinely painted panel escapes it.
+ const panel = el({
+ tagName: "DIV",
+ rect: { width: 240, height: 120, left: 40, top: 120 },
+ });
+ el({
+ id: "ov1",
+ tagName: "DIV",
+ style: { overflow: "visible", overflowX: "visible", overflowY: "visible" },
+ rect: { width: 0, height: 0, left: 40, top: 120 },
+ children: [panel],
+ });
+ const result = trigger();
+ assert(
+ result !== null,
+ "an overflow:visible zero-area portal root with a rendered descendant => open",
+ );
+ assert(
+ result.controlsResolved === 1,
+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,
+ );
+});
+
+test("positive rendered controlled target is open (rendered-area gate control)", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "pr1",
+ },
+ });
+ el({
+ id: "pr1",
+ tagName: "UL",
+ rect: { width: 240, height: 120, left: 40, top: 120 },
+ });
+ const result = trigger();
+ assert(
+ result !== null,
+ "a positive-area controlled target must count as open",
+ );
+ assert(
+ result.controlsResolved === 1,
+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,
+ );
+});
+
+// --- dangling vs absent aria-controls (a present-but-unresolved IDREF is a popup unmounted on
+// close, not a portal wired without aria-controls) ---
+
+test("dangling aria-controls (present IDREF, target unmounted) is closed", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "ghost1",
+ },
+ });
+ // no element with id ghost1 exists (unmounted on close)
+ assert(
+ trigger() === null,
+ "a present-but-dangling aria-controls must be treated as closed, not open",
+ );
+});
+
+test("dangling aria-owns (present IDREF, target unmounted) is closed", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ role: "combobox",
+ "aria-expanded": "true",
+ "aria-owns": "ghostOwned",
+ },
+ });
+ assert(
+ trigger() === null,
+ "a present-but-dangling aria-owns must be treated as closed",
+ );
+});
+
+test("absent aria-controls keeps portal fallback => open (controlsResolved=0)", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: { "aria-haspopup": "listbox", "aria-expanded": "true" },
+ });
+ const result = trigger();
+ assert(result !== null, "absent aria-controls => portal fallback => open");
+ assert(
+ result.controlsResolved === 0,
+ `controlsResolved should be 0 for the no-target fallback, got ${result && result.controlsResolved}`,
+ );
+});
+
+test("empty/whitespace aria-controls is treated as absent => open", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ role: "combobox",
+ "aria-expanded": "true",
+ "aria-controls": " ",
+ },
+ });
+ assert(
+ trigger() !== null,
+ "a whitespace-only aria-controls references no IDREF and must fall back to portal-open",
+ );
+});
+
+test("mixed dangling + resolved-visible target is open (any resolved-visible => open)", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ "aria-haspopup": "listbox",
+ "aria-expanded": "true",
+ "aria-controls": "ghostX visibleX",
+ },
+ });
+ el({ id: "visibleX", tagName: "UL" });
+ const result = trigger();
+ assert(result !== null, "a resolved-visible target among danglers => open");
+ assert(
+ result.controlsResolved === 1,
+ `controlsResolved should count only resolved targets (1), got ${result && result.controlsResolved}`,
+ );
+});
+
+// --- telemetry token bounding: role / aria-haspopup are page-controlled and must not reach the
+// return (and thus telemetry) verbatim; only the matched allowlisted token is emitted ---
+
+test("haspopup-qualified trigger with arbitrary role emits null role, matched hasPopup", () => {
+ el({
+ tagName: "BUTTON",
+ attributes: {
+ role: "'\"><script>arbitrary-attacker-role",
+ "aria-haspopup": "menu",
+ "aria-expanded": "true",
+ },
+ });
+ const result = trigger();
+ assert(result !== null, "haspopup=menu qualifies");
+ assert(
+ result.role === null,
+ `arbitrary role must not be emitted, got ${result && JSON.stringify(result.role)}`,
+ );
+ assert(
+ result.hasPopup === "menu",
+ `matched hasPopup token must be emitted, got ${result && result.hasPopup}`,
+ );
+});
+
+test("combobox-qualified trigger with arbitrary aria-haspopup emits null hasPopup, matched role", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ role: "combobox",
+ "aria-haspopup": "arbitrary-attacker-haspopup-value",
+ "aria-expanded": "true",
+ },
+ });
+ const result = trigger();
+ assert(result !== null, "role=combobox qualifies");
+ assert(
+ result.role === "combobox",
+ `matched role token must be emitted, got ${result && result.role}`,
+ );
+ assert(
+ result.hasPopup === null,
+ `non-allowlisted aria-haspopup must not be emitted, got ${result && JSON.stringify(result.hasPopup)}`,
+ );
+});
+
+test("both matched tokens are emitted (combobox + menu)", () => {
+ el({
+ tagName: "DIV",
+ attributes: {
+ role: "combobox",
+ "aria-haspopup": "menu",
+ "aria-expanded": "true",
+ },
+ });
+ const result = trigger();
+ assert(result !== null, "combobox + menu qualifies");
+ assert(
+ result.role === "combobox",
+ `role should be combobox, got ${result && result.role}`,
+ );
+ assert(
+ result.hasPopup === "menu",
+ `hasPopup should be menu, got ${result && result.hasPopup}`,
+ );
+});
+
+console.log(`\n${passed + failed} tests: ${passed} passed, ${failed} failed`);
+process.exit(failed > 0 ? 1 : 0);
diff --git a/tests/unit/test_dom_scrape_crash_guards.py b/tests/unit/test_dom_scrape_crash_guards.py
index 72442d778..5544e5730 100644
--- a/tests/unit/test_dom_scrape_crash_guards.py
+++ b/tests/unit/test_dom_scrape_crash_guards.py
@@ -49,6 +49,17 @@ class TestDomScrapeCrashGuards:
)
assert result.returncode == 0, f"Failed:\n{result.stdout}\n{result.stderr}"
+ def test_aria_popup_trigger_behavioral(self):
+ script = Path(__file__).parent / "test_aria_popup_trigger_domutils.js"
+ assert script.exists(), f"Missing {script}"
+ result = subprocess.run(
+ [_NODE, str(script)],
+ capture_output=True,
+ text=True,
+ timeout=30,
+ )
+ assert result.returncode == 0, f"Failed:\n{result.stdout}\n{result.stderr}"
+
def test_injection_scope_isolation(self, tmp_path):
from skyvern.webeye.utils.page import load_js_script
diff --git a/tests/unit/test_incompatible_element_guards.py b/tests/unit/test_incompatible_element_guards.py
index b12c9157a..cdf964d21 100644
--- a/tests/unit/test_incompatible_element_guards.py
+++ b/tests/unit/test_incompatible_element_guards.py
@@ -6,7 +6,12 @@ import pytest
from playwright.async_api import Error as PlaywrightError
from skyvern.exceptions import InvalidElementForTextInput
-from skyvern.webeye.actions.handler import get_input_value
+from skyvern.webeye.actions.handler import (
+ _is_selected_engine_error,
+ _is_selected_engine_timeout,
+ get_input_value,
+)
+from skyvern.webeye.browser_engine import BrowserEngineMetadata, BrowserEngineSelection
from skyvern.webeye.utils.dom import SkyvernElement, is_incompatible_text_input_error
_NOT_INPUT_ERROR = "Element is not an <input>, <textarea> or [contenteditable] element"
@@ -15,6 +20,54 @@ _NOT_INPUTELEMENT_ERROR = "Node is not an HTMLInputElement, HTMLTextAreaElement
_TIMEOUT_ERROR = "Timeout 30000ms exceeded"
+class _EngineError(Exception):
+ pass
+
+
+class _EngineTimeout(_EngineError):
+ pass
+
+
+async def _never_start(): # pragma: no cover - never awaited
+ raise AssertionError("start_driver must not be called")
+
+
+def _engine_selection() -> BrowserEngineSelection:
+ return BrowserEngineSelection(
+ name="engine-a",
+ start_driver=_never_start,
+ error_type=_EngineError,
+ timeout_error_type=_EngineTimeout,
+ metadata=BrowserEngineMetadata(name="engine-a", version="0.0.0"),
+ selection_reason="test",
+ )
+
+
+def test_is_selected_engine_timeout_falls_back_to_stock_playwright_when_no_selection() -> None:
+ from playwright.async_api import TimeoutError as PlaywrightTimeoutError
+
+ assert _is_selected_engine_timeout(PlaywrightTimeoutError("boom"), None) is True
+ assert _is_selected_engine_timeout(_EngineTimeout("boom"), None) is False
+
+
+def test_is_selected_engine_timeout_uses_selected_engine_identity() -> None:
+ from playwright.async_api import TimeoutError as PlaywrightTimeoutError
+
+ selection = _engine_selection()
+ assert _is_selected_engine_timeout(_EngineTimeout("boom"), selection) is True
+ # A foreign (stock Playwright) timeout is NOT this engine's timeout.
+ assert _is_selected_engine_timeout(PlaywrightTimeoutError("boom"), selection) is False
+
+
+def test_is_selected_engine_error_uses_selected_engine_identity() -> None:
+ selection = _engine_selection()
+ assert _is_selected_engine_error(_EngineError("boom"), selection) is True
+ assert _is_selected_engine_error(PlaywrightError("boom"), selection) is False
+ # No selection keeps the stock Playwright identity.
+ assert _is_selected_engine_error(PlaywrightError("boom"), None) is True
+ assert _is_selected_engine_error(_EngineError("boom"), None) is False
+
+
def _make_element(tag_name: str, locator: MagicMock) -> SkyvernElement:
return SkyvernElement(locator, MagicMock(), {"id": "el-1", "tagName": tag_name})
@@ -66,6 +119,33 @@ async def test_input_fill_reraises_unrelated_errors() -> None:
await element.input_fill("hello")
+async def test_get_input_value_translates_selected_engine_incompatible_read() -> None:
+ # Under a non-stock selected engine, an incompatible-node read raised as THAT engine's error
+ # must still be translated to "value unknown" (None), not escape raw.
+ locator = MagicMock()
+ locator.input_value = AsyncMock(side_effect=_EngineError(_NOT_INPUTELEMENT_ERROR))
+ assert await get_input_value("input", locator, engine_selection=_engine_selection()) is None
+
+
+async def test_get_input_value_reraises_foreign_error_under_selected_engine() -> None:
+ # A stock Playwright error is foreign to the pinned engine; it must propagate untranslated
+ # instead of being silently swallowed as an incompatible read.
+ locator = MagicMock()
+ locator.input_value = AsyncMock(side_effect=PlaywrightError(_NOT_INPUTELEMENT_ERROR))
+ with pytest.raises(PlaywrightError):
+ await get_input_value("input", locator, engine_selection=_engine_selection())
+
+
+async def test_get_input_value_reraises_selected_engine_non_incompatible_error() -> None:
+ locator = MagicMock()
+ locator.input_value = AsyncMock(side_effect=_EngineError(_TIMEOUT_ERROR))
+ with pytest.raises(_EngineError):
+ await get_input_value("input", locator, engine_selection=_engine_selection())
+
+
@pytest.mark.asyncio
async def test_input_clear_classifies_incompatible_element(monkeypatch: pytest.MonkeyPatch) -> None:
element = _make_element("button", MagicMock())
diff --git a/tests/unit/test_input_text_incremental_timeout_reraise.py b/tests/unit/test_input_text_incremental_timeout_reraise.py
index bfe20a940..b7f22122a 100644
--- a/tests/unit/test_input_text_incremental_timeout_reraise.py
+++ b/tests/unit/test_input_text_incremental_timeout_reraise.py
@@ -21,6 +21,7 @@ from skyvern.forge.sdk.models import StepStatus
from skyvern.webeye.actions.actions import InputOrSelectContext, InputTextAction
from skyvern.webeye.actions.handler import handle_input_text_action
from skyvern.webeye.actions.responses import ActionSuccess
+from skyvern.webeye.browser_engine import BrowserEngineMetadata, BrowserEngineSelection
from skyvern.webeye.scraper.scraper import IncrementalScrapePage
from tests.unit.conftest import make_input_element_mock
from tests.unit.helpers import make_organization, make_step, make_task
@@ -31,6 +32,29 @@ _TASK = make_task(_NOW, _ORG, navigation_payload={}, navigation_goal="Fill the f
_STEP = make_step(_NOW, _TASK, step_id="stp-1", status=StepStatus.created, order=0, output=None)
+class _EngineError(Exception):
+ pass
+
+
+class _EngineTimeout(_EngineError):
+ pass
+
+
+async def _never_start(): # pragma: no cover - never awaited
+ raise AssertionError("start_driver must not be called")
+
+
+def _engine_selection() -> BrowserEngineSelection:
+ return BrowserEngineSelection(
+ name="engine-a",
+ start_driver=_never_start,
+ error_type=_EngineError,
+ timeout_error_type=_EngineTimeout,
+ metadata=BrowserEngineMetadata(name="engine-a", version="0.0.0"),
+ selection_reason="test",
+ )
+
+
@pytest.mark.asyncio
async def test_incremental_element_tree_propagates_when_both_attempts_time_out() -> None:
"""Both the wait-until-finished attempt and the no-wait retry raising ``SkyvernPageAnalysisTimeout``
@@ -50,7 +74,9 @@ async def test_incremental_element_tree_propagates_when_both_attempts_time_out()
assert skyvern_frame.get_incremental_element_tree.await_count == 2
-async def _run_input_with_incremental_error(error: BaseException) -> list:
+async def _run_input_with_incremental_error(
+ error: BaseException, engine_selection: BrowserEngineSelection | None = None
+) -> list:
skyvern_el = make_input_element_mock(element_id="AADC")
dom_instance = MagicMock()
dom_instance.get_skyvern_element_by_id = AsyncMock(return_value=skyvern_el)
@@ -79,6 +105,10 @@ async def _run_input_with_incremental_error(error: BaseException) -> list:
return_value="123456",
),
patch("skyvern.webeye.actions.handler._get_input_or_select_context", new=AsyncMock(return_value=context)),
+ patch(
+ "skyvern.webeye.actions.handler.resolve_engine_selection_for_task",
+ return_value=engine_selection,
+ ),
):
return await handle_input_text_action(
action=action, page=MagicMock(), scraped_page=scraped_page, task=_TASK, step=_STEP
@@ -102,3 +132,36 @@ async def test_input_action_still_swallows_unrelated_incremental_error() -> None
semantic timeout only."""
results = await _run_input_with_incremental_error(RuntimeError("unexpected DOM state"))
assert len(results) == 1 and isinstance(results[0], ActionSuccess)
+
+
+async def test_input_action_tolerates_selected_engine_navigation_error() -> None:
+ """A pinned non-stock engine's navigation error during incremental processing is tolerated
+ exactly like the stock Playwright navigation error (silently continue, action succeeds)."""
+ results = await _run_input_with_incremental_error(
+ _EngineError("Execution context was destroyed"), engine_selection=_engine_selection()
+ )
+ assert len(results) == 1 and isinstance(results[0], ActionSuccess)
+
+
+async def test_input_action_reraises_selected_engine_non_navigation_error() -> None:
+ """A pinned engine's non-navigation error must be re-raised, mirroring the stock
+ Playwright-error contract, instead of being swallowed into an ActionSuccess."""
+ with pytest.raises(_EngineError):
+ await _run_input_with_incremental_error(
+ _EngineError("something genuinely unexpected"), engine_selection=_engine_selection()
+ )
+
+
[email protected]("message", ["Execution context was destroyed", "something genuinely unexpected"])
+async def test_input_action_reraises_foreign_error_under_selected_engine(message: str) -> None:
+ """Under a pinned non-stock engine, a stock Playwright error is foreign — it is not one of this
+ engine's tolerances, so it must propagate exactly as the pre-PR ``except PlaywrightError`` re-raised
+ it, never fall through to a false ``ActionSuccess``. This holds even for a navigation-shaped message:
+ the navigation tolerance belongs to the selected engine's own errors, not to a foreign driver."""
+ from playwright.async_api import Error as PlaywrightError
+
+ with pytest.raises(PlaywrightError):
+ await _run_input_with_incremental_error(PlaywrightError(message), engine_selection=_engine_selection())
diff --git a/tests/unit/test_input_text_wrapper_retarget.py b/tests/unit/test_input_text_wrapper_retarget.py
index 4c5babaf5..db2422fb6 100644
--- a/tests/unit/test_input_text_wrapper_retarget.py
+++ b/tests/unit/test_input_text_wrapper_retarget.py
@@ -92,7 +92,7 @@ async def _run(action: InputTextAction, wrapper: MagicMock, child: MagicMock | N
# successful retarget short-circuits to success, proving the child drove the pipeline.
read_tags: list[str] = []
- async def _get_input_value(tag_name: str, locator: object) -> str:
+ async def _get_input_value(tag_name: str, locator: object, engine_selection: object = None) -> str:
read_tags.append(tag_name)
return _CARD if tag_name == "input" else ""
diff --git a/tests/unit/test_record_artifacts_popup_preserving.py b/tests/unit/test_record_artifacts_popup_preserving.py
new file mode 100644
index 000000000..3bc12e69d
--- /dev/null
+++ b/tests/unit/test_record_artifacts_popup_preserving.py
@@ -0,0 +1,300 @@
+"""Post-action transient-UI capture gate.
+
+record_artifacts_after_action takes the LLM-facing SCREENSHOT_ACTION. Under the treatment arm
+of the PRESERVE_TRANSIENT_UI_CAPTURE experiment, an open ARIA popup forces scrolling_number=0
+so the scroll events of a multi-page capture do not dismiss the popup before the next step sees
+it. Control shadow-detects only (scrolling unchanged); off never detects. CUA keeps its existing
+scrolling_number=0 independently of the experiment arm.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from skyvern.constants import DEFAULT_MAX_SCREENSHOT_SCROLLS
+from skyvern.forge.agent import ForgeAgent
+from skyvern.forge.sdk.core import skyvern_context
+from skyvern.forge.sdk.core.skyvern_context import SkyvernContext
+from skyvern.schemas.run_enums import RunEngine
+
+
+async def _run(
+ *,
+ engine: RunEngine,
+ popup_trigger: dict | None,
+ arm_value: bool | None,
+ context: SkyvernContext | None = None,
+):
+ """Drive record_artifacts_after_action with stubbed IO, yielding (frame_mock, scrolling_number).
+ arm_value maps to the cached experiment assignment: True=treatment, False=control, None=off.
+ ``context`` lets a caller reuse one SkyvernContext across successive post-action captures to
+ exercise the shared per-run suppression cap."""
+ working_page = MagicMock()
+
+ frame_mock = MagicMock()
+ frame_mock.safe_wait_for_animation_end = AsyncMock()
+ frame_mock.get_scroll_x_y = AsyncMock(return_value=(0, 0))
+ frame_mock.safe_scroll_to_x_y = AsyncMock()
+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value=popup_trigger)
+ # Force the html-artifact block to fail fast so we don't need the artifact manager.
+ frame_mock.get_content = AsyncMock(side_effect=RuntimeError("stop after screenshot"))
+
+ browser_state = MagicMock()
+ browser_state.get_working_page = AsyncMock(return_value=working_page)
+ browser_state.take_post_action_screenshot = AsyncMock(side_effect=RuntimeError("stop"))
+
+ task = MagicMock()
+ task.workflow_run_id = "wr_test"
+ task.task_id = "tsk_test"
+ task.organization_id = "o_test"
+ step = MagicMock()
+ step.order = 1
+ step.is_speculative = False
+ action = MagicMock()
+ action.action_id = None
+ action.organization_id = None
+
+ skyvern_context.set(
+ context
+ or SkyvernContext(
+ max_screenshot_scrolls=None,
+ workflow_run_id="wr_test",
+ preserve_transient_ui_capture=arm_value,
+ )
+ )
+
+ try:
+ with contextlib.ExitStack() as stack:
+ skyvern_frame_cls = stack.enter_context(patch("skyvern.forge.agent.SkyvernFrame"))
+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)
+
+ agent = MagicMock(spec=ForgeAgent)
+ await ForgeAgent.record_artifacts_after_action(
+ agent, task=task, step=step, browser_state=browser_state, engine=engine, action=action
+ )
+ scrolling_number = browser_state.take_post_action_screenshot.await_args.kwargs["scrolling_number"]
+ yield frame_mock, scrolling_number
+ finally:
+ skyvern_context.reset()
+
+
+async def test_treatment_open_popup_forces_zero_scroll() -> None:
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == 0, "treatment + open popup must force scrolling_number=0"
+ frame_mock.get_open_aria_popup_trigger.assert_awaited()
+
+
+async def test_control_open_popup_shadow_detects_but_keeps_scroll() -> None:
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=False) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS, "control must not alter scrolling"
+ frame_mock.get_open_aria_popup_trigger.assert_awaited() # shadow detection ran
+
+
+async def test_off_open_popup_keeps_scroll_and_skips_predicate() -> None:
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=None) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited() # off pays no shadow cost
+
+
+async def test_treatment_no_popup_keeps_default_scroll() -> None:
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=None, arm_value=True) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS
+
+
+async def test_cua_engine_stays_zero_and_skips_predicate_regardless_of_arm() -> None:
+ async with _run(engine=RunEngine.openai_cua, popup_trigger={"role": "combobox"}, arm_value=True) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == 0
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()
+
+
+async def test_treatment_zero_scroll_still_reads_scroll_position_legacy() -> None:
+ # Legacy always snapshots the scroll position (regardless of scrolling_number); treatment must
+ # change ONLY the treatment decision (scrolling_number=0), not the legacy read/restore control
+ # flow. Restoring to the unchanged position is a no-op window.scroll, so it does not scroll.
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == 0
+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read preserved
+
+
+async def test_scrolling_capture_snapshots_scroll_position() -> None:
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=None, arm_value=True) as (
+ frame_mock,
+ scrolling_number,
+ ):
+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS
+ frame_mock.get_scroll_x_y.assert_awaited_once()
+
+
+async def test_treatment_zero_scroll_records_screenshot_action_via_legacy_nesting() -> None:
+ # Suppressing the scroll (scrolling_number=0) still records the LLM-facing SCREENSHOT_ACTION,
+ # via the LEGACY nesting: get_scroll_x_y succeeds (0, 0) so x/y are known and the recording runs.
+ from skyvern.forge import app
+
+ frame_mock = MagicMock()
+ frame_mock.safe_wait_for_animation_end = AsyncMock()
+ frame_mock.get_scroll_x_y = AsyncMock(return_value=(0, 0))
+ frame_mock.safe_scroll_to_x_y = AsyncMock()
+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value={"role": "combobox"})
+ frame_mock.get_content = AsyncMock(side_effect=RuntimeError("stop after screenshot"))
+
+ browser_state = MagicMock()
+ browser_state.get_working_page = AsyncMock(return_value=MagicMock())
+ browser_state.take_post_action_screenshot = AsyncMock(return_value=b"img")
+
+ task = MagicMock()
+ task.workflow_run_id = "wr_test"
+ task.task_id = "tsk_test"
+ task.organization_id = "o_test"
+ step = MagicMock()
+ step.order = 1
+ step.is_speculative = False
+ action = MagicMock()
+ action.action_id = None
+ action.organization_id = None
+
+ skyvern_context.set(
+ SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_test", preserve_transient_ui_capture=True)
+ )
+ try:
+ with contextlib.ExitStack() as stack:
+ skyvern_frame_cls = stack.enter_context(patch("skyvern.forge.agent.SkyvernFrame"))
+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)
+ prepare = AsyncMock(return_value=None)
+ stack.enter_context(patch.object(app.ARTIFACT_MANAGER, "prepare_llm_artifact", prepare))
+
+ agent = MagicMock(spec=ForgeAgent)
+ await ForgeAgent.record_artifacts_after_action(
+ agent, task=task, step=step, browser_state=browser_state, engine=RunEngine.skyvern_v1, action=action
+ )
+ assert browser_state.take_post_action_screenshot.await_args.kwargs["scrolling_number"] == 0
+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read
+ frame_mock.safe_scroll_to_x_y.assert_awaited() # legacy restore (no-op to same position)
+ prepare.assert_awaited() # artifact recorded via legacy nesting (x/y known)
+ finally:
+ skyvern_context.reset()
+
+
+async def test_off_scroll_fetch_failure_drops_screenshot_action_legacy() -> None:
+ # OFF must follow EXACT legacy artifact semantics: legacy nested the SCREENSHOT_ACTION recording
+ # inside `if x is not None and y is not None`, so a failing get_scroll_x_y drops the screenshot.
+ # (Decoupling the recording from the scroll snapshot is an unrelated hardening, deferred out of
+ # this PR so OFF keeps legacy control flow.)
+ from skyvern.forge import app
+
+ frame_mock = MagicMock()
+ frame_mock.safe_wait_for_animation_end = AsyncMock()
+ frame_mock.get_scroll_x_y = AsyncMock(side_effect=RuntimeError("scroll read failed"))
+ frame_mock.safe_scroll_to_x_y = AsyncMock()
+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value=None)
+ frame_mock.get_content = AsyncMock(side_effect=RuntimeError("stop after screenshot"))
+
+ browser_state = MagicMock()
+ browser_state.get_working_page = AsyncMock(return_value=MagicMock())
+ browser_state.take_post_action_screenshot = AsyncMock(return_value=b"img")
+
+ task = MagicMock()
+ task.workflow_run_id = "wr_test"
+ task.task_id = "tsk_test"
+ task.organization_id = "o_test"
+ step = MagicMock()
+ step.order = 1
+ step.is_speculative = False
+ action = MagicMock()
+ action.action_id = None
+ action.organization_id = None
+
+ skyvern_context.set(
+ SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_test", preserve_transient_ui_capture=None)
+ )
+ try:
+ with contextlib.ExitStack() as stack:
+ skyvern_frame_cls = stack.enter_context(patch("skyvern.forge.agent.SkyvernFrame"))
+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)
+ prepare = AsyncMock(return_value=None)
+ stack.enter_context(patch.object(app.ARTIFACT_MANAGER, "prepare_llm_artifact", prepare))
+
+ agent = MagicMock(spec=ForgeAgent)
+ await ForgeAgent.record_artifacts_after_action(
+ agent, task=task, step=step, browser_state=browser_state, engine=RunEngine.skyvern_v1, action=action
+ )
+ assert browser_state.take_post_action_screenshot.await_args.kwargs["scrolling_number"] > 0
+ frame_mock.get_scroll_x_y.assert_awaited() # fetch was attempted (and raised)
+ frame_mock.safe_scroll_to_x_y.assert_not_awaited() # no restore (x/y unknown)
+ prepare.assert_not_awaited() # legacy: recording is nested under x/y-not-None, so it is dropped
+ finally:
+ skyvern_context.reset()
+
+
+# --- The post-action capture is the sibling of the agent-step scrape and must honor the same two
+# invariants: bounded telemetry tokens, and the shared per-run consecutive-suppression cap. ---
+
+
+async def test_post_action_routes_telemetry_through_bounded_helper() -> None:
+ # The post-action site must emit popup telemetry only through the shared, bounded
+ # emit_transient_ui_popup_telemetry helper (allowlisted tokens + bounded boolean), never by
+ # writing raw page-controlled role/aria-haspopup to the span. The helper itself is unit-tested
+ # in test_transient_ui_capture_experiment.py; here we assert the wiring.
+ trigger = {"role": "<script>evil", "hasPopup": "arbitrary-attacker-text", "controlsResolved": 0}
+ with patch("skyvern.forge.agent.emit_transient_ui_popup_telemetry") as emit:
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=trigger, arm_value=True):
+ pass
+ emit.assert_called_once()
+ assert emit.call_args.args[1] is trigger
+
+
+async def test_post_action_participates_in_shared_consecutive_cap() -> None:
+ ctx = SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_cap", preserve_transient_ui_capture=True)
+ scrolls = []
+ for _ in range(3):
+ async with _run(
+ engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True, context=ctx
+ ) as (_f, scrolling_number):
+ scrolls.append(scrolling_number)
+ assert scrolls[0] == 0 and scrolls[1] == 0, "first two post-action captures suppress"
+ assert scrolls[2] == DEFAULT_MAX_SCREENSHOT_SCROLLS, "third consecutive capture falls back to legacy scrolling"
+ assert ctx.transient_ui_consecutive_suppressions == 2
+
+
+async def test_post_action_no_popup_resets_shared_counter() -> None:
+ ctx = SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_reset", preserve_transient_ui_capture=True)
+ for _ in range(2):
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True, context=ctx):
+ pass
+ assert ctx.transient_ui_consecutive_suppressions == 2
+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=None, arm_value=True, context=ctx):
+ pass
+ assert ctx.transient_ui_consecutive_suppressions == 0
diff --git a/tests/unit/test_scrape_popup_preserving_scroll.py b/tests/unit/test_scrape_popup_preserving_scroll.py
new file mode 100644
index 000000000..2c84da429
--- /dev/null
+++ b/tests/unit/test_scrape_popup_preserving_scroll.py
@@ -0,0 +1,345 @@
+"""Scrape-site transient-UI capture gate.
+
+Under the treatment arm of PRESERVE_TRANSIENT_UI_CAPTURE, when an ARIA popup (e.g. a portal
+date-picker popup) is open, the next scrape's split screenshot must NOT scroll — scrolling
+fires events that dismiss the portal popup before the just-built element tree's day-cell IDs
+can be acted on. Control shadow-detects only (scrolls as before); off never detects.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from skyvern.forge.agent import ForgeAgent, ScrapeType
+from skyvern.forge.sdk.core import skyvern_context
+from skyvern.forge.sdk.core.skyvern_context import SkyvernContext
+from skyvern.webeye.scraper import scraper
+
+
+def _make_page() -> MagicMock:
+ page = MagicMock()
+ page.url = "https://example.com"
+ page.main_frame.url = "https://example.com"
+ page.main_frame.child_frames = []
+ page.viewport_size = {"width": 1280, "height": 800}
+ return page
+
+
+_UNSET = object()
+
+
+def _span_capture():
+ """Patch traced_span to yield a MagicMock span so tests can inspect emitted attributes."""
+ span = MagicMock()
+
+ @contextlib.contextmanager
+ def _fake_traced_span(_tracer, _name):
+ yield span
+
+ with patch.object(scraper, "traced_span", new=_fake_traced_span):
+ yield span
+
+
+def _span_attrs(span: MagicMock) -> dict:
+ return {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}
+
+
+async def _run_scrape(
+ *,
+ popup_trigger: dict | None,
+ arm_value: bool | None,
+ scroll: bool = True,
+ allow_transient_ui_suppression: object = True,
+ context: SkyvernContext | None = None,
+):
+ """Drive scrape_web_unsafe with the heavy dependencies stubbed, yielding
+ (take_split_screenshots_mock, frame_mock, span_mock). arm_value maps to the cached experiment
+ assignment: True=treatment, False=control, None=off.
+
+ allow_transient_ui_suppression is forwarded to scrape_web_unsafe unless left as ``_UNSET``, which
+ simulates a caller (verify/extract/error path) that never opts into suppression. ``context`` lets a
+ caller reuse one SkyvernContext across successive scrapes to exercise the per-run suppression cap.
+ """
+ page = _make_page()
+ browser_state = MagicMock()
+ browser_state.must_get_working_page = AsyncMock(return_value=page)
+
+ frame_mock = MagicMock()
+ frame_mock.get_scroll_x_y = AsyncMock(return_value=(0, 0))
+ frame_mock.safe_scroll_to_x_y = AsyncMock()
+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value=popup_trigger)
+ frame_mock.get_content = AsyncMock(return_value="<html></html>")
+
+ element = {"id": "btn", "tagName": "button"}
+
+ ctx = context or SkyvernContext(workflow_run_id="wr_test", preserve_transient_ui_capture=arm_value)
+ skyvern_context.set(ctx)
+ try:
+ with contextlib.ExitStack() as stack:
+ skyvern_frame_cls = stack.enter_context(patch.object(scraper, "SkyvernFrame"))
+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)
+ take_split = AsyncMock(return_value=[b"img"])
+ skyvern_frame_cls.take_split_screenshots = take_split
+
+ stack.enter_context(patch.object(scraper, "_wait_for_scrape_ready", new=AsyncMock()))
+ stack.enter_context(
+ patch.object(
+ scraper,
+ "get_interactable_element_tree",
+ new=AsyncMock(return_value=([element], [element], [])),
+ )
+ )
+ stack.enter_context(patch.object(scraper, "trim_element_tree", new=MagicMock(return_value=[])))
+ stack.enter_context(
+ patch.object(scraper, "build_element_dict", new=MagicMock(return_value=({}, {}, {}, {}, {})))
+ )
+ stack.enter_context(patch.object(scraper, "get_frame_text", new=AsyncMock(return_value="")))
+ stack.enter_context(patch.object(scraper, "advance_observation_epoch", new=MagicMock()))
+ stack.enter_context(patch.object(scraper, "_record_scrape_span_attrs", new=MagicMock()))
+ span = stack.enter_context(_span_capture())
+
+ kwargs: dict = dict(
+ browser_state=browser_state,
+ url="https://example.com",
+ cleanup_element_tree=AsyncMock(return_value=[element]),
+ scroll=scroll,
+ )
+ if allow_transient_ui_suppression is not _UNSET:
+ kwargs["allow_transient_ui_suppression"] = allow_transient_ui_suppression
+ await scraper.scrape_web_unsafe(**kwargs)
+ yield take_split, frame_mock, span
+ finally:
+ skyvern_context.reset()
+
+
+async def test_treatment_open_popup_suppresses_only_the_scroll_argument() -> None:
+ # Treatment differs from legacy ONLY in the scroll argument to take_split_screenshots. The
+ # scroll-position read + restore stay exactly as legacy (always-attempt); restoring to the
+ # unchanged position is a no-op window.scroll that does not dismiss the popup.
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True) as (take_split, frame_mock, _span):
+ assert take_split.await_args.kwargs["scroll"] is False, "treatment + open popup must force scroll=False"
+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read preserved
+ frame_mock.safe_scroll_to_x_y.assert_awaited() # legacy restore preserved
+
+
+async def test_off_scroll_false_preserves_legacy_read_and_restore() -> None:
+ # OFF must follow EXACT legacy control flow: legacy read the scroll position and restored it
+ # unconditionally, even when scroll=False. The current effective_scroll gate skips the read
+ # (RED); OFF must keep the legacy always-attempt read + restore.
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=None, scroll=False) as (
+ take_split,
+ frame_mock,
+ _span,
+ ):
+ assert take_split.await_args.kwargs["scroll"] is False
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited() # OFF never detects
+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read
+ frame_mock.safe_scroll_to_x_y.assert_awaited() # legacy restore
+
+
+async def test_treatment_no_popup_scrolls_and_restores_as_before() -> None:
+ async with _run_scrape(popup_trigger=None, arm_value=True) as (take_split, frame_mock, _span):
+ assert take_split.await_args.kwargs["scroll"] is True
+ frame_mock.get_open_aria_popup_trigger.assert_awaited()
+ frame_mock.get_scroll_x_y.assert_awaited()
+ frame_mock.safe_scroll_to_x_y.assert_awaited()
+
+
+async def test_control_open_popup_shadow_detects_but_keeps_scroll() -> None:
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=False) as (take_split, frame_mock, _span):
+ assert take_split.await_args.kwargs["scroll"] is True, "control must not alter scrolling"
+ frame_mock.get_open_aria_popup_trigger.assert_awaited() # shadow detection ran
+
+
+async def test_off_open_popup_scrolls_and_skips_predicate() -> None:
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=None) as (take_split, frame_mock, _span):
+ assert take_split.await_args.kwargs["scroll"] is True, "off must preserve status-quo scrolling"
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()
+
+
+async def test_caller_scroll_false_short_circuits_predicate() -> None:
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, scroll=False) as (
+ take_split,
+ frame_mock,
+ _span,
+ ):
+ assert take_split.await_args.kwargs["scroll"] is False
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()
+
+
+# --- MUST-FIX 3: suppression is opt-in per call. A caller that does not opt in (verify /
+# extraction / error-detection scrapes) keeps legacy scrolling even under treatment + open popup. ---
+
+
+async def test_no_optin_caller_keeps_legacy_scroll_and_skips_detection() -> None:
+ async with _run_scrape(
+ popup_trigger={"role": "combobox"}, arm_value=True, allow_transient_ui_suppression=_UNSET
+ ) as (take_split, frame_mock, _span):
+ assert take_split.await_args.kwargs["scroll"] is True, "non-opted-in scrape must stay legacy"
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited(), "no detection off the agent-step path"
+
+
+async def test_explicit_disallow_keeps_legacy_scroll() -> None:
+ async with _run_scrape(
+ popup_trigger={"role": "combobox"}, arm_value=True, allow_transient_ui_suppression=False
+ ) as (take_split, frame_mock, _span):
+ assert take_split.await_args.kwargs["scroll"] is True
+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()
+
+
+async def test_optin_agent_step_suppresses() -> None:
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, allow_transient_ui_suppression=True) as (
+ take_split,
+ frame_mock,
+ _span,
+ ):
+ assert take_split.await_args.kwargs["scroll"] is False, "opted-in agent-step scrape suppresses"
+
+
+# --- MUST-FIX 2: at most two consecutive suppressing captures per run, then fall back to legacy
+# scrolling; reset the counter when no qualifying popup is detected. ---
+
+
+async def test_consecutive_suppression_capped_at_two_then_legacy() -> None:
+ ctx = SkyvernContext(workflow_run_id="wr_cap", preserve_transient_ui_capture=True)
+ scroll_args = []
+ for _ in range(3):
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, context=ctx) as (
+ take_split,
+ _frame,
+ _span,
+ ):
+ scroll_args.append(take_split.await_args.kwargs["scroll"])
+ assert scroll_args == [False, False, True], f"expected suppress, suppress, fall back; got {scroll_args}"
+ assert ctx.transient_ui_consecutive_suppressions == 2
+
+
+async def test_no_popup_resets_consecutive_counter() -> None:
+ ctx = SkyvernContext(workflow_run_id="wr_reset", preserve_transient_ui_capture=True)
+ # burn the cap
+ for _ in range(2):
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, context=ctx):
+ pass
+ assert ctx.transient_ui_consecutive_suppressions == 2
+ # a scrape with no qualifying popup resets the counter
+ async with _run_scrape(popup_trigger=None, arm_value=True, context=ctx):
+ pass
+ assert ctx.transient_ui_consecutive_suppressions == 0
+ # ...so the next open popup suppresses again
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, context=ctx) as (
+ take_split,
+ _frame,
+ _span,
+ ):
+ assert take_split.await_args.kwargs["scroll"] is False
+
+
+async def test_off_arm_never_touches_counter() -> None:
+ ctx = SkyvernContext(workflow_run_id="wr_off", preserve_transient_ui_capture=None)
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=None, context=ctx) as (
+ take_split,
+ _frame,
+ _span,
+ ):
+ assert take_split.await_args.kwargs["scroll"] is True
+ assert ctx.transient_ui_consecutive_suppressions == 0
+
+
+async def test_control_arm_never_suppresses_and_counter_stays_zero() -> None:
+ ctx = SkyvernContext(workflow_run_id="wr_ctrl", preserve_transient_ui_capture=False)
+ for _ in range(3):
+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=False, context=ctx) as (
+ take_split,
+ _frame,
+ _span,
+ ):
+ assert take_split.await_args.kwargs["scroll"] is True
+ assert ctx.transient_ui_consecutive_suppressions == 0
+
+
+# --- MUST-FIX 4: page-controlled role / aria-haspopup must not reach the span verbatim. Only
+# allowlisted matched tokens are emitted. ---
+
+
+async def test_arbitrary_role_and_haspopup_not_emitted_to_span() -> None:
+ trigger = {"role": "<script>evil", "hasPopup": "arbitrary-attacker-text", "controlsResolved": 0}
+ async with _run_scrape(popup_trigger=trigger, arm_value=True) as (_take_split, _frame, span):
+ attrs = _span_attrs(span)
+ assert attrs.get("transient_ui_role") != "<script>evil"
+ assert attrs.get("transient_ui_haspopup") != "arbitrary-attacker-text"
+ assert "transient_ui_role" not in attrs, "non-allowlisted role must be omitted"
+ assert "transient_ui_haspopup" not in attrs, "non-allowlisted haspopup must be omitted"
+
+
+async def test_allowlisted_role_and_haspopup_are_emitted() -> None:
+ trigger = {"role": "combobox", "hasPopup": "menu", "controlsResolved": 1}
+ async with _run_scrape(popup_trigger=trigger, arm_value=True) as (_take_split, _frame, span):
+ attrs = _span_attrs(span)
+ assert attrs.get("transient_ui_role") == "combobox"
+ assert attrs.get("transient_ui_haspopup") == "menu"
+
+
+# --- Optional 5: bounded boolean controlsResolved telemetry so the false-positive source
+# (no-target portal fallback) is measurable. ---
+
+
+async def test_controls_resolved_boolean_emitted_true_and_false() -> None:
+ resolved = {"role": "combobox", "hasPopup": "menu", "controlsResolved": 2}
+ async with _run_scrape(popup_trigger=resolved, arm_value=True) as (_take_split, _frame, span):
+ assert _span_attrs(span).get("transient_ui_controls_resolved") is True
+ fallback = {"role": "combobox", "hasPopup": None, "controlsResolved": 0}
+ async with _run_scrape(popup_trigger=fallback, arm_value=True) as (_take_split, _frame, span):
+ assert _span_attrs(span).get("transient_ui_controls_resolved") is False
+
+
+# --- MUST-FIX 3 call-site wiring: the agent-step scrape (ForgeAgent._scrape_with_type) is the only
+# call site that opts into suppression. verify / extraction / error-detection paths call
+# scrape_website without the flag, so they inherit the legacy default (proven above). ---
+
+
+async def test_agent_step_scrape_opts_into_suppression() -> None:
+ browser_state = MagicMock()
+ browser_state.scrape_website = AsyncMock(return_value=MagicMock())
+
+ task = MagicMock()
+ task.url = "https://example.com"
+ step = MagicMock()
+
+ fake_app = MagicMock()
+ fake_app.AGENT_FUNCTION.cleanup_element_tree_factory = MagicMock(return_value=AsyncMock())
+ fake_app.scrape_exclude = AsyncMock()
+
+ with patch("skyvern.forge.agent.app", fake_app):
+ # unbound call with a mock self: _scrape_with_type only touches module-level app/settings.
+ await ForgeAgent._scrape_with_type(
+ MagicMock(),
+ task=task,
+ step=step,
+ browser_state=browser_state,
+ scrape_type=ScrapeType.NORMAL,
+ engine=MagicMock(), # not a CUA engine
+ )
+
+ assert browser_state.scrape_website.await_args.kwargs["allow_transient_ui_suppression"] is True
diff --git a/tests/unit/test_transient_ui_capture_experiment.py b/tests/unit/test_transient_ui_capture_experiment.py
new file mode 100644
index 000000000..4067f688f
--- /dev/null
+++ b/tests/unit/test_transient_ui_capture_experiment.py
@@ -0,0 +1,298 @@
+"""Per-run resolution of the PRESERVE_TRANSIENT_UI_CAPTURE experiment.
+
+Off/unenrolled is the safe default: the flag only assigns treatment when a real provider resolves
+True. Undefined, no-provider, resolver error, and missing run identity all stay off/unenrolled
+(the predicate is not evaluated and scrolling is unchanged). An explicit provider False is the
+distinct enrolled control arm (shadow-detect only, still scrolls).
+"""
+
+from __future__ import annotations
+
+import asyncio
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+from skyvern.forge.sdk.core.skyvern_context import SkyvernContext
+from skyvern.forge.sdk.experimentation import transient_ui_capture as tuc
+from skyvern.forge.sdk.experimentation.providers import NoOpExperimentationProvider
+
+
+def _ctx(**kwargs: object) -> SkyvernContext:
+ return SkyvernContext(**kwargs) # type: ignore[arg-type]
+
+
+async def test_provider_true_caches_treatment(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1", organization_id="o_1", workflow_permanent_id="wpid_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert ctx.preserve_transient_ui_capture is True
+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"
+ provider.resolve_feature_flag_cached.assert_awaited_once()
+ # run-level distinct_id, org/wpid as properties (per backend.md convention)
+ call = provider.resolve_feature_flag_cached.await_args
+ assert call.args[0] == tuc.PRESERVE_TRANSIENT_UI_CAPTURE_FLAG
+ assert call.args[1] == "wr_1"
+ assert call.kwargs["properties"]["organization_id"] == "o_1"
+
+
+async def test_provider_false_caches_control(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=False)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert ctx.preserve_transient_ui_capture is False
+ assert tuc.transient_ui_capture_arm(ctx) == "control"
+
+
+async def test_provider_undefined_none_is_off(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=None)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert ctx.preserve_transient_ui_capture is None
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+
+
+async def test_noop_provider_is_off_without_querying_provider(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = NoOpExperimentationProvider()
+ spy = AsyncMock(return_value=True)
+ monkeypatch.setattr(provider, "resolve_feature_flag_cached", spy)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ spy.assert_not_awaited()
+
+
+async def test_provider_error_is_off_and_does_not_raise(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(side_effect=RuntimeError("posthog down"))
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+
+
+async def test_missing_run_identity_is_off_without_querying_provider(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx()
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ provider.resolve_feature_flag_cached.assert_not_awaited()
+
+
+def test_arm_helper_missing_context_is_off() -> None:
+ assert tuc.transient_ui_capture_arm(None) == "off"
+
+
+def test_arm_helper_uninitialized_field_is_off() -> None:
+ assert tuc.transient_ui_capture_arm(_ctx(workflow_run_id="wr_1")) == "off"
+
+
+async def test_treatment_arm_is_sticky_across_provider_change(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"
+ # TTL expiry / mid-run ramp flips the provider result — the run's arm must NOT change.
+ provider.resolve_feature_flag_cached.return_value = False
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"
+ provider.resolve_feature_flag_cached.assert_awaited_once()
+
+
+async def test_control_arm_is_sticky_across_provider_change(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=False)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "control"
+ provider.resolve_feature_flag_cached.return_value = True
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "control"
+ provider.resolve_feature_flag_cached.assert_awaited_once()
+
+
+async def test_off_undefined_arm_is_sticky_across_provider_change(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=None)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ provider.resolve_feature_flag_cached.return_value = True
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ provider.resolve_feature_flag_cached.assert_awaited_once()
+
+
+async def test_resolver_error_arm_is_sticky_off(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(side_effect=RuntimeError("posthog down"))
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ # Provider recovers and would now return treatment — the pinned off must not flip.
+ recovered = AsyncMock(return_value=True)
+ monkeypatch.setattr(provider, "resolve_feature_flag_cached", recovered)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ recovered.assert_not_awaited()
+
+
+async def test_missing_identity_arm_is_sticky_off(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx() # no run identity
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ provider.resolve_feature_flag_cached.assert_not_awaited()
+
+
+async def test_noop_provider_is_sticky_off_without_calls(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_1")
+ provider = NoOpExperimentationProvider()
+ spy = AsyncMock(return_value=True)
+ monkeypatch.setattr(provider, "resolve_feature_flag_cached", spy)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "off"
+ spy.assert_not_awaited()
+
+
+async def test_explicit_distinct_id_override_used_over_context(monkeypatch: pytest.MonkeyPatch) -> None:
+ # An execution-boundary caller (e.g. an inline child workflow) whose context lacks full run
+ # identity may pass distinct_id / organization_id / workflow_permanent_id explicitly.
+ ctx = _ctx(run_id="parent_run", organization_id="o_ctx")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(
+ ctx, distinct_id="wr_child", organization_id="o_child", workflow_permanent_id="wpid_child"
+ )
+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"
+ call = provider.resolve_feature_flag_cached.await_args
+ assert call.args[1] == "wr_child" # explicit distinct_id, NOT the context's parent run_id
+ assert call.kwargs["properties"]["organization_id"] == "o_child"
+ assert call.kwargs["properties"]["workflow_permanent_id"] == "wpid_child"
+
+
+async def test_override_falls_back_to_context_when_not_given(monkeypatch: pytest.MonkeyPatch) -> None:
+ ctx = _ctx(workflow_run_id="wr_ctx", organization_id="o_ctx")
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(return_value=False)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+ await tuc.resolve_transient_ui_capture_arm(ctx)
+ assert tuc.transient_ui_capture_arm(ctx) == "control"
+ assert provider.resolve_feature_flag_cached.await_args.args[1] == "wr_ctx"
+
+
+async def test_concurrent_first_resolve_queries_provider_once(monkeypatch: pytest.MonkeyPatch) -> None:
+ # Two coroutines sharing one SkyvernContext both hit the first-resolve path. A single-flight lock
+ # must keep the provider queried exactly once (the ADR's "provider ≤ once/run" invariant).
+ ctx = _ctx(workflow_run_id="wr_1")
+ calls = {"n": 0}
+
+ async def _slow_resolve(*_a: object, **_k: object) -> bool:
+ calls["n"] += 1
+ await asyncio.sleep(0) # yield so a concurrent first-resolver can interleave before we pin
+ return True
+
+ provider = MagicMock()
+ provider.resolve_feature_flag_cached = AsyncMock(side_effect=_slow_resolve)
+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)
+
+ await asyncio.gather(
+ tuc.resolve_transient_ui_capture_arm(ctx),
+ tuc.resolve_transient_ui_capture_arm(ctx),
+ )
+
+ assert calls["n"] == 1, "two concurrent first-resolvers on one context must query the provider once"
+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"
+
+
+# --- Shared capture helpers used by BOTH the agent-step scrape and the post-action screenshot ---
+
+
+def test_emit_telemetry_omits_arbitrary_page_controlled_tokens() -> None:
+ span = MagicMock()
+ tuc.emit_transient_ui_popup_telemetry(
+ span, {"role": "<script>evil", "hasPopup": "arbitrary-attacker-text", "controlsResolved": 0}
+ )
+ keys = {call.args[0] for call in span.set_attribute.call_args_list}
+ assert "transient_ui_role" not in keys, "non-allowlisted role must be omitted"
+ assert "transient_ui_haspopup" not in keys, "non-allowlisted haspopup must be omitted"
+
+
+def test_emit_telemetry_emits_allowlisted_tokens_and_controls_boolean() -> None:
+ span = MagicMock()
+ tuc.emit_transient_ui_popup_telemetry(span, {"role": "combobox", "hasPopup": "menu", "controlsResolved": 2})
+ attrs = {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}
+ assert attrs["transient_ui_role"] == "combobox"
+ assert attrs["transient_ui_haspopup"] == "menu"
+ assert attrs["transient_ui_controls_resolved"] is True
+
+
+def test_emit_telemetry_controls_boolean_false_for_no_target_fallback() -> None:
+ span = MagicMock()
+ tuc.emit_transient_ui_popup_telemetry(span, {"role": "combobox", "hasPopup": None, "controlsResolved": 0})
+ attrs = {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}
+ assert attrs["transient_ui_controls_resolved"] is False
+
+
+def test_decide_suppression_treatment_caps_at_two_then_falls_back() -> None:
+ ctx = _ctx(workflow_run_id="wr_cap", preserve_transient_ui_capture=True)
+ d1 = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)
+ d2 = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)
+ d3 = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)
+ assert (d1.suppress, d2.suppress, d3.suppress) == (True, True, False)
+ assert d3.capped is True
+ assert ctx.transient_ui_consecutive_suppressions == 2
+
+
+def test_decide_suppression_resets_when_not_detected() -> None:
+ ctx = _ctx(workflow_run_id="wr_reset", preserve_transient_ui_capture=True)
+ tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)
+ tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)
+ assert ctx.transient_ui_consecutive_suppressions == 2
+ d = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=False)
+ assert d.suppress is False and d.capped is False
+ assert ctx.transient_ui_consecutive_suppressions == 0
+
+
+def test_decide_suppression_control_and_off_never_suppress_or_touch_counter() -> None:
+ for arm in (tuc.ARM_CONTROL, tuc.ARM_OFF):
+ ctx = _ctx(workflow_run_id="wr_arm", preserve_transient_ui_capture=False)
+ d = tuc.decide_transient_ui_suppression(ctx, arm, detected=True)
+ assert d.suppress is False and d.capped is False
+ assert ctx.transient_ui_consecutive_suppressions == 0
diff --git a/tests/unit/test_upload_file_wait.py b/tests/unit/test_upload_file_wait.py
index 292ac4c4a..9fc46e746 100644
--- a/tests/unit/test_upload_file_wait.py
+++ b/tests/unit/test_upload_file_wait.py
@@ -15,6 +15,39 @@ from playwright.async_api import TimeoutError as PlaywrightTimeoutError
from skyvern.exceptions import SkyvernPageAnalysisTimeout
from skyvern.webeye.actions import handler as handler_module
from skyvern.webeye.actions.handler import _wait_for_upload_processing
+from skyvern.webeye.browser_engine import BrowserEngineMetadata, BrowserEngineSelection
+
+
+class _EngineError(Exception):
+ pass
+
+
+class _EngineTimeout(_EngineError):
+ pass
+
+
+async def _never_start(): # pragma: no cover - never awaited
+ raise AssertionError("start_driver must not be called")
+
+
+def _engine_selection() -> BrowserEngineSelection:
+ return BrowserEngineSelection(
+ name="engine-a",
+ start_driver=_never_start,
+ error_type=_EngineError,
+ timeout_error_type=_EngineTimeout,
+ metadata=BrowserEngineMetadata(name="engine-a", version="0.0.0"),
+ selection_reason="test",
+ )
+
+
+async def _run_with_page_ready_error(error: BaseException, engine_selection: BrowserEngineSelection | None) -> None:
+ mock_frame = AsyncMock()
+ mock_frame.wait_for_page_ready.side_effect = error
+ with patch("skyvern.webeye.actions.handler.SkyvernFrame.create_instance", new_callable=AsyncMock) as mock_create:
+ mock_create.return_value = mock_frame
+ await _wait_for_upload_processing(AsyncMock(), engine_selection=engine_selection)
+
# ---------------------------------------------------------------------------
# Helper behaviour tests
@@ -88,6 +121,31 @@ async def test_propagates_non_playwright_error() -> None:
await _wait_for_upload_processing(AsyncMock())
+async def test_swallows_selected_engine_timeout() -> None:
+ # A non-stock engine's native timeout must be tolerated like the stock Playwright timeout.
+ await _run_with_page_ready_error(_EngineTimeout("deadline exceeded"), _engine_selection())
+
+
+async def test_swallows_selected_engine_error() -> None:
+ await _run_with_page_ready_error(_EngineError("target closed"), _engine_selection())
+
+
+async def test_propagates_foreign_playwright_error_under_selected_engine() -> None:
+ # Under a pinned non-stock engine, a stock Playwright error is foreign and must propagate,
+ # not be swallowed as an upload-processing tolerance.
+ with pytest.raises(PlaywrightError):
+ await _run_with_page_ready_error(PlaywrightError("navigated away"), _engine_selection())
+
+
+async def test_propagates_foreign_playwright_timeout_under_selected_engine() -> None:
+ with pytest.raises(PlaywrightTimeoutError):
+ await _run_with_page_ready_error(PlaywrightTimeoutError("pw timeout"), _engine_selection())
+
+
# ---------------------------------------------------------------------------
# Static regression guards
# ---------------------------------------------------------------------------
diff --git a/tests/unit/webeye/test_skyvern_frame_aria_popup.py b/tests/unit/webeye/test_skyvern_frame_aria_popup.py
new file mode 100644
index 000000000..d07f75dc3
--- /dev/null
+++ b/tests/unit/webeye/test_skyvern_frame_aria_popup.py
@@ -0,0 +1,51 @@
+"""SkyvernFrame.get_open_aria_popup_trigger seam.
+
+The Python wrapper evaluates the JS predicate through the frame evaluation seam and
+MUST fail open (return None) on any evaluation error so screenshot-scroll policy keeps
+the current scrolling behavior instead of crashing the scrape/action loop.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from skyvern.webeye.utils.page import SkyvernFrame
+
+
+def _frame() -> SkyvernFrame:
+ return SkyvernFrame(frame=MagicMock())
+
+
+async def test_returns_dict_when_predicate_reports_open_popup() -> None:
+ detail = {"role": "combobox", "hasPopup": "dialog", "tag": "div", "controlsResolved": 0}
+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(return_value=detail)) as mock_eval:
+ result = await _frame().get_open_aria_popup_trigger()
+ assert result == detail
+ # The wrapper must call the named JS predicate.
+ assert "getOpenAriaPopupTrigger" in mock_eval.await_args.kwargs["expression"]
+
+
+async def test_returns_none_when_predicate_reports_no_popup() -> None:
+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(return_value=None)):
+ result = await _frame().get_open_aria_popup_trigger()
+ assert result is None
+
+
+async def test_non_dict_result_coerced_to_none() -> None:
+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(return_value="unexpected")):
+ result = await _frame().get_open_aria_popup_trigger()
+ assert result is None
+
+
+async def test_evaluation_exception_fails_open_with_warning() -> None:
+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(side_effect=RuntimeError("boom"))):
+ with patch("skyvern.webeye.utils.page.LOG.warning") as mock_warn:
+ result = await _frame().get_open_aria_popup_trigger()
+ assert result is None, "an evaluation error must fail open to the default (scrolling) behavior"
+ assert mock_warn.called, "the fail-open path must warn"