Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
fix(SKY-12735): preserve open ARIA popups from screenshot-scroll dismissal (#7837)
LawyZheng5e8200f61603
diff --git a/docs/developers/features/browser-sessions.mdx b/docs/developers/features/browser-sessions.mdxindex bc7cb5536..316fd89cd 100644--- a/docs/developers/features/browser-sessions.mdx+++ b/docs/developers/features/browser-sessions.mdx@@ -28,7 +28,7 @@ A Browser Session is a live browser instance that Skyvern keeps running between## Configuration-Sessions support timeouts from 5 minutes to 24 hours (defaulting to 60 minutes), proxy locations across any of Skyvern's [21 supported countries](/developers/going-to-production/proxy-geolocation), Chrome or Edge browser types, and built-in extensions for ad-blocking and CAPTCHA solving. Every session exposes a live VNC stream.+Sessions support timeouts from 5 minutes to 24 hours (defaulting to 60 minutes), proxy locations across any of Skyvern's [21 supported countries](/developers/going-to-production/proxy-geolocation), Chrome or Edge browser types, and built-in extensions for ad-blocking and CAPTCHA solving. The CAPTCHA solver isn't available on the Chrome browser type — use Edge or the managed stealth default (see [Choose the right browser type](/developers/optimization/browser-sessions#choose-the-right-browser-type)). Every session exposes a live VNC stream.## Sessions vs. Profilesdiff --git a/docs/developers/optimization/browser-sessions.mdx b/docs/developers/optimization/browser-sessions.mdxindex 19c8540f3..72a56d16e 100644--- a/docs/developers/optimization/browser-sessions.mdx+++ b/docs/developers/optimization/browser-sessions.mdx@@ -84,7 +84,7 @@ curl -X POST "https://api.skyvern.com/v1/browser_sessions" \|-----------|------|-------------|| `timeout` | integer | Session lifetime in minutes. Min: 5, Max: 1440 (24 hours). Default: `60` || `proxy_location` | string | Geographic proxy location (Cloud only). See [Proxy & Geolocation](/developers/going-to-production/proxy-geolocation) for available options |-| `browser_type` | string | Browser type: `chrome` or `msedge` |+| `browser_type` | string | Browser type: `chrome` or `msedge`. If omitted, Skyvern picks a managed default that may be a hardened stealth Chromium build rather than Chrome. Set it explicitly to select the engine when no `browser_profile_id` is used — a supplied profile's source browser type takes precedence. See [Choose the right browser type](#choose-the-right-browser-type) || `extensions` | array | Extensions to install: `ad-blocker`, `captcha-solver` || `generate_browser_profile` | boolean | REST API only for now — not yet accepted by the SDK methods. Save the session's browser profile (cookies, localStorage, session files) when the session ends, so it can be turned into a reusable [browser profile](/developers/optimization/browser-profiles). Default: `false`. See [Save a session's profile](#save-a-sessions-profile) |@@ -647,33 +647,40 @@ await client.runWorkflow({ body: { workflow_id: "wpid_abc" }, waitForCompletion:### Choose the right browser type-Chrome has the widest compatibility. Use Edge only when a site requires or detects it specifically.+If you omit `browser_type`, Skyvern selects a managed default for your account — this may be a hardened stealth Chromium build rather than Google Chrome, so don't rely on the default being Chrome. Set `browser_type` explicitly when you need a specific engine. Note that if you load a session from a `browser_profile_id`, the profile's source browser type takes precedence over `browser_type`.++Each engine has a trade-off:++- **`chrome`** — Google Chrome. Can work better on some fingerprint-sensitive sites where the managed stealth default gets challenged, but does **not** support the CAPTCHA-solver extension. This varies by site — test for yours.+- **`msedge`** — Microsoft Edge. Supports the CAPTCHA-solver extension, so use it (or the managed stealth default) when a site serves hCaptcha/reCAPTCHA and you need automatic solving.++Site behavior varies: if a site both fingerprints aggressively and serves CAPTCHAs, test both engines to find the best fit.<CodeGroup>```python Python-# Chrome (default) - widest compatibility+# Chrome - can help on some fingerprint-sensitive sites; no CAPTCHA-solver extensionsession = await client.create_browser_session(browser_type="chrome")-# Edge - for sites that require or fingerprint Edge+# Edge - supports the CAPTCHA-solver extension for hCaptcha/reCAPTCHAsession = await client.create_browser_session(browser_type="msedge")``````typescript TypeScript-// Chrome (default) - widest compatibility+// Chrome - can help on some fingerprint-sensitive sites; no CAPTCHA-solver extensionconst chromeSession = await client.createBrowserSession({ browser_type: "chrome" });-// Edge - for sites that require or fingerprint Edge+// Edge - supports the CAPTCHA-solver extension for hCaptcha/reCAPTCHAconst edgeSession = await client.createBrowserSession({ browser_type: "msedge" });``````bash cURL-# Chrome (default) - widest compatibility+# Chrome - can help on some fingerprint-sensitive sites; no CAPTCHA-solver extensioncurl -s -X POST "https://api.skyvern.com/v1/browser_sessions" \-H "x-api-key: $SKYVERN_API_KEY" \-H "Content-Type: application/json" \-d '{"browser_type": "chrome"}'-# Edge - for sites that require or fingerprint Edge+# Edge - supports the CAPTCHA-solver extension for hCaptcha/reCAPTCHAcurl -s -X POST "https://api.skyvern.com/v1/browser_sessions" \-H "x-api-key: $SKYVERN_API_KEY" \-H "Content-Type: application/json" \@@ -683,7 +690,7 @@ curl -s -X POST "https://api.skyvern.com/v1/browser_sessions" \### Use extensions strategically-Extensions add startup time, so only enable them when needed. The ad-blocker removes overlay ads that can interfere with automation. The captcha-solver handles CAPTCHAs automatically but is only available on Cloud.+Extensions add startup time, so only enable them when needed. The ad-blocker removes overlay ads that can interfere with automation. The captcha-solver handles CAPTCHAs automatically but is only available on Cloud, and it requires the `msedge` browser type (or the managed stealth default) — the `chrome` browser type does not support it.<CodeGroup>```python Pythondiff --git a/skyvern/forge/agent.py b/skyvern/forge/agent.pyindex dc3d3722d..e329d909c 100644--- a/skyvern/forge/agent.py+++ b/skyvern/forge/agent.py@@ -114,6 +114,12 @@ from skyvern.forge.sdk.db.enums import TaskTypefrom skyvern.forge.sdk.experimentation.enrich_tree import resolve_enrich_tree_for_contextfrom skyvern.forge.sdk.experimentation.llm_prompt_config import resolve_check_user_goal_handlerfrom skyvern.forge.sdk.experimentation.slim_llm_output import get_slim_output_template_value+from skyvern.forge.sdk.experimentation.transient_ui_capture import (+ decide_transient_ui_suppression,+ emit_transient_ui_popup_telemetry,+ resolve_transient_ui_capture_arm,+ transient_ui_capture_arm,+)from skyvern.forge.sdk.fail_fast.shadow import record_fail_fast_shadowfrom skyvern.forge.sdk.log_artifacts import save_step_logs, save_task_logsfrom skyvern.forge.sdk.models import SpeculativeLLMMetadata, Step, StepStatus@@ -3446,6 +3452,24 @@ class ForgeAgent:if engine in CUA_ENGINES:scrolling_number = 0+ elif scrolling_number > 0 and skyvern_frame is not None:+ arm = transient_ui_capture_arm(context)+ if arm != "off":+ # Shadow-detect an open transient popup in control; only treatment suppresses the+ # scroll that would dismiss it (e.g. a portal date-picker popup) before the next step.+ # This post-action screenshot shares the per-run consecutive-suppression cap with the+ # agent-step scrape so a stale expanded trigger cannot pin captures at one viewport.+ popup_trigger = await skyvern_frame.get_open_aria_popup_trigger()+ _span.set_attribute("transient_ui_arm", arm)+ if popup_trigger is not None:+ _span.set_attribute("transient_ui_detected", True)+ emit_transient_ui_popup_telemetry(_span, popup_trigger)+ decision = decide_transient_ui_suppression(context, arm, detected=popup_trigger is not None)+ if decision.suppress:+ scrolling_number = 0+ _span.set_attribute("transient_ui_scroll_suppressed", True)+ elif decision.capped:+ _span.set_attribute("transient_ui_suppression_capped", True)artifacts: list[BulkArtifactCreationRequest | None] = []screenshot_artifact_id: str | None = None@@ -3661,6 +3685,9 @@ class ForgeAgent:max_screenshot_number=max_screenshot_number,draw_boxes=draw_boxes,scroll=scroll,+ # Only this agent-step scrape opts into transient-UI scroll suppression (SKY-12735);+ # verification / extraction / error-detection scrapes keep legacy scrolling.+ allow_transient_ui_suppression=True,)@traced(name="skyvern.agent.scrape_and_prompt", role="wrapper")@@ -3752,6 +3779,8 @@ class ForgeAgent:)context.use_artifact_bundling = False+ await resolve_transient_ui_capture_arm(context)+# start the async tasks while running scrape_websiteif engine not in CUA_ENGINES:self.async_operation_pool.run_operation(task.task_id, AgentPhase.scrape)diff --git a/skyvern/forge/sdk/core/skyvern_context.py b/skyvern/forge/sdk/core/skyvern_context.pyindex 54c95653a..a49965c2b 100644--- a/skyvern/forge/sdk/core/skyvern_context.py+++ b/skyvern/forge/sdk/core/skyvern_context.py@@ -131,6 +131,25 @@ class SkyvernContext:# PostHog flag ENABLE_LEAN_ELEMENT_TREE, evaluated once per run at scrape time# and read sync from prompt-build sites.enable_lean_element_tree: bool = False+ # PRESERVE_TRANSIENT_UI_CAPTURE experiment arm, resolved per run. Tri-state: True=treatment+ # (suppress a scroll that would dismiss an open transient popup), False=control (shadow-detect+ # only), None=off (undefined/no-provider/error -> current scrolling behavior).+ preserve_transient_ui_capture: bool | None = None+ # Pinned once resolve_transient_ui_capture_arm resolves the arm (including off/None), so a TTL+ # expiry or mid-run flag ramp cannot flip the arm later in the same run.+ preserve_transient_ui_capture_resolved: bool = False+ # Single-flight the first-use provider resolution when parallel blocks/branches share one+ # context, so it is queried at most once per run (mirrors slim_output_variant_lock).+ preserve_transient_ui_capture_lock: asyncio.Lock = field(default_factory=asyncio.Lock)+ # Count of CONSECUTIVE agent-step captures the treatment arm has suppressed scrolling on. Co-owned+ # by the two agent-step capture sites — the agent-step scrape (scrape_web_unsafe with+ # allow_transient_ui_suppression=True) and the post-action screenshot+ # (record_artifacts_after_action) — via decide_transient_ui_suppression: incremented when a+ # capture suppresses, reset to 0 when a qualifying popup is not detected, and frozen at the cap+ # while a stale expanded trigger keeps matching so later captures fall back to legacy scrolling.+ # Both sites for a run run sequentially, so the read-modify-write needs no lock; verification /+ # extraction / error-detection scrapes never touch it.+ transient_ui_consecutive_suppressions: int = 0enrich_tree_mode: EnrichTreeMode = EnrichTreeMode.CONTROLstep_retry_index: int = 0diff --git a/skyvern/forge/sdk/experimentation/transient_ui_capture.py b/skyvern/forge/sdk/experimentation/transient_ui_capture.pynew file mode 100644index 000000000..a9f7f5118--- /dev/null+++ b/skyvern/forge/sdk/experimentation/transient_ui_capture.py@@ -0,0 +1,139 @@+from __future__ import annotations++from dataclasses import dataclass+from typing import Any++import structlog++from skyvern.forge import app+from skyvern.forge.sdk.core import skyvern_context+from skyvern.forge.sdk.experimentation.providers import NoOpExperimentationProvider++LOG = structlog.get_logger()++PRESERVE_TRANSIENT_UI_CAPTURE_FLAG = "PRESERVE_TRANSIENT_UI_CAPTURE"++ARM_TREATMENT = "treatment"+ARM_CONTROL = "control"+ARM_OFF = "off"++# A trigger that stays aria-expanded=true (e.g. a stale/undismissed combobox) would otherwise hold+# every agent-step capture at one viewport for the rest of the run. Allow at most this many+# CONSECUTIVE suppressing captures — enough for the post-action screenshot plus the next agent-step+# scrape — then fall back to legacy scrolling until a scrape sees no qualifying popup (which resets+# the run's consecutive counter).+MAX_CONSECUTIVE_TRANSIENT_UI_SUPPRESSIONS = 2++# role / aria-haspopup are page-controlled attribute values. domUtils.js already returns only the+# matched token, but the capture site re-validates against these allowlists so arbitrary+# attacker-authored text can never reach telemetry even if the detector return shape changes.+TRANSIENT_UI_ROLE_TELEMETRY_ALLOWLIST = frozenset({"combobox"})+TRANSIENT_UI_HASPOPUP_TELEMETRY_ALLOWLIST = frozenset({"true", "listbox", "menu", "dialog", "grid", "tree"})+++async def resolve_transient_ui_capture_arm(+ context: skyvern_context.SkyvernContext,+ *,+ distinct_id: str | None = None,+ organization_id: str | None = None,+ workflow_permanent_id: str | None = None,+) -> None:+ """Resolve the PRESERVE_TRANSIENT_UI_CAPTURE experiment once at execution start and pin the+ tri-state arm on the context. Idempotent: after the first resolution — True, False, or None+ (undefined flag, no provider, missing run identity, or resolver error) — later calls return+ immediately, so a TTL expiry or a mid-run flag ramp cannot flip the arm and the provider is+ queried at most once per run; the capture hot paths only read the cached value. Off is the+ safe default.++ An execution-boundary caller whose context does not yet carry full run identity — notably an+ inline child workflow, whose scoped context has no workflow_run_id/organization_id — may pass+ distinct_id/organization_id/workflow_permanent_id explicitly so the child is attributed to its+ own run rather than the parent's; otherwise they are read from the context.+ """+ if context.preserve_transient_ui_capture_resolved:+ return+ provider = app.EXPERIMENTATION_PROVIDER+ if isinstance(provider, NoOpExperimentationProvider):+ # No experimentation configured (OSS default) -> off, and never query the provider.+ context.preserve_transient_ui_capture_resolved = True+ return+ resolved_distinct_id = (+ distinct_id or context.workflow_run_id or context.task_id or context.task_v2_id or context.run_id+ )+ if not resolved_distinct_id:+ context.preserve_transient_ui_capture_resolved = True+ return+ # Single-flight the provider query: concurrent first-resolvers sharing this context must not+ # both call the provider. Re-check inside the lock in case another coroutine pinned it while we+ # waited (the fast-path check above is unlocked to keep already-resolved contexts lock-free).+ async with context.preserve_transient_ui_capture_lock:+ if context.preserve_transient_ui_capture_resolved:+ return+ try:+ context.preserve_transient_ui_capture = await provider.resolve_feature_flag_cached(+ PRESERVE_TRANSIENT_UI_CAPTURE_FLAG,+ resolved_distinct_id,+ properties={+ "organization_id": organization_id or context.organization_id,+ "workflow_permanent_id": workflow_permanent_id or context.workflow_permanent_id or "not_workflow",+ },+ )+ except Exception:+ LOG.warning("Failed to resolve PRESERVE_TRANSIENT_UI_CAPTURE; defaulting to off/unenrolled", exc_info=True)+ context.preserve_transient_ui_capture = None+ context.preserve_transient_ui_capture_resolved = True+++@dataclass(frozen=True)+class TransientUiSuppressionDecision:+ suppress: bool+ capped: bool+++def decide_transient_ui_suppression(+ context: skyvern_context.SkyvernContext | None, arm: str, *, detected: bool+) -> TransientUiSuppressionDecision:+ """Apply the per-run consecutive-suppression cap for one agent-step capture and update the run+ counter. ``detected`` is whether a qualifying popup was found on THIS capture. Only the treatment+ arm suppresses or touches the counter (control/off shadow-detect but never suppress). The agent-+ step scrape and the post-action screenshot both call this so their consecutive suppressions count+ against ONE budget: suppress until the cap, then fall back to legacy scrolling, and reset the+ counter as soon as a capture sees no qualifying popup."""+ if arm != ARM_TREATMENT:+ return TransientUiSuppressionDecision(suppress=False, capped=False)+ if not detected:+ if context is not None:+ context.transient_ui_consecutive_suppressions = 0+ return TransientUiSuppressionDecision(suppress=False, capped=False)+ if context is None or context.transient_ui_consecutive_suppressions < MAX_CONSECUTIVE_TRANSIENT_UI_SUPPRESSIONS:+ if context is not None:+ context.transient_ui_consecutive_suppressions += 1+ return TransientUiSuppressionDecision(suppress=True, capped=False)+ return TransientUiSuppressionDecision(suppress=False, capped=True)+++def emit_transient_ui_popup_telemetry(span: Any, popup_trigger: dict) -> None:+ """Emit bounded popup attributes to ``span``. role / aria-haspopup are page-controlled, so only+ an allowlisted matched token is emitted (a non-allowlisted value is omitted); controls-resolved+ is a bounded boolean so the no-target portal fallback (the main false-positive source) is+ measurable. Shared by both capture sites so telemetry stays consistent."""+ role_val = popup_trigger.get("role")+ if isinstance(role_val, str) and role_val in TRANSIENT_UI_ROLE_TELEMETRY_ALLOWLIST:+ span.set_attribute("transient_ui_role", role_val)+ haspopup_val = popup_trigger.get("hasPopup")+ if isinstance(haspopup_val, str) and haspopup_val in TRANSIENT_UI_HASPOPUP_TELEMETRY_ALLOWLIST:+ span.set_attribute("transient_ui_haspopup", haspopup_val)+ span.set_attribute("transient_ui_controls_resolved", bool(popup_trigger.get("controlsResolved")))+++def transient_ui_capture_arm(context: skyvern_context.SkyvernContext | None) -> str:+ """Map the cached tri-state assignment to an arm: treatment (suppress scroll on detection),+ control (shadow-detect only), or off (undefined/no-provider/error/missing -> current behavior)."""+ if context is None:+ return ARM_OFF+ value = context.preserve_transient_ui_capture+ if value is True:+ return ARM_TREATMENT+ if value is False:+ return ARM_CONTROL+ return ARM_OFFdiff --git a/skyvern/forge/sdk/workflow/service.py b/skyvern/forge/sdk/workflow/service.pyindex 53d298f68..b89417db2 100644--- a/skyvern/forge/sdk/workflow/service.py+++ b/skyvern/forge/sdk/workflow/service.py@@ -74,6 +74,7 @@ from skyvern.forge.sdk.db.enums import BrowserSeedSource, OrganizationAuthTokenTfrom skyvern.forge.sdk.db.id import generate_output_parameter_id, generate_workflow_parameter_idfrom skyvern.forge.sdk.enterprise_features import collect_enterprise_gated_run_featuresfrom skyvern.forge.sdk.experimentation.enrich_tree import resolve_enrich_tree_for_context+from skyvern.forge.sdk.experimentation.transient_ui_capture import resolve_transient_ui_capture_armfrom skyvern.forge.sdk.models import Step, StepStatusfrom skyvern.forge.sdk.schemas.browser_profiles import BrowserProfilefrom skyvern.forge.sdk.schemas.files import FileInfo@@ -3286,6 +3287,16 @@ class WorkflowService:current_context.generate_script = Falseif workflow_run.code_gen:current_context.generate_script = True+ # Assign the transient-UI capture arm at execution start — before any block+ # capture, and for cached/code runs that never reach agent_step — using the run+ # being executed, so an inline child workflow whose scoped context lacks full+ # identity is attributed to its own run rather than the parent's.+ await resolve_transient_ui_capture_arm(+ current_context,+ distinct_id=workflow_run.workflow_run_id,+ organization_id=organization.organization_id,+ workflow_permanent_id=workflow_run.workflow_permanent_id,+ )workflow_run, blocks_to_update = await self._execute_workflow_blocks(workflow=workflow,workflow_run=workflow_run,diff --git a/skyvern/services/task_v2_service.py b/skyvern/services/task_v2_service.pyindex 01554c19c..723961272 100644--- a/skyvern/services/task_v2_service.py+++ b/skyvern/services/task_v2_service.py@@ -552,9 +552,11 @@ async def run_task_v2(workflow, workflow_run = None, Noneparent_context = skyvern_context.current()current_run_id = parent_context.run_id if parent_context and parent_context.run_id else task_v2_id- # Carry trigger_type + use_flex_llm_routing forward from the parent context (set in- # scripts/run_task_v2.py at worker boot) so flex routing eligibility persists into the- # scoped TaskV2 execution context.+ # Carry trigger_type, use_flex_llm_routing, and the execution-start transient-UI capture arm+ # forward from the parent context (set in scripts/run_task_v2.py at worker boot). scoped() takes+ # a wholesale replacement context, so any field not copied here is dropped for the scope; without+ # the arm fields the boot resolution would be lost and TaskV2 would re-resolve (or run off) before+ # its first capture.context = SkyvernContext(organization_id=organization_id,organization_name=organization.organization_name,@@ -567,6 +569,10 @@ async def run_task_v2(trigger_type=parent_context.trigger_type if parent_context else None,use_flex_llm_routing=parent_context.use_flex_llm_routing if parent_context else False,consecutive_captcha_timeouts=parent_context.consecutive_captcha_timeouts if parent_context else 0,+ preserve_transient_ui_capture=parent_context.preserve_transient_ui_capture if parent_context else None,+ preserve_transient_ui_capture_resolved=(+ parent_context.preserve_transient_ui_capture_resolved if parent_context else False+ ),)# SKY-7005: scoped() restores the parent context on exit, preserving# loop_internal_state so per-iteration download filtering continues todiff --git a/skyvern/webeye/actions/handler.py b/skyvern/webeye/actions/handler.pyindex 194906d2c..2e630958c 100644--- a/skyvern/webeye/actions/handler.py+++ b/skyvern/webeye/actions/handler.py@@ -208,6 +208,22 @@ def resolve_engine_selection_for_task(task: Task) -> BrowserEngineSelection | Noreturn browser_state.engine_selection if browser_state is not None else None+def _is_selected_engine_timeout(exc: BaseException, engine_selection: BrowserEngineSelection | None) -> bool:+ """A driver-native timeout under THIS run's selected engine; the stock Playwright ``TimeoutError``+ identity when no engine is pinned (unchanged default)."""+ if engine_selection is not None:+ return engine_selection.is_engine_timeout_error(exc)+ return isinstance(exc, TimeoutError)+++def _is_selected_engine_error(exc: BaseException, engine_selection: BrowserEngineSelection | None) -> bool:+ """A driver-native error under THIS run's selected engine; the stock Playwright ``Error`` identity+ when no engine is pinned (unchanged default)."""+ if engine_selection is not None:+ return engine_selection.is_engine_error(exc)+ return isinstance(exc, PlaywrightError)++class _CollapseGateResult(NamedTuple):family_enabled: boolassigned: bool | None@@ -4058,13 +4074,14 @@ async def handle_input_text_action(skyvern_element = retargeted_elementcan_input_text = True+ engine_selection = resolve_engine_selection_for_task(task)skyvern_frame = await SkyvernFrame.create_instance(skyvern_element.get_frame())- incremental_scraped = IncrementalScrapePage(- skyvern_frame=skyvern_frame, engine_selection=resolve_engine_selection_for_task(task)- )+ incremental_scraped = IncrementalScrapePage(skyvern_frame=skyvern_frame, engine_selection=engine_selection)timeout = settings.BROWSER_ACTION_TIMEOUT_MS- current_text = await get_input_value(skyvern_element.get_tag_name(), skyvern_element.get_locator())+ current_text = await get_input_value(+ skyvern_element.get_tag_name(), skyvern_element.get_locator(), engine_selection=engine_selection+ )if not is_totp_value and current_text == current_text_target:return [ActionSuccess()]@@ -4196,7 +4213,9 @@ async def handle_input_text_action(if not prefilter_typeahead:try:await skyvern_element.press_key("ArrowDown")- except TimeoutError:+ except Exception as exc:+ if not _is_selected_engine_timeout(exc, engine_selection):+ raise# sometimes we notice `press_key()` raise a timeout but actually the dropdown is opened.LOG.info("Timeout to press ArrowDown to open dropdown, ignore the timeout and continue to execute the action",@@ -4389,17 +4408,11 @@ async def handle_input_text_action(is_date_related = input_or_select_context is not None and input_or_select_context.is_date_related is Truetry:await skyvern_element.input_clear()- except TimeoutError:- LOG.info("None input tag clear timeout", action=action)- return [- ActionFailure(- InvalidElementForTextInput(- element_id=action.element_id, tag_name=tag_name, is_date_related=is_date_related- )- )- ]- except Exception:- LOG.warning("Failed to clear the input field", action=action, exc_info=True)+ except Exception as exc:+ if _is_selected_engine_timeout(exc, engine_selection):+ LOG.info("None input tag clear timeout", action=action)+ else:+ LOG.warning("Failed to clear the input field", action=action, exc_info=True)return [ActionFailure(InvalidElementForTextInput(@@ -4700,35 +4713,50 @@ async def handle_input_text_action(error_message=str(inc_error),)raise inc_error- except PlaywrightError as inc_error:- # Handle Playwright-specific errors during incremental element processing- # (e.g., TOTP form auto-submit, or search-dropdown selection triggering navigation)- error_message = str(inc_error).lower()- if (- "execution context was destroyed" in error_message- or "navigation" in error_message- or "target closed" in error_message- ):- # These are expected during page navigation/auto-submit, silently continue- LOG.debug(- "Playwright error during incremental element processing (likely page navigation)",+ except Exception as inc_error:+ # Driver-native errors during incremental processing (e.g. TOTP form auto-submit, or a+ # search-dropdown selection triggering navigation) are classified against THIS run's+ # selected engine, so a non-stock driver's navigation errors are tolerated identically;+ # missing selection keeps the stock Playwright identity. Non-engine errors stay swallowed.+ if _is_selected_engine_error(inc_error, engine_selection):+ error_message = str(inc_error).lower()+ if (+ "execution context was destroyed" in error_message+ or "navigation" in error_message+ or "target closed" in error_message+ ):+ # These are expected during page navigation/auto-submit, silently continue+ LOG.debug(+ "Engine error during incremental element processing (likely page navigation)",+ error_type=type(inc_error).__name__,+ error_message=error_message,+ )+ else:+ LOG.warning(+ "Unexpected engine error during incremental element processing",+ error_type=type(inc_error).__name__,+ error_message=str(inc_error),+ )+ raise inc_error+ elif isinstance(inc_error, PlaywrightError):+ # A foreign stock-Playwright driver error under a pinned non-stock engine. Under the+ # stock default this arm is unreachable (a PlaywrightError satisfies the engine+ # predicate above), so this is a no-op there. It is not one of the selected engine's+ # tolerances, so propagate it exactly as the pre-PR ``except PlaywrightError`` did+ # instead of falling through to a false ActionSuccess.+ LOG.warning(+ "Foreign driver error during incremental element processing under a pinned engine",error_type=type(inc_error).__name__,- error_message=error_message,+ error_message=str(inc_error),)+ raise inc_errorelse:+ # Handle any other unexpected errors during incremental element processingLOG.warning(- "Unexpected Playwright error during incremental element processing",+ "Unexpected error during incremental element processing",error_type=type(inc_error).__name__,error_message=str(inc_error),)- raise inc_error- except Exception as inc_error:- # Handle any other unexpected errors during incremental element processing- LOG.warning(- "Unexpected error during incremental element processing",- error_type=type(inc_error).__name__,- error_message=str(inc_error),- )finally:# Always stop listeningawait incremental_scraped.stop_listen_dom_increment()@@ -4794,7 +4822,7 @@ def _find_similar_url_in_text(candidate_url: str, text: str) -> str | None:return matched-async def _wait_for_upload_processing(page: Page) -> None:+async def _wait_for_upload_processing(page: Page, engine_selection: BrowserEngineSelection | None = None) -> None:"""Wait for page readiness signals after a file upload.Covers upload-processing UI (spinners, progress bars, DOM updates) beyond@@ -4812,10 +4840,20 @@ async def _wait_for_upload_processing(page: Page) -> None:dom_stable_ms=300,dom_stability_timeout_ms=2000,)- except (TimeoutError, asyncio.TimeoutError, SkyvernPageAnalysisTimeout):+ except (asyncio.TimeoutError, SkyvernPageAnalysisTimeout):LOG.info("Upload processing page-ready wait timed out, continuing")- except PlaywrightError:- LOG.warning("Upload processing page-ready wait interrupted by Playwright error, continuing", exc_info=True)+ except Exception as exc:+ # Classify against THIS run's selected engine so a non-stock driver's timeout/error is+ # tolerated identically; missing selection keeps the stock Playwright identity. Anything+ # that is not an engine error propagates, as before.+ if _is_selected_engine_timeout(exc, engine_selection):+ LOG.info("Upload processing page-ready wait timed out, continuing")+ elif _is_selected_engine_error(exc, engine_selection):+ LOG.warning(+ "Upload processing page-ready wait interrupted by browser engine error, continuing", exc_info=True+ )+ else:+ raise@traced(name="skyvern.agent.action.upload_file")@@ -4893,7 +4931,7 @@ async def handle_upload_file_action(timeout=settings.BROWSER_ACTION_TIMEOUT_MS,)- await _wait_for_upload_processing(page)+ await _wait_for_upload_processing(page, engine_selection=resolve_engine_selection_for_task(task))return [ActionSuccess()]else:@@ -6498,7 +6536,7 @@ async def chain_click(# File chooser opened during this click — upload completed normallyLOG.info("File chooser triggered during this click", action=action)if file:- await _wait_for_upload_processing(page)+ await _wait_for_upload_processing(page, engine_selection=engine_selection)if not has_pending:page.remove_listener("filechooser", fc_func)if context is not None and context.pending_file_chooser is not None:@@ -6534,7 +6572,7 @@ async def chain_click():# A previous UPLOAD_FILE's deferred listener was consumed by this clickLOG.info("Pending file chooser from previous UPLOAD_FILE was consumed by this click", action=action)- await _wait_for_upload_processing(page)+ await _wait_for_upload_processing(page, engine_selection=engine_selection)context.cleanup_pending_file_chooser()else:@@ -7155,7 +7193,9 @@ async def discover_and_select_from_full_dropdown()try:await skyvern_element.press_key("ArrowDown")- except TimeoutError:+ except Exception as exc:+ if not _is_selected_engine_timeout(exc, resolve_engine_selection_for_task(task)):+ raiseLOG.info("Timeout pressing ArrowDown in discover fallback, continuing",element_id=skyvern_element.get_id(),@@ -8254,7 +8294,7 @@ async def _reset_custom_select_combobox_input(locator = element.get_locator()await locator.fill("")await element.click(page=page, engine_selection=engine_selection)- return await get_input_value(element.get_tag_name(), locator) == ""+ return await get_input_value(element.get_tag_name(), locator, engine_selection=engine_selection) == ""except Exception:LOG.info("Failed to reset custom-select combobox input before LLM fallback",@@ -8474,7 +8514,11 @@ async def select_from_emerging_elements()input_element = await dom_after_open.get_skyvern_element_by_id(element_id)await input_element.scroll_into_view()- current_text = await get_input_value(input_element.get_tag_name(), input_element.get_locator())+ current_text = await get_input_value(+ input_element.get_tag_name(),+ input_element.get_locator(),+ engine_selection=resolve_engine_selection_for_task(task),+ )if current_text == actual_value:return ActionSuccess()@@ -8666,7 +8710,11 @@ async def select_from_dropdown(actual_value = get_actual_value_of_parameter_if_secret_with_task(task, value)input_element = await SkyvernElement.create_from_incremental(incremental_scraped, element_id)await input_element.scroll_into_view()- current_text = await get_input_value(input_element.get_tag_name(), input_element.get_locator())+ current_text = await get_input_value(+ input_element.get_tag_name(),+ input_element.get_locator(),+ engine_selection=resolve_engine_selection_for_task(task),+ )if current_text == actual_value:single_select_result.action_result = ActionSuccess()return single_select_result@@ -9776,18 +9824,24 @@ async def click_listbox_option(return False-async def get_input_value(tag_name: str, locator: Locator) -> str | None:+async def get_input_value(+ tag_name: str, locator: Locator, engine_selection: BrowserEngineSelection | None = None+) -> str | None:# input_value() rejects non-<input>/<textarea>/<select> nodes and inner_text() rejects# non-HTMLElement nodes; the live node can disagree with the scraped tag_name after a# re-render. Treat an incompatible read as "value unknown" so the caller's own- # element-type classification runs instead of a raw Playwright exception escaping here.+ # element-type classification runs instead of a raw driver exception escaping here. The+ # incompatible-node identity is matched against THIS run's selected engine; missing selection+ # keeps the stock Playwright identity (unchanged default).try:if tag_name in COMMON_INPUT_TAGS:return await locator.input_value()# for span, div, p or other tags:# we need to trim the unicode space for these tagsreturn (await locator.inner_text()).replace("\xa0", " ").strip()- except PlaywrightError as exc:+ except Exception as exc:+ if not _is_selected_engine_error(exc, engine_selection):+ raiseif is_incompatible_text_input_error(exc):LOG.info("Skipping value read on an incompatible element", tag_name=tag_name, error=str(exc))return Nonediff --git a/skyvern/webeye/browser_state.py b/skyvern/webeye/browser_state.pyindex 5613fff90..9c30404c3 100644--- a/skyvern/webeye/browser_state.py+++ b/skyvern/webeye/browser_state.py@@ -121,4 +121,5 @@ class BrowserState(Protocol):support_empty_page: bool = False,wait_seconds: float = 0,must_included_tags: list[str] | None = None,+ allow_transient_ui_suppression: bool = False,) -> ScrapedPage: ...diff --git a/skyvern/webeye/real_browser_state.py b/skyvern/webeye/real_browser_state.pyindex f9e128032..c1e00fb8c 100644--- a/skyvern/webeye/real_browser_state.py+++ b/skyvern/webeye/real_browser_state.py@@ -599,6 +599,7 @@ class RealBrowserState(BrowserState):support_empty_page: bool = False,wait_seconds: float = 0,must_included_tags: list[str] | None = None,+ allow_transient_ui_suppression: bool = False,) -> ScrapedPage:page = await self.get_working_page()if page is not None:@@ -618,6 +619,7 @@ class RealBrowserState(BrowserState):support_empty_page=support_empty_page,wait_seconds=wait_seconds,must_included_tags=must_included_tags,+ allow_transient_ui_suppression=allow_transient_ui_suppression,)async def close(self, close_browser_on_completion: bool = True, release_driver: bool | None = None) -> None:diff --git a/skyvern/webeye/scraper/domUtils.js b/skyvern/webeye/scraper/domUtils.jsindex 198c44e5a..39288066e 100644--- a/skyvern/webeye/scraper/domUtils.js+++ b/skyvern/webeye/scraper/domUtils.js@@ -765,6 +765,204 @@ function isHiddenOrDisabled(element) {return isHidden(element) || element.disabled;}+// Report an open ARIA popup trigger so one scrolling screenshot can skip scrolling while it is+// open. Combobox / menu / listbox / dialog / grid / tree popups (portal-rendered date/list+// pickers) usually render in a portal that dismisses on scroll, which the post-action+// and next-scrape screenshots would otherwise trigger before the agent can act on the popup.+const ARIA_POPUP_HASPOPUP_VALUES = new Set([+ "true",+ "listbox",+ "menu",+ "dialog",+ "grid",+ "tree",+]);++function isAriaPopupElementVisible(element) {+ if (isHidden(element)) return false;+ const style = getElementComputedStyle(element);+ if (style) {+ if (style.visibility === "hidden" || style.visibility === "collapse") {+ return false;+ }+ const opacity = parseFloat(style.opacity);+ if (!Number.isNaN(opacity) && opacity === 0) return false;+ if (style.contentVisibility === "hidden") return false;+ }+ // Native flat-tree check: Element.checkVisibility (Chromium) rejects an element hidden by an+ // ancestor's display / content-visibility, or a UA shadow root (e.g. a closed <details>), up the+ // flat tree — which the light-DOM parentElement walk below cannot see. WebKit (checkVisibility+ // bug) and unit mocks fall through to the manual walk.+ if (!isElementStyleVisibilityVisible(element, style)) return false;+ // WebKit-safe, mockable fallback: display / opacity / content-visibility do not propagate to a+ // descendant's own computed style, so walk the FULL ancestor chain to reject a hidden portal (a+ // hidden ancestor can sit at any depth — no arbitrary cap). A visited Set (identity) guards a+ // malformed/cyclic parentElement chain: each ancestor is inspected at most once, so traversal+ // terminates in O(distinct ancestors) even on a pathological DOM.+ let ancestor = element.parentElement;+ const seenAncestors = new Set();+ while (ancestor && !seenAncestors.has(ancestor)) {+ seenAncestors.add(ancestor);+ if (ancestor.hidden) return false;+ const ancestorStyle = getElementComputedStyle(ancestor);+ if (ancestorStyle) {+ if (ancestorStyle.display === "none") return false;+ if (+ ancestorStyle.visibility === "hidden" ||+ ancestorStyle.visibility === "collapse"+ ) {+ return false;+ }+ const ancestorOpacity = parseFloat(ancestorStyle.opacity);+ if (!Number.isNaN(ancestorOpacity) && ancestorOpacity === 0) return false;+ if (ancestorStyle.contentVisibility === "hidden") return false;+ }+ ancestor = ancestor.parentElement;+ }+ return true;+}++// aria-controls / aria-owns are whitespace-separated IDREF lists; resolve them against the+// light DOM only (IDREF resolution does not cross shadow boundaries). Returns the resolved+// targets plus hadIdref: whether the trigger named at least one non-empty IDREF. A trigger that+// named an IDREF but resolved nothing is a popup unmounted on close (dangling), which the caller+// must distinguish from a trigger with no aria-controls at all (a portal wired without one).+function resolveAriaPopupTargets(trigger) {+ const targets = [];+ let hadIdref = false;+ for (const attr of ["aria-controls", "aria-owns"]) {+ const raw = trigger.getAttribute(attr);+ if (!raw) continue;+ for (const id of raw.split(/\s+/)) {+ if (!id) continue;+ hadIdref = true;+ const target = document.getElementById(id);+ if (target) targets.push(target);+ }+ }+ return { targets, hadIdref };+}++// A box paints an area only if its border box has positive width/height AND intersects the viewport+// (shared by the trigger gate and the target rendered-area gate below). checkVisibility() and style+// visibility pass for zero-area / transform:scale(0) / clipped elements, so geometry must be checked+// separately or a target that renders nothing would false-positive as an open popup.+function hasPositiveViewportArea(element) {+ const rect = element.getBoundingClientRect();+ return (+ rect.width > 0 &&+ rect.height > 0 &&+ rect.bottom > 0 &&+ rect.right > 0 &&+ rect.top < window.innerHeight &&+ rect.left < window.innerWidth+ );+}++const CLIPPING_OVERFLOW_VALUES = new Set(["hidden", "clip", "scroll", "auto"]);++function clipsOverflow(style) {+ if (!style) return false;+ return (+ CLIPPING_OVERFLOW_VALUES.has(style.overflow) ||+ CLIPPING_OVERFLOW_VALUES.has(style.overflowX) ||+ CLIPPING_OVERFLOW_VALUES.has(style.overflowY)+ );+}++// A descendant is clipped to nothing if a clipping-overflow ancestor between it and `boundary`+// (inclusive) has zero on-screen area, so a zero-size scroll/hidden container swallows it. Identity+// Set guards a malformed/cyclic chain; the walk stops at `boundary` (the resolved target).+function isClippedByZeroAreaAncestor(descendant, boundary) {+ let ancestor = descendant.parentElement;+ const seen = new Set();+ while (ancestor && !seen.has(ancestor)) {+ seen.add(ancestor);+ if (+ clipsOverflow(getElementComputedStyle(ancestor)) &&+ !hasPositiveViewportArea(ancestor)+ ) {+ return true;+ }+ if (ancestor === boundary) break;+ ancestor = ancestor.parentElement;+ }+ return false;+}++// A resolved aria-controls target counts as an open popup only if it actually paints an area: its+// own border box paints, or — when the target is a zero-area container that does NOT clip its+// overflow (an overflow:visible portal mount point) — a genuinely rendered, visible descendant+// escapes it. A zero-area target that clips its overflow (or has no rendered descendant) paints+// nothing, so a stale aria-expanded=true trigger pointing at it is treated as closed. The subtree+// walk needs no depth/count cap: a DOM subtree is finite and an identity Set visits each node once.+function ariaPopupTargetHasRenderedArea(target) {+ if (hasPositiveViewportArea(target)) return true;+ if (clipsOverflow(getElementComputedStyle(target))) return false;+ const stack = Array.from(target.children || []);+ const seen = new Set();+ while (stack.length) {+ const descendant = stack.pop();+ if (!descendant || seen.has(descendant)) continue;+ seen.add(descendant);+ if (+ isAriaPopupElementVisible(descendant) &&+ hasPositiveViewportArea(descendant) &&+ !isClippedByZeroAreaAncestor(descendant, target)+ ) {+ return true;+ }+ for (const child of descendant.children || []) stack.push(child);+ }+ return false;+}++// Returns structural (non-PII) details of the first open ARIA popup trigger, or null. On a trigger+// with no aria-controls/aria-owns at all, none-resolving => open (a sibling portal not wired via+// aria-controls). On a trigger that named an IDREF: any resolved target visible+painting => open,+// otherwise closed — a present-but-dangling IDREF is a popup unmounted on close, not an open one.+// Scans the light DOM only, so triggers inside shadow roots are not detected. role / aria-haspopup+// are page-controlled: only the matched allowlisted token is returned so arbitrary attacker-authored+// attribute text never reaches telemetry.+function getOpenAriaPopupTrigger() {+ const candidates = document.querySelectorAll('[aria-expanded="true"]');+ for (const trigger of candidates) {+ const role = (trigger.getAttribute("role") || "").toLowerCase();+ const hasPopup = (+ trigger.getAttribute("aria-haspopup") || ""+ ).toLowerCase();+ const roleMatched = role === "combobox";+ const hasPopupMatched = ARIA_POPUP_HASPOPUP_VALUES.has(hasPopup);+ if (!roleMatched && !hasPopupMatched) {+ continue;+ }+ if (!isAriaPopupElementVisible(trigger)) continue;+ if (!hasPositiveViewportArea(trigger)) continue;++ const { targets, hadIdref } = resolveAriaPopupTargets(trigger);+ // No aria-controls/aria-owns at all => portal fallback (open). A named IDREF that resolves+ // nothing => dangling => closed (falls through to targets.some over an empty list). A resolved+ // target counts only if it is visible AND paints an area, so a stale trigger pointing at a+ // zero-area/clipped target is treated as closed.+ const open =+ !hadIdref ||+ targets.some(+ (target) =>+ isAriaPopupElementVisible(target) &&+ ariaPopupTargetHasRenderedArea(target),+ );+ if (!open) continue;++ return {+ role: roleMatched ? "combobox" : null,+ hasPopup: hasPopupMatched ? hasPopup : null,+ tag: trigger.tagName ? trigger.tagName.toLowerCase() : null,+ controlsResolved: targets.length,+ };+ }+ return null;+}+function isScriptOrStyle(element) {const tagName = element.tagName.toLowerCase();return tagName === "script" || tagName === "style";diff --git a/skyvern/webeye/scraper/scraper.py b/skyvern/webeye/scraper/scraper.pyindex de748d4f6..10a120483 100644--- a/skyvern/webeye/scraper/scraper.py+++ b/skyvern/webeye/scraper/scraper.py@@ -24,6 +24,11 @@ from skyvern.experimentation.wait_utils import empty_page_retry_waitfrom skyvern.forge.sdk.api.crypto import calculate_sha256from skyvern.forge.sdk.browser_action_preflight import advance_observation_epochfrom skyvern.forge.sdk.core import skyvern_context+from skyvern.forge.sdk.experimentation.transient_ui_capture import (+ decide_transient_ui_suppression,+ emit_transient_ui_popup_telemetry,+ transient_ui_capture_arm,+)from skyvern.forge.sdk.settings_manager import SettingsManagerfrom skyvern.forge.sdk.trace import apply_context_attrs, traced, traced_spanfrom skyvern.utils.image_resizer import Resolution@@ -229,6 +234,7 @@ async def scrape_website(support_empty_page: bool = False,wait_seconds: float = 0,must_included_tags: list[str] | None = None,+ allow_transient_ui_suppression: bool = False,) -> ScrapedPage:"""************************************************************************************************@@ -265,6 +271,7 @@ async def scrape_website(support_empty_page=support_empty_page,wait_seconds=wait_seconds,must_included_tags=must_included_tags,+ allow_transient_ui_suppression=allow_transient_ui_suppression,)except ScrapingFailedBlankPage:raise@@ -306,6 +313,7 @@ async def scrape_website(max_screenshot_number=max_screenshot_number,scroll=scroll,must_included_tags=must_included_tags,+ allow_transient_ui_suppression=allow_transient_ui_suppression,)@@ -433,6 +441,7 @@ async def scrape_web_unsafe(support_empty_page: bool = False,wait_seconds: float = 0,must_included_tags: list[str] | None = None,+ allow_transient_ui_suppression: bool = False,) -> ScrapedPage:"""Asynchronous function that performs web scraping without any built-in error handling. This function is intended@@ -496,6 +505,23 @@ async def scrape_web_unsafe(if token_count > DEFAULT_MAX_TOKENS:max_screenshot_number = min(max_screenshot_number, 1)+ # Shadow-detect an open transient popup only on the agent-step scrape (opt-in via+ # allow_transient_ui_suppression); goal-verification / extraction / error-detection scrapes+ # keep legacy scrolling. Only the treatment arm suppresses the scroll so the popup survives+ # into the just-built tree's next action, and only up to a bounded number of consecutive+ # captures so a stale expanded trigger cannot pin the run at one viewport.+ popup_trigger: dict | None = None+ transient_ui_ctx = skyvern_context.current()+ arm = transient_ui_capture_arm(transient_ui_ctx)+ suppress_scroll = False+ suppression_capped = False+ if allow_transient_ui_suppression and scroll and arm != "off":+ popup_trigger = await skyvern_frame.get_open_aria_popup_trigger()+ decision = decide_transient_ui_suppression(transient_ui_ctx, arm, detected=popup_trigger is not None)+ suppress_scroll = decision.suppress+ suppression_capped = decision.capped+ effective_scroll = scroll and not suppress_scroll+# get current x, y position of the pagex: int | None = Noney: int | None = None@@ -515,6 +541,16 @@ async def scrape_web_unsafe(_ss_span.set_attribute("max_screenshot_number", max_screenshot_number)_ss_span.set_attribute("draw_boxes", draw_boxes)_ss_span.set_attribute("scroll", scroll)+ _ss_span.set_attribute("effective_scroll", effective_scroll)+ if arm != "off":+ _ss_span.set_attribute("transient_ui_arm", arm)+ if popup_trigger is not None:+ _ss_span.set_attribute("transient_ui_detected", True)+ emit_transient_ui_popup_telemetry(_ss_span, popup_trigger)+ if suppress_scroll:+ _ss_span.set_attribute("transient_ui_scroll_suppressed", True)+ if suppression_capped:+ _ss_span.set_attribute("transient_ui_suppression_capped", True)_scrape_ctx = skyvern_context.current()if _scrape_ctx:if _scrape_ctx.scrape_trigger:@@ -526,7 +562,7 @@ async def scrape_web_unsafe(url=url,draw_boxes=draw_boxes,max_number=max_screenshot_number,- scroll=scroll,+ scroll=effective_scroll,)_ss_span.set_attribute("screenshot_count", len(screenshots))_ss_span.set_attribute("screenshot_bytes", sum(len(s) for s in screenshots))diff --git a/skyvern/webeye/utils/page.py b/skyvern/webeye/utils/page.pyindex cfa03bd05..dab46cf6f 100644--- a/skyvern/webeye/utils/page.py+++ b/skyvern/webeye/utils/page.py@@ -1076,6 +1076,23 @@ class SkyvernFrame:js_script = "() => getScrollXY()"return await self.evaluate(frame=self.frame, engine_selection=self.engine_selection, expression=js_script)+ async def get_open_aria_popup_trigger(self) -> dict | None:+ """Return structural details of an open ARIA popup trigger on this frame, or None.++ Fails open (returns None) on any evaluation error so screenshot-scroll policy keeps the+ current scrolling behavior rather than breaking the scrape/action loop. Detection+ semantics live in getOpenAriaPopupTrigger in domUtils.js.+ """+ try:+ result = await self.evaluate(frame=self.frame, expression="() => getOpenAriaPopupTrigger()")+ except Exception:+ LOG.warning(+ "Failed to detect open ARIA popup trigger; using default scrolling behavior",+ exc_info=True,+ )+ return None+ return result if isinstance(result, dict) else None+async def get_scroll_width_and_height(self) -> tuple[int, int]:js_script = "() => getScrollWidthAndHeight()"return await self.evaluate(frame=self.frame, engine_selection=self.engine_selection, expression=js_script)diff --git a/tests/unit/test_aria_popup_trigger_domutils.js b/tests/unit/test_aria_popup_trigger_domutils.jsnew file mode 100644index 000000000..72fcc4539--- /dev/null+++ b/tests/unit/test_aria_popup_trigger_domutils.js@@ -0,0 +1,830 @@+/**+ * Behavioral tests for getOpenAriaPopupTrigger() in domUtils.js.+ *+ * The predicate reports an open ARIA popup trigger (combobox / haspopup) so the+ * screenshot-scroll policy can skip scrolling for one capture and avoid dismissing+ * a just-opened portal popup (e.g. a date-picker overlay or ARIA listbox).+ *+ * Exit 0 = pass, exit 1 = failures on stderr.+ */++const fs = require("fs");+const path = require("path");+const vm = require("vm");++const src = fs.readFileSync(+ path.join(__dirname, "../../skyvern/webeye/scraper/domUtils.js"),+ "utf8",+);++// Node-scope DOM registries the mocked document closures read.+let allElements = [];+const byId = new Map();++function reset() {+ allElements = [];+ byId.clear();+}++function getComputedStyle(element) {+ return {+ display: "block",+ visibility: "visible",+ opacity: "1",+ cursor: "auto",+ pointerEvents: "auto",+ ...(element._style || {}),+ };+}++class TestElement {}++const context = {+ console,+ window: {+ scrollX: 0,+ scrollY: 0,+ innerHeight: 800,+ innerWidth: 1200,+ getComputedStyle,+ },+ document: {+ documentElement: {},+ querySelector() {+ return null;+ },+ querySelectorAll(selector) {+ if (selector === '[aria-expanded="true"]') {+ return allElements.filter(+ (el) => el.getAttribute("aria-expanded") === "true",+ );+ }+ return [];+ },+ getElementById(id) {+ return byId.get(id) || null;+ },+ },+ MutationObserver: class {+ observe() {}+ disconnect() {}+ takeRecords() {+ return [];+ }+ },+ Element: TestElement,+ ShadowRoot: class {},+ Node: { ELEMENT_NODE: 1, TEXT_NODE: 3 },+ Number,+ Set,+};+context.global = context;+vm.runInNewContext(+ `${src}+this.__exports = { getOpenAriaPopupTrigger };`,+ context,+);++function el(opts) {+ const element = new TestElement();+ element.tagName = (opts.tagName || "DIV").toUpperCase();+ element.nodeName = element.tagName;+ element.hidden = Boolean(opts.hidden);+ element.disabled = Boolean(opts.disabled);+ element._style = opts.style || {};+ const rect = opts.rect || { width: 120, height: 32, left: 40, top: 120 };+ element._rect = {+ width: rect.width,+ height: rect.height,+ left: rect.left,+ top: rect.top,+ right: rect.right !== undefined ? rect.right : rect.left + rect.width,+ bottom: rect.bottom !== undefined ? rect.bottom : rect.top + rect.height,+ };+ element._attributes = { ...(opts.attributes || {}) };+ if (opts.id) {+ element._attributes.id = opts.id;+ }+ element.getAttribute = function (name) {+ return name in this._attributes ? this._attributes[name] : null;+ };+ element.hasAttribute = function (name) {+ return name in this._attributes;+ };+ element.getBoundingClientRect = function () {+ return this._rect;+ };+ element.ownerDocument = { defaultView: { getComputedStyle } };+ element.getRootNode = function () {+ return { host: null };+ };+ element.parentElement = opts.parent || null;+ element.children = opts.children || [];+ for (const child of element.children) {+ child.parentElement = element;+ }+ element.closest = function () {+ return null;+ };+ allElements.push(element);+ if (opts.id) {+ byId.set(opts.id, element);+ }+ return element;+}++// Build a chain of nested ancestors: a top ancestor styled by `topStyle`, then `levels` visible+// descendants below it; returns the bottom-most node (use it as a deep element's parent).+function deepChain(topStyle, levels) {+ let node = el({ tagName: "DIV", style: topStyle });+ for (let i = 0; i < levels; i++) {+ node = el({ tagName: "DIV", parent: node });+ }+ return node;+}++let passed = 0;+let failed = 0;++function test(name, fn) {+ reset();+ try {+ fn();+ passed++;+ process.stdout.write(` PASS: ${name}\n`);+ } catch (e) {+ failed++;+ process.stderr.write(` FAIL: ${name}\n ${e.message}\n`);+ }+}++function assert(condition, message) {+ if (!condition) {+ throw new Error(message);+ }+}++const trigger = () => context.__exports.getOpenAriaPopupTrigger();++test("open combobox with no aria-controls is reported (unresolved => open)", () => {+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "true" },+ });+ const result = trigger();+ assert(+ result !== null,+ "an open combobox with no controls should be detected",+ );+ assert(+ result.role === "combobox",+ `role should be combobox, got ${result && result.role}`,+ );+});++test("closed combobox (aria-expanded=false) is not reported", () => {+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "false" },+ });+ assert(trigger() === null, "a closed combobox must not be reported");+});++test("aria-haspopup=listbox with a visible controlled target is reported", () => {+ el({+ tagName: "DIV",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "lb1",+ },+ });+ el({ id: "lb1", tagName: "UL", attributes: { role: "listbox" } });+ const result = trigger();+ assert(+ result !== null,+ "haspopup=listbox with a visible target should be detected",+ );+ assert(+ result.hasPopup === "listbox",+ `hasPopup should be listbox, got ${result && result.hasPopup}`,+ );+ assert(+ result.controlsResolved === 1,+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,+ );+});++test("menu trigger whose only controlled target is display:none is closed", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "menu",+ "aria-expanded": "true",+ "aria-controls": "m1",+ },+ });+ el({ id: "m1", tagName: "DIV", style: { display: "none" } });+ assert(+ trigger() === null,+ "all resolved targets hidden => closed => not reported",+ );+});++test("listbox trigger with one hidden and one visible target is open (any visible => open)", () => {+ el({+ tagName: "DIV",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "hidden1 visible1",+ },+ });+ el({ id: "hidden1", tagName: "DIV", style: { display: "none" } });+ el({ id: "visible1", tagName: "UL" });+ const result = trigger();+ assert(result !== null, "any visible resolved target => open");+ assert(+ result.controlsResolved === 2,+ `controlsResolved should be 2, got ${result && result.controlsResolved}`,+ );+});++test("aria-owns is parsed like aria-controls", () => {+ el({+ tagName: "DIV",+ attributes: {+ role: "combobox",+ "aria-expanded": "true",+ "aria-owns": "own1",+ },+ });+ el({ id: "own1", tagName: "DIV", style: { display: "none" } });+ assert(trigger() === null, "aria-owns target hidden => closed");+});++test("expanded element with no combobox role and no popup role does not qualify", () => {+ el({ tagName: "BUTTON", attributes: { "aria-expanded": "true" } });+ assert(+ trigger() === null,+ "a bare aria-expanded element (accordion) must not qualify",+ );+});++test("open combobox scrolled fully off-screen is not reported", () => {+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "true" },+ rect: { width: 120, height: 32, left: 40, top: 2000 },+ });+ assert(+ trigger() === null,+ "a trigger below the viewport must not be reported",+ );+});++test("open combobox with display:none is not reported", () => {+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "true" },+ style: { display: "none" },+ });+ assert(trigger() === null, "a display:none trigger must not be reported");+});++test("aria-haspopup=true (string) qualifies", () => {+ el({+ tagName: "DIV",+ attributes: { "aria-haspopup": "true", "aria-expanded": "true" },+ });+ assert(trigger() !== null, "aria-haspopup='true' should qualify");+});++test("aria-haspopup=dialog with grid/tree values all qualify", () => {+ for (const value of ["dialog", "grid", "tree"]) {+ reset();+ el({+ tagName: "DIV",+ attributes: { "aria-haspopup": value, "aria-expanded": "true" },+ });+ assert(trigger() !== null, `aria-haspopup='${value}' should qualify`);+ }+});++test("multiple whitespace-separated IDREFs none of which resolve => closed (all dangling)", () => {+ el({+ tagName: "DIV",+ attributes: {+ role: "combobox",+ "aria-expanded": "true",+ "aria-controls": "gone1 gone2\tgone3",+ },+ });+ // The trigger named IDREFs but none resolve: the popup was unmounted on close, so it is closed —+ // distinct from a trigger with no aria-controls at all (which keeps the portal fallback => open).+ assert(+ trigger() === null,+ "present-but-all-dangling IDREFs must be treated as closed, not open",+ );+});++test("target inside a display:none ancestor is closed (ancestor-aware)", () => {+ const hiddenAncestor = el({ tagName: "DIV", style: { display: "none" } });+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "menu",+ "aria-expanded": "true",+ "aria-controls": "t1",+ },+ });+ el({ id: "t1", tagName: "DIV", parent: hiddenAncestor });+ assert(+ trigger() === null,+ "a target under a display:none ancestor must not count as open",+ );+});++test("target inside a visibility:hidden ancestor is closed (ancestor-aware)", () => {+ const hiddenAncestor = el({+ tagName: "DIV",+ style: { visibility: "hidden" },+ });+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "t2",+ },+ });+ el({ id: "t2", tagName: "UL", parent: hiddenAncestor });+ assert(+ trigger() === null,+ "a target under a visibility:hidden ancestor must not count as open",+ );+});++test("target inside an opacity:0 ancestor is closed (ancestor-aware)", () => {+ const hiddenAncestor = el({ tagName: "DIV", style: { opacity: "0" } });+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "dialog",+ "aria-expanded": "true",+ "aria-controls": "t3",+ },+ });+ el({ id: "t3", tagName: "DIV", parent: hiddenAncestor });+ assert(+ trigger() === null,+ "a target under an opacity:0 ancestor must not count as open",+ );+});++test("trigger inside a display:none ancestor is not detected (ancestor-aware)", () => {+ const hiddenAncestor = el({ tagName: "DIV", style: { display: "none" } });+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "true" },+ parent: hiddenAncestor,+ });+ assert(+ trigger() === null,+ "a trigger under a display:none ancestor must not be reported",+ );+});++test("faithful visible portal under a visible ancestor is still open (control)", () => {+ const visibleAncestor = el({+ tagName: "DIV",+ style: { display: "block", opacity: "1" },+ });+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "t4",+ },+ });+ el({ id: "t4", tagName: "UL", parent: visibleAncestor });+ const result = trigger();+ assert(+ result !== null,+ "a target under a visible ancestor must still count as open",+ );+ assert(+ result.controlsResolved === 1,+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,+ );+});++test("target nested >50 levels under a display:none ancestor is closed (full walk to root)", () => {+ const bottom = deepChain({ display: "none" }, 55);+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "menu",+ "aria-expanded": "true",+ "aria-controls": "deepT",+ },+ });+ el({ id: "deepT", tagName: "DIV", parent: bottom });+ assert(+ trigger() === null,+ "a target under a display:none ancestor >50 levels up must not count as open",+ );+});++test("trigger nested >50 levels under a display:none ancestor is not reported (full walk to root)", () => {+ const bottom = deepChain({ display: "none" }, 55);+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "true" },+ parent: bottom,+ });+ assert(+ trigger() === null,+ "a trigger under a display:none ancestor >50 levels up must not be reported",+ );+});++test("valid deep portal (>50 levels, all visible) is still open (control — walk must not over-reject)", () => {+ const bottom = deepChain({ display: "block", opacity: "1" }, 55);+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "deepV",+ },+ });+ el({ id: "deepV", tagName: "UL", parent: bottom });+ const result = trigger();+ assert(+ result !== null,+ "a deep portal with all-visible ancestors must still count as open",+ );+ assert(+ result.controlsResolved === 1,+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,+ );+});++test("target under a content-visibility:hidden ancestor is closed", () => {+ const hiddenAncestor = el({+ tagName: "DIV",+ style: { contentVisibility: "hidden" },+ });+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "menu",+ "aria-expanded": "true",+ "aria-controls": "cv1",+ },+ });+ el({ id: "cv1", tagName: "DIV", parent: hiddenAncestor });+ assert(+ trigger() === null,+ "a target under a content-visibility:hidden ancestor must not count as open",+ );+});++test("trigger under a content-visibility:hidden ancestor is not reported", () => {+ const hiddenAncestor = el({+ tagName: "DIV",+ style: { contentVisibility: "hidden" },+ });+ el({+ tagName: "DIV",+ attributes: { role: "combobox", "aria-expanded": "true" },+ parent: hiddenAncestor,+ });+ assert(+ trigger() === null,+ "a trigger under a content-visibility:hidden ancestor must not be reported",+ );+});++test("target with its own content-visibility:hidden is closed", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "cv2",+ },+ });+ el({ id: "cv2", tagName: "UL", style: { contentVisibility: "hidden" } });+ assert(+ trigger() === null,+ "a target with its own content-visibility:hidden must not count as open",+ );+});++test("content-visibility:visible ancestor does not reject a valid portal (control)", () => {+ const visibleAncestor = el({+ tagName: "DIV",+ style: { contentVisibility: "visible" },+ });+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "cv3",+ },+ });+ el({ id: "cv3", tagName: "UL", parent: visibleAncestor });+ assert(+ trigger() !== null,+ "a content-visibility:visible ancestor must not reject a valid portal",+ );+});++// --- Rendered-area gate for the controlled target ----------------------------------------------+// checkVisibility() / style visibility pass for a zero-area or transform:scale(0) target, so a+// stale aria-expanded=true trigger pointing at a target that paints nothing would false-positive as+// open and suppress a needed scroll under treatment. The target must actually render an area.++test("zero-area controlled target is closed (rendered-area gate)", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "za1",+ },+ });+ el({+ id: "za1",+ tagName: "UL",+ rect: { width: 0, height: 0, left: 40, top: 120 },+ });+ assert(+ trigger() === null,+ "a zero-area controlled target must not count as open",+ );+});++test("transform:scale(0) controlled target is closed (rendered-area gate)", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "sc1",+ },+ });+ // scale(0) collapses the box AND its subtree to zero on-screen area.+ const scaledChild = el({+ tagName: "LI",+ rect: { width: 0, height: 0, left: 40, top: 120 },+ });+ el({+ id: "sc1",+ tagName: "UL",+ style: { transform: "scale(0)" },+ rect: { width: 0, height: 0, left: 40, top: 120 },+ children: [scaledChild],+ });+ assert(+ trigger() === null,+ "a transform:scale(0) controlled target must not count as open",+ );+});++test("zero-size overflow:hidden target with a positive child is closed (clipped)", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "oh1",+ },+ });+ // The child has layout area but is clipped to the 0x0 content box, so nothing paints.+ const clippedChild = el({+ tagName: "LI",+ rect: { width: 240, height: 120, left: 40, top: 120 },+ });+ el({+ id: "oh1",+ tagName: "UL",+ style: { overflow: "hidden", overflowX: "hidden", overflowY: "hidden" },+ rect: { width: 0, height: 0, left: 40, top: 120 },+ children: [clippedChild],+ });+ assert(+ trigger() === null,+ "a zero-size overflow:hidden container clips its child => closed",+ );+});++test("zero-size overflow:visible portal root with a rendered visible descendant is open", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "ov1",+ },+ });+ // A portal mount point that itself has no box, but a genuinely painted panel escapes it.+ const panel = el({+ tagName: "DIV",+ rect: { width: 240, height: 120, left: 40, top: 120 },+ });+ el({+ id: "ov1",+ tagName: "DIV",+ style: { overflow: "visible", overflowX: "visible", overflowY: "visible" },+ rect: { width: 0, height: 0, left: 40, top: 120 },+ children: [panel],+ });+ const result = trigger();+ assert(+ result !== null,+ "an overflow:visible zero-area portal root with a rendered descendant => open",+ );+ assert(+ result.controlsResolved === 1,+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,+ );+});++test("positive rendered controlled target is open (rendered-area gate control)", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "pr1",+ },+ });+ el({+ id: "pr1",+ tagName: "UL",+ rect: { width: 240, height: 120, left: 40, top: 120 },+ });+ const result = trigger();+ assert(+ result !== null,+ "a positive-area controlled target must count as open",+ );+ assert(+ result.controlsResolved === 1,+ `controlsResolved should be 1, got ${result && result.controlsResolved}`,+ );+});++// --- dangling vs absent aria-controls (a present-but-unresolved IDREF is a popup unmounted on+// close, not a portal wired without aria-controls) ---++test("dangling aria-controls (present IDREF, target unmounted) is closed", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "ghost1",+ },+ });+ // no element with id ghost1 exists (unmounted on close)+ assert(+ trigger() === null,+ "a present-but-dangling aria-controls must be treated as closed, not open",+ );+});++test("dangling aria-owns (present IDREF, target unmounted) is closed", () => {+ el({+ tagName: "DIV",+ attributes: {+ role: "combobox",+ "aria-expanded": "true",+ "aria-owns": "ghostOwned",+ },+ });+ assert(+ trigger() === null,+ "a present-but-dangling aria-owns must be treated as closed",+ );+});++test("absent aria-controls keeps portal fallback => open (controlsResolved=0)", () => {+ el({+ tagName: "BUTTON",+ attributes: { "aria-haspopup": "listbox", "aria-expanded": "true" },+ });+ const result = trigger();+ assert(result !== null, "absent aria-controls => portal fallback => open");+ assert(+ result.controlsResolved === 0,+ `controlsResolved should be 0 for the no-target fallback, got ${result && result.controlsResolved}`,+ );+});++test("empty/whitespace aria-controls is treated as absent => open", () => {+ el({+ tagName: "DIV",+ attributes: {+ role: "combobox",+ "aria-expanded": "true",+ "aria-controls": " ",+ },+ });+ assert(+ trigger() !== null,+ "a whitespace-only aria-controls references no IDREF and must fall back to portal-open",+ );+});++test("mixed dangling + resolved-visible target is open (any resolved-visible => open)", () => {+ el({+ tagName: "DIV",+ attributes: {+ "aria-haspopup": "listbox",+ "aria-expanded": "true",+ "aria-controls": "ghostX visibleX",+ },+ });+ el({ id: "visibleX", tagName: "UL" });+ const result = trigger();+ assert(result !== null, "a resolved-visible target among danglers => open");+ assert(+ result.controlsResolved === 1,+ `controlsResolved should count only resolved targets (1), got ${result && result.controlsResolved}`,+ );+});++// --- telemetry token bounding: role / aria-haspopup are page-controlled and must not reach the+// return (and thus telemetry) verbatim; only the matched allowlisted token is emitted ---++test("haspopup-qualified trigger with arbitrary role emits null role, matched hasPopup", () => {+ el({+ tagName: "BUTTON",+ attributes: {+ role: "'\"><script>arbitrary-attacker-role",+ "aria-haspopup": "menu",+ "aria-expanded": "true",+ },+ });+ const result = trigger();+ assert(result !== null, "haspopup=menu qualifies");+ assert(+ result.role === null,+ `arbitrary role must not be emitted, got ${result && JSON.stringify(result.role)}`,+ );+ assert(+ result.hasPopup === "menu",+ `matched hasPopup token must be emitted, got ${result && result.hasPopup}`,+ );+});++test("combobox-qualified trigger with arbitrary aria-haspopup emits null hasPopup, matched role", () => {+ el({+ tagName: "DIV",+ attributes: {+ role: "combobox",+ "aria-haspopup": "arbitrary-attacker-haspopup-value",+ "aria-expanded": "true",+ },+ });+ const result = trigger();+ assert(result !== null, "role=combobox qualifies");+ assert(+ result.role === "combobox",+ `matched role token must be emitted, got ${result && result.role}`,+ );+ assert(+ result.hasPopup === null,+ `non-allowlisted aria-haspopup must not be emitted, got ${result && JSON.stringify(result.hasPopup)}`,+ );+});++test("both matched tokens are emitted (combobox + menu)", () => {+ el({+ tagName: "DIV",+ attributes: {+ role: "combobox",+ "aria-haspopup": "menu",+ "aria-expanded": "true",+ },+ });+ const result = trigger();+ assert(result !== null, "combobox + menu qualifies");+ assert(+ result.role === "combobox",+ `role should be combobox, got ${result && result.role}`,+ );+ assert(+ result.hasPopup === "menu",+ `hasPopup should be menu, got ${result && result.hasPopup}`,+ );+});++console.log(`\n${passed + failed} tests: ${passed} passed, ${failed} failed`);+process.exit(failed > 0 ? 1 : 0);diff --git a/tests/unit/test_dom_scrape_crash_guards.py b/tests/unit/test_dom_scrape_crash_guards.pyindex 72442d778..5544e5730 100644--- a/tests/unit/test_dom_scrape_crash_guards.py+++ b/tests/unit/test_dom_scrape_crash_guards.py@@ -49,6 +49,17 @@ class TestDomScrapeCrashGuards:)assert result.returncode == 0, f"Failed:\n{result.stdout}\n{result.stderr}"+ def test_aria_popup_trigger_behavioral(self):+ script = Path(__file__).parent / "test_aria_popup_trigger_domutils.js"+ assert script.exists(), f"Missing {script}"+ result = subprocess.run(+ [_NODE, str(script)],+ capture_output=True,+ text=True,+ timeout=30,+ )+ assert result.returncode == 0, f"Failed:\n{result.stdout}\n{result.stderr}"+def test_injection_scope_isolation(self, tmp_path):from skyvern.webeye.utils.page import load_js_scriptdiff --git a/tests/unit/test_incompatible_element_guards.py b/tests/unit/test_incompatible_element_guards.pyindex b12c9157a..cdf964d21 100644--- a/tests/unit/test_incompatible_element_guards.py+++ b/tests/unit/test_incompatible_element_guards.py@@ -6,7 +6,12 @@ import pytestfrom playwright.async_api import Error as PlaywrightErrorfrom skyvern.exceptions import InvalidElementForTextInput-from skyvern.webeye.actions.handler import get_input_value+from skyvern.webeye.actions.handler import (+ _is_selected_engine_error,+ _is_selected_engine_timeout,+ get_input_value,+)+from skyvern.webeye.browser_engine import BrowserEngineMetadata, BrowserEngineSelectionfrom skyvern.webeye.utils.dom import SkyvernElement, is_incompatible_text_input_error_NOT_INPUT_ERROR = "Element is not an <input>, <textarea> or [contenteditable] element"@@ -15,6 +20,54 @@ _NOT_INPUTELEMENT_ERROR = "Node is not an HTMLInputElement, HTMLTextAreaElement_TIMEOUT_ERROR = "Timeout 30000ms exceeded"+class _EngineError(Exception):+ pass+++class _EngineTimeout(_EngineError):+ pass+++async def _never_start(): # pragma: no cover - never awaited+ raise AssertionError("start_driver must not be called")+++def _engine_selection() -> BrowserEngineSelection:+ return BrowserEngineSelection(+ name="engine-a",+ start_driver=_never_start,+ error_type=_EngineError,+ timeout_error_type=_EngineTimeout,+ metadata=BrowserEngineMetadata(name="engine-a", version="0.0.0"),+ selection_reason="test",+ )+++def test_is_selected_engine_timeout_falls_back_to_stock_playwright_when_no_selection() -> None:+ from playwright.async_api import TimeoutError as PlaywrightTimeoutError++ assert _is_selected_engine_timeout(PlaywrightTimeoutError("boom"), None) is True+ assert _is_selected_engine_timeout(_EngineTimeout("boom"), None) is False+++def test_is_selected_engine_timeout_uses_selected_engine_identity() -> None:+ from playwright.async_api import TimeoutError as PlaywrightTimeoutError++ selection = _engine_selection()+ assert _is_selected_engine_timeout(_EngineTimeout("boom"), selection) is True+ # A foreign (stock Playwright) timeout is NOT this engine's timeout.+ assert _is_selected_engine_timeout(PlaywrightTimeoutError("boom"), selection) is False+++def test_is_selected_engine_error_uses_selected_engine_identity() -> None:+ selection = _engine_selection()+ assert _is_selected_engine_error(_EngineError("boom"), selection) is True+ assert _is_selected_engine_error(PlaywrightError("boom"), selection) is False+ # No selection keeps the stock Playwright identity.+ assert _is_selected_engine_error(PlaywrightError("boom"), None) is True+ assert _is_selected_engine_error(_EngineError("boom"), None) is False++def _make_element(tag_name: str, locator: MagicMock) -> SkyvernElement:return SkyvernElement(locator, MagicMock(), {"id": "el-1", "tagName": tag_name})@@ -66,6 +119,33 @@ async def test_input_fill_reraises_unrelated_errors() -> None:await element.input_fill("hello")+async def test_get_input_value_translates_selected_engine_incompatible_read() -> None:+ # Under a non-stock selected engine, an incompatible-node read raised as THAT engine's error+ # must still be translated to "value unknown" (None), not escape raw.+ locator = MagicMock()+ locator.input_value = AsyncMock(side_effect=_EngineError(_NOT_INPUTELEMENT_ERROR))+ assert await get_input_value("input", locator, engine_selection=_engine_selection()) is None+++async def test_get_input_value_reraises_foreign_error_under_selected_engine() -> None:+ # A stock Playwright error is foreign to the pinned engine; it must propagate untranslated+ # instead of being silently swallowed as an incompatible read.+ locator = MagicMock()+ locator.input_value = AsyncMock(side_effect=PlaywrightError(_NOT_INPUTELEMENT_ERROR))+ with pytest.raises(PlaywrightError):+ await get_input_value("input", locator, engine_selection=_engine_selection())+++async def test_get_input_value_reraises_selected_engine_non_incompatible_error() -> None:+ locator = MagicMock()+ locator.input_value = AsyncMock(side_effect=_EngineError(_TIMEOUT_ERROR))+ with pytest.raises(_EngineError):+ await get_input_value("input", locator, engine_selection=_engine_selection())++@pytest.mark.asyncioasync def test_input_clear_classifies_incompatible_element(monkeypatch: pytest.MonkeyPatch) -> None:element = _make_element("button", MagicMock())diff --git a/tests/unit/test_input_text_incremental_timeout_reraise.py b/tests/unit/test_input_text_incremental_timeout_reraise.pyindex bfe20a940..b7f22122a 100644--- a/tests/unit/test_input_text_incremental_timeout_reraise.py+++ b/tests/unit/test_input_text_incremental_timeout_reraise.py@@ -21,6 +21,7 @@ from skyvern.forge.sdk.models import StepStatusfrom skyvern.webeye.actions.actions import InputOrSelectContext, InputTextActionfrom skyvern.webeye.actions.handler import handle_input_text_actionfrom skyvern.webeye.actions.responses import ActionSuccess+from skyvern.webeye.browser_engine import BrowserEngineMetadata, BrowserEngineSelectionfrom skyvern.webeye.scraper.scraper import IncrementalScrapePagefrom tests.unit.conftest import make_input_element_mockfrom tests.unit.helpers import make_organization, make_step, make_task@@ -31,6 +32,29 @@ _TASK = make_task(_NOW, _ORG, navigation_payload={}, navigation_goal="Fill the f_STEP = make_step(_NOW, _TASK, step_id="stp-1", status=StepStatus.created, order=0, output=None)+class _EngineError(Exception):+ pass+++class _EngineTimeout(_EngineError):+ pass+++async def _never_start(): # pragma: no cover - never awaited+ raise AssertionError("start_driver must not be called")+++def _engine_selection() -> BrowserEngineSelection:+ return BrowserEngineSelection(+ name="engine-a",+ start_driver=_never_start,+ error_type=_EngineError,+ timeout_error_type=_EngineTimeout,+ metadata=BrowserEngineMetadata(name="engine-a", version="0.0.0"),+ selection_reason="test",+ )++@pytest.mark.asyncioasync def test_incremental_element_tree_propagates_when_both_attempts_time_out() -> None:"""Both the wait-until-finished attempt and the no-wait retry raising ``SkyvernPageAnalysisTimeout``@@ -50,7 +74,9 @@ async def test_incremental_element_tree_propagates_when_both_attempts_time_out()assert skyvern_frame.get_incremental_element_tree.await_count == 2-async def _run_input_with_incremental_error(error: BaseException) -> list:+async def _run_input_with_incremental_error(+ error: BaseException, engine_selection: BrowserEngineSelection | None = None+) -> list:skyvern_el = make_input_element_mock(element_id="AADC")dom_instance = MagicMock()dom_instance.get_skyvern_element_by_id = AsyncMock(return_value=skyvern_el)@@ -79,6 +105,10 @@ async def _run_input_with_incremental_error(error: BaseException) -> list:return_value="123456",),patch("skyvern.webeye.actions.handler._get_input_or_select_context", new=AsyncMock(return_value=context)),+ patch(+ "skyvern.webeye.actions.handler.resolve_engine_selection_for_task",+ return_value=engine_selection,+ ),):return await handle_input_text_action(action=action, page=MagicMock(), scraped_page=scraped_page, task=_TASK, step=_STEP@@ -102,3 +132,36 @@ async def test_input_action_still_swallows_unrelated_incremental_error() -> Nonesemantic timeout only."""results = await _run_input_with_incremental_error(RuntimeError("unexpected DOM state"))assert len(results) == 1 and isinstance(results[0], ActionSuccess)+++async def test_input_action_tolerates_selected_engine_navigation_error() -> None:+ """A pinned non-stock engine's navigation error during incremental processing is tolerated+ exactly like the stock Playwright navigation error (silently continue, action succeeds)."""+ results = await _run_input_with_incremental_error(+ _EngineError("Execution context was destroyed"), engine_selection=_engine_selection()+ )+ assert len(results) == 1 and isinstance(results[0], ActionSuccess)+++async def test_input_action_reraises_selected_engine_non_navigation_error() -> None:+ """A pinned engine's non-navigation error must be re-raised, mirroring the stock+ Playwright-error contract, instead of being swallowed into an ActionSuccess."""+ with pytest.raises(_EngineError):+ await _run_input_with_incremental_error(+ _EngineError("something genuinely unexpected"), engine_selection=_engine_selection()+ )++[email protected]("message", ["Execution context was destroyed", "something genuinely unexpected"])+async def test_input_action_reraises_foreign_error_under_selected_engine(message: str) -> None:+ """Under a pinned non-stock engine, a stock Playwright error is foreign — it is not one of this+ engine's tolerances, so it must propagate exactly as the pre-PR ``except PlaywrightError`` re-raised+ it, never fall through to a false ``ActionSuccess``. This holds even for a navigation-shaped message:+ the navigation tolerance belongs to the selected engine's own errors, not to a foreign driver."""+ from playwright.async_api import Error as PlaywrightError++ with pytest.raises(PlaywrightError):+ await _run_input_with_incremental_error(PlaywrightError(message), engine_selection=_engine_selection())diff --git a/tests/unit/test_input_text_wrapper_retarget.py b/tests/unit/test_input_text_wrapper_retarget.pyindex 4c5babaf5..db2422fb6 100644--- a/tests/unit/test_input_text_wrapper_retarget.py+++ b/tests/unit/test_input_text_wrapper_retarget.py@@ -92,7 +92,7 @@ async def _run(action: InputTextAction, wrapper: MagicMock, child: MagicMock | N# successful retarget short-circuits to success, proving the child drove the pipeline.read_tags: list[str] = []- async def _get_input_value(tag_name: str, locator: object) -> str:+ async def _get_input_value(tag_name: str, locator: object, engine_selection: object = None) -> str:read_tags.append(tag_name)return _CARD if tag_name == "input" else ""diff --git a/tests/unit/test_record_artifacts_popup_preserving.py b/tests/unit/test_record_artifacts_popup_preserving.pynew file mode 100644index 000000000..3bc12e69d--- /dev/null+++ b/tests/unit/test_record_artifacts_popup_preserving.py@@ -0,0 +1,300 @@+"""Post-action transient-UI capture gate.++record_artifacts_after_action takes the LLM-facing SCREENSHOT_ACTION. Under the treatment arm+of the PRESERVE_TRANSIENT_UI_CAPTURE experiment, an open ARIA popup forces scrolling_number=0+so the scroll events of a multi-page capture do not dismiss the popup before the next step sees+it. Control shadow-detects only (scrolling unchanged); off never detects. CUA keeps its existing+scrolling_number=0 independently of the experiment arm.+"""++from __future__ import annotations++import contextlib+from unittest.mock import AsyncMock, MagicMock, patch++import pytest++from skyvern.constants import DEFAULT_MAX_SCREENSHOT_SCROLLS+from skyvern.forge.agent import ForgeAgent+from skyvern.forge.sdk.core import skyvern_context+from skyvern.forge.sdk.core.skyvern_context import SkyvernContext+from skyvern.schemas.run_enums import RunEngine+++async def _run(+ *,+ engine: RunEngine,+ popup_trigger: dict | None,+ arm_value: bool | None,+ context: SkyvernContext | None = None,+):+ """Drive record_artifacts_after_action with stubbed IO, yielding (frame_mock, scrolling_number).+ arm_value maps to the cached experiment assignment: True=treatment, False=control, None=off.+ ``context`` lets a caller reuse one SkyvernContext across successive post-action captures to+ exercise the shared per-run suppression cap."""+ working_page = MagicMock()++ frame_mock = MagicMock()+ frame_mock.safe_wait_for_animation_end = AsyncMock()+ frame_mock.get_scroll_x_y = AsyncMock(return_value=(0, 0))+ frame_mock.safe_scroll_to_x_y = AsyncMock()+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value=popup_trigger)+ # Force the html-artifact block to fail fast so we don't need the artifact manager.+ frame_mock.get_content = AsyncMock(side_effect=RuntimeError("stop after screenshot"))++ browser_state = MagicMock()+ browser_state.get_working_page = AsyncMock(return_value=working_page)+ browser_state.take_post_action_screenshot = AsyncMock(side_effect=RuntimeError("stop"))++ task = MagicMock()+ task.workflow_run_id = "wr_test"+ task.task_id = "tsk_test"+ task.organization_id = "o_test"+ step = MagicMock()+ step.order = 1+ step.is_speculative = False+ action = MagicMock()+ action.action_id = None+ action.organization_id = None++ skyvern_context.set(+ context+ or SkyvernContext(+ max_screenshot_scrolls=None,+ workflow_run_id="wr_test",+ preserve_transient_ui_capture=arm_value,+ )+ )++ try:+ with contextlib.ExitStack() as stack:+ skyvern_frame_cls = stack.enter_context(patch("skyvern.forge.agent.SkyvernFrame"))+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)++ agent = MagicMock(spec=ForgeAgent)+ await ForgeAgent.record_artifacts_after_action(+ agent, task=task, step=step, browser_state=browser_state, engine=engine, action=action+ )+ scrolling_number = browser_state.take_post_action_screenshot.await_args.kwargs["scrolling_number"]+ yield frame_mock, scrolling_number+ finally:+ skyvern_context.reset()+++async def test_treatment_open_popup_forces_zero_scroll() -> None:+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == 0, "treatment + open popup must force scrolling_number=0"+ frame_mock.get_open_aria_popup_trigger.assert_awaited()+++async def test_control_open_popup_shadow_detects_but_keeps_scroll() -> None:+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=False) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS, "control must not alter scrolling"+ frame_mock.get_open_aria_popup_trigger.assert_awaited() # shadow detection ran+++async def test_off_open_popup_keeps_scroll_and_skips_predicate() -> None:+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=None) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited() # off pays no shadow cost+++async def test_treatment_no_popup_keeps_default_scroll() -> None:+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=None, arm_value=True) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS+++async def test_cua_engine_stays_zero_and_skips_predicate_regardless_of_arm() -> None:+ async with _run(engine=RunEngine.openai_cua, popup_trigger={"role": "combobox"}, arm_value=True) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == 0+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()+++async def test_treatment_zero_scroll_still_reads_scroll_position_legacy() -> None:+ # Legacy always snapshots the scroll position (regardless of scrolling_number); treatment must+ # change ONLY the treatment decision (scrolling_number=0), not the legacy read/restore control+ # flow. Restoring to the unchanged position is a no-op window.scroll, so it does not scroll.+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == 0+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read preserved+++async def test_scrolling_capture_snapshots_scroll_position() -> None:+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=None, arm_value=True) as (+ frame_mock,+ scrolling_number,+ ):+ assert scrolling_number == DEFAULT_MAX_SCREENSHOT_SCROLLS+ frame_mock.get_scroll_x_y.assert_awaited_once()+++async def test_treatment_zero_scroll_records_screenshot_action_via_legacy_nesting() -> None:+ # Suppressing the scroll (scrolling_number=0) still records the LLM-facing SCREENSHOT_ACTION,+ # via the LEGACY nesting: get_scroll_x_y succeeds (0, 0) so x/y are known and the recording runs.+ from skyvern.forge import app++ frame_mock = MagicMock()+ frame_mock.safe_wait_for_animation_end = AsyncMock()+ frame_mock.get_scroll_x_y = AsyncMock(return_value=(0, 0))+ frame_mock.safe_scroll_to_x_y = AsyncMock()+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value={"role": "combobox"})+ frame_mock.get_content = AsyncMock(side_effect=RuntimeError("stop after screenshot"))++ browser_state = MagicMock()+ browser_state.get_working_page = AsyncMock(return_value=MagicMock())+ browser_state.take_post_action_screenshot = AsyncMock(return_value=b"img")++ task = MagicMock()+ task.workflow_run_id = "wr_test"+ task.task_id = "tsk_test"+ task.organization_id = "o_test"+ step = MagicMock()+ step.order = 1+ step.is_speculative = False+ action = MagicMock()+ action.action_id = None+ action.organization_id = None++ skyvern_context.set(+ SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_test", preserve_transient_ui_capture=True)+ )+ try:+ with contextlib.ExitStack() as stack:+ skyvern_frame_cls = stack.enter_context(patch("skyvern.forge.agent.SkyvernFrame"))+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)+ prepare = AsyncMock(return_value=None)+ stack.enter_context(patch.object(app.ARTIFACT_MANAGER, "prepare_llm_artifact", prepare))++ agent = MagicMock(spec=ForgeAgent)+ await ForgeAgent.record_artifacts_after_action(+ agent, task=task, step=step, browser_state=browser_state, engine=RunEngine.skyvern_v1, action=action+ )+ assert browser_state.take_post_action_screenshot.await_args.kwargs["scrolling_number"] == 0+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read+ frame_mock.safe_scroll_to_x_y.assert_awaited() # legacy restore (no-op to same position)+ prepare.assert_awaited() # artifact recorded via legacy nesting (x/y known)+ finally:+ skyvern_context.reset()+++async def test_off_scroll_fetch_failure_drops_screenshot_action_legacy() -> None:+ # OFF must follow EXACT legacy artifact semantics: legacy nested the SCREENSHOT_ACTION recording+ # inside `if x is not None and y is not None`, so a failing get_scroll_x_y drops the screenshot.+ # (Decoupling the recording from the scroll snapshot is an unrelated hardening, deferred out of+ # this PR so OFF keeps legacy control flow.)+ from skyvern.forge import app++ frame_mock = MagicMock()+ frame_mock.safe_wait_for_animation_end = AsyncMock()+ frame_mock.get_scroll_x_y = AsyncMock(side_effect=RuntimeError("scroll read failed"))+ frame_mock.safe_scroll_to_x_y = AsyncMock()+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value=None)+ frame_mock.get_content = AsyncMock(side_effect=RuntimeError("stop after screenshot"))++ browser_state = MagicMock()+ browser_state.get_working_page = AsyncMock(return_value=MagicMock())+ browser_state.take_post_action_screenshot = AsyncMock(return_value=b"img")++ task = MagicMock()+ task.workflow_run_id = "wr_test"+ task.task_id = "tsk_test"+ task.organization_id = "o_test"+ step = MagicMock()+ step.order = 1+ step.is_speculative = False+ action = MagicMock()+ action.action_id = None+ action.organization_id = None++ skyvern_context.set(+ SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_test", preserve_transient_ui_capture=None)+ )+ try:+ with contextlib.ExitStack() as stack:+ skyvern_frame_cls = stack.enter_context(patch("skyvern.forge.agent.SkyvernFrame"))+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)+ prepare = AsyncMock(return_value=None)+ stack.enter_context(patch.object(app.ARTIFACT_MANAGER, "prepare_llm_artifact", prepare))++ agent = MagicMock(spec=ForgeAgent)+ await ForgeAgent.record_artifacts_after_action(+ agent, task=task, step=step, browser_state=browser_state, engine=RunEngine.skyvern_v1, action=action+ )+ assert browser_state.take_post_action_screenshot.await_args.kwargs["scrolling_number"] > 0+ frame_mock.get_scroll_x_y.assert_awaited() # fetch was attempted (and raised)+ frame_mock.safe_scroll_to_x_y.assert_not_awaited() # no restore (x/y unknown)+ prepare.assert_not_awaited() # legacy: recording is nested under x/y-not-None, so it is dropped+ finally:+ skyvern_context.reset()+++# --- The post-action capture is the sibling of the agent-step scrape and must honor the same two+# invariants: bounded telemetry tokens, and the shared per-run consecutive-suppression cap. ---+++async def test_post_action_routes_telemetry_through_bounded_helper() -> None:+ # The post-action site must emit popup telemetry only through the shared, bounded+ # emit_transient_ui_popup_telemetry helper (allowlisted tokens + bounded boolean), never by+ # writing raw page-controlled role/aria-haspopup to the span. The helper itself is unit-tested+ # in test_transient_ui_capture_experiment.py; here we assert the wiring.+ trigger = {"role": "<script>evil", "hasPopup": "arbitrary-attacker-text", "controlsResolved": 0}+ with patch("skyvern.forge.agent.emit_transient_ui_popup_telemetry") as emit:+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=trigger, arm_value=True):+ pass+ emit.assert_called_once()+ assert emit.call_args.args[1] is trigger+++async def test_post_action_participates_in_shared_consecutive_cap() -> None:+ ctx = SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_cap", preserve_transient_ui_capture=True)+ scrolls = []+ for _ in range(3):+ async with _run(+ engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True, context=ctx+ ) as (_f, scrolling_number):+ scrolls.append(scrolling_number)+ assert scrolls[0] == 0 and scrolls[1] == 0, "first two post-action captures suppress"+ assert scrolls[2] == DEFAULT_MAX_SCREENSHOT_SCROLLS, "third consecutive capture falls back to legacy scrolling"+ assert ctx.transient_ui_consecutive_suppressions == 2+++async def test_post_action_no_popup_resets_shared_counter() -> None:+ ctx = SkyvernContext(max_screenshot_scrolls=None, workflow_run_id="wr_reset", preserve_transient_ui_capture=True)+ for _ in range(2):+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger={"role": "combobox"}, arm_value=True, context=ctx):+ pass+ assert ctx.transient_ui_consecutive_suppressions == 2+ async with _run(engine=RunEngine.skyvern_v1, popup_trigger=None, arm_value=True, context=ctx):+ pass+ assert ctx.transient_ui_consecutive_suppressions == 0diff --git a/tests/unit/test_scrape_popup_preserving_scroll.py b/tests/unit/test_scrape_popup_preserving_scroll.pynew file mode 100644index 000000000..2c84da429--- /dev/null+++ b/tests/unit/test_scrape_popup_preserving_scroll.py@@ -0,0 +1,345 @@+"""Scrape-site transient-UI capture gate.++Under the treatment arm of PRESERVE_TRANSIENT_UI_CAPTURE, when an ARIA popup (e.g. a portal+date-picker popup) is open, the next scrape's split screenshot must NOT scroll — scrolling+fires events that dismiss the portal popup before the just-built element tree's day-cell IDs+can be acted on. Control shadow-detects only (scrolls as before); off never detects.+"""++from __future__ import annotations++import contextlib+from unittest.mock import AsyncMock, MagicMock, patch++import pytest++from skyvern.forge.agent import ForgeAgent, ScrapeType+from skyvern.forge.sdk.core import skyvern_context+from skyvern.forge.sdk.core.skyvern_context import SkyvernContext+from skyvern.webeye.scraper import scraper+++def _make_page() -> MagicMock:+ page = MagicMock()+ page.url = "https://example.com"+ page.main_frame.url = "https://example.com"+ page.main_frame.child_frames = []+ page.viewport_size = {"width": 1280, "height": 800}+ return page+++_UNSET = object()+++def _span_capture():+ """Patch traced_span to yield a MagicMock span so tests can inspect emitted attributes."""+ span = MagicMock()++ @contextlib.contextmanager+ def _fake_traced_span(_tracer, _name):+ yield span++ with patch.object(scraper, "traced_span", new=_fake_traced_span):+ yield span+++def _span_attrs(span: MagicMock) -> dict:+ return {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}+++async def _run_scrape(+ *,+ popup_trigger: dict | None,+ arm_value: bool | None,+ scroll: bool = True,+ allow_transient_ui_suppression: object = True,+ context: SkyvernContext | None = None,+):+ """Drive scrape_web_unsafe with the heavy dependencies stubbed, yielding+ (take_split_screenshots_mock, frame_mock, span_mock). arm_value maps to the cached experiment+ assignment: True=treatment, False=control, None=off.++ allow_transient_ui_suppression is forwarded to scrape_web_unsafe unless left as ``_UNSET``, which+ simulates a caller (verify/extract/error path) that never opts into suppression. ``context`` lets a+ caller reuse one SkyvernContext across successive scrapes to exercise the per-run suppression cap.+ """+ page = _make_page()+ browser_state = MagicMock()+ browser_state.must_get_working_page = AsyncMock(return_value=page)++ frame_mock = MagicMock()+ frame_mock.get_scroll_x_y = AsyncMock(return_value=(0, 0))+ frame_mock.safe_scroll_to_x_y = AsyncMock()+ frame_mock.get_open_aria_popup_trigger = AsyncMock(return_value=popup_trigger)+ frame_mock.get_content = AsyncMock(return_value="<html></html>")++ element = {"id": "btn", "tagName": "button"}++ ctx = context or SkyvernContext(workflow_run_id="wr_test", preserve_transient_ui_capture=arm_value)+ skyvern_context.set(ctx)+ try:+ with contextlib.ExitStack() as stack:+ skyvern_frame_cls = stack.enter_context(patch.object(scraper, "SkyvernFrame"))+ skyvern_frame_cls.create_instance = AsyncMock(return_value=frame_mock)+ take_split = AsyncMock(return_value=[b"img"])+ skyvern_frame_cls.take_split_screenshots = take_split++ stack.enter_context(patch.object(scraper, "_wait_for_scrape_ready", new=AsyncMock()))+ stack.enter_context(+ patch.object(+ scraper,+ "get_interactable_element_tree",+ new=AsyncMock(return_value=([element], [element], [])),+ )+ )+ stack.enter_context(patch.object(scraper, "trim_element_tree", new=MagicMock(return_value=[])))+ stack.enter_context(+ patch.object(scraper, "build_element_dict", new=MagicMock(return_value=({}, {}, {}, {}, {})))+ )+ stack.enter_context(patch.object(scraper, "get_frame_text", new=AsyncMock(return_value="")))+ stack.enter_context(patch.object(scraper, "advance_observation_epoch", new=MagicMock()))+ stack.enter_context(patch.object(scraper, "_record_scrape_span_attrs", new=MagicMock()))+ span = stack.enter_context(_span_capture())++ kwargs: dict = dict(+ browser_state=browser_state,+ url="https://example.com",+ cleanup_element_tree=AsyncMock(return_value=[element]),+ scroll=scroll,+ )+ if allow_transient_ui_suppression is not _UNSET:+ kwargs["allow_transient_ui_suppression"] = allow_transient_ui_suppression+ await scraper.scrape_web_unsafe(**kwargs)+ yield take_split, frame_mock, span+ finally:+ skyvern_context.reset()+++async def test_treatment_open_popup_suppresses_only_the_scroll_argument() -> None:+ # Treatment differs from legacy ONLY in the scroll argument to take_split_screenshots. The+ # scroll-position read + restore stay exactly as legacy (always-attempt); restoring to the+ # unchanged position is a no-op window.scroll that does not dismiss the popup.+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True) as (take_split, frame_mock, _span):+ assert take_split.await_args.kwargs["scroll"] is False, "treatment + open popup must force scroll=False"+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read preserved+ frame_mock.safe_scroll_to_x_y.assert_awaited() # legacy restore preserved+++async def test_off_scroll_false_preserves_legacy_read_and_restore() -> None:+ # OFF must follow EXACT legacy control flow: legacy read the scroll position and restored it+ # unconditionally, even when scroll=False. The current effective_scroll gate skips the read+ # (RED); OFF must keep the legacy always-attempt read + restore.+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=None, scroll=False) as (+ take_split,+ frame_mock,+ _span,+ ):+ assert take_split.await_args.kwargs["scroll"] is False+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited() # OFF never detects+ frame_mock.get_scroll_x_y.assert_awaited() # legacy always-attempt read+ frame_mock.safe_scroll_to_x_y.assert_awaited() # legacy restore+++async def test_treatment_no_popup_scrolls_and_restores_as_before() -> None:+ async with _run_scrape(popup_trigger=None, arm_value=True) as (take_split, frame_mock, _span):+ assert take_split.await_args.kwargs["scroll"] is True+ frame_mock.get_open_aria_popup_trigger.assert_awaited()+ frame_mock.get_scroll_x_y.assert_awaited()+ frame_mock.safe_scroll_to_x_y.assert_awaited()+++async def test_control_open_popup_shadow_detects_but_keeps_scroll() -> None:+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=False) as (take_split, frame_mock, _span):+ assert take_split.await_args.kwargs["scroll"] is True, "control must not alter scrolling"+ frame_mock.get_open_aria_popup_trigger.assert_awaited() # shadow detection ran+++async def test_off_open_popup_scrolls_and_skips_predicate() -> None:+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=None) as (take_split, frame_mock, _span):+ assert take_split.await_args.kwargs["scroll"] is True, "off must preserve status-quo scrolling"+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()+++async def test_caller_scroll_false_short_circuits_predicate() -> None:+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, scroll=False) as (+ take_split,+ frame_mock,+ _span,+ ):+ assert take_split.await_args.kwargs["scroll"] is False+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()+++# --- MUST-FIX 3: suppression is opt-in per call. A caller that does not opt in (verify /+# extraction / error-detection scrapes) keeps legacy scrolling even under treatment + open popup. ---+++async def test_no_optin_caller_keeps_legacy_scroll_and_skips_detection() -> None:+ async with _run_scrape(+ popup_trigger={"role": "combobox"}, arm_value=True, allow_transient_ui_suppression=_UNSET+ ) as (take_split, frame_mock, _span):+ assert take_split.await_args.kwargs["scroll"] is True, "non-opted-in scrape must stay legacy"+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited(), "no detection off the agent-step path"+++async def test_explicit_disallow_keeps_legacy_scroll() -> None:+ async with _run_scrape(+ popup_trigger={"role": "combobox"}, arm_value=True, allow_transient_ui_suppression=False+ ) as (take_split, frame_mock, _span):+ assert take_split.await_args.kwargs["scroll"] is True+ frame_mock.get_open_aria_popup_trigger.assert_not_awaited()+++async def test_optin_agent_step_suppresses() -> None:+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, allow_transient_ui_suppression=True) as (+ take_split,+ frame_mock,+ _span,+ ):+ assert take_split.await_args.kwargs["scroll"] is False, "opted-in agent-step scrape suppresses"+++# --- MUST-FIX 2: at most two consecutive suppressing captures per run, then fall back to legacy+# scrolling; reset the counter when no qualifying popup is detected. ---+++async def test_consecutive_suppression_capped_at_two_then_legacy() -> None:+ ctx = SkyvernContext(workflow_run_id="wr_cap", preserve_transient_ui_capture=True)+ scroll_args = []+ for _ in range(3):+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, context=ctx) as (+ take_split,+ _frame,+ _span,+ ):+ scroll_args.append(take_split.await_args.kwargs["scroll"])+ assert scroll_args == [False, False, True], f"expected suppress, suppress, fall back; got {scroll_args}"+ assert ctx.transient_ui_consecutive_suppressions == 2+++async def test_no_popup_resets_consecutive_counter() -> None:+ ctx = SkyvernContext(workflow_run_id="wr_reset", preserve_transient_ui_capture=True)+ # burn the cap+ for _ in range(2):+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, context=ctx):+ pass+ assert ctx.transient_ui_consecutive_suppressions == 2+ # a scrape with no qualifying popup resets the counter+ async with _run_scrape(popup_trigger=None, arm_value=True, context=ctx):+ pass+ assert ctx.transient_ui_consecutive_suppressions == 0+ # ...so the next open popup suppresses again+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=True, context=ctx) as (+ take_split,+ _frame,+ _span,+ ):+ assert take_split.await_args.kwargs["scroll"] is False+++async def test_off_arm_never_touches_counter() -> None:+ ctx = SkyvernContext(workflow_run_id="wr_off", preserve_transient_ui_capture=None)+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=None, context=ctx) as (+ take_split,+ _frame,+ _span,+ ):+ assert take_split.await_args.kwargs["scroll"] is True+ assert ctx.transient_ui_consecutive_suppressions == 0+++async def test_control_arm_never_suppresses_and_counter_stays_zero() -> None:+ ctx = SkyvernContext(workflow_run_id="wr_ctrl", preserve_transient_ui_capture=False)+ for _ in range(3):+ async with _run_scrape(popup_trigger={"role": "combobox"}, arm_value=False, context=ctx) as (+ take_split,+ _frame,+ _span,+ ):+ assert take_split.await_args.kwargs["scroll"] is True+ assert ctx.transient_ui_consecutive_suppressions == 0+++# --- MUST-FIX 4: page-controlled role / aria-haspopup must not reach the span verbatim. Only+# allowlisted matched tokens are emitted. ---+++async def test_arbitrary_role_and_haspopup_not_emitted_to_span() -> None:+ trigger = {"role": "<script>evil", "hasPopup": "arbitrary-attacker-text", "controlsResolved": 0}+ async with _run_scrape(popup_trigger=trigger, arm_value=True) as (_take_split, _frame, span):+ attrs = _span_attrs(span)+ assert attrs.get("transient_ui_role") != "<script>evil"+ assert attrs.get("transient_ui_haspopup") != "arbitrary-attacker-text"+ assert "transient_ui_role" not in attrs, "non-allowlisted role must be omitted"+ assert "transient_ui_haspopup" not in attrs, "non-allowlisted haspopup must be omitted"+++async def test_allowlisted_role_and_haspopup_are_emitted() -> None:+ trigger = {"role": "combobox", "hasPopup": "menu", "controlsResolved": 1}+ async with _run_scrape(popup_trigger=trigger, arm_value=True) as (_take_split, _frame, span):+ attrs = _span_attrs(span)+ assert attrs.get("transient_ui_role") == "combobox"+ assert attrs.get("transient_ui_haspopup") == "menu"+++# --- Optional 5: bounded boolean controlsResolved telemetry so the false-positive source+# (no-target portal fallback) is measurable. ---+++async def test_controls_resolved_boolean_emitted_true_and_false() -> None:+ resolved = {"role": "combobox", "hasPopup": "menu", "controlsResolved": 2}+ async with _run_scrape(popup_trigger=resolved, arm_value=True) as (_take_split, _frame, span):+ assert _span_attrs(span).get("transient_ui_controls_resolved") is True+ fallback = {"role": "combobox", "hasPopup": None, "controlsResolved": 0}+ async with _run_scrape(popup_trigger=fallback, arm_value=True) as (_take_split, _frame, span):+ assert _span_attrs(span).get("transient_ui_controls_resolved") is False+++# --- MUST-FIX 3 call-site wiring: the agent-step scrape (ForgeAgent._scrape_with_type) is the only+# call site that opts into suppression. verify / extraction / error-detection paths call+# scrape_website without the flag, so they inherit the legacy default (proven above). ---+++async def test_agent_step_scrape_opts_into_suppression() -> None:+ browser_state = MagicMock()+ browser_state.scrape_website = AsyncMock(return_value=MagicMock())++ task = MagicMock()+ task.url = "https://example.com"+ step = MagicMock()++ fake_app = MagicMock()+ fake_app.AGENT_FUNCTION.cleanup_element_tree_factory = MagicMock(return_value=AsyncMock())+ fake_app.scrape_exclude = AsyncMock()++ with patch("skyvern.forge.agent.app", fake_app):+ # unbound call with a mock self: _scrape_with_type only touches module-level app/settings.+ await ForgeAgent._scrape_with_type(+ MagicMock(),+ task=task,+ step=step,+ browser_state=browser_state,+ scrape_type=ScrapeType.NORMAL,+ engine=MagicMock(), # not a CUA engine+ )++ assert browser_state.scrape_website.await_args.kwargs["allow_transient_ui_suppression"] is Truediff --git a/tests/unit/test_transient_ui_capture_experiment.py b/tests/unit/test_transient_ui_capture_experiment.pynew file mode 100644index 000000000..4067f688f--- /dev/null+++ b/tests/unit/test_transient_ui_capture_experiment.py@@ -0,0 +1,298 @@+"""Per-run resolution of the PRESERVE_TRANSIENT_UI_CAPTURE experiment.++Off/unenrolled is the safe default: the flag only assigns treatment when a real provider resolves+True. Undefined, no-provider, resolver error, and missing run identity all stay off/unenrolled+(the predicate is not evaluated and scrolling is unchanged). An explicit provider False is the+distinct enrolled control arm (shadow-detect only, still scrolls).+"""++from __future__ import annotations++import asyncio+from unittest.mock import AsyncMock, MagicMock++import pytest++from skyvern.forge.sdk.core.skyvern_context import SkyvernContext+from skyvern.forge.sdk.experimentation import transient_ui_capture as tuc+from skyvern.forge.sdk.experimentation.providers import NoOpExperimentationProvider+++def _ctx(**kwargs: object) -> SkyvernContext:+ return SkyvernContext(**kwargs) # type: ignore[arg-type]+++async def test_provider_true_caches_treatment(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1", organization_id="o_1", workflow_permanent_id="wpid_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert ctx.preserve_transient_ui_capture is True+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"+ provider.resolve_feature_flag_cached.assert_awaited_once()+ # run-level distinct_id, org/wpid as properties (per backend.md convention)+ call = provider.resolve_feature_flag_cached.await_args+ assert call.args[0] == tuc.PRESERVE_TRANSIENT_UI_CAPTURE_FLAG+ assert call.args[1] == "wr_1"+ assert call.kwargs["properties"]["organization_id"] == "o_1"+++async def test_provider_false_caches_control(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=False)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert ctx.preserve_transient_ui_capture is False+ assert tuc.transient_ui_capture_arm(ctx) == "control"+++async def test_provider_undefined_none_is_off(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=None)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert ctx.preserve_transient_ui_capture is None+ assert tuc.transient_ui_capture_arm(ctx) == "off"+++async def test_noop_provider_is_off_without_querying_provider(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = NoOpExperimentationProvider()+ spy = AsyncMock(return_value=True)+ monkeypatch.setattr(provider, "resolve_feature_flag_cached", spy)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ spy.assert_not_awaited()+++async def test_provider_error_is_off_and_does_not_raise(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(side_effect=RuntimeError("posthog down"))+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+++async def test_missing_run_identity_is_off_without_querying_provider(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx()+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ provider.resolve_feature_flag_cached.assert_not_awaited()+++def test_arm_helper_missing_context_is_off() -> None:+ assert tuc.transient_ui_capture_arm(None) == "off"+++def test_arm_helper_uninitialized_field_is_off() -> None:+ assert tuc.transient_ui_capture_arm(_ctx(workflow_run_id="wr_1")) == "off"+++async def test_treatment_arm_is_sticky_across_provider_change(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"+ # TTL expiry / mid-run ramp flips the provider result — the run's arm must NOT change.+ provider.resolve_feature_flag_cached.return_value = False+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"+ provider.resolve_feature_flag_cached.assert_awaited_once()+++async def test_control_arm_is_sticky_across_provider_change(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=False)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "control"+ provider.resolve_feature_flag_cached.return_value = True+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "control"+ provider.resolve_feature_flag_cached.assert_awaited_once()+++async def test_off_undefined_arm_is_sticky_across_provider_change(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=None)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ provider.resolve_feature_flag_cached.return_value = True+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ provider.resolve_feature_flag_cached.assert_awaited_once()+++async def test_resolver_error_arm_is_sticky_off(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(side_effect=RuntimeError("posthog down"))+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ # Provider recovers and would now return treatment — the pinned off must not flip.+ recovered = AsyncMock(return_value=True)+ monkeypatch.setattr(provider, "resolve_feature_flag_cached", recovered)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ recovered.assert_not_awaited()+++async def test_missing_identity_arm_is_sticky_off(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx() # no run identity+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ provider.resolve_feature_flag_cached.assert_not_awaited()+++async def test_noop_provider_is_sticky_off_without_calls(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_1")+ provider = NoOpExperimentationProvider()+ spy = AsyncMock(return_value=True)+ monkeypatch.setattr(provider, "resolve_feature_flag_cached", spy)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "off"+ spy.assert_not_awaited()+++async def test_explicit_distinct_id_override_used_over_context(monkeypatch: pytest.MonkeyPatch) -> None:+ # An execution-boundary caller (e.g. an inline child workflow) whose context lacks full run+ # identity may pass distinct_id / organization_id / workflow_permanent_id explicitly.+ ctx = _ctx(run_id="parent_run", organization_id="o_ctx")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=True)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(+ ctx, distinct_id="wr_child", organization_id="o_child", workflow_permanent_id="wpid_child"+ )+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"+ call = provider.resolve_feature_flag_cached.await_args+ assert call.args[1] == "wr_child" # explicit distinct_id, NOT the context's parent run_id+ assert call.kwargs["properties"]["organization_id"] == "o_child"+ assert call.kwargs["properties"]["workflow_permanent_id"] == "wpid_child"+++async def test_override_falls_back_to_context_when_not_given(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _ctx(workflow_run_id="wr_ctx", organization_id="o_ctx")+ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(return_value=False)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)+ await tuc.resolve_transient_ui_capture_arm(ctx)+ assert tuc.transient_ui_capture_arm(ctx) == "control"+ assert provider.resolve_feature_flag_cached.await_args.args[1] == "wr_ctx"+++async def test_concurrent_first_resolve_queries_provider_once(monkeypatch: pytest.MonkeyPatch) -> None:+ # Two coroutines sharing one SkyvernContext both hit the first-resolve path. A single-flight lock+ # must keep the provider queried exactly once (the ADR's "provider ≤ once/run" invariant).+ ctx = _ctx(workflow_run_id="wr_1")+ calls = {"n": 0}++ async def _slow_resolve(*_a: object, **_k: object) -> bool:+ calls["n"] += 1+ await asyncio.sleep(0) # yield so a concurrent first-resolver can interleave before we pin+ return True++ provider = MagicMock()+ provider.resolve_feature_flag_cached = AsyncMock(side_effect=_slow_resolve)+ monkeypatch.setattr(tuc.app, "EXPERIMENTATION_PROVIDER", provider)++ await asyncio.gather(+ tuc.resolve_transient_ui_capture_arm(ctx),+ tuc.resolve_transient_ui_capture_arm(ctx),+ )++ assert calls["n"] == 1, "two concurrent first-resolvers on one context must query the provider once"+ assert tuc.transient_ui_capture_arm(ctx) == "treatment"+++# --- Shared capture helpers used by BOTH the agent-step scrape and the post-action screenshot ---+++def test_emit_telemetry_omits_arbitrary_page_controlled_tokens() -> None:+ span = MagicMock()+ tuc.emit_transient_ui_popup_telemetry(+ span, {"role": "<script>evil", "hasPopup": "arbitrary-attacker-text", "controlsResolved": 0}+ )+ keys = {call.args[0] for call in span.set_attribute.call_args_list}+ assert "transient_ui_role" not in keys, "non-allowlisted role must be omitted"+ assert "transient_ui_haspopup" not in keys, "non-allowlisted haspopup must be omitted"+++def test_emit_telemetry_emits_allowlisted_tokens_and_controls_boolean() -> None:+ span = MagicMock()+ tuc.emit_transient_ui_popup_telemetry(span, {"role": "combobox", "hasPopup": "menu", "controlsResolved": 2})+ attrs = {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}+ assert attrs["transient_ui_role"] == "combobox"+ assert attrs["transient_ui_haspopup"] == "menu"+ assert attrs["transient_ui_controls_resolved"] is True+++def test_emit_telemetry_controls_boolean_false_for_no_target_fallback() -> None:+ span = MagicMock()+ tuc.emit_transient_ui_popup_telemetry(span, {"role": "combobox", "hasPopup": None, "controlsResolved": 0})+ attrs = {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list}+ assert attrs["transient_ui_controls_resolved"] is False+++def test_decide_suppression_treatment_caps_at_two_then_falls_back() -> None:+ ctx = _ctx(workflow_run_id="wr_cap", preserve_transient_ui_capture=True)+ d1 = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)+ d2 = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)+ d3 = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)+ assert (d1.suppress, d2.suppress, d3.suppress) == (True, True, False)+ assert d3.capped is True+ assert ctx.transient_ui_consecutive_suppressions == 2+++def test_decide_suppression_resets_when_not_detected() -> None:+ ctx = _ctx(workflow_run_id="wr_reset", preserve_transient_ui_capture=True)+ tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)+ tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=True)+ assert ctx.transient_ui_consecutive_suppressions == 2+ d = tuc.decide_transient_ui_suppression(ctx, tuc.ARM_TREATMENT, detected=False)+ assert d.suppress is False and d.capped is False+ assert ctx.transient_ui_consecutive_suppressions == 0+++def test_decide_suppression_control_and_off_never_suppress_or_touch_counter() -> None:+ for arm in (tuc.ARM_CONTROL, tuc.ARM_OFF):+ ctx = _ctx(workflow_run_id="wr_arm", preserve_transient_ui_capture=False)+ d = tuc.decide_transient_ui_suppression(ctx, arm, detected=True)+ assert d.suppress is False and d.capped is False+ assert ctx.transient_ui_consecutive_suppressions == 0diff --git a/tests/unit/test_upload_file_wait.py b/tests/unit/test_upload_file_wait.pyindex 292ac4c4a..9fc46e746 100644--- a/tests/unit/test_upload_file_wait.py+++ b/tests/unit/test_upload_file_wait.py@@ -15,6 +15,39 @@ from playwright.async_api import TimeoutError as PlaywrightTimeoutErrorfrom skyvern.exceptions import SkyvernPageAnalysisTimeoutfrom skyvern.webeye.actions import handler as handler_modulefrom skyvern.webeye.actions.handler import _wait_for_upload_processing+from skyvern.webeye.browser_engine import BrowserEngineMetadata, BrowserEngineSelection+++class _EngineError(Exception):+ pass+++class _EngineTimeout(_EngineError):+ pass+++async def _never_start(): # pragma: no cover - never awaited+ raise AssertionError("start_driver must not be called")+++def _engine_selection() -> BrowserEngineSelection:+ return BrowserEngineSelection(+ name="engine-a",+ start_driver=_never_start,+ error_type=_EngineError,+ timeout_error_type=_EngineTimeout,+ metadata=BrowserEngineMetadata(name="engine-a", version="0.0.0"),+ selection_reason="test",+ )+++async def _run_with_page_ready_error(error: BaseException, engine_selection: BrowserEngineSelection | None) -> None:+ mock_frame = AsyncMock()+ mock_frame.wait_for_page_ready.side_effect = error+ with patch("skyvern.webeye.actions.handler.SkyvernFrame.create_instance", new_callable=AsyncMock) as mock_create:+ mock_create.return_value = mock_frame+ await _wait_for_upload_processing(AsyncMock(), engine_selection=engine_selection)+# ---------------------------------------------------------------------------# Helper behaviour tests@@ -88,6 +121,31 @@ async def test_propagates_non_playwright_error() -> None:await _wait_for_upload_processing(AsyncMock())+async def test_swallows_selected_engine_timeout() -> None:+ # A non-stock engine's native timeout must be tolerated like the stock Playwright timeout.+ await _run_with_page_ready_error(_EngineTimeout("deadline exceeded"), _engine_selection())+++async def test_swallows_selected_engine_error() -> None:+ await _run_with_page_ready_error(_EngineError("target closed"), _engine_selection())+++async def test_propagates_foreign_playwright_error_under_selected_engine() -> None:+ # Under a pinned non-stock engine, a stock Playwright error is foreign and must propagate,+ # not be swallowed as an upload-processing tolerance.+ with pytest.raises(PlaywrightError):+ await _run_with_page_ready_error(PlaywrightError("navigated away"), _engine_selection())+++async def test_propagates_foreign_playwright_timeout_under_selected_engine() -> None:+ with pytest.raises(PlaywrightTimeoutError):+ await _run_with_page_ready_error(PlaywrightTimeoutError("pw timeout"), _engine_selection())++# ---------------------------------------------------------------------------# Static regression guards# ---------------------------------------------------------------------------diff --git a/tests/unit/webeye/test_skyvern_frame_aria_popup.py b/tests/unit/webeye/test_skyvern_frame_aria_popup.pynew file mode 100644index 000000000..d07f75dc3--- /dev/null+++ b/tests/unit/webeye/test_skyvern_frame_aria_popup.py@@ -0,0 +1,51 @@+"""SkyvernFrame.get_open_aria_popup_trigger seam.++The Python wrapper evaluates the JS predicate through the frame evaluation seam and+MUST fail open (return None) on any evaluation error so screenshot-scroll policy keeps+the current scrolling behavior instead of crashing the scrape/action loop.+"""++from __future__ import annotations++from unittest.mock import AsyncMock, MagicMock, patch++import pytest++from skyvern.webeye.utils.page import SkyvernFrame+++def _frame() -> SkyvernFrame:+ return SkyvernFrame(frame=MagicMock())+++async def test_returns_dict_when_predicate_reports_open_popup() -> None:+ detail = {"role": "combobox", "hasPopup": "dialog", "tag": "div", "controlsResolved": 0}+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(return_value=detail)) as mock_eval:+ result = await _frame().get_open_aria_popup_trigger()+ assert result == detail+ # The wrapper must call the named JS predicate.+ assert "getOpenAriaPopupTrigger" in mock_eval.await_args.kwargs["expression"]+++async def test_returns_none_when_predicate_reports_no_popup() -> None:+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(return_value=None)):+ result = await _frame().get_open_aria_popup_trigger()+ assert result is None+++async def test_non_dict_result_coerced_to_none() -> None:+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(return_value="unexpected")):+ result = await _frame().get_open_aria_popup_trigger()+ assert result is None+++async def test_evaluation_exception_fails_open_with_warning() -> None:+ with patch.object(SkyvernFrame, "evaluate", new=AsyncMock(side_effect=RuntimeError("boom"))):+ with patch("skyvern.webeye.utils.page.LOG.warning") as mock_warn:+ result = await _frame().get_open_aria_popup_trigger()+ assert result is None, "an evaluation error must fail open to the default (scrolling) behavior"+ assert mock_warn.called, "the fail-open path must warn"