Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(mcp): Add a new tool to wait for several conditions (Chainable)

Karim shoairb68157f3f810
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 57032da..0af83ee 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index b149f9a..9703fb7 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -1,8 +1,8 @@
# Scrapling MCP Server
-The Scrapling MCP server exposes seventeen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
+The Scrapling MCP server exposes eighteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
-Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL). `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, and `browser_press_key` return plain text.
+Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL). `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, `browser_press_key`, and `browser_wait` return plain text.
## Shadow DOM
@@ -237,6 +237,42 @@ Use `["ControlOrMeta+A", "Backspace"]` to select all with the platform's modifie
Returns `Keys pressed.` as plain text without an automatic snapshot or a wait for navigation. An error or cancellation stops the remaining presses; completed actions are not undone or retried. The page reservation is released after success, failure, or cancellation. Use `browser_snapshot` to check partial effects before retrying. Invalid keys, unknown or HTTP sessions, and missing, closed, or busy pages return an error.
+### `browser_wait` -- Chain waits on the current page
+
+Runs native waits in order on the existing page in a dynamic or stealthy browser session without reloading it. Call `browser_fetch` first.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `session_id` | str | required | ID of an open browser session |
+| `actions` | list[object] | required | Nonempty ordered list of `time`, `element`, or `load` waits |
+
+| Action `type` | Fields | Behavior |
+|---------------|--------|----------|
+| `time` | `milliseconds` | Fixed pause in finite nonnegative milliseconds |
+| `element` | `selector`, `state="visible"`, `timeout=30000` | Wait for a nonempty Playwright selector to reach `"visible"`, `"hidden"`, `"attached"`, or `"detached"` |
+| `load` | `state`, `timeout=30000` | Wait for `"domcontentloaded"`, `"load"`, or `"networkidle"` |
+
+Element and load timeouts are finite nonnegative milliseconds per action; 0 disables the limit. Element waits are strict: multiple matches return an error. `attached` means present in the DOM, and `detached` means absent. `visible` requires a nonempty bounding box and no `visibility:hidden`; `hidden` also succeeds when the element is absent.
+
+Load states use the same native browser waits as the fetchers. An already reached state returns immediately for the current document. `domcontentloaded` waits for the DOMContentLoaded event, not all future JavaScript work. `load` waits for the load event. `networkidle` requires no active network connections for at least 500 ms; it does not prove the application is ready. Prefer a specific element state when it signals readiness.
+
+For example, wait for the document, results, and loading overlay in order:
+
+```json
+{
+ "session_id": "browser",
+ "actions": [
+ {"type": "load", "state": "domcontentloaded"},
+ {"type": "element", "selector": "#results", "state": "visible"},
+ {"type": "element", "selector": "#loading", "state": "hidden"}
+ ]
+}
+```
+
+MCP validates the whole list before execution. The page stays reserved for the full sequence and is released after success, failure, or cancellation. The first failure stops the remaining waits and reports the one-based action number, type, and native error. Cancellation also stops the sequence. Completed waits are not undone or retried.
+
+Returns `Wait completed.` as plain text without a snapshot. Unknown or HTTP sessions and missing, closed, or busy pages return an error. Use `browser_snapshot` afterward to inspect the result.
+
### `session_make_request` -- HTTP request through an open requests session
Makes an HTTP request (any method) through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint across calls. Same parameters as `make_request` plus a required `session_id`, minus the session-level `impersonate`, `proxy`, and `proxy_auth`. Raises on a browser session.
@@ -293,6 +329,7 @@ Requires an open browser session. Call `browser_open` first, then pass the `sess
| Move, hover, click, or scroll in order | `browser_mouse` with `session_id` |
| Fill one or more page fields in order | `browser_fill_fields` with `session_id` |
| Chain keys or keyboard shortcuts | `browser_press_key` with `session_id` |
+| Wait for content or a page load state | `browser_wait` with `session_id` |
Start with `make_request` (fastest, lowest resource cost). Escalate to `browser_fetch_once` if content requires JS rendering. Escalate to `browser_stealth_fetch_once` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
@@ -383,7 +420,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com
## Connecting to remote browsers
-`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, `browser_press_key`, and `browser_screenshot` as usual.
+`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, `browser_press_key`, `browser_wait`, and `browser_screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index e32cffd..a2c12b1 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W
## Features
-The Scrapling MCP Server provides seventeen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
+The Scrapling MCP Server provides eighteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
### One-shot tools
@@ -50,6 +50,10 @@ The Scrapling MCP Server provides seventeen tools for web scraping and browser i
- **`browser_fill_fields`**: Fill one or more fields anywhere on the page in one call, including text, checkboxes, radio buttons, and native dropdowns. Target fields through selectors or snapshot references. Replace or clear text, or type at the current caret without clearing existing content. Supports checking or unchecking boxes and choosing one or more dropdown options by their visible labels. Slow mode types text one character at a time with random pauses between characters and fields.
- **`browser_press_key`**: Chain keys and shortcuts on the current page to move between fields, submit forms, or dismiss menus in one call. Supports actions such as selecting all text and then deleting it, using the browser's current focus for each press.
+#### Waiting
+
+- **`browser_wait`**: Chain waits for delayed content, loading overlays, and page load events in one call without reloading the current page. For example, wait for the document to load, results to appear, and a loading overlay to disappear before the next action. Also supports fixed pauses and waiting for network activity to settle.
+
### Shadow DOM
Set `pierce_shadow=true` on `browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`, or `browser_fetch` to include open Shadow DOM content. It defaults to `false`. For persistent sessions, pass it on each `browser_fetch` call. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.
@@ -209,7 +213,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta
```
Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.
```
-Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, `browser_press_key`, and `browser_screenshot` as usual.
+Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, `browser_press_key`, `browser_wait`, and `browser_screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:
```commandline
diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md
index d7296bd..4f71b92 100644
--- a/docs/api-reference/mcp-server.md
+++ b/docs/api-reference/mcp-server.md
@@ -5,7 +5,7 @@ search:
# MCP Server API Reference
-The **Scrapling MCP Server** provides seventeen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
+The **Scrapling MCP Server** provides eighteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
You can start the MCP server by running:
@@ -30,7 +30,7 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")
## Response Model
-The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns image and text content blocks, and `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, and `browser_press_key` return plain text.
+The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns image and text content blocks, and `browser_snapshot`, `browser_mouse`, `browser_fill_fields`, `browser_press_key`, and `browser_wait` return plain text.
## ::: scrapling.core.ai.ResponseModel
handler: python
@@ -60,6 +60,7 @@ The main MCP server class that provides all web scraping tools:
- **`browser_mouse`**: Chain native mouse moves, hovers, clicks, and scrolling in one call. Reveal hover menus and tooltips, click buttons and links, or move over a nested panel and scroll it vertically or horizontally. Moves and clicks target selectors, snapshot references, or coordinates, with automatic waiting and scrolling into view for element targets. Supports coordinate movement in multiple steps, left, right, and middle clicks, plus double-clicks.
- **`browser_fill_fields`**: Fill text fields, check or uncheck boxes, select radio buttons, and choose one or more native dropdown options by visible labels in one call. Targets one or more fields anywhere on the page through selectors or snapshot references. Replace or clear text, or type at the current caret without clearing existing content. Slow mode types text one character at a time with random pauses between characters and fields.
- **`browser_press_key`**: Chain keys and shortcuts in one call to move between fields, submit forms, dismiss menus, or select and delete text. Each press uses the browser's current focus.
+- **`browser_wait`**: Chain waits for page load events, content appearing, and loading overlays disappearing in one call on the current page. Supports fixed pauses and waiting for network activity to settle without reloading the page.
- **`browser_fetch`**: Fetch a page through an open browser session and extract HTML, Markdown, text, or a structured snapshot. Snapshots can cover the whole page or a chosen element and include element references, positions, and sizes for later interaction.
## ::: scrapling.core.ai.ScraplingMCPServer
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 55a4668..6f6f39c 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -146,6 +146,32 @@ class _MouseWheel(TypedDict):
MouseAction = Annotated[Union[_MouseMove, _MouseClick, _MouseWheel], Field(discriminator="type")]
+class _TimeWait(TypedDict):
+ type: Literal["time"]
+ milliseconds: NonNegativeFiniteFloat
+
+
+class _ConditionWait(TypedDict, total=False):
+ timeout: Annotated[NonNegativeFiniteFloat, Field(default=30000, description="Wait limit in ms; 0 disables it.")]
+
+
+class _ElementWait(_ConditionWait):
+ type: Literal["element"]
+ selector: Annotated[NonEmptyString, Field(description="Playwright selector for one element.")]
+ state: NotRequired[Annotated[SelectorWaitStates, Field(default="visible", description="Hidden includes removal.")]]
+
+
+class _LoadWait(_ConditionWait):
+ type: Literal["load"]
+ state: Annotated[
+ Literal["domcontentloaded", "load", "networkidle"],
+ Field(description="Current document readiness; networkidle waits for 500 ms without active connections."),
+ ]
+
+
+WaitAction = Annotated[Union[_TimeWait, _ElementWait, _LoadWait], Field(discriminator="type")]
+
+
class _FormTarget(TypedDict, total=False):
selector: Optional[NonEmptyString]
ref: Optional[NonEmptyString]
@@ -482,6 +508,32 @@ class ScraplingMCPServer:
else:
page_info.mark_ready()
+ async def browser_wait(
+ self,
+ session_id: str,
+ actions: Annotated[List[WaitAction], Field(min_length=1)],
+ ) -> str:
+ """Run time, element, or load waits in order on the current page; return plain text. No navigation or snapshot.
+ Stop on the first error, identifying its action. Load states do not guarantee all later JavaScript has finished.
+
+ :param session_id: ID from `browser_open`; call `browser_fetch` first.
+ :param actions: Ordered waits; each condition has its own timeout. Time actions use milliseconds.
+ """
+ with self._browser_page(session_id) as (_, page):
+ for index, action in enumerate(actions, 1):
+ try:
+ if action["type"] == "time":
+ await page.wait_for_timeout(action["milliseconds"])
+ elif action["type"] == "element":
+ await page.locator(action["selector"]).wait_for(
+ state=action.get("state", "visible"), timeout=action.get("timeout", 30000)
+ )
+ else:
+ await page.wait_for_load_state(action["state"], timeout=action.get("timeout", 30000))
+ except Exception as exc:
+ raise RuntimeError(f"Wait action {index} ({action['type']}) failed: {exc}") from exc
+ return "Wait completed."
+
async def browser_mouse(
self,
session_id: str,
@@ -1408,6 +1460,13 @@ class ScraplingMCPServer:
structured_output=False,
annotations=_FETCH_TOOL_ANNOTATIONS,
)
+ server.add_tool(
+ self.browser_wait,
+ title="Wait",
+ description=self.browser_wait.__doc__,
+ structured_output=False,
+ annotations=_FETCH_TOOL_ANNOTATIONS,
+ )
server.add_tool(
self.browser_mouse,
title="Mouse actions",
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index 8fc1495..f1064bd 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -982,7 +982,7 @@ class TestServerToolRegistration:
@pytest.mark.asyncio
async def test_tools_are_listed_with_expected_schemas(self):
- """All 17 tools are advertised, with plain content for screenshots, snapshots, and input actions"""
+ """All 18 tools are advertised, with plain content for screenshots, snapshots, waits, and input actions"""
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
assert client.instructions
@@ -1005,9 +1005,10 @@ class TestServerToolRegistration:
result = await client.call_tool(name, {})
assert result.is_error
- assert len(tools) == 17
+ assert len(tools) == 18
assert tools["browser_screenshot"].output_schema is None
assert tools["browser_snapshot"].output_schema is None
+ assert tools["browser_wait"].output_schema is None
assert tools["browser_mouse"].output_schema is None
assert tools["browser_press_key"].output_schema is None
assert tools["browser_fill_fields"].output_schema is None
@@ -1040,6 +1041,7 @@ class TestServerToolRegistration:
not in (
"browser_screenshot",
"browser_snapshot",
+ "browser_wait",
"browser_mouse",
"browser_press_key",
"browser_fill_fields",
@@ -1108,7 +1110,7 @@ class TestServerToolRegistration:
assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
- assert len(annotations) == 17
+ assert len(annotations) == 18
for name in (
"make_request",
"bulk_get",
@@ -1120,6 +1122,7 @@ class TestServerToolRegistration:
"session_make_request",
"browser_snapshot",
"browser_screenshot",
+ "browser_wait",
):
assert annotations[name].read_only_hint is True
assert annotations[name].open_world_hint is True
diff --git a/tests/ai/test_browser_wait.py b/tests/ai/test_browser_wait.py
new file mode 100644
index 0000000..be0a1e7
--- /dev/null
+++ b/tests/ai/test_browser_wait.py
@@ -0,0 +1,467 @@
+import asyncio
+from contextlib import asynccontextmanager
+from os import getenv
+from time import monotonic
+from unittest.mock import AsyncMock, Mock, call
+
+import pytest
+from anyio import CancelScope
+from mcp.client import Client
+from mcp.types import TextContent
+from patchright.async_api import TimeoutError as PatchrightTimeoutError
+from playwright.async_api import TimeoutError as PlaywrightTimeoutError
+
+from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core._types import Any, AsyncGenerator
+from scrapling.engines._browsers._base import AsyncSession
+
+
+def _server(session_type: SessionType = "dynamic") -> tuple[ScraplingMCPServer, AsyncSession, Mock]:
+ server = ScraplingMCPServer()
+ session = AsyncSession()
+ session._is_alive = True
+ page = Mock()
+ page.is_closed.return_value = False
+ page.wait_for_timeout = AsyncMock()
+ page.wait_for_load_state = AsyncMock()
+ page.locator.return_value.wait_for = AsyncMock()
+ session.page_pool.add_page(page).mark_ready()
+ server._sessions["browser"] = _SessionEntry(session, session_type)
+ return server, session, page
+
+
+async def test_browser_wait_schema_and_annotations() -> None:
+ async with Client(ScraplingMCPServer()._build_server("127.0.0.1", 8000)) as client:
+ tool = next(tool for tool in (await client.list_tools()).tools if tool.name == "browser_wait")
+ assert tool.title == "Wait"
+ assert tool.input_schema["required"] == ["session_id", "actions"]
+ properties = tool.input_schema["properties"]
+ assert set(properties) == {"session_id", "actions"}
+ actions = properties["actions"]
+ assert actions["type"] == "array" and actions["minItems"] == 1
+ items = actions["items"]
+ assert items["discriminator"]["propertyName"] == "type"
+ variants = {
+ kind: tool.input_schema["$defs"][ref.rsplit("/", 1)[1]]
+ for kind, ref in items["discriminator"]["mapping"].items()
+ }
+ assert set(variants) == {"time", "element", "load"}
+ assert len(items["oneOf"]) == 3
+ assert variants["time"]["required"] == ["type", "milliseconds"]
+ assert variants["time"]["properties"]["milliseconds"]["minimum"] == 0
+ assert variants["element"]["required"] == ["type", "selector"]
+ assert variants["element"]["properties"]["selector"]["minLength"] == 1
+ assert set(variants["element"]["properties"]["state"]["enum"]) == {"attached", "detached", "visible", "hidden"}
+ assert variants["element"]["properties"]["state"]["default"] == "visible"
+ assert variants["load"]["required"] == ["type", "state"]
+ assert set(variants["load"]["properties"]["state"]["enum"]) == {"domcontentloaded", "load", "networkidle"}
+ for kind in ("element", "load"):
+ assert variants[kind]["properties"]["timeout"]["minimum"] == 0
+ assert variants[kind]["properties"]["timeout"]["default"] == 30000
+ assert tool.output_schema is None
+ assert tool.annotations and tool.annotations.read_only_hint and tool.annotations.open_world_hint
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+ "values, method, args, kwargs",
+ [
+ ({"type": "time", "milliseconds": 0}, "wait_for_timeout", (0,), {}),
+ ({"type": "time", "milliseconds": 12.5}, "wait_for_timeout", (12.5,), {}),
+ ({"type": "element", "selector": "#ready"}, "locator", (), {"state": "visible", "timeout": 30000}),
+ *[
+ (
+ {"type": "element", "selector": "#ready", "state": state, "timeout": 0},
+ "locator",
+ (),
+ {"state": state, "timeout": 0},
+ )
+ for state in ("attached", "detached", "visible", "hidden")
+ ],
+ *[
+ ({"type": "load", "state": state, "timeout": 12.5}, "wait_for_load_state", (state,), {"timeout": 12.5})
+ for state in ("domcontentloaded", "load", "networkidle")
+ ],
+ ],
+)
+async def test_browser_wait_forwards_only_requested_wait(
+ session_type: SessionType, values: dict[str, Any], method: str, args: tuple, kwargs: dict[str, Any]
+) -> None:
+ server, session, page = _server(session_type)
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_wait", {"session_id": "browser", "actions": [values]})
+ assert not result.is_error and result.structured_content is None
+ assert result.content == [TextContent(type="text", text="Wait completed.")]
+ for name in ("wait_for_timeout", "wait_for_load_state", "locator"):
+ action = page.locator.return_value.wait_for if name == "locator" else getattr(page, name)
+ if name == method:
+ action.assert_awaited_once_with(*args, **kwargs)
+ else:
+ action.assert_not_awaited()
+ if method == "locator":
+ page.locator.assert_called_once_with("#ready")
+ page.locator.return_value.first.wait_for.assert_not_called()
+ else:
+ page.locator.assert_not_called()
+ for name in ("goto", "reload", "evaluate", "aria_snapshot", "screenshot", "set_default_timeout"):
+ getattr(page, name).assert_not_called()
+ assert session.page_pool.pages[0].state == "ready"
+
+
+async def test_browser_wait_mixed_chain_keeps_order_and_page_reservation() -> None:
+ server, session, page = _server()
+ waited = Mock()
+ for name, method in (
+ ("time", page.wait_for_timeout),
+ ("element", page.locator.return_value.wait_for),
+ ("load", page.wait_for_load_state),
+ ):
+ waited.attach_mock(method, name)
+
+ async def assert_reserved(*args: Any, **kwargs: Any) -> None:
+ assert session.page_pool.pages[0].state == "busy"
+
+ method.side_effect = assert_reserved
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_wait",
+ {
+ "session_id": "browser",
+ "actions": [
+ {"type": "load", "state": "domcontentloaded"},
+ {"type": "element", "selector": "#ready", "timeout": 15},
+ {"type": "time", "milliseconds": 2.5},
+ {"type": "element", "selector": "#ready", "state": "hidden", "timeout": 0},
+ ],
+ },
+ )
+ assert not result.is_error
+ assert waited.mock_calls == [
+ call.load("domcontentloaded", timeout=30000),
+ call.element(state="visible", timeout=15),
+ call.time(2.5),
+ call.element(state="hidden", timeout=0),
+ ]
+ assert page.is_closed.call_count == 2
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "actions",
+ [
+ None,
+ [],
+ {},
+ "load",
+ [None],
+ [{}],
+ [{"type": "unknown"}],
+ *[
+ [{"type": "time", "milliseconds": 0}, invalid]
+ for invalid in (
+ {"type": "time"},
+ {"type": "time", "milliseconds": -1},
+ {"type": "time", "milliseconds": "NaN"},
+ {"type": "time", "milliseconds": "Infinity"},
+ {"type": "element"},
+ {"type": "element", "selector": ""},
+ {"type": "element", "selector": "#ready", "state": "missing"},
+ {"type": "load"},
+ {"type": "load", "state": "commit"},
+ {"type": "load", "state": "load", "timeout": -1},
+ {"type": "load", "state": "load", "timeout": "NaN"},
+ {"type": "element", "selector": "#ready", "timeout": "Infinity"},
+ )
+ ],
+ ],
+)
+async def test_browser_wait_invalid_batch_does_not_start_or_reserve_page(actions: Any) -> None:
+ server, session, page = _server()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_wait", {"session_id": "browser", "actions": actions})
+ assert result.is_error
+ page.is_closed.assert_not_called()
+ page.wait_for_timeout.assert_not_awaited()
+ page.wait_for_load_state.assert_not_awaited()
+ page.locator.assert_not_called()
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "session_type, error", [("dynamic", PlaywrightTimeoutError), ("stealthy", PatchrightTimeoutError)]
+)
+ "values", [{"type": "element", "selector": "#missing"}, {"type": "load", "state": "networkidle"}]
+)
+async def test_browser_wait_native_timeout_reaches_mcp_and_releases_page(
+ session_type: SessionType, error: type[Exception], values: dict[str, Any]
+) -> None:
+ server, session, page = _server(session_type)
+ action = page.locator.return_value.wait_for if values["type"] == "element" else page.wait_for_load_state
+ action.side_effect = error("wait timed out")
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_wait",
+ {
+ "session_id": "browser",
+ "actions": [{"type": "time", "milliseconds": 1}, values, {"type": "time", "milliseconds": 2}],
+ },
+ )
+ assert result.is_error and isinstance(result.content[0], TextContent)
+ assert f"Wait action 2 ({values['type']}) failed: wait timed out" in result.content[0].text
+ assert page.wait_for_timeout.await_args_list == [call(1)]
+ assert session.page_pool.pages[0].state == "ready"
+ assert not (
+ await client.call_tool(
+ "browser_wait", {"session_id": "browser", "actions": [{"type": "time", "milliseconds": 0}]}
+ )
+ ).is_error
+ assert action.await_count == 1
+ with pytest.raises(RuntimeError, match=r"Wait action 1 .* failed: wait timed out") as failed:
+ await server.browser_wait("browser", [values])
+ assert isinstance(failed.value.__cause__, error)
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "state, message",
+ [
+ ("missing", "not found"),
+ ("static", "'static'"),
+ ("dead", "no longer alive"),
+ ("empty", "browser_fetch"),
+ ("closed", "closed page"),
+ ],
+)
+async def test_browser_wait_session_errors(state: str, message: str) -> None:
+ server, session, page = _server("static" if state == "static" else "dynamic")
+ if state == "missing":
+ server._sessions.clear()
+ elif state == "dead":
+ session._is_alive = False
+ elif state == "empty":
+ session.page_pool.clear()
+ elif state == "closed":
+ page.is_closed.return_value = True
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_wait", {"session_id": "browser", "actions": [{"type": "time", "milliseconds": 0}]}
+ )
+ assert result.is_error and isinstance(result.content[0], TextContent)
+ assert message in result.content[0].text
+ page.wait_for_timeout.assert_not_awaited()
+ if state == "closed":
+ assert not session.page_pool.pages
+
+
[email protected]("cancel_mode", ["task", "scope"])
+async def test_browser_wait_reserves_page_until_cancelled(cancel_mode: str) -> None:
+ server, session, page = _server()
+ entered = asyncio.Event()
+ scope = CancelScope()
+
+ async def pending(*args: Any, **kwargs: Any) -> None:
+ entered.set()
+ await asyncio.Event().wait()
+
+ async def wait() -> None:
+ with scope:
+ await server.browser_wait(
+ "browser",
+ [
+ {"type": "time", "milliseconds": 1},
+ {"type": "load", "state": "networkidle", "timeout": 0},
+ {"type": "time", "milliseconds": 2},
+ ],
+ )
+
+ page.wait_for_load_state.side_effect = pending
+ task = asyncio.create_task(wait())
+ try:
+ await asyncio.wait_for(entered.wait(), 5)
+ assert session.page_pool.pages[0].state == "busy"
+ with pytest.raises(RuntimeError, match="busy"):
+ await server.browser_wait("browser", [{"type": "time", "milliseconds": 0}])
+ with pytest.raises(RuntimeError, match="busy"):
+ await server.browser_press_key("browser", ["Enter"])
+ finally:
+ scope.cancel() if cancel_mode == "scope" else task.cancel()
+ if cancel_mode == "scope":
+ await asyncio.wait_for(task, 5)
+ assert scope.cancelled_caught
+ else:
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert session.page_pool.pages[0].state == "ready"
+ assert page.wait_for_timeout.await_args_list == [call(1)]
+ assert await server.browser_wait("browser", [{"type": "time", "milliseconds": 0}]) == "Wait completed."
+
+
+@asynccontextmanager
+async def _browser(session_type: SessionType) -> AsyncGenerator[tuple[Client, Any, Any], None]:
+ server = ScraplingMCPServer(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ opened = await client.call_tool("browser_open", {"session_type": session_type, "session_id": "browser"})
+ assert not opened.is_error
+ try:
+ session = server._sessions["browser"].session
+ await session.context.route(
+ "**/*", lambda route: route.fulfill(content_type="text/html", body="<html><body></body></html>")
+ )
+ fetched = await client.call_tool(
+ "browser_fetch", {"session_id": "browser", "url": "https://wait.test/", "google_search": False}
+ )
+ assert not fetched.is_error
+ yield client, session, session.page_pool.pages[0].page
+ finally:
+ assert not (await client.call_tool("close_session", {"session_id": "browser"})).is_error
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_wait_live_pause_and_element_states(session_type: SessionType) -> None:
+ async with _browser(session_type) as (client, session, page):
+ started = monotonic()
+ assert not (
+ await client.call_tool(
+ "browser_wait", {"session_id": "browser", "actions": [{"type": "time", "milliseconds": 80}]}
+ )
+ ).is_error
+ assert monotonic() - started >= 0.07
+ for state, setup, change in (
+ (
+ "attached",
+ "document.body.innerHTML = ''",
+ "document.body.innerHTML = '<div id=target hidden>Ready</div>'",
+ ),
+ ("visible", "", "document.querySelector('#target').hidden = false"),
+ ("hidden", "", "document.querySelector('#target').hidden = true"),
+ ("detached", "", "document.querySelector('#target').remove()"),
+ ):
+ await page.evaluate(f"() => {{ {setup}; setTimeout(() => {{ {change} }}, 80); }}")
+ result = await client.call_tool(
+ "browser_wait",
+ {
+ "session_id": "browser",
+ "actions": [{"type": "element", "selector": "#target", "state": state, "timeout": 5000}],
+ },
+ )
+ assert not result.is_error
+ assert session.page_pool.pages[0].state == "ready"
+ if state in ("attached", "detached"):
+ assert await page.locator("#target").count() == (1 if state == "attached" else 0)
+ else:
+ assert await page.locator("#target").is_visible() == (state == "visible")
+ for state in ("hidden", "detached"):
+ assert not (
+ await client.call_tool(
+ "browser_wait",
+ {
+ "session_id": "browser",
+ "actions": [{"type": "element", "selector": "#absent", "state": state, "timeout": 100}],
+ },
+ )
+ ).is_error
+ await page.evaluate("document.body.innerHTML = '<div class=duplicate></div><div class=duplicate></div>'")
+ result = await client.call_tool(
+ "browser_wait",
+ {
+ "session_id": "browser",
+ "actions": [{"type": "element", "selector": ".duplicate", "state": "attached", "timeout": 100}],
+ },
+ )
+ assert result.is_error and isinstance(result.content[0], TextContent)
+ assert "strict mode violation" in result.content[0].text
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_wait_live_current_document_load_states(session_type: SessionType) -> None:
+ async with _browser(session_type) as (client, session, page):
+ requested, release = asyncio.Event(), asyncio.Event()
+
+ async def blocked(route: Any) -> None:
+ requested.set()
+ await release.wait()
+ await route.fulfill(body="done")
+
+ async def wait(values: dict[str, Any]) -> Any:
+ return await client.call_tool(
+ "browser_wait", {"session_id": "browser", "actions": [{"type": "load", **values}]}
+ )
+
+ await page.route("**/pending", blocked)
+ await page.route(
+ "**/document",
+ lambda route: route.fulfill(
+ content_type="text/html", body='<html><body><img src="/pending"></body></html>'
+ ),
+ )
+ task = None
+ try:
+ await page.goto("https://wait.test/document", wait_until="commit")
+ await asyncio.wait_for(requested.wait(), 5)
+ assert not (await wait({"state": "domcontentloaded", "timeout": 5000})).is_error
+ assert await page.evaluate("document.readyState") == "interactive"
+ task = asyncio.create_task(wait({"state": "load", "timeout": 5000}))
+ await asyncio.sleep(0.05)
+ assert not task.done()
+ release.set()
+ assert not (await asyncio.wait_for(task, 5)).is_error
+ assert await page.evaluate("document.readyState") == "complete"
+ requested.clear()
+ release.clear()
+ await page.evaluate("() => { window.pending = fetch('/pending').then(response => response.text()); }")
+ await asyncio.wait_for(requested.wait(), 5)
+ result = await wait({"state": "networkidle", "timeout": 100})
+ assert result.is_error and isinstance(result.content[0], TextContent)
+ assert "Timeout" in result.content[0].text
+ assert session.page_pool.pages[0].state == "ready"
+ task = asyncio.create_task(wait({"state": "networkidle", "timeout": 5000}))
+ await asyncio.sleep(0.05)
+ assert not task.done()
+ release.set()
+ assert not (await asyncio.wait_for(task, 5)).is_error
+ assert await page.evaluate("window.pending") == "done"
+ assert session.page_pool.pages[0].state == "ready"
+ finally:
+ release.set()
+ if task is not None and not task.done():
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_wait_live_load_visible_hidden_chain(session_type: SessionType) -> None:
+ async with _browser(session_type) as (client, session, page):
+ await page.evaluate("""() => {
+ document.body.innerHTML = '<div id=target hidden>Ready</div>';
+ setTimeout(() => {
+ document.querySelector('#target').hidden = false;
+ setTimeout(() => document.querySelector('#target').hidden = true, 300);
+ }, 100);
+ }""")
+ result = await client.call_tool(
+ "browser_wait",
+ {
+ "session_id": "browser",
+ "actions": [
+ {"type": "load", "state": "load", "timeout": 5000},
+ {"type": "element", "selector": "#target", "state": "visible", "timeout": 5000},
+ {"type": "element", "selector": "#target", "state": "hidden", "timeout": 5000},
+ ],
+ },
+ )
+ assert not result.is_error
+ assert result.content == [TextContent(type="text", text="Wait completed.")]
+ assert await page.locator("#target").is_hidden()
+ assert session.page_pool.pages[0].state == "ready"