Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(mcp): Add a new tool to execute JS on page

Karim shoair768e615df6d1
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 59cea96..8a44e0a 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 1461efe..ef890b8 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -1,8 +1,8 @@
# Scrapling MCP Server
-The Scrapling MCP server exposes fifteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
+The Scrapling MCP server exposes sixteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
-Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_snapshot` and `browser_actions` return plain text.
+Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_snapshot` and `browser_actions` return plain text. `browser_evaluate` returns compact JSON in one plain text block.
## Shadow DOM
@@ -218,6 +218,35 @@ MCP validates the entire action schema, and all target combinations are checked
Returns `Actions completed.` as plain text without echoing field values or taking a snapshot. Use `browser_snapshot` before retrying after an error or when later actions depend on inspecting a page change. Unknown or HTTP sessions and missing, closed, or busy pages return an error.
+### `browser_evaluate` -- Run JavaScript on the current page
+
+Runs a native JavaScript expression or function on the existing page. Call `browser_open`, then `browser_fetch` first. Both `dynamic` and `stealthy` sessions are accepted. Functions are invoked with the optional JSON object argument, and returned promises are awaited.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `session_id` | str | required | ID of an open browser session with a page |
+| `expression` | str | required | Nonempty JavaScript expression or function to invoke |
+| `arg` | object or null | null | One JSON object passed to the function; nested values can contain any JSON data |
+| `isolated_context` | bool | true | Stealthy only: use a separate JavaScript context; `false` accesses the page's own variables |
+
+In stealthy sessions, the isolated context shares the page's DOM but does not share its JavaScript variables. Use `isolated_context=false` to access application globals. Dynamic sessions always use the page's main context, regardless of this option.
+
+For example, collect text from the first matching links:
+
+```json
+{
+ "session_id": "browser",
+ "expression": "({selector, limit}) => [...document.querySelectorAll(selector)].slice(0, limit).map(element => element.textContent)",
+ "arg": {"selector": "a", "limit": 10}
+}
+```
+
+Return JSON-compatible data: null, strings, booleans, finite numbers, arrays, or objects. Convert special objects such as DOM nodes, dates, maps, and sets into plain data in JavaScript. Both `null` and `undefined` return `null`. Circular data, non-finite numbers, and other results that cannot be encoded as JSON return an error.
+
+The result is compact JSON in one `TextContent` block with `structured_output=False`: strings are JSON-quoted, arrays stay in one block, and null is retained. No automatic snapshot, navigation, reload, retry, or tool timeout is added.
+
+The page stays reserved during evaluation and is released after success, failure, or cancellation. Scripts can change page state or make requests. Errors and cancellation do not undo those effects or guarantee that browser-side JavaScript stops. Use `browser_snapshot` or another read to check effects before repeating a script. Unknown or HTTP sessions and missing, closed, or busy pages return an error; JavaScript errors propagate.
+
### `session_make_request` -- HTTP request through an open requests session
Makes an HTTP request (any method) through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint across calls. Same parameters as `make_request` plus a required `session_id`, minus the session-level `impersonate`, `proxy`, and `proxy_auth`. Raises on a browser session.
@@ -287,6 +316,7 @@ The page stays reserved during capture and is released after success, failure, o
| Capture the current page or an element | `browser_screenshot` after `browser_open` + `browser_fetch` |
| Read the current page's AI ARIA snapshot | `browser_snapshot` with `session_id` |
| Chain mouse, field, keyboard, or wait actions | `browser_actions` with `session_id` |
+| Custom JavaScript extraction or page tasks | `browser_evaluate` with `session_id` |
Start with `make_request` (fastest, lowest resource cost). Escalate to `browser_fetch_once` if content requires JS rendering. Escalate to `browser_stealth_fetch_once` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
@@ -377,7 +407,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com
## Connecting to remote browsers
-`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, and `browser_screenshot` as usual.
+`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, `browser_evaluate`, and `browser_screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index e6d7e6f..57b67b6 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W
## Features
-The Scrapling MCP Server provides fifteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
+The Scrapling MCP Server provides sixteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
### One-shot tools
@@ -52,6 +52,10 @@ The Scrapling MCP Server provides fifteen tools for web scraping and browser int
Wait for results to appear, loading overlays to disappear, or page load events before the next action. Also supports fixed pauses and waiting for network activity to settle.
+#### JavaScript
+
+- **`browser_evaluate`**: Run JavaScript on the current page for custom data extraction, calculations, and page-specific tasks. The AI can collect data from several elements, read application state, or update page content. Supports asynchronous scripts and returns their results as JSON.
+
### Shadow DOM
Set `pierce_shadow=true` on `browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`, or `browser_fetch` to include open Shadow DOM content. It defaults to `false`. For persistent sessions, pass it on each `browser_fetch` call. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.
@@ -211,7 +215,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta
```
Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.
```
-Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, and `browser_screenshot` as usual.
+Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, `browser_evaluate`, and `browser_screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:
```commandline
diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md
index 71b141d..5e55f55 100644
--- a/docs/api-reference/mcp-server.md
+++ b/docs/api-reference/mcp-server.md
@@ -5,7 +5,7 @@ search:
# MCP Server API Reference
-The **Scrapling MCP Server** provides fifteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
+The **Scrapling MCP Server** provides sixteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
You can start the MCP server by running:
@@ -30,7 +30,7 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")
## Response Model
-The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text.
+The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text. `browser_evaluate` returns its result as JSON in a plain text block.
## ::: scrapling.core.ai.ResponseModel
handler: python
@@ -64,6 +64,8 @@ The main MCP server class that provides all web scraping tools:
Target fields anywhere on the page through selectors or snapshot references. Fill text, check or uncheck boxes, select radio buttons, and choose one or more native dropdown options by label. Replace or clear text, or type at the current caret. Keyboard shortcuts use the current focus. Slow mode adds random pauses between typed characters and all actions.
Wait for content, loading overlays, page load events, or settled network activity, or add a fixed pause before continuing.
+
+- **`browser_evaluate`**: Run JavaScript on the current page to extract custom data, calculate results, read application state, or update page content. Supports asynchronous scripts and returns their results as JSON.
- **`browser_fetch`**: Fetch a page through an open browser session and extract HTML, Markdown, text, or a structured snapshot. Snapshots can cover the whole page or a chosen element and include element references, positions, and sizes for later interaction.
## ::: scrapling.core.ai.ScraplingMCPServer
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 6bcd93c..b3054d2 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -1,5 +1,6 @@
from uuid import uuid4
from os import environ
+from json import dumps
from time import monotonic
from random import uniform
from hmac import compare_digest
@@ -625,6 +626,31 @@ class ScraplingMCPServer:
raise RuntimeError(f"Action {index} ({action['type']}) failed: {exc}") from exc
return "Actions completed."
+ async def browser_evaluate(
+ self,
+ session_id: str,
+ expression: NonEmptyString,
+ arg: Optional[Dict[str, Any]] = None,
+ isolated_context: bool = True,
+ ) -> str:
+ """Run JavaScript on the current page, await promises, and return JSON text without a snapshot.
+ Return JSON-compatible values; null/undefined become null. Script effects are not undone on failure.
+ Cancellation stops waiting, but may not stop the script.
+
+ :param session_id: ID from `browser_open`; call `browser_fetch` first.
+ :param expression: JavaScript expression or function to invoke.
+ :param arg: JSON object passed to the function; use properties for scalar or array values.
+ :param isolated_context: Stealthy only: false accesses the page's own JS variables. Dynamic always uses that context.
+ """
+ with self._browser_page(session_id) as (session, page):
+ options = {"isolated_context": isolated_context} if isinstance(session, AsyncStealthySession) else {}
+ return dumps(
+ await page.evaluate(expression, arg=arg, **options),
+ ensure_ascii=False,
+ allow_nan=False,
+ separators=(",", ":"),
+ )
+
async def browser_screenshot(
self,
session_id: str,
@@ -1439,6 +1465,13 @@ class ScraplingMCPServer:
structured_output=False,
annotations=_INPUT_TOOL_ANNOTATIONS,
)
+ server.add_tool(
+ self.browser_evaluate,
+ title="Run JavaScript",
+ description=self.browser_evaluate.__doc__,
+ structured_output=False,
+ annotations=_INPUT_TOOL_ANNOTATIONS,
+ )
# Screenshot tool (returns image + url content blocks, not structured JSON)
server.add_tool(
self.browser_screenshot,
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index d308cd5..b35333d 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -982,7 +982,7 @@ class TestServerToolRegistration:
@pytest.mark.asyncio
async def test_tools_are_listed_with_expected_schemas(self):
- """All 15 tools are advertised, with plain content for screenshots, snapshots, waits, and input actions"""
+ """All 16 tools are advertised, with plain content for screenshots, snapshots, actions, and JavaScript"""
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
assert client.instructions
@@ -1009,10 +1009,11 @@ class TestServerToolRegistration:
result = await client.call_tool(name, {})
assert result.is_error
- assert len(tools) == 15
+ assert len(tools) == 16
assert tools["browser_screenshot"].output_schema is None
assert tools["browser_snapshot"].output_schema is None
assert tools["browser_actions"].output_schema is None
+ assert tools["browser_evaluate"].output_schema is None
assert {
"browser_type",
"browser_fill_form",
@@ -1047,6 +1048,7 @@ class TestServerToolRegistration:
"browser_screenshot",
"browser_snapshot",
"browser_actions",
+ "browser_evaluate",
)
)
@@ -1112,7 +1114,7 @@ class TestServerToolRegistration:
assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
- assert len(annotations) == 15
+ assert len(annotations) == 16
for name in (
"make_request",
"bulk_get",
@@ -1133,7 +1135,7 @@ class TestServerToolRegistration:
assert annotations[name].open_world_hint is True
assert annotations["list_sessions"].read_only_hint is True
assert annotations["list_sessions"].open_world_hint is False
- for name in ("browser_actions",):
+ for name in ("browser_actions", "browser_evaluate"):
assert annotations[name].read_only_hint is False
assert annotations[name].destructive_hint is True
assert annotations[name].idempotent_hint is False
diff --git a/tests/ai/test_browser_evaluate.py b/tests/ai/test_browser_evaluate.py
new file mode 100644
index 0000000..534f181
--- /dev/null
+++ b/tests/ai/test_browser_evaluate.py
@@ -0,0 +1,364 @@
+import asyncio
+from contextlib import suppress
+from json import loads
+from os import getenv
+from unittest.mock import AsyncMock, Mock, call
+
+import pytest
+from anyio import CancelScope
+from mcp.client import Client
+from mcp.types import TextContent
+from patchright.async_api import Error as PatchrightError
+from playwright.async_api import Error as PlaywrightError
+
+from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core._types import Any
+from scrapling.engines._browsers._base import AsyncSession
+from scrapling.fetchers import AsyncStealthySession
+
+
+def _server(session_type: SessionType = "dynamic") -> tuple[ScraplingMCPServer, AsyncSession, Mock]:
+ server = ScraplingMCPServer()
+ session = AsyncStealthySession() if session_type == "stealthy" else AsyncSession()
+ session._is_alive = True
+ page = Mock()
+ page.is_closed.return_value = False
+ page.evaluate = AsyncMock()
+ session.page_pool.add_page(page).mark_ready()
+ server._sessions["browser"] = _SessionEntry(session, session_type)
+ return server, session, page
+
+
+async def test_browser_evaluate_schema_and_annotations() -> None:
+ async with Client(ScraplingMCPServer()._build_server("127.0.0.1", 8000)) as client:
+ tool = next(tool for tool in (await client.list_tools()).tools if tool.name == "browser_evaluate")
+ assert tool.title == "Run JavaScript"
+ assert tool.input_schema["required"] == ["session_id", "expression"]
+ properties = tool.input_schema["properties"]
+ assert set(properties) == {"session_id", "expression", "arg", "isolated_context"}
+ assert properties["expression"]["type"] == "string"
+ assert properties["expression"]["minLength"] == 1
+ assert properties["arg"]["default"] is None
+ assert {option["type"] for option in properties["arg"]["anyOf"]} == {"object", "null"}
+ assert properties["isolated_context"]["type"] == "boolean"
+ assert properties["isolated_context"]["default"] is True
+ assert tool.output_schema is None
+ assert tool.annotations is not None
+ assert tool.annotations.read_only_hint is False
+ assert tool.annotations.destructive_hint is True
+ assert tool.annotations.idempotent_hint is False
+ assert tool.annotations.open_world_hint is True
+
+
+ "value, expected",
+ [
+ (None, "null"),
+ (True, "true"),
+ (False, "false"),
+ (42, "42"),
+ (1.25, "1.25"),
+ ("مرحبا\n", '"مرحبا\\n"'),
+ (["a", None], '["a",null]'),
+ ({"x": "☃", "nested": [1, False]}, '{"x":"☃","nested":[1,false]}'),
+ ],
+)
+async def test_browser_evaluate_returns_one_compact_json_text_block(value: Any, expected: str) -> None:
+ server, session, page = _server()
+
+ async def evaluate(*args: Any, **kwargs: Any) -> Any:
+ assert session.page_pool.pages[0].state == "busy"
+ return value
+
+ page.evaluate.side_effect = evaluate
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_evaluate", {"session_id": "browser", "expression": "document.title"})
+ assert not result.is_error and result.structured_content is None
+ assert result.content == [TextContent(type="text", text=expected)]
+ assert page.mock_calls == [call.is_closed(), call.evaluate("document.title", arg=None), call.is_closed()]
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
[email protected]("options", [{}, {"isolated_context": False}])
+async def test_browser_evaluate_forwards_argument_and_only_stealth_context(
+ session_type: SessionType, options: dict[str, Any]
+) -> None:
+ server, session, page = _server(session_type)
+ expression = "({items}) => items.length"
+ arg = {"items": ["a", "b"], "enabled": True, "empty": None, "strings": ["null", "[1]", '{"x":1}']}
+ page.evaluate.return_value = 2
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_evaluate", {"session_id": "browser", "expression": expression, "arg": arg, **options}
+ )
+ assert not result.is_error
+ expected = {"isolated_context": options.get("isolated_context", True)} if session_type == "stealthy" else {}
+ page.evaluate.assert_awaited_once_with(expression, arg=arg, **expected)
+ assert result.content == [TextContent(type="text", text="2")]
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("arg", [[1, 2], 42])
+async def test_browser_evaluate_rejects_non_object_arguments(arg: Any) -> None:
+ server, session, page = _server()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_evaluate", {"session_id": "browser", "expression": "value => value", "arg": arg}
+ )
+ assert result.is_error
+ page.is_closed.assert_not_called()
+ page.evaluate.assert_not_awaited()
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("expression", ["", None, [], 123])
+async def test_browser_evaluate_invalid_expression_does_not_reserve_page(expression: Any) -> None:
+ server, session, page = _server()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_evaluate", {"session_id": "browser", "expression": expression})
+ assert result.is_error
+ page.is_closed.assert_not_called()
+ page.evaluate.assert_not_awaited()
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "case, error_type",
+ [("nan", ValueError), ("infinite", ValueError), ("unsupported", TypeError), ("circular", ValueError)],
+)
+async def test_browser_evaluate_serialization_error_releases_page(case: str, error_type: type[Exception]) -> None:
+ server, session, page = _server()
+ value: Any = (
+ [] if case == "circular" else {"nan": float("nan"), "infinite": float("inf"), "unsupported": object()}[case]
+ )
+ if case == "circular":
+ value.append(value)
+ page.evaluate.return_value = value
+ with pytest.raises(error_type):
+ await server.browser_evaluate("browser", "value")
+ assert session.page_pool.pages[0].state == "ready"
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_evaluate", {"session_id": "browser", "expression": "value"})
+ assert result.is_error
+ assert page.evaluate.await_count == 2
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "state, message",
+ [
+ ("missing", "not found"),
+ ("dead", "no longer alive"),
+ ("static", "'dynamic' or 'stealthy'"),
+ ("empty", "browser_fetch"),
+ ("busy", "busy"),
+ ("closed", "closed page"),
+ ],
+)
+async def test_browser_evaluate_session_errors(state: str, message: str) -> None:
+ server, session, page = _server("static" if state == "static" else "dynamic")
+ if state == "missing":
+ server._sessions.clear()
+ elif state == "dead":
+ session._is_alive = False
+ elif state == "empty":
+ session.page_pool.clear()
+ elif state == "busy":
+ session.page_pool.pages[0].mark_busy()
+ elif state == "closed":
+ page.is_closed.return_value = True
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_evaluate", {"session_id": "browser", "expression": "1"})
+ assert result.is_error and isinstance(result.content[0], TextContent)
+ assert message in result.content[0].text
+ page.evaluate.assert_not_awaited()
+ if state == "closed":
+ assert session.page_pool.pages_count == 0
+ elif state == "busy":
+ assert session.page_pool.pages[0].state == "busy"
+
+
[email protected]("session_type, error_type", [("dynamic", PlaywrightError), ("stealthy", PatchrightError)])
[email protected]("closed", [False, True])
+async def test_browser_evaluate_native_error_releases_page_without_retry(
+ session_type: SessionType, error_type: type[Exception], closed: bool
+) -> None:
+ server, session, page = _server(session_type)
+ native_error = error_type("script failed")
+
+ async def fail(*args: Any, **kwargs: Any) -> None:
+ page.is_closed.return_value = closed
+ raise native_error
+
+ page.evaluate.side_effect = fail
+ with pytest.raises(error_type) as error:
+ await server.browser_evaluate("browser", "badScript()")
+ assert error.value is native_error
+ page.evaluate.assert_awaited_once()
+ page.close.assert_not_called()
+ if closed:
+ assert session.page_pool.pages_count == 0
+ else:
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("cancel_mode", ["task", "scope"])
+async def test_browser_evaluate_reserves_page_until_cancelled(cancel_mode: str) -> None:
+ server, session, page = _server()
+ entered = asyncio.Event()
+ scope = CancelScope()
+
+ async def pending(*args: Any, **kwargs: Any) -> None:
+ entered.set()
+ await asyncio.Event().wait()
+
+ async def evaluate() -> None:
+ with scope:
+ await server.browser_evaluate("browser", "new Promise(() => {})")
+
+ page.evaluate.side_effect = pending
+ task = asyncio.create_task(evaluate())
+ try:
+ await asyncio.wait_for(entered.wait(), 5)
+ assert session.page_pool.pages[0].state == "busy"
+ with pytest.raises(RuntimeError, match="busy"):
+ await server.browser_evaluate("browser", "1")
+ with pytest.raises(RuntimeError, match="busy"):
+ await server.browser_snapshot("browser")
+ scope.cancel() if cancel_mode == "scope" else task.cancel()
+ if cancel_mode == "scope":
+ await asyncio.wait_for(task, 5)
+ assert scope.cancelled_caught
+ else:
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ finally:
+ if not task.done():
+ task.cancel()
+ with suppress(asyncio.CancelledError):
+ await task
+ page.evaluate.assert_awaited_once()
+ assert session.page_pool.pages[0].state == "ready"
+ page.close.assert_not_called()
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_evaluate_live_expressions_promises_context_and_page_state(session_type: SessionType) -> None:
+ server = ScraplingMCPServer(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))
+ requests: list[str] = []
+ html = """<!DOCTYPE html><html><head><title>Evaluate test</title></head><body>
+ <input id=name><input id=agree type=checkbox><button id=toggle>Menu</button><div id=menu hidden>Choices</div>
+ <output id=status>pending</output><script>
+ window.pageState = {label: 'page world'};
+ document.querySelector('#toggle').onclick = () => {
+ document.querySelector('#menu').hidden = false;
+ history.pushState({}, '', '/edited');
+ };
+ </script></body></html>"""
+
+ async def serve(route: Any) -> None:
+ requests.append(route.request.url)
+ await route.fulfill(content_type="text/html", body=html)
+
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ opened = await client.call_tool("browser_open", {"session_id": "browser", "session_type": session_type})
+ assert not opened.is_error
+ try:
+ session = server._sessions["browser"].session
+ await session.context.route("**/*", serve)
+ fetched = await client.call_tool(
+ "browser_fetch", {"session_id": "browser", "url": "https://evaluate.test/start", "google_search": False}
+ )
+ assert not fetched.is_error
+ page_info = session.page_pool.pages[0]
+ page = page_info.page
+ actions = await client.call_tool(
+ "browser_actions",
+ {
+ "session_id": "browser",
+ "actions": [
+ {"type": "textbox", "selector": "#name", "value": "before"},
+ {"type": "checkbox", "selector": "#agree", "value": True},
+ {"type": "click", "selector": "#toggle"},
+ {"type": "click", "selector": "#name"},
+ ],
+ },
+ )
+ assert not actions.is_error
+ read_state = """() => ({url: location.href, name: document.querySelector('#name').value,
+ checked: document.querySelector('#agree').checked, menu: !document.querySelector('#menu').hidden,
+ focus: document.activeElement.id, status: document.querySelector('#status').textContent})"""
+ before = await page.evaluate(read_state)
+ assert before == {
+ "url": "https://evaluate.test/edited",
+ "name": "before",
+ "checked": True,
+ "menu": True,
+ "focus": "name",
+ "status": "pending",
+ }
+ assert requests == ["https://evaluate.test/start"]
+ for expression, arg, expected in (
+ ("document.title", None, "Evaluate test"),
+ ("undefined", None, None),
+ ("({items}) => items.map(item => item * 2)", {"items": [1, 2]}, [2, 4]),
+ (
+ "({value}) => value",
+ {"value": {"strings": ["null", "[1]", '{"x":1}'], "count": 3, "flag": False}},
+ {"strings": ["null", "[1]", '{"x":1}'], "count": 3, "flag": False},
+ ),
+ (
+ "async ({value}) => { await new Promise(resolve => setTimeout(resolve, 20)); return {value, inputs: document.querySelectorAll('input').length}; }",
+ {"value": "مرحبا"},
+ {"value": "مرحبا", "inputs": 2},
+ ),
+ ("typeof window.pageState", None, "undefined" if session_type == "stealthy" else "object"),
+ ):
+ result = await client.call_tool(
+ "browser_evaluate", {"session_id": "browser", "expression": expression, "arg": arg}
+ )
+ assert not result.is_error and result.structured_content is None
+ assert len(result.content) == 1 and isinstance(result.content[0], TextContent)
+ assert loads(result.content[0].text) == expected
+ assert await page.evaluate(read_state) == before
+ main_world = await client.call_tool(
+ "browser_evaluate",
+ {"session_id": "browser", "expression": "window.pageState.label", "isolated_context": False},
+ )
+ assert not main_world.is_error and isinstance(main_world.content[0], TextContent)
+ assert loads(main_world.content[0].text) == "page world"
+ changed = await client.call_tool(
+ "browser_evaluate",
+ {
+ "session_id": "browser",
+ "expression": "({value}) => { document.querySelector('#name').value = value; document.querySelector('#status').textContent = 'saved'; return document.querySelector('#name').value; }",
+ "arg": {"value": "after"},
+ },
+ )
+ assert not changed.is_error and isinstance(changed.content[0], TextContent)
+ assert loads(changed.content[0].text) == "after"
+ expected_state = {**before, "name": "after", "status": "saved"}
+ for expression, message in (
+ ("() => { throw new Error('script failed'); }", "script failed"),
+ ("Promise.reject(new Error('promise failed'))", "promise failed"),
+ ("NaN", "JSON"),
+ ):
+ failed = await client.call_tool("browser_evaluate", {"session_id": "browser", "expression": expression})
+ assert failed.is_error and isinstance(failed.content[0], TextContent)
+ assert message in failed.content[0].text
+ assert await page.evaluate(read_state) == expected_state
+ assert requests == ["https://evaluate.test/start"]
+ assert session.page_pool.pages == [page_info] and page_info.state == "ready"
+ finally:
+ assert not (await client.call_tool("close_session", {"session_id": "browser"})).is_error