Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(mcp): add a tool to fill all kinds of inputs in a page

Karim shoair6ed271d33fc9
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 39bfb9b..39d45e1 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 1b6c728..9a82475 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -1,8 +1,8 @@
# Scrapling MCP Server
-The Scrapling MCP server exposes nineteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, mouse actions, text input, keyboard shortcuts, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
+The Scrapling MCP server exposes twenty tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, mouse actions, text input, batch field filling, keyboard shortcuts, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`, `browser_stealth_fetch_once`, `browser_stealth_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.
-Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL). `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, and `browser_press_key` return plain text.
+Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL). `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, `browser_fill_fields`, and `browser_press_key` return plain text.
## Shadow DOM
@@ -235,6 +235,30 @@ Pass the exact snapshot reference value, such as `ref="e2"`. Take a new snapshot
Returns `Text entered.` as plain text without echoing the text or taking a snapshot. Errors are not retried, and the page reservation is released after success, failure, or cancellation. Entry or submission may have already changed the page when an error occurs; use `browser_snapshot` to check before retrying.
+### `browser_fill_fields` -- Fill multiple page fields in order
+
+Fills fields anywhere on the existing page in a dynamic or stealthy browser session. Fields do not need a `<form>` parent. Call `browser_fetch` first. Each field needs exactly one nonempty `selector` or snapshot `ref`, plus its `type` and `value`.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `session_id` | str | required | ID of an open browser session |
+| `fields` | list[object] | required | Nonempty ordered list of field targets, types, and values |
+| `timeout` | number | 30000 | Finite nonnegative timeout in milliseconds per operation; 0 disables the limit |
+| `slowly` | bool | false | Replace text with fresh random delays: 50-150 ms per character, 100-300 ms between fields |
+
+| Field `type` | `value` | Behavior |
+|--------------|---------|----------|
+| `textbox` | str | Replace text in an input, textarea, or contenteditable element; an empty string clears it |
+| `checkbox` | bool | `true` checks the box; `false` unchecks it |
+| `radio` | `true` | Select the radio option; `false` is not supported |
+| `combobox` | str or list[str] | Select native `<select>` options by label; use a list for multiple selections or `[]` to clear |
+
+Selectors must match exactly one element. Use current snapshot references, such as `"e2"`. MCP validates all field types and values before execution, and target combinations are checked before reserving the page. One page stays reserved for the whole batch. Native locator `fill`, `set_checked`, and `select_option(label=...)` run in list order with their normal waiting. Combobox fields support native `<select>` elements, not custom dropdown widgets.
+
+With `slowly=true`, text fields are cleared before native `press_sequentially` types each character with a fresh random delay of 50-150 ms. Empty text only clears the field. All field types get a fresh random pause of 100-300 ms before the next field, with no pause before the first or after the last. The timeout applies separately to each native action, including clearing and each character press; pauses between fields are outside it.
+
+Returns `Fields filled.` as plain text without echoing values, submitting forms, or taking a snapshot. An error or cancellation stops the remaining fields; completed changes stay and are not retried. The page reservation is released afterward. Use `browser_snapshot` to check the page before retrying or submitting a form. Unknown or HTTP sessions and missing, closed, or busy pages return an error.
+
### `browser_press_key` -- Chain keys and shortcuts at the current focus
Presses each item in order through native `page.keyboard.press` on the existing page in a dynamic or stealthy browser session. The page stays reserved for the full sequence. Call `browser_fetch` first. Use `browser_click` or `browser_type` to focus a control before pressing when needed; later presses follow any focus changes caused by earlier ones.
@@ -305,6 +329,7 @@ Requires an open browser session. Call `browser_open` first, then pass the `sess
| Scroll a page or nested panel | `browser_mouse_wheel` with `session_id` |
| Click by selector, snapshot ref, or coordinates | `browser_click` with `session_id` |
| Enter text by selector or snapshot ref | `browser_type` with `session_id` |
+| Fill several page fields in order | `browser_fill_fields` with `session_id` |
| Chain keys or keyboard shortcuts | `browser_press_key` with `session_id` |
Start with `make_request` (fastest, lowest resource cost). Escalate to `browser_fetch_once` if content requires JS rendering. Escalate to `browser_stealth_fetch_once` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
@@ -396,7 +421,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com
## Connecting to remote browsers
-`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, `browser_press_key`, and `browser_screenshot` as usual.
+`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, `browser_fill_fields`, `browser_press_key`, and `browser_screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index 04c47d0..380d649 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W
## Features
-The Scrapling MCP Server provides nineteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
+The Scrapling MCP Server provides twenty tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.
### One-shot tools
@@ -50,6 +50,7 @@ The Scrapling MCP Server provides nineteen tools for web scraping and browser in
#### Text Input and Keyboard
- **`browser_type`**: Fill search boxes, login fields, text areas, and other editable content through selectors or snapshot references. Replace or clear existing text, or type one character at a time for fields that react to key events. Can also press Enter to run a search or submit a form.
+- **`browser_fill_fields`**: Fill multiple fields anywhere on the page in one call, including text, checkboxes, radio buttons, and native dropdowns. Target fields through selectors or snapshot references, waiting for each field to be ready. Supports clearing text, checking or unchecking boxes, and choosing one or more dropdown options by their visible labels. Slow mode types text one character at a time with random pauses between characters and fields.
- **`browser_press_key`**: Chain keys and shortcuts on the current page to move between fields, submit forms, or dismiss menus in one call. Supports actions such as selecting all text and then deleting it, using the browser's current focus for each press.
### Shadow DOM
@@ -211,7 +212,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta
```
Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.
```
-Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, `browser_press_key`, and `browser_screenshot` as usual.
+Both browser session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, `browser_fill_fields`, `browser_press_key`, and `browser_screenshot` as usual.
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:
```commandline
diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md
index 234dc18..085d43b 100644
--- a/docs/api-reference/mcp-server.md
+++ b/docs/api-reference/mcp-server.md
@@ -5,7 +5,7 @@ search:
# MCP Server API Reference
-The **Scrapling MCP Server** provides nineteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
+The **Scrapling MCP Server** provides twenty tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
You can start the MCP server by running:
@@ -30,7 +30,7 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")
## Response Model
-The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns image and text content blocks, and `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, and `browser_press_key` return plain text.
+The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns image and text content blocks, and `browser_snapshot`, `browser_mouse_move`, `browser_mouse_wheel`, `browser_click`, `browser_type`, `browser_fill_fields`, and `browser_press_key` return plain text.
## ::: scrapling.core.ai.ResponseModel
handler: python
@@ -61,6 +61,7 @@ The main MCP server class that provides all web scraping tools:
- **`browser_mouse_wheel`**: Scroll pages and nested panels vertically or horizontally with native mouse wheel events. Works at the current pointer position, so mouse movement can choose the scrollable area.
- **`browser_click`**: Click buttons, links, and other controls using selectors, snapshot references, or page coordinates. Supports left, right, and middle clicks, plus double-clicks.
- **`browser_type`**: Fill search boxes, login fields, text areas, and other editable content through selectors or snapshot references. Supports replacing or clearing text, typing one character at a time for key events, and pressing Enter after entry.
+- **`browser_fill_fields`**: Fill text fields, check or uncheck boxes, select radio buttons, and choose one or more native dropdown options by visible labels in one call. Targets fields anywhere on the page through selectors or snapshot references and waits for each field to be ready. Slow mode types text one character at a time with random pauses between characters and fields.
- **`browser_press_key`**: Chain keys and shortcuts in one call to move between fields, submit forms, dismiss menus, or select and delete text. Each press uses the browser's current focus.
- **`browser_fetch`**: Fetch a page through an open browser session and extract HTML, Markdown, text, or a structured snapshot. Snapshots can cover the whole page or a chosen element and include element references, positions, and sizes for later interaction.
diff --git a/ruff.toml b/ruff.toml
index 405614a..8428347 100644
--- a/ruff.toml
+++ b/ruff.toml
@@ -16,6 +16,7 @@ target-version = "py310"
line-length = 120
[lint]
+typing-modules = ["scrapling.core._types"]
select = ["E", "F", "W"]
ignore = ["E501", "F401", "F811"]
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 0244c3c..4c76aaf 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -1,7 +1,8 @@
from uuid import uuid4
from os import environ
+from random import uniform
from hmac import compare_digest
-from asyncio import CancelledError, gather
+from asyncio import CancelledError, gather, sleep
from contextlib import contextmanager
from datetime import datetime, timezone
from dataclasses import dataclass, field
@@ -36,6 +37,7 @@ from scrapling.core._types import (
Tuple,
Mapping,
Dict,
+ TypedDict,
List,
Any,
Annotated,
@@ -106,6 +108,36 @@ _INPUT_TOOL_ANNOTATIONS = ToolAnnotations(
)
+class _FormTarget(TypedDict, total=False):
+ selector: Optional[NonEmptyString]
+ ref: Optional[NonEmptyString]
+
+
+class _TextFormField(_FormTarget):
+ type: Literal["textbox"]
+ value: str
+
+
+class _CheckboxFormField(_FormTarget):
+ type: Literal["checkbox"]
+ value: bool
+
+
+class _RadioFormField(_FormTarget):
+ type: Literal["radio"]
+ value: Literal[True]
+
+
+class _SelectFormField(_FormTarget):
+ type: Literal["combobox"]
+ value: Annotated[Union[str, List[str]], Field(description="Option label(s); [] clears selection.")]
+
+
+FormField = Annotated[
+ Union[_TextFormField, _CheckboxFormField, _RadioFormField, _SelectFormField], Field(discriminator="type")
+]
+
+
class ResponseModel(BaseModel):
"""Request's response information structure."""
@@ -541,6 +573,38 @@ class ScraplingMCPServer:
await page.keyboard.press(key)
return "Keys pressed."
+ async def browser_fill_fields(
+ self,
+ session_id: str,
+ fields: Annotated[List[FormField], Field(min_length=1)],
+ timeout: NonNegativeFiniteFloat = 30000,
+ slowly: bool = False,
+ ) -> str:
+ """Fill fields in order; stop on error and return plain text on success.
+
+ :param session_id: ID from `browser_open`; call `browser_fetch` first.
+ :param fields: Ordered field values; each needs exactly one selector or snapshot ref.
+ :param timeout: Limit per operation in milliseconds; 0 disables it.
+ :param slowly: Replace text with random pauses: 50-150 ms per character, 100-300 ms between fields.
+ """
+ if any((item.get("selector") is None) == (item.get("ref") is None) for item in fields):
+ raise ValueError("Each field must have exactly one target: 'selector' or 'ref'.")
+ with self._browser_page(session_id) as (_, page):
+ for index, item in enumerate(fields):
+ if slowly and index:
+ await sleep(uniform(0.1, 0.3))
+ locator = page.locator(item.get("selector") or f"aria-ref={item.get('ref')}")
+ if item["type"] == "textbox":
+ await locator.fill("" if slowly else item["value"], timeout=timeout)
+ if slowly:
+ for character in item["value"]:
+ await locator.press_sequentially(character, delay=uniform(50, 150), timeout=timeout)
+ elif item["type"] == "combobox":
+ await locator.select_option(label=item["value"], timeout=timeout)
+ else:
+ await locator.set_checked(item["value"], timeout=timeout)
+ return "Fields filled."
+
async def browser_screenshot(
self,
url: str,
@@ -1396,6 +1460,13 @@ class ScraplingMCPServer:
structured_output=False,
annotations=_INPUT_TOOL_ANNOTATIONS,
)
+ server.add_tool(
+ self.browser_fill_fields,
+ title="Fill fields",
+ description=self.browser_fill_fields.__doc__,
+ structured_output=False,
+ annotations=_INPUT_TOOL_ANNOTATIONS,
+ )
# Screenshot tool (returns image + url content blocks, not structured JSON)
server.add_tool(
self.browser_screenshot,
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index 1d28c83..a9d1896 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -982,12 +982,13 @@ class TestServerToolRegistration:
@pytest.mark.asyncio
async def test_tools_are_listed_with_expected_schemas(self):
- """All 19 tools are advertised, with plain content for screenshots, snapshots, and input actions"""
+ """All 20 tools are advertised, with plain content for screenshots, snapshots, and input actions"""
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
assert client.instructions
tools = {tool.name: tool for tool in (await client.list_tools()).tools}
for name in (
+ "browser_fill_form",
"open_session",
"session_fetch",
"screenshot",
@@ -1001,7 +1002,7 @@ class TestServerToolRegistration:
result = await client.call_tool(name, {})
assert result.is_error
- assert len(tools) == 19
+ assert len(tools) == 20
assert tools["browser_screenshot"].output_schema is None
assert tools["browser_snapshot"].output_schema is None
assert tools["browser_mouse_move"].output_schema is None
@@ -1009,7 +1010,9 @@ class TestServerToolRegistration:
assert tools["browser_type"].output_schema is None
assert tools["browser_mouse_wheel"].output_schema is None
assert tools["browser_press_key"].output_schema is None
+ assert tools["browser_fill_fields"].output_schema is None
assert {
+ "browser_fill_form",
"session_snapshot",
"open_session",
"session_fetch",
@@ -1039,6 +1042,7 @@ class TestServerToolRegistration:
"browser_type",
"browser_mouse_wheel",
"browser_press_key",
+ "browser_fill_fields",
)
)
@@ -1104,7 +1108,7 @@ class TestServerToolRegistration:
assert result.ttl_ms == 3_600_000 and result.cache_scope == "public"
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
- assert len(annotations) == 19
+ assert len(annotations) == 20
for name in (
"make_request",
"bulk_get",
@@ -1125,7 +1129,14 @@ class TestServerToolRegistration:
assert annotations[name].open_world_hint is True
assert annotations["list_sessions"].read_only_hint is True
assert annotations["list_sessions"].open_world_hint is False
- for name in ("browser_mouse_move", "browser_click", "browser_type", "browser_mouse_wheel", "browser_press_key"):
+ for name in (
+ "browser_mouse_move",
+ "browser_click",
+ "browser_type",
+ "browser_mouse_wheel",
+ "browser_press_key",
+ "browser_fill_fields",
+ ):
assert annotations[name].read_only_hint is False
assert annotations[name].destructive_hint is True
assert annotations[name].idempotent_hint is False
diff --git a/tests/ai/test_browser_fill_fields.py b/tests/ai/test_browser_fill_fields.py
new file mode 100644
index 0000000..17562ac
--- /dev/null
+++ b/tests/ai/test_browser_fill_fields.py
@@ -0,0 +1,557 @@
+import asyncio
+from contextlib import asynccontextmanager
+from json import loads
+from os import getenv
+from re import search
+from unittest.mock import AsyncMock, Mock, call
+
+import pytest
+from anyio import CancelScope
+from mcp.client import Client
+from mcp.types import TextContent
+from pydantic import ValidationError
+
+from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core._types import Any, AsyncGenerator
+from scrapling.engines._browsers._base import AsyncSession
+
+
+TEXT_FIELD = {"type": "textbox", "selector": "#name", "value": "private value"}
+HTML = """<!DOCTYPE html><html><body><form>
+<label>Name<input id="name" value="initial"></label><label>Notes<textarea id="notes">initial notes</textarea></label>
+<div id="editor" contenteditable="true">initial editor</div>
+<label>Agree<input id="agree" type="checkbox"></label><label>Updates<input id="updates" type="checkbox" checked></label>
+<label>Basic<input id="basic" type="radio" name="plan" checked></label><label>Pro<input id="pro" type="radio" name="plan"></label>
+<button>Submit</button></form>
+<label>Country<select id="country"><option value="us">United States</option><option value="gb">United Kingdom</option></select></label>
+<label>Colors<select id="colors" multiple><option value="r">Red</option><option value="g">Green</option><option value="b">Blue</option></select></label>
+<output id="submitted">0</output><textarea id="events" hidden>[]</textarea><script>
+document.querySelector('form').addEventListener('submit', event => {
+ event.preventDefault();
+ const output = document.querySelector('#submitted');
+ output.textContent = Number(output.textContent) + 1;
+});
+for (const type of ['input', 'change', 'keydown', 'keyup']) {
+ document.addEventListener(type, event => {
+ const events = document.querySelector('#events');
+ events.value = JSON.stringify([...JSON.parse(events.value), {type, target: event.target.id, key: event.key, time: performance.now()}]);
+ });
+}
+</script></body></html>"""
+
+
+def _server(session_type: SessionType = "dynamic") -> tuple[ScraplingMCPServer, AsyncSession, Mock]:
+ server = ScraplingMCPServer()
+ session = AsyncSession()
+ session._is_alive = True
+ page = Mock()
+ page.is_closed.return_value = False
+ for method in ("fill", "press_sequentially", "set_checked", "select_option"):
+ setattr(page.locator.return_value, method, AsyncMock())
+ session.page_pool.add_page(page).mark_ready()
+ server._sessions["browser"] = _SessionEntry(session, session_type)
+ return server, session, page
+
+
+async def test_browser_fill_fields_schema() -> None:
+ async with Client(ScraplingMCPServer()._build_server("127.0.0.1", 8000)) as client:
+ tool = next(tool for tool in (await client.list_tools()).tools if tool.name == "browser_fill_fields")
+ assert tool.title == "Fill fields"
+ schema = tool.input_schema
+ assert set(schema["properties"]) == {"session_id", "fields", "timeout", "slowly"}
+ assert schema["required"] == ["session_id", "fields"]
+ assert schema["properties"]["slowly"]["type"] == "boolean"
+ assert schema["properties"]["slowly"]["default"] is False
+ assert schema["properties"]["timeout"]["default"] == 30000
+ assert schema["properties"]["timeout"]["minimum"] == 0
+ fields = schema["properties"]["fields"]
+ assert fields["type"] == "array" and fields["minItems"] == 1
+ items = fields["items"]
+ assert items["discriminator"]["propertyName"] == "type"
+ variants = {kind: schema["$defs"][ref.rsplit("/", 1)[1]] for kind, ref in items["discriminator"]["mapping"].items()}
+ assert set(variants) == {"textbox", "checkbox", "radio", "combobox"}
+ assert len(items["oneOf"]) == 4
+ for kind, variant in variants.items():
+ assert set(variant["required"]) == {"type", "value"}
+ assert set(variant["properties"]) == {"type", "value", "selector", "ref"}
+ assert variant["properties"]["type"]["const"] == kind
+ for target in ("selector", "ref"):
+ assert {"type": "null"} in variant["properties"][target]["anyOf"]
+ assert any(option.get("minLength") == 1 for option in variant["properties"][target]["anyOf"])
+ assert variants["textbox"]["properties"]["value"]["type"] == "string"
+ assert variants["checkbox"]["properties"]["value"]["type"] == "boolean"
+ assert variants["radio"]["properties"]["value"]["const"] is True
+ assert {option["type"] for option in variants["combobox"]["properties"]["value"]["anyOf"]} == {"string", "array"}
+
+
[email protected]("options, timeout", [({}, 30000), ({"timeout": 0}, 0), ({"timeout": 123.5}, 123.5)])
+async def test_browser_fill_fields_forwards_native_actions_in_order(
+ options: dict[str, Any], timeout: float, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ server, session, page = _server()
+ pause = AsyncMock()
+ monkeypatch.setattr("scrapling.core.ai.sleep", pause)
+ fields = [
+ {**TEXT_FIELD, "ref": None},
+ {"type": "textbox", "selector": None, "ref": "e2", "value": ""},
+ {"type": "checkbox", "selector": "#agree", "value": True},
+ {"type": "checkbox", "selector": "#agree", "value": False},
+ {"type": "checkbox", "selector": "#agree", "value": "false"},
+ {"type": "radio", "selector": "#pro", "value": True},
+ {"type": "combobox", "selector": "#country", "value": "United Kingdom"},
+ {"type": "combobox", "selector": "#colors", "value": ["Red", "Blue"]},
+ {"type": "combobox", "selector": "#colors", "value": []},
+ ]
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_fill_fields", {"session_id": "browser", "fields": fields, **options})
+ assert not result.is_error
+ assert result.structured_content is None
+ assert result.content == [TextContent(type="text", text="Fields filled.")]
+ assert page.locator.mock_calls == [
+ call("#name"),
+ call().fill("private value", timeout=timeout),
+ call("aria-ref=e2"),
+ call().fill("", timeout=timeout),
+ call("#agree"),
+ call().set_checked(True, timeout=timeout),
+ call("#agree"),
+ call().set_checked(False, timeout=timeout),
+ call("#agree"),
+ call().set_checked(False, timeout=timeout),
+ call("#pro"),
+ call().set_checked(True, timeout=timeout),
+ call("#country"),
+ call().select_option(label="United Kingdom", timeout=timeout),
+ call("#colors"),
+ call().select_option(label=["Red", "Blue"], timeout=timeout),
+ call("#colors"),
+ call().select_option(label=[], timeout=timeout),
+ ]
+ page.keyboard.press.assert_not_called()
+ page.locator.return_value.press.assert_not_called()
+ page.aria_snapshot.assert_not_called()
+ page.screenshot.assert_not_called()
+ pause.assert_not_awaited()
+ assert session.page_pool.pages[0].state == "ready"
+
+
+async def test_browser_fill_fields_slow_pacing_and_empty_text(monkeypatch: pytest.MonkeyPatch) -> None:
+ server, session, page = _server()
+ actions = Mock()
+ actions.attach_mock(page.locator, "locator")
+ actions.pause = AsyncMock()
+ monkeypatch.setattr("scrapling.core.ai.sleep", actions.pause)
+ intervals = Mock(side_effect=[60, 140, 0.12, 0.27, 0.18, 0.22])
+ monkeypatch.setattr("scrapling.core.ai.uniform", intervals)
+ fields = [
+ {**TEXT_FIELD, "value": "ab"},
+ {"type": "textbox", "ref": "e2", "value": ""},
+ {"type": "checkbox", "selector": "#agree", "value": True},
+ {"type": "radio", "selector": "#pro", "value": True},
+ {"type": "combobox", "selector": "#country", "value": "United Kingdom"},
+ ]
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_fill_fields", {"session_id": "browser", "fields": fields, "slowly": True, "timeout": 0}
+ )
+ assert not result.is_error
+ assert result.content == [TextContent(type="text", text="Fields filled.")]
+ assert actions.mock_calls == [
+ call.locator("#name"),
+ call.locator().fill("", timeout=0),
+ call.locator().press_sequentially("a", delay=60, timeout=0),
+ call.locator().press_sequentially("b", delay=140, timeout=0),
+ call.pause(0.12),
+ call.locator("aria-ref=e2"),
+ call.locator().fill("", timeout=0),
+ call.pause(0.27),
+ call.locator("#agree"),
+ call.locator().set_checked(True, timeout=0),
+ call.pause(0.18),
+ call.locator("#pro"),
+ call.locator().set_checked(True, timeout=0),
+ call.pause(0.22),
+ call.locator("#country"),
+ call.locator().select_option(label="United Kingdom", timeout=0),
+ ]
+ assert intervals.call_args_list == [call(50, 150), call(50, 150), *[call(0.1, 0.3)] * 4]
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("method", ["fill", "press_sequentially"])
+async def test_browser_fill_fields_slow_failure_stops_before_next_pause(
+ method: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ server, session, page = _server()
+ getattr(page.locator.return_value, method).side_effect = (
+ [None, RuntimeError("text failed")] if method == "press_sequentially" else RuntimeError("text failed")
+ )
+ pause = AsyncMock()
+ monkeypatch.setattr("scrapling.core.ai.sleep", pause)
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_fill_fields", {"session_id": "browser", "fields": [TEXT_FIELD, TEXT_FIELD], "slowly": True}
+ )
+ assert result.is_error
+ page.locator.assert_called_once_with("#name")
+ pause.assert_not_awaited()
+ if method == "fill":
+ page.locator.return_value.press_sequentially.assert_not_awaited()
+ else:
+ assert [args.args[0] for args in page.locator.return_value.press_sequentially.await_args_list] == ["p", "r"]
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "invalid",
+ [
+ {"selector": None},
+ {"ref": "e1"},
+ {"selector": ""},
+ {"selector": None, "ref": ""},
+ {"type": "unknown"},
+ {"type": None},
+ {"value": None},
+ {"value": False},
+ {"type": "checkbox", "value": "invalid"},
+ {"type": "radio", "value": False},
+ {"type": "combobox", "value": True},
+ {"type": "combobox", "value": ["Red", None]},
+ ],
+)
+async def test_browser_fill_fields_invalid_later_field_prevents_all_actions(invalid: dict[str, Any]) -> None:
+ server, session, page = _server()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool(
+ "browser_fill_fields", {"session_id": "browser", "fields": [TEXT_FIELD, {**TEXT_FIELD, **invalid}]}
+ )
+ assert result.is_error
+ page.is_closed.assert_not_called()
+ page.locator.assert_not_called()
+ assert session.page_pool.pages[0].state == "ready"
+
+
+ "values",
+ [
+ {},
+ {"fields": []},
+ {"fields": None},
+ {"fields": TEXT_FIELD},
+ {"fields": [{"type": "textbox", "selector": "#name"}]},
+ {"fields": [{"selector": "#name", "value": "value"}]},
+ {"fields": [TEXT_FIELD], "timeout": -1},
+ {"fields": [TEXT_FIELD], "slowly": "invalid"},
+ ],
+)
+async def test_browser_fill_fields_invalid_arguments_do_not_reserve_page(values: dict[str, Any]) -> None:
+ server, _, page = _server()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_fill_fields", {"session_id": "browser", **values})
+ assert result.is_error
+ page.is_closed.assert_not_called()
+ page.locator.assert_not_called()
+
+
[email protected]("timeout", [float("nan"), float("inf"), float("-inf")])
+def test_browser_fill_fields_rejects_nonfinite_timeout(timeout: float) -> None:
+ tool = ScraplingMCPServer()._build_server("127.0.0.1", 8000)._tool_manager.get_tool("browser_fill_fields")
+ assert tool is not None
+ with pytest.raises(ValidationError):
+ tool.fn_metadata.arg_model.model_validate({"session_id": "browser", "fields": [TEXT_FIELD], "timeout": timeout})
+
+
+ "state, message", [("missing", "not found"), ("static", "'static'"), ("empty", "browser_fetch")]
+)
+async def test_browser_fill_fields_session_errors_reach_mcp(state: str, message: str) -> None:
+ server, session, page = _server("static" if state == "static" else "dynamic")
+ if state == "missing":
+ server._sessions.clear()
+ elif state == "empty":
+ session.page_pool.clear()
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_fill_fields", {"session_id": "browser", "fields": [TEXT_FIELD]})
+ assert result.is_error
+ assert result.content and isinstance(result.content[0], TextContent)
+ assert message in result.content[0].text
+ page.locator.assert_not_called()
+
+
+async def test_browser_fill_fields_error_stops_remaining_fields_and_releases_page() -> None:
+ server, session, page = _server()
+ page.locator.return_value.fill.side_effect = [None, RuntimeError("field failed")]
+ fields = [{**TEXT_FIELD, "value": value} for value in ("first", "second", "third")]
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ result = await client.call_tool("browser_fill_fields", {"session_id": "browser", "fields": fields})
+ assert result.is_error
+ assert result.content and isinstance(result.content[0], TextContent)
+ assert "field failed" in result.content[0].text
+ assert page.locator.return_value.fill.await_args_list == [
+ call("first", timeout=30000),
+ call("second", timeout=30000),
+ ]
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("cancel_mode", ["task", "scope"])
+async def test_browser_fill_fields_cancellation_stops_remaining_fields(cancel_mode: str) -> None:
+ server, session, page = _server()
+ entered = asyncio.Event()
+ scope = CancelScope()
+ fields = [{**TEXT_FIELD, "value": value} for value in ("first", "second", "third")]
+
+ async def pending(value: str, **kwargs: Any) -> None:
+ if value == "second":
+ entered.set()
+ await asyncio.Event().wait()
+
+ async def fill() -> None:
+ with scope:
+ await server.browser_fill_fields("browser", fields)
+
+ page.locator.return_value.fill.side_effect = pending
+ task = asyncio.create_task(fill())
+ try:
+ await asyncio.wait_for(entered.wait(), 5)
+ assert session.page_pool.pages[0].state == "busy"
+ with pytest.raises(RuntimeError, match="busy"):
+ await server.browser_press_key("browser", ["Enter"])
+ finally:
+ scope.cancel() if cancel_mode == "scope" else task.cancel()
+ if cancel_mode == "scope":
+ await asyncio.wait_for(task, 5)
+ assert scope.cancelled_caught
+ else:
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert page.locator.return_value.fill.await_args_list == [
+ call("first", timeout=30000),
+ call("second", timeout=30000),
+ ]
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("stage", ["typing", "pause"])
[email protected]("cancel_mode", ["task", "scope"])
+async def test_browser_fill_fields_slow_cancellation_releases_page(
+ stage: str, cancel_mode: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ server, session, page = _server()
+ entered = asyncio.Event()
+ scope = CancelScope()
+
+ async def pending(*args: Any, **kwargs: Any) -> None:
+ entered.set()
+ await asyncio.Event().wait()
+
+ async def fill() -> None:
+ with scope:
+ await server.browser_fill_fields("browser", [TEXT_FIELD, TEXT_FIELD], slowly=True)
+
+ if stage == "typing":
+ page.locator.return_value.press_sequentially.side_effect = pending
+ else:
+ monkeypatch.setattr("scrapling.core.ai.sleep", pending)
+ task = asyncio.create_task(fill())
+ try:
+ await asyncio.wait_for(entered.wait(), 5)
+ assert session.page_pool.pages[0].state == "busy"
+ with pytest.raises(RuntimeError, match="busy"):
+ await server.browser_press_key("browser", ["Enter"])
+ finally:
+ scope.cancel() if cancel_mode == "scope" else task.cancel()
+ if cancel_mode == "scope":
+ await asyncio.wait_for(task, 5)
+ assert scope.cancelled_caught
+ else:
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ page.locator.assert_called_once_with("#name")
+ assert session.page_pool.pages[0].state == "ready"
+
+
+@asynccontextmanager
+async def _browser(session_type: SessionType) -> AsyncGenerator[tuple[Client, Any, Any], None]:
+ server = ScraplingMCPServer(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))
+
+ async def serve(route: Any) -> None:
+ await route.fulfill(status=200, content_type="text/html", body=HTML)
+
+ async with Client(server._build_server("127.0.0.1", 8000)) as client:
+ opened = await client.call_tool("browser_open", {"session_type": session_type, "session_id": "browser"})
+ assert not opened.is_error
+ try:
+ session = server._sessions["browser"].session
+ await session.context.route("**/*", serve)
+ fetched = await client.call_tool(
+ "browser_fetch", {"session_id": "browser", "url": "https://form.test/", "google_search": False}
+ )
+ assert not fetched.is_error
+ yield client, session, session.page_pool.pages[0].page
+ finally:
+ closed = await client.call_tool("close_session", {"session_id": "browser"})
+ assert not closed.is_error
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_fill_fields_live_mixed_fields_and_clearing(session_type: SessionType) -> None:
+ async with _browser(session_type) as (client, session, page):
+ snapshot = await client.call_tool("browser_snapshot", {"session_id": "browser"})
+ assert not snapshot.is_error
+ assert snapshot.content and isinstance(snapshot.content[0], TextContent)
+ name = search(r'textbox "Name".*?\[ref=([^\]]+)\]', snapshot.content[0].text)
+ agree = search(r'checkbox "Agree".*?\[ref=([^\]]+)\]', snapshot.content[0].text)
+ assert name is not None and agree is not None
+ fields = [
+ {"type": "textbox", "ref": name[1], "value": "new α name"},
+ {"type": "textbox", "selector": 'xpath=//textarea[@id="notes"]', "value": "line one\nline two"},
+ {"type": "textbox", "selector": "#editor", "value": "edited content"},
+ {"type": "checkbox", "ref": agree[1], "value": True},
+ {"type": "checkbox", "selector": "#updates", "value": False},
+ {"type": "radio", "selector": "#pro", "value": True},
+ {"type": "combobox", "selector": "#country", "value": "United Kingdom"},
+ {"type": "combobox", "selector": "#colors", "value": ["Red", "Blue"]},
+ ]
+ filled = await client.call_tool("browser_fill_fields", {"session_id": "browser", "fields": fields})
+ assert not filled.is_error
+ assert await page.locator("#name").input_value() == "new α name"
+ assert await page.locator("#notes").input_value() == "line one\nline two"
+ assert await page.locator("#editor").inner_text() == "edited content"
+ for selector, expected in (("#agree", True), ("#updates", False), ("#pro", True), ("#basic", False)):
+ assert await page.locator(selector).is_checked() is expected
+ assert await page.locator("#country").input_value() == "gb"
+ assert await page.locator("#colors").evaluate(
+ "element => [...element.selectedOptions].map(option => option.value)"
+ ) == ["r", "b"]
+ events = loads(await page.locator("#events").input_value())
+ assert list(dict.fromkeys(event["target"] for event in events)) == [
+ "name",
+ "notes",
+ "editor",
+ "agree",
+ "updates",
+ "pro",
+ "country",
+ "colors",
+ ]
+ assert {event["target"] for event in events if event["type"] == "change"} >= {
+ "agree",
+ "updates",
+ "pro",
+ "country",
+ "colors",
+ }
+ cleared = await client.call_tool(
+ "browser_fill_fields",
+ {
+ "session_id": "browser",
+ "fields": [
+ {"type": "textbox", "selector": "#name", "value": ""},
+ {"type": "checkbox", "selector": "#agree", "value": False},
+ {"type": "checkbox", "selector": "#updates", "value": True},
+ {"type": "radio", "selector": "#basic", "value": True},
+ {"type": "combobox", "selector": "#colors", "value": []},
+ ],
+ },
+ )
+ assert not cleared.is_error
+ assert await page.locator("#name").input_value() == ""
+ for selector, expected in (("#agree", False), ("#updates", True), ("#basic", True), ("#pro", False)):
+ assert await page.locator(selector).is_checked() is expected
+ assert await page.locator("#colors").evaluate("element => element.selectedOptions.length") == 0
+ assert await page.locator("#submitted").inner_text() == "0"
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_fill_fields_live_slow_typing_and_field_delays(session_type: SessionType) -> None:
+ async with _browser(session_type) as (client, session, page):
+ snapshot = await client.call_tool("browser_snapshot", {"session_id": "browser"})
+ assert not snapshot.is_error
+ assert snapshot.content and isinstance(snapshot.content[0], TextContent)
+ name = search(r'textbox "Name".*?\[ref=([^\]]+)\]', snapshot.content[0].text)
+ assert name is not None
+ fields = [
+ {"type": "textbox", "ref": name[1], "value": "abc"},
+ {"type": "textbox", "selector": "#notes", "value": "x\ny"},
+ {"type": "textbox", "selector": "#editor", "value": "xyα🙂"},
+ {"type": "checkbox", "selector": "#agree", "value": True},
+ {"type": "radio", "selector": "#pro", "value": True},
+ {"type": "combobox", "selector": "#country", "value": "United Kingdom"},
+ ]
+ result = await client.call_tool(
+ "browser_fill_fields", {"session_id": "browser", "fields": fields, "slowly": True}
+ )
+ assert not result.is_error
+ assert await page.locator("#name").input_value() == "abc"
+ assert await page.locator("#notes").input_value() == "x\ny"
+ assert await page.locator("#editor").inner_text() == "xyα🙂"
+ assert await page.locator("#agree").is_checked()
+ assert await page.locator("#pro").is_checked()
+ assert await page.locator("#country").input_value() == "gb"
+ events = loads(await page.locator("#events").input_value())
+ keys = [
+ event
+ for event in events
+ if event["target"] == "name" and event["type"] == "keydown" and event["key"] in "abc"
+ ]
+ assert [event["key"] for event in keys] == ["a", "b", "c"]
+ assert all(second["time"] - first["time"] >= 40 for first, second in zip(keys, keys[1:]))
+ for previous, following in zip(
+ ("name", "notes", "editor", "agree", "pro"), ("notes", "editor", "agree", "pro", "country")
+ ):
+ previous_time = max(
+ event["time"] for event in events if event["target"] == previous and event["type"] == "input"
+ )
+ next_time = min(
+ event["time"] for event in events if event["target"] == following and event["type"] == "input"
+ )
+ assert next_time - previous_time >= 80
+ cleared = await client.call_tool(
+ "browser_fill_fields",
+ {"session_id": "browser", "fields": [{**TEXT_FIELD, "value": ""}], "slowly": True},
+ )
+ assert not cleared.is_error
+ assert await page.locator("#name").input_value() == ""
+ assert await page.locator("#submitted").inner_text() == "0"
+ assert session.page_pool.pages[0].state == "ready"
+
+
[email protected]("session_type", ["dynamic", "stealthy"])
+async def test_browser_fill_fields_live_failed_field_preserves_prior_changes(session_type: SessionType) -> None:
+ async with _browser(session_type) as (client, session, page):
+ for selector in ("#missing", "input"):
+ await page.locator("#name").fill("initial")
+ result = await client.call_tool(
+ "browser_fill_fields",
+ {
+ "session_id": "browser",
+ "timeout": 100,
+ "fields": [
+ {**TEXT_FIELD, "value": "changed first"},
+ {**TEXT_FIELD, "selector": selector, "value": "fails"},
+ {**TEXT_FIELD, "selector": "#notes", "value": "must not run"},
+ ],
+ },
+ )
+ assert result.is_error
+ assert result.content and isinstance(result.content[0], TextContent)
+ assert ("Timeout" if selector == "#missing" else "strict mode violation") in result.content[0].text
+ assert await page.locator("#name").input_value() == "changed first"
+ assert await page.locator("#notes").input_value() == "initial notes"
+ assert await page.locator("#submitted").inner_text() == "0"
+ assert session.page_pool.pages[0].state == "ready"