Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

refactor(mcp): Move all AI related files to one submodule

Karim shoaird82f835d31de
diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md
index b25f91d..01c5b15 100644
--- a/docs/api-reference/mcp-server.md
+++ b/docs/api-reference/mcp-server.md
@@ -32,7 +32,7 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")
The standardized response structure returned by the fetch and HTTP request tools, with one response per call. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON using the models below. `browser_evaluate` returns its result as JSON in a plain text block.
-## ::: scrapling.core.ai.ResponseModel
+## ::: scrapling.core.ai.server.ResponseModel
handler: python
:docstring:
@@ -42,15 +42,15 @@ The standardized response structure returned by the fetch and HTTP request tools
`browser_network_request` returns `NetworkRequestModel` with `request_id`, `part`, `data`, and `note`. Summaries and headers are objects; the full saved body text is returned as a string, including JSON response text. `note` explains missing body data or partial headers.
-## ::: scrapling.core.ai.NetworkRequestInfo
+## ::: scrapling.core.ai._network_formatting.NetworkRequestInfo
handler: python
:docstring:
-## ::: scrapling.core.ai.NetworkRequestsModel
+## ::: scrapling.core.ai.server.NetworkRequestsModel
handler: python
:docstring:
-## ::: scrapling.core.ai.NetworkRequestModel
+## ::: scrapling.core.ai._network_formatting.NetworkRequestModel
handler: python
:docstring:
@@ -58,15 +58,15 @@ The standardized response structure returned by the fetch and HTTP request tools
Model classes for session management:
-## ::: scrapling.core.ai.SessionInfo
+## ::: scrapling.core.ai.server.SessionInfo
handler: python
:docstring:
-## ::: scrapling.core.ai.SessionCreatedModel
+## ::: scrapling.core.ai.server.SessionCreatedModel
handler: python
:docstring:
-## ::: scrapling.core.ai.SessionClosedModel
+## ::: scrapling.core.ai.server.SessionClosedModel
handler: python
:docstring:
@@ -91,6 +91,6 @@ The main MCP server class that provides all web scraping tools:
- **`browser_network_request`**: Read a recorded request's summary, headers, sent data, or text response body as structured JSON without repeating the request. Returns the selected part in full, including saved body text. Saved responses remain available after page changes, including HTTP 4xx and 5xx responses. Entries appear after capture finishes; reading history or closing the session does not wait. Captures still running at navigation or closure may be omitted. Saved bytes come from Playwright and can differ from the server's bytes. Recorded text follows Playwright's UTF-8 decoding when valid, with the declared charset as a fallback. Only response bodies with a supported text `Content-Type` are read and saved. Binary, missing, or unrecognized types keep metadata and headers without body reads; browser loading is unchanged. Size limits bound the saved history, and an optional note explains non-text, oversized, or unreadable bodies. Read needed details before closing the MCP session.
- **`browser_fetch`**: Fetch a page through an open stealthy browser session and extract HTML, Markdown, text, or a structured snapshot. Snapshots can cover the whole page or a chosen element and include element references, positions, and sizes for later interaction.
-## ::: scrapling.core.ai.ScraplingMCPServer
+## ::: scrapling.core.ai.server.ScraplingMCPServer
handler: python
:docstring:
diff --git a/scrapling/core/ai/__init__.py b/scrapling/core/ai/__init__.py
new file mode 100644
index 0000000..1a85ae5
--- /dev/null
+++ b/scrapling/core/ai/__init__.py
@@ -0,0 +1,15 @@
+from ._browser_actions import BrowserAction
+from ._network_formatting import NetworkPart, NetworkRequestInfo, NetworkRequestModel
+from .server import (
+ MCP_AUTH_TOKEN_ENV,
+ MCP_EXECUTABLE_PATH_ENV,
+ NetworkRequestsModel,
+ ResponseModel,
+ ScraplingMCPServer,
+ ScreenshotType,
+ SessionClosedModel,
+ SessionCreatedModel,
+ SessionExtractionType,
+ SessionInfo,
+ SessionType,
+)
diff --git a/scrapling/core/_browser_actions.py b/scrapling/core/ai/_browser_actions.py
similarity index 100%
rename from scrapling/core/_browser_actions.py
rename to scrapling/core/ai/_browser_actions.py
diff --git a/scrapling/core/_network_formatting.py b/scrapling/core/ai/_network_formatting.py
similarity index 100%
rename from scrapling/core/_network_formatting.py
rename to scrapling/core/ai/_network_formatting.py
diff --git a/scrapling/core/ai.py b/scrapling/core/ai/server.py
similarity index 99%
rename from scrapling/core/ai.py
rename to scrapling/core/ai/server.py
index 2412f34..62a3a48 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai/server.py
@@ -25,14 +25,14 @@ from patchright.async_api import Error as PatchrightError
from scrapling import __version__
from scrapling.core.utils import log
-from scrapling.core._browser_actions import (
+from ._browser_actions import (
BrowserAction,
NonEmptyString,
NonNegativeFiniteFloat,
_run_actions,
_validate_actions,
)
-from scrapling.core._network_formatting import NetworkPart, NetworkRequestInfo, NetworkRequestModel, _request_details
+from ._network_formatting import NetworkPart, NetworkRequestInfo, NetworkRequestModel, _request_details
from scrapling.core.shell import Convertor, _CONTROL_CHARS_PATTERN
from scrapling.engines.toolbelt.custom import Response as _ScraplingResponse
from scrapling.fetchers import FetcherSession, AsyncStealthySession
diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.py
index 6540320..e79d8b8 100644
--- a/tests/ai/test_ai_mcp.py
+++ b/tests/ai/test_ai_mcp.py
@@ -24,6 +24,8 @@ from scrapling.core.ai import (
SessionCreatedModel,
SessionClosedModel,
SessionType,
+)
+from scrapling.core.ai.server import (
_SessionEntry,
_normalize_credentials,
_session_settings,
@@ -133,7 +135,7 @@ class TestOneShotTools:
sessions.append(session)
return session
- monkeypatch.setattr("scrapling.core.ai.FetcherSession", create_session)
+ monkeypatch.setattr("scrapling.core.ai.server.FetcherSession", create_session)
server = ScraplingMCPServer()
async with Client(server._build_server("127.0.0.1", 8000)) as client:
fetched = await client.call_tool("make_request", {"url": f"{httpbin.url}/html", "css_selector": "h1"})
@@ -174,7 +176,7 @@ class TestOneShotTools:
sessions.append(session)
return session
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", create_session)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", create_session)
server = ScraplingMCPServer()
html = b"""<html><body><p id="answer">Initial</p><p>Outside</p><script>
setTimeout(() => { const p = document.querySelector('#answer'); p.textContent = 'Rendered'; p.id = 'ready'; }, 50);
@@ -211,7 +213,7 @@ setTimeout(() => { const p = document.querySelector('#answer'); p.textContent =
context = AsyncMock()
context.__aenter__.return_value = session
factory = Mock(return_value=context)
- monkeypatch.setattr("scrapling.core.ai.FetcherSession", factory)
+ monkeypatch.setattr("scrapling.core.ai.server.FetcherSession", factory)
server = ScraplingMCPServer()
existing = _SessionEntry(Mock(_is_alive=True), "static")
server._sessions["existing"] = existing
@@ -258,7 +260,9 @@ setTimeout(() => { const p = document.querySelector('#answer'); p.textContent =
request.side_effect = RuntimeError("fetch failed")
factory = Mock(return_value=context)
monkeypatch.setattr(
- "scrapling.core.ai.FetcherSession" if tool == "make_request" else "scrapling.core.ai.AsyncStealthySession",
+ "scrapling.core.ai.server.FetcherSession"
+ if tool == "make_request"
+ else "scrapling.core.ai.server.AsyncStealthySession",
factory,
)
server = ScraplingMCPServer()
@@ -309,7 +313,7 @@ class TestRequestProfiles:
@pytest.mark.parametrize("impersonate", ["unknown-browser", ["chrome", "unknown-browser"]])
async def test_invalid_profiles_fail_before_opening_sessions(self, monkeypatch, tool, impersonate):
factory = Mock()
- monkeypatch.setattr("scrapling.core.ai.FetcherSession", factory)
+ monkeypatch.setattr("scrapling.core.ai.server.FetcherSession", factory)
server = ScraplingMCPServer()
args = {"url": "https://example.com"} if tool == "make_request" else {"session_id": "profiles"}
args["impersonate"] = impersonate
@@ -599,7 +603,7 @@ class TestStaticSessionManagement:
@pytest.mark.asyncio
async def test_session_ids_are_shared_across_both_open_tools(self, server, monkeypatch):
"""A requests session and a browser session can't share the same ID"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
await server.open_request_session(session_id="shared")
with pytest.raises(ValueError, match="already exists"):
await server.browser_open(session_id="shared")
@@ -608,7 +612,7 @@ class TestStaticSessionManagement:
@pytest.mark.asyncio
async def test_session_make_request_requires_a_static_session(self, server, monkeypatch):
"""session_make_request rejects browser sessions"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
await server.browser_open(session_id="browser")
with pytest.raises(ValueError, match="requires a 'static' session"):
await server.session_make_request(url="https://example.com", session_id="browser")
@@ -635,7 +639,7 @@ class TestExecutablePath:
@pytest.mark.asyncio
async def test_browser_open_passes_executable_path(self, monkeypatch):
"""browser_open forwards per-session executable_path to the stealthy session"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
created = await server.browser_open(executable_path="/tmp/chrome")
@@ -647,7 +651,7 @@ class TestExecutablePath:
async def test_browser_open_uses_environment_default(self, monkeypatch):
"""browser_open uses SCRAPLING_EXECUTABLE_PATH when no per-call value is provided"""
monkeypatch.setenv("SCRAPLING_EXECUTABLE_PATH", "/opt/custom-chromium")
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
created = await server.browser_open()
@@ -658,7 +662,7 @@ class TestExecutablePath:
@pytest.mark.asyncio
async def test_browser_open_overrides_global_executable_path(self, monkeypatch):
"""browser_open prefers the supplied executable path to the server default."""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer(executable_path="/opt/default-chromium")
created = await server.browser_open(executable_path="/opt/request-chromium")
@@ -669,7 +673,7 @@ class TestExecutablePath:
@pytest.mark.asyncio
async def test_browser_open_uses_global_executable_path(self, monkeypatch):
"""browser_open forwards the server executable path default."""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer(executable_path="/opt/default-chromium")
created = await server.browser_open()
@@ -680,7 +684,7 @@ class TestExecutablePath:
@pytest.mark.asyncio
async def test_browser_fetch_once_overrides_global_executable_path(self, monkeypatch):
"""browser_fetch_once forwards a per-call executable_path instead of the server default"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer(executable_path="/opt/default-chromium")
result = await server.browser_fetch_once(url="https://example.com", executable_path="/opt/request-chromium")
@@ -691,7 +695,7 @@ class TestExecutablePath:
@pytest.mark.asyncio
async def test_browser_fetch_once_uses_global_executable_path(self, monkeypatch):
"""browser_fetch_once forwards the server executable_path default"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer(executable_path="/opt/default-chromium")
result = await server.browser_fetch_once(url="https://example.com")
@@ -702,7 +706,7 @@ class TestExecutablePath:
@pytest.mark.asyncio
async def test_browser_fetch_once_uses_environment_default(self, monkeypatch):
monkeypatch.setenv("SCRAPLING_EXECUTABLE_PATH", "/opt/environment-chromium")
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
await ScraplingMCPServer().browser_fetch_once(url="https://example.com")
assert _FakeStealthySession.instances[0].kwargs["executable_path"] == "/opt/environment-chromium"
assert not _FakeStealthySession.instances[0]._is_alive
@@ -744,7 +748,7 @@ class TestOneShotBrowserForwarding:
)
async def test_each_call_uses_a_closed_stealth_session_with_all_options(self, monkeypatch, options):
monkeypatch.delenv("SCRAPLING_EXECUTABLE_PATH", raising=False)
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
_FakeStealthySession.instances = []
server = ScraplingMCPServer()
existing = _SessionEntry(Mock(_is_alive=True), "stealthy")
@@ -804,7 +808,7 @@ class TestBrowserFetchForwarding:
@pytest.mark.asyncio
async def test_stealthy_browser_fetch_forwards_the_per_request_params(self, monkeypatch):
"""A stealthy session receives every stealthy per-request param, including explicit None values"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
opened = await server.browser_open()
@@ -830,7 +834,7 @@ class TestBrowserFetchForwarding:
@pytest.mark.asyncio
async def test_stealthy_browser_fetch_forwards_solve_cloudflare(self, monkeypatch):
"""A stealthy session additionally receives solve_cloudflare"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
opened = await server.browser_open()
@@ -842,7 +846,7 @@ class TestBrowserFetchForwarding:
@pytest.mark.asyncio
async def test_supplied_values_are_forwarded(self, monkeypatch):
"""Per-request overrides reach the session as given"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
options = {
"wait": 250,
@@ -925,7 +929,7 @@ class TestModeSplitContract:
@pytest.mark.asyncio
async def test_browser_open_forwards_proxy_to_the_session(self, monkeypatch):
"""The session-level proxy reaches the underlying session so it applies to every fetch"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
_FakeStealthySession.instances = []
server = ScraplingMCPServer()
@@ -935,7 +939,7 @@ class TestModeSplitContract:
@pytest.mark.asyncio
async def test_browser_open_preserves_stealth_session_options(self, monkeypatch):
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
options = {
"headless": False,
@@ -1000,7 +1004,7 @@ class TestSessionSettingsReceipt:
@pytest.mark.asyncio
async def test_browser_open_and_list_report_the_receipt(self, monkeypatch):
"""browser_open returns the receipt and list_sessions reports the same one"""
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()
created = await server.browser_open()
@@ -1405,8 +1409,8 @@ class TestServerToolRegistration:
async def test_one_shot_tools_reject_batches_before_opening_sessions(self, monkeypatch, tool, args):
static = Mock()
browser = Mock()
- monkeypatch.setattr("scrapling.core.ai.FetcherSession", static)
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", browser)
+ monkeypatch.setattr("scrapling.core.ai.server.FetcherSession", static)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", browser)
async with Client(ScraplingMCPServer()._build_server("127.0.0.1", 8000)) as client:
result = await client.call_tool(tool, args)
assert result.is_error
@@ -1479,7 +1483,7 @@ class TestServerToolRegistration:
@pytest.mark.asyncio
async def test_shadow_option_changes_per_session_request(monkeypatch):
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
opened = await client.call_tool("browser_open", {"session_id": "shadow-test"})
@@ -1499,7 +1503,7 @@ async def test_shadow_option_changes_per_session_request(monkeypatch):
@pytest.mark.asyncio
@pytest.mark.parametrize("enabled", [False, True])
async def test_shadow_option_reaches_browser_session(monkeypatch, enabled):
- monkeypatch.setattr("scrapling.core.ai.AsyncStealthySession", _FakeStealthySession)
+ monkeypatch.setattr("scrapling.core.ai.server.AsyncStealthySession", _FakeStealthySession)
server = ScraplingMCPServer()._build_server("127.0.0.1", 8000)
async with Client(server) as client:
result = await client.call_tool("browser_fetch_once", {"url": "https://example.com", "pierce_shadow": enabled})
diff --git a/tests/ai/test_browser_actions.py b/tests/ai/test_browser_actions.py
index f9c7703..0e1299d 100644
--- a/tests/ai/test_browser_actions.py
+++ b/tests/ai/test_browser_actions.py
@@ -6,7 +6,8 @@ import pytest
from mcp.client import Client
from mcp.types import TextContent
-from scrapling.core.ai import ScraplingMCPServer, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any
from scrapling.engines._browsers._base import AsyncSession
from tests.ai.test_browser_wait import _browser
@@ -36,8 +37,8 @@ async def test_mixed_actions_keep_order_and_sample_each_delay(slowly: bool, monk
events = Mock()
pause = AsyncMock()
random = Mock(side_effect=[55, 145, 0.11, 0.12, 0.13, 0.14, 0.15])
- monkeypatch.setattr("scrapling.core._browser_actions.uniform", random)
- monkeypatch.setattr("scrapling.core._browser_actions.sleep", pause)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.uniform", random)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.sleep", pause)
async def reserved(*args: Any, **kwargs: Any) -> None:
assert session.page_pool.pages[0].state == "busy"
diff --git a/tests/ai/test_browser_evaluate.py b/tests/ai/test_browser_evaluate.py
index 4896111..a2e8729 100644
--- a/tests/ai/test_browser_evaluate.py
+++ b/tests/ai/test_browser_evaluate.py
@@ -10,7 +10,8 @@ from mcp.client import Client
from mcp.types import TextContent
from patchright.async_api import Error as PatchrightError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any
from scrapling.engines._browsers._base import AsyncSession
diff --git a/tests/ai/test_browser_fill_fields.py b/tests/ai/test_browser_fill_fields.py
index b833b17..a1e3db9 100644
--- a/tests/ai/test_browser_fill_fields.py
+++ b/tests/ai/test_browser_fill_fields.py
@@ -12,7 +12,8 @@ from mcp.types import TextContent
from patchright.async_api import TimeoutError as PatchrightTimeoutError
from pydantic import ValidationError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any, AsyncGenerator
from scrapling.engines._browsers._base import AsyncSession
@@ -110,7 +111,7 @@ async def test_browser_actions_fields_forwards_native_actions_in_order(
) -> None:
server, session, page = _server()
pause = AsyncMock()
- monkeypatch.setattr("scrapling.core._browser_actions.sleep", pause)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.sleep", pause)
actions = [
{**TEXT_FIELD, "ref": None},
{"type": "textbox", "selector": None, "ref": "e2", "value": "", "clear": True},
@@ -164,7 +165,7 @@ async def test_browser_actions_fields_types_without_clearing(
slowly: bool, value: str, monkeypatch: pytest.MonkeyPatch
) -> None:
server, session, page = _server()
- monkeypatch.setattr("scrapling.core._browser_actions.uniform", Mock(side_effect=[60, 140]))
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.uniform", Mock(side_effect=[60, 140]))
async with Client(server._build_server("127.0.0.1", 8000)) as client:
result = await client.call_tool(
"browser_actions",
@@ -187,9 +188,9 @@ async def test_browser_actions_fields_slow_pacing_and_empty_text(monkeypatch: py
calls = Mock()
calls.attach_mock(page.locator, "locator")
calls.pause = AsyncMock()
- monkeypatch.setattr("scrapling.core._browser_actions.sleep", calls.pause)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.sleep", calls.pause)
intervals = Mock(side_effect=[60, 140, 0.12, 0.27, 0.18, 0.22])
- monkeypatch.setattr("scrapling.core._browser_actions.uniform", intervals)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.uniform", intervals)
actions = [
{**TEXT_FIELD, "value": "ab"},
{"type": "textbox", "ref": "e2", "value": ""},
@@ -236,7 +237,7 @@ async def test_browser_actions_fields_slow_failure_stops_before_next_pause(
[None, RuntimeError("text failed")] if method == "press_sequentially" else RuntimeError("text failed")
)
pause = AsyncMock()
- monkeypatch.setattr("scrapling.core._browser_actions.sleep", pause)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.sleep", pause)
async with Client(server._build_server("127.0.0.1", 8000)) as client:
result = await client.call_tool(
"browser_actions", {"session_id": "browser", "actions": [TEXT_FIELD, TEXT_FIELD], "slowly": True}
@@ -453,7 +454,7 @@ async def test_browser_actions_fields_slow_cancellation_releases_page(
if stage == "typing":
page.locator.return_value.press_sequentially.side_effect = pending
else:
- monkeypatch.setattr("scrapling.core._browser_actions.sleep", pending)
+ monkeypatch.setattr("scrapling.core.ai._browser_actions.sleep", pending)
task = asyncio.create_task(fill())
try:
await asyncio.wait_for(entered.wait(), 5)
diff --git a/tests/ai/test_browser_mouse.py b/tests/ai/test_browser_mouse.py
index bd51bd4..9d085fe 100644
--- a/tests/ai/test_browser_mouse.py
+++ b/tests/ai/test_browser_mouse.py
@@ -13,7 +13,8 @@ from patchright.async_api import Error as PatchrightError
from patchright.async_api import TimeoutError as PatchrightTimeoutError
from pydantic import ValidationError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any
from scrapling.engines._browsers._base import AsyncSession
diff --git a/tests/ai/test_browser_network.py b/tests/ai/test_browser_network.py
index adeb0da..803666c 100644
--- a/tests/ai/test_browser_network.py
+++ b/tests/ai/test_browser_network.py
@@ -10,8 +10,9 @@ from mcp.client import Client
from mcp.types import CallToolResult, TextContent
from scrapling.core._types import Any
-from scrapling.core._network_formatting import NetworkRequestModel, _request_details
-from scrapling.core.ai import NetworkRequestsModel, ScraplingMCPServer, _SessionEntry
+from scrapling.core.ai._network_formatting import _request_details
+from scrapling.core.ai import NetworkRequestModel, NetworkRequestsModel, ScraplingMCPServer
+from scrapling.core.ai.server import _SessionEntry
from scrapling.engines._browsers._network import NetworkRecorder
from scrapling.engines.toolbelt.custom import Response
diff --git a/tests/ai/test_browser_press_key.py b/tests/ai/test_browser_press_key.py
index c1ea7f5..e4d19c4 100644
--- a/tests/ai/test_browser_press_key.py
+++ b/tests/ai/test_browser_press_key.py
@@ -9,7 +9,8 @@ from anyio import CancelScope
from mcp.client import Client
from mcp.types import TextContent
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any, AsyncGenerator
from scrapling.engines._browsers._base import AsyncSession
diff --git a/tests/ai/test_browser_screenshot.py b/tests/ai/test_browser_screenshot.py
index 5c23ad1..8494f1f 100644
--- a/tests/ai/test_browser_screenshot.py
+++ b/tests/ai/test_browser_screenshot.py
@@ -14,7 +14,8 @@ from mcp.types import ImageContent, TextContent
from patchright.async_api import Error as PatchrightError
from pydantic import ValidationError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any
from scrapling.engines._browsers._base import AsyncSession
@@ -108,7 +109,7 @@ async def test_browser_screenshot_forwards_element_target_without_full_page(
target: dict[str, Any], options: dict[str, Any], monkeypatch: pytest.MonkeyPatch
) -> None:
server, session, page = _server()
- monkeypatch.setattr("scrapling.core.ai.monotonic", lambda: 10)
+ monkeypatch.setattr("scrapling.core.ai.server.monotonic", lambda: 10)
async def capture(**kwargs: Any) -> bytes:
assert session.page_pool.pages[0].state == "busy"
@@ -144,7 +145,7 @@ async def test_browser_screenshot_element_resolution_uses_capture_budget(
timeout: float, elapsed: float, remaining: float, monkeypatch: pytest.MonkeyPatch
) -> None:
server, session, page = _server()
- monkeypatch.setattr("scrapling.core.ai.monotonic", Mock(side_effect=[10, 10 + elapsed]))
+ monkeypatch.setattr("scrapling.core.ai.server.monotonic", Mock(side_effect=[10, 10 + elapsed]))
await server.browser_screenshot("browser", selector="#card", timeout=timeout)
locator = page.locator.return_value
locator.element_handle.assert_awaited_once_with(timeout=timeout)
diff --git a/tests/ai/test_browser_snapshot.py b/tests/ai/test_browser_snapshot.py
index e9ff594..cc3623e 100644
--- a/tests/ai/test_browser_snapshot.py
+++ b/tests/ai/test_browser_snapshot.py
@@ -8,7 +8,8 @@ from mcp.client import Client
from mcp.types import TextContent
from patchright.async_api import TimeoutError as PatchrightTimeoutError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any
from scrapling.engines._browsers._base import AsyncSession
from scrapling.engines.toolbelt.custom import Response
diff --git a/tests/ai/test_browser_wait.py b/tests/ai/test_browser_wait.py
index e105e1d..f2f370a 100644
--- a/tests/ai/test_browser_wait.py
+++ b/tests/ai/test_browser_wait.py
@@ -10,7 +10,8 @@ from mcp.client import Client
from mcp.types import TextContent
from patchright.async_api import TimeoutError as PatchrightTimeoutError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any, AsyncGenerator
from scrapling.engines._browsers._base import AsyncSession
diff --git a/tests/ai/test_browser_wheel.py b/tests/ai/test_browser_wheel.py
index 4723015..2bd01d5 100644
--- a/tests/ai/test_browser_wheel.py
+++ b/tests/ai/test_browser_wheel.py
@@ -10,7 +10,8 @@ from mcp.client import Client
from mcp.types import TextContent
from pydantic import ValidationError
-from scrapling.core.ai import ScraplingMCPServer, SessionType, _SessionEntry
+from scrapling.core.ai import ScraplingMCPServer, SessionType
+from scrapling.core.ai.server import _SessionEntry
from scrapling.core._types import Any, AsyncGenerator
from scrapling.engines._browsers._base import AsyncSession
diff --git a/tests/ai/test_tool_errors.py b/tests/ai/test_tool_errors.py
index 9b59b5f..68ae65d 100644
--- a/tests/ai/test_tool_errors.py
+++ b/tests/ai/test_tool_errors.py
@@ -9,7 +9,8 @@ from mcp.server.mcpserver.exceptions import ToolError
from mcp.types import TextContent
from patchright.async_api import Error as PatchrightError
-from scrapling.core.ai import ScraplingMCPServer, _mcp_tool
+from scrapling.core.ai import ScraplingMCPServer
+from scrapling.core.ai.server import _mcp_tool
from scrapling.core._types import Any
diff --git a/tests/fetchers/test_network_browser.py b/tests/fetchers/test_network_browser.py
index edee3f0..90ba4df 100644
--- a/tests/fetchers/test_network_browser.py
+++ b/tests/fetchers/test_network_browser.py
@@ -19,7 +19,7 @@ from playwright.async_api import Response as AsyncPlaywrightResponse
from playwright.sync_api import Response as PlaywrightResponse
from scrapling.core._types import Any, Generator
-from scrapling.core._network_formatting import _request_details
+from scrapling.core.ai._network_formatting import _request_details
from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, DynamicSession, StealthySession
from scrapling.engines.toolbelt.custom import Response
from scrapling.engines.toolbelt.proxy_rotation import ProxyRotator