🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
fix: require authentication by default for the HTTP transport (#414)
Mrityunjay SIngh029d098c065a
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex 87aacaf..9905e40 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -203,9 +203,11 @@ Or with Streamable HTTP transport:```bashscrapling-mcp --http-scrapling-mcp --http --host 127.0.0.1 --port 8000+scrapling-mcp --http --host 0.0.0.0 --port 8000```+The host defaults to `127.0.0.1`, so the server only accepts connections from the same machine. Pass `--host 0.0.0.0` to make it reachable from the network.+Docker alternative:```bash@@ -256,7 +258,7 @@ The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed br## Authentication-The stdio transport is only reachable by the program that started it, but with Streamable HTTP anyone who can reach the port can call every tool, including fetching any URL from the machine running the server. If the server listens on anything other than localhost, give it a token:+The stdio transport is only reachable by the program that started it, but with Streamable HTTP anyone who can reach the port can call every tool, including fetching any URL from the machine running the server. That's why Streamable HTTP requires authentication, so `--http` on its own refuses to start and asks you for a token:```bashscrapling-mcp --http --auth-token "$(openssl rand -hex 32)"@@ -284,6 +286,14 @@ export SCRAPLING_MCP_AUTH_TOKEN="<your-token>"scrapling-mcp --http```+If you really want an unauthenticated server, for example while testing locally on the default `127.0.0.1`, you have to ask for it with `--no-auth`:++```bash+scrapling-mcp --http --no-auth+```++Combining `--no-auth` with `--host 0.0.0.0` leaves every tool open to anyone who can reach the port, so avoid that pair outside a trusted network.+When the server listens on a public address, also tell it which host names to accept, which turns on protection against DNS-rebinding attacks. The option can be repeated:```bash@@ -295,4 +305,5 @@ scrapling-mcp --http --allowed-host 'your-server.example.com:8000'- Authentication applies to the Streamable HTTP transport only. It's ignored with stdio, and the server logs a warning to say so.- Plain HTTP sends the token in cleartext, so put the server behind a reverse proxy that terminates TLS before exposing it to the internet.- This is a single shared key, not per-client credentials, so every client uses the same token and rotating it means restarting the server.-- Starting with `--http` and no token still works for local use, but logs a warning that it's unauthenticated.\ No newline at end of file+- Starting with `--http --no-auth` still logs a warning that it's unauthenticated.+- Passing both `--auth-token` and `--no-auth` keeps the token, so the server stays authenticated instead of quietly dropping it.\ No newline at end of filediff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex 78d3151..ae39008 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -206,14 +206,15 @@ Use the following to enable 'Streamable HTTP' transport mode:```bashscrapling-mcp --http```-Hence, the default value for the host the server is listening to is '0.0.0.0' and the port is 8000, which both can be configured as below:+Hence, the default value for the host the server is listening to is '127.0.0.1' and the port is 8000, which both can be configured as below:```bash-scrapling-mcp --http --host '127.0.0.1' --port 8000+scrapling-mcp --http --host '0.0.0.0' --port 8000```+The default only accepts connections from the same machine. Pass `--host '0.0.0.0'` when you want the server to be reachable from the network, which is a separate decision from authentication below.### Authentication-The 'stdio' transport is only reachable by the program that started it, but the moment you switch to 'Streamable HTTP', anyone who can reach the port can call every tool, and that includes fetching any URL from the machine running the server. So if the server is listening on anything other than localhost, give it a token:+The 'stdio' transport is only reachable by the program that started it, but the moment you switch to 'Streamable HTTP', anyone who can reach the port can call every tool, and that includes fetching any URL from the machine running the server. That's why 'Streamable HTTP' requires authentication, so `--http` on its own refuses to start and asks you for a token:```bashscrapling-mcp --http --auth-token "$(openssl rand -hex 32)"```@@ -235,6 +236,12 @@ Passing the token on the command line leaves it in your shell history and in theexport SCRAPLING_MCP_AUTH_TOKEN="<your-token>"scrapling-mcp --http```+If you really want an unauthenticated server, for example while testing locally on the default '127.0.0.1', you have to ask for it with `--no-auth`:+```bash+scrapling-mcp --http --no-auth+```+Combining `--no-auth` with `--host '0.0.0.0'` leaves every tool open to anyone who can reach the port, so avoid that pair outside a trusted network.+When the server listens on a public address, you should also tell it which host names to accept, which turns on protection against DNS-rebinding attacks (a website your browser visits trying to talk to your server). The option can be repeated:```bashscrapling-mcp --http --allowed-host 'your-server.example.com:8000'@@ -245,7 +252,8 @@ scrapling-mcp --http --allowed-host 'your-server.example.com:8000'* Authentication applies to the 'Streamable HTTP' transport only. It's ignored with 'stdio', and the server logs a warning to tell you so.<br/>* Plain HTTP sends the token in cleartext, so put the server behind a reverse proxy that terminates TLS before exposing it to the internet.<br/>* This is a single shared key, not per-client credentials, so every client uses the same token, and rotating it means restarting the server.<br/>- * Starting the server with `--http` and no token still works for local use, but it logs a warning telling you that it's unauthenticated.+ * Starting the server with `--http --no-auth` still logs a warning telling you that it's unauthenticated.<br/>+ * Passing both `--auth-token` and `--no-auth` keeps the token, so the server stays authenticated instead of quietly dropping it.## Examplesdiff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.mdindex 952f15c..4f28ad0 100644--- a/docs/api-reference/mcp-server.md+++ b/docs/api-reference/mcp-server.md@@ -19,7 +19,7 @@ Or import the server class directly:from scrapling.core.ai import ScraplingMCPServerserver = ScraplingMCPServer()-server.serve(http=False, host="0.0.0.0", port=8000)+server.serve(http=False, host="127.0.0.1", port=8000)```To set a custom Chromium-compatible browser executable for browser-based MCP tools, pass `executable_path`:diff --git a/scrapling/cli.py b/scrapling/cli.pyindex 32a8243..8a4d905 100644--- a/scrapling/cli.py+++ b/scrapling/cli.py@@ -12,7 +12,7 @@ from scrapling.core._types import List, Optional, Dict, Tuple, Any, Callablefrom orjson import loads as json_loads, JSONDecodeErrortry:- from click import command, option, Choice, group, argument, version_option+ from click import command, option, Choice, group, argument, version_option, UsageErrorexcept (ImportError, ModuleNotFoundError) as e:raise ModuleNotFoundError("You need to install scrapling with any of the extras to enable Shell commands. See: https://scrapling.readthedocs.io/en/latest/#installation"@@ -152,8 +152,9 @@ def install(force): # pragma: no cover@option("--host",type=str,- default="0.0.0.0",- help="The host to use if streamable-http transport is enabled (Default: '0.0.0.0')",+ default="127.0.0.1",+ help="The host to use if streamable-http transport is enabled. Pass '0.0.0.0' to accept connections from "+ "the network (Default: '127.0.0.1')",)@option("--port", type=int, default=8000, help="The port to use if streamable-http transport is enabled (Default: 8000)"@@ -178,11 +179,22 @@ def install(force): # pragma: no coverhelp="Enable DNS-rebinding protection and accept requests for this host only, like 'mcp.example.com:8000' ""(repeatable). Recommended whenever the server listens on a public address",)-def mcp(http, host, port, executable_path, auth_token, allowed_host):+@option(+ "--no-auth",+ is_flag=True,+ default=False,+ help="Serve the streamable-http transport without authentication (not recommended). Without it, `--http` "+ "refuses to start unless a token is given",+)+def mcp(http, host, port, executable_path, auth_token, allowed_host, no_auth):from scrapling.core.ai import ScraplingMCPServerserver = ScraplingMCPServer(executable_path=executable_path, auth_token=auth_token)- server.serve(http, host, port, allowed_hosts=allowed_host)+ try:+ server.serve(http, host, port, allowed_hosts=allowed_host, allow_unauthenticated=no_auth)+ except ValueError as e:+ # Turn the unauthenticated-HTTP refusal into a clean CLI error instead of a traceback+ raise UsageError(str(e)) from e@command(help="Interactive scraping console")diff --git a/scrapling/core/ai.py b/scrapling/core/ai.pyindex 3253990..fa6376b 100644--- a/scrapling/core/ai.py+++ b/scrapling/core/ai.py@@ -1017,17 +1017,44 @@ class ScraplingMCPServer:)return server- def serve(self, http: bool, host: str, port: int, allowed_hosts: Sequence[str] = ()):- """Serve the MCP server."""- if http and not self._auth_token:- log.warning(- f"The MCP server is running over HTTP without authentication, so anyone who can reach "- f"{host}:{port} can use every tool, including fetching arbitrary URLs from this machine. "- f"Pass `--auth-token` (or set the {MCP_AUTH_TOKEN_ENV} environment variable) to require a bearer token."+ def serve(+ self,+ http: bool,+ host: str,+ port: int,+ allowed_hosts: Sequence[str] = (),+ allow_unauthenticated: bool = False,+ ):+ """Serve the MCP server.++ :param http: Serve over the streamable-http transport instead of stdio.+ :param host: The host to bind to when `http` is enabled.+ :param port: The port to bind to when `http` is enabled.+ :param allowed_hosts: Host names to accept, which turns on DNS-rebinding protection.+ :param allow_unauthenticated: Start the streamable-http transport without a token. The transport+ requires authentication by default, so this is the explicit opt-out.+ """+ if not http:+ if self._auth_token:+ log.warning(+ "The authentication token only applies to the streamable-http transport, so it's ignored with stdio."+ )+ elif self._auth_token:+ if allow_unauthenticated:+ log.warning(+ "An authentication token was given, so it takes precedence and the server still requires it."+ )+ elif not allow_unauthenticated:+ raise ValueError(+ f"Refusing to serve the MCP server over HTTP without authentication because anyone who can reach "+ f"{host}:{port} would be able to use every tool, including fetching arbitrary URLs from this machine. "+ f"Pass `--auth-token` (or set the {MCP_AUTH_TOKEN_ENV} environment variable) to require a bearer "+ f"token, or `--no-auth` to serve it unauthenticated anyway.")- elif self._auth_token and not http:+ else:log.warning(- "The authentication token only applies to the streamable-http transport, so it's ignored with stdio."+ f"The MCP server is running over HTTP without authentication, so anyone who can reach "+ f"{host}:{port} can use every tool, including fetching arbitrary URLs from this machine.")server = self._build_server(host, port)diff --git a/tests/ai/test_ai_mcp.py b/tests/ai/test_ai_mcp.pyindex 572ec94..7ba645f 100644--- a/tests/ai/test_ai_mcp.py+++ b/tests/ai/test_ai_mcp.py@@ -4,10 +4,12 @@ from typing import Anyfrom contextlib import contextmanagerfrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServerfrom threading import Thread+from unittest.mock import patchimport pytestimport pytest_httpbinfrom mcp.client import Client+from mcp.server import MCPServerfrom mcp.types import ImageContent, TextContentfrom scrapling import __version__ as scrapling_version@@ -574,6 +576,46 @@ class TestMCPServerAuthentication:assert len(built._tool_manager.list_tools()) == 10+ def test_http_without_a_token_refuses_to_serve(self, monkeypatch):+ """The streamable-http transport requires authentication unless the caller explicitly opts out"""+ monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)+ server = ScraplingMCPServer()++ with pytest.raises(ValueError, match="without authentication"):+ server.serve(True, "0.0.0.0", 8000)++ def test_stdio_without_a_token_still_serves(self, monkeypatch):+ """stdio is only reachable by the program that started it, so it stays unauthenticated"""+ monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)+ server = ScraplingMCPServer()++ with patch.object(MCPServer, "run") as mocked_run:+ server.serve(False, "0.0.0.0", 8000)++ mocked_run.assert_called_once_with()++ def test_http_serves_unauthenticated_when_explicitly_allowed(self, monkeypatch):+ """`--no-auth` is the opt-out, and the server still warns that it's unprotected"""+ monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)+ server = ScraplingMCPServer()++ with patch.object(MCPServer, "run") as mocked_run:+ server.serve(True, "0.0.0.0", 8000, allow_unauthenticated=True)++ assert mocked_run.call_args.kwargs["transport"] == "streamable-http"+ assert server._build_server("0.0.0.0", 8000).settings.auth is None++ def test_token_wins_over_the_opt_out(self, monkeypatch):+ """Passing both keeps authentication on instead of silently dropping the token"""+ monkeypatch.delenv(MCP_AUTH_TOKEN_ENV, raising=False)+ server = ScraplingMCPServer(auth_token=SHARED_KEY)++ with patch.object(MCPServer, "run") as mocked_run:+ server.serve(True, "0.0.0.0", 8000, allow_unauthenticated=True)++ assert mocked_run.call_args.kwargs["transport"] == "streamable-http"+ assert server._build_server("0.0.0.0", 8000).settings.auth is not None+def test_allowed_hosts_enable_dns_rebinding_protection(self):assert ScraplingMCPServer._transport_security(()) is Nonediff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.pyindex f9bdad4..dde4db6 100644--- a/tests/cli/test_cli.py+++ b/tests/cli/test_cli.py@@ -56,7 +56,9 @@ class TestCLI:result = runner.invoke(mcp)assert result.exit_code == 0mock_server.assert_called_once_with(executable_path=None, auth_token=None)- mock_instance.serve.assert_called_once_with(False, "0.0.0.0", 8000, allowed_hosts=())+ mock_instance.serve.assert_called_once_with(+ False, "127.0.0.1", 8000, allowed_hosts=(), allow_unauthenticated=False+ )def test_mcp_command_with_executable_path(self, runner):"""Test MCP command with a custom browser executable"""@@ -67,7 +69,9 @@ class TestCLI:result = runner.invoke(mcp, ["--executable-path", "/opt/custom-chromium"])assert result.exit_code == 0mock_server.assert_called_once_with(executable_path="/opt/custom-chromium", auth_token=None)- mock_instance.serve.assert_called_once_with(False, "0.0.0.0", 8000, allowed_hosts=())+ mock_instance.serve.assert_called_once_with(+ False, "127.0.0.1", 8000, allowed_hosts=(), allow_unauthenticated=False+ )def test_mcp_command_with_auth_token(self, runner):"""Test MCP command with a shared authentication token"""@@ -79,7 +83,45 @@ class TestCLI:result = runner.invoke(mcp, ["--http", "--auth-token", shared_key])assert result.exit_code == 0mock_server.assert_called_once_with(executable_path=None, auth_token=shared_key)- mock_instance.serve.assert_called_once_with(True, "0.0.0.0", 8000, allowed_hosts=())+ mock_instance.serve.assert_called_once_with(+ True, "127.0.0.1", 8000, allowed_hosts=(), allow_unauthenticated=False+ )++ def test_mcp_command_with_no_auth(self, runner):+ """Test MCP command opting out of the streamable-http authentication"""+ with patch("scrapling.core.ai.ScraplingMCPServer") as mock_server:+ mock_instance = MagicMock()+ mock_server.return_value = mock_instance++ result = runner.invoke(mcp, ["--http", "--no-auth"])+ assert result.exit_code == 0+ mock_instance.serve.assert_called_once_with(+ True, "127.0.0.1", 8000, allowed_hosts=(), allow_unauthenticated=True+ )++ def test_mcp_command_binds_loopback_unless_asked_otherwise(self, runner):+ """`--http` stays off the network by default, so `--no-auth` can't expose the tools by accident"""+ with patch("scrapling.core.ai.ScraplingMCPServer") as mock_server:+ mock_instance = MagicMock()+ mock_server.return_value = mock_instance++ runner.invoke(mcp, ["--http", "--no-auth"])+ assert mock_instance.serve.call_args.args[1] == "127.0.0.1"++ mock_instance.serve.reset_mock()+ runner.invoke(mcp, ["--http", "--no-auth", "--host", "0.0.0.0"])+ assert mock_instance.serve.call_args.args[1] == "0.0.0.0"++ def test_mcp_command_reports_the_unauthenticated_refusal(self, runner):+ """The refusal raised by `serve` is shown as a CLI usage error instead of a traceback"""+ with patch("scrapling.core.ai.ScraplingMCPServer") as mock_server:+ mock_instance = MagicMock()+ mock_instance.serve.side_effect = ValueError("Refusing to serve without authentication")+ mock_server.return_value = mock_instance++ result = runner.invoke(mcp, ["--http"])+ assert result.exit_code == 2+ assert "Refusing to serve without authentication" in result.outputdef test_mcp_command_with_allowed_hosts(self, runner):"""Test MCP command with repeated allowed hosts"""@@ -92,7 +134,11 @@ class TestCLI:)assert result.exit_code == 0mock_instance.serve.assert_called_once_with(- True, "0.0.0.0", 8000, allowed_hosts=("mcp.example.com:8000", "127.0.0.1:8000")+ True,+ "127.0.0.1",+ 8000,+ allowed_hosts=("mcp.example.com:8000", "127.0.0.1:8000"),+ allow_unauthenticated=False,)def test_extract_get_command(self, runner, tmp_path, html_url):