🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs(agent): update skill with the latest changes
Karim shoaira61113fc2929
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex b03d956..8daa704 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.mdindex 6b3cbf7..d4519bb 100644--- a/agent-skill/Scrapling-Skill/SKILL.md+++ b/agent-skill/Scrapling-Skill/SKILL.md@@ -1,7 +1,7 @@---name: scrapling-officialdescription: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.-version: "0.4.4"+version: "0.4.5"license: Complete terms in LICENSE.txtmetadata:homepage: "https://scrapling.readthedocs.io/en/latest/index.html"@@ -40,7 +40,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers foCreate a virtual Python environment through any way available, like `venv`, then inside the environment do:-`pip install "scrapling[all]>=0.4.4"`+`pip install "scrapling[all]>=0.4.5"`Then do this to download all the browsers' dependencies:@@ -104,7 +104,7 @@ Those options are shared between the 4 HTTP request commands:| --proxy | TEXT | Proxy URL in format "http://username:password@host:port" || -s, --css-selector | TEXT | CSS selector to extract specific content from the page. It returns all matches. || -p, --params | TEXT | Query parameters in format "key=value" (can be used multiple times) |-| --follow-redirects / --no-follow-redirects | None | Whether to follow redirects (default: True) |+| --follow-redirects / --no-follow-redirects | None | Whether to follow redirects (default: "safe", rejects redirects to internal/private IPs) || --verify / --no-verify | None | Whether to verify SSL certificates (default: True) || --impersonate | TEXT | Browser to impersonate. Can be a single browser (e.g., Chrome) or a comma-separated list for random selection (e.g., Chrome, Firefox, Safari). || --stealthy-headers / --no-stealthy-headers | None | Use stealthy browser headers (default: True) |diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.mdindex 0c2f031..344dc0f 100644--- a/agent-skill/Scrapling-Skill/examples/README.md+++ b/agent-skill/Scrapling-Skill/examples/README.md@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.Make sure Scrapling is installed:```bash-pip install "scrapling[all]>=0.4.4"+pip install "scrapling[all]>=0.4.5"scrapling install --force```diff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.mdindex fa4d2a5..4115483 100644--- a/agent-skill/Scrapling-Skill/references/fetching/static.md+++ b/agent-skill/Scrapling-Skill/references/fetching/static.md@@ -15,7 +15,7 @@ All methods for making requests here share some arguments, so let's discuss them- **url**: The targeted URL- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.-- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default**+- **follow_redirects**: Controls redirect behavior. **Defaults to `"safe"`**, which follows redirects but rejects those targeting internal/private IPs (SSRF protection). Pass `True` to follow all redirects without restriction, or `False` to disable redirects entirely.- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.- **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**.@@ -50,7 +50,7 @@ Examples are the best way to explain this:>>> from scrapling.fetchers import Fetcher>>> # Basic GET>>> page = Fetcher.get('https://example.com')->>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')>>> # With parameters>>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})@@ -69,7 +69,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic GET>>> page = await AsyncFetcher.get('https://example.com')->>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')>>> # With parameters>>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})@@ -105,7 +105,7 @@ The `page` object in all cases is a [Response](choosing.md#response-object) obje>>> from scrapling.fetchers import Fetcher>>> # Basic POST>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})->>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")>>> # Another example of form-encoded data>>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)@@ -117,7 +117,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic POST>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})->>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")>>> # Another example of form-encoded data>>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)@@ -129,7 +129,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import Fetcher>>> # Basic PUT>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'})->>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')>>> # Another example of form-encoded data>>> page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})@@ -139,7 +139,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic PUT>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})->>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')>>> # Another example of form-encoded data>>> page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})@@ -149,14 +149,14 @@ And for asynchronous requests, it's a small adjustment```python>>> from scrapling.fetchers import Fetcher>>> page = Fetcher.delete('https://example.com/resource/123')->>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")>>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')```And for asynchronous requests, it's a small adjustment```python>>> from scrapling.fetchers import AsyncFetcher>>> page = await AsyncFetcher.delete('https://example.com/resource/123')->>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")>>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')```diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex 0a63642..7bbc86b 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -27,7 +27,7 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo| `retry_delay` | int | 1 | Seconds between retries || `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer || `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) |-| `follow_redirects` | bool | true | Follow HTTP redirects |+| `follow_redirects` | bool or "safe" | "safe" | Follow redirects. "safe" rejects redirects to internal/private IPs || `max_redirects` | int | 30 | Max redirects (-1 for unlimited) || `headers` | dict or null | null | Custom request headers || `cookies` | dict or null | null | Request cookies |