🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: update pages with the new changes
Karim shoairf756e519f38b
diff --git a/docs/fetching/static.md b/docs/fetching/static.mdindex 587ebd1..7d2a522 100644--- a/docs/fetching/static.md+++ b/docs/fetching/static.md@@ -21,7 +21,7 @@ All methods for making requests here share some arguments, so let's discuss them- **url**: The targeted URL- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.-- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default**+- **follow_redirects**: Controls redirect behavior. **Defaults to `"safe"`**, which follows redirects but rejects those targeting internal/private IPs (SSRF protection). Pass `True` to follow all redirects without restriction, or `False` to disable redirects entirely.- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.- **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**.@@ -57,7 +57,7 @@ Examples are the best way to explain this:>>> from scrapling.fetchers import Fetcher>>> # Basic GET>>> page = Fetcher.get('https://example.com')->>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')>>> # With parameters>>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})@@ -76,7 +76,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic GET>>> page = await AsyncFetcher.get('https://example.com')->>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')>>> # With parameters>>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})@@ -112,7 +112,7 @@ Needless to say, the `page` object in all cases is [Response](choosing.md#respon>>> from scrapling.fetchers import Fetcher>>> # Basic POST>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})->>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")>>> # Another example of form-encoded data>>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)@@ -124,7 +124,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic POST>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})->>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")>>> # Another example of form-encoded data>>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)@@ -136,7 +136,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import Fetcher>>> # Basic PUT>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'})->>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')>>> # Another example of form-encoded data>>> page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})@@ -146,7 +146,7 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic PUT>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})->>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')>>> # Another example of form-encoded data>>> page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})@@ -156,14 +156,14 @@ And for asynchronous requests, it's a small adjustment```python>>> from scrapling.fetchers import Fetcher>>> page = Fetcher.delete('https://example.com/resource/123')->>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")>>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')```And for asynchronous requests, it's a small adjustment```python>>> from scrapling.fetchers import AsyncFetcher>>> page = await AsyncFetcher.delete('https://example.com/resource/123')->>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True, impersonate="chrome")+>>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")>>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')```diff --git a/docs/overview.md b/docs/overview.mdindex b0d1a4e..c62c7c3 100644--- a/docs/overview.md+++ b/docs/overview.md@@ -264,7 +264,7 @@ page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chroWith that out of the way, here's how to do all HTTP methods:```python>>> from scrapling.fetchers import Fetcher->>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')>>> page = Fetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})>>> page = Fetcher.delete('https://scrapling.requestcatcher.com/delete')@@ -272,7 +272,7 @@ With that out of the way, here's how to do all HTTP methods:For Async requests, you will replace the import like below:```python>>> from scrapling.fetchers import AsyncFetcher->>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')>>> page = await AsyncFetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})>>> page = await AsyncFetcher.delete('https://scrapling.requestcatcher.com/delete')