🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: replace the test website
Karim shoair8edcd611d351
diff --git a/docs/cli/interactive-shell.md b/docs/cli/interactive-shell.mdindex ab8d70d..e56939b 100644--- a/docs/cli/interactive-shell.md+++ b/docs/cli/interactive-shell.md@@ -128,7 +128,7 @@ The shell provides a few functions to help you convert curl commands from the br- **Convert Curl command to Request Object**```python- >>> curl_cmd = '''curl 'https://httpbin.org/post' \+ >>> curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \... -X POST \... -H 'Content-Type: application/json' \... -d '{"name": "test", "value": 123}' '''@@ -137,7 +137,7 @@ The shell provides a few functions to help you convert curl commands from the br>>> request.method'post'>>> request.url- 'https://httpbin.org/post'+ 'https://scrapling.requestcatcher.com/post'>>> request.headers{'Content-Type': 'application/json'}```diff --git a/docs/fetching/static.md b/docs/fetching/static.mdindex b338332..524e63e 100644--- a/docs/fetching/static.md+++ b/docs/fetching/static.md@@ -48,8 +48,8 @@ Examples are the best way to explain this, as follows.>>> from scrapling.fetchers import Fetcher>>> # Basic GET>>> page = Fetcher.get('https://example.com')->>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)->>> page = Fetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')+>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')>>> # With parameters>>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})>>>@@ -67,8 +67,8 @@ And for asynchronous requests, it's a small adjustment>>> from scrapling.fetchers import AsyncFetcher>>> # Basic GET>>> page = await AsyncFetcher.get('https://example.com')->>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)->>> page = await AsyncFetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')+>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')>>> # With parameters>>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})>>>@@ -102,9 +102,9 @@ Needless to say, the `page` object in all cases is [Response](choosing.md#respon```python>>> from scrapling.fetchers import Fetcher>>> # Basic POST->>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, params={'q': 'query'})->>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)->>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")+>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})+>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")>>> # Another example of form-encoded data>>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)>>> # JSON data@@ -114,9 +114,9 @@ And for asynchronous requests, it's a small adjustment```python>>> from scrapling.fetchers import AsyncFetcher>>> # Basic POST->>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'})->>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)->>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")+>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})+>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")>>> # Another example of form-encoded data>>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)>>> # JSON data@@ -130,7 +130,7 @@ And for asynchronous requests, it's a small adjustment>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome")>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')>>> # Another example of form-encoded data->>> page = Fetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})+>>> page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})```And for asynchronous requests, it's a small adjustment```python@@ -140,7 +140,7 @@ And for asynchronous requests, it's a small adjustment>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome")>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')>>> # Another example of form-encoded data->>> page = await AsyncFetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})+>>> page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})```#### DELETE@@ -176,8 +176,8 @@ with FetcherSession(retries=3) as session:# Make multiple requests with the same settings- page1 = session.get('https://httpbin.org/get')- page2 = session.post('https://httpbin.org/post', data={'key': 'value'})+ page1 = session.get('https://scrapling.requestcatcher.com/get')+ page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})page3 = session.get('https://api.github.com/events')# All requests share the same session and connection pool@@ -189,9 +189,9 @@ And here's an async exampleasync with FetcherSession(impersonate='firefox', http3=True) as session:# All standard HTTP methods availableresponse = async session.get('https://example.com')- response = async session.post('https://httpbin.org/post', json={'data': 'value'})- response = async session.put('https://httpbin.org/put', data={'update': 'info'})- response = async session.delete('https://httpbin.org/delete')+ response = async session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})+ response = async session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})+ response = async session.delete('https://scrapling.requestcatcher.com/delete')```or better```pythondiff --git a/docs/overview.md b/docs/overview.mdindex 51e9cac..acde1b3 100644--- a/docs/overview.md+++ b/docs/overview.md@@ -245,23 +245,23 @@ A fetcher is made for every use case.For simple HTTP requests, there's a `Fetcher` class that can be imported and used as below:```pythonfrom scrapling.fetchers import Fetcher-page = Fetcher.get('https://httpbin.org/get', impersonate="chrome")+page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")```With that out of the way, here's how to do all HTTP methods:```python>>> from scrapling.fetchers import Fetcher->>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)->>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')->>> page = Fetcher.put('https://httpbin.org/put', data={'key': 'value'})->>> page = Fetcher.delete('https://httpbin.org/delete')+>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')+>>> page = Fetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})+>>> page = Fetcher.delete('https://scrapling.requestcatcher.com/delete')```For Async requests, you will replace the import like below:```python>>> from scrapling.fetchers import AsyncFetcher->>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)->>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')->>> page = await AsyncFetcher.put('https://httpbin.org/put', data={'key': 'value'})->>> page = await AsyncFetcher.delete('https://httpbin.org/delete')+>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True)+>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')+>>> page = await AsyncFetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})+>>> page = await AsyncFetcher.delete('https://scrapling.requestcatcher.com/delete')```> Notes: