🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: Updating the README to reflect the new changes
Karim shoaird7ef555d9958
diff --git a/README.md b/README.mdindex 88a7fda..3eebf98 100644--- a/README.md+++ b/README.md@@ -6,7 +6,7 @@ Dealing with failing web scrapers due to anti-bot protections or website changesScrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity.```python->> from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher+>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher# Fetch websites' source under the radar!>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)>> print(page.status)@@ -200,18 +200,24 @@ Fetchers are interfaces built on top of other libraries with added features that### FeaturesYou might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way```python-from scrapling.fetchers import Fetcher, StealthyFetcher, PlayWrightFetcher+>>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher```-All of them can take these initialization arguments: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class.--If you don't want to pass arguments to the generated `Adaptor` object and want to use the default values, you can use this import instead for cleaner code:+then use it right away without initializing like this and it will use the default parser settings:+```python+>>> page = StealthyFetcher.fetch('https://example.com')+```+If you want to configure the parser (Adaptor class) that will be used on the response before returning it for you then do this first:```python-from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher+>>> StealthyFetcher.configure(auto_match=True, keep_comments=False)```-then use it right away without initializing like:+or```python-page = StealthyFetcher.fetch('https://example.com')+>>> StealthyFetcher.auto_match = True+>>> StealthyFetcher.keep_comments = False```+Then continue your code as normal.++The available configuration arguments are: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class.Also, the `Response` object returned from all fetchers is the same as the `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, `history`, and `request_headers`. All `cookies`, `headers`, and `request_headers` are always of type `dictionary`.> [!NOTE]@@ -225,26 +231,26 @@ For all methods, you have `stealthy_headers` which makes `Fetcher` create and usYou can route all traffic (HTTP and HTTPS) to a proxy for any of these methods in this format `http://username:password@localhost:8030````python->> page = Fetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)->> page = Fetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')->> page = Fetcher().put('https://httpbin.org/put', data={'key': 'value'})->> page = Fetcher().delete('https://httpbin.org/delete')+>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)+>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')+>> page = Fetcher.put('https://httpbin.org/put', data={'key': 'value'})+>> page = Fetcher.delete('https://httpbin.org/delete')```For Async requests, you will just replace the import like below:```python>> from scrapling.fetchers import AsyncFetcher->> page = await AsyncFetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)->> page = await AsyncFetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')->> page = await AsyncFetcher().put('https://httpbin.org/put', data={'key': 'value'})->> page = await AsyncFetcher().delete('https://httpbin.org/delete')+>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)+>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')+>> page = await AsyncFetcher.put('https://httpbin.org/put', data={'key': 'value'})+>> page = await AsyncFetcher.delete('https://httpbin.org/delete')```### StealthyFetcherThis class is built on top of [Camoufox](https://github.com/daijro/camoufox), bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to increase performance and undetectability even further.```python->> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection') # Running headless by default+>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default>> page.status == 200True->> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection') # the async version of fetch+>> page = await StealthyFetcher.async_fetch('https://www.browserscan.net/bot-detection') # the async version of fetch>> page.status == 200True```@@ -281,10 +287,10 @@ This list isn't final so expect a lot more additions and flexibility to be added### PlayWrightFetcherThis class is built on top of [Playwright](https://playwright.dev/python/) which currently provides 4 main run options but they can be mixed as you want.```python->> page = PlayWrightFetcher().fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option+>> page = PlayWrightFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option>> page.css_first("#search a::attr(href)")'https://github.com/D4Vinci/Scrapling'->> page = await PlayWrightFetcher().async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # the async version of fetch+>> page = await PlayWrightFetcher.async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # the async version of fetch>> page.css_first("#search a::attr(href)")'https://github.com/D4Vinci/Scrapling'```@@ -386,7 +392,7 @@ You can search for a specific ancestor of an element that satisfies a function,### Content-based Selection & Finding Similar ElementsYou can select elements by their text content in multiple ways, here's a full example on another website:```python->>> page = Fetcher().get('https://books.toscrape.com/index.html')+>>> page = Fetcher.get('https://books.toscrape.com/index.html')>>> page.find_by_text('Tipping the Velvet') # Find the first element whose text fully matches this text<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>@@ -566,7 +572,7 @@ Examples to clear any confusion :)```python>> from scrapling.fetchers import Fetcher->> page = Fetcher().get('https://quotes.toscrape.com/')+>> page = Fetcher.get('https://quotes.toscrape.com/')# Find all elements with tag name `div`.>> page.find_all('div')[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,