π·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs(README file): Better more-friendly description
Karim shoair21291958f178
diff --git a/README.md b/README.mdindex ba123fd..81702a5 100644--- a/README.md+++ b/README.md@@ -92,27 +92,27 @@ Scrapling is a high-performance, intelligent web scraping library for Python tha## Key Features### Fetch websites as you prefer with async support-- **HTTP requests**: Stealthy and fast HTTP requests with `Fetcher`-- **Stealthy fetcher**: Annoying anti-bot protection? No problem! Scrapling can bypass almost all of them with `StealthyFetcher` with default configuration!-- **Your preferred browser**: Use your real browser with CDP, [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless, PlayWright with stealth mode, or even vanilla PlayWright - All is possible with `PlayWrightFetcher`!+- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class.+- **Dynamic Loading & Automation**: Fetch dynamic websites with the `PlayWrightFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless!+- **Anti-bot Protections Bypass**: Easily bypass protections with `StealthyFetcher` and `PlayWrightFetcher` classes.### Adaptive Scraping-- π **Smart Element Tracking**: Locate previously identified elements after website structure changes, using an intelligent similarity system and integrated storage.-- π― **Flexible Querying**: Use CSS selectors, XPath, Elements filters, text search, or regex - chain them however you want!-- π **Find Similar Elements**: Automatically locate elements similar to the element you want on the page (Ex: other products like the product you found on the page).+- π **Smart Element Tracking**: Relocate elements after website changes, using an intelligent similarity system and integrated storage.+- π― **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search and more.+- π **Find Similar Elements**: Automatically locate elements similar to the element you found!- π§ **Smart Content Scraping**: Extract data from multiple websites without specific selectors using Scrapling powerful features.-### Performance-- π **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries (outperforming BeautifulSoup in parsing by up to 620x in our tests).+### High Performance+- π **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries.- π **Memory Efficient**: Optimized data structures for minimal memory footprint.-- β‘ **Fast JSON serialization**: 10x faster JSON serialization than the standard json library with more options.+- β‘ **Fast JSON serialization**: 10x faster than standard library.-### Developing Experience-- π οΈ **Powerful Navigation API**: Traverse the DOM tree easily in all directions and get the info you want (parent, ancestors, sibling, children, next/previous element, and more).-- 𧬠**Rich Text Processing**: All strings have built-in methods for regex matching, cleaning, and more. All elements' attributes are read-only dictionaries that are faster than standard dictionaries with added methods.-- π **Automatic Selector Generation**: Create robust CSS/XPath selectors for any element.-- π **API Similar to Scrapy/BeautifulSoup**: Familiar methods and similar pseudo-elements for Scrapy and BeautifulSoup users.-- π **Type hints and test coverage**: Complete type coverage and almost full test coverage for better IDE support and fewer bugs, respectively.+### Developer Friendly+- π οΈ **Powerful Navigation API**: Easy DOM traversal in all directions.+- 𧬠**Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that takes less memory than standard dictionaries with added methods.+- π **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element.+- π **Familiar API**: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy.+- π **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support.## Getting Started@@ -121,21 +121,22 @@ from scrapling import Fetcherfetcher = Fetcher(auto_match=False)-# Fetch a web page and create an Adaptor instance+# Do http GET request to a web page and create an Adaptor instancepage = fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True)-# Get all strings in the full page+# Get all text content from all HTML tags in the page except `script` and `style` tagspage.get_all_text(ignore_tags=('script', 'style'))-# Get all quotes, any of these methods will return a list of strings (TextHandlers)+# Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers)quotes = page.css('.quote .text::text') # CSS selectorquotes = page.xpath('//span[@class="text"]/text()') # XPathquotes = page.css('.quote').css('.text::text') # Chained selectorsquotes = [element.text for element in page.css('.quote .text')] # Slower than bulk query above# Get the first quote element-quote = page.css_first('.quote') # / page.css('.quote').first / page.css('.quote')[0]+quote = page.css_first('.quote') # same as page.css('.quote').first or page.css('.quote')[0]# Tired of selectors? Use find_all/find+# Get all 'div' HTML tags that one of its 'class' values is 'quote'quotes = page.find_all('div', {'class': 'quote'})# Same asquotes = page.find_all('div', class_='quote')@@ -143,10 +144,10 @@ quotes = page.find_all(['div'], class_='quote')quotes = page.find_all(class_='quote') # and so on...# Working with elements-quote.html_content # Inner HTML-quote.prettify() # Prettified version of Inner HTML-quote.attrib # Element attributes-quote.path # DOM path to element (List)+quote.html_content # Get Inner HTML of this element+quote.prettify() # Prettified version of Inner HTML above+quote.attrib # Get that element's attributes+quote.path # DOM path to element (List of all ancestors from <html> tag till the element itself)```To keep it simple, all methods can be chained on top of each other!