🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
TODOs
- Add more tests and increase the code coverage.
- Structure the tests folder in a better way.
- Add more documentation.
- Add the browsing ability.
- Create detailed documentation for the 'readthedocs' website, preferably add GitHub action for deploying it.
- Create a Scrapy plugin/decorator to make it replace parsel in the response argument when needed.
- Need to add more functionality to
AttributesHandlerand more navigation functions toSelectorobject (ex: functions similar to map, filter, and reduce functions but here pass it to the element and the function is executed on children, siblings, next elements, etc...) - Add
.filtermethod toSelectorsobject and other similar methods. - Add functionality to automatically detect pagination URLs
- Add the ability to auto-detect schemas in pages and manipulate them.
- Add
analyzerability that tries to learn about the page through meta-elements and return what it learned - Add the ability to generate a regex from a group of elements (Like for all href attributes)