🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
Commits on main
fix(fetcher): solve running browser from elevated shell on Windows
0449bd3 Karim shoair committed 1 month ago
fix: disambiguate generated selectors for first same-tag siblings (#495)
5e1383d Huang Qi committed 1 month ago
fix(parser)!: remove any left traces of `keep_cdata` parameter
e09f595 Karim shoair committed 1 month ago
fix: remove no-op strip_cdata kwarg from lxml HTMLParser to silence DeprecationWarning (#493)
609f519 Suyadi committed 1 month ago
fix(shell): apply curl -A and pass -x proxies as a URL string (#490)
bd90176 Suliman Abdulrazzaq committed 1 month ago
fix(cli): write md and txt extract output as UTF-8 (#488)
71f567e Suliman Abdulrazzaq committed 1 month ago
fix(fetchers): let request headers override session headers regardless of case (#487)
71b5794 Suliman Abdulrazzaq committed 1 month ago
fix(spiders): keep the checkpoint when a crawl ends with an exception (#486)
1537332 Suliman Abdulrazzaq committed 1 month ago
fix(cli): keep falsy json bodies in extract commandline requests (#484)
b3d0dc7 Suliman Abdulrazzaq committed 1 month ago
fix(spiders): let LinkExtractor restrict selectors match the link element itself (#483)
353fc70 Suliman Abdulrazzaq committed 1 month ago
refactor(mcp): move tools descriptions to the definition itself to save tokens on definition
0f27a77 Karim shoair committed 1 month ago
docs: fix website main page image and add automatic stars update again
9b042d0 Karim shoair committed 1 month ago
docs(agent): update the skill to catch up with all the changes
965415a Karim shoair committed 1 month ago
docs: update all pages to be clearer, fix mistakes, and improve for SEO
c8568a4 Karim shoair committed 1 month ago
feat(CrawlSpider): Add crawl strategy with values BFS or DFS
7183891 Karim shoair committed 1 month ago
refactor(rag): Renamed SiteToMarkdownSpider to ContentSpider
9cf4280 Karim shoair committed 1 month ago
docs: update the docs accordingly with the latest breaking changes
2228316 Karim shoair committed 1 month ago