🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
Commits on main
fix(shell): add missing parameters to the shell signature
0678ed1 Karim shoair committed 7 months ago
fix: emit valid XPath node test for ID elements in full-path mode
273c8c2 sjhddh committed 7 months ago
fix: prevent duplicate ID segments in full-path selector generation
1ac2673 sjhddh committed 7 months ago
fix(spider): save checkpoint before cancel_scope.cancel() on force-stop to prevent data loss (#230)
fad9efd Karim shoair committed 7 months ago
fix: save checkpoint before cancel_scope.cancel() on force-stop to prevent data loss
eaa0e8c voidborne-d committed 7 months ago
feat(security): default follow_redirects to "safe" for SSRF protection
e7f9adb Karim shoair committed 7 months ago
style(spider): removing excessive docstrings and unifying the style with the rest of the repo
9383bec Karim shoair committed 7 months ago
fix(spider): only allocate _domain_delays when robots_txt_obey is enabled
070338c Karim shoair committed 7 months ago
refactor(spiders): Make Robots.txt compliance turned off by default
ea2dd78 Karim shoair committed 7 months ago
refactor(spider): prefetch robots.txt from start_urls only
556a90f Karim shoair committed 7 months ago
fix(spider): Make delay in robots file don't affect user's concurrency settings
ec487d3 Karim shoair committed 7 months ago