🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
Commits on main
fix(spider robots): solve multiple issues with cache prefetch
af83a11 Karim shoair committed 7 months ago
style(spiders robots feat): Adjustments for maintainability
854daac Karim shoair committed 7 months ago
feat(spiders): pre-warm robots.txt cache before crawl loop starts
a86e970 Abdullah committed 7 months ago
refactor(spiders): simplify robots.txt cache to domain-only key
e2b293f Abdullah committed 7 months ago
fix(requests): handle missing type hint returned by curl-cffi
eaf7ec2 Karim shoair committed 7 months ago
test(spiders): add comprehensive test suite for robots.txt compliance
132f33c Abdullah committed 7 months ago
feat(spiders): integrate robots.txt compliance into the crawl engine
5c40c6a Abdullah committed 7 months ago
feat(spiders): implement RobotsTxtManager with concurrent fetch deduplication
0bbe62f Abdullah committed 7 months ago
feat(deps): add protego for robots.txt parsing and fix pyright type error in static.py
1d15349 Abdullah committed 7 months ago
fix(agent): update skill zip file with the latest changes
b354be0 Karim shoair committed 7 months ago
fix(agent): update skill zip file with the latest changes
6598664 Karim shoair committed 7 months ago
docs(mcp): add section about prompt injection in docs and skill
3ac7f76 Karim shoair committed 7 months ago