🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
Commits on main
feat(proxy control): Force a proxy at request level at any given point
f885027 Karim shoair committed 9 months ago
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
ef8c5bc Karim shoair committed 9 months ago
fix(browsers): Improve stealth and speed by adjusting flags
65f1a23 Karim shoair committed 9 months ago
feat(parser): replacing `tldextract` with `tld` library
90c52c4 Karim shoair committed 10 months ago
fix(requests): return the cached fingerprint on second call
5dbbd84 Karim shoair committed 10 months ago
feat(spiders/requests): Use a reliable method for fingerprint generation
122923c Karim shoair committed 10 months ago
style: update post requests type hints to be more accurate
5aa178a Karim shoair committed 10 months ago
test: remove the tests for the removed html_utils module
d0f52c3 Karim shoair committed 10 months ago
fix(spiders): handle errors with manual pause during stream mode
25850db Karim shoair committed 10 months ago
fix(spiders): Fix for when the pausing system has large queue
4ecf8a6 Karim shoair committed 10 months ago
fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
bf5aa02 Karim shoair committed 10 months ago
feat(spiders): Change items hook to make it for processing items + add a stat for this
afe6b2e Karim shoair committed 10 months ago