🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
Commits on main
feat(spiders): Add option to logging format and date format
35d5701 Karim shoair committed 10 months ago
feat(spiders): Add a stream mode to collect items in realtime
e0f94cf Karim shoair committed 10 months ago
style: remove unwanted import from spiders main init file
f2c6216 Karim shoair committed 10 months ago
feat(spiders): add simple export system for the results
72c3c27 Karim shoair committed 10 months ago
feat(browsers): Add option to retry tabs that gives errors
a17bb1d Karim shoair committed 10 months ago
fix: make `retry_blocked_request` take the response as well
5a11c24 Karim shoair committed 10 months ago
feat(spiders): Add logic to detect blocked requests and retry them
736873a Karim shoair committed 10 months ago
feat(spiders system): a prototype of the new spiders system
059a708 Karim shoair committed 10 months ago
feat(response): Option to set the referer as the previous response url.
dd5c30c Karim shoair committed 10 months ago
fix: handle a bug with response follow referer flow argument
1443a46 Karim shoair committed 10 months ago
feat: Change logger to be flexible enough for the spiders classes
aa4817f Karim shoair committed 10 months ago
fix(browsers): solve a bug with setting referer on the request level
c5bc82d Karim shoair committed 10 months ago
refactor: internal API changes to be easily used as indicators for spiders
6a73f9d Karim shoair committed 10 months ago