🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
Commits on main
fix(spiders): ignore a leading byte-order mark in robots.txt and CSV feeds (#453)
6ee3322 Suliman Abdulrazzaq committed 2 months ago
fix(spiders): return no links when a LinkExtractor restriction matches nothing (#452)
f7b3137 Suliman Abdulrazzaq committed 2 months ago
fix: robots.txt fetch failures cached as allow-all (#443)
e0be0c5 Prateek Rawat committed 2 months ago
fix(spiders): cache responses only after blocked-response checks (#446)
ce29406 Kuang-xianxin committed 2 months ago
fix(spiders): include params in the request fingerprint (#448)
2beadcf Yuval Dinodia committed 2 months ago
fix(spiders): skip non-element nodes in sitemap XML instead of crashing on comments (#450)
4702b5a Александр М. committed 2 months ago
fix: keep the stealth flags when the user passes extra_flags (#430)
79f2427 Alexandr Kazmin committed 2 months ago
feat(mcp/cli): Include open Shadow DOM content in the response HTML.
6bbc6e0 Karim shoair committed 2 months ago
feat(browsers): Include open Shadow DOM content in the response HTML.
5edd203 Karim shoair committed 2 months ago
fix(static): send the request once when retries is below 1 (#420)
ac18622 Yiğit ERDOĞAN committed 2 months ago
fix(spiders): keep the request meta on cached responses (#419)
63cdc99 Yiğit ERDOĞAN committed 2 months ago
feat(fetchers): keep browser tabs open and reuse them across requests
f164713 Karim shoair committed 2 months ago