Scrapy, a fast high-level web crawling & scraping framework for Python.
Commits on master
Document crawl exceptions and bootstrap_failed for crawler runners (#8340)
6305c80 Adrian committed 1 month ago
Show how to crawl quotes.toscrape.com faster in the tutorial (#8334)
ffd0c46 Adrian committed 1 month ago
Make BaseDownloadHandler implement DownloadHandlerProtocol (#8333)
ca9ab94 Adrian committed 1 month ago
Stop using get_event_loop() in AsyncCrawler*. (#8327)
fbfc89a Andrey Rakhmatullin committed 1 month ago
Return partial responses on timeout when fail_on_dataloss is False (#8084)
dac8584 Adrian committed 1 month ago
Add HTTPCACHE_SCOPE to allow sharing the HTTP cache across spiders (#7964)
6ab6b90 Adrian committed 1 month ago
Add SitemapSpider.sitemap_request() to build requests from sitemap entries (#8175)
c38f20c Adrian committed 1 month ago
Use the asyncio reactor's loop when no loop is running. (#8325)
db2404b Andrey Rakhmatullin committed 1 month ago
Make the media pipeline checksum algorithm configurable (#7951)
bd0f99c Adrian committed 1 month ago
Remove the dead UnicodeDammit test and assert the exact replacement in test_replace_wrong_encoding (#8320)
e38ae3b Hygor luz committed 1 month ago
Refuse download_async() and crawl() during engine close. (#8319)
d23e63e Andrey Rakhmatullin committed 1 month ago
Recursively apply nested items' own serializers in item exporters (#8057)
4635b1d Adrian committed 1 month ago
Handle exceptions and cancellation in parallel helpers. (#8318)
2aa4668 Andrey Rakhmatullin committed 1 month ago
Make ExecutionEngine._slot request changes symmetrical. (#8315)
60e8595 Andrey Rakhmatullin committed 1 month ago
Simplify MockFTPServer to avoid flaky failures. (#8317)
5969555 Andrey Rakhmatullin committed 1 month ago