Scrapy, a fast high-level web crawling & scraping framework for Python.
Commits on master
Add DiskDupeFilter, a disk-backed duplicate request filter (#7952)
7b8eec6 Adrian committed 1 month ago
Fix counting free downloader slots on asyncio. (#8257)
bb6dc3e Andrey Rakhmatullin committed 1 month ago
Embed the MCP server introduction video in the agents docs (#8253)
5fc5de3 Adrian committed 1 month ago
Document the trust boundary of JOBDIR and HTTPCACHE_DIR (#8252)
55c3e31 Adrian committed 1 month ago
Make closing the spider stop the engine, other state transition fixes (#8217)
bb1b5c6 Andrey Rakhmatullin committed 1 month ago
Let refcounting free responses after selector use or callback errors (#8245)
b4279e9 Adrian committed 1 month ago
Log Scrapy info after add-ons, document extending LOG_VERSIONS (#8246)
84a82af Adrian committed 1 month ago
Deprecate scrapy.utils.url.add_http_if_no_scheme() in favor of w3lib (#8189)
cbc2de9 Adrian committed 1 month ago
Fix duplicated setting entries when merging parallel doc reads (#8238)
ed55178 Adrian committed 1 month ago
Use monkeypatch for os.environ manipulations. (#8236)
e2c18d9 Andrey Rakhmatullin committed 1 month ago
Remove remaining mockserver class attrs (except feedexport tests). (#8235)
74064bc Andrey Rakhmatullin committed 1 month ago
Start item-processing workers on demand instead of one per CONCURRENT_ITEMS (#7959)
c8b09e5 Adrian committed 1 month ago
Run walltime benchmarks on PRs only when compression code changes (#8227)
45dcb7d Adrian committed 1 month ago
Move the referrer policy scenarios to module-level constants. (#8228)
8006f33 Andrey Rakhmatullin committed 1 month ago