Scrapy, a fast high-level web crawling & scraping framework for Python.
Commits on master
runspider: support process substitution and other extensionless file paths (#8036)
5b72ba3 Adrian committed 2 months ago
Add name/from_rules to CrawlSpider Rule for rule chaining (#8066)
03e65d7 Adrian committed 2 months ago
Support batches and multiple @url per callback in contracts (#8043)
46a20db Adrian committed 2 months ago
Support disabling file/image expiration with a negative FILES_EXPIRES/IMAGES_EXPIRES (#8046)
29484a1 Adrian committed 2 months ago
Raise on unresolvable component priority dictionary keys (#8159)
42030b2 Adrian committed 2 months ago
Init redirect middlewares in spider_opened instead of engine_started. (#8174)
5e30da4 Andrey Rakhmatullin committed 2 months ago
Build the scheduler before assinging engine.spider. (#8169)
cdc5a75 Andrey Rakhmatullin committed 2 months ago
Wait for engine_started in the reactorless shell init (#8168)
9008b6d Andrey Rakhmatullin committed 2 months ago
Close the downloader when closing the engine without a spider. (#8167)
ee9a21b Andrey Rakhmatullin committed 2 months ago
Align SPIDER_MIDDLEWARES_BASE docs with the value. (#8166)
7bb7372 Andrey Rakhmatullin committed 2 months ago
Document how to set up a Scrapy development environment (#8165)
e89eb70 Adrian committed 2 months ago
Document adjusting concurrency and delays from the telnet console (#8164)
14bf448 Adrian committed 2 months ago
Document how blocking callback code inflates download latency (#8160)
381003e Adrian committed 2 months ago
Gate CI on pre-commit and static checks, keeping PyPy and CodSpeed ungated (#8156)
bb89d98 Adrian committed 2 months ago
Document that item pipelines open and close concurrently (#8158)
49fbca3 Adrian committed 2 months ago