Scrapy, a fast high-level web crawling & scraping framework for Python.
Commits on master
Add unit tests that check download handlers behave like basic http clients (#6946) (#7059)
e98c164 Wes Galbraith committed 1 year ago
Deprecate scrapy.core.downloader.handlers.http. (#7079)
b49aa2f Andrey Rakhmatullin committed 1 year ago
UTF-8 BOM at the beginning of the file ignored (#7095)
7b215c6 Vasiliy Kiryanov committed 1 year ago
Move LogCounterHandler into a new extension (#7046)
1c6ba00 Andrey Rakhmatullin committed 1 year ago
Improve performance of deprecated spider arg checks. (#7045)
798390c Andrey Rakhmatullin committed 1 year ago
Deprecate sync methods of ItemPipelineManager/ITEM_PROCESSOR. (#7043)
dd0b071 Andrey Rakhmatullin committed 1 year ago
Drop PyPy 3.10 support, restore coverage for subprocesses (#7050)
393d715 Andrey Rakhmatullin committed 1 year ago
Fix get_meta_refresh to correctly resolve relative URLs using <base> tag (#7047)
d239fcf Ashar Khan committed 1 year ago
Deprecate the spider arg of downloader middleware methods. (#7037)
2ad81a0 Andrey Rakhmatullin committed 1 year ago
Log when AsyncCrawlerProcess or CrawlerProcess is instantiated. (#7034)
80beec4 Andrey Rakhmatullin committed 1 year ago
Add unit tests for the StatsMailer extension (#7013)
00b2be0 M Ikram Ullah Khan committed 1 year ago
Raise NotConfigured when FILES_STORE or IMAGES_STORE is None or empty (#6969)
3ee4a52 M Ikram Ullah Khan committed 1 year ago
Deprecate spider args of middlewares and pipelines. (#7006)
ed63fa9 Andrey Rakhmatullin committed 1 year ago
Deprecate spider arguments of StatsCollector methods (#7011)
b683308 Andrey Rakhmatullin committed 1 year ago
Refactor RobotsTxtMiddleware to use async def process_request(). (#6802)
edba1ad Andrey Rakhmatullin committed 1 year ago
Add ItemPipelineManager.process_item_async() and ensure_awaitable(). (#7005)
d3e15a1 Andrey Rakhmatullin committed 1 year ago
Deprecate setting overrides through spider attributes (#6994)
d27d0a4 NavpreetSSidhu committed 1 year ago
Rewrite ExecutionEngine.close_spider() to inlineCallbacks. (#6986)
a3daa36 Andrey Rakhmatullin committed 1 year ago
Deprecate various `spider` args in `MiddlewareManager` subclasses (#6984)
baa579d Andrey Rakhmatullin committed 1 year ago