Scrapy, a fast high-level web crawling & scraping framework for Python.
Slight improvements for the signal docs.
Andrey Rakhmatullinbf1bfaaa3e58
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 59742ffeb..a815ffb43 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -60,6 +60,7 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:.. code-block:: pythonimport scrapy+ import treqclass SignalSpider(scrapy.Spider):@@ -103,6 +104,7 @@ Built-in signals referenceHere's the list of Scrapy built-in signals and their meaning.+Engine signals--------------@@ -114,7 +116,7 @@ engine_startedSent when the Scrapy engine has started crawling.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`... note:: This signal may be fired *after* the :signal:`spider_opened` signal,depending on how the spider was started. So **don't** rely on this signal@@ -129,7 +131,7 @@ engine_stoppedSent when the Scrapy engine is stopped (for example, when a crawlingprocess has finished).- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.scheduler_empty~~~~~~~~~~~~~~~@@ -144,6 +146,9 @@ scheduler_emptySee :ref:`start-requests-lazy` for an example.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.++Item signals------------@@ -164,7 +169,7 @@ item_scrapedSent when an item has been scraped, after it has passed all the:ref:`topics-item-pipeline` stages (without being dropped).- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.:param item: the scraped item:type item: :ref:`item object <item-types>`@@ -185,7 +190,7 @@ item_droppedSent after an item has been dropped from the :ref:`topics-item-pipeline`when some stage raised a :exc:`~scrapy.exceptions.DropItem` exception.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.:param item: the item dropped from the :ref:`topics-item-pipeline`:type item: :ref:`item object <item-types>`@@ -211,7 +216,7 @@ item_errorSent when a :ref:`topics-item-pipeline` generates an error (i.e. raisesan exception), except :exc:`~scrapy.exceptions.DropItem` exception.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.:param item: the item that caused the error in the :ref:`topics-item-pipeline`:type item: :ref:`item object <item-types>`@@ -227,6 +232,7 @@ item_error:param failure: the exception raised:type failure: twisted.python.failure.Failure+Spider signals--------------@@ -239,7 +245,7 @@ spider_closedSent after a spider has been closed. This can be used to release per-spiderresources reserved on :signal:`spider_opened`.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.:param spider: the spider which has been closed:type spider: :class:`~scrapy.Spider` object@@ -263,7 +269,7 @@ spider_openedreserve per-spider resources, but can be used for any task that needs to beperformed when a spider is opened.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.:param spider: the spider which has been opened:type spider: :class:`~scrapy.Spider` object@@ -294,16 +300,16 @@ spider_idleaccordingly (e.g. setting it to 'too_few_results' instead of'finished').- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param spider: the spider which has gone idle:type spider: :class:`~scrapy.Spider` object-.. note:: Scheduling some requests in your :signal:`spider_idle` handler does- **not** guarantee that it can prevent the spider from being closed,- although it sometimes can. That's because the spider may still remain idle- if all the scheduled requests are rejected by the scheduler (e.g. filtered- due to duplication).+ .. note:: Scheduling some requests in your :signal:`spider_idle` handler does+ **not** guarantee that it can prevent the spider from being closed,+ although it sometimes can. That's because the spider may still remain idle+ if all the scheduled requests are rejected by the scheduler (e.g. filtered+ due to duplication).spider_error~~~~~~~~~~~~@@ -313,7 +319,7 @@ spider_errorSent when a spider callback generates an error (i.e. raises an exception).- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param failure: the exception raised:type failure: twisted.python.failure.Failure@@ -332,12 +338,11 @@ feed_slot_closedSent when a :ref:`feed exports <topics-feed-exports>` slot is closed.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.:param slot: the slot closed:type slot: scrapy.extensions.feedexport.FeedSlot-feed_exporter_closed~~~~~~~~~~~~~~~~~~~~@@ -348,7 +353,7 @@ feed_exporter_closedduring the handling of the :signal:`spider_closed` signal by the extension,after all feed exporting has been handled.- This signal supports asynchronous handlers.+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.Request signals@@ -367,7 +372,7 @@ request_scheduledRaise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before itreaches the scheduler.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`... versionadded:: 2.11.2Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`.@@ -387,7 +392,7 @@ request_droppedSent when a :class:`~scrapy.Request`, scheduled by the engine to bedownloaded later, is rejected by the scheduler.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param request: the request that reached the scheduler:type request: :class:`~scrapy.Request` object@@ -403,7 +408,7 @@ request_reached_downloaderSent when a :class:`~scrapy.Request` reached downloader.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param request: the request that reached downloader:type request: :class:`~scrapy.Request` object@@ -422,7 +427,7 @@ request_left_downloaderSent when a :class:`~scrapy.Request` leaves the downloader, even in case offailure.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param request: the request that reached the downloader:type request: :class:`~scrapy.Request` object@@ -433,11 +438,11 @@ request_left_downloaderbytes_received~~~~~~~~~~~~~~-.. versionadded:: 2.2-.. signal:: bytes_received.. function:: bytes_received(data, request, spider)+ .. versionadded:: 2.2+Sent by the HTTP 1.1 and S3 download handlers when a group of bytes isreceived for a specific request. This signal might be fired multipletimes for the same request, with partial data each time. For instance,@@ -449,7 +454,7 @@ bytes_receivedexception. Please refer to the :ref:`topics-stop-response-download` topicfor additional information and examples.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param data: the data received by the download handler:type data: :class:`bytes` object@@ -463,11 +468,11 @@ bytes_receivedheaders_received~~~~~~~~~~~~~~~~-.. versionadded:: 2.5-.. signal:: headers_received.. function:: headers_received(headers, body_length, request, spider)+ .. versionadded:: 2.5+Sent by the HTTP 1.1 and S3 download handlers when the response headers areavailable for a given request, before downloading any additional content.@@ -476,7 +481,7 @@ headers_receivedexception. Please refer to the :ref:`topics-stop-response-download` topicfor additional information and examples.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param headers: the headers received by the download handler:type headers: :class:`scrapy.http.headers.Headers` object@@ -490,6 +495,7 @@ headers_received:param spider: the spider associated with the response:type spider: :class:`~scrapy.Spider` object+Response signals----------------@@ -502,7 +508,7 @@ response_receivedSent when the engine receives a new :class:`~scrapy.http.Response` from thedownloader.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param response: the response received:type response: :class:`~scrapy.http.Response` object@@ -524,9 +530,9 @@ response_downloaded.. signal:: response_downloaded.. function:: response_downloaded(response, request, spider)- Sent by the downloader right after a ``HTTPResponse`` is downloaded.+ Sent by the downloader right after a :class:`~scrapy.http.Response` is downloaded.- This signal does not support returning deferreds from its handlers.+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.:param response: the response downloaded:type response: :class:`~scrapy.http.Response` objectdiff --git a/scrapy/signalmanager.py b/scrapy/signalmanager.pyindex 7fd172535..283060074 100644--- a/scrapy/signalmanager.py+++ b/scrapy/signalmanager.py@@ -53,7 +53,8 @@ class SignalManager:self, signal: Any, **kwargs: Any) -> Deferred[list[tuple[Any, Any]]]:"""- Like :meth:`send_catch_log` but supports asynchronous signal handlers.+ Like :meth:`send_catch_log` but supports :ref:`asynchronous signal+ handlers <signal-deferred>`.Returns a Deferred that gets fired once all signal handlershave finished. Send a signal, catch exceptions and log them.@@ -68,7 +69,8 @@ class SignalManager:self, signal: Any, **kwargs: Any) -> list[tuple[Any, Any]]:"""- Like :meth:`send_catch_log` but supports asynchronous signal handlers.+ Like :meth:`send_catch_log` but supports :ref:`asynchronous signal+ handlers <signal-deferred>`.Returns a coroutine that completes once all signal handlershave finished. Send a signal, catch exceptions and log them.diff --git a/scrapy/utils/signal.py b/scrapy/utils/signal.pyindex d6b0a671b..552fbaa90 100644--- a/scrapy/utils/signal.py+++ b/scrapy/utils/signal.py@@ -30,7 +30,7 @@ def send_catch_log(*arguments: TypingAny,**named: TypingAny,) -> list[tuple[TypingAny, TypingAny]]:- """Like pydispatcher.robust.sendRobust but it also logs errors and returns+ """Like ``pydispatcher.robust.sendRobust()`` but it also logs errors and returnsFailures instead of exceptions."""dont_log = named.pop("dont_log", ())@@ -73,7 +73,8 @@ def send_catch_log_deferred(*arguments: TypingAny,**named: TypingAny,) -> Generator[Deferred[TypingAny], TypingAny, list[tuple[TypingAny, TypingAny]]]:- """Like send_catch_log but supports asynchronous signal handlers.+ """Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers+ <signal-deferred>`.Returns a deferred that gets fired once all signal handlers have finished."""@@ -115,7 +116,8 @@ async def send_catch_log_async(*arguments: TypingAny,**named: TypingAny,) -> list[tuple[TypingAny, TypingAny]]:- """Like send_catch_log but supports asynchronous signal handlers.+ """Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers+ <signal-deferred>`.Returns a coroutine that completes once all signal handlers have finished."""@@ -126,7 +128,7 @@ async def send_catch_log_async(def disconnect_all(signal: TypingAny = Any, sender: TypingAny = Any) -> None:"""Disconnect all signal handlers. Useful for cleaning up after running- tests+ tests."""for receiver in liveReceivers(getAllReceivers(sender, signal)):disconnect(receiver, signal=signal, sender=sender)