Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Slight improvements for the signal docs.

Andrey Rakhmatullinbf1bfaaa3e58
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 59742ffeb..a815ffb43 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -60,6 +60,7 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
.. code-block:: python
import scrapy
+ import treq
class SignalSpider(scrapy.Spider):
@@ -103,6 +104,7 @@ Built-in signals reference
Here's the list of Scrapy built-in signals and their meaning.
+
Engine signals
--------------
@@ -114,7 +116,7 @@ engine_started
Sent when the Scrapy engine has started crawling.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
.. note:: This signal may be fired *after* the :signal:`spider_opened` signal,
depending on how the spider was started. So **don't** rely on this signal
@@ -129,7 +131,7 @@ engine_stopped
Sent when the Scrapy engine is stopped (for example, when a crawling
process has finished).
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
scheduler_empty
~~~~~~~~~~~~~~~
@@ -144,6 +146,9 @@ scheduler_empty
See :ref:`start-requests-lazy` for an example.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
+
+
Item signals
------------
@@ -164,7 +169,7 @@ item_scraped
Sent when an item has been scraped, after it has passed all the
:ref:`topics-item-pipeline` stages (without being dropped).
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the scraped item
:type item: :ref:`item object <item-types>`
@@ -185,7 +190,7 @@ item_dropped
Sent after an item has been dropped from the :ref:`topics-item-pipeline`
when some stage raised a :exc:`~scrapy.exceptions.DropItem` exception.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the item dropped from the :ref:`topics-item-pipeline`
:type item: :ref:`item object <item-types>`
@@ -211,7 +216,7 @@ item_error
Sent when a :ref:`topics-item-pipeline` generates an error (i.e. raises
an exception), except :exc:`~scrapy.exceptions.DropItem` exception.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the item that caused the error in the :ref:`topics-item-pipeline`
:type item: :ref:`item object <item-types>`
@@ -227,6 +232,7 @@ item_error
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
+
Spider signals
--------------
@@ -239,7 +245,7 @@ spider_closed
Sent after a spider has been closed. This can be used to release per-spider
resources reserved on :signal:`spider_opened`.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.Spider` object
@@ -263,7 +269,7 @@ spider_opened
reserve per-spider resources, but can be used for any task that needs to be
performed when a spider is opened.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.Spider` object
@@ -294,16 +300,16 @@ spider_idle
accordingly (e.g. setting it to 'too_few_results' instead of
'finished').
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has gone idle
:type spider: :class:`~scrapy.Spider` object
-.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
- **not** guarantee that it can prevent the spider from being closed,
- although it sometimes can. That's because the spider may still remain idle
- if all the scheduled requests are rejected by the scheduler (e.g. filtered
- due to duplication).
+ .. note:: Scheduling some requests in your :signal:`spider_idle` handler does
+ **not** guarantee that it can prevent the spider from being closed,
+ although it sometimes can. That's because the spider may still remain idle
+ if all the scheduled requests are rejected by the scheduler (e.g. filtered
+ due to duplication).
spider_error
~~~~~~~~~~~~
@@ -313,7 +319,7 @@ spider_error
Sent when a spider callback generates an error (i.e. raises an exception).
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
@@ -332,12 +338,11 @@ feed_slot_closed
Sent when a :ref:`feed exports <topics-feed-exports>` slot is closed.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param slot: the slot closed
:type slot: scrapy.extensions.feedexport.FeedSlot
-
feed_exporter_closed
~~~~~~~~~~~~~~~~~~~~
@@ -348,7 +353,7 @@ feed_exporter_closed
during the handling of the :signal:`spider_closed` signal by the extension,
after all feed exporting has been handled.
- This signal supports asynchronous handlers.
+ This signal supports :ref:`asynchronous handlers <signal-deferred>`.
Request signals
@@ -367,7 +372,7 @@ request_scheduled
Raise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before it
reaches the scheduler.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
.. versionadded:: 2.11.2
Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`.
@@ -387,7 +392,7 @@ request_dropped
Sent when a :class:`~scrapy.Request`, scheduled by the engine to be
downloaded later, is rejected by the scheduler.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.Request` object
@@ -403,7 +408,7 @@ request_reached_downloader
Sent when a :class:`~scrapy.Request` reached downloader.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached downloader
:type request: :class:`~scrapy.Request` object
@@ -422,7 +427,7 @@ request_left_downloader
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
failure.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached the downloader
:type request: :class:`~scrapy.Request` object
@@ -433,11 +438,11 @@ request_left_downloader
bytes_received
~~~~~~~~~~~~~~
-.. versionadded:: 2.2
-
.. signal:: bytes_received
.. function:: bytes_received(data, request, spider)
+ .. versionadded:: 2.2
+
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
received for a specific request. This signal might be fired multiple
times for the same request, with partial data each time. For instance,
@@ -449,7 +454,7 @@ bytes_received
exception. Please refer to the :ref:`topics-stop-response-download` topic
for additional information and examples.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param data: the data received by the download handler
:type data: :class:`bytes` object
@@ -463,11 +468,11 @@ bytes_received
headers_received
~~~~~~~~~~~~~~~~
-.. versionadded:: 2.5
-
.. signal:: headers_received
.. function:: headers_received(headers, body_length, request, spider)
+ .. versionadded:: 2.5
+
Sent by the HTTP 1.1 and S3 download handlers when the response headers are
available for a given request, before downloading any additional content.
@@ -476,7 +481,7 @@ headers_received
exception. Please refer to the :ref:`topics-stop-response-download` topic
for additional information and examples.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param headers: the headers received by the download handler
:type headers: :class:`scrapy.http.headers.Headers` object
@@ -490,6 +495,7 @@ headers_received
:param spider: the spider associated with the response
:type spider: :class:`~scrapy.Spider` object
+
Response signals
----------------
@@ -502,7 +508,7 @@ response_received
Sent when the engine receives a new :class:`~scrapy.http.Response` from the
downloader.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param response: the response received
:type response: :class:`~scrapy.http.Response` object
@@ -524,9 +530,9 @@ response_downloaded
.. signal:: response_downloaded
.. function:: response_downloaded(response, request, spider)
- Sent by the downloader right after a ``HTTPResponse`` is downloaded.
+ Sent by the downloader right after a :class:`~scrapy.http.Response` is downloaded.
- This signal does not support returning deferreds from its handlers.
+ This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param response: the response downloaded
:type response: :class:`~scrapy.http.Response` object
diff --git a/scrapy/signalmanager.py b/scrapy/signalmanager.py
index 7fd172535..283060074 100644
--- a/scrapy/signalmanager.py
+++ b/scrapy/signalmanager.py
@@ -53,7 +53,8 @@ class SignalManager:
self, signal: Any, **kwargs: Any
) -> Deferred[list[tuple[Any, Any]]]:
"""
- Like :meth:`send_catch_log` but supports asynchronous signal handlers.
+ Like :meth:`send_catch_log` but supports :ref:`asynchronous signal
+ handlers <signal-deferred>`.
Returns a Deferred that gets fired once all signal handlers
have finished. Send a signal, catch exceptions and log them.
@@ -68,7 +69,8 @@ class SignalManager:
self, signal: Any, **kwargs: Any
) -> list[tuple[Any, Any]]:
"""
- Like :meth:`send_catch_log` but supports asynchronous signal handlers.
+ Like :meth:`send_catch_log` but supports :ref:`asynchronous signal
+ handlers <signal-deferred>`.
Returns a coroutine that completes once all signal handlers
have finished. Send a signal, catch exceptions and log them.
diff --git a/scrapy/utils/signal.py b/scrapy/utils/signal.py
index d6b0a671b..552fbaa90 100644
--- a/scrapy/utils/signal.py
+++ b/scrapy/utils/signal.py
@@ -30,7 +30,7 @@ def send_catch_log(
*arguments: TypingAny,
**named: TypingAny,
) -> list[tuple[TypingAny, TypingAny]]:
- """Like pydispatcher.robust.sendRobust but it also logs errors and returns
+ """Like ``pydispatcher.robust.sendRobust()`` but it also logs errors and returns
Failures instead of exceptions.
"""
dont_log = named.pop("dont_log", ())
@@ -73,7 +73,8 @@ def send_catch_log_deferred(
*arguments: TypingAny,
**named: TypingAny,
) -> Generator[Deferred[TypingAny], TypingAny, list[tuple[TypingAny, TypingAny]]]:
- """Like send_catch_log but supports asynchronous signal handlers.
+ """Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers
+ <signal-deferred>`.
Returns a deferred that gets fired once all signal handlers have finished.
"""
@@ -115,7 +116,8 @@ async def send_catch_log_async(
*arguments: TypingAny,
**named: TypingAny,
) -> list[tuple[TypingAny, TypingAny]]:
- """Like send_catch_log but supports asynchronous signal handlers.
+ """Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers
+ <signal-deferred>`.
Returns a coroutine that completes once all signal handlers have finished.
"""
@@ -126,7 +128,7 @@ async def send_catch_log_async(
def disconnect_all(signal: TypingAny = Any, sender: TypingAny = Any) -> None:
"""Disconnect all signal handlers. Useful for cleaning up after running
- tests
+ tests.
"""
for receiver in liveReceivers(getAllReceivers(sender, signal)):
disconnect(receiver, signal=signal, sender=sender)