Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add doc sections for callbacks and errbacks (#7821)

Adrianf02a99fe71dd
diff --git a/docs/faq.rst b/docs/faq.rst
index 0446a6868..1a574e5da 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -97,7 +97,7 @@ handler documentation.
How can I scrape an item with attributes in different pages?
------------------------------------------------------------
-See :ref:`topics-request-response-ref-request-callback-arguments`.
+See :ref:`callback-data`.
How can I simulate a user login in my spider?
---------------------------------------------
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index c4e04364b..eaf492c95 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -769,7 +769,7 @@ crawlers on top of it.
Also, a common pattern is to build an item with data from more than one page,
using a :ref:`trick to pass additional data to the callbacks
-<topics-request-response-ref-request-callback-arguments>`.
+<callback-data>`.
Using spider arguments
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 9dcd9d69c..b7ddb0a57 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -21,7 +21,9 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
.. versionadded:: 2.13
-- :class:`~scrapy.Request` callbacks.
+- :class:`~scrapy.Request` :ref:`callbacks <callbacks>`, which may
+ also be defined as :term:`asynchronous generators <asynchronous
+ generator>`.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst
index c9916110d..dcff10772 100644
--- a/docs/topics/jobs.rst
+++ b/docs/topics/jobs.rst
@@ -96,9 +96,13 @@ Request serialization
---------------------
For persistence to work, :class:`~scrapy.Request` objects must be
-serializable with :mod:`pickle`, except for the ``callback`` and ``errback``
-values passed to their ``__init__`` method, which must be methods of the
-running :class:`~scrapy.Spider` class.
+serializable with :mod:`pickle`, except for the :ref:`callback
+<callbacks>` and :ref:`errback
+<errbacks>` values passed to their ``__init__``
+method, which must be methods of the running :class:`~scrapy.Spider` class.
+
+Requests that cannot be serialized are kept in memory only: they are still
+sent, but they are lost when the crawl is paused.
If you wish to log the requests that couldn't be serialized, you can set the
:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 8e565907f..75158440b 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -205,10 +205,11 @@ Request objects
Request metadata can also be accessed through the
:attr:`~scrapy.http.Response.meta` attribute of a response.
- To pass data from one spider callback to another, consider using
- :attr:`cb_kwargs` instead. However, request metadata may be the right
- choice in certain scenarios, such as to maintain some debugging data
- across all follow-up requests (e.g. the source URL).
+ To pass your own data from one spider callback to another, use
+ :attr:`cb_kwargs` instead, see :ref:`callback-data`. However, request
+ metadata may be the right choice in certain scenarios, such as to
+ maintain some debugging data across all follow-up requests (e.g. the
+ source URL).
A common use of request metadata is to define request-specific
parameters for Scrapy components (extensions, middlewares, etc.). For
@@ -248,7 +249,7 @@ Request objects
.. method:: Request.copy()
Return a new Request which is a copy of this Request. See also:
- :ref:`topics-request-response-ref-request-callback-arguments`.
+ :ref:`callback-data`.
.. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs, cls])
@@ -256,7 +257,7 @@ Request objects
given new values by whichever keyword arguments are specified. The
:attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta` attributes are shallow
copied by default (unless new values are given as arguments). See also
- :ref:`topics-request-response-ref-request-callback-arguments`.
+ :ref:`callback-data`.
.. automethod:: from_curl
@@ -347,160 +348,6 @@ Other functions related to requests
.. autofunction:: scrapy.utils.httpobj.urlparse_cached
-.. _topics-request-response-ref-request-callback-arguments:
-
-Passing additional data to callback functions
----------------------------------------------
-
-The callback of a request is a function that will be called when the response
-of that request is downloaded. The callback function will be called with the
-downloaded :class:`Response` object as its first argument.
-
-Example:
-
-.. code-block:: python
-
- def parse_page1(self, response):
- return scrapy.Request(
- "http://www.example.com/some_page.html", callback=self.parse_page2
- )
-
-
- def parse_page2(self, response):
- # this would log http://www.example.com/some_page.html
- self.logger.info("Visited %s", response.url)
-
-In some cases you may be interested in passing arguments to those callback
-functions so you can receive the arguments later, in the second callback.
-The following example shows how to achieve this by using the
-:attr:`.Request.cb_kwargs` attribute:
-
-.. code-block:: python
-
- def parse(self, response):
- request = scrapy.Request(
- "http://www.example.com/index.html",
- callback=self.parse_page2,
- cb_kwargs=dict(main_url=response.url),
- )
- request.cb_kwargs["foo"] = "bar" # add more arguments for the callback
- yield request
-
-
- def parse_page2(self, response, main_url, foo):
- yield dict(
- main_url=main_url,
- other_url=response.url,
- foo=foo,
- )
-
-.. caution:: :attr:`.Request.cb_kwargs` was introduced in version ``1.7``.
- Prior to that, using :attr:`.Request.meta` was recommended for passing
- information around callbacks. After ``1.7``, :attr:`.Request.cb_kwargs`
- became the preferred way for handling user information, leaving :attr:`.Request.meta`
- for communication with components like middlewares and extensions.
-
-.. _topics-request-response-ref-errbacks:
-
-Using errbacks to catch exceptions in request processing
---------------------------------------------------------
-
-The errback of a request is a function that will be called when an exception
-is raise while processing it.
-
-It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can
-be used to track connection establishment timeouts, DNS errors etc.
-
-Here's an example spider logging all errors and catching some specific
-errors if needed:
-
-.. code-block:: python
-
- import scrapy
-
- from scrapy.spidermiddlewares.httperror import HttpError
- from twisted.internet.error import DNSLookupError
- from twisted.internet.error import TimeoutError, TCPTimedOutError
-
-
- class ErrbackSpider(scrapy.Spider):
- name = "errback_example"
- start_urls = [
- "http://www.httpbin.org/", # HTTP 200 expected
- "http://www.httpbin.org/status/404", # Not found error
- "http://www.httpbin.org/status/500", # server issue
- "http://www.httpbin.org:12345/", # non-responding host, timeout expected
- "https://example.invalid/", # DNS error expected
- ]
-
- async def start(self):
- for u in self.start_urls:
- yield scrapy.Request(
- u,
- callback=self.parse_httpbin,
- errback=self.errback_httpbin,
- dont_filter=True,
- )
-
- def parse_httpbin(self, response):
- self.logger.info(f"Got successful response from {response.url}")
- # do something useful here...
-
- def errback_httpbin(self, failure):
- # log all failures
- self.logger.error(repr(failure))
-
- # in case you want to do something special for some errors,
- # you may need the failure's type:
-
- if failure.check(HttpError):
- # these exceptions come from HttpError spider middleware
- # you can get the non-200 response
- response = failure.value.response
- self.logger.error("HttpError on %s", response.url)
-
- elif failure.check(DNSLookupError):
- # this is the original request
- request = failure.request
- self.logger.error("DNSLookupError on %s", request.url)
-
- elif failure.check(TimeoutError, TCPTimedOutError):
- request = failure.request
- self.logger.error("TimeoutError on %s", request.url)
-
-
-.. _errback-cb_kwargs:
-
-Accessing additional data in errback functions
-----------------------------------------------
-
-In case of a failure to process the request, you may be interested in
-accessing arguments to the callback functions so you can process further
-based on the arguments in the errback. The following example shows how to
-achieve this by using ``Failure.request.cb_kwargs``:
-
-.. code-block:: python
-
- def parse(self, response):
- request = scrapy.Request(
- "http://www.example.com/index.html",
- callback=self.parse_page2,
- errback=self.errback_page2,
- cb_kwargs=dict(main_url=response.url),
- )
- yield request
-
-
- def parse_page2(self, response, main_url):
- pass
-
-
- def errback_page2(self, failure):
- yield dict(
- main_url=failure.request.cb_kwargs["main_url"],
- )
-
-
.. _request-fingerprints:
Request fingerprints
@@ -702,6 +549,319 @@ The following built-in Scrapy components have such restrictions:
45-character-long keys must be supported.
+.. _callbacks:
+
+Callbacks
+=========
+
+A callback is a function that Scrapy calls with the :class:`Response` of a
+:class:`~scrapy.Request` once that request has been downloaded, so that you can
+extract data from that response and generate additional requests to continue
+the crawl:
+
+.. code-block:: python
+
+ from scrapy import Request, Spider
+
+
+ class BookSpider(Spider):
+ name = "books"
+
+ async def start(self):
+ yield Request("https://books.toscrape.com/", callback=self.parse_home)
+
+ def parse_home(self, response):
+ for url in response.css("h3 a::attr(href)").getall():
+ yield Request(response.urljoin(url), callback=self.parse_book)
+
+ def parse_book(self, response):
+ yield {"title": response.css("h1::text").get()}
+
+Requests may also define an :ref:`errback <errbacks>`, which Scrapy calls
+instead of the callback when an exception is raised while processing the
+request or its response, e.g. a connection error or, by default, a non-2xx
+response.
+
+
+.. _callback-assignment:
+
+Assigning a callback to a request
+---------------------------------
+
+To assign a callback to a request, use the ``callback`` parameter of
+:class:`~scrapy.Request`, which sets the :attr:`.Request.callback` attribute:
+
+.. code-block:: python
+
+ from scrapy import Request
+
+
+ def parse_home(response): ...
+
+
+ request = Request("https://books.toscrape.com/", callback=parse_home)
+
+Requests with no callback, i.e. with :attr:`~scrapy.Request.callback` set to
+``None``, are handled by the :meth:`~scrapy.Spider.parse` method of the spider:
+
+.. code-block:: python
+
+ request = Request("https://books.toscrape.com/") # Handled by parse()
+
+If a request is never meant to reach a spider callback, e.g. because a
+:ref:`component <topics-components>` sends it and handles its response itself,
+assign the special :func:`~scrapy.http.request.NO_CALLBACK` value to it
+instead, so that :ref:`downloader middlewares <topics-downloader-middleware>`
+can tell such requests apart.
+
+While :attr:`~scrapy.Request.callback` only accepts callables, some spider
+classes let you also define a callback by name: both :attr:`CrawlSpider.rules
+<scrapy.spiders.CrawlSpider.rules>` and :attr:`SitemapSpider.sitemap_rules
+<scrapy.spiders.SitemapSpider.sitemap_rules>` accept the name of a spider
+method as a string.
+
+
+.. _writing-callbacks:
+
+Writing a callback
+------------------
+
+Any callable can be a callback, as long as it takes the response as its first
+positional parameter, and any :ref:`additional callback data <callback-data>`
+as keyword parameters. Spider methods are the most common choice, but plain
+functions, lambda expressions and other callable objects work as well.
+
+.. note:: If you enable :ref:`job persistence <topics-jobs>` through the
+ :setting:`JOBDIR` setting, callbacks must be methods of the running spider.
+ Requests with any other callback cannot be serialized, so they are kept in
+ memory only and lost when you pause the crawl. See
+ :ref:`request-serialization`.
+
+A callback can be:
+
+- A regular function:
+
+ .. code-block:: python
+
+ def parse(self, response):
+ return {"url": response.url}
+
+- A generator function:
+
+ .. code-block:: python
+
+ def parse(self, response):
+ yield {"url": response.url}
+
+- A coroutine function, i.e. defined with ``async def``:
+
+ .. code-block:: python
+
+ async def parse(self, response):
+ return {"url": response.url}
+
+- An asynchronous generator function:
+
+ .. code-block:: python
+
+ async def parse(self, response):
+ yield {"url": response.url}
+
+The last two allow using ``await``, ``async for`` and ``async with`` in your
+callback. See :ref:`topics-coroutines`.
+
+
+.. _callback-output:
+
+Callback output
+---------------
+
+A callback may return or yield any of the following:
+
+- ``None``, which does nothing.
+
+ Callbacks that produce no output at all, e.g. callbacks that only log
+ information about the response, are perfectly valid. ``None`` values within
+ an iterable of callback output are ignored as well.
+
+- A :class:`~scrapy.Request` object, which Scrapy schedules, downloads and
+ eventually sends to its own callback.
+
+- An :ref:`item object <topics-items>`, which Scrapy sends to the
+ :ref:`item pipelines <topics-item-pipeline>`.
+
+ Any object that is neither ``None`` nor a :class:`~scrapy.Request` object
+ is treated as an item.
+
+- An iterable of any of the values above, e.g. a list or, more commonly, a
+ generator.
+
+ :term:`Asynchronous iterables <asynchronous iterable>`, e.g. an
+ :term:`asynchronous generator`, are also supported.
+
+.. note:: When a callback *returns* an object, Scrapy iterates that object if
+ it supports iteration, except for :class:`dict`, :class:`~scrapy.Item`,
+ :class:`str` and :class:`bytes` objects, which are always handled as single
+ items.
+
+.. note:: In a generator callback, a ``return`` statement with a value does not
+ produce any output, since such a value is not part of what the generator
+ yields. Scrapy logs a warning when it detects such a callback, see
+ :setting:`WARN_ON_GENERATOR_RETURN_VALUE`.
+
+Before Scrapy acts on the output of a callback, that output goes through the
+:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
+of your :ref:`spider middlewares <topics-spider-middleware>`, which may modify
+it or drop part of it.
+
+If a callback raises an exception, the :attr:`~scrapy.Request.errback` of the
+request is *not* called. The exception goes through the
+:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
+method of your spider middlewares instead and, unless one of them handles it,
+Scrapy logs it and sends the :signal:`spider_error` signal.
+
+
+.. _callback-data:
+.. _topics-request-response-ref-request-callback-arguments:
+
+Passing additional data to callback functions
+---------------------------------------------
+
+In some cases you may be interested in passing data to a callback in addition
+to the response, e.g. data extracted from the response that triggered the
+request. The following example shows how to achieve this by using the
+:attr:`.Request.cb_kwargs` attribute:
+
+.. code-block:: python
+
+ from scrapy import Request
+
+
+ def parse(self, response):
+ request = Request(
+ "http://www.example.com/index.html",
+ callback=self.parse_page2,
+ cb_kwargs=dict(main_url=response.url),
+ )
+ request.cb_kwargs["foo"] = "bar" # add more arguments for the callback
+ yield request
+
+
+ def parse_page2(self, response, main_url, foo):
+ yield dict(
+ main_url=main_url,
+ other_url=response.url,
+ foo=foo,
+ )
+
+:attr:`.Request.cb_kwargs` is the recommended way to pass your own data to a
+callback. Use :attr:`.Request.meta` only for data aimed at :ref:`components
+<topics-components>`, such as middlewares and extensions.
+
+.. _errbacks:
+.. _topics-request-response-ref-errbacks:
+
+Errbacks
+========
+
+The errback of a request is a function that will be called when an exception
+is raise while processing it.
+
+It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can
+be used to track connection establishment timeouts, DNS errors etc.
+
+Here's an example spider logging all errors and catching some specific
+errors if needed:
+
+.. code-block:: python
+
+ from scrapy import Request, Spider
+ from scrapy.spidermiddlewares.httperror import HttpError
+ from twisted.internet.error import DNSLookupError
+ from twisted.internet.error import TimeoutError, TCPTimedOutError
+
+
+ class ErrbackSpider(Spider):
+ name = "errback_example"
+ start_urls = [
+ "http://www.httpbin.org/", # HTTP 200 expected
+ "http://www.httpbin.org/status/404", # Not found error
+ "http://www.httpbin.org/status/500", # server issue
+ "http://www.httpbin.org:12345/", # non-responding host, timeout expected
+ "https://example.invalid/", # DNS error expected
+ ]
+
+ async def start(self):
+ for u in self.start_urls:
+ yield Request(
+ u,
+ callback=self.parse_httpbin,
+ errback=self.errback_httpbin,
+ dont_filter=True,
+ )
+
+ def parse_httpbin(self, response):
+ self.logger.info(f"Got successful response from {response.url}")
+ # do something useful here...
+
+ def errback_httpbin(self, failure):
+ # log all failures
+ self.logger.error(repr(failure))
+
+ # in case you want to do something special for some errors,
+ # you may need the failure's type:
+
+ if failure.check(HttpError):
+ # these exceptions come from HttpError spider middleware
+ # you can get the non-200 response
+ response = failure.value.response
+ self.logger.error("HttpError on %s", response.url)
+
+ elif failure.check(DNSLookupError):
+ # this is the original request
+ request = failure.request
+ self.logger.error("DNSLookupError on %s", request.url)
+
+ elif failure.check(TimeoutError, TCPTimedOutError):
+ request = failure.request
+ self.logger.error("TimeoutError on %s", request.url)
+
+
+.. _errback-cb_kwargs:
+
+Accessing additional data in errback functions
+----------------------------------------------
+
+In case of a failure to process the request, you may be interested in
+accessing arguments to the callback functions so you can process further
+based on the arguments in the errback. The following example shows how to
+achieve this by using ``Failure.request.cb_kwargs``:
+
+.. code-block:: python
+
+ from scrapy import Request
+
+
+ def parse(self, response):
+ request = Request(
+ "http://www.example.com/index.html",
+ callback=self.parse_page2,
+ errback=self.errback_page2,
+ cb_kwargs=dict(main_url=response.url),
+ )
+ yield request
+
+
+ def parse_page2(self, response, main_url):
+ pass
+
+
+ def errback_page2(self, failure):
+ yield dict(
+ main_url=failure.request.cb_kwargs["main_url"],
+ )
+
+
.. _topics-request-meta:
Request.meta special keys
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 69ff08fa1..8fbf0c52d 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -4,43 +4,31 @@
Spiders
=======
-Spiders are classes which define how a certain site (or a group of sites) will be
-scraped, including how to perform the crawl (i.e. follow links) and how to
-extract structured data from their pages (i.e. scraping items). In other words,
-Spiders are the place where you define the custom behaviour for crawling and
-parsing pages for a particular site (or, in some cases, a group of sites).
-
-For spiders, the scraping cycle goes through something like this:
-
-1. You start by generating the initial requests to crawl the first URLs, and
- specify a callback function to be called with the response downloaded from
- those requests.
-
- The first requests to perform are obtained by iterating the
- :meth:`~scrapy.Spider.start` method, which by default yields a
- :class:`~scrapy.Request` object for each URL in the
- :attr:`~scrapy.Spider.start_urls` spider attribute, with the
- :attr:`~scrapy.Spider.parse` method set as :attr:`~scrapy.Request.callback`
- function to handle each :class:`~scrapy.http.Response`.
-
-2. In the callback function, you parse the response (web page) and return
- :ref:`item objects <topics-items>`,
- :class:`~scrapy.Request` objects, or an iterable of these objects.
- Those Requests will also contain a callback (maybe
- the same) and will then be downloaded by Scrapy and then their
- response handled by the specified callback.
-
-3. In callback functions, you parse the page contents, typically using
- :ref:`topics-selectors` (but you can also use BeautifulSoup, lxml or whatever
- mechanism you prefer) and generate items with the parsed data.
-
-4. Finally, the items returned from the spider will be typically persisted to a
- database (in some :ref:`Item Pipeline <topics-item-pipeline>`) or written to
- a file using :ref:`topics-feed-exports`.
-
-Even though this cycle applies (more or less) to any kind of spider, there are
-different kinds of default spiders bundled into Scrapy for different purposes.
-We will talk about those types here.
+Spiders are classes that define how a site, or a group of sites, is scraped:
+which requests to send, and how to parse their responses to extract data and to
+send additional requests.
+
+A crawl goes as follows:
+
+1. Scrapy iterates the :meth:`~scrapy.Spider.start` method of the spider to
+ get the initial requests. By default, that method yields a
+ :class:`~scrapy.Request` object for each URL in
+ :attr:`~scrapy.Spider.start_urls`, with :meth:`~scrapy.Spider.parse` as
+ :ref:`callback <callbacks>`.
+
+2. Scrapy downloads each request and calls its callback with the resulting
+ :class:`~scrapy.http.Response`.
+
+3. Callbacks parse the response, typically using :ref:`topics-selectors`, and
+ return or yield :ref:`item objects <topics-items>` with the extracted data
+ and :class:`~scrapy.Request` objects to continue the crawl, which go back
+ to step 2. See :ref:`callback-output`.
+
+4. Items go through :ref:`item pipelines <topics-item-pipeline>`, and are
+ usually stored through :ref:`topics-feed-exports`.
+
+Scrapy includes different spider classes for different purposes, described
+below.
.. _topics-spiders-ref:
@@ -191,22 +179,7 @@ scrapy.Spider
.. automethod:: start
- .. method:: parse(response)
-
- This is the default callback used by Scrapy to process downloaded
- responses, when their requests don't specify a callback.
-
- The ``parse`` method is in charge of processing the response and returning
- scraped data and/or more URLs to follow. Other Requests callbacks have
- the same requirements as the :class:`~scrapy.Spider` class.
-
- This method, as well as any other Request callback, must return a
- :class:`~scrapy.Request` object, an :ref:`item object <topics-items>`, an
- iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects
- <topics-items>`, or ``None``.
-
- :param response: the response to parse
- :type response: :class:`~scrapy.http.Response`
+ .. automethod:: parse
.. method:: closed(reason)
diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py
index 68847283a..7d67bb6d7 100644
--- a/scrapy/http/request/__init__.py
+++ b/scrapy/http/request/__init__.py
@@ -169,7 +169,8 @@ class Request(object_ref):
#:
#: The callable must expect the response as its first parameter, and
#: support any additional keyword arguments set through
- #: :attr:`cb_kwargs`.
+ #: :attr:`cb_kwargs`. See :ref:`writing-callbacks` and
+ #: :ref:`callback-output`.
#:
#: In addition to an arbitrary callable, the following values are also
#: supported:
@@ -190,8 +191,7 @@ class Request(object_ref):
#: raises exceptions for non-2xx responses by default, sending them
#: to the :attr:`errback` instead.
#:
- #: .. seealso::
- #: :ref:`topics-request-response-ref-request-callback-arguments`
+ #: .. seealso:: :ref:`callbacks`
self.callback: CallbackT | None = callback
#: :class:`~collections.abc.Callable` to handle exceptions raised
@@ -200,7 +200,7 @@ class Request(object_ref):
#: The callable must expect a :exc:`~twisted.python.failure.Failure` as
#: its first parameter.
#:
- #: .. seealso:: :ref:`topics-request-response-ref-errbacks`
+ #: .. seealso:: :ref:`errbacks`
self.errback: Callable[[Failure], Any] | None = errback
self._cookies: CookiesT | None = cookies or None
diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py
index 02dfa2ac6..6244e3264 100644
--- a/scrapy/spiders/__init__.py
+++ b/scrapy/spiders/__init__.py
@@ -143,6 +143,22 @@ class Spider(object_ref):
else:
def parse(self, response: Response, **kwargs: Any) -> Any:
+ """Process *response*, i.e. extract data from it and generate new
+ requests.
+
+ This is the default :ref:`callback <callbacks>`: Scrapy uses
+ it for the response to any request that does not define a
+ :attr:`~scrapy.Request.callback`, such as the requests that
+ :meth:`start` yields by default.
+
+ Any :attr:`~scrapy.Request.cb_kwargs` of the request are passed as
+ keyword parameters.
+
+ Spiders must define this method, unless every request that they
+ send defines a callback.
+
+ See :ref:`callback-output` about the supported return values.
+ """
raise NotImplementedError(
f"{self.__class__.__name__}.parse callback is not defined"
)