Scrapy, a fast high-level web crawling & scraping framework for Python.
Add doc sections for callbacks and errbacks (#7821)
Adrianf02a99fe71dd
diff --git a/docs/faq.rst b/docs/faq.rstindex 0446a6868..1a574e5da 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -97,7 +97,7 @@ handler documentation.How can I scrape an item with attributes in different pages?-------------------------------------------------------------See :ref:`topics-request-response-ref-request-callback-arguments`.+See :ref:`callback-data`.How can I simulate a user login in my spider?---------------------------------------------diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex c4e04364b..eaf492c95 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -769,7 +769,7 @@ crawlers on top of it.Also, a common pattern is to build an item with data from more than one page,using a :ref:`trick to pass additional data to the callbacks-<topics-request-response-ref-request-callback-arguments>`.+<callback-data>`.Using spider argumentsdiff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 9dcd9d69c..b7ddb0a57 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -21,7 +21,9 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):.. versionadded:: 2.13-- :class:`~scrapy.Request` callbacks.+- :class:`~scrapy.Request` :ref:`callbacks <callbacks>`, which may+ also be defined as :term:`asynchronous generators <asynchronous+ generator>`.- The :meth:`process_item` method of:ref:`item pipelines <topics-item-pipeline>`.diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rstindex c9916110d..dcff10772 100644--- a/docs/topics/jobs.rst+++ b/docs/topics/jobs.rst@@ -96,9 +96,13 @@ Request serialization---------------------For persistence to work, :class:`~scrapy.Request` objects must be-serializable with :mod:`pickle`, except for the ``callback`` and ``errback``-values passed to their ``__init__`` method, which must be methods of the-running :class:`~scrapy.Spider` class.+serializable with :mod:`pickle`, except for the :ref:`callback+<callbacks>` and :ref:`errback+<errbacks>` values passed to their ``__init__``+method, which must be methods of the running :class:`~scrapy.Spider` class.++Requests that cannot be serialized are kept in memory only: they are still+sent, but they are lost when the crawl is paused.If you wish to log the requests that couldn't be serialized, you can set the:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 8e565907f..75158440b 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -205,10 +205,11 @@ Request objectsRequest metadata can also be accessed through the:attr:`~scrapy.http.Response.meta` attribute of a response.- To pass data from one spider callback to another, consider using- :attr:`cb_kwargs` instead. However, request metadata may be the right- choice in certain scenarios, such as to maintain some debugging data- across all follow-up requests (e.g. the source URL).+ To pass your own data from one spider callback to another, use+ :attr:`cb_kwargs` instead, see :ref:`callback-data`. However, request+ metadata may be the right choice in certain scenarios, such as to+ maintain some debugging data across all follow-up requests (e.g. the+ source URL).A common use of request metadata is to define request-specificparameters for Scrapy components (extensions, middlewares, etc.). For@@ -248,7 +249,7 @@ Request objects.. method:: Request.copy()Return a new Request which is a copy of this Request. See also:- :ref:`topics-request-response-ref-request-callback-arguments`.+ :ref:`callback-data`... method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs, cls])@@ -256,7 +257,7 @@ Request objectsgiven new values by whichever keyword arguments are specified. The:attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta` attributes are shallowcopied by default (unless new values are given as arguments). See also- :ref:`topics-request-response-ref-request-callback-arguments`.+ :ref:`callback-data`... automethod:: from_curl@@ -347,160 +348,6 @@ Other functions related to requests.. autofunction:: scrapy.utils.httpobj.urlparse_cached-.. _topics-request-response-ref-request-callback-arguments:--Passing additional data to callback functions------------------------------------------------The callback of a request is a function that will be called when the response-of that request is downloaded. The callback function will be called with the-downloaded :class:`Response` object as its first argument.--Example:--.. code-block:: python-- def parse_page1(self, response):- return scrapy.Request(- "http://www.example.com/some_page.html", callback=self.parse_page2- )--- def parse_page2(self, response):- # this would log http://www.example.com/some_page.html- self.logger.info("Visited %s", response.url)--In some cases you may be interested in passing arguments to those callback-functions so you can receive the arguments later, in the second callback.-The following example shows how to achieve this by using the-:attr:`.Request.cb_kwargs` attribute:--.. code-block:: python-- def parse(self, response):- request = scrapy.Request(- "http://www.example.com/index.html",- callback=self.parse_page2,- cb_kwargs=dict(main_url=response.url),- )- request.cb_kwargs["foo"] = "bar" # add more arguments for the callback- yield request--- def parse_page2(self, response, main_url, foo):- yield dict(- main_url=main_url,- other_url=response.url,- foo=foo,- )--.. caution:: :attr:`.Request.cb_kwargs` was introduced in version ``1.7``.- Prior to that, using :attr:`.Request.meta` was recommended for passing- information around callbacks. After ``1.7``, :attr:`.Request.cb_kwargs`- became the preferred way for handling user information, leaving :attr:`.Request.meta`- for communication with components like middlewares and extensions.--.. _topics-request-response-ref-errbacks:--Using errbacks to catch exceptions in request processing-----------------------------------------------------------The errback of a request is a function that will be called when an exception-is raise while processing it.--It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can-be used to track connection establishment timeouts, DNS errors etc.--Here's an example spider logging all errors and catching some specific-errors if needed:--.. code-block:: python-- import scrapy-- from scrapy.spidermiddlewares.httperror import HttpError- from twisted.internet.error import DNSLookupError- from twisted.internet.error import TimeoutError, TCPTimedOutError--- class ErrbackSpider(scrapy.Spider):- name = "errback_example"- start_urls = [- "http://www.httpbin.org/", # HTTP 200 expected- "http://www.httpbin.org/status/404", # Not found error- "http://www.httpbin.org/status/500", # server issue- "http://www.httpbin.org:12345/", # non-responding host, timeout expected- "https://example.invalid/", # DNS error expected- ]-- async def start(self):- for u in self.start_urls:- yield scrapy.Request(- u,- callback=self.parse_httpbin,- errback=self.errback_httpbin,- dont_filter=True,- )-- def parse_httpbin(self, response):- self.logger.info(f"Got successful response from {response.url}")- # do something useful here...-- def errback_httpbin(self, failure):- # log all failures- self.logger.error(repr(failure))-- # in case you want to do something special for some errors,- # you may need the failure's type:-- if failure.check(HttpError):- # these exceptions come from HttpError spider middleware- # you can get the non-200 response- response = failure.value.response- self.logger.error("HttpError on %s", response.url)-- elif failure.check(DNSLookupError):- # this is the original request- request = failure.request- self.logger.error("DNSLookupError on %s", request.url)-- elif failure.check(TimeoutError, TCPTimedOutError):- request = failure.request- self.logger.error("TimeoutError on %s", request.url)---.. _errback-cb_kwargs:--Accessing additional data in errback functions-------------------------------------------------In case of a failure to process the request, you may be interested in-accessing arguments to the callback functions so you can process further-based on the arguments in the errback. The following example shows how to-achieve this by using ``Failure.request.cb_kwargs``:--.. code-block:: python-- def parse(self, response):- request = scrapy.Request(- "http://www.example.com/index.html",- callback=self.parse_page2,- errback=self.errback_page2,- cb_kwargs=dict(main_url=response.url),- )- yield request--- def parse_page2(self, response, main_url):- pass--- def errback_page2(self, failure):- yield dict(- main_url=failure.request.cb_kwargs["main_url"],- )--.. _request-fingerprints:Request fingerprints@@ -702,6 +549,319 @@ The following built-in Scrapy components have such restrictions:45-character-long keys must be supported.+.. _callbacks:++Callbacks+=========++A callback is a function that Scrapy calls with the :class:`Response` of a+:class:`~scrapy.Request` once that request has been downloaded, so that you can+extract data from that response and generate additional requests to continue+the crawl:++.. code-block:: python++ from scrapy import Request, Spider+++ class BookSpider(Spider):+ name = "books"++ async def start(self):+ yield Request("https://books.toscrape.com/", callback=self.parse_home)++ def parse_home(self, response):+ for url in response.css("h3 a::attr(href)").getall():+ yield Request(response.urljoin(url), callback=self.parse_book)++ def parse_book(self, response):+ yield {"title": response.css("h1::text").get()}++Requests may also define an :ref:`errback <errbacks>`, which Scrapy calls+instead of the callback when an exception is raised while processing the+request or its response, e.g. a connection error or, by default, a non-2xx+response.+++.. _callback-assignment:++Assigning a callback to a request+---------------------------------++To assign a callback to a request, use the ``callback`` parameter of+:class:`~scrapy.Request`, which sets the :attr:`.Request.callback` attribute:++.. code-block:: python++ from scrapy import Request+++ def parse_home(response): ...+++ request = Request("https://books.toscrape.com/", callback=parse_home)++Requests with no callback, i.e. with :attr:`~scrapy.Request.callback` set to+``None``, are handled by the :meth:`~scrapy.Spider.parse` method of the spider:++.. code-block:: python++ request = Request("https://books.toscrape.com/") # Handled by parse()++If a request is never meant to reach a spider callback, e.g. because a+:ref:`component <topics-components>` sends it and handles its response itself,+assign the special :func:`~scrapy.http.request.NO_CALLBACK` value to it+instead, so that :ref:`downloader middlewares <topics-downloader-middleware>`+can tell such requests apart.++While :attr:`~scrapy.Request.callback` only accepts callables, some spider+classes let you also define a callback by name: both :attr:`CrawlSpider.rules+<scrapy.spiders.CrawlSpider.rules>` and :attr:`SitemapSpider.sitemap_rules+<scrapy.spiders.SitemapSpider.sitemap_rules>` accept the name of a spider+method as a string.+++.. _writing-callbacks:++Writing a callback+------------------++Any callable can be a callback, as long as it takes the response as its first+positional parameter, and any :ref:`additional callback data <callback-data>`+as keyword parameters. Spider methods are the most common choice, but plain+functions, lambda expressions and other callable objects work as well.++.. note:: If you enable :ref:`job persistence <topics-jobs>` through the+ :setting:`JOBDIR` setting, callbacks must be methods of the running spider.+ Requests with any other callback cannot be serialized, so they are kept in+ memory only and lost when you pause the crawl. See+ :ref:`request-serialization`.++A callback can be:++- A regular function:++ .. code-block:: python++ def parse(self, response):+ return {"url": response.url}++- A generator function:++ .. code-block:: python++ def parse(self, response):+ yield {"url": response.url}++- A coroutine function, i.e. defined with ``async def``:++ .. code-block:: python++ async def parse(self, response):+ return {"url": response.url}++- An asynchronous generator function:++ .. code-block:: python++ async def parse(self, response):+ yield {"url": response.url}++The last two allow using ``await``, ``async for`` and ``async with`` in your+callback. See :ref:`topics-coroutines`.+++.. _callback-output:++Callback output+---------------++A callback may return or yield any of the following:++- ``None``, which does nothing.++ Callbacks that produce no output at all, e.g. callbacks that only log+ information about the response, are perfectly valid. ``None`` values within+ an iterable of callback output are ignored as well.++- A :class:`~scrapy.Request` object, which Scrapy schedules, downloads and+ eventually sends to its own callback.++- An :ref:`item object <topics-items>`, which Scrapy sends to the+ :ref:`item pipelines <topics-item-pipeline>`.++ Any object that is neither ``None`` nor a :class:`~scrapy.Request` object+ is treated as an item.++- An iterable of any of the values above, e.g. a list or, more commonly, a+ generator.++ :term:`Asynchronous iterables <asynchronous iterable>`, e.g. an+ :term:`asynchronous generator`, are also supported.++.. note:: When a callback *returns* an object, Scrapy iterates that object if+ it supports iteration, except for :class:`dict`, :class:`~scrapy.Item`,+ :class:`str` and :class:`bytes` objects, which are always handled as single+ items.++.. note:: In a generator callback, a ``return`` statement with a value does not+ produce any output, since such a value is not part of what the generator+ yields. Scrapy logs a warning when it detects such a callback, see+ :setting:`WARN_ON_GENERATOR_RETURN_VALUE`.++Before Scrapy acts on the output of a callback, that output goes through the+:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method+of your :ref:`spider middlewares <topics-spider-middleware>`, which may modify+it or drop part of it.++If a callback raises an exception, the :attr:`~scrapy.Request.errback` of the+request is *not* called. The exception goes through the+:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`+method of your spider middlewares instead and, unless one of them handles it,+Scrapy logs it and sends the :signal:`spider_error` signal.+++.. _callback-data:+.. _topics-request-response-ref-request-callback-arguments:++Passing additional data to callback functions+---------------------------------------------++In some cases you may be interested in passing data to a callback in addition+to the response, e.g. data extracted from the response that triggered the+request. The following example shows how to achieve this by using the+:attr:`.Request.cb_kwargs` attribute:++.. code-block:: python++ from scrapy import Request+++ def parse(self, response):+ request = Request(+ "http://www.example.com/index.html",+ callback=self.parse_page2,+ cb_kwargs=dict(main_url=response.url),+ )+ request.cb_kwargs["foo"] = "bar" # add more arguments for the callback+ yield request+++ def parse_page2(self, response, main_url, foo):+ yield dict(+ main_url=main_url,+ other_url=response.url,+ foo=foo,+ )++:attr:`.Request.cb_kwargs` is the recommended way to pass your own data to a+callback. Use :attr:`.Request.meta` only for data aimed at :ref:`components+<topics-components>`, such as middlewares and extensions.++.. _errbacks:+.. _topics-request-response-ref-errbacks:++Errbacks+========++The errback of a request is a function that will be called when an exception+is raise while processing it.++It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can+be used to track connection establishment timeouts, DNS errors etc.++Here's an example spider logging all errors and catching some specific+errors if needed:++.. code-block:: python++ from scrapy import Request, Spider+ from scrapy.spidermiddlewares.httperror import HttpError+ from twisted.internet.error import DNSLookupError+ from twisted.internet.error import TimeoutError, TCPTimedOutError+++ class ErrbackSpider(Spider):+ name = "errback_example"+ start_urls = [+ "http://www.httpbin.org/", # HTTP 200 expected+ "http://www.httpbin.org/status/404", # Not found error+ "http://www.httpbin.org/status/500", # server issue+ "http://www.httpbin.org:12345/", # non-responding host, timeout expected+ "https://example.invalid/", # DNS error expected+ ]++ async def start(self):+ for u in self.start_urls:+ yield Request(+ u,+ callback=self.parse_httpbin,+ errback=self.errback_httpbin,+ dont_filter=True,+ )++ def parse_httpbin(self, response):+ self.logger.info(f"Got successful response from {response.url}")+ # do something useful here...++ def errback_httpbin(self, failure):+ # log all failures+ self.logger.error(repr(failure))++ # in case you want to do something special for some errors,+ # you may need the failure's type:++ if failure.check(HttpError):+ # these exceptions come from HttpError spider middleware+ # you can get the non-200 response+ response = failure.value.response+ self.logger.error("HttpError on %s", response.url)++ elif failure.check(DNSLookupError):+ # this is the original request+ request = failure.request+ self.logger.error("DNSLookupError on %s", request.url)++ elif failure.check(TimeoutError, TCPTimedOutError):+ request = failure.request+ self.logger.error("TimeoutError on %s", request.url)+++.. _errback-cb_kwargs:++Accessing additional data in errback functions+----------------------------------------------++In case of a failure to process the request, you may be interested in+accessing arguments to the callback functions so you can process further+based on the arguments in the errback. The following example shows how to+achieve this by using ``Failure.request.cb_kwargs``:++.. code-block:: python++ from scrapy import Request+++ def parse(self, response):+ request = Request(+ "http://www.example.com/index.html",+ callback=self.parse_page2,+ errback=self.errback_page2,+ cb_kwargs=dict(main_url=response.url),+ )+ yield request+++ def parse_page2(self, response, main_url):+ pass+++ def errback_page2(self, failure):+ yield dict(+ main_url=failure.request.cb_kwargs["main_url"],+ )++.. _topics-request-meta:Request.meta special keysdiff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 69ff08fa1..8fbf0c52d 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -4,43 +4,31 @@Spiders=======-Spiders are classes which define how a certain site (or a group of sites) will be-scraped, including how to perform the crawl (i.e. follow links) and how to-extract structured data from their pages (i.e. scraping items). In other words,-Spiders are the place where you define the custom behaviour for crawling and-parsing pages for a particular site (or, in some cases, a group of sites).--For spiders, the scraping cycle goes through something like this:--1. You start by generating the initial requests to crawl the first URLs, and- specify a callback function to be called with the response downloaded from- those requests.-- The first requests to perform are obtained by iterating the- :meth:`~scrapy.Spider.start` method, which by default yields a- :class:`~scrapy.Request` object for each URL in the- :attr:`~scrapy.Spider.start_urls` spider attribute, with the- :attr:`~scrapy.Spider.parse` method set as :attr:`~scrapy.Request.callback`- function to handle each :class:`~scrapy.http.Response`.--2. In the callback function, you parse the response (web page) and return- :ref:`item objects <topics-items>`,- :class:`~scrapy.Request` objects, or an iterable of these objects.- Those Requests will also contain a callback (maybe- the same) and will then be downloaded by Scrapy and then their- response handled by the specified callback.--3. In callback functions, you parse the page contents, typically using- :ref:`topics-selectors` (but you can also use BeautifulSoup, lxml or whatever- mechanism you prefer) and generate items with the parsed data.--4. Finally, the items returned from the spider will be typically persisted to a- database (in some :ref:`Item Pipeline <topics-item-pipeline>`) or written to- a file using :ref:`topics-feed-exports`.--Even though this cycle applies (more or less) to any kind of spider, there are-different kinds of default spiders bundled into Scrapy for different purposes.-We will talk about those types here.+Spiders are classes that define how a site, or a group of sites, is scraped:+which requests to send, and how to parse their responses to extract data and to+send additional requests.++A crawl goes as follows:++1. Scrapy iterates the :meth:`~scrapy.Spider.start` method of the spider to+ get the initial requests. By default, that method yields a+ :class:`~scrapy.Request` object for each URL in+ :attr:`~scrapy.Spider.start_urls`, with :meth:`~scrapy.Spider.parse` as+ :ref:`callback <callbacks>`.++2. Scrapy downloads each request and calls its callback with the resulting+ :class:`~scrapy.http.Response`.++3. Callbacks parse the response, typically using :ref:`topics-selectors`, and+ return or yield :ref:`item objects <topics-items>` with the extracted data+ and :class:`~scrapy.Request` objects to continue the crawl, which go back+ to step 2. See :ref:`callback-output`.++4. Items go through :ref:`item pipelines <topics-item-pipeline>`, and are+ usually stored through :ref:`topics-feed-exports`.++Scrapy includes different spider classes for different purposes, described+below... _topics-spiders-ref:@@ -191,22 +179,7 @@ scrapy.Spider.. automethod:: start- .. method:: parse(response)-- This is the default callback used by Scrapy to process downloaded- responses, when their requests don't specify a callback.-- The ``parse`` method is in charge of processing the response and returning- scraped data and/or more URLs to follow. Other Requests callbacks have- the same requirements as the :class:`~scrapy.Spider` class.-- This method, as well as any other Request callback, must return a- :class:`~scrapy.Request` object, an :ref:`item object <topics-items>`, an- iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects- <topics-items>`, or ``None``.-- :param response: the response to parse- :type response: :class:`~scrapy.http.Response`+ .. automethod:: parse.. method:: closed(reason)diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.pyindex 68847283a..7d67bb6d7 100644--- a/scrapy/http/request/__init__.py+++ b/scrapy/http/request/__init__.py@@ -169,7 +169,8 @@ class Request(object_ref):#:#: The callable must expect the response as its first parameter, and#: support any additional keyword arguments set through- #: :attr:`cb_kwargs`.+ #: :attr:`cb_kwargs`. See :ref:`writing-callbacks` and+ #: :ref:`callback-output`.#:#: In addition to an arbitrary callable, the following values are also#: supported:@@ -190,8 +191,7 @@ class Request(object_ref):#: raises exceptions for non-2xx responses by default, sending them#: to the :attr:`errback` instead.#:- #: .. seealso::- #: :ref:`topics-request-response-ref-request-callback-arguments`+ #: .. seealso:: :ref:`callbacks`self.callback: CallbackT | None = callback#: :class:`~collections.abc.Callable` to handle exceptions raised@@ -200,7 +200,7 @@ class Request(object_ref):#: The callable must expect a :exc:`~twisted.python.failure.Failure` as#: its first parameter.#:- #: .. seealso:: :ref:`topics-request-response-ref-errbacks`+ #: .. seealso:: :ref:`errbacks`self.errback: Callable[[Failure], Any] | None = errbackself._cookies: CookiesT | None = cookies or Nonediff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.pyindex 02dfa2ac6..6244e3264 100644--- a/scrapy/spiders/__init__.py+++ b/scrapy/spiders/__init__.py@@ -143,6 +143,22 @@ class Spider(object_ref):else:def parse(self, response: Response, **kwargs: Any) -> Any:+ """Process *response*, i.e. extract data from it and generate new+ requests.++ This is the default :ref:`callback <callbacks>`: Scrapy uses+ it for the response to any request that does not define a+ :attr:`~scrapy.Request.callback`, such as the requests that+ :meth:`start` yields by default.++ Any :attr:`~scrapy.Request.cb_kwargs` of the request are passed as+ keyword parameters.++ Spiders must define this method, unless every request that they+ send defines a callback.++ See :ref:`callback-output` about the supported return values.+ """raise NotImplementedError(f"{self.__class__.__name__}.parse callback is not defined")