Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

[docs] document that process_item can return Deferred

Pawel Miechb2bfd1e5c536
diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst
index b9b4c2058..a31fe7423 100644
--- a/docs/topics/item-pipeline.rst
+++ b/docs/topics/item-pipeline.rst
@@ -27,9 +27,10 @@ Each item pipeline component is a Python class that must implement the following
.. method:: process_item(self, item, spider)
- This method is called for every item pipeline component and must either return
- a dict with data, :class:`~scrapy.item.Item` (or any descendant class) object
- or raise a :exc:`~scrapy.exceptions.DropItem` exception. Dropped items are no longer
+ This method is called for every item pipeline component. :meth:`process_item`
+ must either: return a dict with data, return an :class:`~scrapy.item.Item`
+ (or any descendant class) object, return a `Twisted Deferred`_ or raise
+ :exc:`~scrapy.exceptions.DropItem` exception. Dropped items are no longer
processed by further pipeline components.
:param item: the item scraped
@@ -66,6 +67,8 @@ Additionally, they may also implement the following methods:
:type crawler: :class:`~scrapy.crawler.Crawler` object
+.. _Twisted Deferred: https://twistedmatrix.com/documents/current/core/howto/defer.html
+
Item pipeline example
=====================
@@ -163,6 +166,52 @@ method and how to clean up the resources properly.
.. _MongoDB: https://www.mongodb.org/
.. _pymongo: https://api.mongodb.org/python/current/
+
+Take screenshot of item
+-----------------------
+
+This example demonstrates how to return Deferred_ from :meth:`process_item` method.
+It uses Splash_ to render screenshot of item url. Pipeline
+makes request to locally running instance of Splash_. After request is downloaded
+and Deferred callback fires, it saves item to a file and adds filename to an item.
+
+::
+
+ import scrapy
+
+
+ class ScreenshotPipeline(object):
+ """Pipeline that uses Splash to render screenshot of
+ every Scrapy item."""
+
+ SPLASH_URL = "http://localhost:8050/render.png?url={}"
+
+ def process_item(self, item, spider):
+ item_url = item["url"]
+ screenshot_url = self.SPLASH_URL.format(item_url)
+ request = scrapy.Request(screenshot_url)
+ dfd = spider.crawler.engine.download(request, spider)
+ dfd.addBoth(self.return_item, item)
+ return dfd
+
+ def return_item(self, response, item):
+ if response.status != 200:
+ # Error happened, return item.
+ return item
+
+ # Save screenshot to file.
+ filename = "item_file_name.png"
+ with open(filename, "wb") as f:
+ f.write(response.body)
+
+ # Store filename in item.
+ item["screenshot_filename"] = filename
+ return item
+
+
+.. _Splash: http://splash.readthedocs.io/en/stable/
+.. _Deferred: https://twistedmatrix.com/documents/current/core/howto/defer.html
+
Duplicates filter
-----------------