Scrapy, a fast high-level web crawling & scraping framework for Python.
Add a FAQ entry about splitting items in item pipelines
Adrián Chaves6ea62cacc71d
diff --git a/docs/faq.rst b/docs/faq.rstindex 7a0628f88..69edb0993 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -319,6 +319,29 @@ I'm scraping a XML document and my XPath selector doesn't return any itemsYou may need to remove namespaces. See :ref:`removing-namespaces`.+How to split an item into multiple items in an item pipeline?+-------------------------------------------------------------++:ref:`Item pipelines <topics-item-pipeline>` cannot yield multiple items per+input item. :ref:`Create a spider middleware <custom-spider-middleware>`+instead, and use its+:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`+method for this puspose. For example::++ from copy import deepcopy++ from scrapy.item import BaseItem+++ class MultiplyItemsMiddleware:++ def process_spider_output(self, response, result, spider):+ for item in result:+ if isinstance(item, (BaseItem, dict)):+ for _ in range(item['multiply_by']):+ yield deepcopy(item)++.. _user agents: https://en.wikipedia.org/wiki/User_agent.. _LIFO: https://en.wikipedia.org/wiki/Stack_(abstract_data_type).. _DFO order: https://en.wikipedia.org/wiki/Depth-first_searchdiff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 30b8638bd..0e8210130 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -54,6 +54,8 @@ value. For example, if you want to disable the off-site middleware::Finally, keep in mind that some middlewares may need to be enabled through aparticular setting. See each middleware documentation for more info.+.. _custom-spider-middleware:+Writing your own spider middleware==================================