Scrapy, a fast high-level web crawling & scraping framework for Python.
Include signal example
Darshan Chaudharya2e64525544c
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex 4ee4f1758..5ed6ce97d 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -7,7 +7,7 @@ Downloading and processing files and images.. currentmodule:: scrapy.pipelines.imagesScrapy provides reusable :doc:`item pipelines </topics/item-pipeline>` for-downloading fies attached to a particular item (for example, when you scrape+downloading files attached to a particular item (for example, when you scrapeproducts and also want to download their images locally). These pipelines sharea bit of functionality and structure (we refer to them as media pipelines), buttypically you'll either use the Files Pipeline or the Images Pipeline.diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 5dd3b9ef5..19d5e8df9 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -16,6 +16,37 @@ deliver the arguments that the handler receives.You can connect to signals (or send your own) through the:ref:`topics-api-signals`.+Here is a simple example showing how you can catch signals and perform some action:+::++ from scrapy import signals+ from scrapy import Spider+++ class DmozSpider(Spider):+ name = "dmoz"+ allowed_domains = ["dmoz.org"]+ start_urls = [+ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",+ "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",+ ]+++ @classmethod+ def from_crawler(cls, crawler, *args, **kwargs):+ spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)+ crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)+ return spider+++ def spider_closed(self, spider):+ spider.logger.info('Spider closed: %s', spider.name)+++ def parse(self, response):+ pass++Deferred signal handlers========================diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.pyindex 303a413d8..f20070b5d 100644--- a/scrapy/utils/misc.py+++ b/scrapy/utils/misc.py@@ -1,4 +1,4 @@-"""Helper functions which doesn't fit anywhere else"""+"""Helper functions which don't fit anywhere else"""import reimport hashlibfrom importlib import import_module