Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: Remove top-level reactor imports from CrawlerProces/CrawlerRunner examples
Laerte Pereirae676cd3ce0d4
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex cd359b147..7731180fe 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -92,7 +92,6 @@ reactor after ``MySpider`` has finished running... code-block:: python- from twisted.internet import reactorimport scrapyfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.log import configure_logging@@ -107,6 +106,33 @@ reactor after ``MySpider`` has finished running.runner = CrawlerRunner()d = runner.crawl(MySpider)+ from twisted.internet import reactor++ d.addBoth(lambda _: reactor.stop())+ reactor.run() # the script will block here until the crawling is finished++Same example but using a non-default reactor, is only necessary call ``install_reactor`` if you are using ``CrawlerRunner`` since ``CrawlerProcess`` already does this automatically.++.. code-block:: python++ import scrapy+ from scrapy.crawler import CrawlerRunner+ from scrapy.utils.log import configure_logging+++ class MySpider(scrapy.Spider):+ # Your spider definition+ ...+++ configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})+ from scrapy.utils.reactor import install_reactor++ install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")+ runner = CrawlerRunner()+ d = runner.crawl(MySpider)+ from twisted.internet import reactor+d.addBoth(lambda _: reactor.stop())reactor.run() # the script will block here until the crawling is finished@@ -151,7 +177,6 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:.. code-block:: pythonimport scrapy- from twisted.internet import reactorfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.log import configure_loggingfrom scrapy.utils.project import get_project_settings@@ -173,6 +198,8 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:runner.crawl(MySpider1)runner.crawl(MySpider2)d = runner.join()+ from twisted.internet import reactor+d.addBoth(lambda _: reactor.stop())reactor.run() # the script will block here until all crawling jobs are finished@@ -181,7 +208,7 @@ Same example but running the spiders sequentially by chaining the deferreds:.. code-block:: python- from twisted.internet import reactor, defer+ from twisted.internet import deferfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.log import configure_loggingfrom scrapy.utils.project import get_project_settings@@ -209,6 +236,8 @@ Same example but running the spiders sequentially by chaining the deferreds:reactor.stop()+ from twisted.internet import reactor+crawl()reactor.run() # the script will block here until the last crawl call is finished