Scrapy, a fast high-level web crawling & scraping framework for Python.
Document crawl exceptions and bootstrap_failed for crawler runners (#8340)
Adrian6305c804abfd
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex ffbbe2d97..0503cefab 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -108,6 +108,9 @@ use :func:`twisted.internet.task.react` to start and stop the reactor, thoughit may be easier to just use :class:`~scrapy.crawler.AsyncCrawlerProcess` or:class:`~scrapy.crawler.CrawlerProcess` instead.+Exceptions that interrupt a crawl reach the code that awaits that task or that+adds callbacks to that Deferred, which is where you should handle them.+Here's an example of using :class:`~scrapy.crawler.AsyncCrawlerRunner` togetherwith simple reactor management code:diff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex f57143a0b..a5f7467f4 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -505,6 +505,8 @@ class CrawlerRunnerBase(ABC):self.spider_loader: SpiderLoaderProtocol = get_spider_loader(settings)self._crawlers: set[Crawler] = set()self.bootstrap_failed = False+ """``True`` if any crawl started by :meth:`crawl` was interrupted by an+ exception, including a failure to create the spider object."""@propertydef crawlers(self) -> set[Crawler]: