Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document crawl exceptions and bootstrap_failed for crawler runners (#8340)

Adrian6305c804abfd
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index ffbbe2d97..0503cefab 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -108,6 +108,9 @@ use :func:`twisted.internet.task.react` to start and stop the reactor, though
it may be easier to just use :class:`~scrapy.crawler.AsyncCrawlerProcess` or
:class:`~scrapy.crawler.CrawlerProcess` instead.
+Exceptions that interrupt a crawl reach the code that awaits that task or that
+adds callbacks to that Deferred, which is where you should handle them.
+
Here's an example of using :class:`~scrapy.crawler.AsyncCrawlerRunner` together
with simple reactor management code:
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index f57143a0b..a5f7467f4 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -505,6 +505,8 @@ class CrawlerRunnerBase(ABC):
self.spider_loader: SpiderLoaderProtocol = get_spider_loader(settings)
self._crawlers: set[Crawler] = set()
self.bootstrap_failed = False
+ """``True`` if any crawl started by :meth:`crawl` was interrupted by an
+ exception, including a failure to create the spider object."""
@property
def crawlers(self) -> set[Crawler]: