Scrapy, a fast high-level web crawling & scraping framework for Python.
Documentation added for Spider State in extensions.rst (#6522)
Rohit Kumar Singh7701e590fbc5
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex 7b34a19d5..9cbc9663d 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -243,6 +243,32 @@ An extension for debugging memory usage. It collects information about:To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. Theinfo will be stored in the stats.+.. _topics-extensions-ref-spiderstate:++Spider state extension+~~~~~~~~~~~~~~~~~~~~~~++.. module:: scrapy.extensions.spiderstate+ :synopsis: Spider state extension++.. class:: SpiderState++Manages spider state data by loading it before a crawl and saving it after.++Give a value to the :setting:`JOBDIR` setting to enable this extension.+When enabled, this extension manages the :attr:`~scrapy.Spider.state`+attribute of your :class:`~scrapy.Spider` instance:++- When your spider closes (:signal:`spider_closed`), the contents of its+ :attr:`~scrapy.Spider.state` attribute are serialized into a file named+ ``spider.state`` in the :setting:`JOBDIR` folder.+- When your spider opens (:signal:`spider_opened`), if a previously-generated+ ``spider.state`` file exists in the :setting:`JOBDIR` folder, it is loaded+ into the :attr:`~scrapy.Spider.state` attribute.+++For an example, see :ref:`topics-keeping-persistent-state-between-batches`.+Close spider extension~~~~~~~~~~~~~~~~~~~~~~diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rstindex c7fc1ea48..0e705dc64 100644--- a/docs/topics/jobs.rst+++ b/docs/topics/jobs.rst@@ -46,7 +46,7 @@ Keeping persistent state between batchesSometimes you'll want to keep some persistent spider state between pause/resumebatches. You can use the ``spider.state`` attribute for that, which should be a-dict. There's a built-in extension that takes care of serializing, storing and+dict. There's :ref:`a built-in extension <topics-extensions-ref-spiderstate>` that takes care of serializing, storing andloading that attribute from the job directory, when the spider starts andstops.