Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Documentation added for Spider State in extensions.rst (#6522)

Rohit Kumar Singh7701e590fbc5
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 7b34a19d5..9cbc9663d 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -243,6 +243,32 @@ An extension for debugging memory usage. It collects information about:
To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The
info will be stored in the stats.
+.. _topics-extensions-ref-spiderstate:
+
+Spider state extension
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. module:: scrapy.extensions.spiderstate
+ :synopsis: Spider state extension
+
+.. class:: SpiderState
+
+Manages spider state data by loading it before a crawl and saving it after.
+
+Give a value to the :setting:`JOBDIR` setting to enable this extension.
+When enabled, this extension manages the :attr:`~scrapy.Spider.state`
+attribute of your :class:`~scrapy.Spider` instance:
+
+- When your spider closes (:signal:`spider_closed`), the contents of its
+ :attr:`~scrapy.Spider.state` attribute are serialized into a file named
+ ``spider.state`` in the :setting:`JOBDIR` folder.
+- When your spider opens (:signal:`spider_opened`), if a previously-generated
+ ``spider.state`` file exists in the :setting:`JOBDIR` folder, it is loaded
+ into the :attr:`~scrapy.Spider.state` attribute.
+
+
+For an example, see :ref:`topics-keeping-persistent-state-between-batches`.
+
Close spider extension
~~~~~~~~~~~~~~~~~~~~~~
diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst
index c7fc1ea48..0e705dc64 100644
--- a/docs/topics/jobs.rst
+++ b/docs/topics/jobs.rst
@@ -46,7 +46,7 @@ Keeping persistent state between batches
Sometimes you'll want to keep some persistent spider state between pause/resume
batches. You can use the ``spider.state`` attribute for that, which should be a
-dict. There's a built-in extension that takes care of serializing, storing and
+dict. There's :ref:`a built-in extension <topics-extensions-ref-spiderstate>` that takes care of serializing, storing and
loading that attribute from the job directory, when the spider starts and
stops.