Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add Jobdir documentation (#5260)

Martin Schimandl66fe5de1395d
diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst
index 50bcaa6d6..2a976e91d 100644
--- a/docs/topics/jobs.rst
+++ b/docs/topics/jobs.rst
@@ -17,15 +17,21 @@ facilities:
* an extension that keeps some spider state (key/value pairs) persistent
between batches
+.. _job-dir:
+
Job directory
=============
-To enable persistence support you just need to define a *job directory* through
-the ``JOBDIR`` setting. This directory will be for storing all required data to
-keep the state of a single job (i.e. a spider run). It's important to note that
-this directory must not be shared by different spiders, or even different
-jobs/runs of the same spider, as it's meant to be used for storing the state of
-a *single* job.
+To enable persistence support, define a *job directory* through the
+:setting:`JOBDIR` setting.
+
+The job directory will store all required data to keep the state of a *single*
+job (i.e. a spider run), so that if stopped cleanly, it can be resumed later.
+
+.. warning:: This directory must *not* be shared by different spiders, or even
+ different jobs of the same spider.
+
+See also :ref:`job-dir-contents`.
How to use it
=============
@@ -65,6 +71,14 @@ Persistence gotchas
There are a few things to keep in mind if you want to be able to use the Scrapy
persistence support:
+Pause limitations
+-----------------
+
+Job pausing and resuming is only supported when the spider is paused by
+stopping it cleanly. Forced, sudden or otherwise unclean shutdown can lead to
+data corruption in the job directory, which may prevent the spider from
+resuming correctly.
+
Cookies expiration
------------------
@@ -72,7 +86,6 @@ Cookies may expire. So, if you don't resume your spider quickly the requests
scheduled may no longer work. This won't be an issue if your spider doesn't rely
on cookies.
-
.. _request-serialization:
Request serialization
@@ -86,3 +99,52 @@ running :class:`~scrapy.Spider` class.
If you wish to log the requests that couldn't be serialized, you can set the
:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.
It is ``False`` by default.
+
+.. _job-dir-contents:
+
+Job directory contents
+======================
+
+The contents of a job directory depend on the components used during the job.
+Components known to write in the job directory include the :ref:`scheduler
+<topics-scheduler>` and the :class:`~scrapy.extensions.spiderstate.SpiderState`
+extension. See the reference documentation of the corresponding components for
+details.
+
+For example, with default settings, the job directory may look like this:
+
+.. code-block:: none
+
+ ├── requests.queue
+ | ├── active.json
+ | └── {hostname}-{hash}
+ | └── {priority}{s?}
+ | ├── q{00000}
+ | └── info.json
+ ├── requests.seen
+ └── spider.state
+
+Where:
+
+- :class:`~scrapy.core.scheduler.Scheduler` creates the ``requests.queue/``
+ directory and the ``active.json`` file, the latter containing the state
+ data returned by :meth:`DownloaderAwarePriorityQueue.close()
+ <scrapy.pqueues.DownloaderAwarePriorityQueue.close>` the last time the job
+ was paused.
+
+- :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` creates the
+ ``{hostname}-{hash}`` directories.
+
+- :class:`~scrapy.pqueues.ScrapyPriorityQueue` creates the ``{priority}{s?}``
+ directories.
+
+- :class:`scrapy.squeues.PickleLifoDiskQueue`, a subclass of
+ :class:`queuelib.LifoDiskQueue` that uses :mod:`pickle` to serialize
+ :class:`dict` representations of :class:`scrapy.Request` objects, creates
+ the ``info.json`` and ``q{00000}`` files.
+
+- :class:`~scrapy.dupefilters.RFPDupeFilter` creates the ``requests.seen``
+ file.
+
+- :class:`~scrapy.extensions.spiderstate.SpiderState` creates the
+ ``spider.state`` file.
diff --git a/docs/topics/scheduler.rst b/docs/topics/scheduler.rst
index b6e54ebd7..b79d6de1f 100644
--- a/docs/topics/scheduler.rst
+++ b/docs/topics/scheduler.rst
@@ -32,3 +32,10 @@ Default scheduler
.. autoclass:: Scheduler()
:members:
:special-members: __init__, __len__
+
+
+Priority queues
+===============
+
+.. autoclass:: scrapy.pqueues.DownloaderAwarePriorityQueue
+.. autoclass:: scrapy.pqueues.ScrapyPriorityQueue
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 5357fa72b..5b4a9b5f9 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1702,10 +1702,10 @@ the user agent to use in the robots.txt file.
SCHEDULER
---------
-Default: ``'scrapy.core.scheduler.Scheduler'``
+Default: :class:`~scrapy.core.scheduler.Scheduler`
-The scheduler class to be used for crawling.
-See the :ref:`topics-scheduler` topic for details.
+The scheduler class to be used for crawling. See :ref:`topics-scheduler` for
+details.
.. setting:: SCHEDULER_DEBUG
@@ -1755,12 +1755,14 @@ Type of in-memory queue used by the scheduler. Other available type is:
SCHEDULER_PRIORITY_QUEUE
------------------------
-Default: ``'scrapy.pqueues.DownloaderAwarePriorityQueue'``
+Default: :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`
-Type of priority queue used by the scheduler. Another available type is
-``scrapy.pqueues.ScrapyPriorityQueue``.
-``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than
-``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different
+Type of priority queue used by the scheduler.
+
+Another available type is :class:`~scrapy.pqueues.ScrapyPriorityQueue`.
+
+:class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` works better than
+:class:`~scrapy.pqueues.ScrapyPriorityQueue` when you crawl many different
domains in parallel.
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
index 491e7a8a6..5a0aa2197 100644
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -4,7 +4,7 @@ import json
import logging
from abc import abstractmethod
from pathlib import Path
-from typing import TYPE_CHECKING, Any, cast
+from typing import TYPE_CHECKING, Any
from warnings import warn
# working around https://github.com/sphinx-doc/sphinx/issues/10400
@@ -128,7 +128,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
class Scheduler(BaseScheduler):
- """Default scheduler.
+ r"""Default scheduler.
Requests are stored into priority queues
(:setting:`SCHEDULER_PRIORITY_QUEUE`) that sort requests by
@@ -190,7 +190,8 @@ class Scheduler(BaseScheduler):
following :ref:`settings <topics-settings>`:
| :setting:`DEPTH_PRIORITY` = ``1``
- | :setting:`SCHEDULER_DISK_QUEUE` = ``"scrapy.squeues.PickleFifoDiskQueue"``
+ | :setting:`SCHEDULER_DISK_QUEUE` =
+ ``"scrapy.squeues.PickleFifoDiskQueue"``
| :setting:`SCHEDULER_MEMORY_QUEUE` = ``"scrapy.squeues.FifoMemoryQueue"``
.. _BFO order: https://en.wikipedia.org/wiki/Breadth-first_search
@@ -219,6 +220,37 @@ class Scheduler(BaseScheduler):
order. Lowering those settings to ``1`` enforces the desired order except
for the very first request, but it significantly slows down the crawl as a
whole.
+
+ Job directory contents
+ ======================
+
+ .. warning:: The files that this class generates in the :ref:`job directory
+ <job-dir>` are an implementation detail, and may change without a
+ warning in a future version of Scrapy. Do not rely on the following
+ information for anything other than debugging purposes.
+
+ When using :setting:`JOBDIR`, this scheduler class:
+
+ - Creates a directory named ``requests.queue`` inside the :ref:`job
+ directory <job-dir>`, meant to keep track of all requests stored in
+ the scheduler (i.e. not downloaded yet).
+
+ - Generates inside that directory an ``active.json`` file with a JSON
+ representation of the state (``startprios``) of
+ :setting:`SCHEDULER_PRIORITY_QUEUE`.
+
+ The file is generated whenever the job stops (cleanly) and is loaded
+ when resuming the job.
+
+ - Instantiates the configured :setting:`SCHEDULER_PRIORITY_QUEUE` with
+ ``requests.queue/`` as persistence directory (*key*) and
+ :setting:`SCHEDULER_DISK_QUEUE` as *downstream_queue_cls*. The priority
+ queue may create additional files and directories inside that
+ directory, directly or though instances of
+ :setting:`SCHEDULER_DISK_QUEUE`.
+
+ This scheduler class also uses the configured :setting:`DUPEFILTER_CLASS`,
+ which may also write data inside the job directory.
"""
@classmethod
@@ -486,13 +518,13 @@ class Scheduler(BaseScheduler):
return str(dqdir)
return None
- def _read_dqs_state(self, dqdir: str) -> list[int]:
+ def _read_dqs_state(self, dqdir: str) -> Any:
path = Path(dqdir, "active.json")
if not path.exists():
return []
with path.open(encoding="utf-8") as f:
- return cast("list[int]", json.load(f))
+ return json.load(f)
- def _write_dqs_state(self, dqdir: str, state: list[int]) -> None:
+ def _write_dqs_state(self, dqdir: str, state: Any) -> None:
with Path(dqdir, "active.json").open("w", encoding="utf-8") as f:
json.dump(state, f)
diff --git a/scrapy/dupefilters.py b/scrapy/dupefilters.py
index f0a6988c8..36fb0f97d 100644
--- a/scrapy/dupefilters.py
+++ b/scrapy/dupefilters.py
@@ -55,6 +55,18 @@ class RFPDupeFilter(BaseDupeFilter):
filters out requests with the canonical
(:func:`w3lib.url.canonicalize_url`) :attr:`~scrapy.http.Request.url`,
:attr:`~scrapy.http.Request.method` and :attr:`~scrapy.http.Request.body`.
+
+ Job directory contents
+ ======================
+
+ .. warning:: The files that this class generates in the :ref:`job directory
+ <job-dir>` are an implementation detail, and may change without a
+ warning in a future version of Scrapy. Do not rely on the following
+ information for anything other than debugging purposes.
+
+ When using :setting:`JOBDIR`, seen fingerprints are tracked in a file named
+ ``requests.seen`` in the :ref:`job directory <job-dir>`, which contains 1
+ request fingerprint per line.
"""
def __init__(
diff --git a/scrapy/pqueues.py b/scrapy/pqueues.py
index 42c53a527..ed57091ba 100644
--- a/scrapy/pqueues.py
+++ b/scrapy/pqueues.py
@@ -51,16 +51,17 @@ class QueueProtocol(Protocol):
class ScrapyPriorityQueue:
"""A priority queue implemented using multiple internal queues (typically,
- FIFO queues). It uses one internal queue for each priority value. The internal
- queue must implement the following methods:
+ FIFO queues). It uses one internal queue for each priority value. The
+ internal queue must implement the following methods:
* push(obj)
* pop()
* close()
* __len__()
- Optionally, the queue could provide a ``peek`` method, that should return the
- next object to be returned by ``pop``, but without removing it from the queue.
+ Optionally, the queue could provide a ``peek`` method, that should return
+ the next object to be returned by ``pop``, but without removing it from the
+ queue.
``__init__`` method of ScrapyPriorityQueue receives a downstream_queue_cls
argument, which is a class used to instantiate a new (internal) queue when
@@ -72,6 +73,28 @@ class ScrapyPriorityQueue:
startprios is a sequence of priorities to start with. If the queue was
previously closed leaving some priority buckets non-empty, those priorities
should be passed in startprios.
+
+ Disk persistence
+ ================
+
+ .. warning:: The files that this class generates on disk are an
+ implementation detail, and may change without a warning in a future
+ version of Scrapy. Do not rely on the following information for
+ anything other than debugging purposes.
+
+ When a component instantiates this class with a non-empty *key* argument,
+ *key* is used as a persistence directory.
+
+ For every request enqueued, this class checks:
+
+ - Whether the request is a :ref:`start request <start-requests>` or not.
+
+ - The :data:`~scrapy.Request.priority` of the request.
+
+ For each combination of the above seen, this class creates an instance of
+ *downstream_queue_cls* with *key* set to a subdirectory of the persistence
+ directory, named as the request priority (e.g. ``1``), with an ``s`` suffix
+ in case of a start request (e.g. ``1s``).
"""
@classmethod
@@ -255,6 +278,26 @@ class DownloaderAwarePriorityQueue:
"""PriorityQueue which takes Downloader activity into account:
domains (slots) with the least amount of active downloads are dequeued
first.
+
+ Disk persistence
+ ================
+
+ .. warning:: The files that this class generates on disk are an
+ implementation detail, and may change without a warning in a future
+ version of Scrapy. Do not rely on the following information for
+ anything other than debugging purposes.
+
+ When a component instantiates this class with a non-empty *key* argument,
+ *key* is used as a persistence directory, and inside that directory this
+ class creates a subdirectory per download slot (domain).
+
+ Those subdirectories are named after the corresponding download slot, with
+ path-unsafe characters replaced by underscores and an MD5 hash suffix to
+ avoid collisions.
+
+ For each download slot, this class creates an instance of
+ :class:`ScrapyPriorityQueue` with the download slot subdirectory as *key*
+ and its own *downstream_queue_cls*.
"""
@classmethod