Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add a stats reference (#7814)

Adrian6cefaa5434da
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 311df7052..c27348c7a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -27,6 +27,6 @@ repos:
hooks:
- id: sphinx-lint
- repo: https://github.com/scrapy/sphinx-scrapy
- rev: 0.8.8
+ rev: 0.8.9
hooks:
- id: sphinx-scrapy
diff --git a/docs/requirements.in b/docs/requirements.in
index a1f3a7468..257365380 100644
--- a/docs/requirements.in
+++ b/docs/requirements.in
@@ -5,4 +5,4 @@ sphinx
sphinx-notfound-page
sphinx-rtd-theme
sphinx-rtd-dark-mode
-sphinx-scrapy @ git+https://github.com/scrapy/[email protected]
+sphinx-scrapy @ git+https://github.com/scrapy/[email protected]
diff --git a/docs/requirements.txt b/docs/requirements.txt
index a5cbad302..87634cea9 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -153,7 +153,7 @@ sphinx-rtd-theme==3.1.0
# via
# -r docs/requirements.in
# sphinx-rtd-dark-mode
-sphinx-scrapy @ git+https://github.com/scrapy/sphinx-scrapy.git@c0b2ac815afc3cb8857d575cecb5d55c05e6b737
+sphinx-scrapy @ git+https://github.com/scrapy/sphinx-scrapy.git@912ed0507405e16ac60a47dd08195a1cd0ced984
# via -r docs/requirements.in
sphinx-sitemap==2.9.0
# via sphinx-scrapy
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 5598ab983..78b38cc3f 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -136,18 +136,10 @@ Core Stats extension
Enable the collection of core statistics, provided the stats collection is
enabled (see :ref:`topics-stats`).
-The following stats are collected:
-
-* ``start_time``: start date/time of the crawl (:class:`~datetime.datetime`).
-* ``finish_time``: end date/time of the crawl (:class:`~datetime.datetime`).
-* ``elapsed_time_seconds``: total crawl duration in seconds (:class:`float`).
-* ``finish_reason``: the closing reason string (e.g. ``"finished"``,
- ``"closespider_timeout"``).
-* ``item_scraped_count``: total number of items that passed all pipelines.
-* ``item_dropped_count``: total number of items dropped by a pipeline.
-* ``item_dropped_reasons_count/<ExceptionName>``: per-exception drop count
- (e.g. ``item_dropped_reasons_count/DropItem``).
-* ``response_received_count``: total number of HTTP responses received.
+The following stats are collected: :stat:`elapsed_time_seconds`,
+:stat:`finish_reason`, :stat:`finish_time`, :stat:`item_dropped_count`,
+:stat:`item_dropped_reasons_count/{exception}`, :stat:`item_scraped_count`,
+:stat:`response_received_count`, :stat:`start_time`.
Log Count extension
~~~~~~~~~~~~~~~~~~~
@@ -190,7 +182,7 @@ Monitors the memory used by the Scrapy process that runs the spider and:
1. sends a :signal:`memusage_warning_reached` signal when it exceeds
:setting:`MEMUSAGE_WARNING_MB`
-2. closes the spider with the `"memusage_exceeded"` reason when it exceeds
+2. closes the spider with the ``"memusage_exceeded"`` reason when it exceeds
:setting:`MEMUSAGE_LIMIT_MB`
This extension is enabled by the :setting:`MEMUSAGE_ENABLED` setting and
@@ -214,7 +206,8 @@ An extension for debugging memory usage. It collects information about:
* objects left alive that shouldn't. For more info, see :ref:`topics-leaks-trackrefs`
To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The
-info will be stored in the stats.
+info will be stored in the :stat:`memdebug/gc_garbage_count` and
+:stat:`memdebug/live_refs/{cls}` stats.
.. _topics-extensions-ref-spiderstate:
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 81055afc2..b07dff180 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1871,7 +1871,8 @@ Default: ``False``
Setting to ``True`` will log debug information about the requests scheduler.
This currently logs (only once) if the requests cannot be serialized to disk.
-Stats counter (``scheduler/unserializable``) tracks the number of times this happens.
+The :stat:`scheduler/unserializable` stat tracks the number of times this
+happens.
Example entry in logs::
diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst
index 0cf4a72cc..c702cefe7 100644
--- a/docs/topics/stats.rst
+++ b/docs/topics/stats.rst
@@ -21,6 +21,8 @@ using the Stats Collector from.
Another feature of the Stats Collector is that it's very efficient (when
enabled) and extremely efficient (almost unnoticeable) when disabled.
+See :ref:`topics-stats-reference` below for the stats that Scrapy sets.
+
.. _topics-stats-usecases:
Common Stats Collector uses
@@ -101,3 +103,642 @@ DummyStatsCollector
-------------------
.. autoclass:: DummyStatsCollector
+
+.. _topics-stats-reference:
+
+Built-in stats reference
+========================
+
+Scrapy sets the following :ref:`stats <topics-stats>`. Components other than
+those built into Scrapy may set additional stats; see their documentation.
+
+Stat keys that contain a ``{placeholder}`` below stand for a family of stats,
+one per actual value of the placeholder.
+
+.. note:: Most stats are set by a specific :ref:`component
+ <topics-components>`, and are only present if that component is enabled and
+ its code path is reached. A stat that is missing from
+ :meth:`~scrapy.statscollectors.StatsCollector.get_stats` output is
+ equivalent to a counter of 0.
+
+.. stat:: downloader/exception_count
+
+``downloader/exception_count``
+ Number of exceptions raised while downloading requests.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/exception_type_count/{exception_type}
+
+``downloader/exception_type_count/{exception_type}``
+ Number of exceptions raised while downloading requests, per exception type,
+ where ``{exception_type}`` is the import path of the exception class, e.g.
+ ``twisted.internet.error.DNSLookupError``.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/request_bytes
+
+``downloader/request_bytes``
+ Total size, in bytes, of the requests sent, counting the request line, the
+ headers and the body. As with :stat:`downloader/request_count`, requests
+ served from the cache are also counted.
+
+ It is an approximation, reconstructed from each :class:`~scrapy.Request`
+ object instead of measured on the wire, so it does not account for the
+ actual bytes that the :ref:`download handler
+ <topics-download-handlers>` sends, e.g. transport-level overhead.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/request_count
+
+``downloader/request_count``
+ Number of requests sent.
+
+ Requests that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`
+ serves from the cache are also counted, even though they are never sent,
+ because it handles requests after
+ :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/request_method_count/{method}
+
+``downloader/request_method_count/{method}``
+ Number of requests sent, per HTTP method, e.g. ``GET`` or ``POST``. As with
+ :stat:`downloader/request_count`, requests served from the cache are also
+ counted.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/response_bytes
+
+``downloader/response_bytes``
+ Total size, in bytes, of the responses received, counting the status line,
+ the headers and the body. It covers the same responses as
+ :stat:`downloader/response_count`.
+
+ The body is counted as received, i.e. still compressed for responses that
+ used ``Content-Encoding``, because
+ :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` handles
+ responses before
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
+ decompresses them. See :stat:`httpcompression/response_bytes` for
+ decompressed sizes.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/response_count
+
+``downloader/response_count``
+ Number of responses received.
+
+ It counts responses that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`
+ serves from the cache, even though they do not come from the network, and
+ responses that a downloader middleware consumes before they reach your
+ spider, e.g. redirect responses that :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware`
+ turns into new requests. Compare with :stat:`response_received_count`.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: downloader/response_status_count/{status_code}
+
+``downloader/response_status_count/{status_code}``
+ Number of responses received, per HTTP status code, e.g. ``200`` or
+ ``404``. It covers the same responses as :stat:`downloader/response_count`.
+
+ Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
+
+.. stat:: dupefilter/filtered
+
+``dupefilter/filtered``
+ Number of requests dropped as duplicates.
+
+ Set by :class:`~scrapy.dupefilters.RFPDupeFilter`.
+
+.. stat:: elapsed_time_seconds
+
+``elapsed_time_seconds``
+ Time, as a :class:`float`, in seconds, between the :signal:`spider_opened`
+ and the :signal:`spider_closed` signals.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: feedexport/failed_count/{storage}
+
+``feedexport/failed_count/{storage}``
+ Number of :ref:`feeds <topics-feed-exports>` that could not be stored, per
+ :ref:`storage backend <topics-feed-storage-backends>`, where ``{storage}``
+ is the class name of the storage backend, e.g. ``FileFeedStorage``.
+
+.. stat:: feedexport/success_count/{storage}
+
+``feedexport/success_count/{storage}``
+ Number of :ref:`feeds <topics-feed-exports>` stored successfully, per
+ :ref:`storage backend <topics-feed-storage-backends>`, where ``{storage}``
+ is the class name of the storage backend, e.g. ``FileFeedStorage``.
+
+.. stat:: file_count
+
+``file_count``
+ Number of files handled by the :ref:`media pipelines
+ <topics-media-pipeline>`.
+
+.. stat:: file_status_count/{status}
+
+``file_status_count/{status}``
+ Number of files handled by the :ref:`media pipelines
+ <topics-media-pipeline>`, per status, where ``{status}`` is one of:
+
+ - ``downloaded``: the file was downloaded.
+
+ - ``cached``: the file came from the
+ :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`
+ cache.
+
+ - ``uptodate``: the file was already in the storage backend and had not
+ :ref:`expired <file-expiration>`, so it was not downloaded again.
+
+.. stat:: finish_reason
+
+``finish_reason``
+ String indicating why the crawl finished. It matches the *reason* argument
+ of the :signal:`spider_closed` signal.
+
+ Scrapy uses the following reasons:
+
+ - ``cancelled``: the spider was closed without a more specific reason,
+ e.g. because :exc:`~scrapy.exceptions.CloseSpider` was raised without
+ one.
+
+ - ``closespider_errorcount``: see :setting:`CLOSESPIDER_ERRORCOUNT`.
+
+ - ``closespider_itemcount``: see :setting:`CLOSESPIDER_ITEMCOUNT`.
+
+ - ``closespider_pagecount``: see :setting:`CLOSESPIDER_PAGECOUNT`.
+
+ - ``closespider_pagecount_no_item``: see
+ :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM`.
+
+ - ``closespider_timeout``: see :setting:`CLOSESPIDER_TIMEOUT`.
+
+ - ``closespider_timeout_no_item``: see
+ :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM`.
+
+ - ``finished``: the spider became idle with no pending requests, i.e. it
+ finished normally.
+
+ - ``memusage_exceeded``: see :setting:`MEMUSAGE_LIMIT_MB`.
+
+ - ``shutdown``: the crawl was interrupted, e.g. by a system signal such
+ as ``SIGINT`` (:kbd:`Ctrl-C`).
+
+ Third-party components and your own code may use any other reason, e.g. by
+ raising :exc:`~scrapy.exceptions.CloseSpider` with it.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: finish_time
+
+``finish_time``
+ Timezone-aware :class:`~datetime.datetime` object, in UTC, indicating when
+ the :signal:`spider_closed` signal was sent.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: httpcache/errorrecovery
+
+``httpcache/errorrecovery``
+ Number of times that a stale cached response was used because downloading a
+ fresh response raised an exception.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/firsthand
+
+``httpcache/firsthand``
+ Number of responses that were downloaded without a matching cache entry to
+ validate against, i.e. responses for requests counted in
+ :stat:`httpcache/miss`.
+
+ It is lower than :stat:`httpcache/miss` when some of those requests yield
+ no response, either because they are dropped (see
+ :stat:`httpcache/ignore`) or because their download fails.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/hit
+
+``httpcache/hit``
+ Number of requests served from the cache.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/ignore
+
+``httpcache/ignore``
+ Number of requests dropped because they were not in the cache and
+ :setting:`HTTPCACHE_IGNORE_MISSING` is ``True``.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/invalidate
+
+``httpcache/invalidate``
+ Number of times that a cached response failed validation and was replaced
+ with a freshly downloaded response.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/miss
+
+``httpcache/miss``
+ Number of requests for which no cache entry could be read, either because
+ there was none or because reading it failed, in which case the request is
+ also counted in :stat:`httpcache/retrieve_error`. Those requests are
+ downloaded (see :stat:`httpcache/firsthand`), or dropped if
+ :setting:`HTTPCACHE_IGNORE_MISSING` is ``True`` (see
+ :stat:`httpcache/ignore`).
+
+ Requests with a stale cache entry are not counted here; see
+ :stat:`httpcache/revalidate` and :stat:`httpcache/invalidate`.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/retrieve_error
+
+``httpcache/retrieve_error``
+ Number of cache entries that could not be read, and hence were treated as
+ cache misses. Those requests are also counted in :stat:`httpcache/miss`.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/revalidate
+
+``httpcache/revalidate``
+ Number of times that a cached response was successfully validated against
+ the target server, and hence used instead of the fresh response.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/store
+
+``httpcache/store``
+ Number of responses stored in the cache.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcache/uncacheable
+
+``httpcache/uncacheable``
+ Number of responses not stored in the cache because the
+ :setting:`HTTPCACHE_POLICY` did not allow it.
+
+ Every response considered for caching is counted either here or in
+ :stat:`httpcache/store`, so ``httpcache/store + httpcache/uncacheable``
+ equals ``httpcache/firsthand + httpcache/invalidate``.
+
+ Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
+
+.. stat:: httpcompression/response_bytes
+
+``httpcompression/response_bytes``
+ Total size, in bytes, of decompressed response bodies, counting only the
+ body and only responses that were actually decompressed. Compare with
+ :stat:`downloader/response_bytes`.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`.
+
+.. stat:: httpcompression/response_count
+
+``httpcompression/response_count``
+ Number of decompressed responses.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`.
+
+.. stat:: httperror/response_ignored_count
+
+``httperror/response_ignored_count``
+ Number of responses dropped because of their HTTP status code.
+
+ Set by :class:`~scrapy.spidermiddlewares.httperror.HttpErrorMiddleware`.
+
+.. stat:: httperror/response_ignored_status_count/{status_code}
+
+``httperror/response_ignored_status_count/{status_code}``
+ Number of responses dropped because of their HTTP status code, per HTTP
+ status code, e.g. ``404``.
+
+ Set by :class:`~scrapy.spidermiddlewares.httperror.HttpErrorMiddleware`.
+
+.. stat:: item_dropped_count
+
+``item_dropped_count``
+ Number of items dropped by an :ref:`item pipeline
+ <topics-item-pipeline>`, i.e. number of times that the
+ :signal:`item_dropped` signal was sent.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: item_dropped_reasons_count/{exception}
+
+``item_dropped_reasons_count/{exception}``
+ Number of items dropped, per exception, where ``{exception}`` is the class
+ name of the exception that caused the item to be dropped.
+
+ Only :exc:`~scrapy.exceptions.DropItem` and its subclasses drop items, and
+ each one is counted under its own class name, e.g.
+ ``item_dropped_reasons_count/DropItem`` for
+ :exc:`~scrapy.exceptions.DropItem` itself and
+ ``item_dropped_reasons_count/MyDropItem`` for a ``MyDropItem`` subclass of
+ it. Any other exception raised by an :ref:`item pipeline
+ <topics-item-pipeline>` triggers the :signal:`item_error` signal instead of
+ :signal:`item_dropped`, and is not counted here or in
+ :stat:`item_dropped_count`.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: item_scraped_count
+
+``item_scraped_count``
+ Number of items that passed all :ref:`item pipelines
+ <topics-item-pipeline>`, i.e. number of times that the
+ :signal:`item_scraped` signal was sent.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: items_per_minute
+
+``items_per_minute``
+ Average number of items scraped per minute during the crawl.
+
+ It is ``None`` if the crawl took less than a minute.
+
+ Set by :class:`~scrapy.extensions.logstats.LogStats`.
+
+.. stat:: log_count/{level}
+
+``log_count/{level}``
+ Number of log messages, per logging level name, e.g. ``INFO`` or
+ ``WARNING``.
+
+ Only messages that the :setting:`LOG_LEVEL` setting allows are counted.
+
+ Set by :class:`~scrapy.extensions.logcount.LogCount`.
+
+.. stat:: memdebug/gc_garbage_count
+
+``memdebug/gc_garbage_count``
+ Number of objects in :data:`gc.garbage` when the spider is closed.
+
+ Set by :class:`~scrapy.extensions.memdebug.MemoryDebugger`, which requires
+ :setting:`MEMDEBUG_ENABLED` to be ``True``.
+
+.. stat:: memdebug/live_refs/{cls}
+
+``memdebug/live_refs/{cls}``
+ Number of live objects of class ``{cls}`` when the spider is closed, as
+ reported by :ref:`trackref <topics-leaks-trackrefs>`, e.g.
+ ``memdebug/live_refs/HtmlResponse``.
+
+ Only set for classes with at least 1 live object.
+
+ Set by :class:`~scrapy.extensions.memdebug.MemoryDebugger`, which requires
+ :setting:`MEMDEBUG_ENABLED` to be ``True``.
+
+.. stat:: memusage/limit_reached
+
+``memusage/limit_reached``
+ ``1`` if memory usage exceeded :setting:`MEMUSAGE_LIMIT_MB`, which also
+ stops the crawl.
+
+ Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
+
+.. stat:: memusage/max
+
+``memusage/max``
+ Maximum peak memory usage, in bytes, observed during the crawl.
+
+ Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
+
+.. stat:: memusage/startup
+
+``memusage/startup``
+ Peak memory usage, in bytes, when the engine started.
+
+ Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
+
+.. stat:: memusage/warning_reached
+
+``memusage/warning_reached``
+ ``1`` if memory usage exceeded :setting:`MEMUSAGE_WARNING_MB`.
+
+ Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
+
+.. stat:: offsite/domains
+
+``offsite/domains``
+ Number of distinct domains for which at least 1 request was dropped for
+ being offsite.
+
+ Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.
+
+.. stat:: offsite/filtered
+
+``offsite/filtered``
+ Number of requests dropped for being offsite.
+
+ Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.
+
+.. stat:: request_depth_count/{depth}
+
+``request_depth_count/{depth}``
+ Number of requests scheduled at depth ``{depth}``, e.g.
+ ``request_depth_count/2``.
+
+ Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`, which
+ requires :setting:`DEPTH_STATS_VERBOSE` to be ``True`` for this stat.
+
+.. stat:: request_depth_max
+
+``request_depth_max``
+ Maximum depth reached.
+
+ Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`.
+
+.. stat:: response_received_count
+
+``response_received_count``
+ Number of responses received, i.e. number of times that the
+ :signal:`response_received` signal was sent.
+
+ Unlike :stat:`downloader/response_count`, it does not count responses that
+ a downloader middleware consumes before they reach the engine, e.g.
+ redirect responses that :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware`
+ turns into new requests. Both count responses that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`
+ serves from the cache.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: responses_per_minute
+
+``responses_per_minute``
+ Average number of responses received per minute during the crawl.
+
+ It is ``None`` if the crawl took less than a minute.
+
+ Set by :class:`~scrapy.extensions.logstats.LogStats`.
+
+.. stat:: retry/count
+
+``retry/count``
+ Number of requests retried.
+
+ Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which
+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.
+
+.. stat:: retry/max_reached
+
+``retry/max_reached``
+ Number of requests that were not retried because they had already been
+ retried :setting:`RETRY_TIMES` times.
+
+ Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which
+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.
+
+.. stat:: retry/reason_count/{reason}
+
+``retry/reason_count/{reason}``
+ Number of requests retried, per reason, e.g.
+ ``retry/reason_count/twisted.internet.error.TimeoutError`` or
+ ``retry/reason_count/504 Gateway Time-out``.
+
+ Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which
+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.
+
+.. note:: Code calling
+ :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` may pass a
+ custom *stats_base_key*, in which case ``retry`` is replaced with that key
+ in the 3 stats above.
+
+.. stat:: robotstxt/exception_count/{exception_type}
+
+``robotstxt/exception_count/{exception_type}``
+ Number of exceptions raised while downloading ``robots.txt`` files, per
+ exception type, where ``{exception_type}`` is the string representation of
+ the exception class, e.g. ``<class
+ 'twisted.internet.error.DNSLookupError'>``.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
+
+.. stat:: robotstxt/forbidden
+
+``robotstxt/forbidden``
+ Number of requests dropped for being disallowed by ``robots.txt``.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
+
+.. stat:: robotstxt/request_count
+
+``robotstxt/request_count``
+ Number of ``robots.txt`` files requested, i.e. 1 per network location for
+ which at least 1 request was sent.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
+
+.. stat:: robotstxt/response_count
+
+``robotstxt/response_count``
+ Number of ``robots.txt`` responses received.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
+
+.. stat:: robotstxt/response_status_count/{status_code}
+
+``robotstxt/response_status_count/{status_code}``
+ Number of ``robots.txt`` responses received, per HTTP status code, e.g.
+ ``404``.
+
+ Set by
+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
+
+.. stat:: scheduler/dequeued
+
+``scheduler/dequeued``
+ Number of requests read from the :ref:`scheduler <topics-scheduler>`.
+
+.. stat:: scheduler/dequeued/disk
+
+``scheduler/dequeued/disk``
+ Number of requests read from the disk queue of the :ref:`scheduler
+ <topics-scheduler>`.
+
+.. stat:: scheduler/dequeued/memory
+
+``scheduler/dequeued/memory``
+ Number of requests read from the memory queue of the :ref:`scheduler
+ <topics-scheduler>`.
+
+.. stat:: scheduler/enqueued
+
+``scheduler/enqueued``
+ Number of requests stored into the :ref:`scheduler <topics-scheduler>`.
+
+.. stat:: scheduler/enqueued/disk
+
+``scheduler/enqueued/disk``
+ Number of requests stored into the disk queue of the :ref:`scheduler
+ <topics-scheduler>`.
+
+.. stat:: scheduler/enqueued/memory
+
+``scheduler/enqueued/memory``
+ Number of requests stored into the memory queue of the :ref:`scheduler
+ <topics-scheduler>`.
+
+.. stat:: scheduler/unserializable
+
+``scheduler/unserializable``
+ Number of requests that could not be stored into the disk queue of the
+ :ref:`scheduler <topics-scheduler>` because they could not be
+ :ref:`serialized <request-serialization>`, and hence were stored into the
+ memory queue instead.
+
+.. stat:: spider_exceptions/count
+
+``spider_exceptions/count``
+ Number of unhandled exceptions raised by spider callbacks.
+
+ Set by the :ref:`scraper <topics-architecture>`.
+
+.. stat:: spider_exceptions/{exception}
+
+``spider_exceptions/{exception}``
+ Number of unhandled exceptions raised by spider callbacks, per exception,
+ where ``{exception}`` is the class name of the exception, e.g.
+ ``spider_exceptions/ValueError``.
+
+ Set by the :ref:`scraper <topics-architecture>`.
+
+.. stat:: start_time
+
+``start_time``
+ Timezone-aware :class:`~datetime.datetime` object, in UTC, indicating when
+ the :signal:`spider_opened` signal was sent.
+
+ Set by :class:`~scrapy.extensions.corestats.CoreStats`.
+
+.. stat:: urllength/request_ignored_count
+
+``urllength/request_ignored_count``
+ Number of requests dropped for having a URL longer than
+ :setting:`URLLENGTH_LIMIT`.
+
+ Set by :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`.
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
index 42c517222..a511b0c7b 100644
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -366,8 +366,8 @@ class Scheduler(BaseScheduler):
Unless the received request is filtered out by the Dupefilter, attempt to push
it into the disk queue, falling back to pushing it into the memory queue.
- Increment the appropriate stats, such as: ``scheduler/enqueued``,
- ``scheduler/enqueued/disk``, ``scheduler/enqueued/memory``.
+ Increment the appropriate stats, such as: :stat:`scheduler/enqueued`,
+ :stat:`scheduler/enqueued/disk`, :stat:`scheduler/enqueued/memory`.
Return ``True`` if the request was stored successfully, ``False`` otherwise.
"""
@@ -390,8 +390,8 @@ class Scheduler(BaseScheduler):
falling back to the disk queue if the memory queue is empty.
Return ``None`` if there are no more enqueued requests.
- Increment the appropriate stats, such as: ``scheduler/dequeued``,
- ``scheduler/dequeued/disk``, ``scheduler/dequeued/memory``.
+ Increment the appropriate stats, such as: :stat:`scheduler/dequeued`,
+ :stat:`scheduler/dequeued/disk`, :stat:`scheduler/dequeued/memory`.
"""
request: Request | None = self.mqs.pop()
assert self.stats is not None
diff --git a/scrapy/extensions/logcount.py b/scrapy/extensions/logcount.py
index e6d51a7d8..fcce64438 100644
--- a/scrapy/extensions/logcount.py
+++ b/scrapy/extensions/logcount.py
@@ -20,7 +20,7 @@ class LogCount:
"""Install a log handler that counts log messages by level.
The handler installed is :class:`scrapy.utils.log.LogCounterHandler`.
- The counts are stored in stats as ``log_count/<level>``.
+ The counts are stored in the :stat:`log_count/{level}` stat.
.. versionadded:: 2.14
"""
diff --git a/tox.ini b/tox.ini
index 18e1579c9..b59221340 100644
--- a/tox.ini
+++ b/tox.ini
@@ -5,7 +5,7 @@
[tox]
requires =
- sphinx-scrapy[tox] @ git+https://github.com/scrapy/[email protected]
+ sphinx-scrapy[tox] @ git+https://github.com/scrapy/[email protected]
envlist =
pre-commit
pylint