Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Provide complete API documentation coverage of scrapy.extensions

Adrián Chaves2828cb769f06
diff --git a/docs/conf.py b/docs/conf.py
index eba416cd6..fa257dead 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -256,6 +256,12 @@ coverage_ignore_pyobjects = [
# Methods of BaseItemExporter subclasses are only documented in
# BaseItemExporter.
r'^scrapy\.exporters\.(?!BaseItemExporter\b)\w*?\.',
+
+ # Extension behavior is only modified through settings. Methods of
+ # extension classes, as well as helper functions, are implementation
+ # details that are not documented.
+ r'^scrapy\.extensions\.[a-z]\w*?\.[A-Z]\w*?\.', # methods
+ r'^scrapy\.extensions\.[a-z]\w*?\.[a-z]', # helper functions
]
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index c25e12bda..0845ef6e4 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -365,24 +365,25 @@ HttpCacheMiddleware
You can also avoid caching a response on every policy using :reqmeta:`dont_cache` meta key equals ``True``.
+.. module:: scrapy.extensions.httpcache
+ :noindex:
+
.. _httpcache-policy-dummy:
Dummy policy (default)
~~~~~~~~~~~~~~~~~~~~~~
-This policy has no awareness of any HTTP Cache-Control directives.
-Every request and its corresponding response are cached. When the same
-request is seen again, the response is returned without transferring
-anything from the Internet.
-
-The Dummy policy is useful for testing spiders faster (without having
-to wait for downloads every time) and for trying your spider offline,
-when an Internet connection is not available. The goal is to be able to
-"replay" a spider run *exactly as it ran before*.
+.. class:: DummyPolicy
-In order to use this policy, set:
+ This policy has no awareness of any HTTP Cache-Control directives.
+ Every request and its corresponding response are cached. When the same
+ request is seen again, the response is returned without transferring
+ anything from the Internet.
-* :setting:`HTTPCACHE_POLICY` to ``scrapy.extensions.httpcache.DummyPolicy``
+ The Dummy policy is useful for testing spiders faster (without having
+ to wait for downloads every time) and for trying your spider offline,
+ when an Internet connection is not available. The goal is to be able to
+ "replay" a spider run *exactly as it ran before*.
.. _httpcache-policy-rfc2616:
@@ -390,45 +391,44 @@ In order to use this policy, set:
RFC2616 policy
~~~~~~~~~~~~~~
-This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP
-Cache-Control awareness, aimed at production and used in continuous
-runs to avoid downloading unmodified data (to save bandwidth and speed up crawls).
+.. class:: RFC2616Policy
-what is implemented:
+ This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP
+ Cache-Control awareness, aimed at production and used in continuous
+ runs to avoid downloading unmodified data (to save bandwidth and speed up
+ crawls).
-* Do not attempt to store responses/requests with ``no-store`` cache-control directive set
-* Do not serve responses from cache if ``no-cache`` cache-control directive is set even for fresh responses
-* Compute freshness lifetime from ``max-age`` cache-control directive
-* Compute freshness lifetime from ``Expires`` response header
-* Compute freshness lifetime from ``Last-Modified`` response header (heuristic used by Firefox)
-* Compute current age from ``Age`` response header
-* Compute current age from ``Date`` header
-* Revalidate stale responses based on ``Last-Modified`` response header
-* Revalidate stale responses based on ``ETag`` response header
-* Set ``Date`` header for any received response missing it
-* Support ``max-stale`` cache-control directive in requests
+ What is implemented:
- This allows spiders to be configured with the full RFC2616 cache policy,
- but avoid revalidation on a request-by-request basis, while remaining
- conformant with the HTTP spec.
+ * Do not attempt to store responses/requests with ``no-store`` cache-control directive set
+ * Do not serve responses from cache if ``no-cache`` cache-control directive is set even for fresh responses
+ * Compute freshness lifetime from ``max-age`` cache-control directive
+ * Compute freshness lifetime from ``Expires`` response header
+ * Compute freshness lifetime from ``Last-Modified`` response header (heuristic used by Firefox)
+ * Compute current age from ``Age`` response header
+ * Compute current age from ``Date`` header
+ * Revalidate stale responses based on ``Last-Modified`` response header
+ * Revalidate stale responses based on ``ETag`` response header
+ * Set ``Date`` header for any received response missing it
+ * Support ``max-stale`` cache-control directive in requests
- Example:
+ This allows spiders to be configured with the full RFC2616 cache policy,
+ but avoid revalidation on a request-by-request basis, while remaining
+ conformant with the HTTP spec.
- Add ``Cache-Control: max-stale=600`` to Request headers to accept responses that
- have exceeded their expiration time by no more than 600 seconds.
+ Example:
- See also: RFC2616, 14.9.3
+ Add ``Cache-Control: max-stale=600`` to Request headers to accept responses that
+ have exceeded their expiration time by no more than 600 seconds.
-what is missing:
+ See also: RFC2616, 14.9.3
-* ``Pragma: no-cache`` support https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
-* ``Vary`` header support https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6
-* Invalidation after updates or deletes https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10
-* ... probably others ..
+ What is missing:
-In order to use this policy, set:
-
-* :setting:`HTTPCACHE_POLICY` to ``scrapy.extensions.httpcache.RFC2616Policy``
+ * ``Pragma: no-cache`` support https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
+ * ``Vary`` header support https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6
+ * Invalidation after updates or deletes https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10
+ * ... probably others ..
.. _httpcache-storage-fs:
@@ -436,67 +436,68 @@ In order to use this policy, set:
Filesystem storage backend (default)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-File system storage backend is available for the HTTP cache middleware.
+.. class:: FilesystemCacheStorage
+
+ File system storage backend is available for the HTTP cache middleware.
-In order to use this storage backend, set:
+ Each request/response pair is stored in a different directory containing
+ the following files:
-* :setting:`HTTPCACHE_STORAGE` to ``scrapy.extensions.httpcache.FilesystemCacheStorage``
+ * ``request_body`` - the plain request body
-Each request/response pair is stored in a different directory containing
-the following files:
+ * ``request_headers`` - the request headers (in raw HTTP format)
- * ``request_body`` - the plain request body
- * ``request_headers`` - the request headers (in raw HTTP format)
- * ``response_body`` - the plain response body
- * ``response_headers`` - the request headers (in raw HTTP format)
- * ``meta`` - some metadata of this cache resource in Python ``repr()`` format
- (grep-friendly format)
- * ``pickled_meta`` - the same metadata in ``meta`` but pickled for more
- efficient deserialization
+ * ``response_body`` - the plain response body
-The directory name is made from the request fingerprint (see
-``scrapy.utils.request.fingerprint``), and one level of subdirectories is
-used to avoid creating too many files into the same directory (which is
-inefficient in many file systems). An example directory could be::
+ * ``response_headers`` - the request headers (in raw HTTP format)
- /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
+ * ``meta`` - some metadata of this cache resource in Python ``repr()``
+ format (grep-friendly format)
+
+ * ``pickled_meta`` - the same metadata in ``meta`` but pickled for more
+ efficient deserialization
+
+ The directory name is made from the request fingerprint (see
+ ``scrapy.utils.request.fingerprint``), and one level of subdirectories is
+ used to avoid creating too many files into the same directory (which is
+ inefficient in many file systems). An example directory could be::
+
+ /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
.. _httpcache-storage-dbm:
DBM storage backend
~~~~~~~~~~~~~~~~~~~
-.. versionadded:: 0.13
-
-A DBM_ storage backend is also available for the HTTP cache middleware.
+.. class:: DbmCacheStorage
-By default, it uses the anydbm_ module, but you can change it with the
-:setting:`HTTPCACHE_DBM_MODULE` setting.
+ .. versionadded:: 0.13
-In order to use this storage backend, set:
+ A DBM_ storage backend is also available for the HTTP cache middleware.
-* :setting:`HTTPCACHE_STORAGE` to ``scrapy.extensions.httpcache.DbmCacheStorage``
+ By default, it uses the anydbm_ module, but you can change it with the
+ :setting:`HTTPCACHE_DBM_MODULE` setting.
.. _httpcache-storage-leveldb:
LevelDB storage backend
~~~~~~~~~~~~~~~~~~~~~~~
-.. versionadded:: 0.23
+.. class:: LeveldbCacheStorage
-A LevelDB_ storage backend is also available for the HTTP cache middleware.
+ .. versionadded:: 0.23
-This backend is not recommended for development because only one process can
-access LevelDB databases at the same time, so you can't run a crawl and open
-the scrapy shell in parallel for the same spider.
+ A LevelDB_ storage backend is also available for the HTTP cache middleware.
-In order to use this storage backend:
+ This backend is not recommended for development because only one process
+ can access LevelDB databases at the same time, so you can't run a crawl and
+ open the scrapy shell in parallel for the same spider.
-* set :setting:`HTTPCACHE_STORAGE` to ``scrapy.extensions.httpcache.LeveldbCacheStorage``
-* install `LevelDB python bindings`_ like ``pip install leveldb``
+ In order to use this storage backend, install the `LevelDB python
+ bindings`_ (e.g. ``pip install leveldb``).
-.. _LevelDB: https://github.com/google/leveldb
-.. _leveldb python bindings: https://pypi.python.org/pypi/leveldb
+ .. _LevelDB: https://github.com/google/leveldb
+ .. _leveldb python bindings: https://pypi.python.org/pypi/leveldb
.. _httpcache-storage-custom:
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index d6e7452a1..72c2290b5 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -183,7 +183,7 @@ Telnet console extension
.. module:: scrapy.extensions.telnet
:synopsis: Telnet console
-.. class:: scrapy.extensions.telnet.TelnetConsole
+.. class:: TelnetConsole
Provides a telnet console for getting into a Python interpreter inside the
currently running Scrapy process, which can be very useful for debugging.
@@ -200,7 +200,7 @@ Memory usage extension
.. module:: scrapy.extensions.memusage
:synopsis: Memory usage extension
-.. class:: scrapy.extensions.memusage.MemoryUsage
+.. class:: MemoryUsage
.. note:: This extension does not work in Windows.
@@ -228,7 +228,7 @@ Memory debugger extension
.. module:: scrapy.extensions.memdebug
:synopsis: Memory debugger extension
-.. class:: scrapy.extensions.memdebug.MemoryDebugger
+.. class:: MemoryDebugger
An extension for debugging memory usage. It collects information about:
@@ -244,7 +244,7 @@ Close spider extension
.. module:: scrapy.extensions.closespider
:synopsis: Close spider extension
-.. class:: scrapy.extensions.closespider.CloseSpider
+.. class:: CloseSpider
Closes a spider automatically when some conditions are met, using a specific
closing reason for each condition.
@@ -317,7 +317,7 @@ StatsMailer extension
.. module:: scrapy.extensions.statsmailer
:synopsis: StatsMailer extension
-.. class:: scrapy.extensions.statsmailer.StatsMailer
+.. class:: StatsMailer
This simple extension can be used to send a notification e-mail every time a
domain has finished scraping, including the Scrapy stats collected. The email
@@ -333,7 +333,7 @@ Debugging extensions
Stack trace dump extension
~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. class:: scrapy.extensions.debug.StackTraceDump
+.. class:: StackTraceDump
Dumps information about the running process when a `SIGQUIT`_ or `SIGUSR2`_
signal is received. The information dumped is the following:
@@ -362,7 +362,7 @@ There are at least two ways to send Scrapy the `SIGQUIT`_ signal:
Debugger extension
~~~~~~~~~~~~~~~~~~
-.. class:: scrapy.extensions.debug.Debugger
+.. class:: Debugger
Invokes a `Python debugger`_ inside a running Scrapy process when a `SIGUSR2`_
signal is received. After the debugger is exited, the Scrapy process continues