Scrapy, a fast high-level web crawling & scraping framework for Python.
Provide complete API documentation coverage of scrapy.extensions
Adrián Chaves2828cb769f06
diff --git a/docs/conf.py b/docs/conf.pyindex eba416cd6..fa257dead 100644--- a/docs/conf.py+++ b/docs/conf.py@@ -256,6 +256,12 @@ coverage_ignore_pyobjects = [# Methods of BaseItemExporter subclasses are only documented in# BaseItemExporter.r'^scrapy\.exporters\.(?!BaseItemExporter\b)\w*?\.',++ # Extension behavior is only modified through settings. Methods of+ # extension classes, as well as helper functions, are implementation+ # details that are not documented.+ r'^scrapy\.extensions\.[a-z]\w*?\.[A-Z]\w*?\.', # methods+ r'^scrapy\.extensions\.[a-z]\w*?\.[a-z]', # helper functions]diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex c25e12bda..0845ef6e4 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -365,24 +365,25 @@ HttpCacheMiddlewareYou can also avoid caching a response on every policy using :reqmeta:`dont_cache` meta key equals ``True``.+.. module:: scrapy.extensions.httpcache+ :noindex:+.. _httpcache-policy-dummy:Dummy policy (default)~~~~~~~~~~~~~~~~~~~~~~-This policy has no awareness of any HTTP Cache-Control directives.-Every request and its corresponding response are cached. When the same-request is seen again, the response is returned without transferring-anything from the Internet.--The Dummy policy is useful for testing spiders faster (without having-to wait for downloads every time) and for trying your spider offline,-when an Internet connection is not available. The goal is to be able to-"replay" a spider run *exactly as it ran before*.+.. class:: DummyPolicy-In order to use this policy, set:+ This policy has no awareness of any HTTP Cache-Control directives.+ Every request and its corresponding response are cached. When the same+ request is seen again, the response is returned without transferring+ anything from the Internet.-* :setting:`HTTPCACHE_POLICY` to ``scrapy.extensions.httpcache.DummyPolicy``+ The Dummy policy is useful for testing spiders faster (without having+ to wait for downloads every time) and for trying your spider offline,+ when an Internet connection is not available. The goal is to be able to+ "replay" a spider run *exactly as it ran before*... _httpcache-policy-rfc2616:@@ -390,45 +391,44 @@ In order to use this policy, set:RFC2616 policy~~~~~~~~~~~~~~-This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP-Cache-Control awareness, aimed at production and used in continuous-runs to avoid downloading unmodified data (to save bandwidth and speed up crawls).+.. class:: RFC2616Policy-what is implemented:+ This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP+ Cache-Control awareness, aimed at production and used in continuous+ runs to avoid downloading unmodified data (to save bandwidth and speed up+ crawls).-* Do not attempt to store responses/requests with ``no-store`` cache-control directive set-* Do not serve responses from cache if ``no-cache`` cache-control directive is set even for fresh responses-* Compute freshness lifetime from ``max-age`` cache-control directive-* Compute freshness lifetime from ``Expires`` response header-* Compute freshness lifetime from ``Last-Modified`` response header (heuristic used by Firefox)-* Compute current age from ``Age`` response header-* Compute current age from ``Date`` header-* Revalidate stale responses based on ``Last-Modified`` response header-* Revalidate stale responses based on ``ETag`` response header-* Set ``Date`` header for any received response missing it-* Support ``max-stale`` cache-control directive in requests+ What is implemented:- This allows spiders to be configured with the full RFC2616 cache policy,- but avoid revalidation on a request-by-request basis, while remaining- conformant with the HTTP spec.+ * Do not attempt to store responses/requests with ``no-store`` cache-control directive set+ * Do not serve responses from cache if ``no-cache`` cache-control directive is set even for fresh responses+ * Compute freshness lifetime from ``max-age`` cache-control directive+ * Compute freshness lifetime from ``Expires`` response header+ * Compute freshness lifetime from ``Last-Modified`` response header (heuristic used by Firefox)+ * Compute current age from ``Age`` response header+ * Compute current age from ``Date`` header+ * Revalidate stale responses based on ``Last-Modified`` response header+ * Revalidate stale responses based on ``ETag`` response header+ * Set ``Date`` header for any received response missing it+ * Support ``max-stale`` cache-control directive in requests- Example:+ This allows spiders to be configured with the full RFC2616 cache policy,+ but avoid revalidation on a request-by-request basis, while remaining+ conformant with the HTTP spec.- Add ``Cache-Control: max-stale=600`` to Request headers to accept responses that- have exceeded their expiration time by no more than 600 seconds.+ Example:- See also: RFC2616, 14.9.3+ Add ``Cache-Control: max-stale=600`` to Request headers to accept responses that+ have exceeded their expiration time by no more than 600 seconds.-what is missing:+ See also: RFC2616, 14.9.3-* ``Pragma: no-cache`` support https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1-* ``Vary`` header support https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6-* Invalidation after updates or deletes https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10-* ... probably others ..+ What is missing:-In order to use this policy, set:--* :setting:`HTTPCACHE_POLICY` to ``scrapy.extensions.httpcache.RFC2616Policy``+ * ``Pragma: no-cache`` support https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1+ * ``Vary`` header support https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6+ * Invalidation after updates or deletes https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10+ * ... probably others .... _httpcache-storage-fs:@@ -436,67 +436,68 @@ In order to use this policy, set:Filesystem storage backend (default)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-File system storage backend is available for the HTTP cache middleware.+.. class:: FilesystemCacheStorage++ File system storage backend is available for the HTTP cache middleware.-In order to use this storage backend, set:+ Each request/response pair is stored in a different directory containing+ the following files:-* :setting:`HTTPCACHE_STORAGE` to ``scrapy.extensions.httpcache.FilesystemCacheStorage``+ * ``request_body`` - the plain request body-Each request/response pair is stored in a different directory containing-the following files:+ * ``request_headers`` - the request headers (in raw HTTP format)- * ``request_body`` - the plain request body- * ``request_headers`` - the request headers (in raw HTTP format)- * ``response_body`` - the plain response body- * ``response_headers`` - the request headers (in raw HTTP format)- * ``meta`` - some metadata of this cache resource in Python ``repr()`` format- (grep-friendly format)- * ``pickled_meta`` - the same metadata in ``meta`` but pickled for more- efficient deserialization+ * ``response_body`` - the plain response body-The directory name is made from the request fingerprint (see-``scrapy.utils.request.fingerprint``), and one level of subdirectories is-used to avoid creating too many files into the same directory (which is-inefficient in many file systems). An example directory could be::+ * ``response_headers`` - the request headers (in raw HTTP format)- /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7+ * ``meta`` - some metadata of this cache resource in Python ``repr()``+ format (grep-friendly format)++ * ``pickled_meta`` - the same metadata in ``meta`` but pickled for more+ efficient deserialization++ The directory name is made from the request fingerprint (see+ ``scrapy.utils.request.fingerprint``), and one level of subdirectories is+ used to avoid creating too many files into the same directory (which is+ inefficient in many file systems). An example directory could be::++ /path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7.. _httpcache-storage-dbm:DBM storage backend~~~~~~~~~~~~~~~~~~~-.. versionadded:: 0.13--A DBM_ storage backend is also available for the HTTP cache middleware.+.. class:: DbmCacheStorage-By default, it uses the anydbm_ module, but you can change it with the-:setting:`HTTPCACHE_DBM_MODULE` setting.+ .. versionadded:: 0.13-In order to use this storage backend, set:+ A DBM_ storage backend is also available for the HTTP cache middleware.-* :setting:`HTTPCACHE_STORAGE` to ``scrapy.extensions.httpcache.DbmCacheStorage``+ By default, it uses the anydbm_ module, but you can change it with the+ :setting:`HTTPCACHE_DBM_MODULE` setting... _httpcache-storage-leveldb:LevelDB storage backend~~~~~~~~~~~~~~~~~~~~~~~-.. versionadded:: 0.23+.. class:: LeveldbCacheStorage-A LevelDB_ storage backend is also available for the HTTP cache middleware.+ .. versionadded:: 0.23-This backend is not recommended for development because only one process can-access LevelDB databases at the same time, so you can't run a crawl and open-the scrapy shell in parallel for the same spider.+ A LevelDB_ storage backend is also available for the HTTP cache middleware.-In order to use this storage backend:+ This backend is not recommended for development because only one process+ can access LevelDB databases at the same time, so you can't run a crawl and+ open the scrapy shell in parallel for the same spider.-* set :setting:`HTTPCACHE_STORAGE` to ``scrapy.extensions.httpcache.LeveldbCacheStorage``-* install `LevelDB python bindings`_ like ``pip install leveldb``+ In order to use this storage backend, install the `LevelDB python+ bindings`_ (e.g. ``pip install leveldb``).-.. _LevelDB: https://github.com/google/leveldb-.. _leveldb python bindings: https://pypi.python.org/pypi/leveldb+ .. _LevelDB: https://github.com/google/leveldb+ .. _leveldb python bindings: https://pypi.python.org/pypi/leveldb.. _httpcache-storage-custom:diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex d6e7452a1..72c2290b5 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -183,7 +183,7 @@ Telnet console extension.. module:: scrapy.extensions.telnet:synopsis: Telnet console-.. class:: scrapy.extensions.telnet.TelnetConsole+.. class:: TelnetConsoleProvides a telnet console for getting into a Python interpreter inside thecurrently running Scrapy process, which can be very useful for debugging.@@ -200,7 +200,7 @@ Memory usage extension.. module:: scrapy.extensions.memusage:synopsis: Memory usage extension-.. class:: scrapy.extensions.memusage.MemoryUsage+.. class:: MemoryUsage.. note:: This extension does not work in Windows.@@ -228,7 +228,7 @@ Memory debugger extension.. module:: scrapy.extensions.memdebug:synopsis: Memory debugger extension-.. class:: scrapy.extensions.memdebug.MemoryDebugger+.. class:: MemoryDebuggerAn extension for debugging memory usage. It collects information about:@@ -244,7 +244,7 @@ Close spider extension.. module:: scrapy.extensions.closespider:synopsis: Close spider extension-.. class:: scrapy.extensions.closespider.CloseSpider+.. class:: CloseSpiderCloses a spider automatically when some conditions are met, using a specificclosing reason for each condition.@@ -317,7 +317,7 @@ StatsMailer extension.. module:: scrapy.extensions.statsmailer:synopsis: StatsMailer extension-.. class:: scrapy.extensions.statsmailer.StatsMailer+.. class:: StatsMailerThis simple extension can be used to send a notification e-mail every time adomain has finished scraping, including the Scrapy stats collected. The email@@ -333,7 +333,7 @@ Debugging extensionsStack trace dump extension~~~~~~~~~~~~~~~~~~~~~~~~~~-.. class:: scrapy.extensions.debug.StackTraceDump+.. class:: StackTraceDumpDumps information about the running process when a `SIGQUIT`_ or `SIGUSR2`_signal is received. The information dumped is the following:@@ -362,7 +362,7 @@ There are at least two ways to send Scrapy the `SIGQUIT`_ signal:Debugger extension~~~~~~~~~~~~~~~~~~-.. class:: scrapy.extensions.debug.Debugger+.. class:: DebuggerInvokes a `Python debugger`_ inside a running Scrapy process when a `SIGUSR2`_signal is received. After the debugger is exited, the Scrapy process continues