Scrapy, a fast high-level web crawling & scraping framework for Python.
More quick doc fixes.
Andrey Rakhmatullin27f5f3513437
diff --git a/docs/topics/addons.rst b/docs/topics/addons.rstindex 25d7da50b..523f6e86e 100644--- a/docs/topics/addons.rst+++ b/docs/topics/addons.rst@@ -40,10 +40,10 @@ This is an example where an internal add-on and two third-party add-ons (in thiscase with one requiring no configuration) are enabled/configured in a project's``settings.py``::- ADDONS = (- 'httpcache',- 'path.to.some.addon',- )+ ADDONS = {+ 'httpcache': 0,+ 'path.to.some.addon': 0,+ }HTTPCACHE = {'expiration_secs': 60,@@ -385,126 +385,3 @@ Forward to other add-ons depending on Python version::_addon = 'path.to.addon'else:_addon = 'path.to.other.addon'---Built-in add-on reference-=========================--Scrapy comes with gateway add-ons that you can use to configure the built-in-middlewares and extensions. For example, to activate and configure the-:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`, instead-of placing this in your ``settings.py``::-- HTTPCACHE_ENABLED = True- HTTPCACHE_EXPIRATION_SECS = 60- HTTPCACHE_IGNORE_HTTP_CODES = [404]--you can also use the add-on framework::-- ADDONS = (- # ...,- 'httpcache',- )-- HTTPCACHE = {- 'expiration_secs': 60,- 'ignore_http_codes': [404],- }--Note that you *must* enable built-in addons by placing them in your-``ADDONS`` setting before you can use them for configuring built-in-components. I.e., configuring the ``HTTPCACHE`` setting will have no effect-when ``httpcache`` is not listed in ``ADDONS``.--In general, the add-on names match the lowercase name of the component, with its-type suffix removed (i.e. the add-on configuring the-:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` is called-``httpcache``), and the configuration option names match the names of the-settings they map to, with the component prefix removed (i.e.-``expiration_secs`` maps to :setting:`HTTPCACHE_EXPIRATION_SECS`, as above).-The available add-ons are:---+--------------------------------------+--------------------------------------+-| Add-on | Notes |-+======================================+======================================+-| **Spider middlewares** |-+--------------------------------------+--------------------------------------+-| depth (:class:`~scrapy.spidermi\ | |-| ddlewares.depth.DepthMiddleware`) | |-+--------------------------------------+--------------------------------------+-| httperror (:class:`~scrapy.spid\ | |-| ermiddlewares.httperror.HttpErrorMi\ | |-| ddleware`) | |-+--------------------------------------+--------------------------------------+-| offsite (:class:`~scrapy.spid\ | |-| ermiddlewares.offsite.OffsiteMiddle\ | |-| ware`) | |-+--------------------------------------+--------------------------------------+-| referer (:class:`~scrapy.spid\ | |-| ermiddlewares.referer.RefererMiddle\ | |-| ware`) | |-+--------------------------------------+--------------------------------------+-| urllength (:class:`~scrapy.spid\ | |-| ermiddlewares.urllength.UrlLengthMi\ | |-| ddleware`) | |-+--------------------------------------+--------------------------------------+-| **Downloader middlewares** |-+--------------------------------------+--------------------------------------+-| ajaxcrawl (:class:`~scrapy.download\ | |-| ermiddlewares.ajaxcrawl.AjaxCrawlMi\ | |-| ddleware`) | |-+--------------------------------------+--------------------------------------+-| chunked (:class:`~scrapy.download\ | |-| ermiddlewares.chunked.ChunkedTrans\ | |-| ferMiddleware`) | |-+--------------------------------------+--------------------------------------+-| cookies (:class:`~scrapy.download\ | |-| ermiddlewares.cookies.CookiesMiddle\ | |-| ware`) | |-+--------------------------------------+--------------------------------------+-| defaultheaders (:class:`~scrapy.down\| Every configuration entry is treated |-| loadermiddlewares.defaultheaders.Def\| as a default header. |-| aultHeadersMiddleware`) | |-+--------------------------------------+--------------------------------------+-| **Extensions** |-+--------------------------------------+--------------------------------------+-| autothrottle | Installing sets |-| (:ref:`topics-autothrottle`) | :setting:`AUTOTHROTTLE_ENABLED` to |-| | ``True``. |-+--------------------------------------+--------------------------------------+-| corestats (:class:`~scrapy.exten\ | |-| sions.corestats.CoreStats`) | |-+--------------------------------------+--------------------------------------+-| closespider (:class:`~scrapy.exten\ | |-| sions.closespider.CloseSpider`) | |-+--------------------------------------+--------------------------------------+-| debugger (:class:`~scrapy.exten\ | |-| sions.debug.Debugger`) | |-+--------------------------------------+--------------------------------------+-| feedexport (:ref:`topics-feed-expor\ | |-| ts`) | |-+--------------------------------------+--------------------------------------+-| logstats (:class:`~scrapy.exten\ | |-| sions.logstats.LogStats`) | |-+--------------------------------------+--------------------------------------+-| memdebug (:class:`~scrapy.exten\ | Installing sets |-| sions.memdebug.MemoryDebugger`) | :setting:`MEMDEBUG_ENABLED` to |-| | ``True``. |-+--------------------------------------+--------------------------------------+-| memusage (:class:`~scrapy.exten\ | Installing sets |-| sions.memusage.MemoryUsage`) | :setting:`MEMUSAGE_ENABLED` to |-| | ``True``. |-+--------------------------------------+--------------------------------------+-| spiderstate (:class:`~scrapy.exten\ | |-| sions.spiderstate.SpiderState`) | |-+--------------------------------------+--------------------------------------+-| stacktracedump (:class:`~scrapy.ext\ | |-| ensions.debug.StackTraceDump`) | |-+--------------------------------------+--------------------------------------+-| statsmailer (:class:`~scrapy.exten\ | |-| sions.statsmailer.StatsMailer`) | |-+--------------------------------------+--------------------------------------+-| telnetconsole (:ref:`topics-telnet\ | |-| console`) | |-+--------------------------------------+--------------------------------------+diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex aa09abbd5..143002360 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -206,10 +206,10 @@ component must be enabled in order for the setting to have any effect.ADDONS-------Default: ``()``+Default: ``{}``-A tuple containing paths to the add-ons enabled in your project. For more-information, see :ref:`topics-addons`.+A dict containing paths to the add-ons enabled in your project and their+priorities. For more information, see :ref:`topics-addons`... setting:: AWS_ACCESS_KEY_ID