Scrapy, a fast high-level web crawling & scraping framework for Python.
Assorted docs fixes, part 1. (#7710)
Andrey Rakhmatullinbdf3067935fd
diff --git a/docs/README.rst b/docs/README.rstdeleted file mode 100644index 4d2236b53..000000000--- a/docs/README.rst+++ /dev/null@@ -1,68 +0,0 @@-:orphan:--======================================-Scrapy documentation quick start guide-======================================--This file provides a quick guide on how to compile the Scrapy documentation.---Setup the environment------------------------To compile the documentation you need Sphinx Python library. To install it-and all its dependencies run the following command from this dir--::-- pip install -r requirements.txt---Compile the documentation----------------------------To compile the documentation (to classic HTML output) run the following command-from this dir::-- make html--Documentation will be generated (in HTML format) inside the ``build/html`` dir.---View the documentation-------------------------To view the documentation run the following command::-- make htmlview--This command will fire up your default browser and open the main page of your-(previously generated) HTML documentation.---Start over-------------To clean up all generated documentation files and start from scratch run::-- make clean--Keep in mind that this command won't touch any documentation source files.---Recreating documentation on the fly--------------------------------------There is a way to recreate the doc automatically when you make changes, you-need to install watchdog (``pip install watchdog``) and then use::-- make watch--Alternative method using tox-------------------------------To compile the documentation to HTML run the following command::-- tox -e docs--Documentation will be generated inside the ``docs/_build/all`` dir.diff --git a/docs/contributing.rst b/docs/contributing.rstindex c868a0ac4..b43d30796 100644--- a/docs/contributing.rst+++ b/docs/contributing.rst@@ -323,9 +323,10 @@ deprecation removals are documented in the :ref:`release notes <news>`.Tests=====-Tests are implemented using the :doc:`Twisted unit-testing framework-<twisted:development/test-standard>`. Running tests requires-:doc:`tox <tox:index>`.+Tests are implemented using pytest_. Running tests requires :doc:`tox+<tox:index>`.++.. _pytest: https://pytest.org.. _running-tests:diff --git a/docs/faq.rst b/docs/faq.rstindex 6eeb81a31..a396a8212 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -285,7 +285,8 @@ consume a lot of memory.In order to avoid parsing all the entire feed at once in memory, you can usethe :func:`~scrapy.utils.iterators.xmliter_lxml` and:func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what-:class:`~scrapy.spiders.XMLFeedSpider` uses.+:class:`~scrapy.spiders.XMLFeedSpider` and+:class:`~scrapy.spiders.CSVFeedSpider` use... autofunction:: scrapy.utils.iterators.xmliter_lxml@@ -360,10 +361,11 @@ method for this purpose. For example:def process_spider_output(self, response, result):for item_or_request in result:if isinstance(item_or_request, Request):+ yield item_or_requestcontinue- adapter = ItemAdapter(item)+ adapter = ItemAdapter(item_or_request)for _ in range(adapter["multiply_by"]):- yield deepcopy(item)+ yield deepcopy(item_or_request)Does Scrapy support IPv6 addresses?-----------------------------------diff --git a/docs/index.rst b/docs/index.rstindex 8e8624a22..688cab81b 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -24,7 +24,7 @@ Having trouble? We'd like to help!* Ask or search questions in `StackOverflow using the scrapy tag`_.* Ask or search questions in the `Scrapy subreddit`_.* Search for questions on the archives of the `scrapy-users mailing list`_.-* Ask a question in the `#scrapy IRC channel`_,+* Ask a question in the `#scrapy IRC channel`_.* Report bugs with Scrapy in our `issue tracker`_.* Join the Discord community `Scrapy Discord`_.@@ -91,15 +91,15 @@ Basic concepts:doc:`topics/selectors`Extract the data from web pages using XPath.-:doc:`topics/shell`- Test your extraction code in an interactive environment.-:doc:`topics/items`Define the data you want to scrape.:doc:`topics/loaders`Populate your items with the extracted data.+:doc:`topics/shell`+ Test your extraction code in an interactive environment.+:doc:`topics/item-pipeline`Post-process and store your scraped data.diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex ee91ce7ca..488e1c2e3 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -90,9 +90,10 @@ to figure these settings out automatically... note::This is using :ref:`feed exports <topics-feed-exports>` to generate the- JSON file, you can easily change the export format (XML or CSV, for example) or the- storage backend (FTP or `Amazon S3`_, for example). You can also write an- :ref:`item pipeline <topics-item-pipeline>` to store the items in a database.+ JSON Lines file, you can easily change the export format (XML or CSV, for+ example) or the storage backend (FTP or `Amazon S3`_, for example). You can+ also write an :ref:`item pipeline <topics-item-pipeline>` to store the+ items in a database... _topics-whatelse:diff --git a/docs/news.rst b/docs/news.rstindex 7d2b07b09..63f0d69e4 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -1035,7 +1035,7 @@ DeprecationsNew features~~~~~~~~~~~~-- Added a new setting, :setting:`REFERER_POLICIES`, to allow customizing+- Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizingsupported referrer policies.Bug fixes@@ -6046,7 +6046,7 @@ The following changes may impact custom priority queue classes:* A new keyword parameter has been added: ``key``. It is a stringthat is always an empty string for memory queues and indicates the- :setting:`JOB_DIR` value for disk queues.+ :setting:`JOBDIR` value for disk queues.* The parameter for disk queues that contains data from the previouscrawl, ``startprios`` or ``slot_startprios``, is now passed as a@@ -6600,7 +6600,7 @@ New features``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be:ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significantscheduling improvement on crawls targeting multiple web domains, at the- cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)+ cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:issue:`3520`)* A new :attr:`.Request.cb_kwargs` attributeprovides a cleaner way to pass keyword arguments to callback methods@@ -9079,7 +9079,7 @@ New features and settings- In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`)- Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`)- ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by:- - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, :setting:`CONCURRENT_REQUESTS_PER_IP`+ - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP``- check the documentation for more details- Added builtin caching DNS resolver (:rev:`2728`)- Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`)diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex 1a151044f..17b322b5a 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -105,6 +105,9 @@ Scrapy API requires passing a Deferred to it) using the following helpers:.. autofunction:: scrapy.utils.defer.deferred_from_coro.. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f++The following function helps with a reverse wrapping:+.. autofunction:: scrapy.utils.defer.ensure_awaitable@@ -190,7 +193,7 @@ in future Scrapy versions. The following features are not available::class:`~scrapy.crawler.CrawlerProcess`(:class:`~scrapy.crawler.AsyncCrawlerProcess` and:class:`~scrapy.crawler.AsyncCrawlerRunner` are available)-* Twisted-specific DNS resolvers (the :setting:`DNS_RESOLVER` setting)+* Twisted-specific DNS resolvers (the :setting:`TWISTED_DNS_RESOLVER` setting)* User and 3rd-party code that requires a reactor (see :ref:`below<asyncio-without-reactor-migrate>` for examples)@@ -315,8 +318,7 @@ implementations, :class:`~asyncio.ProactorEventLoop` (default) and:class:`~asyncio.SelectorEventLoop` works with Twisted.Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop`-automatically when you change the :setting:`TWISTED_REACTOR` setting or call-:func:`~scrapy.utils.reactor.install_reactor`.+automatically when installing the asyncio reactor... note:: Other libraries you use may require:class:`~asyncio.ProactorEventLoop`, e.g. because it supportsdiff --git a/docs/topics/commands.rst b/docs/topics/commands.rstindex 8d1351eb9..270d9859d 100644--- a/docs/topics/commands.rst+++ b/docs/topics/commands.rst@@ -199,6 +199,7 @@ Global commands:* :command:`fetch`* :command:`view`* :command:`version`+* :command:`bench`Project-only commands:@@ -207,7 +208,6 @@ Project-only commands:* :command:`list`* :command:`edit`* :command:`parse`-* :command:`bench`.. command:: startproject@@ -476,7 +476,7 @@ Supported options:* ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider-* ``--a NAME=VALUE``: set spider argument (may be repeated)+* ``-a NAME=VALUE``: set spider argument (may be repeated)* ``--callback`` or ``-c``: spider method to use as callback for parsing theresponsediff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 116aac323..8aaf0aa31 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -268,5 +268,5 @@ You can also send multiple requests in parallel:yield {"h1": response.css("h1::text").get(),"price": responses[0].css(".price::text").get(),- "price2": responses[1].css(".color::text").get(),+ "color": responses[1].css(".color::text").get(),}diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 054c1130c..383f331b5 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -505,7 +505,7 @@ Filesystem storage backend (default)* ``response_body`` - the plain response body- * ``response_headers`` - the request headers (in raw HTTP format)+ * ``response_headers`` - the response headers (in raw HTTP format)* ``meta`` - some metadata of this cache resource in Python ``repr()``format (grep-friendly format)@@ -547,7 +547,7 @@ defines the methods described below... method:: open_spider(spider)This method gets called after a spider has been opened for crawling. It handles- the :signal:`open_spider <spider_opened>` signal.+ the :signal:`spider_opened` signal.:param spider: the spider which has been opened:type spider: :class:`~scrapy.Spider` object@@ -555,7 +555,7 @@ defines the methods described below... method:: close_spider(spider)This method gets called after a spider has been closed. It handles- the :signal:`close_spider <spider_closed>` signal.+ the :signal:`spider_closed` signal.:param spider: the spider which has been closed:type spider: :class:`~scrapy.Spider` object@@ -814,7 +814,6 @@ HttpProxyMiddleware settings~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. setting:: HTTPPROXY_ENABLED-.. setting:: HTTPPROXY_AUTH_ENCODINGHTTPPROXY_ENABLED^^^^^^^^^^^^^^^^^@@ -823,6 +822,8 @@ Default: ``True``Whether or not to enable the :class:`HttpProxyMiddleware`.+.. setting:: HTTPPROXY_AUTH_ENCODING+HTTPPROXY_AUTH_ENCODING^^^^^^^^^^^^^^^^^^^^^^^@@ -984,7 +985,7 @@ Whether the Meta Refresh middleware will be enabled.METAREFRESH_IGNORE_TAGS^^^^^^^^^^^^^^^^^^^^^^^-Default: ``[]``+Default: ``["noscript"]``Meta tags within these tags are ignored.@@ -1091,7 +1092,7 @@ Default::'twisted.internet.error.ConnectionDone','twisted.internet.error.ConnectError','twisted.internet.error.ConnectionLost',- IOError,+ OSError,'scrapy.core.downloader.handlers.http11.TunnelError',]diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rstindex 5a399e094..30b6536c7 100644--- a/docs/topics/dynamic-content.rst+++ b/docs/topics/dynamic-content.rst@@ -133,7 +133,7 @@ data from it depends on the type of response:.. code-block:: python- selector = Selector(data["html"])+ selector = Selector(text=data["html"])- If the response is JavaScript, or HTML with a ``<script/>`` elementcontaining the desired data, see :ref:`topics-parsing-javascript`.diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rstindex cc690d609..4c56a0f14 100644--- a/docs/topics/exceptions.rst+++ b/docs/topics/exceptions.rst@@ -31,7 +31,7 @@ For example:.. code-block:: pythondef parse_page(self, response):- if "Bandwidth exceeded" in response.body:+ if "Bandwidth exceeded" in response.text:raise CloseSpider("bandwidth_exceeded")DontCloseSpiderdiff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rstindex c4cd05683..f0b704b9d 100644--- a/docs/topics/exporters.rst+++ b/docs/topics/exporters.rst@@ -153,7 +153,7 @@ output examples, which assume you're exporting these two items:BaseItemExporter-----------------.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding='utf-8', indent=0, dont_fail=False)+.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding=None, indent=None, dont_fail=False)This is the (abstract) base class for all Item Exporters. It providessupport for common features used by all (concrete) Item Exporters, such as@@ -239,7 +239,7 @@ BaseItemExporter.. attribute:: indent- Amount of spaces used to indent the output on each level. Defaults to ``0``.+ Amount of spaces used to indent the output on each level. Defaults to ``None``.* ``indent=None`` selects the most compact representation,all items in the same line with no indentation@@ -328,7 +328,7 @@ CsvItemExporter:param join_multivalued: The char (or chars) that will be used for joiningmulti-valued fields, if found.- :type include_headers_line: str+ :type join_multivalued: str:param errors: The optional string that specifies how encoding and decodingerrors are to be handled. For more information see@@ -342,14 +342,14 @@ CsvItemExporterA typical output of this exporter would be::- product,price+ name,priceColor TV,1200DVD player,200PickleItemExporter-------------------.. class:: PickleItemExporter(file, protocol=0, **kwargs)+.. class:: PickleItemExporter(file, protocol=4, **kwargs)Exports items in pickle format to the given file-like object.diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex 5a05f67d9..5598ab983 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -341,9 +341,6 @@ closing the spider. If the spider generates more than that number of errors,it will be closed with the reason ``closespider_errorcount``. If zero (or nonset), spiders won't be closed by number of errors.-.. module:: scrapy.extensions.debug- :synopsis: Extensions for debugging Scrapy-.. module:: scrapy.extensions.periodic_log:synopsis: Periodic stats logging@@ -418,7 +415,7 @@ Example extension configuration:custom_settings = {"LOG_LEVEL": "INFO","PERIODIC_LOG_STATS": {- "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],+ "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count"],},"PERIODIC_LOG_DELTA": {"include": ["downloader/"]},"PERIODIC_LOG_TIMING_ENABLED": True,@@ -463,6 +460,9 @@ Default: ``False``Debugging extensions--------------------+.. module:: scrapy.extensions.debug+ :synopsis: Extensions for debugging Scrapy+Stack trace dump extension~~~~~~~~~~~~~~~~~~~~~~~~~~diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex 9aa1b8d5f..db6dd127c 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -615,6 +615,7 @@ Default:"file": "scrapy.extensions.feedexport.FileFeedStorage","stdout": "scrapy.extensions.feedexport.StdoutFeedStorage","s3": "scrapy.extensions.feedexport.S3FeedStorage",+ "gs": "scrapy.extensions.feedexport.GCSFeedStorage","ftp": "scrapy.extensions.feedexport.FTPFeedStorage",}diff --git a/docs/topics/items.rst b/docs/topics/items.rstindex 7a1c1f907..b21d6b903 100644--- a/docs/topics/items.rst+++ b/docs/topics/items.rst@@ -376,7 +376,9 @@ Creating dicts from items:>>> dict(product) # create a dict from all populated values{'price': 1000, 'name': 'Desktop PC'}- Creating items from dicts:+Creating items from dicts:++.. code-block:: pycon>>> Product({"name": "Laptop PC", "price": 1500})Product(price=1500, name='Laptop PC')diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rstindex 0913a3310..70d563fdd 100644--- a/docs/topics/leaks.rst+++ b/docs/topics/leaks.rst@@ -93,7 +93,7 @@ You can get the oldest object of each class using theWhich objects are tracked?---------------------------The objects tracked by ``trackrefs`` are all from these classes (and all its+The objects tracked by ``trackref`` are all from these classes (and all itssubclasses):* :class:`scrapy.Request`@@ -187,7 +187,7 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.Inherit from this class if you want to track liveinstances with the ``trackref`` module.-.. function:: print_live_refs(class_name, ignore=NoneType)+.. function:: print_live_refs(ignore=NoneType)Print a report of live references, grouped by class name.@@ -203,9 +203,9 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module... function:: iter_all(class_name)- Return an iterator over all objects alive with the given class name, or- ``None`` if none is found. Use :func:`print_live_refs` first to get a list- of all tracked live objects per class name.+ Return an iterator over all objects alive with the given class name. Use+ :func:`print_live_refs` first to get a list of all tracked live objects+ per class name... skip: enddiff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rstindex 3ccc90bf9..801dd6583 100644--- a/docs/topics/loaders.rst+++ b/docs/topics/loaders.rst@@ -76,7 +76,7 @@ data that will be assigned to the ``name`` field later.Afterwards, similar calls are used for ``price`` and ``stock`` fields(the latter using a CSS selector with the :meth:`~ItemLoader.add_css` method),-and finally the ``last_update`` field is populated directly with a literal value+and finally the ``last_updated`` field is populated directly with a literal value(``today``) using a different method: :meth:`~ItemLoader.add_value`.Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is@@ -273,8 +273,8 @@ The precedence order, for both input and output processors, is as follows:1. Item Loader field-specific attributes: ``field_in`` and ``field_out`` (mostprecedence)2. Field metadata (``input_processor`` and ``output_processor`` key)-3. Item Loader defaults: :meth:`ItemLoader.default_input_processor` and- :meth:`ItemLoader.default_output_processor` (least precedence)+3. Item Loader defaults: :attr:`ItemLoader.default_input_processor` and+ :attr:`ItemLoader.default_output_processor` (least precedence)See also: :ref:`topics-loaders-extending`.@@ -323,8 +323,8 @@ There are several ways to modify Item Loader context values:loader = ItemLoader(product, unit="cm")3. On Item Loader declaration, for those input/output processors that support- instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of- them:+ instantiating them with an Item Loader context.+ :class:`~itemloaders.processors.MapCompose` is one of them:.. code-block:: pythondiff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex bbb5d0458..b486eab63 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -4,11 +4,6 @@Logging=======-.. note::- :mod:`scrapy.log` has been deprecated alongside its functions in favor of- explicit calls to the Python standard logging. Keep reading to learn more- about the new logging system.-Scrapy uses :mod:`logging` for event logging. We'llprovide some simple examples to get you started, but for more advanceduse-cases it's strongly suggested to read thoroughly its documentation.diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex c67d2d627..04a744b0e 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -371,11 +371,12 @@ For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/orIf you need something more complex and want to override the custom pipelinebehaviour, see :ref:`topics-media-pipeline-override`.-If you have multiple image pipelines inheriting from ImagePipeline and you want-to have different settings in different pipelines you can set setting keys-preceded with uppercase name of your pipeline class. E.g. if your pipeline is-called MyPipeline and you want to have custom IMAGES_URLS_FIELD you define-setting MYPIPELINE_IMAGES_URLS_FIELD and your custom settings will be used.+If you have multiple image pipelines inheriting from :class:`ImagesPipeline`+and you want to have different settings in different pipelines you can set+setting keys preceded with uppercase name of your pipeline class. E.g. if your+pipeline is called ``MyPipeline`` and you want to have custom+:setting:`IMAGES_URLS_FIELD` you define setting+``MYPIPELINE_IMAGES_URLS_FIELD`` and your custom settings will be used.Additional featuresdiff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 44bc09b7f..7796755af 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -247,7 +247,7 @@ Request objectsReturn a new Request which is a copy of this Request. See also::ref:`topics-request-response-ref-request-callback-arguments`.- .. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs])+ .. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs, cls])Return a Request object with the same members, except for those membersgiven new values by whichever keyword arguments are specified. The@@ -717,6 +717,7 @@ Those are:* :reqmeta:`download_fail_on_dataloss`* :reqmeta:`download_latency`* :reqmeta:`download_maxsize`+* :reqmeta:`download_slot`* :reqmeta:`download_warnsize`* :reqmeta:`download_timeout`* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)@@ -1085,7 +1086,7 @@ Response objects.. attribute:: Response.flagsA list that contains flags for this response. Flags are labels used for- tagging Responses. For example: ``'cached'``, ``'redirected``', etc. And+ tagging Responses. For example: ``'cached'``, ``'redirected'``', etc. Andthey're shown on the string representation of the Response (``__str__()``method) which is used by the engine for logging.@@ -1119,7 +1120,7 @@ Response objectsReturns a new Response which is a copy of this Response.- .. method:: Response.replace([url, status, headers, body, request, flags, cls])+ .. method:: Response.replace([url, status, headers, body, request, flags, certificate, ip_address, protocol, cls])Returns a Response object with the same members, except for those membersgiven new values by whichever keyword arguments are specified. Thediff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rstindex 0b174341d..6d4954c2d 100644--- a/docs/topics/selectors.rst+++ b/docs/topics/selectors.rst@@ -308,7 +308,7 @@ Examples:* ``*::text`` selects all descendant text nodes of the current selector context:-..skip: next+.. skip: next.. code-block:: pycon>>> response.css("#images *::text").getall()diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex a5148ee26..6084fdb5b 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -409,6 +409,34 @@ Default: ``{}``A dict containing paths to the add-ons enabled in your project and theirpriorities. For more information, see :ref:`topics-addons`.+.. setting:: ASYNCIO_EVENT_LOOP++ASYNCIO_EVENT_LOOP+------------------++Default: ``None``++Import path of a given ``asyncio`` event loop class.++If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the+asyncio event loop to be used with it. Set the setting to the import path of the+desired asyncio event loop class. If the setting is set to ``None`` the default asyncio+event loop will be used.++If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`+function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop+class to be used.++Note that the event loop class must inherit from :class:`asyncio.AbstractEventLoop`.++.. caution:: Please be aware that, when using a non-default event loop+ (either defined via :setting:`ASYNCIO_EVENT_LOOP` or installed with+ :func:`~scrapy.utils.reactor.install_reactor`), Scrapy will call+ :func:`asyncio.set_event_loop`, which will set the specified event loop+ as the current loop for the current OS thread.++.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: AWS_ACCESS_KEY_IDAWS_ACCESS_KEY_ID@@ -419,6 +447,24 @@ Default: ``None``The AWS access key used by code that requires access to `Amazon Web services`_,such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.+.. setting:: AWS_ENDPOINT_URL++AWS_ENDPOINT_URL+----------------++Default: ``None``++Endpoint URL used for S3-like storage, for example Minio or s3.scality.++.. setting:: AWS_REGION_NAME++AWS_REGION_NAME+---------------++Default: ``None``++The name of the region associated with the AWS client.+.. setting:: AWS_SECRET_ACCESS_KEYAWS_SECRET_ACCESS_KEY@@ -442,15 +488,6 @@ such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using.. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html-.. setting:: AWS_ENDPOINT_URL--AWS_ENDPOINT_URL-------------------Default: ``None``--Endpoint URL used for S3-like storage, for example Minio or s3.scality.-.. setting:: AWS_USE_SSLAWS_USE_SSL@@ -471,43 +508,6 @@ Default: ``None``Verify SSL connection between Scrapy and S3 or S3-like storage. By defaultSSL verification will occur.-.. setting:: AWS_REGION_NAME--AWS_REGION_NAME------------------Default: ``None``--The name of the region associated with the AWS client.--.. setting:: ASYNCIO_EVENT_LOOP--ASYNCIO_EVENT_LOOP---------------------Default: ``None``--Import path of a given ``asyncio`` event loop class.--If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the-asyncio event loop to be used with it. Set the setting to the import path of the-desired asyncio event loop class. If the setting is set to ``None`` the default asyncio-event loop will be used.--If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`-function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop-class to be used.--Note that the event loop class must inherit from :class:`asyncio.AbstractEventLoop`.--.. caution:: Please be aware that, when using a non-default event loop- (either defined via :setting:`ASYNCIO_EVENT_LOOP` or installed with- :func:`~scrapy.utils.reactor.install_reactor`), Scrapy will call- :func:`asyncio.set_event_loop`, which will set the specified event loop- as the current loop for the current OS thread.--.. note:: This is a :ref:`reactor setting <reactor-settings>`.-.. setting:: BOT_NAMEBOT_NAME@@ -593,7 +593,7 @@ When writing an item pipeline, you can force a different log level by settingDEFAULT_ITEM_CLASS-------------------Default: ``'scrapy.Item'``+Default: ``'scrapy.item.Item'``The default class that will be used for instantiating items in the :ref:`theScrapy shell <topics-shell>`.@@ -687,7 +687,7 @@ Whether to enable DNS in-memory cache.:class:`~scrapy.resolver.CachingThreadedResolver` and:class:`~scrapy.resolver.CachingHostnameResolver`. It has no effect when:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect- either when :setting:`DNS_RESOLVER` is set to a different resolver.+ either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver... note:: This is a :ref:`reactor setting <reactor-settings>`.@@ -702,25 +702,6 @@ DNS in-memory cache size, see :setting:`DNSCACHE_ENABLED`... note:: This is a :ref:`reactor setting <reactor-settings>`.-.. setting:: TWISTED_DNS_RESOLVER--TWISTED_DNS_RESOLVER-----------------------Default: ``'scrapy.resolver.CachingThreadedResolver'``--The class to be used by Twisted to resolve DNS names. The default-``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for-DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4-addresses. Scrapy provides an alternative resolver,-``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not-take the :setting:`DNS_TIMEOUT` setting into account.--.. note::- This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.--.. note:: This is a :ref:`reactor setting <reactor-settings>`.-.. setting:: DNS_TIMEOUTDNS_TIMEOUT@@ -734,7 +715,7 @@ Timeout for processing of DNS queries in seconds. Float is supported.This setting is only used by:class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect- either when :setting:`DNS_RESOLVER` is set to a different resolver.+ either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver... note:: This is a :ref:`reactor setting <reactor-settings>`.@@ -1330,6 +1311,7 @@ Default:{"scrapy.extensions.corestats.CoreStats": 0,+ "scrapy.extensions.logcount.LogCount": 0,"scrapy.extensions.telnet.TelnetConsole": 0,"scrapy.extensions.memusage.MemoryUsage": 0,"scrapy.extensions.memdebug.MemoryDebugger": 0,@@ -1352,6 +1334,8 @@ and the :ref:`list of available extensions <topics-extensions-ref>`.FEED_TEMPDIR------------+Default: ``None``+The Feed Temp dir allows you to set a custom folder to save crawlertemporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and:ref:`Amazon S3 <topics-feed-storage-s3>`.@@ -1361,6 +1345,8 @@ temporary files before uploading with :ref:`FTP feed storage <topics-feed-storagFEED_STORAGE_GCS_ACL--------------------+Default: ``""``+The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.@@ -1629,6 +1615,8 @@ The following special items are also supported:- ``Python``+- ``pyOpenSSL``+.. setting:: LOGSTATS_INTERVALLOGSTATS_INTERVAL@@ -1756,7 +1744,7 @@ significant similarities in the time between their requests.The randomization policy is the same used by `wget`_ ``--random-wait`` option.-If :setting:`DOWNLOAD_DELAY` is zero (default) this option has no effect.+If :setting:`DOWNLOAD_DELAY` is zero this option has no effect... _wget: https://www.gnu.org/software/wget/manual/wget.html@@ -1817,7 +1805,7 @@ The parser backend to use for parsing ``robots.txt`` files. For more information.. setting:: ROBOTSTXT_USER_AGENTROBOTSTXT_USER_AGENT-^^^^^^^^^^^^^^^^^^^^+--------------------Default: ``None``@@ -1972,6 +1960,8 @@ Default:{"scrapy.contracts.default.UrlContract": 1,+ "scrapy.contracts.default.CallbackKeywordArgumentsContract": 1,+ "scrapy.contracts.default.MetadataContract": 1,"scrapy.contracts.default.ReturnsContract": 2,"scrapy.contracts.default.ScrapesContract": 3,}@@ -2036,6 +2026,7 @@ Default:.. code-block:: python{+ "scrapy.spidermiddlewares.start.StartSpiderMiddleware": 25,"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,@@ -2111,6 +2102,25 @@ command.The project name must not conflict with the name of custom files or directoriesin the ``project`` subdirectory.+.. setting:: TWISTED_DNS_RESOLVER++TWISTED_DNS_RESOLVER+--------------------++Default: ``'scrapy.resolver.CachingThreadedResolver'``++The class to be used by Twisted to resolve DNS names. The default+``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for+DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4+addresses. Scrapy provides an alternative resolver,+``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not+take the :setting:`DNS_TIMEOUT` setting into account.++.. note::+ This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.++.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: TWISTED_REACTOR_ENABLEDTWISTED_REACTOR_ENABLED@@ -2252,7 +2262,7 @@ URLLENGTH_LIMITDefault: ``2083``-Scope: ``spidermiddlewares.urllength``+Scope: ``scrapy.spidermiddlewares.urllength``The maximum URL length to allow for crawled URLs.diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex d13733623..bd14f5993 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -60,6 +60,8 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:.. skip: next.. code-block:: python+ import json+import scrapyimport treqdiff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex e2cd0f986..870adbc8c 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -46,7 +46,7 @@ previous (or subsequent) middleware being applied.If you want to disable a builtin middleware (the ones defined in:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define itin your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its-value. For example, if you want to disable the off-site middleware:+value. For example, if you want to disable the referer middleware:.. code-block:: pythondiff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 506daf930..bb5952826 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -208,7 +208,7 @@ scrapy.Spider:param response: the response to parse:type response: :class:`~scrapy.http.Response`- .. method:: log(message, [level, component])+ .. method:: log(message, [level])Wrapper that sends a log message through the Spider's :attr:`logger`,kept for backward compatibility. For more information see@@ -335,8 +335,8 @@ The above example can also be written as follows:If you are :ref:`running Scrapy from a script <run-from-script>`, you canspecify spider arguments when calling-:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or-:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:+:meth:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or+:meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:.. skip: next.. code-block:: python@@ -593,7 +593,7 @@ Let's now take a look at an example CrawlSpider with rules:This spider would start crawling example.com's home page, collecting categorylinks, and item links, parsing the latter with the ``parse_item`` method. Foreach item response, some data will be extracted from the HTML using XPath, and-an :class:`~scrapy.Item` will be filled with it.+a dictionary will be filled with it.XMLFeedSpider-------------@@ -953,6 +953,7 @@ Combine SitemapSpider with other sources of urls:.. code-block:: python+ from scrapy import Requestfrom scrapy.spiders import SitemapSpiderdiff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rstindex a30258112..72188c03f 100644--- a/docs/topics/telnetconsole.rst+++ b/docs/topics/telnetconsole.rst@@ -36,9 +36,9 @@ disable it if you want. For more information about the extension itself seeHow to access the telnet console================================-The telnet console listens in the TCP port defined in the-:setting:`TELNETCONSOLE_PORT` setting, which defaults to ``6023``. To access-the console you need to type::+The telnet console listens on the first available TCP port from the range+defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to+``[6023, 6073]``. To access the console you need to type::telnet localhost 6023Trying localhost...diff --git a/docs/versioning.rst b/docs/versioning.rstindex 0c6498244..e44496799 100644--- a/docs/versioning.rst+++ b/docs/versioning.rst@@ -39,8 +39,8 @@ API stabilityAPI stability was one of the major goals for the *1.0* release.-Methods or functions that start with a single dash (``_``) are private and-should never be relied as stable.+Methods or functions that start with a single underscore (``_``) are private+and should never be relied upon as stable.Also, keep in mind that stable doesn't mean complete: stable APIs could grownew methods or functionality but the existing methods should keep working thediff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.pyindex 049921ba7..3a7dfd018 100644--- a/scrapy/settings/default_settings.py+++ b/scrapy/settings/default_settings.py@@ -240,7 +240,7 @@ BOT_NAME = "scrapybot"CLOSESPIDER_ERRORCOUNT = 0CLOSESPIDER_ITEMCOUNT = 0CLOSESPIDER_PAGECOUNT = 0-CLOSESPIDER_TIMEOUT = 0+CLOSESPIDER_TIMEOUT = 0.0CLOSESPIDER_PAGECOUNT_NO_ITEM = 0CLOSESPIDER_TIMEOUT_NO_ITEM = 0