Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Assorted docs fixes, part 1. (#7710)

Andrey Rakhmatullinbdf3067935fd
diff --git a/docs/README.rst b/docs/README.rst
deleted file mode 100644
index 4d2236b53..000000000
--- a/docs/README.rst
+++ /dev/null
@@ -1,68 +0,0 @@
-:orphan:
-
-======================================
-Scrapy documentation quick start guide
-======================================
-
-This file provides a quick guide on how to compile the Scrapy documentation.
-
-
-Setup the environment
----------------------
-
-To compile the documentation you need Sphinx Python library. To install it
-and all its dependencies run the following command from this dir
-
-::
-
- pip install -r requirements.txt
-
-
-Compile the documentation
--------------------------
-
-To compile the documentation (to classic HTML output) run the following command
-from this dir::
-
- make html
-
-Documentation will be generated (in HTML format) inside the ``build/html`` dir.
-
-
-View the documentation
-----------------------
-
-To view the documentation run the following command::
-
- make htmlview
-
-This command will fire up your default browser and open the main page of your
-(previously generated) HTML documentation.
-
-
-Start over
-----------
-
-To clean up all generated documentation files and start from scratch run::
-
- make clean
-
-Keep in mind that this command won't touch any documentation source files.
-
-
-Recreating documentation on the fly
------------------------------------
-
-There is a way to recreate the doc automatically when you make changes, you
-need to install watchdog (``pip install watchdog``) and then use::
-
- make watch
-
-Alternative method using tox
-----------------------------
-
-To compile the documentation to HTML run the following command::
-
- tox -e docs
-
-Documentation will be generated inside the ``docs/_build/all`` dir.
diff --git a/docs/contributing.rst b/docs/contributing.rst
index c868a0ac4..b43d30796 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -323,9 +323,10 @@ deprecation removals are documented in the :ref:`release notes <news>`.
Tests
=====
-Tests are implemented using the :doc:`Twisted unit-testing framework
-<twisted:development/test-standard>`. Running tests requires
-:doc:`tox <tox:index>`.
+Tests are implemented using pytest_. Running tests requires :doc:`tox
+<tox:index>`.
+
+.. _pytest: https://pytest.org
.. _running-tests:
diff --git a/docs/faq.rst b/docs/faq.rst
index 6eeb81a31..a396a8212 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -285,7 +285,8 @@ consume a lot of memory.
In order to avoid parsing all the entire feed at once in memory, you can use
the :func:`~scrapy.utils.iterators.xmliter_lxml` and
:func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what
-:class:`~scrapy.spiders.XMLFeedSpider` uses.
+:class:`~scrapy.spiders.XMLFeedSpider` and
+:class:`~scrapy.spiders.CSVFeedSpider` use.
.. autofunction:: scrapy.utils.iterators.xmliter_lxml
@@ -360,10 +361,11 @@ method for this purpose. For example:
def process_spider_output(self, response, result):
for item_or_request in result:
if isinstance(item_or_request, Request):
+ yield item_or_request
continue
- adapter = ItemAdapter(item)
+ adapter = ItemAdapter(item_or_request)
for _ in range(adapter["multiply_by"]):
- yield deepcopy(item)
+ yield deepcopy(item_or_request)
Does Scrapy support IPv6 addresses?
-----------------------------------
diff --git a/docs/index.rst b/docs/index.rst
index 8e8624a22..688cab81b 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -24,7 +24,7 @@ Having trouble? We'd like to help!
* Ask or search questions in `StackOverflow using the scrapy tag`_.
* Ask or search questions in the `Scrapy subreddit`_.
* Search for questions on the archives of the `scrapy-users mailing list`_.
-* Ask a question in the `#scrapy IRC channel`_,
+* Ask a question in the `#scrapy IRC channel`_.
* Report bugs with Scrapy in our `issue tracker`_.
* Join the Discord community `Scrapy Discord`_.
@@ -91,15 +91,15 @@ Basic concepts
:doc:`topics/selectors`
Extract the data from web pages using XPath.
-:doc:`topics/shell`
- Test your extraction code in an interactive environment.
-
:doc:`topics/items`
Define the data you want to scrape.
:doc:`topics/loaders`
Populate your items with the extracted data.
+:doc:`topics/shell`
+ Test your extraction code in an interactive environment.
+
:doc:`topics/item-pipeline`
Post-process and store your scraped data.
diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst
index ee91ce7ca..488e1c2e3 100644
--- a/docs/intro/overview.rst
+++ b/docs/intro/overview.rst
@@ -90,9 +90,10 @@ to figure these settings out automatically.
.. note::
This is using :ref:`feed exports <topics-feed-exports>` to generate the
- JSON file, you can easily change the export format (XML or CSV, for example) or the
- storage backend (FTP or `Amazon S3`_, for example). You can also write an
- :ref:`item pipeline <topics-item-pipeline>` to store the items in a database.
+ JSON Lines file, you can easily change the export format (XML or CSV, for
+ example) or the storage backend (FTP or `Amazon S3`_, for example). You can
+ also write an :ref:`item pipeline <topics-item-pipeline>` to store the
+ items in a database.
.. _topics-whatelse:
diff --git a/docs/news.rst b/docs/news.rst
index 7d2b07b09..63f0d69e4 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -1035,7 +1035,7 @@ Deprecations
New features
~~~~~~~~~~~~
-- Added a new setting, :setting:`REFERER_POLICIES`, to allow customizing
+- Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizing
supported referrer policies.
Bug fixes
@@ -6046,7 +6046,7 @@ The following changes may impact custom priority queue classes:
* A new keyword parameter has been added: ``key``. It is a string
that is always an empty string for memory queues and indicates the
- :setting:`JOB_DIR` value for disk queues.
+ :setting:`JOBDIR` value for disk queues.
* The parameter for disk queues that contains data from the previous
crawl, ``startprios`` or ``slot_startprios``, is now passed as a
@@ -6600,7 +6600,7 @@ New features
``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be
:ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant
scheduling improvement on crawls targeting multiple web domains, at the
- cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)
+ cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:issue:`3520`)
* A new :attr:`.Request.cb_kwargs` attribute
provides a cleaner way to pass keyword arguments to callback methods
@@ -9079,7 +9079,7 @@ New features and settings
- In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`)
- Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`)
- ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by:
- - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, :setting:`CONCURRENT_REQUESTS_PER_IP`
+ - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP``
- check the documentation for more details
- Added builtin caching DNS resolver (:rev:`2728`)
- Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`)
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index 1a151044f..17b322b5a 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -105,6 +105,9 @@ Scrapy API requires passing a Deferred to it) using the following helpers:
.. autofunction:: scrapy.utils.defer.deferred_from_coro
.. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f
+
+The following function helps with a reverse wrapping:
+
.. autofunction:: scrapy.utils.defer.ensure_awaitable
@@ -190,7 +193,7 @@ in future Scrapy versions. The following features are not available:
:class:`~scrapy.crawler.CrawlerProcess`
(:class:`~scrapy.crawler.AsyncCrawlerProcess` and
:class:`~scrapy.crawler.AsyncCrawlerRunner` are available)
-* Twisted-specific DNS resolvers (the :setting:`DNS_RESOLVER` setting)
+* Twisted-specific DNS resolvers (the :setting:`TWISTED_DNS_RESOLVER` setting)
* User and 3rd-party code that requires a reactor (see :ref:`below
<asyncio-without-reactor-migrate>` for examples)
@@ -315,8 +318,7 @@ implementations, :class:`~asyncio.ProactorEventLoop` (default) and
:class:`~asyncio.SelectorEventLoop` works with Twisted.
Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop`
-automatically when you change the :setting:`TWISTED_REACTOR` setting or call
-:func:`~scrapy.utils.reactor.install_reactor`.
+automatically when installing the asyncio reactor.
.. note:: Other libraries you use may require
:class:`~asyncio.ProactorEventLoop`, e.g. because it supports
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index 8d1351eb9..270d9859d 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -199,6 +199,7 @@ Global commands:
* :command:`fetch`
* :command:`view`
* :command:`version`
+* :command:`bench`
Project-only commands:
@@ -207,7 +208,6 @@ Project-only commands:
* :command:`list`
* :command:`edit`
* :command:`parse`
-* :command:`bench`
.. command:: startproject
@@ -476,7 +476,7 @@ Supported options:
* ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider
-* ``--a NAME=VALUE``: set spider argument (may be repeated)
+* ``-a NAME=VALUE``: set spider argument (may be repeated)
* ``--callback`` or ``-c``: spider method to use as callback for parsing the
response
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 116aac323..8aaf0aa31 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -268,5 +268,5 @@ You can also send multiple requests in parallel:
yield {
"h1": response.css("h1::text").get(),
"price": responses[0].css(".price::text").get(),
- "price2": responses[1].css(".color::text").get(),
+ "color": responses[1].css(".color::text").get(),
}
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 054c1130c..383f331b5 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -505,7 +505,7 @@ Filesystem storage backend (default)
* ``response_body`` - the plain response body
- * ``response_headers`` - the request headers (in raw HTTP format)
+ * ``response_headers`` - the response headers (in raw HTTP format)
* ``meta`` - some metadata of this cache resource in Python ``repr()``
format (grep-friendly format)
@@ -547,7 +547,7 @@ defines the methods described below.
.. method:: open_spider(spider)
This method gets called after a spider has been opened for crawling. It handles
- the :signal:`open_spider <spider_opened>` signal.
+ the :signal:`spider_opened` signal.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.Spider` object
@@ -555,7 +555,7 @@ defines the methods described below.
.. method:: close_spider(spider)
This method gets called after a spider has been closed. It handles
- the :signal:`close_spider <spider_closed>` signal.
+ the :signal:`spider_closed` signal.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.Spider` object
@@ -814,7 +814,6 @@ HttpProxyMiddleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. setting:: HTTPPROXY_ENABLED
-.. setting:: HTTPPROXY_AUTH_ENCODING
HTTPPROXY_ENABLED
^^^^^^^^^^^^^^^^^
@@ -823,6 +822,8 @@ Default: ``True``
Whether or not to enable the :class:`HttpProxyMiddleware`.
+.. setting:: HTTPPROXY_AUTH_ENCODING
+
HTTPPROXY_AUTH_ENCODING
^^^^^^^^^^^^^^^^^^^^^^^
@@ -984,7 +985,7 @@ Whether the Meta Refresh middleware will be enabled.
METAREFRESH_IGNORE_TAGS
^^^^^^^^^^^^^^^^^^^^^^^
-Default: ``[]``
+Default: ``["noscript"]``
Meta tags within these tags are ignored.
@@ -1091,7 +1092,7 @@ Default::
'twisted.internet.error.ConnectionDone',
'twisted.internet.error.ConnectError',
'twisted.internet.error.ConnectionLost',
- IOError,
+ OSError,
'scrapy.core.downloader.handlers.http11.TunnelError',
]
diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst
index 5a399e094..30b6536c7 100644
--- a/docs/topics/dynamic-content.rst
+++ b/docs/topics/dynamic-content.rst
@@ -133,7 +133,7 @@ data from it depends on the type of response:
.. code-block:: python
- selector = Selector(data["html"])
+ selector = Selector(text=data["html"])
- If the response is JavaScript, or HTML with a ``<script/>`` element
containing the desired data, see :ref:`topics-parsing-javascript`.
diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst
index cc690d609..4c56a0f14 100644
--- a/docs/topics/exceptions.rst
+++ b/docs/topics/exceptions.rst
@@ -31,7 +31,7 @@ For example:
.. code-block:: python
def parse_page(self, response):
- if "Bandwidth exceeded" in response.body:
+ if "Bandwidth exceeded" in response.text:
raise CloseSpider("bandwidth_exceeded")
DontCloseSpider
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index c4cd05683..f0b704b9d 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -153,7 +153,7 @@ output examples, which assume you're exporting these two items:
BaseItemExporter
----------------
-.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding='utf-8', indent=0, dont_fail=False)
+.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding=None, indent=None, dont_fail=False)
This is the (abstract) base class for all Item Exporters. It provides
support for common features used by all (concrete) Item Exporters, such as
@@ -239,7 +239,7 @@ BaseItemExporter
.. attribute:: indent
- Amount of spaces used to indent the output on each level. Defaults to ``0``.
+ Amount of spaces used to indent the output on each level. Defaults to ``None``.
* ``indent=None`` selects the most compact representation,
all items in the same line with no indentation
@@ -328,7 +328,7 @@ CsvItemExporter
:param join_multivalued: The char (or chars) that will be used for joining
multi-valued fields, if found.
- :type include_headers_line: str
+ :type join_multivalued: str
:param errors: The optional string that specifies how encoding and decoding
errors are to be handled. For more information see
@@ -342,14 +342,14 @@ CsvItemExporter
A typical output of this exporter would be::
- product,price
+ name,price
Color TV,1200
DVD player,200
PickleItemExporter
------------------
-.. class:: PickleItemExporter(file, protocol=0, **kwargs)
+.. class:: PickleItemExporter(file, protocol=4, **kwargs)
Exports items in pickle format to the given file-like object.
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 5a05f67d9..5598ab983 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -341,9 +341,6 @@ closing the spider. If the spider generates more than that number of errors,
it will be closed with the reason ``closespider_errorcount``. If zero (or non
set), spiders won't be closed by number of errors.
-.. module:: scrapy.extensions.debug
- :synopsis: Extensions for debugging Scrapy
-
.. module:: scrapy.extensions.periodic_log
:synopsis: Periodic stats logging
@@ -418,7 +415,7 @@ Example extension configuration:
custom_settings = {
"LOG_LEVEL": "INFO",
"PERIODIC_LOG_STATS": {
- "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
+ "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count"],
},
"PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
"PERIODIC_LOG_TIMING_ENABLED": True,
@@ -463,6 +460,9 @@ Default: ``False``
Debugging extensions
--------------------
+.. module:: scrapy.extensions.debug
+ :synopsis: Extensions for debugging Scrapy
+
Stack trace dump extension
~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index 9aa1b8d5f..db6dd127c 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -615,6 +615,7 @@ Default:
"file": "scrapy.extensions.feedexport.FileFeedStorage",
"stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
"s3": "scrapy.extensions.feedexport.S3FeedStorage",
+ "gs": "scrapy.extensions.feedexport.GCSFeedStorage",
"ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
}
diff --git a/docs/topics/items.rst b/docs/topics/items.rst
index 7a1c1f907..b21d6b903 100644
--- a/docs/topics/items.rst
+++ b/docs/topics/items.rst
@@ -376,7 +376,9 @@ Creating dicts from items:
>>> dict(product) # create a dict from all populated values
{'price': 1000, 'name': 'Desktop PC'}
- Creating items from dicts:
+Creating items from dicts:
+
+.. code-block:: pycon
>>> Product({"name": "Laptop PC", "price": 1500})
Product(price=1500, name='Laptop PC')
diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst
index 0913a3310..70d563fdd 100644
--- a/docs/topics/leaks.rst
+++ b/docs/topics/leaks.rst
@@ -93,7 +93,7 @@ You can get the oldest object of each class using the
Which objects are tracked?
--------------------------
-The objects tracked by ``trackrefs`` are all from these classes (and all its
+The objects tracked by ``trackref`` are all from these classes (and all its
subclasses):
* :class:`scrapy.Request`
@@ -187,7 +187,7 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
Inherit from this class if you want to track live
instances with the ``trackref`` module.
-.. function:: print_live_refs(class_name, ignore=NoneType)
+.. function:: print_live_refs(ignore=NoneType)
Print a report of live references, grouped by class name.
@@ -203,9 +203,9 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
.. function:: iter_all(class_name)
- Return an iterator over all objects alive with the given class name, or
- ``None`` if none is found. Use :func:`print_live_refs` first to get a list
- of all tracked live objects per class name.
+ Return an iterator over all objects alive with the given class name. Use
+ :func:`print_live_refs` first to get a list of all tracked live objects
+ per class name.
.. skip: end
diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst
index 3ccc90bf9..801dd6583 100644
--- a/docs/topics/loaders.rst
+++ b/docs/topics/loaders.rst
@@ -76,7 +76,7 @@ data that will be assigned to the ``name`` field later.
Afterwards, similar calls are used for ``price`` and ``stock`` fields
(the latter using a CSS selector with the :meth:`~ItemLoader.add_css` method),
-and finally the ``last_update`` field is populated directly with a literal value
+and finally the ``last_updated`` field is populated directly with a literal value
(``today``) using a different method: :meth:`~ItemLoader.add_value`.
Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is
@@ -273,8 +273,8 @@ The precedence order, for both input and output processors, is as follows:
1. Item Loader field-specific attributes: ``field_in`` and ``field_out`` (most
precedence)
2. Field metadata (``input_processor`` and ``output_processor`` key)
-3. Item Loader defaults: :meth:`ItemLoader.default_input_processor` and
- :meth:`ItemLoader.default_output_processor` (least precedence)
+3. Item Loader defaults: :attr:`ItemLoader.default_input_processor` and
+ :attr:`ItemLoader.default_output_processor` (least precedence)
See also: :ref:`topics-loaders-extending`.
@@ -323,8 +323,8 @@ There are several ways to modify Item Loader context values:
loader = ItemLoader(product, unit="cm")
3. On Item Loader declaration, for those input/output processors that support
- instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of
- them:
+ instantiating them with an Item Loader context.
+ :class:`~itemloaders.processors.MapCompose` is one of them:
.. code-block:: python
diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst
index bbb5d0458..b486eab63 100644
--- a/docs/topics/logging.rst
+++ b/docs/topics/logging.rst
@@ -4,11 +4,6 @@
Logging
=======
-.. note::
- :mod:`scrapy.log` has been deprecated alongside its functions in favor of
- explicit calls to the Python standard logging. Keep reading to learn more
- about the new logging system.
-
Scrapy uses :mod:`logging` for event logging. We'll
provide some simple examples to get you started, but for more advanced
use-cases it's strongly suggested to read thoroughly its documentation.
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index c67d2d627..04a744b0e 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -371,11 +371,12 @@ For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
If you need something more complex and want to override the custom pipeline
behaviour, see :ref:`topics-media-pipeline-override`.
-If you have multiple image pipelines inheriting from ImagePipeline and you want
-to have different settings in different pipelines you can set setting keys
-preceded with uppercase name of your pipeline class. E.g. if your pipeline is
-called MyPipeline and you want to have custom IMAGES_URLS_FIELD you define
-setting MYPIPELINE_IMAGES_URLS_FIELD and your custom settings will be used.
+If you have multiple image pipelines inheriting from :class:`ImagesPipeline`
+and you want to have different settings in different pipelines you can set
+setting keys preceded with uppercase name of your pipeline class. E.g. if your
+pipeline is called ``MyPipeline`` and you want to have custom
+:setting:`IMAGES_URLS_FIELD` you define setting
+``MYPIPELINE_IMAGES_URLS_FIELD`` and your custom settings will be used.
Additional features
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 44bc09b7f..7796755af 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -247,7 +247,7 @@ Request objects
Return a new Request which is a copy of this Request. See also:
:ref:`topics-request-response-ref-request-callback-arguments`.
- .. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs])
+ .. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs, cls])
Return a Request object with the same members, except for those members
given new values by whichever keyword arguments are specified. The
@@ -717,6 +717,7 @@ Those are:
* :reqmeta:`download_fail_on_dataloss`
* :reqmeta:`download_latency`
* :reqmeta:`download_maxsize`
+* :reqmeta:`download_slot`
* :reqmeta:`download_warnsize`
* :reqmeta:`download_timeout`
* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
@@ -1085,7 +1086,7 @@ Response objects
.. attribute:: Response.flags
A list that contains flags for this response. Flags are labels used for
- tagging Responses. For example: ``'cached'``, ``'redirected``', etc. And
+ tagging Responses. For example: ``'cached'``, ``'redirected'``', etc. And
they're shown on the string representation of the Response (``__str__()``
method) which is used by the engine for logging.
@@ -1119,7 +1120,7 @@ Response objects
Returns a new Response which is a copy of this Response.
- .. method:: Response.replace([url, status, headers, body, request, flags, cls])
+ .. method:: Response.replace([url, status, headers, body, request, flags, certificate, ip_address, protocol, cls])
Returns a Response object with the same members, except for those members
given new values by whichever keyword arguments are specified. The
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 0b174341d..6d4954c2d 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -308,7 +308,7 @@ Examples:
* ``*::text`` selects all descendant text nodes of the current selector context:
-..skip: next
+.. skip: next
.. code-block:: pycon
>>> response.css("#images *::text").getall()
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index a5148ee26..6084fdb5b 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -409,6 +409,34 @@ Default: ``{}``
A dict containing paths to the add-ons enabled in your project and their
priorities. For more information, see :ref:`topics-addons`.
+.. setting:: ASYNCIO_EVENT_LOOP
+
+ASYNCIO_EVENT_LOOP
+------------------
+
+Default: ``None``
+
+Import path of a given ``asyncio`` event loop class.
+
+If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the
+asyncio event loop to be used with it. Set the setting to the import path of the
+desired asyncio event loop class. If the setting is set to ``None`` the default asyncio
+event loop will be used.
+
+If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`
+function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop
+class to be used.
+
+Note that the event loop class must inherit from :class:`asyncio.AbstractEventLoop`.
+
+.. caution:: Please be aware that, when using a non-default event loop
+ (either defined via :setting:`ASYNCIO_EVENT_LOOP` or installed with
+ :func:`~scrapy.utils.reactor.install_reactor`), Scrapy will call
+ :func:`asyncio.set_event_loop`, which will set the specified event loop
+ as the current loop for the current OS thread.
+
+.. note:: This is a :ref:`reactor setting <reactor-settings>`.
+
.. setting:: AWS_ACCESS_KEY_ID
AWS_ACCESS_KEY_ID
@@ -419,6 +447,24 @@ Default: ``None``
The AWS access key used by code that requires access to `Amazon Web services`_,
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
+.. setting:: AWS_ENDPOINT_URL
+
+AWS_ENDPOINT_URL
+----------------
+
+Default: ``None``
+
+Endpoint URL used for S3-like storage, for example Minio or s3.scality.
+
+.. setting:: AWS_REGION_NAME
+
+AWS_REGION_NAME
+---------------
+
+Default: ``None``
+
+The name of the region associated with the AWS client.
+
.. setting:: AWS_SECRET_ACCESS_KEY
AWS_SECRET_ACCESS_KEY
@@ -442,15 +488,6 @@ such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using
.. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
-.. setting:: AWS_ENDPOINT_URL
-
-AWS_ENDPOINT_URL
-----------------
-
-Default: ``None``
-
-Endpoint URL used for S3-like storage, for example Minio or s3.scality.
-
.. setting:: AWS_USE_SSL
AWS_USE_SSL
@@ -471,43 +508,6 @@ Default: ``None``
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
SSL verification will occur.
-.. setting:: AWS_REGION_NAME
-
-AWS_REGION_NAME
----------------
-
-Default: ``None``
-
-The name of the region associated with the AWS client.
-
-.. setting:: ASYNCIO_EVENT_LOOP
-
-ASYNCIO_EVENT_LOOP
-------------------
-
-Default: ``None``
-
-Import path of a given ``asyncio`` event loop class.
-
-If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the
-asyncio event loop to be used with it. Set the setting to the import path of the
-desired asyncio event loop class. If the setting is set to ``None`` the default asyncio
-event loop will be used.
-
-If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`
-function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop
-class to be used.
-
-Note that the event loop class must inherit from :class:`asyncio.AbstractEventLoop`.
-
-.. caution:: Please be aware that, when using a non-default event loop
- (either defined via :setting:`ASYNCIO_EVENT_LOOP` or installed with
- :func:`~scrapy.utils.reactor.install_reactor`), Scrapy will call
- :func:`asyncio.set_event_loop`, which will set the specified event loop
- as the current loop for the current OS thread.
-
-.. note:: This is a :ref:`reactor setting <reactor-settings>`.
-
.. setting:: BOT_NAME
BOT_NAME
@@ -593,7 +593,7 @@ When writing an item pipeline, you can force a different log level by setting
DEFAULT_ITEM_CLASS
------------------
-Default: ``'scrapy.Item'``
+Default: ``'scrapy.item.Item'``
The default class that will be used for instantiating items in the :ref:`the
Scrapy shell <topics-shell>`.
@@ -687,7 +687,7 @@ Whether to enable DNS in-memory cache.
:class:`~scrapy.resolver.CachingThreadedResolver` and
:class:`~scrapy.resolver.CachingHostnameResolver`. It has no effect when
:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect
- either when :setting:`DNS_RESOLVER` is set to a different resolver.
+ either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver.
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
@@ -702,25 +702,6 @@ DNS in-memory cache size, see :setting:`DNSCACHE_ENABLED`.
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
-.. setting:: TWISTED_DNS_RESOLVER
-
-TWISTED_DNS_RESOLVER
---------------------
-
-Default: ``'scrapy.resolver.CachingThreadedResolver'``
-
-The class to be used by Twisted to resolve DNS names. The default
-``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for
-DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4
-addresses. Scrapy provides an alternative resolver,
-``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not
-take the :setting:`DNS_TIMEOUT` setting into account.
-
-.. note::
- This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
-
-.. note:: This is a :ref:`reactor setting <reactor-settings>`.
-
.. setting:: DNS_TIMEOUT
DNS_TIMEOUT
@@ -734,7 +715,7 @@ Timeout for processing of DNS queries in seconds. Float is supported.
This setting is only used by
:class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when
:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect
- either when :setting:`DNS_RESOLVER` is set to a different resolver.
+ either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver.
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
@@ -1330,6 +1311,7 @@ Default:
{
"scrapy.extensions.corestats.CoreStats": 0,
+ "scrapy.extensions.logcount.LogCount": 0,
"scrapy.extensions.telnet.TelnetConsole": 0,
"scrapy.extensions.memusage.MemoryUsage": 0,
"scrapy.extensions.memdebug.MemoryDebugger": 0,
@@ -1352,6 +1334,8 @@ and the :ref:`list of available extensions <topics-extensions-ref>`.
FEED_TEMPDIR
------------
+Default: ``None``
+
The Feed Temp dir allows you to set a custom folder to save crawler
temporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and
:ref:`Amazon S3 <topics-feed-storage-s3>`.
@@ -1361,6 +1345,8 @@ temporary files before uploading with :ref:`FTP feed storage <topics-feed-storag
FEED_STORAGE_GCS_ACL
--------------------
+Default: ``""``
+
The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.
@@ -1629,6 +1615,8 @@ The following special items are also supported:
- ``Python``
+- ``pyOpenSSL``
+
.. setting:: LOGSTATS_INTERVAL
LOGSTATS_INTERVAL
@@ -1756,7 +1744,7 @@ significant similarities in the time between their requests.
The randomization policy is the same used by `wget`_ ``--random-wait`` option.
-If :setting:`DOWNLOAD_DELAY` is zero (default) this option has no effect.
+If :setting:`DOWNLOAD_DELAY` is zero this option has no effect.
.. _wget: https://www.gnu.org/software/wget/manual/wget.html
@@ -1817,7 +1805,7 @@ The parser backend to use for parsing ``robots.txt`` files. For more information
.. setting:: ROBOTSTXT_USER_AGENT
ROBOTSTXT_USER_AGENT
-^^^^^^^^^^^^^^^^^^^^
+--------------------
Default: ``None``
@@ -1972,6 +1960,8 @@ Default:
{
"scrapy.contracts.default.UrlContract": 1,
+ "scrapy.contracts.default.CallbackKeywordArgumentsContract": 1,
+ "scrapy.contracts.default.MetadataContract": 1,
"scrapy.contracts.default.ReturnsContract": 2,
"scrapy.contracts.default.ScrapesContract": 3,
}
@@ -2036,6 +2026,7 @@ Default:
.. code-block:: python
{
+ "scrapy.spidermiddlewares.start.StartSpiderMiddleware": 25,
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
@@ -2111,6 +2102,25 @@ command.
The project name must not conflict with the name of custom files or directories
in the ``project`` subdirectory.
+.. setting:: TWISTED_DNS_RESOLVER
+
+TWISTED_DNS_RESOLVER
+--------------------
+
+Default: ``'scrapy.resolver.CachingThreadedResolver'``
+
+The class to be used by Twisted to resolve DNS names. The default
+``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for
+DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4
+addresses. Scrapy provides an alternative resolver,
+``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not
+take the :setting:`DNS_TIMEOUT` setting into account.
+
+.. note::
+ This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
+
+.. note:: This is a :ref:`reactor setting <reactor-settings>`.
+
.. setting:: TWISTED_REACTOR_ENABLED
TWISTED_REACTOR_ENABLED
@@ -2252,7 +2262,7 @@ URLLENGTH_LIMIT
Default: ``2083``
-Scope: ``spidermiddlewares.urllength``
+Scope: ``scrapy.spidermiddlewares.urllength``
The maximum URL length to allow for crawled URLs.
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index d13733623..bd14f5993 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -60,6 +60,8 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
.. skip: next
.. code-block:: python
+ import json
+
import scrapy
import treq
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index e2cd0f986..870adbc8c 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -46,7 +46,7 @@ previous (or subsequent) middleware being applied.
If you want to disable a builtin middleware (the ones defined in
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
-value. For example, if you want to disable the off-site middleware:
+value. For example, if you want to disable the referer middleware:
.. code-block:: python
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 506daf930..bb5952826 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -208,7 +208,7 @@ scrapy.Spider
:param response: the response to parse
:type response: :class:`~scrapy.http.Response`
- .. method:: log(message, [level, component])
+ .. method:: log(message, [level])
Wrapper that sends a log message through the Spider's :attr:`logger`,
kept for backward compatibility. For more information see
@@ -335,8 +335,8 @@ The above example can also be written as follows:
If you are :ref:`running Scrapy from a script <run-from-script>`, you can
specify spider arguments when calling
-:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
-:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
+:meth:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
+:meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
.. skip: next
.. code-block:: python
@@ -593,7 +593,7 @@ Let's now take a look at an example CrawlSpider with rules:
This spider would start crawling example.com's home page, collecting category
links, and item links, parsing the latter with the ``parse_item`` method. For
each item response, some data will be extracted from the HTML using XPath, and
-an :class:`~scrapy.Item` will be filled with it.
+a dictionary will be filled with it.
XMLFeedSpider
-------------
@@ -953,6 +953,7 @@ Combine SitemapSpider with other sources of urls:
.. code-block:: python
+ from scrapy import Request
from scrapy.spiders import SitemapSpider
diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst
index a30258112..72188c03f 100644
--- a/docs/topics/telnetconsole.rst
+++ b/docs/topics/telnetconsole.rst
@@ -36,9 +36,9 @@ disable it if you want. For more information about the extension itself see
How to access the telnet console
================================
-The telnet console listens in the TCP port defined in the
-:setting:`TELNETCONSOLE_PORT` setting, which defaults to ``6023``. To access
-the console you need to type::
+The telnet console listens on the first available TCP port from the range
+defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to
+``[6023, 6073]``. To access the console you need to type::
telnet localhost 6023
Trying localhost...
diff --git a/docs/versioning.rst b/docs/versioning.rst
index 0c6498244..e44496799 100644
--- a/docs/versioning.rst
+++ b/docs/versioning.rst
@@ -39,8 +39,8 @@ API stability
API stability was one of the major goals for the *1.0* release.
-Methods or functions that start with a single dash (``_``) are private and
-should never be relied as stable.
+Methods or functions that start with a single underscore (``_``) are private
+and should never be relied upon as stable.
Also, keep in mind that stable doesn't mean complete: stable APIs could grow
new methods or functionality but the existing methods should keep working the
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 049921ba7..3a7dfd018 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -240,7 +240,7 @@ BOT_NAME = "scrapybot"
CLOSESPIDER_ERRORCOUNT = 0
CLOSESPIDER_ITEMCOUNT = 0
CLOSESPIDER_PAGECOUNT = 0
-CLOSESPIDER_TIMEOUT = 0
+CLOSESPIDER_TIMEOUT = 0.0
CLOSESPIDER_PAGECOUNT_NO_ITEM = 0
CLOSESPIDER_TIMEOUT_NO_ITEM = 0