Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Remove docs mentioning deprecated spider attrs. (#7176)

Andrey Rakhmatullin588f3d4f65e2
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index 4994fe1d6..b8bf41402 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -163,8 +163,8 @@ information on which commands must be run from inside projects, and which not.
Also keep in mind that some commands may have slightly different behaviours
when running them from inside projects. For example, the fetch command will use
-spider-overridden behaviours (such as the ``user_agent`` attribute to override
-the user-agent) if the url being fetched is associated with some specific
+spider-overridden behaviours (such as the ``custom_settings`` attribute to
+override settings) if the url being fetched is associated with some specific
spider. This is intentional, as the ``fetch`` command is meant to be used to
check how spiders are downloading pages.
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 6d24482ec..44c7ac17b 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -290,13 +290,12 @@ DownloadTimeoutMiddleware
.. class:: DownloadTimeoutMiddleware
This middleware sets the download timeout for requests specified in the
- :setting:`DOWNLOAD_TIMEOUT` setting or :attr:`download_timeout`
- spider attribute.
+ :setting:`DOWNLOAD_TIMEOUT` setting.
.. note::
- You can also set download timeout per-request using
- :reqmeta:`download_timeout` Request.meta key; this is supported
+ You can also set download timeout per-request using the
+ :reqmeta:`download_timeout` :attr:`.Request.meta` key; this is supported
even when DownloadTimeoutMiddleware is disabled.
HttpAuthMiddleware
@@ -1212,9 +1211,8 @@ UserAgentMiddleware
.. class:: UserAgentMiddleware
- Middleware that allows spiders to override the default user agent.
+ Middleware that sets the ``User-Agent`` header.
- In order for a spider to override the default user agent, its ``user_agent``
- attribute must be set.
+ The header value is taken from the :setting:`USER_AGENT` setting.
.. _DBM: https://en.wikipedia.org/wiki/Dbm
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 48ba573d7..64e5af250 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1007,9 +1007,8 @@ The amount of time (in secs) that the downloader will wait before timing out.
.. note::
- This timeout can be set per spider using :attr:`download_timeout`
- spider attribute and per-request using :reqmeta:`download_timeout`
- Request.meta key.
+ This timeout can be per-request using the :reqmeta:`download_timeout`
+ :attr:`.Request.meta` key.
.. setting:: DOWNLOAD_MAXSIZE
.. reqmeta:: download_maxsize
@@ -1028,9 +1027,10 @@ ignored.
Use ``0`` to disable this limit.
-This limit can be set per spider using the :attr:`download_maxsize` spider
-attribute and per request using the :reqmeta:`download_maxsize` Request.meta
-key.
+.. note::
+
+ This limit can be set per-request using the :reqmeta:`download_maxsize`
+ :attr:`.Request.meta` key.
.. setting:: DOWNLOAD_WARNSIZE
.. reqmeta:: download_warnsize
@@ -1045,9 +1045,10 @@ warning will be logged about it.
Use ``0`` to disable this limit.
-This limit can be set per spider using the :attr:`download_warnsize` spider
-attribute and per request using the :reqmeta:`download_warnsize` Request.meta
-key.
+.. note::
+
+ This limit can be set per-request using the :reqmeta:`download_warnsize`
+ :attr:`.Request.meta` key.
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 0a83b248e..b4dfb3757 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -355,11 +355,9 @@ Otherwise, you would cause iteration over a ``start_urls`` string
resulting in each character being seen as a separate url.
A valid use case is to set the http auth credentials
-used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
-or the user agent
-used by :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`::
+used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`::
- scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot
+ scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword
Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
See `Scrapyd documentation`_.