Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Advise setting an identifying user agent (#7890)

Adriana6c017c2ca2b
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index eaf492c95..efade47e6 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -72,6 +72,11 @@ This will create a ``tutorial`` directory with the following contents::
spiders/ # a directory where you'll later put your spiders
__init__.py
+Before crawling anything, open ``settings.py`` and uncomment the
+:setting:`USER_AGENT` line to identify yourself, e.g. a project name plus a URL
+or an email address. Website owners who take issue with your crawler can then
+ask you to adjust it, rather than block it.
+
Our first Spider
================
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index aeaf322c2..971bb9106 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -530,32 +530,41 @@ modules by separating them with commas.
Avoiding getting banned
=======================
-Some websites implement certain measures to prevent bots from crawling them,
-with varying degrees of sophistication. Getting around those measures can be
-difficult and tricky, and may sometimes require special infrastructure. Please
-consider contacting `commercial support`_ if in doubt.
-
-Here are some tips to keep in mind when dealing with these kinds of sites:
-
-* rotate your user agent from a pool of well-known ones from browsers (Google
- around to get a list of them)
-* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
- cookies to spot bot behaviour
-* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
-* if possible, use `Common Crawl`_ to fetch pages, instead of hitting the sites
- directly
-* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
+Websites tell regular visitors and crawlers apart by how their traffic looks:
+the headers it carries, how fast it arrives, how many requests come from the
+same place. Traffic that stands out can be blocked even when the crawling
+itself would be welcome.
+
+Where the website allows crawling, the most effective thing you can do is make
+yourself known: set :setting:`USER_AGENT` to a value that identifies you and
+lets its owners reach you, so that they can ask you to adjust your crawler
+rather than block it.
+
+Where that is not enough, the following make your traffic resemble that of a
+regular visitor:
+
+* rotate your user agent among those of common browsers, so that your requests
+ do not all look alike (search the web for an up-to-date list)
+* disable cookies (see :setting:`COOKIES_ENABLED`), so that a session
+ identifier does not tie all your requests together
+* space out your requests, 2 seconds apart or more, with the
+ :setting:`DOWNLOAD_DELAY` setting, to keep your pace closer to that of a
+ person browsing
+* where possible, read pages from `Common Crawl`_, which sends no traffic to
+ the website at all
+* spread your requests over a pool of IP addresses, so that none of them
+ accounts for your whole crawl. For example, the free `Tor project`_ or paid
services like `ProxyMesh`_.
-* for HTTPS websites, if blocking appears related to TLS behavior, consider
- adjusting the :setting:`DOWNLOAD_TLS_MIN_VERSION` and
- :setting:`DOWNLOAD_TLS_MAX_VERSION` settings, since some websites may respond
- differently depending on the TLS method used by the client.
-* use a ban avoidance service, such as `Zyte API`_, which provides a `Scrapy
- plugin <https://github.com/scrapy-plugins/scrapy-zyte-api>`__ and additional
+* match the TLS behavior of a browser: some websites respond differently
+ depending on the TLS version of the client, which you can adjust with the
+ :setting:`DOWNLOAD_TLS_MIN_VERSION` and :setting:`DOWNLOAD_TLS_MAX_VERSION`
+ settings.
+* let a service take care of all of the above, such as `Zyte API`_, which
+ provides a `Scrapy plugin
+ <https://github.com/scrapy-plugins/scrapy-zyte-api>`__ and additional
features, like `AI web scraping <https://www.zyte.com/ai-web-scraping/>`__
-If you are still unable to prevent your bot getting banned, consider contacting
-`commercial support`_.
+If your crawler still gets blocked, consider contacting `commercial support`_.
.. _static-analysis:
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index dc09fdeeb..ec222d999 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -2338,6 +2338,11 @@ also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware
if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and
there is no overriding User-Agent header specified for the request.
+Set it to a value that identifies you, including a URL or an email address
+where website owners can reach you, e.g. ``"MyProject
+(+https://example.com/bot)"``, so that they can ask you to adjust your crawler
+rather than block it.
+
.. setting:: WARN_ON_GENERATOR_RETURN_VALUE
WARN_ON_GENERATOR_RETURN_VALUE