Scrapy, a fast high-level web crawling & scraping framework for Python.
Advise setting an identifying user agent (#7890)
Adriana6c017c2ca2b
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex eaf492c95..efade47e6 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -72,6 +72,11 @@ This will create a ``tutorial`` directory with the following contents::spiders/ # a directory where you'll later put your spiders__init__.py+Before crawling anything, open ``settings.py`` and uncomment the+:setting:`USER_AGENT` line to identify yourself, e.g. a project name plus a URL+or an email address. Website owners who take issue with your crawler can then+ask you to adjust it, rather than block it.+Our first Spider================diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex aeaf322c2..971bb9106 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -530,32 +530,41 @@ modules by separating them with commas.Avoiding getting banned=======================-Some websites implement certain measures to prevent bots from crawling them,-with varying degrees of sophistication. Getting around those measures can be-difficult and tricky, and may sometimes require special infrastructure. Please-consider contacting `commercial support`_ if in doubt.--Here are some tips to keep in mind when dealing with these kinds of sites:--* rotate your user agent from a pool of well-known ones from browsers (Google- around to get a list of them)-* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use- cookies to spot bot behaviour-* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.-* if possible, use `Common Crawl`_ to fetch pages, instead of hitting the sites- directly-* use a pool of rotating IPs. For example, the free `Tor project`_ or paid+Websites tell regular visitors and crawlers apart by how their traffic looks:+the headers it carries, how fast it arrives, how many requests come from the+same place. Traffic that stands out can be blocked even when the crawling+itself would be welcome.++Where the website allows crawling, the most effective thing you can do is make+yourself known: set :setting:`USER_AGENT` to a value that identifies you and+lets its owners reach you, so that they can ask you to adjust your crawler+rather than block it.++Where that is not enough, the following make your traffic resemble that of a+regular visitor:++* rotate your user agent among those of common browsers, so that your requests+ do not all look alike (search the web for an up-to-date list)+* disable cookies (see :setting:`COOKIES_ENABLED`), so that a session+ identifier does not tie all your requests together+* space out your requests, 2 seconds apart or more, with the+ :setting:`DOWNLOAD_DELAY` setting, to keep your pace closer to that of a+ person browsing+* where possible, read pages from `Common Crawl`_, which sends no traffic to+ the website at all+* spread your requests over a pool of IP addresses, so that none of them+ accounts for your whole crawl. For example, the free `Tor project`_ or paidservices like `ProxyMesh`_.-* for HTTPS websites, if blocking appears related to TLS behavior, consider- adjusting the :setting:`DOWNLOAD_TLS_MIN_VERSION` and- :setting:`DOWNLOAD_TLS_MAX_VERSION` settings, since some websites may respond- differently depending on the TLS method used by the client.-* use a ban avoidance service, such as `Zyte API`_, which provides a `Scrapy- plugin <https://github.com/scrapy-plugins/scrapy-zyte-api>`__ and additional+* match the TLS behavior of a browser: some websites respond differently+ depending on the TLS version of the client, which you can adjust with the+ :setting:`DOWNLOAD_TLS_MIN_VERSION` and :setting:`DOWNLOAD_TLS_MAX_VERSION`+ settings.+* let a service take care of all of the above, such as `Zyte API`_, which+ provides a `Scrapy plugin+ <https://github.com/scrapy-plugins/scrapy-zyte-api>`__ and additionalfeatures, like `AI web scraping <https://www.zyte.com/ai-web-scraping/>`__-If you are still unable to prevent your bot getting banned, consider contacting-`commercial support`_.+If your crawler still gets blocked, consider contacting `commercial support`_... _static-analysis:diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex dc09fdeeb..ec222d999 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -2338,6 +2338,11 @@ also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddlewareif :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` andthere is no overriding User-Agent header specified for the request.+Set it to a value that identifies you, including a URL or an email address+where website owners can reach you, e.g. ``"MyProject+(+https://example.com/bot)"``, so that they can ask you to adjust your crawler+rather than block it.+.. setting:: WARN_ON_GENERATOR_RETURN_VALUEWARN_ON_GENERATOR_RETURN_VALUE