Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Move documentation about avoiding bans into a topic of its own (#4039)

Adrián Chaves2d2581c68f35
diff --git a/docs/index.rst b/docs/index.rst
index 433798aa8..7647b3781 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -155,6 +155,7 @@ Solving specific problems
topics/debug
topics/contracts
topics/practices
+ topics/avoiding-bans
topics/broad-crawls
topics/developer-tools
topics/dynamic-content
@@ -179,6 +180,9 @@ Solving specific problems
:doc:`topics/practices`
Get familiar with some Scrapy common practices.
+:doc:`topics/avoiding-bans`
+ Avoid getting banned from websites.
+
:doc:`topics/broad-crawls`
Tune Scrapy for crawling a lot domains in parallel.
diff --git a/docs/topics/avoiding-bans.rst b/docs/topics/avoiding-bans.rst
new file mode 100644
index 000000000..59f0da191
--- /dev/null
+++ b/docs/topics/avoiding-bans.rst
@@ -0,0 +1,340 @@
+.. _bans:
+
+=============
+Avoiding bans
+=============
+
+This topic covers some of the strategies that you can follow to avoid getting
+different or bad responses from a website that you are crawling due to filters
+such as regional filters, web browser filters, etc.
+
+.. _avoiding-crawls:
+
+Avoiding crawls
+===============
+
+The best way not to be banned from a website is not to send requests to it in
+the first place.
+
+One way to avoid crawling a website is to find the desired dataset through
+other means. For example, you can use Google’s `dataset search engine`_.
+
+If the target website is the only or best source of the desired information,
+and you only need to extract the data on a monthly basis or a lower frequency,
+you may be able to crawl a public snapshot of the target website instead.
+`Common Crawl`_ is an open repository of web crawl data that you can access
+freely. It contains monthly snapshots of a wide variety of websites and, if you
+are lucky, your target website will be among them.
+
+.. _Common Crawl: https://commoncrawl.org/
+.. _dataset search engine: https://datasetsearch.research.google.com/
+
+
+.. _being-polite:
+
+Being polite
+============
+
+To avoid being banned, you should first avoid giving a website reasons to ban
+you.
+
+.. _identifying-yourself:
+
+Identifying yourself
+--------------------
+
+If your crawling has a noticeable negative impact on a website or you crawl
+content that should not be crawled, website administrators will need to do
+something.
+
+Set :setting:`USER_AGENT` to a value that uniquely identifies your spider and
+includes contact information, so that website administrators can contact you.
+
+
+.. _following-robotstxt:
+
+Following robots.txt guidelines
+-------------------------------
+
+Some websites provide a ``robots.txt`` file at their root path (e.g.
+``http://example.com/robots.txt``) that describes the guidelines that they wish
+bots to follow when crawling their website.
+
+Before you start writing a spider for a website, read their ``robots.txt``
+file and implement your spider following its guidelines. See the `robots.txt
+standard draft`_ or the `robots.txt Google specification`_ for information on
+how to read ``robots.txt`` files.
+
+To ensure that your spider does not crawl pages restricted by ``robots.txt``
+guidelines, set :setting:`ROBOTSTXT_OBEY` to ``True`` to enable the
+:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
+middleware. When you do, if your spider attempts to crawl a restricted page,
+this middleware aborts that request with the following message::
+
+ Forbidden by robots.txt
+
+Also set :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
+:setting:`DOWNLOAD_DELAY` to values that comply with the ``Crawl-Delay`` or
+``Request-Rate`` directives from the ``robots.txt`` guidelines.
+
+You may also use the :ref:`AutoThrottle extension <topics-autothrottle>` on top
+of that, so that when the target website experiences a high load, your spider
+automatically switches to higher download delays.
+
+.. _robots.txt Google specification: https://developers.google.com/search/reference/robots_txt
+.. _robots.txt standard draft: https://tools.ietf.org/html/draft-koster-rep-00
+
+
+.. _choosing-crawl-speed:
+
+Finding the right guidelines on your own
+----------------------------------------
+
+If a website does not specify a desired download delay, or does not provide a
+``robots.txt`` file, you should make an effort to find out the right values for
+:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY` that
+will not have a noticeable negative impact on the target website.
+
+Use a service like `SimilarWeb`_ to find out the amount of monthly traffic that
+the target website receives, and choose concurrency and delay values that will
+not cause a noticeable traffic increase.
+
+.. _SimilarWeb: https://www.similarweb.com
+
+
+.. _filters-and-challenges:
+
+Bypassing filters and solving challenges
+========================================
+
+Some websites implement filters and challenges that aim to deny access or alter
+their content based on aspects of the visitor, such as the country where they
+are or the web browsing tool they use.
+
+.. _regional-filter:
+
+Bypassing regional filters
+--------------------------
+
+Some websites send different or bad responses based on the region or country
+associated to your `IP address`_.
+
+To bypass these filters, get access to a `proxy server`_ that has an outgoing
+IP address from a region that gets the desired responses.
+
+Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
+middleware to configure your spider to use that proxy.
+
+.. _IP address: https://en.wikipedia.org/wiki/IP_address
+.. _proxy server: https://en.wikipedia.org/wiki/Proxy_server
+
+
+.. _web-browser-filter:
+
+Bypassing web browser filters
+-----------------------------
+
+Some websites send different or bad responses if they detect that your request
+does not come from a web browser.
+
+To bypass these filters, switch your :setting:`USER_AGENT` to a value copied
+from those that popular web browsers use. In some rare cases, you may need a
+user agent string from a specific web browser.
+
+There are multiple Scrapy plugins that can rotate your requests through popular
+web browser user agent strings, such as scrapy-fake-useragent_,
+scrapy-random-useragent_ or Scrapy-UserAgents_.
+
+For advanced web browser filters,
+:ref:`pre-rendering JavaScript <topics-javascript-rendering>` or
+:ref:`using a headless browser <topics-headless-browsing>` may be necessary.
+Use these options only as a last resort, however, because they cause a higher
+load per request on the target website.
+
+.. _scrapy-fake-useragent: https://github.com/alecxe/scrapy-fake-useragent
+.. _scrapy-random-useragent: https://github.com/cleocn/scrapy-random-useragent
+.. _Scrapy-UserAgents: https://pypi.org/project/Scrapy-UserAgents/
+
+
+.. _request-delay-filter:
+
+Bypassing request delay filters
+-------------------------------
+
+Some websites may ban your IP after they detect that your requests use a
+constant download delay.
+
+To help bypassing these filters, the :setting:`RANDOMIZE_DOWNLOAD_DELAY`
+setting is enabled by default. When that is not enough, an
+:ref:`IP address rotation solution <ip-rotation>` may be much more effective.
+
+
+.. _isp-filter:
+
+Bypassing internet service provider filters
+-------------------------------------------
+
+Some websites send different or bad responses if they detect that your request
+comes from an IP address that belongs to a `data center`_, as opposed to a
+residential IP address from an `internet service provider`_ or a mobile IP
+address from a `mobile network`_.
+
+To bypass these filters, get access to a proxy server that has an outgoing IP
+address that is either residential or mobile. Note that you may also get
+different responses depending on whether your IP address is residential or
+mobile.
+
+Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
+middleware to configure your spider to use that proxy.
+
+.. _data center: https://en.wikipedia.org/wiki/Data_center
+.. _internet service provider: https://en.wikipedia.org/wiki/Internet_service_provider
+.. _mobile network: https://en.wikipedia.org/wiki/Cellular_network
+
+
+.. _captcha:
+
+Solving CAPTCHA challenges
+--------------------------
+
+Some websites require you to solve a `CAPTCHA challenge`_ to get the desired
+response.
+
+To bypass these filters, several options exist:
+
+- You could have your spider present the CAPTCHA challenge to you and wait
+ for you to solve it manually.
+
+- Some CAPTCHA challenges can be solved using an `optical character
+ recognition`_ (OCR) solution such as pytesseract_.
+
+- Paid CAPTCHA solving services exist.
+
+Whichever solution you choose, implement it as a :ref:`downloader middleware
+<topics-downloader-middleware>` that automatically detects CAPTCHA challenges
+in responses and solves them, so that your spider code only receives successful
+responses.
+
+.. _CAPTCHA challenge: https://en.wikipedia.org/wiki/CAPTCHA
+.. _optical character recognition: https://en.wikipedia.org/wiki/Optical_character_recognition
+.. _pytesseract: https://github.com/madmaze/pytesseract
+
+
+.. _ip-rotation:
+
+IP address rotation solutions
+=============================
+
+See below some of the different solutions there are to have your requests use
+different outgoing IP addresses.
+
+When using this approach, remember to set :setting:`COOKIES_ENABLED` to
+``False`` to disable global cookie handling. This prevents websites from
+identifying two requests as coming from the same user agent even if they come
+from different IP addresses and have different user-agent strings. You can
+still include some cookies manually in your requests. Define them through the
+``Cookies`` header of your requests. See
+:class:`Request.headers <scrapy.http.Request.headers>`.
+
+.. _smart-proxy:
+
+Smart proxies
+-------------
+
+An increasing number of websites use solutions that apply many of the above
+filters and challenges at the same time.
+
+There are paid proxy services, like `Zyte Smart Proxy Manager`_, that
+automatically bypass website filters and challenges, so that your spider only
+gets successful responses. They also allow managing sessions to simulate user
+behavior.
+
+For Zyte Smart Proxy Manager, installing scrapy-crawlera_ will offer advanced
+integration with Scrapy. For other services, use the
+:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` middleware
+or implement your own :ref:`downloader middleware
+<topics-downloader-middleware>`.
+
+.. _scrapy-crawlera: https://scrapy-crawlera.readthedocs.io/en/latest/
+.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/
+
+
+.. _rotating-proxy:
+
+Rotating proxies
+----------------
+
+Rotating proxy services like ProxyMesh_ send different requests through
+different proxies. This can decrease the likelihood of being affected by some
+filters or challenges.
+
+.. _ProxyMesh: https://proxymesh.com/
+
+
+.. _free-proxies:
+
+Free proxies
+------------
+
+You can easily find lists of free proxies in the internet, and you can use
+a solution like `scrapy-rotating-proxies`_ to configure multiple proxies in
+your spider and have requests rotate through them automatically.
+
+This approach, however, has serious drawbacks:
+
+- Free proxies may stop working at any moment. You need to implement a way to
+ refresh your list of free proxies.
+
+- In addition to handling occasional bad responses from websites, you
+ need to handle all kinds of bad responses from proxies. You may even need
+ to inspect the response body to determine if a response comes from the
+ target website or from a misbehaving proxy.
+
+- Advanced antibot solutions may automatically detect and filter out traffic
+ from free proxies.
+
+.. _scrapy-rotating-proxies: https://github.com/TeamHG-Memex/scrapy-rotating-proxies
+
+
+.. _custom-rotating-proxy:
+
+Custom rotating proxy
+---------------------
+
+If you have spare servers, you can set them up as proxies and use scrapoxy_ to
+build a custom proxy that rotates traffic through them. However, the initial
+setup can be complex, and your requests will be vulnerable to
+:ref:`internet service provider filtering <isp-filter>`.
+
+.. _scrapoxy: https://scrapoxy.io/
+
+
+.. _tor:
+
+The Tor network
+---------------
+
+It is possible to send requests through the `Tor network`_.
+
+The initial setup to have Scrapy working with Tor is not straightforward.
+Use a search engine to find up-to-date documentation specific to using
+Scrapy and Tor together.
+
+The main drawback of using the Tor network is that traffic can be extremely
+slow.
+
+.. _Tor network: https://en.wikipedia.org/wiki/Tor_(anonymity_network)
+
+
+.. _commercial-support:
+
+Seeking professional help
+=========================
+
+Avoiding bans, filters and challenges can be difficult and tricky, and may
+sometimes require special infrastructure.
+
+If you find yourself unable to prevent your spider from getting bad responses,
+consider contacting `commercial support`_.
+
+.. _commercial support: https://scrapy.org/support/
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 732eba587..a7a6fd129 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -226,39 +226,5 @@ crawl::
curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
-.. _bans:
-
-Avoiding getting banned
-=======================
-
-Some websites implement certain measures to prevent bots from crawling them,
-with varying degrees of sophistication. Getting around those measures can be
-difficult and tricky, and may sometimes require special infrastructure. Please
-consider contacting `commercial support`_ if in doubt.
-
-Here are some tips to keep in mind when dealing with these kinds of sites:
-
-* rotate your user agent from a pool of well-known ones from browsers (google
- around to get a list of them)
-* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
- cookies to spot bot behaviour
-* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
-* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
- directly
-* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
- services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a
- super proxy that you can attach your own proxies to.
-* use a highly distributed downloader that circumvents bans internally, so you
- can just focus on parsing clean pages. One example of such downloaders is
- `Zyte Smart Proxy Manager`_
-
-If you are still unable to prevent your bot getting banned, consider contacting
-`commercial support`_.
-
-.. _Tor project: https://www.torproject.org/
-.. _commercial support: https://scrapy.org/support/
-.. _ProxyMesh: https://proxymesh.com/
-.. _Google cache: http://www.googleguide.com/cached_pages.html
+
.. _testspiders: https://github.com/scrapinghub/testspiders
-.. _scrapoxy: https://scrapoxy.io/
-.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/