Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Revert "Move documentation about avoiding bans into a topic of its own (#4039)"

Andrey Rakhmatullincd17c829cf0d
diff --git a/docs/index.rst b/docs/index.rst
index 7647b3781..433798aa8 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -155,7 +155,6 @@ Solving specific problems
topics/debug
topics/contracts
topics/practices
- topics/avoiding-bans
topics/broad-crawls
topics/developer-tools
topics/dynamic-content
@@ -180,9 +179,6 @@ Solving specific problems
:doc:`topics/practices`
Get familiar with some Scrapy common practices.
-:doc:`topics/avoiding-bans`
- Avoid getting banned from websites.
-
:doc:`topics/broad-crawls`
Tune Scrapy for crawling a lot domains in parallel.
diff --git a/docs/topics/avoiding-bans.rst b/docs/topics/avoiding-bans.rst
deleted file mode 100644
index 59f0da191..000000000
--- a/docs/topics/avoiding-bans.rst
+++ /dev/null
@@ -1,340 +0,0 @@
-.. _bans:
-
-=============
-Avoiding bans
-=============
-
-This topic covers some of the strategies that you can follow to avoid getting
-different or bad responses from a website that you are crawling due to filters
-such as regional filters, web browser filters, etc.
-
-.. _avoiding-crawls:
-
-Avoiding crawls
-===============
-
-The best way not to be banned from a website is not to send requests to it in
-the first place.
-
-One way to avoid crawling a website is to find the desired dataset through
-other means. For example, you can use Google’s `dataset search engine`_.
-
-If the target website is the only or best source of the desired information,
-and you only need to extract the data on a monthly basis or a lower frequency,
-you may be able to crawl a public snapshot of the target website instead.
-`Common Crawl`_ is an open repository of web crawl data that you can access
-freely. It contains monthly snapshots of a wide variety of websites and, if you
-are lucky, your target website will be among them.
-
-.. _Common Crawl: https://commoncrawl.org/
-.. _dataset search engine: https://datasetsearch.research.google.com/
-
-
-.. _being-polite:
-
-Being polite
-============
-
-To avoid being banned, you should first avoid giving a website reasons to ban
-you.
-
-.. _identifying-yourself:
-
-Identifying yourself
---------------------
-
-If your crawling has a noticeable negative impact on a website or you crawl
-content that should not be crawled, website administrators will need to do
-something.
-
-Set :setting:`USER_AGENT` to a value that uniquely identifies your spider and
-includes contact information, so that website administrators can contact you.
-
-
-.. _following-robotstxt:
-
-Following robots.txt guidelines
--------------------------------
-
-Some websites provide a ``robots.txt`` file at their root path (e.g.
-``http://example.com/robots.txt``) that describes the guidelines that they wish
-bots to follow when crawling their website.
-
-Before you start writing a spider for a website, read their ``robots.txt``
-file and implement your spider following its guidelines. See the `robots.txt
-standard draft`_ or the `robots.txt Google specification`_ for information on
-how to read ``robots.txt`` files.
-
-To ensure that your spider does not crawl pages restricted by ``robots.txt``
-guidelines, set :setting:`ROBOTSTXT_OBEY` to ``True`` to enable the
-:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
-middleware. When you do, if your spider attempts to crawl a restricted page,
-this middleware aborts that request with the following message::
-
- Forbidden by robots.txt
-
-Also set :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
-:setting:`DOWNLOAD_DELAY` to values that comply with the ``Crawl-Delay`` or
-``Request-Rate`` directives from the ``robots.txt`` guidelines.
-
-You may also use the :ref:`AutoThrottle extension <topics-autothrottle>` on top
-of that, so that when the target website experiences a high load, your spider
-automatically switches to higher download delays.
-
-.. _robots.txt Google specification: https://developers.google.com/search/reference/robots_txt
-.. _robots.txt standard draft: https://tools.ietf.org/html/draft-koster-rep-00
-
-
-.. _choosing-crawl-speed:
-
-Finding the right guidelines on your own
-----------------------------------------
-
-If a website does not specify a desired download delay, or does not provide a
-``robots.txt`` file, you should make an effort to find out the right values for
-:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY` that
-will not have a noticeable negative impact on the target website.
-
-Use a service like `SimilarWeb`_ to find out the amount of monthly traffic that
-the target website receives, and choose concurrency and delay values that will
-not cause a noticeable traffic increase.
-
-.. _SimilarWeb: https://www.similarweb.com
-
-
-.. _filters-and-challenges:
-
-Bypassing filters and solving challenges
-========================================
-
-Some websites implement filters and challenges that aim to deny access or alter
-their content based on aspects of the visitor, such as the country where they
-are or the web browsing tool they use.
-
-.. _regional-filter:
-
-Bypassing regional filters
---------------------------
-
-Some websites send different or bad responses based on the region or country
-associated to your `IP address`_.
-
-To bypass these filters, get access to a `proxy server`_ that has an outgoing
-IP address from a region that gets the desired responses.
-
-Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
-middleware to configure your spider to use that proxy.
-
-.. _IP address: https://en.wikipedia.org/wiki/IP_address
-.. _proxy server: https://en.wikipedia.org/wiki/Proxy_server
-
-
-.. _web-browser-filter:
-
-Bypassing web browser filters
------------------------------
-
-Some websites send different or bad responses if they detect that your request
-does not come from a web browser.
-
-To bypass these filters, switch your :setting:`USER_AGENT` to a value copied
-from those that popular web browsers use. In some rare cases, you may need a
-user agent string from a specific web browser.
-
-There are multiple Scrapy plugins that can rotate your requests through popular
-web browser user agent strings, such as scrapy-fake-useragent_,
-scrapy-random-useragent_ or Scrapy-UserAgents_.
-
-For advanced web browser filters,
-:ref:`pre-rendering JavaScript <topics-javascript-rendering>` or
-:ref:`using a headless browser <topics-headless-browsing>` may be necessary.
-Use these options only as a last resort, however, because they cause a higher
-load per request on the target website.
-
-.. _scrapy-fake-useragent: https://github.com/alecxe/scrapy-fake-useragent
-.. _scrapy-random-useragent: https://github.com/cleocn/scrapy-random-useragent
-.. _Scrapy-UserAgents: https://pypi.org/project/Scrapy-UserAgents/
-
-
-.. _request-delay-filter:
-
-Bypassing request delay filters
--------------------------------
-
-Some websites may ban your IP after they detect that your requests use a
-constant download delay.
-
-To help bypassing these filters, the :setting:`RANDOMIZE_DOWNLOAD_DELAY`
-setting is enabled by default. When that is not enough, an
-:ref:`IP address rotation solution <ip-rotation>` may be much more effective.
-
-
-.. _isp-filter:
-
-Bypassing internet service provider filters
--------------------------------------------
-
-Some websites send different or bad responses if they detect that your request
-comes from an IP address that belongs to a `data center`_, as opposed to a
-residential IP address from an `internet service provider`_ or a mobile IP
-address from a `mobile network`_.
-
-To bypass these filters, get access to a proxy server that has an outgoing IP
-address that is either residential or mobile. Note that you may also get
-different responses depending on whether your IP address is residential or
-mobile.
-
-Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
-middleware to configure your spider to use that proxy.
-
-.. _data center: https://en.wikipedia.org/wiki/Data_center
-.. _internet service provider: https://en.wikipedia.org/wiki/Internet_service_provider
-.. _mobile network: https://en.wikipedia.org/wiki/Cellular_network
-
-
-.. _captcha:
-
-Solving CAPTCHA challenges
---------------------------
-
-Some websites require you to solve a `CAPTCHA challenge`_ to get the desired
-response.
-
-To bypass these filters, several options exist:
-
-- You could have your spider present the CAPTCHA challenge to you and wait
- for you to solve it manually.
-
-- Some CAPTCHA challenges can be solved using an `optical character
- recognition`_ (OCR) solution such as pytesseract_.
-
-- Paid CAPTCHA solving services exist.
-
-Whichever solution you choose, implement it as a :ref:`downloader middleware
-<topics-downloader-middleware>` that automatically detects CAPTCHA challenges
-in responses and solves them, so that your spider code only receives successful
-responses.
-
-.. _CAPTCHA challenge: https://en.wikipedia.org/wiki/CAPTCHA
-.. _optical character recognition: https://en.wikipedia.org/wiki/Optical_character_recognition
-.. _pytesseract: https://github.com/madmaze/pytesseract
-
-
-.. _ip-rotation:
-
-IP address rotation solutions
-=============================
-
-See below some of the different solutions there are to have your requests use
-different outgoing IP addresses.
-
-When using this approach, remember to set :setting:`COOKIES_ENABLED` to
-``False`` to disable global cookie handling. This prevents websites from
-identifying two requests as coming from the same user agent even if they come
-from different IP addresses and have different user-agent strings. You can
-still include some cookies manually in your requests. Define them through the
-``Cookies`` header of your requests. See
-:class:`Request.headers <scrapy.http.Request.headers>`.
-
-.. _smart-proxy:
-
-Smart proxies
--------------
-
-An increasing number of websites use solutions that apply many of the above
-filters and challenges at the same time.
-
-There are paid proxy services, like `Zyte Smart Proxy Manager`_, that
-automatically bypass website filters and challenges, so that your spider only
-gets successful responses. They also allow managing sessions to simulate user
-behavior.
-
-For Zyte Smart Proxy Manager, installing scrapy-crawlera_ will offer advanced
-integration with Scrapy. For other services, use the
-:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` middleware
-or implement your own :ref:`downloader middleware
-<topics-downloader-middleware>`.
-
-.. _scrapy-crawlera: https://scrapy-crawlera.readthedocs.io/en/latest/
-.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/
-
-
-.. _rotating-proxy:
-
-Rotating proxies
-----------------
-
-Rotating proxy services like ProxyMesh_ send different requests through
-different proxies. This can decrease the likelihood of being affected by some
-filters or challenges.
-
-.. _ProxyMesh: https://proxymesh.com/
-
-
-.. _free-proxies:
-
-Free proxies
-------------
-
-You can easily find lists of free proxies in the internet, and you can use
-a solution like `scrapy-rotating-proxies`_ to configure multiple proxies in
-your spider and have requests rotate through them automatically.
-
-This approach, however, has serious drawbacks:
-
-- Free proxies may stop working at any moment. You need to implement a way to
- refresh your list of free proxies.
-
-- In addition to handling occasional bad responses from websites, you
- need to handle all kinds of bad responses from proxies. You may even need
- to inspect the response body to determine if a response comes from the
- target website or from a misbehaving proxy.
-
-- Advanced antibot solutions may automatically detect and filter out traffic
- from free proxies.
-
-.. _scrapy-rotating-proxies: https://github.com/TeamHG-Memex/scrapy-rotating-proxies
-
-
-.. _custom-rotating-proxy:
-
-Custom rotating proxy
----------------------
-
-If you have spare servers, you can set them up as proxies and use scrapoxy_ to
-build a custom proxy that rotates traffic through them. However, the initial
-setup can be complex, and your requests will be vulnerable to
-:ref:`internet service provider filtering <isp-filter>`.
-
-.. _scrapoxy: https://scrapoxy.io/
-
-
-.. _tor:
-
-The Tor network
----------------
-
-It is possible to send requests through the `Tor network`_.
-
-The initial setup to have Scrapy working with Tor is not straightforward.
-Use a search engine to find up-to-date documentation specific to using
-Scrapy and Tor together.
-
-The main drawback of using the Tor network is that traffic can be extremely
-slow.
-
-.. _Tor network: https://en.wikipedia.org/wiki/Tor_(anonymity_network)
-
-
-.. _commercial-support:
-
-Seeking professional help
-=========================
-
-Avoiding bans, filters and challenges can be difficult and tricky, and may
-sometimes require special infrastructure.
-
-If you find yourself unable to prevent your spider from getting bad responses,
-consider contacting `commercial support`_.
-
-.. _commercial support: https://scrapy.org/support/
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index a7a6fd129..732eba587 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -226,5 +226,39 @@ crawl::
curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
-
+.. _bans:
+
+Avoiding getting banned
+=======================
+
+Some websites implement certain measures to prevent bots from crawling them,
+with varying degrees of sophistication. Getting around those measures can be
+difficult and tricky, and may sometimes require special infrastructure. Please
+consider contacting `commercial support`_ if in doubt.
+
+Here are some tips to keep in mind when dealing with these kinds of sites:
+
+* rotate your user agent from a pool of well-known ones from browsers (google
+ around to get a list of them)
+* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
+ cookies to spot bot behaviour
+* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
+* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
+ directly
+* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
+ services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a
+ super proxy that you can attach your own proxies to.
+* use a highly distributed downloader that circumvents bans internally, so you
+ can just focus on parsing clean pages. One example of such downloaders is
+ `Zyte Smart Proxy Manager`_
+
+If you are still unable to prevent your bot getting banned, consider contacting
+`commercial support`_.
+
+.. _Tor project: https://www.torproject.org/
+.. _commercial support: https://scrapy.org/support/
+.. _ProxyMesh: https://proxymesh.com/
+.. _Google cache: http://www.googleguide.com/cached_pages.html
.. _testspiders: https://github.com/scrapinghub/testspiders
+.. _scrapoxy: https://scrapoxy.io/
+.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/