Scrapy, a fast high-level web crawling & scraping framework for Python.
Revert "Move documentation about avoiding bans into a topic of its own (#4039)"
Andrey Rakhmatullincd17c829cf0d
diff --git a/docs/index.rst b/docs/index.rstindex 7647b3781..433798aa8 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -155,7 +155,6 @@ Solving specific problemstopics/debugtopics/contractstopics/practices- topics/avoiding-banstopics/broad-crawlstopics/developer-toolstopics/dynamic-content@@ -180,9 +179,6 @@ Solving specific problems:doc:`topics/practices`Get familiar with some Scrapy common practices.-:doc:`topics/avoiding-bans`- Avoid getting banned from websites.-:doc:`topics/broad-crawls`Tune Scrapy for crawling a lot domains in parallel.diff --git a/docs/topics/avoiding-bans.rst b/docs/topics/avoiding-bans.rstdeleted file mode 100644index 59f0da191..000000000--- a/docs/topics/avoiding-bans.rst+++ /dev/null@@ -1,340 +0,0 @@-.. _bans:--=============-Avoiding bans-=============--This topic covers some of the strategies that you can follow to avoid getting-different or bad responses from a website that you are crawling due to filters-such as regional filters, web browser filters, etc.--.. _avoiding-crawls:--Avoiding crawls-===============--The best way not to be banned from a website is not to send requests to it in-the first place.--One way to avoid crawling a website is to find the desired dataset through-other means. For example, you can use Google’s `dataset search engine`_.--If the target website is the only or best source of the desired information,-and you only need to extract the data on a monthly basis or a lower frequency,-you may be able to crawl a public snapshot of the target website instead.-`Common Crawl`_ is an open repository of web crawl data that you can access-freely. It contains monthly snapshots of a wide variety of websites and, if you-are lucky, your target website will be among them.--.. _Common Crawl: https://commoncrawl.org/-.. _dataset search engine: https://datasetsearch.research.google.com/---.. _being-polite:--Being polite-============--To avoid being banned, you should first avoid giving a website reasons to ban-you.--.. _identifying-yourself:--Identifying yourself-----------------------If your crawling has a noticeable negative impact on a website or you crawl-content that should not be crawled, website administrators will need to do-something.--Set :setting:`USER_AGENT` to a value that uniquely identifies your spider and-includes contact information, so that website administrators can contact you.---.. _following-robotstxt:--Following robots.txt guidelines----------------------------------Some websites provide a ``robots.txt`` file at their root path (e.g.-``http://example.com/robots.txt``) that describes the guidelines that they wish-bots to follow when crawling their website.--Before you start writing a spider for a website, read their ``robots.txt``-file and implement your spider following its guidelines. See the `robots.txt-standard draft`_ or the `robots.txt Google specification`_ for information on-how to read ``robots.txt`` files.--To ensure that your spider does not crawl pages restricted by ``robots.txt``-guidelines, set :setting:`ROBOTSTXT_OBEY` to ``True`` to enable the-:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`-middleware. When you do, if your spider attempts to crawl a restricted page,-this middleware aborts that request with the following message::-- Forbidden by robots.txt--Also set :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and-:setting:`DOWNLOAD_DELAY` to values that comply with the ``Crawl-Delay`` or-``Request-Rate`` directives from the ``robots.txt`` guidelines.--You may also use the :ref:`AutoThrottle extension <topics-autothrottle>` on top-of that, so that when the target website experiences a high load, your spider-automatically switches to higher download delays.--.. _robots.txt Google specification: https://developers.google.com/search/reference/robots_txt-.. _robots.txt standard draft: https://tools.ietf.org/html/draft-koster-rep-00---.. _choosing-crawl-speed:--Finding the right guidelines on your own-------------------------------------------If a website does not specify a desired download delay, or does not provide a-``robots.txt`` file, you should make an effort to find out the right values for-:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY` that-will not have a noticeable negative impact on the target website.--Use a service like `SimilarWeb`_ to find out the amount of monthly traffic that-the target website receives, and choose concurrency and delay values that will-not cause a noticeable traffic increase.--.. _SimilarWeb: https://www.similarweb.com---.. _filters-and-challenges:--Bypassing filters and solving challenges-========================================--Some websites implement filters and challenges that aim to deny access or alter-their content based on aspects of the visitor, such as the country where they-are or the web browsing tool they use.--.. _regional-filter:--Bypassing regional filters-----------------------------Some websites send different or bad responses based on the region or country-associated to your `IP address`_.--To bypass these filters, get access to a `proxy server`_ that has an outgoing-IP address from a region that gets the desired responses.--Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`-middleware to configure your spider to use that proxy.--.. _IP address: https://en.wikipedia.org/wiki/IP_address-.. _proxy server: https://en.wikipedia.org/wiki/Proxy_server---.. _web-browser-filter:--Bypassing web browser filters--------------------------------Some websites send different or bad responses if they detect that your request-does not come from a web browser.--To bypass these filters, switch your :setting:`USER_AGENT` to a value copied-from those that popular web browsers use. In some rare cases, you may need a-user agent string from a specific web browser.--There are multiple Scrapy plugins that can rotate your requests through popular-web browser user agent strings, such as scrapy-fake-useragent_,-scrapy-random-useragent_ or Scrapy-UserAgents_.--For advanced web browser filters,-:ref:`pre-rendering JavaScript <topics-javascript-rendering>` or-:ref:`using a headless browser <topics-headless-browsing>` may be necessary.-Use these options only as a last resort, however, because they cause a higher-load per request on the target website.--.. _scrapy-fake-useragent: https://github.com/alecxe/scrapy-fake-useragent-.. _scrapy-random-useragent: https://github.com/cleocn/scrapy-random-useragent-.. _Scrapy-UserAgents: https://pypi.org/project/Scrapy-UserAgents/---.. _request-delay-filter:--Bypassing request delay filters----------------------------------Some websites may ban your IP after they detect that your requests use a-constant download delay.--To help bypassing these filters, the :setting:`RANDOMIZE_DOWNLOAD_DELAY`-setting is enabled by default. When that is not enough, an-:ref:`IP address rotation solution <ip-rotation>` may be much more effective.---.. _isp-filter:--Bypassing internet service provider filters----------------------------------------------Some websites send different or bad responses if they detect that your request-comes from an IP address that belongs to a `data center`_, as opposed to a-residential IP address from an `internet service provider`_ or a mobile IP-address from a `mobile network`_.--To bypass these filters, get access to a proxy server that has an outgoing IP-address that is either residential or mobile. Note that you may also get-different responses depending on whether your IP address is residential or-mobile.--Use the :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`-middleware to configure your spider to use that proxy.--.. _data center: https://en.wikipedia.org/wiki/Data_center-.. _internet service provider: https://en.wikipedia.org/wiki/Internet_service_provider-.. _mobile network: https://en.wikipedia.org/wiki/Cellular_network---.. _captcha:--Solving CAPTCHA challenges-----------------------------Some websites require you to solve a `CAPTCHA challenge`_ to get the desired-response.--To bypass these filters, several options exist:--- You could have your spider present the CAPTCHA challenge to you and wait- for you to solve it manually.--- Some CAPTCHA challenges can be solved using an `optical character- recognition`_ (OCR) solution such as pytesseract_.--- Paid CAPTCHA solving services exist.--Whichever solution you choose, implement it as a :ref:`downloader middleware-<topics-downloader-middleware>` that automatically detects CAPTCHA challenges-in responses and solves them, so that your spider code only receives successful-responses.--.. _CAPTCHA challenge: https://en.wikipedia.org/wiki/CAPTCHA-.. _optical character recognition: https://en.wikipedia.org/wiki/Optical_character_recognition-.. _pytesseract: https://github.com/madmaze/pytesseract---.. _ip-rotation:--IP address rotation solutions-=============================--See below some of the different solutions there are to have your requests use-different outgoing IP addresses.--When using this approach, remember to set :setting:`COOKIES_ENABLED` to-``False`` to disable global cookie handling. This prevents websites from-identifying two requests as coming from the same user agent even if they come-from different IP addresses and have different user-agent strings. You can-still include some cookies manually in your requests. Define them through the-``Cookies`` header of your requests. See-:class:`Request.headers <scrapy.http.Request.headers>`.--.. _smart-proxy:--Smart proxies----------------An increasing number of websites use solutions that apply many of the above-filters and challenges at the same time.--There are paid proxy services, like `Zyte Smart Proxy Manager`_, that-automatically bypass website filters and challenges, so that your spider only-gets successful responses. They also allow managing sessions to simulate user-behavior.--For Zyte Smart Proxy Manager, installing scrapy-crawlera_ will offer advanced-integration with Scrapy. For other services, use the-:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` middleware-or implement your own :ref:`downloader middleware-<topics-downloader-middleware>`.--.. _scrapy-crawlera: https://scrapy-crawlera.readthedocs.io/en/latest/-.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/---.. _rotating-proxy:--Rotating proxies-------------------Rotating proxy services like ProxyMesh_ send different requests through-different proxies. This can decrease the likelihood of being affected by some-filters or challenges.--.. _ProxyMesh: https://proxymesh.com/---.. _free-proxies:--Free proxies---------------You can easily find lists of free proxies in the internet, and you can use-a solution like `scrapy-rotating-proxies`_ to configure multiple proxies in-your spider and have requests rotate through them automatically.--This approach, however, has serious drawbacks:--- Free proxies may stop working at any moment. You need to implement a way to- refresh your list of free proxies.--- In addition to handling occasional bad responses from websites, you- need to handle all kinds of bad responses from proxies. You may even need- to inspect the response body to determine if a response comes from the- target website or from a misbehaving proxy.--- Advanced antibot solutions may automatically detect and filter out traffic- from free proxies.--.. _scrapy-rotating-proxies: https://github.com/TeamHG-Memex/scrapy-rotating-proxies---.. _custom-rotating-proxy:--Custom rotating proxy------------------------If you have spare servers, you can set them up as proxies and use scrapoxy_ to-build a custom proxy that rotates traffic through them. However, the initial-setup can be complex, and your requests will be vulnerable to-:ref:`internet service provider filtering <isp-filter>`.--.. _scrapoxy: https://scrapoxy.io/---.. _tor:--The Tor network------------------It is possible to send requests through the `Tor network`_.--The initial setup to have Scrapy working with Tor is not straightforward.-Use a search engine to find up-to-date documentation specific to using-Scrapy and Tor together.--The main drawback of using the Tor network is that traffic can be extremely-slow.--.. _Tor network: https://en.wikipedia.org/wiki/Tor_(anonymity_network)---.. _commercial-support:--Seeking professional help-=========================--Avoiding bans, filters and challenges can be difficult and tricky, and may-sometimes require special infrastructure.--If you find yourself unable to prevent your spider from getting bad responses,-consider contacting `commercial support`_.--.. _commercial support: https://scrapy.org/support/diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex a7a6fd129..732eba587 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -226,5 +226,39 @@ crawl::curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3-+.. _bans:++Avoiding getting banned+=======================++Some websites implement certain measures to prevent bots from crawling them,+with varying degrees of sophistication. Getting around those measures can be+difficult and tricky, and may sometimes require special infrastructure. Please+consider contacting `commercial support`_ if in doubt.++Here are some tips to keep in mind when dealing with these kinds of sites:++* rotate your user agent from a pool of well-known ones from browsers (google+ around to get a list of them)+* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use+ cookies to spot bot behaviour+* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.+* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites+ directly+* use a pool of rotating IPs. For example, the free `Tor project`_ or paid+ services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a+ super proxy that you can attach your own proxies to.+* use a highly distributed downloader that circumvents bans internally, so you+ can just focus on parsing clean pages. One example of such downloaders is+ `Zyte Smart Proxy Manager`_++If you are still unable to prevent your bot getting banned, consider contacting+`commercial support`_.++.. _Tor project: https://www.torproject.org/+.. _commercial support: https://scrapy.org/support/+.. _ProxyMesh: https://proxymesh.com/+.. _Google cache: http://www.googleguide.com/cached_pages.html.. _testspiders: https://github.com/scrapinghub/testspiders+.. _scrapoxy: https://scrapoxy.io/+.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/