Scrapy, a fast high-level web crawling & scraping framework for Python.
Recommend Common Crawl instead of Google Cache
Florentinccdbb795ff2f
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex 1a9d56143..d0207fd18 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -262,7 +262,7 @@ Here are some tips to keep in mind when dealing with these kinds of sites:* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may usecookies to spot bot behaviour* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.-* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites+* if possible, use `Common Crawl`_ to fetch pages, instead of hitting the sitesdirectly* use a pool of rotating IPs. For example, the free `Tor project`_ or paidservices like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a@@ -277,7 +277,7 @@ If you are still unable to prevent your bot getting banned, consider contacting.. _Tor project: https://www.torproject.org/.. _commercial support: https://scrapy.org/support/.. _ProxyMesh: https://proxymesh.com/-.. _Google cache: http://www.googleguide.com/cached_pages.html+.. _Common Crawl: https://commoncrawl.org/.. _testspiders: https://github.com/scrapinghub/testspiders.. _scrapoxy: https://scrapoxy.io/.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/