Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

mention scrapoxy in best practices doc

Pablo Hoffmand62776a8589a
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 7dae68470..25ae4b5ba 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -238,7 +238,8 @@ Here are some tips to keep in mind when dealing with these kinds of sites:
* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
directly
* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
- services like `ProxyMesh`_
+ services like `ProxyMesh`_. An open source alterantive is `scrapoxy`_, a
+ super proxy that you can attach your own proxies to.
* use a highly distributed downloader that circumvents bans internally, so you
can just focus on parsing clean pages. One example of such downloaders is
`Crawlera`_
@@ -253,3 +254,4 @@ If you are still unable to prevent your bot getting banned, consider contacting
.. _testspiders: https://github.com/scrapinghub/testspiders
.. _Twisted Reactor Overview: https://twistedmatrix.com/documents/current/core/howto/reactor-basics.html
.. _Crawlera: http://scrapinghub.com/crawlera
+.. _scrapoxy: http://scrapoxy.io/