Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge pull request #1724 from scrapy/robotstxt-default

Paul Tremberth0d368c5d6fd4