Scrapy, a fast high-level web crawling & scraping framework for Python.
Merge pull request #1902 from starrify/case-insensitive-robots-txt-for-sitemap
Paul Tremberth642fedb3d605
Scrapy, a fast high-level web crawling & scraping framework for Python.