Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge pull request #1902 from starrify/case-insensitive-robots-txt-for-sitemap

Paul Tremberth642fedb3d605