Scrapy, a fast high-level web crawling & scraping framework for Python.
Merge pull request #6298 from lorenzoverardo/bug-6292/robots.txt-non-UTF-8
Andrey Rakhmatullin5d31e89262a2
Scrapy, a fast high-level web crawling & scraping framework for Python.