Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge pull request #6298 from lorenzoverardo/bug-6292/robots.txt-non-UTF-8

Andrey Rakhmatullin5d31e89262a2