Scrapy, a fast high-level web crawling & scraping framework for Python.
Added: Making it case-insensitive when extracting sitemap URLs from a robots.txt
Pengyu CHEN103f6eaa88f2
diff --git a/scrapy/utils/sitemap.py b/scrapy/utils/sitemap.pyindex bbf37bc28..008196435 100644--- a/scrapy/utils/sitemap.py+++ b/scrapy/utils/sitemap.py@@ -39,5 +39,5 @@ def sitemap_urls_from_robots(robots_text):robots.txt file"""for line in robots_text.splitlines():- if line.lstrip().startswith('Sitemap:'):+ if line.lstrip().lower().startswith('sitemap:'):yield line.split(':', 1)[1].strip()