Scrapy, a fast high-level web crawling & scraping framework for Python.
documenting sitemap entries as suggested by @kmike
Victor Torres10f46bca54b2
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 918f1cc36..9d4ed6ca6 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -703,6 +703,16 @@ SitemapSpiderThis would retrieve only ``entries`` modified on 2005 and the followingyears.+ Entries are dict objects extracted from the sitemap document.+ Usually, the key is the tag name and the value is the text inside it.++ It's important to notice that:++ - as the loc attribute is required, entries without this tag are discarded+ - alternate links are stored in a list with the key ``alternate``+ (see ``sitemap_alternate_links``)+ - namespaces are removed, so lxml tags named as ``{foo}bar`` become only ``bar``+If you omit this method, all entries found in sitemaps will beprocessed, observing other attributes and their settings.