Scrapy, a fast high-level web crawling & scraping framework for Python.
Add and run pre-commit hook 'blacken-docs'
pankaj1707kc1bbb299d7dc
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yamlindex f5fc1285f..729682392 100644--- a/.pre-commit-config.yaml+++ b/.pre-commit-config.yaml@@ -16,3 +16,9 @@ repos:rev: 5.12.0hooks:- id: isort+- repo: https://github.com/adamchainz/blacken-docs+ rev: 1.13.0+ hooks:+ - id: blacken-docs+ additional_dependencies:+ - black==22.12.0diff --git a/docs/faq.rst b/docs/faq.rstindex 8a9ba809b..836420567 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -35,8 +35,9 @@ for parsing HTML responses in Scrapy callbacks.You just have to feed the response's body into a ``BeautifulSoup`` objectand extract whatever data you need from it.-Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser::+Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser:+.. code-block:: pythonfrom bs4 import BeautifulSoupimport scrapy@@ -45,17 +46,12 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parsclass ExampleSpider(scrapy.Spider):name = "example"allowed_domains = ["example.com"]- start_urls = (- 'http://www.example.com/',- )+ start_urls = ("http://www.example.com/",)def parse(self, response):# use lxml to get decent HTML parsing speed- soup = BeautifulSoup(response.text, 'lxml')- yield {- "url": response.url,- "title": soup.h1.string- }+ soup = BeautifulSoup(response.text, "lxml")+ yield {"url": response.url, "title": soup.h1.string}.. note::@@ -109,11 +105,13 @@ basically means that it crawls in `DFO order`_. This order is more convenientin most cases.If you do want to crawl in true `BFO order`_, you can do it by-setting the following settings::+setting the following settings:++.. code-block:: pythonDEPTH_PRIORITY = 1- SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'- SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'+ SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"+ SCHEDULER_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"While pending requests are below the configured values of:setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or@@ -159,11 +157,13 @@ See also other suggestions at `StackOverflow`_... note:: Remember to disable:class:`scrapy.spidermiddlewares.offsite.OffsiteMiddleware` when you enable- your custom implementation::+ your custom implementation:++ .. code-block:: pythonSPIDER_MIDDLEWARES = {- 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,- 'myproject.middlewares.CustomOffsiteMiddleware': 500,+ "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,+ "myproject.middlewares.CustomOffsiteMiddleware": 500,}.. _meet the installation requirements: https://github.com/andreasvc/pyre2#installation@@ -235,11 +235,13 @@ What does the response status code 999 means?999 is a custom response status code used by Yahoo sites to throttle requests.Try slowing down the crawling speed by using a download delay of ``2`` (or-higher) in your spider::+higher) in your spider:++.. code-block:: pythonclass MySpider(CrawlSpider):- name = 'myspider'+ name = "myspider"download_delay = 2@@ -351,19 +353,21 @@ How to split an item into multiple items in an item pipeline?input item. :ref:`Create a spider middleware <custom-spider-middleware>`instead, and use its:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`-method for this purpose. For example::+method for this purpose. For example:++.. code-block:: pythonfrom copy import deepcopyfrom itemadapter import is_item, ItemAdapter- class MultiplyItemsMiddleware:+ class MultiplyItemsMiddleware:def process_spider_output(self, response, result, spider):for item in result:if is_item(item):adapter = ItemAdapter(item)- for _ in range(adapter['multiply_by']):+ for _ in range(adapter["multiply_by"]):yield deepcopy(item)Does Scrapy support IPv6 addresses?diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex cfa6bfa83..495aad091 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -20,22 +20,24 @@ In order to show you what Scrapy brings to the table, we'll walk you through anexample of a Scrapy Spider using the simplest way to run a spider.Here's the code for a spider that scrapes famous quotes from website-https://quotes.toscrape.com, following the pagination::+https://quotes.toscrape.com, following the pagination++.. code-block:: pythonimport scrapyclass QuotesSpider(scrapy.Spider):- name = 'quotes'+ name = "quotes"start_urls = [- 'https://quotes.toscrape.com/tag/humor/',+ "https://quotes.toscrape.com/tag/humor/",]def parse(self, response):- for quote in response.css('div.quote'):+ for quote in response.css("div.quote"):yield {- 'author': quote.xpath('span/small/text()').get(),- 'text': quote.css('span.text::text').get(),+ "author": quote.xpath("span/small/text()").get(),+ "text": quote.css("span.text::text").get(),}next_page = response.css('li.next a::attr("href")').get()diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex 901a170b4..f5e9b372e 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -177,7 +177,9 @@ that generates :class:`scrapy.Request <scrapy.Request>` objects from URLs,you can just define a :attr:`~scrapy.Spider.start_urls` class attributewith a list of URLs. This list will then be used by the default implementationof :meth:`~scrapy.Spider.start_requests` to create the initial requests-for your spider::+for your spider.++.. code-block:: pythonfrom pathlib import Path@@ -187,13 +189,13 @@ for your spider::class QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'https://quotes.toscrape.com/page/1/',- 'https://quotes.toscrape.com/page/2/',+ "https://quotes.toscrape.com/page/1/",+ "https://quotes.toscrape.com/page/2/",]def parse(self, response):page = response.url.split("/")[-2]- filename = f'quotes-{page}.html'+ filename = f"quotes-{page}.html"Path(filename).write_bytes(response.body)The :meth:`~scrapy.Spider.parse` method will be called to handle each@@ -438,7 +440,9 @@ extraction logic above into our spider.A Scrapy spider typically generates many dictionaries containing the dataextracted from the page. To do that, we use the ``yield`` Python keyword-in the callback, as you can see below::+in the callback, as you can see below:++.. code-block:: pythonimport scrapy@@ -446,16 +450,16 @@ in the callback, as you can see below::class QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'https://quotes.toscrape.com/page/1/',- 'https://quotes.toscrape.com/page/2/',+ "https://quotes.toscrape.com/page/1/",+ "https://quotes.toscrape.com/page/2/",]def parse(self, response):- for quote in response.css('div.quote'):+ for quote in response.css("div.quote"):yield {- 'text': quote.css('span.text::text').get(),- 'author': quote.css('small.author::text').get(),- 'tags': quote.css('div.tags a.tag::text').getall(),+ "text": quote.css("span.text::text").get(),+ "author": quote.css("small.author::text").get(),+ "tags": quote.css("div.tags a.tag::text").getall(),}If you run this spider, it will output the extracted data with the log::@@ -543,7 +547,9 @@ There is also an ``attrib`` property available'/page/2/'Let's see now our spider modified to recursively follow the link to the next-page, extracting data from it::+page, extracting data from it:++.. code-block:: pythonimport scrapy@@ -551,18 +557,18 @@ page, extracting data from it::class QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'https://quotes.toscrape.com/page/1/',+ "https://quotes.toscrape.com/page/1/",]def parse(self, response):- for quote in response.css('div.quote'):+ for quote in response.css("div.quote"):yield {- 'text': quote.css('span.text::text').get(),- 'author': quote.css('small.author::text').get(),- 'tags': quote.css('div.tags a.tag::text').getall(),+ "text": quote.css("span.text::text").get(),+ "author": quote.css("small.author::text").get(),+ "tags": quote.css("div.tags a.tag::text").getall(),}- next_page = response.css('li.next a::attr(href)').get()+ next_page = response.css("li.next a::attr(href)").get()if next_page is not None:next_page = response.urljoin(next_page)yield scrapy.Request(next_page, callback=self.parse)@@ -594,7 +600,9 @@ A shortcut for creating Requests--------------------------------As a shortcut for creating Request objects you can use-:meth:`response.follow <scrapy.http.TextResponse.follow>`::+:meth:`response.follow <scrapy.http.TextResponse.follow>`++.. code-block:: pythonimport scrapy@@ -602,18 +610,18 @@ As a shortcut for creating Request objects you can useclass QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'https://quotes.toscrape.com/page/1/',+ "https://quotes.toscrape.com/page/1/",]def parse(self, response):- for quote in response.css('div.quote'):+ for quote in response.css("div.quote"):yield {- 'text': quote.css('span.text::text').get(),- 'author': quote.css('span small::text').get(),- 'tags': quote.css('div.tags a.tag::text').getall(),+ "text": quote.css("span.text::text").get(),+ "author": quote.css("span small::text").get(),+ "tags": quote.css("div.tags a.tag::text").getall(),}- next_page = response.css('li.next a::attr(href)').get()+ next_page = response.css("li.next a::attr(href)").get()if next_page is not None:yield response.follow(next_page, callback=self.parse)@@ -622,57 +630,67 @@ need to call urljoin. Note that ``response.follow`` just returns a Requestinstance; you still have to yield this Request.You can also pass a selector to ``response.follow`` instead of a string;-this selector should extract necessary attributes::+this selector should extract necessary attributes:++.. code-block:: python- for href in response.css('ul.pager a::attr(href)'):+ for href in response.css("ul.pager a::attr(href)"):yield response.follow(href, callback=self.parse)For ``<a>`` elements there is a shortcut: ``response.follow`` uses their href-attribute automatically. So the code can be shortened further::+attribute automatically. So the code can be shortened further:- for a in response.css('ul.pager a'):+.. code-block:: python++ for a in response.css("ul.pager a"):yield response.follow(a, callback=self.parse)To create multiple requests from an iterable, you can use-:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` instead::+:meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` instead:++.. code-block:: python- anchors = response.css('ul.pager a')+ anchors = response.css("ul.pager a")yield from response.follow_all(anchors, callback=self.parse)-or, shortening it further::+or, shortening it further:++.. code-block:: python- yield from response.follow_all(css='ul.pager a', callback=self.parse)+ yield from response.follow_all(css="ul.pager a", callback=self.parse)More examples and patterns--------------------------Here is another spider that illustrates callbacks and following links,-this time for scraping author information::+this time for scraping author information:++.. code-block:: pythonimport scrapyclass AuthorSpider(scrapy.Spider):- name = 'author'+ name = "author"- start_urls = ['https://quotes.toscrape.com/']+ start_urls = ["https://quotes.toscrape.com/"]def parse(self, response):- author_page_links = response.css('.author + a')+ author_page_links = response.css(".author + a")yield from response.follow_all(author_page_links, self.parse_author)- pagination_links = response.css('li.next a')+ pagination_links = response.css("li.next a")yield from response.follow_all(pagination_links, self.parse)def parse_author(self, response):def extract_with_css(query):- return response.css(query).get(default='').strip()+ return response.css(query).get(default="").strip()yield {- 'name': extract_with_css('h3.author-title::text'),- 'birthdate': extract_with_css('.author-born-date::text'),- 'bio': extract_with_css('.author-description::text'),+ "name": extract_with_css("h3.author-title::text"),+ "birthdate": extract_with_css(".author-born-date::text"),+ "bio": extract_with_css(".author-description::text"),}This spider will start from the main page, it will follow all the links to the@@ -720,7 +738,9 @@ spider attributes by default.In this example, the value provided for the ``tag`` argument will be availablevia ``self.tag``. You can use this to make your spider fetch only quotes-with a specific tag, building the URL based on the argument::+with a specific tag, building the URL based on the argument:++.. code-block:: pythonimport scrapy@@ -729,20 +749,20 @@ with a specific tag, building the URL based on the argument::name = "quotes"def start_requests(self):- url = 'https://quotes.toscrape.com/'- tag = getattr(self, 'tag', None)+ url = "https://quotes.toscrape.com/"+ tag = getattr(self, "tag", None)if tag is not None:- url = url + 'tag/' + tag+ url = url + "tag/" + tagyield scrapy.Request(url, self.parse)def parse(self, response):- for quote in response.css('div.quote'):+ for quote in response.css("div.quote"):yield {- 'text': quote.css('span.text::text').get(),- 'author': quote.css('small.author::text').get(),+ "text": quote.css("span.text::text").get(),+ "author": quote.css("small.author::text").get(),}- next_page = response.css('li.next a::attr(href)').get()+ next_page = response.css("li.next a::attr(href)").get()if next_page is not None:yield response.follow(next_page, self.parse)diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex dbee7146d..7713b1af1 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -106,12 +106,14 @@ Enforcing asyncio as a requirementIf you are writing a :ref:`component <topics-components>` that requires asyncioto work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to:ref:`enforce it as a requirement <enforce-component-requirements>`. For-example::+example:++.. code-block:: pythonfrom scrapy.utils.reactor import is_asyncio_reactor_installed- class MyComponent:+ class MyComponent:def __init__(self):if not is_asyncio_reactor_installed():raise ValueError(diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rstindex 0927ac2d2..8be89feb2 100644--- a/docs/topics/broad-crawls.rst+++ b/docs/topics/broad-crawls.rst@@ -48,9 +48,11 @@ Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQIt works best during single-domain crawl. It does not work well with crawlingmany different domains in parallel-To apply the recommended priority queue use::+To apply the recommended priority queue use:- SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'+.. code-block:: python++ SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue".. _broad-crawls-concurrency:@@ -71,7 +73,9 @@ many different domains in parallel, so you will want to increase it. How muchto increase it will depend on how much CPU and memory your crawler will haveavailable.-A good starting point is ``100``::+A good starting point is ``100``:++.. code-block:: pythonCONCURRENT_REQUESTS = 100@@ -92,7 +96,9 @@ hitting DNS resolver timeouts. Possible solution to increase the number ofthreads handling DNS queries. The DNS queue will be processed faster speedingup establishing of connection and crawling overall.-To increase maximum thread pool size use::+To increase maximum thread pool size use:++.. code-block:: pythonREACTOR_THREADPOOL_MAXSIZE = 20@@ -114,9 +120,11 @@ should not use ``DEBUG`` log level when preforming large broad crawls inproduction. Using ``DEBUG`` level when developing your (broad) crawler may befine though.-To set the log level use::+To set the log level use:- LOG_LEVEL = 'INFO'+.. code-block:: python++ LOG_LEVEL = "INFO"Disable cookies===============@@ -126,7 +134,9 @@ doing broad crawls (search engine crawlers ignore them), and they improveperformance by saving some CPU cycles and reducing the memory footprint of yourScrapy crawler.-To disable cookies use::+To disable cookies use:++.. code-block:: pythonCOOKIES_ENABLED = False@@ -138,7 +148,9 @@ when sites causes are very slow (or fail) to respond, thus causing a timeouterror which gets retried many times, unnecessarily, preventing crawler capacityto be reused for other domains.-To disable retries use::+To disable retries use:++.. code-block:: pythonRETRY_ENABLED = False@@ -149,7 +161,9 @@ Unless you are crawling from a very slow connection (which shouldn't be thecase for broad crawls) reduce the download timeout so that stuck requests arediscarded quickly and free up capacity to process the next ones.-To reduce the download timeout use::+To reduce the download timeout use:++.. code-block:: pythonDOWNLOAD_TIMEOUT = 15@@ -162,7 +176,9 @@ revisiting the site at a later crawl. This also help to keep the number ofrequest constant per crawl batch, otherwise redirect loops may cause thecrawler to dedicate too many resources on any specific domain.-To disable redirects use::+To disable redirects use:++.. code-block:: pythonREDIRECT_ENABLED = False@@ -179,7 +195,9 @@ Pages can indicate it in two ways:"main", "index" website pages.Scrapy handles (1) automatically; to handle (2) enable-:ref:`AjaxCrawlMiddleware <ajaxcrawl-middleware>`::+:ref:`AjaxCrawlMiddleware <ajaxcrawl-middleware>`:++.. code-block:: pythonAJAXCRAWL_ENABLED = Truediff --git a/docs/topics/commands.rst b/docs/topics/commands.rstindex 362190116..54fd5d663 100644--- a/docs/topics/commands.rst+++ b/docs/topics/commands.rst@@ -617,7 +617,7 @@ Example:.. code-block:: python- COMMANDS_MODULE = 'mybot.commands'+ COMMANDS_MODULE = "mybot.commands".. _Deploying your project: https://scrapyd.readthedocs.io/en/latest/deploy.html@@ -636,10 +636,11 @@ The following example adds ``my_command`` command:from setuptools import setup, find_packages- setup(name='scrapy-mymodule',- entry_points={- 'scrapy.commands': [- 'my_command=my_scrapy_module.commands:MyCommand',- ],- },- )+ setup(+ name="scrapy-mymodule",+ entry_points={+ "scrapy.commands": [+ "my_command=my_scrapy_module.commands:MyCommand",+ ],+ },+ )diff --git a/docs/topics/components.rst b/docs/topics/components.rstindex ca301b827..1ed55f000 100644--- a/docs/topics/components.rst+++ b/docs/topics/components.rst@@ -66,16 +66,18 @@ version mismatch, while :exc:`ValueError` may be better if the issue is thevalue of a setting.If your requirement is a minimum Scrapy version, you may use-:attr:`scrapy.__version__` to enforce your requirement. For example::+:attr:`scrapy.__version__` to enforce your requirement. For example:++.. code-block:: pythonfrom pkg_resources import parse_versionimport scrapy- class MyComponent:+ class MyComponent:def __init__(self):- if parse_version(scrapy.__version__) < parse_version('2.7'):+ if parse_version(scrapy.__version__) < parse_version("2.7"):raise RuntimeError(f"{MyComponent.__qualname__} requires Scrapy 2.7 or "f"later, which allow defining the process_spider_output "diff --git a/docs/topics/contracts.rst b/docs/topics/contracts.rstindex c29a3a410..211a0f5f2 100644--- a/docs/topics/contracts.rst+++ b/docs/topics/contracts.rst@@ -11,10 +11,13 @@ integrated way of testing your spiders by the means of contracts.This allows you to test each callback of your spider by hardcoding a sample urland check various constraints for how the callback processes the response. Eachcontract is prefixed with an ``@`` and included in the docstring. See the-following example::+following example:++.. code-block:: pythondef parse(self, response):- """ This function parses a sample response. Some contracts are mingled+ """+ This function parses a sample response. Some contracts are mingledwith this docstring.@url http://www.amazon.com/s?field-keywords=selfish+gene@@ -64,11 +67,13 @@ Custom ContractsIf you find you need more power than the built-in Scrapy contracts you cancreate and load your own contracts in the project by using the-:setting:`SPIDER_CONTRACTS` setting::+:setting:`SPIDER_CONTRACTS` setting:++.. code-block:: pythonSPIDER_CONTRACTS = {- 'myproject.contracts.ResponseCheck': 10,- 'myproject.contracts.ItemValidate': 10,+ "myproject.contracts.ResponseCheck": 10,+ "myproject.contracts.ItemValidate": 10,}Each contract must inherit from :class:`~scrapy.contracts.Contract` and can@@ -111,22 +116,26 @@ Raise :class:`~scrapy.exceptions.ContractFail` from.. autoclass:: scrapy.exceptions.ContractFailHere is a demo contract which checks the presence of a custom header in the-response received::+response received:++.. code-block:: pythonfrom scrapy.contracts import Contractfrom scrapy.exceptions import ContractFail+class HasHeaderContract(Contract):- """ Demo contract which checks the presence of a custom header- @has_header X-CustomHeader+ """+ Demo contract which checks the presence of a custom header+ @has_header X-CustomHeader"""- name = 'has_header'+ name = "has_header"def pre_process(self, response):for header in self.args:if header not in response.headers:- raise ContractFail('X-CustomHeader not present')+ raise ContractFail("X-CustomHeader not present").. _detecting-contract-check-runs:@@ -135,14 +144,17 @@ Detecting check runsWhen ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable isset to the ``true`` string. You can use :data:`os.environ` to perform any change to-your spiders or your settings when ``scrapy check`` is used::+your spiders or your settings when ``scrapy check`` is used:++.. code-block:: pythonimport osimport scrapy+class ExampleSpider(scrapy.Spider):- name = 'example'+ name = "example"def __init__(self):- if os.environ.get('SCRAPY_CHECK'):+ if os.environ.get("SCRAPY_CHECK"):pass # Do some scraper adjustments when a check is runningdiff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex a1ba4ba5c..a0c005204 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -58,49 +58,58 @@ There are several use cases for coroutines in Scrapy.Code that would return Deferreds when written for previous Scrapy versions,such as downloader middlewares and signal handlers, can be rewritten to be-shorter and cleaner::+shorter and cleaner:++.. code-block:: pythonfrom itemadapter import ItemAdapter+class DbPipeline:def _update_item(self, data, item):adapter = ItemAdapter(item)- adapter['field'] = data+ adapter["field"] = datareturn itemdef process_item(self, item, spider):adapter = ItemAdapter(item)- dfd = db.get_some_data(adapter['id'])+ dfd = db.get_some_data(adapter["id"])dfd.addCallback(self._update_item, item)return dfd-becomes::+becomes:++.. code-block:: pythonfrom itemadapter import ItemAdapter+class DbPipeline:async def process_item(self, item, spider):adapter = ItemAdapter(item)- adapter['field'] = await db.get_some_data(adapter['id'])+ adapter["field"] = await db.get_some_data(adapter["id"])return itemCoroutines may be used to call asynchronous code. This includes othercoroutines, functions that return Deferreds and functions that return:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.-This means you can use many useful Python libraries providing such code::+This means you can use many useful Python libraries providing such code:++.. code-block:: pythonclass MySpiderDeferred(Spider):# ...async def parse(self, response):- additional_response = await treq.get('https://additional.url')+ additional_response = await treq.get("https://additional.url")additional_data = await treq.content(additional_response)# ... use response and additional_data to yield items and requests+class MySpiderAsyncio(Spider):# ...async def parse(self, response):async with aiohttp.ClientSession() as session:- async with session.get('https://additional.url') as additional_response:+ async with session.get("https://additional.url") as additional_response:additional_data = await additional_response.text()# ... use response and additional_data to yield items and requests@@ -192,7 +201,9 @@ while maintaining support for older Scrapy versions, you may define:term:`asynchronous generator` version of that method with an alternative name:``process_spider_output_async``.-For example::+For example:++.. code-block:: pythonclass UniversalSpiderMiddleware:def process_spider_output(self, response, result, spider):diff --git a/docs/topics/debug.rst b/docs/topics/debug.rstindex edbcaf432..b133fcc1e 100644--- a/docs/topics/debug.rst+++ b/docs/topics/debug.rst@@ -5,21 +5,24 @@ Debugging Spiders=================This document explains the most common techniques for debugging spiders.-Consider the following Scrapy spider below::+Consider the following Scrapy spider below:++.. code-block:: pythonimport scrapyfrom myproject.items import MyItem+class MySpider(scrapy.Spider):- name = 'myspider'+ name = "myspider"start_urls = (- 'http://example.com/page1',- 'http://example.com/page2',- )+ "http://example.com/page1",+ "http://example.com/page2",+ )def parse(self, response):# <processing code not shown>- # collect `item_urls`+ # collect `item_urls`for item_url in item_urls:yield scrapy.Request(item_url, self.parse_item)@@ -28,7 +31,9 @@ Consider the following Scrapy spider below::item = MyItem()# populate `item` fields# and extract item_details_url- yield scrapy.Request(item_details_url, self.parse_details, cb_kwargs={'item': item})+ yield scrapy.Request(+ item_details_url, self.parse_details, cb_kwargs={"item": item}+ )def parse_details(self, response, item):# populate more `item` fields@@ -103,10 +108,13 @@ showing the response received and the output. How to debug the situation when.. highlight:: pythonFortunately, the :command:`shell` is your bread and butter in this case (see-:ref:`topics-shell-inspect-response`)::+:ref:`topics-shell-inspect-response`):++.. code-block:: pythonfrom scrapy.shell import inspect_response+def parse_details(self, response, item=None):if item:# populate more `item` fields@@ -121,10 +129,13 @@ Open in browserSometimes you just want to see how a certain response looks in a browser, youcan use the ``open_in_browser`` function for that. Here is an example of how-you would use it::+you would use it:++.. code-block:: pythonfrom scrapy.utils.response import open_in_browser+def parse_details(self, response):if "item name" not in response.body:open_in_browser(response)@@ -138,14 +149,16 @@ LoggingLogging is another useful option for getting information about your spider run.Although not as convenient, it comes with the advantage that the logs will be-available in all future runs should they be necessary again::+available in all future runs should they be necessary again:++.. code-block:: pythondef parse_details(self, response, item=None):if item:# populate more `item` fieldsreturn itemelse:- self.logger.warning('No item received for %s', response.url)+ self.logger.warning("No item received for %s", response.url)For more information, check the :ref:`topics-logging` section.diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rstindex 9bf97c628..39e7b7d3c 100644--- a/docs/topics/developer-tools.rst+++ b/docs/topics/developer-tools.rst@@ -237,17 +237,19 @@ on the request and open ``Open in new tab`` to get a better overview.:alt: JSON-object returned from the quotes.toscrape APIWith this response we can now easily parse the JSON-object and-also request each page to get every quote on the site::+also request each page to get every quote on the site:++.. code-block:: pythonimport scrapyimport jsonclass QuoteSpider(scrapy.Spider):- name = 'quote'- allowed_domains = ['quotes.toscrape.com']+ name = "quote"+ allowed_domains = ["quotes.toscrape.com"]page = 1- start_urls = ['https://quotes.toscrape.com/api/quotes?page=1']+ start_urls = ["https://quotes.toscrape.com/api/quotes?page=1"]def parse(self, response):data = json.loads(response.text)@@ -275,7 +277,9 @@ requests, as we could need to add ``headers`` or ``cookies`` to make it work.In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_format, by right-clicking on each of them in the network tool and using the:meth:`~scrapy.Request.from_curl()` method to generate an equivalent-request::+request:++.. code-block:: pythonfrom scrapy import Request@@ -286,7 +290,8 @@ request::"-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM""zEwZTAxLTk5MWUtNDFiNC1iZWRmLTJjNGI4M2ZiNDBmNDpAVEstMDMzMTBlMDEtOTkxZS00MW""I0LWJlZGYtMmM0YjgzZmI0MGY0' -H 'Connection: keep-alive' -H 'Referer: http"- "://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'")+ "://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'"+ )Alternatively, if you want to know the arguments needed to recreate thatrequest you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs`diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 986da0476..e1c481c37 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -17,10 +17,12 @@ To activate a downloader middleware component, add it to the:setting:`DOWNLOADER_MIDDLEWARES` setting, which is a dict whose keys are themiddleware class paths and their values are the middleware orders.-Here's an example::+Here's an example:++.. code-block:: pythonDOWNLOADER_MIDDLEWARES = {- 'myproject.middlewares.CustomDownloaderMiddleware': 543,+ "myproject.middlewares.CustomDownloaderMiddleware": 543,}The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the@@ -42,11 +44,13 @@ previous (or subsequent) middleware being applied.If you want to disable a built-in middleware (the ones defined in:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define itin your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None``-as its value. For example, if you want to disable the user-agent middleware::+as its value. For example, if you want to disable the user-agent middleware:++.. code-block:: pythonDOWNLOADER_MIDDLEWARES = {- 'myproject.middlewares.CustomDownloaderMiddleware': 543,- 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,+ "myproject.middlewares.CustomDownloaderMiddleware": 543,+ "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,}Finally, keep in mind that some middlewares may need to be enabled through a@@ -226,20 +230,25 @@ There is support for keeping multiple cookie sessions per spider by using the:reqmeta:`cookiejar` Request meta key. By default it uses a single cookie jar(session), but you can pass an identifier to use different ones.-For example::+For example:++.. code-block:: pythonfor i, url in enumerate(urls):- yield scrapy.Request(url, meta={'cookiejar': i},- callback=self.parse_page)+ yield scrapy.Request(url, meta={"cookiejar": i}, callback=self.parse_page)Keep in mind that the :reqmeta:`cookiejar` meta key is not "sticky". You need to keep-passing it along on subsequent requests. For example::+passing it along on subsequent requests. For example:++.. code-block:: pythondef parse_page(self, response):# do some processing- return scrapy.Request("http://www.example.com/otherpage",- meta={'cookiejar': response.meta['cookiejar']},- callback=self.parse_other_page)+ return scrapy.Request(+ "http://www.example.com/otherpage",+ meta={"cookiejar": response.meta["cookiejar"]},+ callback=self.parse_other_page,+ ).. setting:: COOKIES_ENABLED@@ -339,16 +348,19 @@ HttpAuthMiddlewaredomain of the first request, which will work for some spiders but notfor others. In the future the middleware will produce an error instead.- Example::+ Example:++ .. code-block:: pythonfrom scrapy.spiders import CrawlSpider+class SomeIntranetSiteSpider(CrawlSpider):- http_user = 'someuser'- http_pass = 'somepass'- http_auth_domain = 'intranet.example.com'- name = 'intranet.example.com'+ http_user = "someuser"+ http_pass = "somepass"+ http_auth_domain = "intranet.example.com"+ name = "intranet.example.com"# .. rest of the spider code omitted ...@@ -792,7 +804,9 @@ If you want to handle some redirect status codes in your spider, you canspecify these in the ``handle_httpstatus_list`` spider attribute.For example, if you want the redirect middleware to ignore 301 and 302-responses (and pass them through to your spider) you can do this::+responses (and pass them through to your spider) you can do this:++.. code-block:: pythonclass MySpider(CrawlSpider):handle_httpstatus_list = [301, 302]diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rstindex ea5d06210..9be0ed058 100644--- a/docs/topics/dynamic-content.rst+++ b/docs/topics/dynamic-content.rst@@ -119,16 +119,20 @@ data from it depends on the type of response:<topics-selectors>` as usual.- If the response is JSON, use :func:`json.loads` to load the desired data from- :attr:`response.text <scrapy.http.TextResponse.text>`::+ :attr:`response.text <scrapy.http.TextResponse.text>`:++ .. code-block:: pythondata = json.loads(response.text)If the desired data is inside HTML or XML code embedded within JSON data,you can load that HTML or XML code into a:class:`~scrapy.Selector` and then- :ref:`use it <topics-selectors>` as usual::+ :ref:`use it <topics-selectors>` as usual:++ .. code-block:: python- selector = Selector(data['html'])+ selector = Selector(data["html"])- If the response is JavaScript, or HTML with a ``<script/>`` elementcontaining the desired data, see :ref:`topics-parsing-javascript`.@@ -250,11 +254,14 @@ automation. By installing the :ref:`asyncio reactor <install-asyncio>`,it is possible to integrate ``asyncio``-based libraries which handle headless browsers.One such library is `playwright-python`_ (an official Python port of `playwright`_).-The following is a simple snippet to illustrate its usage within a Scrapy spider::+The following is a simple snippet to illustrate its usage within a Scrapy spider:++.. code-block:: pythonimport scrapyfrom playwright.async_api import async_playwright+class PlaywrightSpider(scrapy.Spider):name = "playwright"start_urls = ["data:,"] # avoid using the default Scrapy downloaderdiff --git a/docs/topics/email.rst b/docs/topics/email.rstindex 1a2bc6330..62a5a65bd 100644--- a/docs/topics/email.rst+++ b/docs/topics/email.rst@@ -19,19 +19,31 @@ Quick example=============There are two ways to instantiate the mail sender. You can instantiate it using-the standard ``__init__`` method::+the standard ``__init__`` method:++.. code-block:: pythonfrom scrapy.mail import MailSender+mailer = MailSender()Or you can instantiate it passing a Scrapy settings object, which will respect-the :ref:`settings <topics-email-settings>`::+the :ref:`settings <topics-email-settings>`:++.. code-block:: pythonmailer = MailSender.from_settings(settings)-And here is how to use it to send an e-mail (without attachments)::+And here is how to use it to send an e-mail (without attachments):++.. code-block:: python- mailer.send(to=["[email protected]"], subject="Some subject", body="Some body", cc=["[email protected]"])+ mailer.send(+ to=["[email protected]"],+ subject="Some subject",+ body="Some body",+ cc=["[email protected]"],+ )MailSender class reference==========================diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rstindex 9150ca7d9..ea64edbe6 100644--- a/docs/topics/exceptions.rst+++ b/docs/topics/exceptions.rst@@ -26,11 +26,13 @@ CloseSpider:param reason: the reason for closing:type reason: str-For example::+For example:++.. code-block:: pythondef parse_page(self, response):- if 'Bandwidth exceeded' in response.body:- raise CloseSpider('bandwidth_exceeded')+ if "Bandwidth exceeded" in response.body:+ raise CloseSpider("bandwidth_exceeded")DontCloseSpider---------------diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rstindex 9360ecf37..d09a6274f 100644--- a/docs/topics/exporters.rst+++ b/docs/topics/exporters.rst@@ -38,11 +38,14 @@ the end of the exporting processHere you can see an :doc:`Item Pipeline <item-pipeline>` which uses multipleItem Exporters to group scraped items to different files according to the-value of one of their fields::+value of one of their fields:++.. code-block:: pythonfrom itemadapter import ItemAdapterfrom scrapy.exporters import XmlItemExporter+class PerYearXmlExportPipeline:"""Distribute items across multiple XML files according to their 'year' field"""@@ -56,9 +59,9 @@ value of one of their fields::def _exporter_for_item(self, item):adapter = ItemAdapter(item)- year = adapter['year']+ year = adapter["year"]if year not in self.year_to_exporter:- xml_file = open(f'{year}.xml', 'wb')+ xml_file = open(f"{year}.xml", "wb")exporter = XmlItemExporter(xml_file)exporter.start_exporting()self.year_to_exporter[year] = (exporter, xml_file)@@ -94,12 +97,16 @@ If you use :class:`~scrapy.Item` you can declare a serializer in the:ref:`field metadata <topics-items-fields>`. The serializer must bea callable which receives a value and returns its serialized form.-Example::+Example:++.. code-block:: pythonimport scrapy+def serialize_price(value):- return f'$ {str(value)}'+ return f"$ {str(value)}"+class Product(scrapy.Item):name = scrapy.Field()@@ -115,15 +122,17 @@ customize how your field value will be exported.Make sure you call the base class :meth:`~BaseItemExporter.serialize_field()` methodafter your custom code.-Example::+Example:++.. code-block:: pythonfrom scrapy.exporters import XmlItemExporter- class ProductXmlExporter(XmlItemExporter):+ class ProductXmlExporter(XmlItemExporter):def serialize_field(self, field, name, value):- if name == 'price':- return f'$ {str(value)}'+ if name == "price":+ return f"$ {str(value)}"return super().serialize_field(field, name, value).. _topics-exporters-reference:@@ -132,10 +141,12 @@ Built-in Item Exporters reference=================================Here is a list of the Item Exporters bundled with Scrapy. Some of them contain-output examples, which assume you're exporting these two items::+output examples, which assume you're exporting these two items:++.. code-block:: python- Item(name='Color TV', price='1200')- Item(name='DVD player', price='200')+ Item(name="Color TV", price="1200")+ Item(name="DVD player", price="200")BaseItemExporter----------------diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex 130657b0b..96e0216b8 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -31,11 +31,13 @@ initialization code must be performed in the class ``__init__`` method.To make an extension available, add it to the :setting:`EXTENSIONS` setting inyour Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented-by a string: the full Python path to the extension's class name. For example::+by a string: the full Python path to the extension's class name. For example:++.. code-block:: pythonEXTENSIONS = {- 'scrapy.extensions.corestats.CoreStats': 500,- 'scrapy.extensions.telnet.TelnetConsole': 500,+ "scrapy.extensions.corestats.CoreStats": 500,+ "scrapy.extensions.telnet.TelnetConsole": 500,}@@ -64,10 +66,12 @@ Disabling an extensionIn order to disable an extension that comes enabled by default (i.e. thoseincluded in the :setting:`EXTENSIONS_BASE` setting) you must set its order to-``None``. For example::+``None``. For example:++.. code-block:: pythonEXTENSIONS = {- 'scrapy.extensions.corestats.CoreStats': None,+ "scrapy.extensions.corestats.CoreStats": None,}Writing your own extension@@ -98,7 +102,9 @@ in the previous section. This extension will log a message every time:The extension will be enabled through the ``MYEXT_ENABLED`` setting and thenumber of items will be specified through the ``MYEXT_ITEMCOUNT`` setting.-Here is the code of such extension::+Here is the code of such extension:++.. code-block:: pythonimport loggingfrom scrapy import signals@@ -106,8 +112,8 @@ Here is the code of such extension::logger = logging.getLogger(__name__)- class SpiderOpenCloseLogging:+ class SpiderOpenCloseLogging:def __init__(self, item_count):self.item_count = item_countself.items_scraped = 0@@ -116,11 +122,11 @@ Here is the code of such extension::def from_crawler(cls, crawler):# first check if the extension should be enabled and raise# NotConfigured otherwise- if not crawler.settings.getbool('MYEXT_ENABLED'):+ if not crawler.settings.getbool("MYEXT_ENABLED"):raise NotConfigured# get the number of items from settings- item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000)+ item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000)# instantiate the extension objectext = cls(item_count)diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex 8f96b1154..554fc2976 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -290,10 +290,11 @@ class, which is the default value of the ``item_filter`` :ref:`feed option <feedYou can create your own custom filtering class by implementing :class:`~scrapy.extensions.feedexport.ItemFilter`'smethod ``accepts`` and taking ``feed_options`` as an argument.-For instance::+For instance:- class MyCustomFilter:+.. code-block:: python+ class MyCustomFilter:def __init__(self, feed_options):self.feed_options = feed_options@@ -594,23 +595,27 @@ For a complete list of available values, access the `Canned ACL`_ section on AmaFEED_STORAGES_BASE-------------------Default::+Default:++.. code-block:: python{- '': 'scrapy.extensions.feedexport.FileFeedStorage',- 'file': 'scrapy.extensions.feedexport.FileFeedStorage',- 'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage',- 's3': 'scrapy.extensions.feedexport.S3FeedStorage',- 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',+ "": "scrapy.extensions.feedexport.FileFeedStorage",+ "file": "scrapy.extensions.feedexport.FileFeedStorage",+ "stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",+ "s3": "scrapy.extensions.feedexport.S3FeedStorage",+ "ftp": "scrapy.extensions.feedexport.FTPFeedStorage",}A dict containing the built-in feed storage backends supported by Scrapy. Youcan disable any of these backends by assigning ``None`` to their URI scheme in:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend-(without replacement), place this in your ``settings.py``::+(without replacement), place this in your ``settings.py``:++.. code-block:: pythonFEED_STORAGES = {- 'ftp': None,+ "ftp": None,}.. setting:: FEED_EXPORTERS@@ -628,26 +633,30 @@ serialization formats and the values are paths to :ref:`Item exporterFEED_EXPORTERS_BASE--------------------Default::+Default:++.. code-block:: python{- 'json': 'scrapy.exporters.JsonItemExporter',- 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',- 'jsonl': 'scrapy.exporters.JsonLinesItemExporter',- 'jl': 'scrapy.exporters.JsonLinesItemExporter',- 'csv': 'scrapy.exporters.CsvItemExporter',- 'xml': 'scrapy.exporters.XmlItemExporter',- 'marshal': 'scrapy.exporters.MarshalItemExporter',- 'pickle': 'scrapy.exporters.PickleItemExporter',+ "json": "scrapy.exporters.JsonItemExporter",+ "jsonlines": "scrapy.exporters.JsonLinesItemExporter",+ "jsonl": "scrapy.exporters.JsonLinesItemExporter",+ "jl": "scrapy.exporters.JsonLinesItemExporter",+ "csv": "scrapy.exporters.CsvItemExporter",+ "xml": "scrapy.exporters.XmlItemExporter",+ "marshal": "scrapy.exporters.MarshalItemExporter",+ "pickle": "scrapy.exporters.PickleItemExporter",}A dict containing the built-in feed exporters supported by Scrapy. You candisable any of these exporters by assigning ``None`` to their serializationformat in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter-(without replacement), place this in your ``settings.py``::+(without replacement), place this in your ``settings.py``:++.. code-block:: pythonFEED_EXPORTERS = {- 'csv': None,+ "csv": None,}@@ -677,7 +686,9 @@ generated:number by introducing leading zeroes as needed, use ``%(batch_id)05d``(e.g. ``3`` becomes ``00003``, ``123`` becomes ``00123``).-For instance, if your settings include::+For instance, if your settings include:++.. code-block:: pythonFEED_EXPORT_BATCH_ITEM_COUNT = 100@@ -746,16 +757,20 @@ The function signature should be as follows:For example, to include the :attr:`name <scrapy.Spider.name>` of thesource spider in the feed URI:-#. Define the following function somewhere in your project::+#. Define the following function somewhere in your project:++ .. code-block:: python# myproject/utils.pydef uri_params(params, spider):- return {**params, 'spider_name': spider.name}+ return {**params, "spider_name": spider.name}++#. Point :setting:`FEED_URI_PARAMS` to that function in your settings:-#. Point :setting:`FEED_URI_PARAMS` to that function in your settings::+ .. code-block:: python# myproject/settings.py- FEED_URI_PARAMS = 'myproject.utils.uri_params'+ FEED_URI_PARAMS = "myproject.utils.uri_params"#. Use ``%(spider_name)s`` in your feed URI::diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rstindex fa19d2f4c..791560fb5 100644--- a/docs/topics/item-pipeline.rst+++ b/docs/topics/item-pipeline.rst@@ -81,19 +81,23 @@ Price validation and dropping items with no pricesLet's take a look at the following hypothetical pipeline that adjusts the``price`` attribute for those items that do not include VAT(``price_excludes_vat`` attribute), and drops those items which don't-contain a price::+contain a price:++.. code-block:: pythonfrom itemadapter import ItemAdapterfrom scrapy.exceptions import DropItem++class PricePipeline:vat_factor = 1.15def process_item(self, item, spider):adapter = ItemAdapter(item)- if adapter.get('price'):- if adapter.get('price_excludes_vat'):- adapter['price'] = adapter['price'] * self.vat_factor+ if adapter.get("price"):+ if adapter.get("price_excludes_vat"):+ adapter["price"] = adapter["price"] * self.vat_factorreturn itemelse:raise DropItem(f"Missing price in {item}")@@ -104,16 +108,18 @@ Write items to a JSON lines fileThe following pipeline stores all scraped items (from all spiders) into asingle ``items.jsonl`` file, containing one item per line serialized in JSON-format::+format:++.. code-block:: pythonimport jsonfrom itemadapter import ItemAdapter- class JsonWriterPipeline:+ class JsonWriterPipeline:def open_spider(self, spider):- self.file = open('items.jsonl', 'w')+ self.file = open("items.jsonl", "w")def close_spider(self, spider):self.file.close()@@ -135,14 +141,17 @@ MongoDB address and database name are specified in Scrapy settings;MongoDB collection is named after item class.The main point of this example is to show how to use :meth:`from_crawler`-method and how to clean up the resources properly.::+method and how to clean up the resources properly.++.. code-block:: pythonimport pymongofrom itemadapter import ItemAdapter+class MongoPipeline:- collection_name = 'scrapy_items'+ collection_name = "scrapy_items"def __init__(self, mongo_uri, mongo_db):self.mongo_uri = mongo_uri@@ -151,8 +160,8 @@ method and how to clean up the resources properly.::@classmethoddef from_crawler(cls, crawler):return cls(- mongo_uri=crawler.settings.get('MONGO_URI'),- mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')+ mongo_uri=crawler.settings.get("MONGO_URI"),+ mongo_db=crawler.settings.get("MONGO_DATABASE", "items"),)def open_spider(self, spider):@@ -183,7 +192,7 @@ render a screenshot of the item URL. After the request response is downloaded,the item pipeline saves the screenshot to a file and adds the filename to theitem.-::+.. code-block:: pythonimport hashlibfrom pathlib import Path@@ -231,23 +240,24 @@ Duplicates filterA filter that looks for duplicate items, and drops those items that werealready processed. Let's say that our items have a unique id, but our spider-returns multiples items with the same id::+returns multiples items with the same id:+.. code-block:: pythonfrom itemadapter import ItemAdapterfrom scrapy.exceptions import DropItem- class DuplicatesPipeline:+ class DuplicatesPipeline:def __init__(self):self.ids_seen = set()def process_item(self, item, spider):adapter = ItemAdapter(item)- if adapter['id'] in self.ids_seen:+ if adapter["id"] in self.ids_seen:raise DropItem(f"Duplicate item found: {item!r}")else:- self.ids_seen.add(adapter['id'])+ self.ids_seen.add(adapter["id"])return item@@ -255,11 +265,13 @@ Activating an Item Pipeline component=====================================To activate an Item Pipeline component you must add its class to the-:setting:`ITEM_PIPELINES` setting, like in the following example::+:setting:`ITEM_PIPELINES` setting, like in the following example:++.. code-block:: pythonITEM_PIPELINES = {- 'myproject.pipelines.PricePipeline': 300,- 'myproject.pipelines.JsonWriterPipeline': 800,+ "myproject.pipelines.PricePipeline": 300,+ "myproject.pipelines.JsonWriterPipeline": 800,}The integer values you assign to classes in this setting determine thediff --git a/docs/topics/items.rst b/docs/topics/items.rstindex 167014381..7b6dfc5c1 100644--- a/docs/topics/items.rst+++ b/docs/topics/items.rst@@ -76,10 +76,13 @@ make it the most feature-complete item type::class:`Field` objects used in the :ref:`Item declaration<topics-items-declaring>`.-Example::+Example:++.. code-block:: pythonfrom scrapy.item import Item, Field+class CustomItem(Item):one_field = Field()another_field = Field()@@ -102,10 +105,13 @@ Additionally, ``dataclass`` items also allow to:* define custom field metadata through :func:`dataclasses.field`, which can be used to:ref:`customize serialization <topics-exporters-field-serialization>`.-Example::+Example:++.. code-block:: pythonfrom dataclasses import dataclass+@dataclassclass CustomItem:one_field: str@@ -133,10 +139,13 @@ Additionally, ``attr.s`` items also allow to:In order to use this type, the :doc:`attrs package <attrs:index>` needs to be installed.-Example::+Example:++.. code-block:: pythonimport attr+@attr.sclass CustomItem:one_field = attr.ib()@@ -152,10 +161,13 @@ Declaring Item subclasses-------------------------Item subclasses are declared using a simple class definition syntax and-:class:`Field` objects. Here is an example::+:class:`Field` objects. Here is an example:++.. code-block:: pythonimport scrapy+class Product(scrapy.Item):name = scrapy.Field()price = scrapy.Field()@@ -347,17 +359,21 @@ Extending Item subclassesYou can extend Items (to add more fields or to change some metadata for somefields) by declaring a subclass of your original Item.-For example::+For example:++.. code-block:: pythonclass DiscountedProduct(Product):discount_percent = scrapy.Field(serializer=str)discount_expiration_date = scrapy.Field()You can also extend field metadata by using the previous field metadata and-appending more values, or changing existing values, like this::+appending more values, or changing existing values, like this:++.. code-block:: pythonclass SpecificProduct(Product):- name = scrapy.Field(Product.fields['name'], serializer=my_serializer)+ name = scrapy.Field(Product.fields["name"], serializer=my_serializer)That adds (or replaces) the ``serializer`` metadata key for the ``name`` field,keeping all the previously existing metadata values.diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rstindex f16d306c7..c7fc1ea48 100644--- a/docs/topics/jobs.rst+++ b/docs/topics/jobs.rst@@ -51,11 +51,13 @@ loading that attribute from the job directory, when the spider starts andstops.Here's an example of a callback that uses the spider state (other spider code-is omitted for brevity)::+is omitted for brevity):++.. code-block:: pythondef parse_item(self, response):# parse item here- self.state['items_count'] = self.state.get('items_count', 0) + 1+ self.state["items_count"] = self.state.get("items_count", 0) + 1Persistence gotchas===================diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rstindex e12ad45e0..6bdc4bb9d 100644--- a/docs/topics/link-extractors.rst+++ b/docs/topics/link-extractors.rst@@ -18,7 +18,9 @@ through a set of :class:`~scrapy.spiders.Rule` objects.You can also use link extractors in regular spiders. For example, you can instantiate:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class-variable in your spider, and use it from your spider callbacks::+variable in your spider, and use it from your spider callbacks:++.. code-block:: pythondef parse(self, response):for link in self.link_extractor.extract_links(response):@@ -132,7 +134,9 @@ LxmlLinkExtractor.. highlight:: python- You can use the following function in ``process_value``::+ You can use the following function in ``process_value``:++ .. code-block:: pythondef process_value(value):m = re.search("javascript:goToPage\('(.*?)'", value)diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rstindex 0d63700c8..eaad926d1 100644--- a/docs/topics/loaders.rst+++ b/docs/topics/loaders.rst@@ -46,18 +46,21 @@ using a proper processing function.Here is a typical Item Loader usage in a :ref:`Spider <topics-spiders>`, usingthe :ref:`Product item <topics-items-declaring>` declared in the :ref:`Items-chapter <topics-items>`::+chapter <topics-items>`:++.. code-block:: pythonfrom scrapy.loader import ItemLoaderfrom myproject.items import Product+def parse(self, response):l = ItemLoader(item=Product(), response=response)- l.add_xpath('name', '//div[@class="product_name"]')- l.add_xpath('name', '//div[@class="product_title"]')- l.add_xpath('price', '//p[@id="price"]')- l.add_css('stock', 'p#stock')- l.add_value('last_updated', 'today') # you can also use literal values+ l.add_xpath("name", '//div[@class="product_name"]')+ l.add_xpath("name", '//div[@class="product_title"]')+ l.add_xpath("price", '//p[@id="price"]')+ l.add_css("stock", "p#stock")+ l.add_value("last_updated", "today") # you can also use literal valuesreturn l.load_item()By quickly looking at that code, we can see the ``name`` field is being@@ -93,11 +96,14 @@ will be populated incrementally using the loader's :meth:`~ItemLoader.add_xpath`:meth:`~ItemLoader.add_css` and :meth:`~ItemLoader.add_value` methods.One approach to overcome this is to define items using the-:func:`~dataclasses.field` function, with a ``default`` argument::+:func:`~dataclasses.field` function, with a ``default`` argument:++.. code-block:: pythonfrom dataclasses import dataclass, fieldfrom typing import Optional+@dataclassclass InventoryItem:name: Optional[str] = field(default=None)@@ -122,14 +128,16 @@ processor). The result of the output processor is the final value that getsassigned to the item.Let's see an example to illustrate how the input and output processors are-called for a particular field (the same applies for any other field)::+called for a particular field (the same applies for any other field):++.. code-block:: pythonl = ItemLoader(Product(), some_selector)- l.add_xpath('name', xpath1) # (1)- l.add_xpath('name', xpath2) # (2)- l.add_css('name', css) # (3)- l.add_value('name', 'test') # (4)- return l.load_item() # (5)+ l.add_xpath("name", xpath1) # (1)+ l.add_xpath("name", xpath2) # (2)+ l.add_css("name", css) # (3)+ l.add_value("name", "test") # (4)+ return l.load_item() # (5)So what happens is:@@ -184,11 +192,14 @@ processors <itemloaders:built-in-processors>` built-in for convenience.Declaring Item Loaders======================-Item Loaders are declared using a class definition syntax. Here is an example::+Item Loaders are declared using a class definition syntax. Here is an example:++.. code-block:: pythonfrom itemloaders.processors import TakeFirst, MapCompose, Joinfrom scrapy.loader import ItemLoader+class ProductLoader(ItemLoader):default_output_processor = TakeFirst()@@ -215,16 +226,20 @@ As seen in the previous section, input and output processors can be declared inthe Item Loader definition, and it's very common to declare input processorsthis way. However, there is one more place where you can specify the input andoutput processors to use: in the :ref:`Item Field <topics-items-fields>`-metadata. Here is an example::+metadata. Here is an example:++.. code-block:: pythonimport scrapyfrom itemloaders.processors import Join, MapCompose, TakeFirstfrom w3lib.html import remove_tags+def filter_price(value):if value.isdigit():return value+class Product(scrapy.Item):name = scrapy.Field(input_processor=MapCompose(remove_tags),@@ -263,10 +278,12 @@ declaring, instantiating or using Item Loader. They are used to modify thebehaviour of the input/output processors.For example, suppose you have a function ``parse_length`` which receives a text-value and extracts a length from it::+value and extracts a length from it:++.. code-block:: pythondef parse_length(text, loader_context):- unit = loader_context.get('unit', 'm')+ unit = loader_context.get("unit", "m")# ... length parsing code goes here ...return parsed_length@@ -278,22 +295,28 @@ function (``parse_length`` in this case) can thus use them.There are several ways to modify Item Loader context values:1. By modifying the currently active Item Loader context- (:attr:`~ItemLoader.context` attribute)::+ (:attr:`~ItemLoader.context` attribute):++ .. code-block:: pythonloader = ItemLoader(product)- loader.context['unit'] = 'cm'+ loader.context["unit"] = "cm"2. On Item Loader instantiation (the keyword arguments of Item Loader- ``__init__`` method are stored in the Item Loader context)::+ ``__init__`` method are stored in the Item Loader context):- loader = ItemLoader(product, unit='cm')+ .. code-block:: python++ loader = ItemLoader(product, unit="cm")3. On Item Loader declaration, for those input/output processors that supportinstantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of- them::+ them:++ .. code-block:: pythonclass ProductLoader(ItemLoader):- length_out = MapCompose(parse_length, unit='cm')+ length_out = MapCompose(parse_length, unit="cm")ItemLoader objects@@ -323,25 +346,29 @@ Example::Without nested loaders, you need to specify the full xpath (or css) for each valuethat you wish to extract.-Example::+Example:++.. code-block:: pythonloader = ItemLoader(item=Item())# load stuff not in the footer- loader.add_xpath('social', '//footer/a[@class = "social"]/@href')- loader.add_xpath('email', '//footer/a[@class = "email"]/@href')+ loader.add_xpath("social", '//footer/a[@class = "social"]/@href')+ loader.add_xpath("email", '//footer/a[@class = "email"]/@href')loader.load_item()Instead, you can create a nested loader with the footer selector and add valuesrelative to the footer. The functionality is the same but you avoid repeatingthe footer selector.-Example::+Example:++.. code-block:: pythonloader = ItemLoader(item=Item())# load stuff not in the footer- footer_loader = loader.nested_xpath('//footer')- footer_loader.add_xpath('social', 'a[@class = "social"]/@href')- footer_loader.add_xpath('email', 'a[@class = "email"]/@href')+ footer_loader = loader.nested_xpath("//footer")+ footer_loader.add_xpath("social", 'a[@class = "social"]/@href')+ footer_loader.add_xpath("email", 'a[@class = "email"]/@href')# no need to call footer_loader.load_item()loader.load_item()@@ -370,25 +397,32 @@ three dashes (e.g. ``---Plasma TV---``) and you don't want to end up scrapingthose dashes in the final product names.Here's how you can remove those dashes by reusing and extending the default-Product Item Loader (``ProductLoader``)::+Product Item Loader (``ProductLoader``):++.. code-block:: pythonfrom itemloaders.processors import MapComposefrom myproject.ItemLoaders import ProductLoader+def strip_dashes(x):- return x.strip('-')+ return x.strip("-")+class SiteSpecificLoader(ProductLoader):name_in = MapCompose(strip_dashes, ProductLoader.name_in)Another case where extending Item Loaders can be very helpful is when you havemultiple source formats, for example XML and HTML. In the XML version you may-want to remove ``CDATA`` occurrences. Here's an example of how to do it::+want to remove ``CDATA`` occurrences. Here's an example of how to do it:++.. code-block:: pythonfrom itemloaders.processors import MapComposefrom myproject.ItemLoaders import ProductLoaderfrom myproject.utils.xml import remove_cdata+class XmlProductLoader(ProductLoader):name_in = MapCompose(remove_cdata, ProductLoader.name_in)diff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex 3bf23d5f5..5e1de4425 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -39,16 +39,22 @@ How to log messages===================Here's a quick example of how to log a message using the ``logging.WARNING``-level::+level:++.. code-block:: pythonimport logging+logging.warning("This is a warning")There are shortcuts for issuing log messages on any of the standard 5 levels,and there's also a general ``logging.log`` method which takes a given level as-argument. If needed, the last example could be rewritten as::+argument. If needed, the last example could be rewritten as:++.. code-block:: pythonimport logging+logging.log(logging.WARNING, "This is a warning")On top of that, you can create different "loggers" to encapsulate messages. (For@@ -59,24 +65,33 @@ constructions.The previous examples use the root logger behind the scenes, which is a top levellogger where all messages are propagated to (unless otherwise specified). Using``logging`` helpers is merely a shortcut for getting the root logger-explicitly, so this is also an equivalent of the last snippets::+explicitly, so this is also an equivalent of the last snippets:++.. code-block:: pythonimport logging+logger = logging.getLogger()logger.warning("This is a warning")You can use a different logger just by getting its name with the-``logging.getLogger`` function::+``logging.getLogger`` function:++.. code-block:: pythonimport logging- logger = logging.getLogger('mycustomlogger')++ logger = logging.getLogger("mycustomlogger")logger.warning("This is a warning")Finally, you can ensure having a custom logger for any module you're working onby using the ``__name__`` variable, which is populated with current module's-path::+path:++.. code-block:: pythonimport logging+logger = logging.getLogger(__name__)logger.warning("This is a warning")@@ -94,33 +109,39 @@ Logging from Spiders====================Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider-instance, which can be accessed and used like this::+instance, which can be accessed and used like this:++.. code-block:: pythonimport scrapy+class MySpider(scrapy.Spider):- name = 'myspider'- start_urls = ['https://scrapy.org']+ name = "myspider"+ start_urls = ["https://scrapy.org"]def parse(self, response):- self.logger.info('Parse function called on %s', response.url)+ self.logger.info("Parse function called on %s", response.url)That logger is created using the Spider's name, but you can use any custom-Python logger you want. For example::+Python logger you want. For example:++.. code-block:: pythonimport loggingimport scrapy- logger = logging.getLogger('mycustomlogger')+ logger = logging.getLogger("mycustomlogger")+class MySpider(scrapy.Spider):- name = 'myspider'- start_urls = ['https://scrapy.org']+ name = "myspider"+ start_urls = ["https://scrapy.org"]def parse(self, response):- logger.info('Parse function called on %s', response.url)+ logger.info("Parse function called on %s", response.url).. _topics-logging-configuration:@@ -229,7 +250,9 @@ the crawl.Next, we can see that the message has INFO level. To hide itwe should set logging level for ``scrapy.spidermiddlewares.httperror``higher than INFO; next level after INFO is WARNING. It could be done-e.g. in the spider's ``__init__`` method::+e.g. in the spider's ``__init__`` method:++.. code-block:: pythonimport loggingimport scrapy@@ -238,7 +261,7 @@ e.g. in the spider's ``__init__`` method::class MySpider(scrapy.Spider):# ...def __init__(self, *args, **kwargs):- logger = logging.getLogger('scrapy.spidermiddlewares.httperror')+ logger = logging.getLogger("scrapy.spidermiddlewares.httperror")logger.setLevel(logging.WARNING)super().__init__(*args, **kwargs)@@ -249,43 +272,53 @@ You can also filter log records by :class:`~logging.LogRecord` data. Forexample, you can filter log records by message content using a substring ora regular expression. Create a :class:`logging.Filter` subclassand equip it with a regular expression pattern to-filter out unwanted messages::+filter out unwanted messages:++.. code-block:: pythonimport loggingimport re-++class ContentFilter(logging.Filter):def filter(self, record):- match = re.search(r'\d{3} [Ee]rror, retrying', record.message)+ match = re.search(r"\d{3} [Ee]rror, retrying", record.message)if match:return False-+A project-level filter may be attached to the roothandler created by Scrapy, this is a wieldy way tofilter all loggers in different parts of the project-(middlewares, spider, etc.)::+(middlewares, spider, etc.):- import logging- import scrapy+.. code-block:: python++ import logging+ import scrapy+++ class MySpider(scrapy.Spider):+ # ...+ def __init__(self, *args, **kwargs):+ for handler in logging.root.handlers:+ handler.addFilter(ContentFilter())- class MySpider(scrapy.Spider):- # ...- def __init__(self, *args, **kwargs):- for handler in logging.root.handlers:- handler.addFilter(ContentFilter())-Alternatively, you may choose a specific logger-and hide it without affecting other loggers::+and hide it without affecting other loggers:++.. code-block:: pythonimport loggingimport scrapy-++class MySpider(scrapy.Spider):# ...def __init__(self, *args, **kwargs):- logger = logging.getLogger('my_logger')+ logger = logging.getLogger("my_logger")logger.addFilter(ContentFilter())-++scrapy.utils.log module=======================@@ -306,14 +339,14 @@ scrapy.utils.log moduleso it is recommended to only use :func:`logging.basicConfig` together with:class:`~scrapy.crawler.CrawlerRunner`.- This is an example on how to redirect ``INFO`` or higher messages to a file::+ This is an example on how to redirect ``INFO`` or higher messages to a file:++ .. code-block:: pythonimport logginglogging.basicConfig(- filename='log.txt',- format='%(levelname)s: %(message)s',- level=logging.INFO+ filename="log.txt", format="%(levelname)s: %(message)s", level=logging.INFO)Refer to :ref:`run-from-script` for more details about using Scrapy thisdiff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex a528746b0..d7510ce0f 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -87,13 +87,17 @@ Enabling your Media PipelineTo enable your media pipeline you must first add it to your project:setting:`ITEM_PIPELINES` setting.-For Images Pipeline, use::+For Images Pipeline, use:- ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}+.. code-block:: python++ ITEM_PIPELINES = {"scrapy.pipelines.images.ImagesPipeline": 1}-For Files Pipeline, use::+For Files Pipeline, use:++.. code-block:: python- ITEM_PIPELINES = {'scrapy.pipelines.files.FilesPipeline': 1}+ ITEM_PIPELINES = {"scrapy.pipelines.files.FilesPipeline": 1}.. note::You can also use both the Files and Images Pipeline at the same time.@@ -103,13 +107,17 @@ Then, configure the target storage setting to a valid value that will be usedfor storing the downloaded images. Otherwise the pipeline will remain disabled,even if you include it in the :setting:`ITEM_PIPELINES` setting.-For the Files Pipeline, set the :setting:`FILES_STORE` setting::+For the Files Pipeline, set the :setting:`FILES_STORE` setting:- FILES_STORE = '/path/to/valid/dir'+.. code-block:: python++ FILES_STORE = "/path/to/valid/dir"-For the Images Pipeline, set the :setting:`IMAGES_STORE` setting::+For the Images Pipeline, set the :setting:`IMAGES_STORE` setting:- IMAGES_STORE = '/path/to/valid/dir'+.. code-block:: python++ IMAGES_STORE = "/path/to/valid/dir".. _topics-file-naming:@@ -157,10 +165,11 @@ By overriding ``file_path`` like this:import hashlib+def file_path(self, request, response=None, info=None, *, item=None):image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5)- image_perspective = request.url.split('/')[-2]- image_filename = f'{image_url_hash}_{image_perspective}.jpg'+ image_perspective = request.url.split("/")[-2]+ image_filename = f"{image_url_hash}_{image_perspective}.jpg"return image_filename@@ -233,30 +242,38 @@ If botocore_ >= 1.4.87 is installed, :setting:`FILES_STORE` and:setting:`IMAGES_STORE` can represent an Amazon S3 bucket. Scrapy willautomatically upload the files to the bucket.-For example, this is a valid :setting:`IMAGES_STORE` value::+For example, this is a valid :setting:`IMAGES_STORE` value:++.. code-block:: python- IMAGES_STORE = 's3://bucket/images'+ IMAGES_STORE = "s3://bucket/images"You can modify the Access Control List (ACL) policy used for the stored files,which is defined by the :setting:`FILES_STORE_S3_ACL` and:setting:`IMAGES_STORE_S3_ACL` settings. By default, the ACL is set to``private``. To make the files publicly available use the ``public-read``-policy::+policy:++.. code-block:: python- IMAGES_STORE_S3_ACL = 'public-read'+ IMAGES_STORE_S3_ACL = "public-read"For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.You can also use other S3-like storages. Storages like self-hosted `Minio`_ or`s3.scality`_. All you need to do is set endpoint option in you Scrapy-settings::+settings:- AWS_ENDPOINT_URL = 'http://minio.example.com:9000'+.. code-block:: python++ AWS_ENDPOINT_URL = "http://minio.example.com:9000"-For self-hosting you also might feel the need not to use SSL and not to verify SSL connection::+For self-hosting you also might feel the need not to use SSL and not to verify SSL connection:++.. code-block:: python- AWS_USE_SSL = False # or True (None by default)- AWS_VERIFY = False # or True (None by default)+ AWS_USE_SSL = False # or True (None by default)+ AWS_VERIFY = False # or True (None by default).. _botocore: https://github.com/boto/botocore.. _canned ACLs: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl@@ -277,10 +294,12 @@ bucket. Scrapy will automatically upload the files to the bucket. (requires `goo.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python-For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings::+For example, these are valid :setting:`IMAGES_STORE` and :setting:`GCS_PROJECT_ID` settings:- IMAGES_STORE = 'gs://bucket/images/'- GCS_PROJECT_ID = 'project_id'+.. code-block:: python++ IMAGES_STORE = "gs://bucket/images/"+ GCS_PROJECT_ID = "project_id"For information about authentication, see this `documentation`_.@@ -291,9 +310,11 @@ which is defined by the :setting:`FILES_STORE_GCS_ACL` and:setting:`IMAGES_STORE_GCS_ACL` settings. By default, the ACL is set to``''`` (empty string) which means that Cloud Storage applies the bucket's default object ACL to the object.To make the files publicly available use the ``publicRead``-policy::+policy:- IMAGES_STORE_GCS_ACL = 'publicRead'+.. code-block:: python++ IMAGES_STORE_GCS_ACL = "publicRead"For more information, see `Predefined ACLs`_ in the Google Cloud Platform Developer Guide.@@ -318,10 +339,13 @@ respectively), the pipeline will put the results under the respective fieldWhen using :ref:`item types <item-types>` for which fields are defined beforehand,you must define both the URLs field and the results field. For example, whenusing the images pipeline, items must define both the ``image_urls`` and the-``images`` field. For instance, using the :class:`~scrapy.Item` class::+``images`` field. For instance, using the :class:`~scrapy.Item` class:++.. code-block:: pythonimport scrapy+class MyItem(scrapy.Item):# ... other item fields ...image_urls = scrapy.Field()@@ -331,16 +355,20 @@ If you want to use another field name for the URLs key or for the results key,it is also possible to override it.For the Files Pipeline, set :setting:`FILES_URLS_FIELD` and/or-:setting:`FILES_RESULT_FIELD` settings::+:setting:`FILES_RESULT_FIELD` settings:++.. code-block:: python- FILES_URLS_FIELD = 'field_name_for_your_files_urls'- FILES_RESULT_FIELD = 'field_name_for_your_processed_files'+ FILES_URLS_FIELD = "field_name_for_your_files_urls"+ FILES_RESULT_FIELD = "field_name_for_your_processed_files"For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or-:setting:`IMAGES_RESULT_FIELD` settings::+:setting:`IMAGES_RESULT_FIELD` settings:- IMAGES_URLS_FIELD = 'field_name_for_your_images_urls'- IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images'+.. code-block:: python++ IMAGES_URLS_FIELD = "field_name_for_your_images_urls"+ IMAGES_RESULT_FIELD = "field_name_for_your_processed_images"If you need something more complex and want to override the custom pipelinebehaviour, see :ref:`topics-media-pipeline-override`.@@ -366,7 +394,9 @@ File expirationThe Image Pipeline avoids downloading files that were downloaded recently. Toadjust this retention delay use the :setting:`FILES_EXPIRES` setting (or:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which-specifies the delay in number of days::+specifies the delay in number of days:++.. code-block:: python# 120 days of delay for files expirationFILES_EXPIRES = 120@@ -400,11 +430,13 @@ images.In order to use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionarywhere the keys are the thumbnail names and the values are their dimensions.-For example::+For example:++.. code-block:: pythonIMAGES_THUMBS = {- 'small': (50, 50),- 'big': (270, 270),+ "small": (50, 50),+ "big": (270, 270),}When you use this feature, the Images Pipeline will create thumbnails of the@@ -495,17 +527,19 @@ See here the methods that you can override in your custom Files Pipeline:For example, if file URLs end like regular paths (e.g.``https://example.com/a/b/c/foo.png``), you can use the followingapproach to download all files into the ``files`` folder with their- original filenames (e.g. ``files/foo.png``)::+ original filenames (e.g. ``files/foo.png``):++ .. code-block:: pythonfrom pathlib import PurePosixPathfrom urllib.parse import urlparsefrom scrapy.pipelines.files import FilesPipeline- class MyFilesPipeline(FilesPipeline):+ class MyFilesPipeline(FilesPipeline):def file_path(self, request, response=None, info=None, *, item=None):- return 'files/' + PurePosixPath(urlparse(request.url).path).name+ return "files/" + PurePosixPath(urlparse(request.url).path).nameSimilarly, you can use the ``item`` to determine the file path based on some itemproperty.@@ -521,13 +555,16 @@ See here the methods that you can override in your custom Files Pipeline:As seen on the workflow, the pipeline will get the URLs of the images todownload from the item. In order to do this, you can override the:meth:`~get_media_requests` method and return a Request for each- file URL::+ file URL:++ .. code-block:: pythonfrom itemadapter import ItemAdapter+def get_media_requests(self, item, info):adapter = ItemAdapter(item)- for file_url in adapter['file_urls']:+ for file_url in adapter["file_urls"]:yield scrapy.Request(file_url)Those requests will be processed by the pipeline and, when they have finished@@ -567,15 +604,22 @@ See here the methods that you can override in your custom Files Pipeline:guaranteed to retain the same order of the requests returned from the:meth:`~get_media_requests` method.- Here's a typical value of the ``results`` argument::+ Here's a typical value of the ``results`` argument:++ .. code-block:: python- [(True,- {'checksum': '2b00042f7481c7b056c4b410d28f33cf',- 'path': 'full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg',- 'url': 'http://www.example.com/files/product1.pdf',- 'status': 'downloaded'}),- (False,- Failure(...))]+ [+ (+ True,+ {+ "checksum": "2b00042f7481c7b056c4b410d28f33cf",+ "path": "full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg",+ "url": "http://www.example.com/files/product1.pdf",+ "status": "downloaded",+ },+ ),+ (False, Failure(...)),+ ]By default the :meth:`get_media_requests` method returns ``None`` whichmeans there are no files to download for the item.@@ -592,17 +636,20 @@ See here the methods that you can override in your custom Files Pipeline:Here is an example of the :meth:`~item_completed` method where westore the downloaded file paths (passed in results) in the ``file_paths``- item field, and we drop the item if it doesn't contain any files::+ item field, and we drop the item if it doesn't contain any files:++ .. code-block:: pythonfrom itemadapter import ItemAdapterfrom scrapy.exceptions import DropItem+def item_completed(self, results, item, info):- file_paths = [x['path'] for ok, x in results if ok]+ file_paths = [x["path"] for ok, x in results if ok]if not file_paths:raise DropItem("Item contains no files")adapter = ItemAdapter(item)- adapter['file_paths'] = file_paths+ adapter["file_paths"] = file_pathsreturn itemBy default, the :meth:`item_completed` method returns the item.@@ -634,17 +681,19 @@ See here the methods that you can override in your custom Images Pipeline:For example, if file URLs end like regular paths (e.g.``https://example.com/a/b/c/foo.png``), you can use the followingapproach to download all files into the ``files`` folder with their- original filenames (e.g. ``files/foo.png``)::+ original filenames (e.g. ``files/foo.png``):++ .. code-block:: pythonfrom pathlib import PurePosixPathfrom urllib.parse import urlparsefrom scrapy.pipelines.images import ImagesPipeline- class MyImagesPipeline(ImagesPipeline):+ class MyImagesPipeline(ImagesPipeline):def file_path(self, request, response=None, info=None, *, item=None):- return 'files/' + PurePosixPath(urlparse(request.url).path).name+ return "files/" + PurePosixPath(urlparse(request.url).path).nameSimilarly, you can use the ``item`` to determine the file path based on some itemproperty.@@ -700,33 +749,35 @@ Custom Images pipeline example==============================Here is a full example of the Images Pipeline whose methods are exemplified-above::+above:++.. code-block:: pythonimport scrapyfrom itemadapter import ItemAdapterfrom scrapy.exceptions import DropItemfrom scrapy.pipelines.images import ImagesPipeline- class MyImagesPipeline(ImagesPipeline):+ class MyImagesPipeline(ImagesPipeline):def get_media_requests(self, item, info):- for image_url in item['image_urls']:+ for image_url in item["image_urls"]:yield scrapy.Request(image_url)def item_completed(self, results, item, info):- image_paths = [x['path'] for ok, x in results if ok]+ image_paths = [x["path"] for ok, x in results if ok]if not image_paths:raise DropItem("Item contains no images")adapter = ItemAdapter(item)- adapter['image_paths'] = image_paths+ adapter["image_paths"] = image_pathsreturn itemTo enable your custom media pipeline component you must add its class import path to the-:setting:`ITEM_PIPELINES` setting, like in the following example::+:setting:`ITEM_PIPELINES` setting, like in the following example:- ITEM_PIPELINES = {- 'myproject.pipelines.MyImagesPipeline': 300- }+.. code-block:: python++ ITEM_PIPELINES = {"myproject.pipelines.MyImagesPipeline": 300}.. _MD5 hash: https://en.wikipedia.org/wiki/MD5diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex 7313c9246..69e2d4f5e 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -25,23 +25,27 @@ the one used by all Scrapy commands.Here's an example showing how to run a single spider with it.-::+.. code-block:: pythonimport scrapyfrom scrapy.crawler import CrawlerProcess+class MySpider(scrapy.Spider):# Your spider definition...- process = CrawlerProcess(settings={- "FEEDS": {- "items.json": {"format": "json"},- },- })++ process = CrawlerProcess(+ settings={+ "FEEDS": {+ "items.json": {"format": "json"},+ },+ }+ )process.crawl(MySpider)- process.start() # the script will block here until the crawling is finished+ process.start() # the script will block here until the crawling is finishedDefine settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess`documentation to get acquainted with its usage details.@@ -55,7 +59,7 @@ instance with your project settings.What follows is a working example of how to do that, using the `testspiders`_project as example.-::+.. code-block:: pythonfrom scrapy.crawler import CrawlerProcessfrom scrapy.utils.project import get_project_settings@@ -63,8 +67,8 @@ project as example.process = CrawlerProcess(get_project_settings())# 'followall' is the name of one of the spiders of the project.- process.crawl('followall', domain='scrapy.org')- process.start() # the script will block here until the crawling is finished+ process.crawl("followall", domain="scrapy.org")+ process.start() # the script will block here until the crawling is finishedThere's another Scrapy utility that provides more control over the crawlingprocess: :class:`scrapy.crawler.CrawlerRunner`. This class is a thin wrapper@@ -84,23 +88,25 @@ returned by the :meth:`CrawlerRunner.crawlHere's an example of its usage, along with a callback to manually stop thereactor after ``MySpider`` has finished running.-::+.. code-block:: pythonfrom twisted.internet import reactorimport scrapyfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.log import configure_logging+class MySpider(scrapy.Spider):# Your spider definition...- configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})++ configure_logging({"LOG_FORMAT": "%(levelname)s: %(message)s"})runner = CrawlerRunner()d = runner.crawl(MySpider)d.addBoth(lambda _: reactor.stop())- reactor.run() # the script will block here until the crawling is finished+ reactor.run() # the script will block here until the crawling is finished.. seealso:: :doc:`twisted:core/howto/reactor-basics`@@ -115,29 +121,32 @@ the :ref:`internal API <topics-api>`.Here is an example that runs multiple spiders simultaneously:-::+.. code-block:: pythonimport scrapyfrom scrapy.crawler import CrawlerProcessfrom scrapy.utils.project import get_project_settings+class MySpider1(scrapy.Spider):# Your first spider definition...+class MySpider2(scrapy.Spider):# Your second spider definition...+settings = get_project_settings()process = CrawlerProcess(settings)process.crawl(MySpider1)process.crawl(MySpider2)- process.start() # the script will block here until all crawling jobs are finished+ process.start() # the script will block here until all crawling jobs are finishedSame example using :class:`~scrapy.crawler.CrawlerRunner`:-::+.. code-block:: pythonimport scrapyfrom twisted.internet import reactor@@ -145,14 +154,17 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:from scrapy.utils.log import configure_loggingfrom scrapy.utils.project import get_project_settings+class MySpider1(scrapy.Spider):# Your first spider definition...+class MySpider2(scrapy.Spider):# Your second spider definition...+configure_logging()settings = get_project_settings()runner = CrawlerRunner(settings)@@ -161,37 +173,42 @@ Same example using :class:`~scrapy.crawler.CrawlerRunner`:d = runner.join()d.addBoth(lambda _: reactor.stop())- reactor.run() # the script will block here until all crawling jobs are finished+ reactor.run() # the script will block here until all crawling jobs are finishedSame example but running the spiders sequentially by chaining the deferreds:-::+.. code-block:: pythonfrom twisted.internet import reactor, deferfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.log import configure_loggingfrom scrapy.utils.project import get_project_settings+class MySpider1(scrapy.Spider):# Your first spider definition...+class MySpider2(scrapy.Spider):# Your second spider definition...+settings = get_project_settings()configure_logging(settings)runner = CrawlerRunner(settings)+@defer.inlineCallbacksdef crawl():yield runner.crawl(MySpider1)yield runner.crawl(MySpider2)reactor.stop()+crawl()- reactor.run() # the script will block here until the last crawl call is finished+ reactor.run() # the script will block here until the last crawl call is finishedDifferent spiders can set different values for the same setting, but when theyrun in the same process it may be impossible, by design or because of somediff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 766710d66..090554e6e 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -78,23 +78,27 @@ Request objects:param cookies: the request cookies. These can be sent in two forms.- 1. Using a dict::+ 1. Using a dict:++ .. code-block:: pythonrequest_with_cookies = Request(url="http://www.example.com",- cookies={'currency': 'USD', 'country': 'UY'},+ cookies={"currency": "USD", "country": "UY"},)- 2. Using a list of dicts::+ 2. Using a list of dicts:++ .. code-block:: pythonrequest_with_cookies = Request(url="http://www.example.com",cookies=[{- 'name': 'currency',- 'value': 'USD',- 'domain': 'example.com',- 'path': '/currency',+ "name": "currency",+ "value": "USD",+ "domain": "example.com",+ "path": "/currency",},],)@@ -114,12 +118,14 @@ Request objectsin :attr:`request.meta <scrapy.Request.meta>`.Example of a request that sends manually-defined cookies and ignores- cookie storage::+ cookie storage:++ .. code-block:: pythonRequest(url="http://www.example.com",- cookies={'currency': 'USD', 'country': 'UY'},- meta={'dont_merge_cookies': True},+ cookies={"currency": "USD", "country": "UY"},+ meta={"dont_merge_cookies": True},)For more info see :ref:`cookies-mw`.@@ -259,11 +265,15 @@ The callback of a request is a function that will be called when the responseof that request is downloaded. The callback function will be called with thedownloaded :class:`Response` object as its first argument.-Example::+Example:++.. code-block:: pythondef parse_page1(self, response):- return scrapy.Request("http://www.example.com/some_page.html",- callback=self.parse_page2)+ return scrapy.Request(+ "http://www.example.com/some_page.html", callback=self.parse_page2+ )+def parse_page2(self, response):# this would log http://www.example.com/some_page.html@@ -274,15 +284,18 @@ functions so you can receive the arguments later, in the second callback.The following example shows how to achieve this by using the:attr:`Request.cb_kwargs` attribute:-::+.. code-block:: pythondef parse(self, response):- request = scrapy.Request('http://www.example.com/index.html',- callback=self.parse_page2,- cb_kwargs=dict(main_url=response.url))- request.cb_kwargs['foo'] = 'bar' # add more arguments for the callback+ request = scrapy.Request(+ "http://www.example.com/index.html",+ callback=self.parse_page2,+ cb_kwargs=dict(main_url=response.url),+ )+ request.cb_kwargs["foo"] = "bar" # add more arguments for the callbackyield request+def parse_page2(self, response, main_url, foo):yield dict(main_url=main_url,@@ -308,7 +321,9 @@ It receives a :exc:`~twisted.python.failure.Failure` as first parameter and canbe used to track connection establishment timeouts, DNS errors etc.Here's an example spider logging all errors and catching some specific-errors if needed::+errors if needed++.. code-block:: pythonimport scrapy@@ -316,24 +331,28 @@ errors if needed::from twisted.internet.error import DNSLookupErrorfrom twisted.internet.error import TimeoutError, TCPTimedOutError+class ErrbackSpider(scrapy.Spider):name = "errback_example"start_urls = [- "http://www.httpbin.org/", # HTTP 200 expected- "http://www.httpbin.org/status/404", # Not found error- "http://www.httpbin.org/status/500", # server issue- "http://www.httpbin.org:12345/", # non-responding host, timeout expected- "https://example.invalid/", # DNS error expected+ "http://www.httpbin.org/", # HTTP 200 expected+ "http://www.httpbin.org/status/404", # Not found error+ "http://www.httpbin.org/status/500", # server issue+ "http://www.httpbin.org:12345/", # non-responding host, timeout expected+ "https://example.invalid/", # DNS error expected]def start_requests(self):for u in self.start_urls:- yield scrapy.Request(u, callback=self.parse_httpbin,- errback=self.errback_httpbin,- dont_filter=True)+ yield scrapy.Request(+ u,+ callback=self.parse_httpbin,+ errback=self.errback_httpbin,+ dont_filter=True,+ )def parse_httpbin(self, response):- self.logger.info('Got successful response from {}'.format(response.url))+ self.logger.info("Got successful response from {}".format(response.url))# do something useful here...def errback_httpbin(self, failure):@@ -347,16 +366,16 @@ errors if needed::# these exceptions come from HttpError spider middleware# you can get the non-200 responseresponse = failure.value.response- self.logger.error('HttpError on %s', response.url)+ self.logger.error("HttpError on %s", response.url)elif failure.check(DNSLookupError):# this is the original requestrequest = failure.request- self.logger.error('DNSLookupError on %s', request.url)+ self.logger.error("DNSLookupError on %s", request.url)elif failure.check(TimeoutError, TCPTimedOutError):request = failure.request- self.logger.error('TimeoutError on %s', request.url)+ self.logger.error("TimeoutError on %s", request.url).. _errback-cb_kwargs:@@ -367,21 +386,27 @@ Accessing additional data in errback functionsIn case of a failure to process the request, you may be interested inaccessing arguments to the callback functions so you can process furtherbased on the arguments in the errback. The following example shows how to-achieve this by using ``Failure.request.cb_kwargs``::+achieve this by using ``Failure.request.cb_kwargs``++.. code-block:: pythondef parse(self, response):- request = scrapy.Request('http://www.example.com/index.html',- callback=self.parse_page2,- errback=self.errback_page2,- cb_kwargs=dict(main_url=response.url))+ request = scrapy.Request(+ "http://www.example.com/index.html",+ callback=self.parse_page2,+ errback=self.errback_page2,+ cb_kwargs=dict(main_url=response.url),+ )yield request+def parse_page2(self, response, main_url):pass+def errback_page2(self, failure):yield dict(- main_url=failure.request.cb_kwargs['main_url'],+ main_url=failure.request.cb_kwargs["main_url"],)@@ -528,18 +553,20 @@ in your :meth:`fingerprint` method implementation:.. autofunction:: scrapy.utils.request.fingerprintFor example, to take the value of a request header named ``X-ID`` into-account::+account:++.. code-block:: python# my_project/settings.py- REQUEST_FINGERPRINTER_CLASS = 'my_project.utils.RequestFingerprinter'+ REQUEST_FINGERPRINTER_CLASS = "my_project.utils.RequestFingerprinter"# my_project/utils.pyfrom scrapy.utils.request import fingerprint- class RequestFingerprinter:+ class RequestFingerprinter:def fingerprint(self, request):- return fingerprint(request, include_headers=['X-ID'])+ return fingerprint(request, include_headers=["X-ID"])You can also write your own fingerprinting logic from scratch.@@ -555,13 +582,16 @@ you use :class:`~weakref.WeakKeyDictionary` to cache request fingerprints:references to them in your cache dictionary.For example, to take into account only the URL of a request, without any prior-URL canonicalization or taking the request method or body into account::+URL canonicalization or taking the request method or body into account:++.. code-block:: pythonfrom hashlib import sha1from weakref import WeakKeyDictionaryfrom scrapy.utils.python import to_bytes+class RequestFingerprinter:cache = WeakKeyDictionary()@@ -577,21 +607,25 @@ If you need to be able to override the request fingerprinting for arbitraryrequests from your spider callbacks, you may implement a request fingerprinterthat reads fingerprints from :attr:`request.meta <scrapy.http.Request.meta>`when available, and then falls back to-:func:`scrapy.utils.request.fingerprint`. For example::+:func:`scrapy.utils.request.fingerprint`. For example:++.. code-block:: pythonfrom scrapy.utils.request import fingerprint- class RequestFingerprinter:+ class RequestFingerprinter:def fingerprint(self, request):- if 'fingerprint' in request.meta:- return request.meta['fingerprint']+ if "fingerprint" in request.meta:+ return request.meta["fingerprint"]return fingerprint(request)If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6without using the deprecated ``'2.6'`` value of the:setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` setting, use the following-request fingerprinter::+request fingerprinter:++.. code-block:: pythonfrom hashlib import sha1from weakref import WeakKeyDictionary@@ -599,6 +633,7 @@ request fingerprinter::from scrapy.utils.python import to_bytesfrom w3lib.url import canonicalize_url+class RequestFingerprinter:cache = WeakKeyDictionary()@@ -608,7 +643,7 @@ request fingerprinter::fp = sha1()fp.update(to_bytes(request.method))fp.update(to_bytes(canonicalize_url(request.url)))- fp.update(request.body or b'')+ fp.update(request.body or b"")self.cache[request] = fp.digest()return self.cache[request]@@ -737,7 +772,9 @@ Stopping the download of a ResponseRaising a :exc:`~scrapy.exceptions.StopDownload` exception from a handler for the:class:`~scrapy.signals.bytes_received` or :class:`~scrapy.signals.headers_received`-signals will stop the download of a given response. See the following example::+signals will stop the download of a given response. See the following example:++.. code-block:: pythonimport scrapy@@ -749,7 +786,9 @@ signals will stop the download of a given response. See the following example::@classmethoddef from_crawler(cls, crawler):spider = super().from_crawler(crawler)- crawler.signals.connect(spider.on_bytes_received, signal=scrapy.signals.bytes_received)+ crawler.signals.connect(+ spider.on_bytes_received, signal=scrapy.signals.bytes_received+ )return spiderdef parse(self, response):@@ -878,11 +917,17 @@ Using FormRequest to send data via HTTP POSTIf you want to simulate a HTML Form POST in your spider and send a couple ofkey-value fields, you can return a :class:`FormRequest` object (from your-spider) like this::+spider) like this:++.. code-block:: python- return [FormRequest(url="http://www.example.com/post/action",- formdata={'name': 'John Doe', 'age': '27'},- callback=self.after_post)]+ return [+ FormRequest(+ url="http://www.example.com/post/action",+ formdata={"name": "John Doe", "age": "27"},+ callback=self.after_post,+ )+ ].. _topics-request-response-ref-request-userlogin:@@ -894,25 +939,28 @@ type="hidden">`` elements, such as session related data or authenticationtokens (for login pages). When scraping, you'll want these fields to beautomatically pre-populated and only override a couple of them, such as theuser name and password. You can use the :meth:`FormRequest.from_response`-method for this job. Here's an example spider which uses it::+method for this job. Here's an example spider which uses it:+.. code-block:: pythonimport scrapy+def authentication_failed(response):# TODO: Check the contents of the response and return True if it failed# or False if it succeeded.pass+class LoginSpider(scrapy.Spider):- name = 'example.com'- start_urls = ['http://www.example.com/users/login.php']+ name = "example.com"+ start_urls = ["http://www.example.com/users/login.php"]def parse(self, response):return scrapy.FormRequest.from_response(response,- formdata={'username': 'john', 'password': 'secret'},- callback=self.after_login+ formdata={"username": "john", "password": "secret"},+ callback=self.after_login,)def after_login(self, response):@@ -952,13 +1000,15 @@ dealing with JSON requests.JsonRequest usage example--------------------------Sending a JSON POST request with a JSON payload::+Sending a JSON POST request with a JSON payload:++.. code-block:: pythondata = {- 'name1': 'value1',- 'name2': 'value2',+ "name1": "value1",+ "name2": "value2",}- yield JsonRequest(url='http://www.example.com/post/action', data=data)+ yield JsonRequest(url="http://www.example.com/post/action", data=data)Response objectsdiff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rstindex 574d4568c..ede3ca7cd 100644--- a/docs/topics/selectors.rst+++ b/docs/topics/selectors.rst@@ -981,25 +981,34 @@ Selector examples on HTML responseHere are some :class:`Selector` examples to illustrate several concepts.In all cases, we assume there is already a :class:`Selector` instantiated with-a :class:`~scrapy.http.HtmlResponse` object like this::+a :class:`~scrapy.http.HtmlResponse` object like this:++.. code-block:: pythonsel = Selector(html_response)1. Select all ``<h1>`` elements from an HTML response body, returning a list of- :class:`Selector` objects (i.e. a :class:`SelectorList` object)::+ :class:`Selector` objects (i.e. a :class:`SelectorList` object):++ .. code-block:: pythonsel.xpath("//h1")2. Extract the text of all ``<h1>`` elements from an HTML response body,- returning a list of strings::+ returning a list of strings:- sel.xpath("//h1").getall() # this includes the h1 tag+ .. code-block:: python++ sel.xpath("//h1").getall() # this includes the h1 tagsel.xpath("//h1/text()").getall() # this excludes the h1 tag-3. Iterate over all ``<p>`` tags and print their class attribute::+3. Iterate over all ``<p>`` tags and print their class attribute:+++ .. code-block:: pythonfor node in sel.xpath("//p"):- print(node.attrib['class'])+ print(node.attrib["class"]).. _selector-examples-xml:@@ -1008,17 +1017,23 @@ Selector examples on XML response---------------------------------Here are some examples to illustrate concepts for :class:`Selector` objects-instantiated with an :class:`~scrapy.http.XmlResponse` object::+instantiated with an :class:`~scrapy.http.XmlResponse` object:++.. code-block:: pythonsel = Selector(xml_response)1. Select all ``<product>`` elements from an XML response body, returning a list- of :class:`Selector` objects (i.e. a :class:`SelectorList` object)::+ of :class:`Selector` objects (i.e. a :class:`SelectorList` object):++ .. code-block:: pythonsel.xpath("//product")2. Extract all prices from a `Google Base XML feed`_ which requires registering- a namespace::+ a namespace:++ .. code-block:: pythonsel.register_namespace("g", "http://base.google.com/ns/1.0")sel.xpath("//g:price").getall()diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex e147a943b..d636dc301 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -67,13 +67,15 @@ Example::Spiders (See the :ref:`topics-spiders` chapter for reference) can define theirown settings that will take precedence and override the project ones. They can-do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute::+do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute:++.. code-block:: pythonclass MySpider(scrapy.Spider):- name = 'myspider'+ name = "myspider"custom_settings = {- 'SOME_SETTING': 'some value',+ "SOME_SETTING": "some value",}3. Project settings module@@ -115,14 +117,17 @@ class or a function, there are two different ways you can specify that object:- As the object itself-For example::+For example:++.. code-block:: pythonfrom mybot.pipelines.validate import ValidateMyItem+ITEM_PIPELINES = {# passing the classname...ValidateMyItem: 300,# ...equals passing the class path- 'mybot.pipelines.validate.ValidateMyItem': 300,+ "mybot.pipelines.validate.ValidateMyItem": 300,}.. note:: Passing non-callable objects is not supported.@@ -133,11 +138,13 @@ How to access settings.. highlight:: python-In a spider, the settings are available through ``self.settings``::+In a spider, the settings are available through ``self.settings``:++.. code-block:: pythonclass MySpider(scrapy.Spider):- name = 'myspider'- start_urls = ['http://example.com']+ name = "myspider"+ start_urls = ["http://example.com"]def parse(self, response):print(f"Existing settings: {self.settings.attributes.keys()}")@@ -150,7 +157,9 @@ In a spider, the settings are available through ``self.settings``::Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`attribute of the Crawler that is passed to ``from_crawler`` method in-extensions, middlewares and item pipelines::+extensions, middlewares and item pipelines:++.. code-block:: pythonclass MyExtension:def __init__(self, log_is_enabled=False):@@ -160,7 +169,7 @@ extensions, middlewares and item pipelines::@classmethoddef from_crawler(cls, crawler):settings = crawler.settings- return cls(settings.getbool('LOG_ENABLED'))+ return cls(settings.getbool("LOG_ENABLED"))The settings object can be used like a dict (e.g.,``settings['LOG_ENABLED']``), but it's usually preferred to extract the setting@@ -365,11 +374,13 @@ Scrapy shell <topics-shell>`.DEFAULT_REQUEST_HEADERS------------------------Default::+Default:++.. code-block:: python{- 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',- 'Accept-Language': 'en',+ "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",+ "Accept-Language": "en",}The default headers used for Scrapy HTTP Requests. They're populated in the@@ -404,9 +415,11 @@ Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware``An integer that is used to adjust the :attr:`~scrapy.Request.priority` ofa :class:`~scrapy.Request` based on its depth.-The priority of a request is adjusted as follows::+The priority of a request is adjusted as follows:- request.priority = request.priority - ( depth * DEPTH_PRIORITY )+.. code-block:: python++ request.priority = request.priority - (depth * DEPTH_PRIORITY)As depth increases, positive values of ``DEPTH_PRIORITY`` decrease requestpriority (BFO), while negative values increase request priority (DFO). See@@ -595,23 +608,25 @@ orders. For more info see :ref:`topics-downloader-middleware-setting`.DOWNLOADER_MIDDLEWARES_BASE----------------------------Default::+Default:++.. code-block:: python{- 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,- 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,- 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,- 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,- 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,- 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,- 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,- 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,- 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,- 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,- 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,- 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,- 'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,- 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,+ "scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware": 100,+ "scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware": 300,+ "scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware": 350,+ "scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": 400,+ "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 500,+ "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,+ "scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware": 560,+ "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,+ "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,+ "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,+ "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,+ "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,+ "scrapy.downloadermiddlewares.stats.DownloaderStats": 850,+ "scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware": 900,}A dict containing the downloader middlewares enabled by default in Scrapy. Low@@ -687,15 +702,17 @@ See :setting:`DOWNLOAD_HANDLERS_BASE` for example format.DOWNLOAD_HANDLERS_BASE-----------------------Default::+Default:++.. code-block:: python{- 'data': 'scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler',- 'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler',- 'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',- 'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',- 's3': 'scrapy.core.downloader.handlers.s3.S3DownloadHandler',- 'ftp': 'scrapy.core.downloader.handlers.ftp.FTPDownloadHandler',+ "data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler",+ "file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",+ "http": "scrapy.core.downloader.handlers.http.HTTPDownloadHandler",+ "https": "scrapy.core.downloader.handlers.http.HTTPDownloadHandler",+ "s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",+ "ftp": "scrapy.core.downloader.handlers.ftp.FTPDownloadHandler",}@@ -705,10 +722,12 @@ You should never modify this setting in your project, modifyYou can disable any of these download handlers by assigning ``None`` to theirURI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP-handler (without replacement), place this in your ``settings.py``::+handler (without replacement), place this in your ``settings.py``:++.. code-block:: pythonDOWNLOAD_HANDLERS = {- 'ftp': None,+ "ftp": None,}.. _http2:@@ -718,10 +737,12 @@ The default HTTPS handler uses HTTP/1.1. To use HTTP/2:#. Install ``Twisted[http2]>=17.9.0`` to install the packages required toenable HTTP/2 support in Twisted.-#. Update :setting:`DOWNLOAD_HANDLERS` as follows::+#. Update :setting:`DOWNLOAD_HANDLERS` as follows:++ .. code-block:: pythonDOWNLOAD_HANDLERS = {- 'https': 'scrapy.core.downloader.handlers.http2.H2DownloadHandler',+ "https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",}.. warning::@@ -890,18 +911,20 @@ A dict containing the extensions enabled in your project, and their orders.EXTENSIONS_BASE----------------Default::+Default:++.. code-block:: python{- 'scrapy.extensions.corestats.CoreStats': 0,- 'scrapy.extensions.telnet.TelnetConsole': 0,- 'scrapy.extensions.memusage.MemoryUsage': 0,- 'scrapy.extensions.memdebug.MemoryDebugger': 0,- 'scrapy.extensions.closespider.CloseSpider': 0,- 'scrapy.extensions.feedexport.FeedExporter': 0,- 'scrapy.extensions.logstats.LogStats': 0,- 'scrapy.extensions.spiderstate.SpiderState': 0,- 'scrapy.extensions.throttle.AutoThrottle': 0,+ "scrapy.extensions.corestats.CoreStats": 0,+ "scrapy.extensions.telnet.TelnetConsole": 0,+ "scrapy.extensions.memusage.MemoryUsage": 0,+ "scrapy.extensions.memdebug.MemoryDebugger": 0,+ "scrapy.extensions.closespider.CloseSpider": 0,+ "scrapy.extensions.feedexport.FeedExporter": 0,+ "scrapy.extensions.logstats.LogStats": 0,+ "scrapy.extensions.spiderstate.SpiderState": 0,+ "scrapy.extensions.throttle.AutoThrottle": 0,}A dict containing the extensions available by default in Scrapy, and their@@ -988,11 +1011,13 @@ A dict containing the item pipelines to use, and their orders. Order values arearbitrary, but it is customary to define them in the 0-1000 range. Lower ordersprocess before higher orders.-Example::+Example:++.. code-block:: pythonITEM_PIPELINES = {- 'mybot.pipelines.validate.ValidateMyItem': 300,- 'mybot.pipelines.validate.StoreMyItem': 800,+ "mybot.pipelines.validate.ValidateMyItem": 300,+ "mybot.pipelines.validate.StoreMyItem": 800,}.. setting:: ITEM_PIPELINES_BASE@@ -1417,12 +1442,14 @@ testing spiders. For more info see :ref:`topics-contracts`.SPIDER_CONTRACTS_BASE----------------------Default::+Default:++.. code-block:: python{- 'scrapy.contracts.default.UrlContract' : 1,- 'scrapy.contracts.default.ReturnsContract': 2,- 'scrapy.contracts.default.ScrapesContract': 3,+ "scrapy.contracts.default.UrlContract": 1,+ "scrapy.contracts.default.ReturnsContract": 2,+ "scrapy.contracts.default.ScrapesContract": 3,}A dict containing the Scrapy contracts enabled by default in Scrapy. You should@@ -1431,10 +1458,12 @@ instead. For more info see :ref:`topics-contracts`.You can disable any of these contracts by assigning ``None`` to their classpath in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in-``ScrapesContract``, place this in your ``settings.py``::+``ScrapesContract``, place this in your ``settings.py``:++.. code-block:: pythonSPIDER_CONTRACTS = {- 'scrapy.contracts.default.ScrapesContract': None,+ "scrapy.contracts.default.ScrapesContract": None,}.. setting:: SPIDER_LOADER_CLASS@@ -1483,14 +1512,16 @@ orders. For more info see :ref:`topics-spider-middleware-setting`.SPIDER_MIDDLEWARES_BASE------------------------Default::+Default:++.. code-block:: python{- 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,- 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,- 'scrapy.spidermiddlewares.referer.RefererMiddleware': 700,- 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware': 800,- 'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,+ "scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,+ "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": 500,+ "scrapy.spidermiddlewares.referer.RefererMiddleware": 700,+ "scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,+ "scrapy.spidermiddlewares.depth.DepthMiddleware": 900,}A dict containing the spider middlewares enabled by default in Scrapy, and@@ -1506,9 +1537,11 @@ Default: ``[]``A list of modules where Scrapy will look for spiders.-Example::+Example:++.. code-block:: python- SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev']+ SPIDER_MODULES = ["mybot.spiders_prod", "mybot.spiders_dev"].. setting:: STATS_CLASS@@ -1597,60 +1630,65 @@ If a reactor is already installed,third-party libraries will make Scrapy raise :exc:`Exception` whenit checks which reactor is installed.-In order to use the reactor installed by Scrapy::+In order to use the reactor installed by Scrapy:++.. code-block:: pythonimport scrapyfrom twisted.internet import reactorclass QuotesSpider(scrapy.Spider):- name = 'quotes'+ name = "quotes"def __init__(self, *args, **kwargs):- self.timeout = int(kwargs.pop('timeout', '60'))+ self.timeout = int(kwargs.pop("timeout", "60"))super(QuotesSpider, self).__init__(*args, **kwargs)def start_requests(self):reactor.callLater(self.timeout, self.stop)- urls = ['https://quotes.toscrape.com/page/1']+ urls = ["https://quotes.toscrape.com/page/1"]for url in urls:yield scrapy.Request(url=url, callback=self.parse)def parse(self, response):- for quote in response.css('div.quote'):- yield {'text': quote.css('span.text::text').get()}+ for quote in response.css("div.quote"):+ yield {"text": quote.css("span.text::text").get()}def stop(self):- self.crawler.engine.close_spider(self, 'timeout')+ self.crawler.engine.close_spider(self, "timeout")-which raises :exc:`Exception`, becomes::+which raises :exc:`Exception`, becomes:++.. code-block:: pythonimport scrapyclass QuotesSpider(scrapy.Spider):- name = 'quotes'+ name = "quotes"def __init__(self, *args, **kwargs):- self.timeout = int(kwargs.pop('timeout', '60'))+ self.timeout = int(kwargs.pop("timeout", "60"))super(QuotesSpider, self).__init__(*args, **kwargs)def start_requests(self):from twisted.internet import reactor+reactor.callLater(self.timeout, self.stop)- urls = ['https://quotes.toscrape.com/page/1']+ urls = ["https://quotes.toscrape.com/page/1"]for url in urls:yield scrapy.Request(url=url, callback=self.parse)def parse(self, response):- for quote in response.css('div.quote'):- yield {'text': quote.css('span.text::text').get()}+ for quote in response.css("div.quote"):+ yield {"text": quote.css("span.text::text").get()}def stop(self):- self.crawler.engine.close_spider(self, 'timeout')+ self.crawler.engine.close_spider(self, "timeout")The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, whichdiff --git a/docs/topics/shell.rst b/docs/topics/shell.rstindex 007e9fc2f..69a90f620 100644--- a/docs/topics/shell.rst+++ b/docs/topics/shell.rst@@ -242,7 +242,9 @@ getting there.This can be achieved by using the ``scrapy.shell.inspect_response`` function.-Here's an example of how you would call it from your spider::+Here's an example of how you would call it from your spider:++.. code-block:: pythonimport scrapy@@ -259,6 +261,7 @@ Here's an example of how you would call it from your spider::# We want to inspect one specific response.if ".org" in response.url:from scrapy.shell import inspect_response+inspect_response(response, self)# Rest of parsing code.diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 17bd16156..906aecdee 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -16,7 +16,9 @@ deliver the arguments that the handler receives.You can connect to signals (or send your own) through the:ref:`topics-api-signals`.-Here is a simple example showing how you can catch signals and perform some action::+Here is a simple example showing how you can catch signals and perform some action:++.. code-block:: pythonfrom scrapy import signalsfrom scrapy import Spider@@ -30,17 +32,14 @@ Here is a simple example showing how you can catch signals and perform some acti"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",]-@classmethoddef from_crawler(cls, crawler, *args, **kwargs):spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)return spider-def spider_closed(self, spider):- spider.logger.info('Spider closed: %s', spider.name)-+ spider.logger.info("Spider closed: %s", spider.name)def parse(self, response):pass@@ -56,11 +55,13 @@ you to run asynchronous code that does not block Scrapy. If a signalhandler returns one of these objects, Scrapy waits for that asynchronousoperation to finish.-Let's take an example using :ref:`coroutines <topics-coroutines>`::+Let's take an example using :ref:`coroutines <topics-coroutines>`:++.. code-block:: pythonclass SignalSpider(scrapy.Spider):- name = 'signals'- start_urls = ['https://quotes.toscrape.com/page/1/']+ name = "signals"+ start_urls = ["https://quotes.toscrape.com/page/1/"]@classmethoddef from_crawler(cls, crawler, *args, **kwargs):@@ -71,19 +72,19 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`::async def item_scraped(self, item):# Send the scraped item to the serverresponse = await treq.post(- 'http://example.com/post',- json.dumps(item).encode('ascii'),- headers={b'Content-Type': [b'application/json']}+ "http://example.com/post",+ json.dumps(item).encode("ascii"),+ headers={b"Content-Type": [b"application/json"]},)return responsedef parse(self, response):- for quote in response.css('div.quote'):+ for quote in response.css("div.quote"):yield {- 'text': quote.css('span.text::text').get(),- 'author': quote.css('small.author::text').get(),- 'tags': quote.css('div.tags a.tag::text').getall(),+ "text": quote.css("span.text::text").get(),+ "author": quote.css("small.author::text").get(),+ "tags": quote.css("div.tags a.tag::text").getall(),}See the :ref:`topics-signals-ref` below to know which signals supportdiff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 303401a3c..94bb4d44d 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -18,10 +18,12 @@ To activate a spider middleware component, add it to the:setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are themiddleware class path and their values are the middleware orders.-Here's an example::+Here's an example:++.. code-block:: pythonSPIDER_MIDDLEWARES = {- 'myproject.middlewares.CustomSpiderMiddleware': 543,+ "myproject.middlewares.CustomSpiderMiddleware": 543,}The :setting:`SPIDER_MIDDLEWARES` setting is merged with the@@ -44,11 +46,13 @@ previous (or subsequent) middleware being applied.If you want to disable a builtin middleware (the ones defined in:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define itin your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its-value. For example, if you want to disable the off-site middleware::+value. For example, if you want to disable the off-site middleware:++.. code-block:: pythonSPIDER_MIDDLEWARES = {- 'myproject.middlewares.CustomSpiderMiddleware': 543,- 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,+ "myproject.middlewares.CustomSpiderMiddleware": 543,+ "scrapy.spidermiddlewares.offsite.OffsiteMiddleware": None,}Finally, keep in mind that some middlewares may need to be enabled through a@@ -261,7 +265,9 @@ specify which response codes the spider is able to handle using the:setting:`HTTPERROR_ALLOWED_CODES` setting.For example, if you want your spider to handle 404 responses you can do-this::+this:++.. code-block:: pythonclass MySpider(CrawlSpider):handle_httpstatus_list = [404]diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 7aa8555d5..91f783b45 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -157,15 +157,21 @@ scrapy.SpiderIf you want to change the Requests used to start scraping a domain, this isthe method to override. For example, if you need to start by logging in using- a POST request, you could do::+ a POST request, you could do:++ .. code-block:: pythonclass MySpider(scrapy.Spider):- name = 'myspider'+ name = "myspider"def start_requests(self):- return [scrapy.FormRequest("http://www.example.com/login",- formdata={'user': 'john', 'pass': 'secret'},- callback=self.logged_in)]+ return [+ scrapy.FormRequest(+ "http://www.example.com/login",+ formdata={"user": "john", "pass": "secret"},+ callback=self.logged_in,+ )+ ]def logged_in(self, response):# here you would extract links to follow and return Requests for@@ -200,63 +206,71 @@ scrapy.SpiderCalled when the spider closes. This method provides a shortcut tosignals.connect() for the :signal:`spider_closed` signal.-Let's see an example::+Let's see an example:++.. code-block:: pythonimport scrapyclass MySpider(scrapy.Spider):- name = 'example.com'- allowed_domains = ['example.com']+ name = "example.com"+ allowed_domains = ["example.com"]start_urls = [- 'http://www.example.com/1.html',- 'http://www.example.com/2.html',- 'http://www.example.com/3.html',+ "http://www.example.com/1.html",+ "http://www.example.com/2.html",+ "http://www.example.com/3.html",]def parse(self, response):- self.logger.info('A response from %s just arrived!', response.url)+ self.logger.info("A response from %s just arrived!", response.url)-Return multiple Requests and items from a single callback::+Return multiple Requests and items from a single callback:++.. code-block:: pythonimport scrapy+class MySpider(scrapy.Spider):- name = 'example.com'- allowed_domains = ['example.com']+ name = "example.com"+ allowed_domains = ["example.com"]start_urls = [- 'http://www.example.com/1.html',- 'http://www.example.com/2.html',- 'http://www.example.com/3.html',+ "http://www.example.com/1.html",+ "http://www.example.com/2.html",+ "http://www.example.com/3.html",]def parse(self, response):- for h3 in response.xpath('//h3').getall():+ for h3 in response.xpath("//h3").getall():yield {"title": h3}- for href in response.xpath('//a/@href').getall():+ for href in response.xpath("//a/@href").getall():yield scrapy.Request(response.urljoin(href), self.parse)Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;-to give data more structure you can use :class:`~scrapy.Item` objects::+to give data more structure you can use :class:`~scrapy.Item` objects:++.. code-block:: pythonimport scrapyfrom myproject.items import MyItem+class MySpider(scrapy.Spider):- name = 'example.com'- allowed_domains = ['example.com']+ name = "example.com"+ allowed_domains = ["example.com"]def start_requests(self):- yield scrapy.Request('http://www.example.com/1.html', self.parse)- yield scrapy.Request('http://www.example.com/2.html', self.parse)- yield scrapy.Request('http://www.example.com/3.html', self.parse)+ yield scrapy.Request("http://www.example.com/1.html", self.parse)+ yield scrapy.Request("http://www.example.com/2.html", self.parse)+ yield scrapy.Request("http://www.example.com/3.html", self.parse)def parse(self, response):- for h3 in response.xpath('//h3').getall():+ for h3 in response.xpath("//h3").getall():yield MyItem(title=h3)- for href in response.xpath('//a/@href').getall():+ for href in response.xpath("//a/@href").getall():yield scrapy.Request(response.urljoin(href), self.parse).. _spiderargs:@@ -274,34 +288,42 @@ Spider arguments are passed through the :command:`crawl` command using thescrapy crawl myspider -a category=electronics-Spiders can access arguments in their `__init__` methods::+Spiders can access arguments in their `__init__` methods:++.. code-block:: pythonimport scrapy+class MySpider(scrapy.Spider):- name = 'myspider'+ name = "myspider"def __init__(self, category=None, *args, **kwargs):super(MySpider, self).__init__(*args, **kwargs)- self.start_urls = [f'http://www.example.com/categories/{category}']+ self.start_urls = [f"http://www.example.com/categories/{category}"]# ...The default `__init__` method will take any spider argumentsand copy them to the spider as attributes.-The above example can also be written as follows::+The above example can also be written as follows:++.. code-block:: pythonimport scrapy+class MySpider(scrapy.Spider):- name = 'myspider'+ name = "myspider"def start_requests(self):- yield scrapy.Request(f'http://www.example.com/categories/{self.category}')+ yield scrapy.Request(f"http://www.example.com/categories/{self.category}")If you are :ref:`running Scrapy from a script <run-from-script>`, you canspecify spider arguments when calling:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or-:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`::+:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:++.. code-block:: pythonprocess = CrawlerProcess()process.crawl(MySpider, category="electronics")@@ -337,10 +359,13 @@ common scraping cases, like following all links on a site based on certainrules, crawling from `Sitemaps`_, or parsing an XML/CSV feed.For the examples used in the following spiders, we'll assume you have a project-with a ``TestItem`` declared in a ``myproject.items`` module::+with a ``TestItem`` declared in a ``myproject.items`` module:++.. code-block:: pythonimport scrapy+class TestItem(scrapy.Item):id = scrapy.Field()name = scrapy.Field()@@ -436,38 +461,46 @@ Crawling rulesCrawlSpider example~~~~~~~~~~~~~~~~~~~-Let's now take a look at an example CrawlSpider with rules::+Let's now take a look at an example CrawlSpider with rules:++.. code-block:: pythonimport scrapyfrom scrapy.spiders import CrawlSpider, Rulefrom scrapy.linkextractors import LinkExtractor+class MySpider(CrawlSpider):- name = 'example.com'- allowed_domains = ['example.com']- start_urls = ['http://www.example.com']+ name = "example.com"+ allowed_domains = ["example.com"]+ start_urls = ["http://www.example.com"]rules = (# Extract links matching 'category.php' (but not matching 'subsection.php')# and follow links from them (since no callback means follow=True by default).- Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),-+ Rule(LinkExtractor(allow=("category\.php",), deny=("subsection\.php",))),# Extract links matching 'item.php' and parse them with the spider's method parse_item- Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),+ Rule(LinkExtractor(allow=("item\.php",)), callback="parse_item"),)def parse_item(self, response):- self.logger.info('Hi, this is an item page! %s', response.url)+ self.logger.info("Hi, this is an item page! %s", response.url)item = scrapy.Item()- item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')- item['name'] = response.xpath('//td[@id="item_name"]/text()').get()- item['description'] = response.xpath('//td[@id="item_description"]/text()').get()- item['link_text'] = response.meta['link_text']+ item["id"] = response.xpath('//td[@id="item_id"]/text()').re(r"ID: (\d+)")+ item["name"] = response.xpath('//td[@id="item_name"]/text()').get()+ item["description"] = response.xpath(+ '//td[@id="item_description"]/text()'+ ).get()+ item["link_text"] = response.meta["link_text"]url = response.xpath('//td[@id="additional_data"]/@href').get()- return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item))+ return response.follow(+ url, self.parse_additional_page, cb_kwargs=dict(item=item)+ )def parse_additional_page(self, response, item):- item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get()+ item["additional_data"] = response.xpath(+ '//p[@id="additional_data"]/text()'+ ).get()return item@@ -568,25 +601,30 @@ XMLFeedSpiderXMLFeedSpider example~~~~~~~~~~~~~~~~~~~~~-These spiders are pretty easy to use, let's have a look at one example::+These spiders are pretty easy to use, let's have a look at one example:++.. code-block:: pythonfrom scrapy.spiders import XMLFeedSpiderfrom myproject.items import TestItem+class MySpider(XMLFeedSpider):- name = 'example.com'- allowed_domains = ['example.com']- start_urls = ['http://www.example.com/feed.xml']- iterator = 'iternodes' # This is actually unnecessary, since it's the default value- itertag = 'item'+ name = "example.com"+ allowed_domains = ["example.com"]+ start_urls = ["http://www.example.com/feed.xml"]+ iterator = "iternodes" # This is actually unnecessary, since it's the default value+ itertag = "item"def parse_node(self, response, node):- self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.getall()))+ self.logger.info(+ "Hi, this is a <%s> node!: %s", self.itertag, "".join(node.getall())+ )item = TestItem()- item['id'] = node.xpath('@id').get()- item['name'] = node.xpath('name').get()- item['description'] = node.xpath('description').get()+ item["id"] = node.xpath("@id").get()+ item["name"] = node.xpath("name").get()+ item["description"] = node.xpath("description").get()return itemBasically what we did up there was to create a spider that downloads a feed from@@ -627,26 +665,29 @@ CSVFeedSpider example~~~~~~~~~~~~~~~~~~~~~Let's see an example similar to the previous one, but using a-:class:`CSVFeedSpider`::+:class:`CSVFeedSpider`:++.. code-block:: pythonfrom scrapy.spiders import CSVFeedSpiderfrom myproject.items import TestItem+class MySpider(CSVFeedSpider):- name = 'example.com'- allowed_domains = ['example.com']- start_urls = ['http://www.example.com/feed.csv']- delimiter = ';'+ name = "example.com"+ allowed_domains = ["example.com"]+ start_urls = ["http://www.example.com/feed.csv"]+ delimiter = ";"quotechar = "'"- headers = ['id', 'name', 'description']+ headers = ["id", "name", "description"]def parse_row(self, response, row):- self.logger.info('Hi, this is a row!: %r', row)+ self.logger.info("Hi, this is a row!: %r", row)item = TestItem()- item['id'] = row['id']- item['name'] = row['name']- item['description'] = row['description']+ item["id"] = row["id"]+ item["name"] = row["name"]+ item["description"] = row["description"]return item@@ -728,19 +769,22 @@ SitemapSpider<lastmod>2005-01-01</lastmod></url>- We can define a ``sitemap_filter`` function to filter ``entries`` by date::+ We can define a ``sitemap_filter`` function to filter ``entries`` by date:++ .. code-block:: pythonfrom datetime import datetimefrom scrapy.spiders import SitemapSpider+class FilteredSitemapSpider(SitemapSpider):- name = 'filtered_sitemap_spider'- allowed_domains = ['example.com']- sitemap_urls = ['http://example.com/sitemap.xml']+ name = "filtered_sitemap_spider"+ allowed_domains = ["example.com"]+ sitemap_urls = ["http://example.com/sitemap.xml"]def sitemap_filter(self, entries):for entry in entries:- date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')+ date_time = datetime.strptime(entry["lastmod"], "%Y-%m-%d")if date_time.year >= 2005:yield entry@@ -765,60 +809,72 @@ SitemapSpider examples~~~~~~~~~~~~~~~~~~~~~~Simplest example: process all urls discovered through sitemaps using the-``parse`` callback::+``parse`` callback:++.. code-block:: pythonfrom scrapy.spiders import SitemapSpider+class MySpider(SitemapSpider):- sitemap_urls = ['http://www.example.com/sitemap.xml']+ sitemap_urls = ["http://www.example.com/sitemap.xml"]def parse(self, response):- pass # ... scrape item here ...+ pass # ... scrape item here ...Process some urls with certain callback and other urls with a different-callback::+callback:++.. code-block:: pythonfrom scrapy.spiders import SitemapSpider+class MySpider(SitemapSpider):- sitemap_urls = ['http://www.example.com/sitemap.xml']+ sitemap_urls = ["http://www.example.com/sitemap.xml"]sitemap_rules = [- ('/product/', 'parse_product'),- ('/category/', 'parse_category'),+ ("/product/", "parse_product"),+ ("/category/", "parse_category"),]def parse_product(self, response):- pass # ... scrape product ...+ pass # ... scrape product ...def parse_category(self, response):- pass # ... scrape category ...+ pass # ... scrape category ...Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps-whose url contains ``/sitemap_shop``::+whose url contains ``/sitemap_shop``:++.. code-block:: pythonfrom scrapy.spiders import SitemapSpider+class MySpider(SitemapSpider):- sitemap_urls = ['http://www.example.com/robots.txt']+ sitemap_urls = ["http://www.example.com/robots.txt"]sitemap_rules = [- ('/shop/', 'parse_shop'),+ ("/shop/", "parse_shop"),]- sitemap_follow = ['/sitemap_shops']+ sitemap_follow = ["/sitemap_shops"]def parse_shop(self, response):- pass # ... scrape shop here ...+ pass # ... scrape shop here ...++Combine SitemapSpider with other sources of urls:-Combine SitemapSpider with other sources of urls::+.. code-block:: pythonfrom scrapy.spiders import SitemapSpider+class MySpider(SitemapSpider):- sitemap_urls = ['http://www.example.com/robots.txt']+ sitemap_urls = ["http://www.example.com/robots.txt"]sitemap_rules = [- ('/shop/', 'parse_shop'),+ ("/shop/", "parse_shop"),]- other_urls = ['http://www.example.com/about']+ other_urls = ["http://www.example.com/about"]def start_requests(self):requests = list(super(MySpider, self).start_requests())@@ -826,10 +882,10 @@ Combine SitemapSpider with other sources of urls::return requestsdef parse_shop(self, response):- pass # ... scrape shop here ...+ pass # ... scrape shop here ...def parse_other(self, response):- pass # ... scrape other here ...+ pass # ... scrape other here ..... _Sitemaps: https://www.sitemaps.org/index.html.. _Sitemap index files: https://www.sitemaps.org/protocol.html#indexdiff --git a/docs/topics/stats.rst b/docs/topics/stats.rstindex af848b402..f8b0c4d2a 100644--- a/docs/topics/stats.rst+++ b/docs/topics/stats.rst@@ -30,10 +30,11 @@ Common Stats Collector uses===========================Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats`-attribute. Here is an example of an extension that access stats::+attribute. Here is an example of an extension that access stats:- class ExtensionThatAccessStats:+.. code-block:: python+ class ExtensionThatAccessStats:def __init__(self, stats):self.stats = stats@@ -41,21 +42,29 @@ attribute. Here is an example of an extension that access stats::def from_crawler(cls, crawler):return cls(crawler.stats)-Set stat value::+Set stat value:++.. code-block:: python++ stats.set_value("hostname", socket.gethostname())++Increment stat value:++.. code-block:: python- stats.set_value('hostname', socket.gethostname())+ stats.inc_value("custom_count")-Increment stat value::+Set stat value only if greater than previous:- stats.inc_value('custom_count')+.. code-block:: python-Set stat value only if greater than previous::+ stats.max_value("max_items_scraped", value)- stats.max_value('max_items_scraped', value)+Set stat value only if lower than previous:-Set stat value only if lower than previous::+.. code-block:: python- stats.min_value('min_free_memory_percent', value)+ stats.min_value("min_free_memory_percent", value)Get stat value:diff --git a/sep/sep-001.rst b/sep/sep-001.rstindex f704e113f..e6673b360 100644--- a/sep/sep-001.rst+++ b/sep/sep-001.rst@@ -1,274 +1,275 @@-======= ============================================-SEP 1-Title API for populating item fields (comparison)-Author Ismael Carnales, Pablo Hoffman, Daniel Grana-Created 2009-07-19-Status Obsoleted by :ref:`sep-008`-======= ============================================--=====================================================-SEP-001 - API for populating item fields (comparison)-=====================================================--This page shows different usage scenarios for the two new proposed API for-populating item field values (which will replace the old deprecated !RobustItem-API) and compares them. One of these will be chosen as the recommended (and-supported) mechanism in Scrapy 0.7.--Candidates and their API-========================--RobustItem (old, deprecated)-------------------------------- ``attribute(field_name, selector_or_value, **modifiers_and_adaptor_args)``--.. note:: ``attribute()`` modifiers (like ``add=True``) are passed together- with adaptor args as keyword arguments (this is ugly)--ItemForm------------ ``__init__(response, item=None, **adaptor_args)``- - instantiate an ``ItemForm`` with a item instance with predefined adaptor arguments-- ``__setitem__(field_name, selector_or_value)``- - set field value-- ``__getitem__(field_name)``- - return the "computed" value of a field (the one that would be set to the item).- returns ``None`` if not set.-- ``get_item()``- - return the item populated with the data provided so far--ItemBuilder--------------- ``__init__(response, item=None, **adaptor_args)``- - instantiate an ``ItemBuilder`` with predefined adaptor arguments-- ``add_value(field_name, selector_or_value, **adaptor_args)``- - add value to field-- ``replace_value(field_name, selector_or_value, **adaptor_args)``- - replace existing field value-- ``get_value(field_name)``- - return the "computed" value of a field (the one that would be set to the- item). returns ``None`` if not set.-- ``get_item()``- - return the item populated with the data provided so far--Pros and cons of each candidate-===============================--ItemForm-----------Pros:-- same API used for Items (see https://docs.scrapy.org/en/latest/topics/items.html)-- some people consider setitem API more elegant than methods API--Cons:-- doesn't allow passing run-time arguments to adaptors on assign, you have to- override the adaptors for your spider if you need specific parameters, which- can be an overhead. Example:--Neutral:-- solves the add=True problem using standard ``__add__`` and ``list.append()`` method--ItemBuilder--------------Pros:-- allows passing run-time arguments to adaptors on assigned--Cons:-- some people consider setitem API more elegant than methods API--Neutral:-- solves the "add=True" problem by implementing different methods per action- (replacing or adding)--Usage Scenarios for each candidate-==================================--Defining adaptors--------------------ItemForm-~~~~~~~~--::-- #!python- class NewsForm(ItemForm):- item_class = NewsItem-- url = adaptor(extract, remove_tags(), unquote(), strip)- headline = adaptor(extract, remove_tags(), unquote(), strip)--ItemBuilder-~~~~~~~~~~~--::-- #!python- class NewsBuilder(ItemBuilder):- item_class = NewsItem-- url = adaptor(extract, remove_tags(), unquote(), strip)- headline = adaptor(extract, remove_tags(), unquote(), strip)--Creating an Item-------------------ItemForm-~~~~~~~~--::-- #!python- ia = NewsForm(response)- ia['url'] = response.url- ia['headline'] = x.x('//h1[@class="headline"]')-- # if we want to add another value to the same field- ia['headline'] += x.x('//h1[@class="headline2"]')-- # if we want to replace the field value other value to the same field- ia['headline'] = x.x('//h1[@class="headline3"]')-- return ia.get_item()--ItemBuilder-~~~~~~~~~~~--::-- #!python- il = NewsBuilder(response)- il.add_value('url', response.url)- il.add_value('headline', x.x('//h1[@class="headline"]'))-- # if we want to add another value to the same field- il.add_value('headline', x.x('//h1[@class="headline2"]'))-- # if we want to replace the field value other value to the same field- il.replace_value('headline', x.x('//h1[@class="headline3"]'))-- return il.get_item()--Using different adaptors per Spider/Site-------------------------------------------ItemForm-~~~~~~~~--::-- #!python- class SiteNewsFrom(NewsForm):- published = adaptor(HtmlNewsForm.published, to_date('%d.%m.%Y'))--ItemBuilder-~~~~~~~~~~~--::-- #!python- class SiteNewsBuilder(NewsBuilder):- published = adaptor(HtmlNewsBuilder.published, to_date('%d.%m.%Y'))--Check the value of an item being-extracted---------------------------------------------ItemForm-~~~~~~~~--::-- #!python- ia = NewsForm(response)- ia['headline'] = x.x('//h1[@class="headline"]')- if not ia['headline']:- ia['headline'] = x.x('//h1[@class="title"]')--ItemBuilder-~~~~~~~~~~~--::-- #!python- il = NewsBuilder(response)- il.add_value('headline', x.x('//h1[@class="headline"]'))- if not nf.get_value('headline'):- il.add_value('headline', x.x('//h1[@class="title"]'))--Adding a value to a list attribute/field-------------------------------------------ItemForm-~~~~~~~~--::-- #!python- ia['headline'] += x.x('//h1[@class="headline"]')--ItemBuilder-~~~~~~~~~~~--::-- #!python- il.add_value('headline', x.x('//h1[@class="headline"]'))--Passing run-time arguments to adaptors-----------------------------------------ItemForm-~~~~~~~~--::-- #!python- # Only approach is passing arguments when instantiating the form- ia = NewsForm(response, default_unit='cm')- ia['width'] = x.x('//p[@class="width"]')--ItemBuilder-~~~~~~~~~~~--::-- #!python- il.add_value('width', x.x('//p[@class="width"]'), default_unit='cm')-- # an alternative approach (more efficient)- il = NewsBuilder(response, default_unit='cm')- il.add_value('width', x.x('//p[@class="width"]'))--Passing run-time arguments to adaptors (same argument name)--------------------------------------------------------------ItemForm-~~~~~~~~--::-- #!python- class MySiteForm(ItemForm):- width = adaptor(ItemForm.width, default_unit='cm')- volume = adaptor(ItemForm.width, default_unit='lt')-- ia['width'] = x.x('//p[@class="width"]')- ia['volume'] = x.x('//p[@class="volume"]')-- # another example passing parameters on instance- ia = NewsForm(response, encoding='utf-8')- ia['name'] = x.x('//p[@class="name"]')--ItemBuilder-~~~~~~~~~~~--::-- #!python- il.add_value('width', x.x('//p[@class="width"]'), default_unit='cm')- il.add_value('volume', x.x('//p[@class="volume"]'), default_unit='lt')+======= ============================================+SEP 1+Title API for populating item fields (comparison)+Author Ismael Carnales, Pablo Hoffman, Daniel Grana+Created 2009-07-19+Status Obsoleted by :ref:`sep-008`+======= ============================================++=====================================================+SEP-001 - API for populating item fields (comparison)+=====================================================++This page shows different usage scenarios for the two new proposed API for+populating item field values (which will replace the old deprecated !RobustItem+API) and compares them. One of these will be chosen as the recommended (and+supported) mechanism in Scrapy 0.7.++Candidates and their API+========================++RobustItem (old, deprecated)+----------------------------++- ``attribute(field_name, selector_or_value, **modifiers_and_adaptor_args)``++.. note:: ``attribute()`` modifiers (like ``add=True``) are passed together+ with adaptor args as keyword arguments (this is ugly)++ItemForm+--------++- ``__init__(response, item=None, **adaptor_args)``+ - instantiate an ``ItemForm`` with a item instance with predefined adaptor arguments+- ``__setitem__(field_name, selector_or_value)``+ - set field value+- ``__getitem__(field_name)``+ - return the "computed" value of a field (the one that would be set to the item).+ returns ``None`` if not set.+- ``get_item()``+ - return the item populated with the data provided so far++ItemBuilder+-----------++- ``__init__(response, item=None, **adaptor_args)``+ - instantiate an ``ItemBuilder`` with predefined adaptor arguments+- ``add_value(field_name, selector_or_value, **adaptor_args)``+ - add value to field+- ``replace_value(field_name, selector_or_value, **adaptor_args)``+ - replace existing field value+- ``get_value(field_name)``+ - return the "computed" value of a field (the one that would be set to the+ item). returns ``None`` if not set.+- ``get_item()``+ - return the item populated with the data provided so far++Pros and cons of each candidate+===============================++ItemForm+--------++Pros:+- same API used for Items (see https://docs.scrapy.org/en/latest/topics/items.html)+- some people consider setitem API more elegant than methods API++Cons:+- doesn't allow passing run-time arguments to adaptors on assign, you have to+ override the adaptors for your spider if you need specific parameters, which+ can be an overhead. Example:++Neutral:+- solves the add=True problem using standard ``__add__`` and ``list.append()`` method++ItemBuilder+-----------++Pros:+- allows passing run-time arguments to adaptors on assigned++Cons:+- some people consider setitem API more elegant than methods API++Neutral:+- solves the "add=True" problem by implementing different methods per action+ (replacing or adding)++Usage Scenarios for each candidate+==================================++Defining adaptors+-----------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ class NewsForm(ItemForm):+ item_class = NewsItem++ url = adaptor(extract, remove_tags(), unquote(), strip)+ headline = adaptor(extract, remove_tags(), unquote(), strip)++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ class NewsBuilder(ItemBuilder):+ item_class = NewsItem++ url = adaptor(extract, remove_tags(), unquote(), strip)+ headline = adaptor(extract, remove_tags(), unquote(), strip)++Creating an Item+----------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ ia = NewsForm(response)+ ia["url"] = response.url+ ia["headline"] = x.x('//h1[@class="headline"]')++ # if we want to add another value to the same field+ ia["headline"] += x.x('//h1[@class="headline2"]')++ # if we want to replace the field value other value to the same field+ ia["headline"] = x.x('//h1[@class="headline3"]')++ return ia.get_item()++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ il = NewsBuilder(response)+ il.add_value("url", response.url)+ il.add_value("headline", x.x('//h1[@class="headline"]'))++ # if we want to add another value to the same field+ il.add_value("headline", x.x('//h1[@class="headline2"]'))++ # if we want to replace the field value other value to the same field+ il.replace_value("headline", x.x('//h1[@class="headline3"]'))++ return il.get_item()++Using different adaptors per Spider/Site+----------------------------------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ class SiteNewsFrom(NewsForm):+ published = adaptor(HtmlNewsForm.published, to_date("%d.%m.%Y"))++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ class SiteNewsBuilder(NewsBuilder):+ published = adaptor(HtmlNewsBuilder.published, to_date("%d.%m.%Y"))++Check the value of an item being-extracted+------------------------------------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ ia = NewsForm(response)+ ia["headline"] = x.x('//h1[@class="headline"]')+ if not ia["headline"]:+ ia["headline"] = x.x('//h1[@class="title"]')++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ il = NewsBuilder(response)+ il.add_value("headline", x.x('//h1[@class="headline"]'))+ if not nf.get_value("headline"):+ il.add_value("headline", x.x('//h1[@class="title"]'))++Adding a value to a list attribute/field+----------------------------------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ ia["headline"] += x.x('//h1[@class="headline"]')++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ il.add_value("headline", x.x('//h1[@class="headline"]'))++Passing run-time arguments to adaptors+--------------------------------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ # Only approach is passing arguments when instantiating the form+ ia = NewsForm(response, default_unit="cm")+ ia["width"] = x.x('//p[@class="width"]')++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ il.add_value("width", x.x('//p[@class="width"]'), default_unit="cm")++ # an alternative approach (more efficient)+ il = NewsBuilder(response, default_unit="cm")+ il.add_value("width", x.x('//p[@class="width"]'))++Passing run-time arguments to adaptors (same argument name)+-----------------------------------------------------------++ItemForm+~~~~~~~~++.. code-block:: python++ #!python+ class MySiteForm(ItemForm):+ width = adaptor(ItemForm.width, default_unit="cm")+ volume = adaptor(ItemForm.width, default_unit="lt")+++ ia["width"] = x.x('//p[@class="width"]')+ ia["volume"] = x.x('//p[@class="volume"]')++ # another example passing parameters on instance+ ia = NewsForm(response, encoding="utf-8")+ ia["name"] = x.x('//p[@class="name"]')++ItemBuilder+~~~~~~~~~~~++.. code-block:: python++ #!python+ il.add_value("width", x.x('//p[@class="width"]'), default_unit="cm")+ il.add_value("volume", x.x('//p[@class="volume"]'), default_unit="lt")diff --git a/sep/sep-002.rst b/sep/sep-002.rstindex c467cb402..cb9ed926d 100644--- a/sep/sep-002.rst+++ b/sep/sep-002.rst@@ -16,18 +16,19 @@ called !ListField.Proposed Implementation=======================-::+.. code-block:: python#!pythonfrom scrapy.item.fields import BaseField+class ListField(BaseField):def __init__(self, field, default=None):self._field = fieldsuper(ListField, self).__init__(default)def to_python(self, value):- if hasattr(value, '__iter__'): # str/unicode not allowed+ if hasattr(value, "__iter__"): # str/unicode not allowedreturn [self._field.to_python(v) for v in value]else:raise TypeError("Expected iterable, got %s" % type(value).__name__)@@ -42,12 +43,13 @@ Usage ScenariosDefining a list field----------------------::+.. code-block:: python#!pythonfrom scrapy.item.models import Itemfrom scrapy.item.fields import ListField, TextField, DateField, IntegerField+class Article(Item):categories = ListField(TextField)dates = ListField(DateField, default=[])@@ -56,57 +58,59 @@ Defining a list fieldAnother case of products and variants which highlights the fact that it'simportant to instantiate !ListField with field instances, not classes:-::+.. code-block:: python#!pythonfrom scrapy.item.models import Itemfrom scrapy.item.fields import ListField, TextField+class Variant(Item):name = TextField()+class Product(Variant):variants = ListField(ItemField(Variant))Assigning a list field-----------------------::+.. code-block:: python#!pythoni = Article()- i['categories'] = []- i['categories'] = ['politics', 'sport']- i['categories'] = ['test', 1] -> raises TypeError- i['categories'] = asd -> raises TypeError+ i["categories"] = []+ i["categories"] = ["politics", "sport"]+ i["categories"] = ["test", 1] # -> raises TypeError+ i["categories"] = asd # -> raises TypeError- i['dates'] = []- i['dates'] = ['2009-01-01'] # raises TypeError? (depends on TextField)+ i["dates"] = []+ i["dates"] = ["2009-01-01"] # raises TypeError? (depends on TextField)- i['numbers'] = ['1', 2, '3']- i['numbers'] # returns [1, 2, 3]+ i["numbers"] = ["1", 2, "3"]+ i["numbers"] # returns [1, 2, 3]Default values---------------::+.. code-block:: python#!pythoni = Article()- i['categories'] # raises KeyError- i.get('categories') # returns None+ i["categories"] # raises KeyError+ i.get("categories") # returns None- i['numbers'] # returns []+ i["numbers"] # returns []Appending values-----------------::+.. code-block:: python#!pythoni = Article()- i['categories'] = ['one', 'two']- i['categories'].append(3) # XXX: should this fail?+ i["categories"] = ["one", "two"]+ i["categories"].append(3) # XXX: should this fail?diff --git a/sep/sep-003.rst b/sep/sep-003.rstindex e6357313d..9152c7d15 100644--- a/sep/sep-003.rst+++ b/sep/sep-003.rst@@ -27,18 +27,21 @@ This API proposal relies on the following API:Proposed Implementation of ItemField====================================-::+.. code-block:: python#!pythonfrom scrapy.item.fields import BaseField+class ItemField(BaseField):def __init__(self, item_type, default=None):self._item_type = item_typesuper(ItemField, self).__init__(default)def to_python(self, value):- return self._item_type(value) if not isinstance(value, self._item_type) else value+ return (+ self._item_type(value) if not isinstance(value, self._item_type) else value+ )def get_default(self):# WARNING: returns default item instead of a copy - this must be@@ -54,25 +57,28 @@ Usage ScenariosDefining an item containing ItemField's----------------------------------------::+.. code-block:: python#!pythonfrom scrapy.item.models import Itemfrom scrapy.item.fields import ListField, ItemField, TextField, UrlField, DecimalField+class Supplier(Item):name = TextField(default="anonymous supplier")url = UrlField()+class Variant(Item):name = TextField(required=True)url = UrlField()price = DecimalField()+class Product(Variant):- supplier = ItemField(Supplier, default=Supplier(name="default supplier")+ supplier = ItemField(Supplier, default=Supplier(name="default supplier"))variants = ListField(ItemField(Variant))-+# these ones are used for documenting default value examplessupplier2 = ItemField(Supplier)variants2 = ListField(ItemField(Variant), default=[])@@ -81,16 +87,16 @@ It's important to note here that the (perhaps most intuitive) way of defining aProduct-Variant relationship (i.e. defining a recursive !ItemField) doesn'twork. For example, this fails to compile:-::+.. code-block:: python#!pythonclass Product(Item):- variants = ItemField(Product) # Fails to compile+ variants = ItemField(Product) # Fails to compileAssigning an item field------------------------::+.. code-block:: python#!pythonsupplier = Supplier(name="Supplier 1", url="http://example.com")@@ -98,69 +104,69 @@ Assigning an item fieldp = Product()# standard assignment- p['supplier'] = supplier+ p["supplier"] = supplier# this also works as it tries to instantiate a Supplier with the given dict- p['supplier'] = {'name': 'Supplier 1' url='http://example.com'}- # this fails because it can't instantiate a Supplier- p['supplier'] = 'Supplier 1'+ p["supplier"] = {"name": "Supplier 1", url: "http://example.com"}+ # this fails because it can't instantiate a Supplier+ p["supplier"] = "Supplier 1"# this fails because url doesn't have the valid type- p['supplier'] = {'name': 'Supplier 1' url=123}+ p["supplier"] = {"name": "Supplier 1", url: 123}v1 = Variant()- v1['name'] = "lala"- v1['price'] = Decimal("100")+ v1["name"] = "lala"+ v1["price"] = Decimal("100")v2 = Variant()- v2['name'] = "lolo"- v2['price'] = Decimal("150")+ v2["name"] = "lolo"+ v2["price"] = Decimal("150")# standard assignment- p['variants'] = [v1, v2] # OK+ p["variants"] = [v1, v2] # OK# can also instantiate at assignment time- p['variants'] = [v1, Variant(name="lolo", price=Decimal("150")]+ p["variants"] = [v1, Variant(name="lolo", price=Decimal("150"))]# this also works as it tries to instantiate a Variant with the given dict- p['variants'] = [v1, {'name': 'lolo', 'price': Decimal("150")]- # this fails because it can't instantiate a Variant- p['variants'] = [v1, 'test']+ p["variants"] = [v1, {"name": "lolo", "price": Decimal("150")}]+ # this fails because it can't instantiate a Variant+ p["variants"] = [v1, "test"]# this fails because 'coco' is not a valid value for price- p['variants'] = [v1, {'name': 'lolo', 'price': 'coco']+ p["variants"] = [v1, {"name": "lolo", "price": "coco"}]Default values---------------::+.. code-block:: python#!pythonp = Product()- p['supplier'] # returns: Supplier(name='default supplier')- p['supplier2'] # raises KeyError- p['supplier2'] = Supplier()- p['supplier2'] # returns: Supplier(name='anonymous supplier')+ p["supplier"] # returns: Supplier(name='default supplier')+ p["supplier2"] # raises KeyError+ p["supplier2"] = Supplier()+ p["supplier2"] # returns: Supplier(name='anonymous supplier')- p['variants'] # raises KeyError- p['variants2'] # returns []+ p["variants"] # raises KeyError+ p["variants2"] # returns []- p['categories'] # raises KeyError- p.get('categories') # returns None+ p["categories"] # raises KeyError+ p.get("categories") # returns None- p['numbers'] # returns []+ p["numbers"] # returns []Accessing and changing nested item values-----------------------------------------::+.. code-block:: python#!pythonp = Product(supplier=Supplier(name="some name", url="http://example.com"))- p['supplier']['url'] # returns 'http://example.com'- p['supplier']['url'] = "http://www.other.com" # works as expected- p['supplier']['url'] = 123 # fails: wrong type for supplier url+ p["supplier"]["url"] # returns 'http://example.com'+ p["supplier"]["url"] = "http://www.other.com" # works as expected+ p["supplier"]["url"] = 123 # fails: wrong type for supplier url- p['variants'] = [v1, v2]- p['variants'][0]['name'] # returns v1 name- p['variants'][1]['name'] # returns v2 name+ p["variants"] = [v1, v2]+ p["variants"][0]["name"] # returns v1 name+ p["variants"][1]["name"] # returns v2 name# XXX: decide what to do about these cases:- p['variants'].append(v3) # works but doesn't check type of v3- p['variants'].append(1) # works but shouldn't?+ p["variants"].append(v3) # works but doesn't check type of v3+ p["variants"].append(1) # works but shouldn't?diff --git a/sep/sep-004.rst b/sep/sep-004.rstindex 05b0eb99c..b1cef2600 100644--- a/sep/sep-004.rst+++ b/sep/sep-004.rst@@ -26,7 +26,7 @@ Proposed APIHere's a simple proof-of-concept code of such script:-::+.. code-block:: python#!/usr/bin/env pythonfrom scrapy.http import Request@@ -35,21 +35,24 @@ Here's a simple proof-of-concept code of such script:# a container to hold scraped itemsscraped_items = []+def parse_start_page(response):# collect urls to follow into urls_to_follow listrequests = [Request(url, callback=parse_other_page) for url in urls_to_follow]return requests+def parse_other_page(response):# ... parse items from response content ...scraped_items.extend(parsed_items)+start_urls = ["http://www.example.com/start_page.html"]cr = Crawler(start_urls, callback=parse_start_page)- cr.run() # blocking call - this populates scraped_items+ cr.run() # blocking call - this populates scraped_items- print "%d items scraped" % len(scraped_items)+ print("%d items scraped" % len(scraped_items))# ... do something more interesting with scraped_items ...The behaviour of the Scrapy crawler would be controller by the Scrapy settings,diff --git a/sep/sep-005.rst b/sep/sep-005.rstindex 08ed367b3..aded289a5 100644--- a/sep/sep-005.rst+++ b/sep/sep-005.rst@@ -12,7 +12,7 @@ SEP-005: Detailed ``ItemBuilder`` API useItem class for examples:-::+.. code-block:: python#!pythonclass NewsItem(Item):@@ -25,7 +25,7 @@ Item class for examples:gSetting expanders==================-::+.. code-block:: python#!pythonclass NewsItemBuilder(ItemBuilder):@@ -44,7 +44,7 @@ on their Item Field class:gSetting reducers=================-::+.. code-block:: python#!pythonclass NewsItemBuilder(ItemBuilder):@@ -60,7 +60,7 @@ contentgSetting expanders/reducers new way===================================-::+.. code-block:: python#!pythonclass NewsItemBuilder(ItemBuilder):@@ -76,28 +76,29 @@ gSetting expanders/reducers new waygExtending ``ItemBuilder``==========================-::+.. code-block:: python#!pythonclass SiteNewsItemBuilder(NewsItemBuilder):- published = reducers.Reducer(extract, remove_tags(), unquote(),- strip, to_date('%d.%m.%Y'))+ published = reducers.Reducer(+ extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")+ )gExtending ``ItemBuilder`` using statich methods================================================-::+.. code-block:: python#!pythonclass SiteNewsItemBuilder(NewsItemBuilder):- published = reducers.Reducer(NewsItemBuilder.published, to_date('%d.%m.%Y'))+ published = reducers.Reducer(NewsItemBuilder.published, to_date("%d.%m.%Y"))gUsing default_builder======================-::+.. code-block:: python#!pythonclass DefaultedNewsItemBuilder(ItemBuilder):@@ -112,7 +113,7 @@ As a reducer is not set reducers will be set based on Item Field classes.gReset default_builder for a field==================================-::+.. code-block:: python#!pythonclass DefaultedNewsItemBuilder(ItemBuilder):@@ -125,18 +126,20 @@ gReset default_builder for a fieldgExtending default ``ItemBuilder``==================================-::+.. code-block:: python#!pythonclass SiteNewsItemBuilder(NewsItemBuilder):- published = reducers.Reducer(extract, remove_tags(), unquote(), strip, to_date('%d.%m.%Y'))+ published = reducers.Reducer(+ extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")+ )gExtending default ``ItemBuilder`` using static methods=======================================================-::+.. code-block:: python#!pythonclass SiteNewsItemBuilder(NewsItemBuilder):- published = reducers.Reducer(NewsItemBuilder.default_builder, to_date('%d.%m.%Y'))+ published = reducers.Reducer(NewsItemBuilder.default_builder, to_date("%d.%m.%Y"))diff --git a/sep/sep-008.rst b/sep/sep-008.rstindex b28bb548e..be5987e39 100644--- a/sep/sep-008.rst+++ b/sep/sep-008.rst@@ -87,11 +87,12 @@ Alternative Public API ProposalUsage example: declaring Item Parsers=====================================-::+.. code-block:: python#!pythonfrom scrapy.contrib.itemparser import XPathItemParser, parsers+class ProductParser(XPathItemParser):name_in = parsers.MapConcat(removetags, filterx)price_in = parsers.MapConcat(...)@@ -101,7 +102,7 @@ Usage example: declaring Item ParsersUsage example: declaring parsers in Fields==========================================-::+.. code-block:: python#!pythonclass Product(Item):diff --git a/sep/sep-009.rst b/sep/sep-009.rstindex da87fa9aa..e7b200d6e 100644--- a/sep/sep-009.rst+++ b/sep/sep-009.rst@@ -76,28 +76,30 @@ which we haven't documented so far (partly because of this).So, for a typical middleware ``__init__`` method code, instead of this:-::+.. code-block:: python#!pythonfrom scrapy.core.exceptions import NotConfiguredfrom scrapy.conf import settings+class SomeMiddleware(object):- def __init__(self):- if not settings.getbool('SOMEMIDDLEWARE_ENABLED'):- raise NotConfigured+ def __init__(self):+ if not settings.getbool("SOMEMIDDLEWARE_ENABLED"):+ raise NotConfiguredWe'd write this:-::+.. code-block:: python#!pythonfrom scrapy.core.exceptions import NotConfigured+class SomeMiddleware(object):- def __init__(self, crawler):- if not crawler.settings.getbool('SOMEMIDDLEWARE_ENABLED'):- raise NotConfigured+ def __init__(self, crawler):+ if not crawler.settings.getbool("SOMEMIDDLEWARE_ENABLED"):+ raise NotConfiguredRunning from command line=========================diff --git a/sep/sep-012.rst b/sep/sep-012.rstindex 8953cbf99..d41d0cf69 100644--- a/sep/sep-012.rst+++ b/sep/sep-012.rst@@ -83,10 +83,10 @@ example:$ cat project/spiders/google.py-::+.. code-block:: pythonclass GooglecomSpider(BaseSpider):- name = 'google'- allowed_domains = ['google.com']+ name = "google"+ allowed_domains = ["google.com"].. note:: ``spider_allowed_domains`` becomes optional as only ``OffsiteMiddleware`` uses it.diff --git a/sep/sep-014.rst b/sep/sep-014.rstindex 2521aa0e5..b84ee51c9 100644--- a/sep/sep-014.rst+++ b/sep/sep-014.rst@@ -92,7 +92,7 @@ Usage ExamplesBasic Crawling---------------::+.. code-block:: python#!python#@@ -101,20 +101,20 @@ Basic Crawlingclass SampleSpider(CrawlSpider):rules = [# The dispatcher uses first-match policy- Rule(UrlRegexMatch(r'product\.html\?id=\d+'), 'parse_item', follow=False),+ Rule(UrlRegexMatch(r"product\.html\?id=\d+"), "parse_item", follow=False),# by default, if the first param is string is wrapped into UrlRegexMatch- Rule(r'.+', 'parse_page'),- ]+ Rule(r".+", "parse_page"),+ ]request_extractors = [# crawl all links looking for products and imagesSgmlRequestExtractor(),- ]+ ]request_processors = [# canonicalize all requests' urlsCanonicalize(),- ]+ ]def parse_item(self, response):# parse and extract items from response@@ -127,7 +127,7 @@ Basic CrawlingCustom Processor and External Callback---------------------------------------::+.. code-block:: python#!python#@@ -137,30 +137,32 @@ Custom Processor and External Callback# Custom Processordef filter_today_links(requests):# only crawl today links- today = datetime.datetime.today().strftime('%Y-%m-%d')+ today = datetime.datetime.today().strftime("%Y-%m-%d")return [r for r in requests if today in r.url]+# Callback defined out of spiderdef my_external_callback(response):- # process item+ # process itempass+class SampleSpider(CrawlSpider):rules = [# The dispatcher uses first-match policy- Rule(UrlRegexMatch(r'/news/(.+)/'), my_external_callback),- ]+ Rule(UrlRegexMatch(r"/news/(.+)/"), my_external_callback),+ ]request_extractors = [- RegexRequestExtractor(r'/sections/.+'),- RegexRequestExtractor(r'/news/.+'),- ]+ RegexRequestExtractor(r"/sections/.+"),+ RegexRequestExtractor(r"/news/.+"),+ ]request_processors = [# canonicalize all requests' urlsCanonicalize(),filter_today_links,- ]+ ]Implementation==============@@ -199,7 +201,7 @@ Package StructureRequest/Response Matchers--------------------------::+.. code-block:: python#!python"""@@ -208,6 +210,7 @@ Request/Response MatchersPerform evaluation to Request or Response attributes"""+class BaseMatcher(object):"""Base matcher. Returns True by default."""@@ -229,11 +232,11 @@ Request/Response Matchersdef matches_url(self, url):"""Returns True if given url is equal to matcher's url"""- return self._url url+ return self._url == urldef matches_request(self, request):"""Returns True if Request's url matches initial url"""- return self.matches_url(request.url)+ return self.matches_url(request.url)def matches_response(self, response):"""REturns True if Response's url matches initial url"""@@ -254,7 +257,7 @@ Request/Response MatchersRequest Extractor------------------::+.. code-block:: python#!python#@@ -262,21 +265,21 @@ Request Extractor# Extractors receive response and return list of Requests#+class BaseSgmlRequestExtractor(FixedSGMLParser):"""Base SGML Request Extractor"""- def __init__(self, tag='a', attr='href'):+ def __init__(self, tag="a", attr="href"):"""Initialize attributes"""FixedSGMLParser.__init__(self)- self.scan_tag = tag if callable(tag) else lambda t: t tag- self.scan_attr = attr if callable(attr) else lambda a: a attr+ self.scan_tag = tag if callable(tag) else lambda t: t = tag+ self.scan_attr = attr if callable(attr) else lambda a: a = attrself.current_request = Nonedef extract_requests(self, response):"""Returns list of requests extracted from response"""- return self._extract_requests(response.body, response.url,- response.encoding)+ return self._extract_requests(response.body, response.url, response.encoding)def _extract_requests(self, response_text, response_url, response_encoding):"""Extract requests with absolute urls"""@@ -303,20 +306,19 @@ Request Extractordef _fix_link_text_encoding(self, encoding):"""Convert link_text to unicode for each request"""for req in self.requests:- req.meta.setdefault('link_text', '')- req.meta['link_text'] = str_to_unicode(req.meta['link_text'],- encoding)+ req.meta.setdefault("link_text", "")+ req.meta["link_text"] = str_to_unicode(req.meta["link_text"], encoding)def reset(self):"""Reset state"""FixedSGMLParser.reset(self)self.requests = []self.base_url = None-+def unknown_starttag(self, tag, attrs):"""Process unknown start tag"""- if 'base' tag:- self.base_url = dict(attrs).get('href')+ if "base" == tag:+ self.base_url = dict(attrs).get("href")if self.scan_tag(tag):for attr, value in attrs:@@ -333,8 +335,8 @@ Request Extractordef handle_data(self, data):"""Process data"""current = self.current_request- if current and not 'link_text' in current.meta:- current.meta['link_text'] = data.strip()+ if current and not "link_text" in current.meta:+ current.meta["link_text"] = data.strip()class SgmlRequestExtractor(BaseSgmlRequestExtractor):@@ -343,8 +345,8 @@ Request Extractordef __init__(self, tags=None, attrs=None):"""Initialize with custom tag & attribute function checkers"""# defaults- tags = tuple(tags) if tags else ('a', 'area')- attrs = tuple(attrs) if attrs else ('href', )+ tags = tuple(tags) if tags else ("a", "area")+ attrs = tuple(attrs) if attrs else ("href",)tag_func = lambda x: x in tagsattr_func = lambda x: x in attrs@@ -362,25 +364,26 @@ Request Extractordef extract_requests(self, response):"""Restrict to XPath regions"""hxs = HtmlXPathSelector(response)- fragments = (''.join(- html_frag for html_frag in hxs.select(xpath).extract()- ) for xpath in self.restrict_xpaths)- html_slice = ''.join(html_frag for html_frag in fragments)- return self._extract_requests(html_slice, response.url,- response.encoding)+ fragments = (+ "".join(html_frag for html_frag in hxs.select(xpath).extract())+ for xpath in self.restrict_xpaths+ )+ html_slice = "".join(html_frag for html_frag in fragments)+ return self._extract_requests(html_slice, response.url, response.encoding)Request Processor------------------::+.. code-block:: python#!python#- # Request Processors+ # Request Processors# Processors receive list of requests and return list of requests#"""Request Processors"""+class Canonicalize(object):"""Canonicalize Request Processor"""@@ -390,14 +393,14 @@ Request Processor# replace in-placereq.url = canonicalize_url(req.url)yield req-+class Unique(object):"""Filter duplicate Requests"""def __init__(self, *attributes):"""Initialize comparison attributes"""- self._attributes = attributes or ['url']+ self._attributes = attributes or ["url"]def _requests_equal(self, req1, req2):"""Attribute comparison helper"""@@ -430,20 +433,24 @@ Request Processor"""Filter request's domain"""def __init__(self, allow=(), deny=()):- """Initialize allow/deny attributes"""- self.allow = tuple(arg_to_iter(allow))- self.deny = tuple(arg_to_iter(deny))+ """Initialize allow/deny attributes"""+ self.allow = tuple(arg_to_iter(allow))+ self.deny = tuple(arg_to_iter(deny))def __call__(self, requests):"""Filter domains"""processed = (req for req in requests)if self.allow:- processed = (req for req in requests- if url_is_from_any_domain(req.url, self.allow))+ processed = (+ req for req in requests if url_is_from_any_domain(req.url, self.allow)+ )if self.deny:- processed = (req for req in requests- if not url_is_from_any_domain(req.url, self.deny))+ processed = (+ req+ for req in requests+ if not url_is_from_any_domain(req.url, self.deny)+ )return processed@@ -453,24 +460,28 @@ Request Processordef __init__(self, allow=(), deny=()):"""Initialize allow/deny attributes"""- _re_type = type(re.compile('', 0))+ _re_type = type(re.compile("", 0))- self.allow_res = [x if isinstance(x, _re_type) else re.compile(x)- for x in arg_to_iter(allow)]- self.deny_res = [x if isinstance(x, _re_type) else re.compile(x)- for x in arg_to_iter(deny)]+ self.allow_res = [+ x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(allow)+ ]+ self.deny_res = [+ x if isinstance(x, _re_type) else re.compile(x) for x in arg_to_iter(deny)+ ]def __call__(self, requests):"""Filter request's url based on allow/deny rules"""- #TODO: filter valid urls here?+ # TODO: filter valid urls here?processed = (req for req in requests)if self.allow_res:- processed = (req for req in requests- if self._matches(req.url, self.allow_res))+ processed = (+ req for req in requests if self._matches(req.url, self.allow_res)+ )if self.deny_res:- processed = (req for req in requests- if not self._matches(req.url, self.deny_res))+ processed = (+ req for req in requests if not self._matches(req.url, self.deny_res)+ )return processed@@ -481,7 +492,7 @@ Request ProcessorRule Object------------::+.. code-block:: python#!python#@@ -490,8 +501,10 @@ Rule Object#class Rule(object):"""Crawler Rule"""- def __init__(self, matcher, callback=None, cb_args=None,- cb_kwargs=None, follow=True):++ def __init__(+ self, matcher, callback=None, cb_args=None, cb_kwargs=None, follow=True+ ):"""Store attributes"""self.matcher = matcherself.callback = callback@@ -499,12 +512,14 @@ Rule Objectself.cb_kwargs = cb_kwargs if cb_kwargs else {}self.follow = follow+## Rules Manager takes list of Rule objects and normalize matcher and callback# into CompiledRule#class CompiledRule(object):"""Compiled version of Rule"""+def __init__(self, matcher, callback=None, follow=False):"""Initialize attributes checking type"""assert isinstance(matcher, BaseMatcher)@@ -518,15 +533,16 @@ Rule ObjectRules Manager--------------::+.. code-block:: python#!python## Handles rules matcher/callbacks# Resolve rule for given response- #+ #class RulesManager(object):"""Rules Manager"""+def __init__(self, rules, spider, default_matcher=UrlRegexMatcher):"""Initialize rules using spider and default matcher"""self._rules = tuple()@@ -542,8 +558,9 @@ Rules Manager# instance default matchermatcher = default_matcher(rule.matcher)else:- raise ValueError('Not valid matcher given %r in %r' \- % (rule.matcher, rule))+ raise ValueError(+ "Not valid matcher given %r in %r" % (rule.matcher, rule)+ )# prepare callbackif callable(rule.callback):@@ -553,8 +570,9 @@ Rules Managercallback = getattr(spider, rule.callback)if not callable(callback):- raise AttributeError('Invalid callback %r can not be resolved' \- % callback)+ raise AttributeError(+ "Invalid callback %r can not be resolved" % callback+ )else:callback = None@@ -564,7 +582,7 @@ Rules Manager# append compiled rule to rules listcrule = CompiledRule(matcher, callback, follow=rule.follow)- self._rules += (crule, )+ self._rules += (crule,)def get_rule(self, response):"""Returns first rule that matches response"""@@ -575,7 +593,7 @@ Rules ManagerRequest Generator------------------::+.. code-block:: python#!python#@@ -605,7 +623,7 @@ Request Generator``CrawlSpider``------------------::+.. code-block:: python#!python#@@ -625,9 +643,9 @@ Request Generator# wrap rulesself._rulesman = RulesManager(self.rules, spider=self)# generates new requests with given callback- self._reqgen = RequestGenerator(self.request_extractors,- self.request_processors,- self.parse)+ self._reqgen = RequestGenerator(+ self.request_extractors, self.request_processors, self.parse+ )def parse(self, response):"""Dispatch callback and generate requests"""diff --git a/sep/sep-016.rst b/sep/sep-016.rstindex a60ab30dd..e8ac57d11 100644--- a/sep/sep-016.rst+++ b/sep/sep-016.rst@@ -67,21 +67,21 @@ Regex (HTML) Link ExtractorA typical application of LegSpider's is to build Link Extractors. For example:-::+.. code-block:: python#!pythonclass RegexHtmlLinkExtractor(LegSpider):-def process_response(self, response):if isinstance(response, HtmlResponse):allowed_regexes = self.spider.url_regexes_to_follow# extract urls to follow using allowed_regexesreturn [Request(x) for x in urls_to_follow]+class MySpider(LegSpider):legs = [RegexHtmlLinkExtractor()]- url_regexes_to_follow = ['/product.php?.*']+ url_regexes_to_follow = ["/product.php?.*"]def parse_response(self, response):# parse response and extract items@@ -92,13 +92,12 @@ RSS2 link extractorThis is a Leg Spider that can be used for following links from RSS2 feeds.-::+.. code-block:: python#!pythonclass Rss2LinkExtractor(LegSpider):-def process_response(self, response):- if response.headers.get('Content-type') 'application/rss+xml':+ if response.headers.get("Content-type") == "application/rss+xml":xs = XmlXPathSelector(response)urls = xs.select("//item/link/text()").extract()return [Request(x) for x in urls]@@ -108,11 +107,10 @@ Callback dispatcher based on rulesAnother example could be to build a callback dispatcher based on rules:-::+.. code-block:: python#!pythonclass CallbackRules(LegSpider):-def __init__(self, *a, **kw):super(CallbackRules, self).__init__(*a, **kw)for regex, method_name in self.spider.callback_rules.items():@@ -128,12 +126,13 @@ Another example could be to build a callback dispatcher based on rules:return method(response)return []+class MySpider(LegSpider):legs = [CallbackRules()]callback_rules = {- '/product.php.*': 'parse_product',- '/category.php.*': 'parse_category',+ "/product.php.*": "parse_product",+ "/category.php.*": "parse_category",}def parse_product(self, response):@@ -145,19 +144,19 @@ URL CanonicalizersAnother example could be for building URL canonicalizers:-::+.. code-block:: python#!pythonclass CanonicalizeUrl(LegSpider):-def process_request(self, request):curl = canonicalize_url(request.url, rules=self.spider.canonicalization_rules)return request.replace(url=curl)+class MySpider(LegSpider):legs = [CanonicalizeUrl()]- canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]+ canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]# ...@@ -167,22 +166,22 @@ Setting item identifierAnother example could be for setting a unique identifier to items, based oncertain fields:-::+.. code-block:: python#!pythonclass ItemIdSetter(LegSpider):-def process_item(self, item):id_field = self.spider.id_fieldid_fields_to_hash = self.spider.id_fields_to_hashitem[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)return item+class MySpider(LegSpider):legs = [ItemIdSetter()]- id_field = 'guid'- id_fields_to_hash = ['supplier_name', 'supplier_id']+ id_field = "guid"+ id_fields_to_hash = ["supplier_name", "supplier_id"]def process_response(self, item):# extract item from response@@ -193,24 +192,24 @@ Combining multiple leg spidersHere's an example that combines functionality from multiple leg spiders:-::+.. code-block:: python#!pythonclass MySpider(LegSpider):legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()]- url_regexes_to_follow = ['/product.php?.*']+ url_regexes_to_follow = ["/product.php?.*"]parse_rules = {- '/product.php.*': 'parse_product',- '/category.php.*': 'parse_category',+ "/product.php.*": "parse_product",+ "/category.php.*": "parse_category",}- canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]+ canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]- id_field = 'guid'- id_fields_to_hash = ['supplier_name', 'supplier_id']+ id_field = "guid"+ id_fields_to_hash = ["supplier_name", "supplier_id"]def process_product(self, item):# extract item from response@@ -249,7 +248,7 @@ important to keep in mind their scope and limitations, such as:Here's a proof-of-concept implementation of ``LegSpider``:-::+.. code-block:: python#!pythonfrom scrapy.http import Requestdiff --git a/sep/sep-017.rst b/sep/sep-017.rstindex 86005e3c9..28e20a86e 100644--- a/sep/sep-017.rst+++ b/sep/sep-017.rst@@ -35,16 +35,15 @@ gExample URL for simple callbackThe ``parse_product`` callback must return items containing the fields given in``@scrapes``.-::+.. code-block:: python#!pythonclass ProductSpider(BaseSpider):-def parse_product(self, response):"""@url http://www.example.com/store/product.php?id=123@scrapes name, price, description- """"+ """gChained callbacks------------------@@ -55,11 +54,10 @@ other for scraping user profile info.The contracts assert that the first callback returns a Request and the secondone scrape ``user, name, email`` fields.-::+.. code-block:: python#!pythonclass UserProfileSpider(BaseSpider):-def parse_login_page(self, response):"""@url http://www.example.com/login.php@@ -71,7 +69,7 @@ one scrape ``user, name, email`` fields."""@after parse_login_page@scrapes user, name, email- """"+ """# ...Tags referencediff --git a/sep/sep-018.rst b/sep/sep-018.rstindex 96df82f6b..1c1e00e63 100644--- a/sep/sep-018.rst+++ b/sep/sep-018.rst@@ -166,27 +166,32 @@ written, it should work both globally and per spider.Here's an example that combines functionality from multiple middlewares intothe same spider:-::+.. code-block:: python#!pythonclass MySpider(BaseSpider):- middlewares = [RegexLinkExtractor(), CallbackRules(), CanonicalizeUrl(),- ItemIdSetter(), OffsiteMiddleware()]+ middlewares = [+ RegexLinkExtractor(),+ CallbackRules(),+ CanonicalizeUrl(),+ ItemIdSetter(),+ OffsiteMiddleware(),+ ]- allowed_domains = ['example.com', 'sub.example.com']+ allowed_domains = ["example.com", "sub.example.com"]- url_regexes_to_follow = ['/product.php?.*']+ url_regexes_to_follow = ["/product.php?.*"]callback_rules = {- '/product.php.*': 'parse_product',- '/category.php.*': 'parse_category',+ "/product.php.*": "parse_product",+ "/category.php.*": "parse_category",}- canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]+ canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]- id_field = 'guid'- id_fields_to_hash = ['supplier_name', 'supplier_id']+ id_field = "guid"+ id_fields_to_hash = ["supplier_name", "supplier_id"]def parse_product(self, item):# extract item from response@@ -234,35 +239,34 @@ Regex (HTML) Link ExtractorA typical application of spider middlewares could be to build Link Extractors.For example:-::+.. code-block:: python#!pythonclass RegexHtmlLinkExtractor(object):-def process_response(self, response, request, spider):if isinstance(response, HtmlResponse):allowed_regexes = spider.url_regexes_to_follow# extract urls to follow using allowed_regexesreturn [Request(x) for x in urls_to_follow]+# Example spider using this middlewareclass MySpider(BaseSpider):middlewares = [RegexHtmlLinkExtractor()]- url_regexes_to_follow = ['/product.php?.*']+ url_regexes_to_follow = ["/product.php?.*"]# parsing callbacks belowRSS2 link extractor--------------------::+.. code-block:: python#!pythonclass Rss2LinkExtractor(object):-def process_response(self, response, request, spider):- if response.headers.get('Content-type') 'application/rss+xml':+ if response.headers.get("Content-type") == "application/rss+xml":xs = XmlXPathSelector(response)urls = xs.select("//item/link/text()").extract()return [Request(x) for x in urls]@@ -272,11 +276,10 @@ Callback dispatcher based on rulesAnother example could be to build a callback dispatcher based on rules:-::+.. code-block:: python#!pythonclass CallbackRules(object):-def __init__(self):self.rules = {}dispatcher.connect(signals.spider_opened, self.spider_opened)@@ -300,13 +303,14 @@ Another example could be to build a callback dispatcher based on rules:return method(response)return []+# Example spider using this middlewareclass MySpider(BaseSpider):middlewares = [CallbackRules()]callback_rules = {- '/product.php.*': 'parse_product',- '/category.php.*': 'parse_category',+ "/product.php.*": "parse_product",+ "/category.php.*": "parse_category",}def parse_product(self, response):@@ -318,22 +322,20 @@ URL CanonicalizersAnother example could be for building URL canonicalizers:-::+.. code-block:: python#!pythonclass CanonicalizeUrl(object):-def process_request(self, request, response, spider):- curl = canonicalize_url(request.url,- rules=spider.canonicalization_rules)+ curl = canonicalize_url(request.url, rules=spider.canonicalization_rules)return request.replace(url=curl)+# Example spider using this middlewareclass MySpider(BaseSpider):middlewares = [CanonicalizeUrl()]- canonicalization_rules = ['sort-query-args',- 'normalize-percent-encoding', ...]+ canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]# ...@@ -343,23 +345,23 @@ Setting item identifierAnother example could be for setting a unique identifier to items, based oncertain fields:-::+.. code-block:: python#!pythonclass ItemIdSetter(object):-def process_item(self, item, response, spider):id_field = spider.id_fieldid_fields_to_hash = spider.id_fields_to_hashitem[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)return item+# Example spider using this middlewareclass MySpider(BaseSpider):middlewares = [ItemIdSetter()]- id_field = 'guid'- id_fields_to_hash = ['supplier_name', 'supplier_id']+ id_field = "guid"+ id_fields_to_hash = ["supplier_name", "supplier_id"]def parse(self, response):# extract item from response@@ -370,11 +372,10 @@ robots.txt exclusionA spider middleware to avoid visiting pages forbidden by robots.txt:-::+.. code-block:: python#!pythonclass SpiderInfo(object):-def __init__(self, useragent):self.useragent = useragentself.parsers = {}@@ -382,7 +383,6 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:class AllowAllParser(object):-def can_fetch(useragent, url):return True@@ -397,7 +397,7 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:dispatcher.connect(self.spider_closed, signal=signals.spider_closed)def process_request(self, request, response, spider):- return self.process_start_request(self, request)+ return self.process_start_request(request)def process_start_request(self, request, spider):info = self.spiders[spider]@@ -415,17 +415,21 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:res = Noneelse:robotsurl = "%s://%s/robots.txt" % (url.scheme, netloc)- meta = {'spider': spider, {'handle_httpstatus_list': [403, 404, 500]}- res = Request(robotsurl, callback=self.parse_robots,- meta=meta, priority=self.REQUEST_PRIORITY)+ meta = {"spider": spider, "handle_httpstatus_list": [403, 404, 500]}+ res = Request(+ robotsurl,+ callback=self.parse_robots,+ meta=meta,+ priority=self.REQUEST_PRIORITY,+ )info.pending[netloc].append(request)return resdef parse_robots(self, response):- spider = response.request.meta['spider']- netloc urlparse_cached(response).netloc+ spider = response.request.meta["spider"]+ netloc = urlparse_cached(response).netlocinfo = self.spiders[spider]- if response.status 200;+ if response.status == 200:rp = robotparser.RobotFileParser(response.url)rp.parse(response.body.splitlines())info.parsers[netloc] = rp@@ -434,7 +438,7 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:return info.pending[netloc]def spider_opened(self, spider):- ua = getattr(spider, 'user_agent', None) or settings['USER_AGENT']+ ua = getattr(spider, "user_agent", None) or settings["USER_AGENT"]self.spiders[spider] = SpiderInfo(ua)def spider_closed(self, spider):@@ -445,18 +449,16 @@ Offsite middlewareThis is a port of the Offsite middleware to the new spider middleware API:-::+.. code-block:: python#!pythonclass SpiderInfo(object):-def __init__(self, host_regex):self.host_regex = host_regexself.hosts_seen = set()class OffsiteMiddleware(object):-def __init__(self):self.spiders = {}dispatcher.connect(self.spider_opened, signal=signals.spider_opened)@@ -472,19 +474,19 @@ This is a port of the Offsite middleware to the new spider middleware API:info = self.spiders[spider]host = urlparse_cached(x).hostnameif host and host not in info.hosts_seen:- spider.log("Filtered offsite request to %r: %s" % (host, request))- info.hosts_seen.add(host)-+ spider.log("Filtered offsite request to %r: %s" % (host, request))+ info.hosts_seen.add(host)+def should_follow(self, request, spider):info = self.spiders[spider]# hostname can be None for wrong urls (like javascript links)- host = urlparse_cached(request).hostname or ''+ host = urlparse_cached(request).hostname or ""return bool(info.regex.search(host))def get_host_regex(self, spider):"""Override this method to implement a different offsite policy"""- domains = [d.replace('.', r'\.') for d in spider.allowed_domains]- regex = r'^(.*\.)?(%s)$' % '|'.join(domains)+ domains = [d.replace(".", r"\.") for d in spider.allowed_domains]+ regex = r"^(.*\.)?(%s)$" % "|".join(domains)return re.compile(regex)def spider_opened(self, spider):@@ -499,35 +501,36 @@ Limit URL lengthA middleware to filter out requests with long urls:-::+.. code-block:: python#!python- class LimitUrlLength(object):+ class LimitUrlLength(object):def __init__(self):- self.maxlength = settings.getint('URLLENGTH_LIMIT')+ self.maxlength = settings.getint("URLLENGTH_LIMIT")def process_request(self, request, response, spider):return self.process_start_request(self, request)def process_start_request(self, request, spider):- if len(request.url) <= self.maxlength:+ if len(request.url) <= self.maxlength:return request- spider.log("Ignoring request (url length > %d): %s " % (self.maxlength, request.url))+ spider.log(+ "Ignoring request (url length > %d): %s " % (self.maxlength, request.url)+ )Set Referer-----------A middleware to set the Referer:-::+.. code-block:: python#!pythonclass SetReferer(object):-def process_request(self, request, response, spider):- request.headers.setdefault('Referer', response.url)+ request.headers.setdefault("Referer", response.url)return requestSet and limit crawling depth@@ -536,23 +539,22 @@ Set and limit crawling depthA middleware to set (and limit) the request/response depth, taken from thestart requests:-::+.. code-block:: python#!pythonclass SetLimitDepth(object):-def __init__(self, maxdepth=0):- self.maxdepth = maxdepth or settings.getint('DEPTH_LIMIT')+ self.maxdepth = maxdepth or settings.getint("DEPTH_LIMIT")def process_request(self, request, response, spider):- depth = response.request.meta['depth'] + 1- request.meta['depth'] = depth+ depth = response.request.meta["depth"] + 1+ request.meta["depth"] = depthif not self.maxdepth or depth <= self.maxdepth:return request- spider.log("Ignoring link (depth > %d): %s " % (self.maxdepth, request)+ spider.log("Ignoring link (depth > %d): %s " % (self.maxdepth, request))def process_start_request(self, request, spider):- request.meta['depth'] = 0+ request.meta["depth"] = 0return requestFilter duplicate requests@@ -560,17 +562,16 @@ Filter duplicate requestsA middleware to filter out requests already seen:-::+.. code-block:: python#!pythonclass FilterDuplicates(object):-def __init__(self):- clspath = settings.get('DUPEFILTER_CLASS')+ clspath = settings.get("DUPEFILTER_CLASS")self.dupefilter = load_object(clspath)()dispatcher.connect(self.spider_opened, signal=signals.spider_opened)dispatcher.connect(self.spider_closed, signal=signals.spider_closed)-+def enqueue_request(self, spider, request):seen = self.dupefilter.request_seen(spider, request)if not seen or request.dont_filter:@@ -587,22 +588,25 @@ Scrape data using ParsleyA middleware to Scrape data using Parsley as described in UsingParsley-::+.. code-block:: python#!pythonfrom pyparsley import PyParsley- class ParsleyExtractor(object):+ class ParsleyExtractor(object):def __init__(self, parsley_json_code):parsley = json.loads(parselet_json_code)+class ParsleyItem(Item):def __init__(self, *a, **kw):for name in parsley.keys():self.fields[name] = Field()+super(ParsleyItem, self).__init__(*a, **kw)- self.item_class = ParsleyItem- self.parsley = PyParsley(parsley, output='python')++ self.item_class = ParsleyItem+ self.parsley = PyParsley(parsley, output="python")def process_response(self, response, request, spider):return self.item_class(self.parsley.parse(string=response.body))diff --git a/sep/sep-019.rst b/sep/sep-019.rstindex 84f3a96c3..5bc97ac40 100644--- a/sep/sep-019.rst+++ b/sep/sep-019.rst@@ -15,10 +15,11 @@ consistent way, while taking the chance to refactor the settings populationand whole crawl workflow.In short, you will be able to overwrite settings (on a per-spider basis) by-implementing a class method in your spider::+implementing a class method in your spider:- class MySpider(Spider):+.. code-block:: python+ class MySpider(Spider):@classmethoddef custom_settings(cls):return {@@ -197,10 +198,11 @@ SpidersA new class method ``custom_settings`` is proposed, that could be use tooverride project and default settings before they're used to instantiate the-crawler::+crawler:- class MySpider(Spider):+.. code-block:: python+ class MySpider(Spider):@classmethoddef custom_settings(cls):return {diff --git a/sep/sep-020.rst b/sep/sep-020.rstindex 52d78097b..c6c25bda2 100644--- a/sep/sep-020.rst+++ b/sep/sep-020.rst@@ -54,18 +54,18 @@ required.Before-------::+.. code-block:: pythonxpath = '//div[@class="geeks"]/dl/dt[contains(text(),"%s")]/following-sibling::dd[1]//text()'gl = XPathItemLoader(response=response, item=dict())gl.default_output_processor = Compose(TakeFirst(), lambda v: v.strip())- gl.add_xpath('hacker', xpath % 'hacker')- gl.add_xpath('nerd', xpath % 'nerd')+ gl.add_xpath("hacker", xpath % "hacker")+ gl.add_xpath("nerd", xpath % "nerd")After------::+.. code-block:: pythonbil = BulkItemLoader(response=response)bil.parse_dl('//div[@class="geeks"]/dl')@@ -75,33 +75,34 @@ Code ProposalThis is a working code sample that covers just the basics.-::+.. code-block:: pythonfrom scrapy.contrib.loader import XPathItemLoaderfrom scrapy.contrib.loader.processor import MapCompose+class BulkItemLoader(XPathItemLoader):- """ Item loader based on specified pattern recognition- """+ """Item loader based on specified pattern recognition"""+default_item_class = dict- base_xpath = '//body'+ base_xpath = "//body"ignore = ()def _get_label(self, entity):- """ Pull the text label out of selected markup+ """Pull the text label out of selected markup:param entity: Found markup:type entity: Selector"""- label = ' '.join(entity.xpath('.//text()').extract())- label = label.encode('ascii', 'xmlcharrefreplace') if label else ''- label = label.strip(' ') if ' ' in label else label- label = label.strip(':') if ':' in label else label+ label = " ".join(entity.xpath(".//text()").extract())+ label = label.encode("ascii", "xmlcharrefreplace") if label else ""+ label = label.strip(" ") if " " in label else label+ label = label.strip(":") if ":" in label else labellabel = label.strip()return labeldef _get_entities(self, xpath):- """ Retrieve the list of selectors for a given sub-pattern+ """Retrieve the list of selectors for a given sub-pattern:param xpath: The xpath to select:type xpath: String@@ -110,20 +111,21 @@ This is a working code sample that covers just the basics."""return self.selector.xpath(self.base_xpath + xpath)- def parse_dl(self, xpath=u'//dl'):- """ Look for the specified definition list pattern and store all found+ def parse_dl(self, xpath="//dl"):+ """Look for the specified definition list pattern and store all foundvalues for the enclosed terms and descriptions.:param xpath: The xpath to select:type xpath: String"""- for term in self._get_entities(xpath + '/dt'):+ for term in self._get_entities(xpath + "/dt"):label = self._get_label(term)if label and label not in self.ignore:- value = term.xpath('following-sibling::dd[1]//text()')+ value = term.xpath("following-sibling::dd[1]//text()")if value:- self.add_value(label, value.extract(),- MapCompose(lambda v: v.strip()))+ self.add_value(+ label, value.extract(), MapCompose(lambda v: v.strip())+ )Example Spider==============@@ -133,22 +135,24 @@ This spider uses the bulk loader above.Spider code------------::+.. code-block:: pythonfrom scrapy.spider import BaseSpiderfrom scrapy.contrib.loader.bulk import BulkItemLoader+class W3cSpider(BaseSpider):name = "w3c"allowed_domains = ["w3.org"]- start_urls = ('http://www.w3.org/TR/html401/struct/lists.html',)+ start_urls = ("http://www.w3.org/TR/html401/struct/lists.html",)def parse(self, response):el = BulkItemLoader(response=response)- el.parse_dl('//dl[2]')+ el.parse_dl("//dl[2]")item = el.load_item()from pprint import pprint+pprint(item)Log Outputdiff --git a/sep/sep-021.rst b/sep/sep-021.rstindex d56bc26af..e8affa943 100644--- a/sep/sep-021.rst+++ b/sep/sep-021.rst@@ -76,14 +76,18 @@ addon_configureReceives the Settings object and modifies it to enable the required components.If it raises an exception, Scrapy will print it and exit.-Examples::+Examples:++.. code-block:: pythondef addon_configure(settings):- settings.overrides['DOWNLOADER_MIDDLEWARES'].update({- 'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,- })+ settings.overrides["DOWNLOADER_MIDDLEWARES"].update(+ {+ "scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware": 900,+ }+ )-::+.. code-block:: pythondef addon_configure(settings):try:@@ -100,8 +104,10 @@ is meant to be used to perform post-initialization checks like making sure theextension and its dependencies were configured properly. If it raises anexception, Scrapy will print and exit.-Examples::+Examples:++.. code-block:: pythondef crawler_ready(crawler):- if 'some.other.addon' not in crawler.extensions.enabled:+ if "some.other.addon" not in crawler.extensions.enabled:raise RuntimeError("Some other addon is required to use this addon")