Scrapy, a fast high-level web crawling & scraping framework for Python.
Update black version and fix minor typos and punctuations
pankaj1707k03f32c018f76
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yamlindex cfb5a46dc..4b90233e5 100644--- a/.pre-commit-config.yaml+++ b/.pre-commit-config.yaml@@ -21,4 +21,4 @@ repos:hooks:- id: blacken-docsadditional_dependencies:- - black==22.12.0+ - black==23.1.0diff --git a/docs/faq.rst b/docs/faq.rstindex 836420567..241a588a8 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -240,7 +240,6 @@ higher) in your spider:.. code-block:: pythonclass MySpider(CrawlSpider):-name = "myspider"download_delay = 2diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex 495aad091..542760b4f 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -20,7 +20,7 @@ In order to show you what Scrapy brings to the table, we'll walk you through anexample of a Scrapy Spider using the simplest way to run a spider.Here's the code for a spider that scrapes famous quotes from website-https://quotes.toscrape.com, following the pagination+https://quotes.toscrape.com, following the pagination:.. code-block:: pythondiff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex e565c3ebb..5904482a5 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -635,7 +635,7 @@ A shortcut for creating Requests--------------------------------As a shortcut for creating Request objects you can use-:meth:`response.follow <scrapy.http.TextResponse.follow>`+:meth:`response.follow <scrapy.http.TextResponse.follow>`:.. code-block:: pythondiff --git a/docs/topics/api.rst b/docs/topics/api.rstindex 1eac444e6..268344879 100644--- a/docs/topics/api.rst+++ b/docs/topics/api.rst@@ -132,8 +132,6 @@ Settings APIprecedence over lesser ones when setting and retrieving values in the:class:`~scrapy.settings.Settings` class.- .. highlight:: python-.. code-block:: pythonSETTINGS_PRIORITIES = {diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex e1c481c37..a7957dd3f 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -356,7 +356,6 @@ HttpAuthMiddlewareclass SomeIntranetSiteSpider(CrawlSpider):-http_user = "someuser"http_pass = "somepass"http_auth_domain = "intranet.example.com"diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rstindex 791560fb5..e9423d64d 100644--- a/docs/topics/item-pipeline.rst+++ b/docs/topics/item-pipeline.rst@@ -90,7 +90,6 @@ contain a price:class PricePipeline:-vat_factor = 1.15def process_item(self, item, spider):@@ -150,7 +149,6 @@ method and how to clean up the resources properly.class MongoPipeline:-collection_name = "scrapy_items"def __init__(self, mongo_uri, mongo_db):diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rstindex 45e19ef07..443785758 100644--- a/docs/topics/loaders.rst+++ b/docs/topics/loaders.rst@@ -201,7 +201,6 @@ Item Loaders are declared using a class definition syntax. Here is an example:class ProductLoader(ItemLoader):-default_output_processor = TakeFirst()name_in = MapCompose(str.title)diff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex 5e1de4425..fe1c4d162 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -117,7 +117,6 @@ instance, which can be accessed and used like this:class MySpider(scrapy.Spider):-name = "myspider"start_urls = ["https://scrapy.org"]@@ -136,7 +135,6 @@ Python logger you want. For example:class MySpider(scrapy.Spider):-name = "myspider"start_urls = ["https://scrapy.org"]diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex b516a6bef..f05e27c73 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -321,7 +321,7 @@ It receives a :exc:`~twisted.python.failure.Failure` as first parameter and canbe used to track connection establishment timeouts, DNS errors etc.Here's an example spider logging all errors and catching some specific-errors if needed+errors if needed:.. code-block:: python@@ -386,7 +386,7 @@ Accessing additional data in errback functionsIn case of a failure to process the request, you may be interested inaccessing arguments to the callback functions so you can process furtherbased on the arguments in the errback. The following example shows how to-achieve this by using ``Failure.request.cb_kwargs``+achieve this by using ``Failure.request.cb_kwargs``:.. code-block:: python@@ -593,7 +593,6 @@ URL canonicalization or taking the request method or body into account:class RequestFingerprinter:-cache = WeakKeyDictionary()def fingerprint(self, request):@@ -635,7 +634,6 @@ request fingerprinter:class RequestFingerprinter:-cache = WeakKeyDictionary()def fingerprint(self, request):diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rstindex 7f21e1583..c25c75d17 100644--- a/docs/topics/selectors.rst+++ b/docs/topics/selectors.rst@@ -320,8 +320,8 @@ Examples:>>> response.css("img::text").getall()[]- is means ``.css('foo::text').get()`` could return None even if an element- ists. Use ``default=''`` if you always want a string:+ This means ``.css('foo::text').get()`` could return None even if an element+ exists. Use ``default=''`` if you always want a string:.. code-block:: pycondiff --git a/sep/sep-014.rst b/sep/sep-014.rstindex b84ee51c9..e03a2b0f6 100644--- a/sep/sep-014.rst+++ b/sep/sep-014.rst@@ -134,6 +134,7 @@ Custom Processor and External Callback# Using external callbacks#+# Custom Processordef filter_today_links(requests):# only crawl today linksdiff --git a/sep/sep-016.rst b/sep/sep-016.rstindex e8ac57d11..39bd91645 100644--- a/sep/sep-016.rst+++ b/sep/sep-016.rst@@ -79,7 +79,6 @@ A typical application of LegSpider's is to build Link Extractors. For example:class MySpider(LegSpider):-legs = [RegexHtmlLinkExtractor()]url_regexes_to_follow = ["/product.php?.*"]@@ -128,7 +127,6 @@ Another example could be to build a callback dispatcher based on rules:class MySpider(LegSpider):-legs = [CallbackRules()]callback_rules = {"/product.php.*": "parse_product",@@ -154,7 +152,6 @@ Another example could be for building URL canonicalizers:class MySpider(LegSpider):-legs = [CanonicalizeUrl()]canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]@@ -178,7 +175,6 @@ certain fields:class MySpider(LegSpider):-legs = [ItemIdSetter()]id_field = "guid"id_fields_to_hash = ["supplier_name", "supplier_id"]@@ -196,7 +192,6 @@ Here's an example that combines functionality from multiple leg spiders:#!pythonclass MySpider(LegSpider):-legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()]url_regexes_to_follow = ["/product.php?.*"]diff --git a/sep/sep-018.rst b/sep/sep-018.rstindex 1c1e00e63..9ac62c090 100644--- a/sep/sep-018.rst+++ b/sep/sep-018.rst@@ -170,7 +170,6 @@ the same spider:#!pythonclass MySpider(BaseSpider):-middlewares = [RegexLinkExtractor(),CallbackRules(),@@ -252,7 +251,6 @@ For example:# Example spider using this middlewareclass MySpider(BaseSpider):-middlewares = [RegexHtmlLinkExtractor()]url_regexes_to_follow = ["/product.php?.*"]@@ -306,7 +304,6 @@ Another example could be to build a callback dispatcher based on rules:# Example spider using this middlewareclass MySpider(BaseSpider):-middlewares = [CallbackRules()]callback_rules = {"/product.php.*": "parse_product",@@ -333,7 +330,6 @@ Another example could be for building URL canonicalizers:# Example spider using this middlewareclass MySpider(BaseSpider):-middlewares = [CanonicalizeUrl()]canonicalization_rules = ["sort-query-args", "normalize-percent-encoding", ...]@@ -358,7 +354,6 @@ certain fields:# Example spider using this middlewareclass MySpider(BaseSpider):-middlewares = [ItemIdSetter()]id_field = "guid"id_fields_to_hash = ["supplier_name", "supplier_id"]@@ -388,7 +383,6 @@ A spider middleware to avoid visiting pages forbidden by robots.txt:class RobotsTxtMiddleware(object):-REQUEST_PRIORITY = 1000def __init__(self):