Scrapy, a fast high-level web crawling & scraping framework for Python.
Remove trailing whitespace
Adrián Chavesbee74fb753af
diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.ymlindex 8e01ffd88..d1589f4f7 100644--- a/.github/workflows/publish.yml+++ b/.github/workflows/publish.yml@@ -22,7 +22,7 @@ jobs:- uses: actions/setup-python@v5with:python-version: "3.13"- - run: |+ - run: |python -m pip install --upgrade buildpython -m build- name: Publish to PyPIdiff --git a/.pre-commit-config.yaml b/.pre-commit-config.yamlindex 18402b908..0d1a76247 100644--- a/.pre-commit-config.yaml+++ b/.pre-commit-config.yaml@@ -11,3 +11,7 @@ repos:- id: blacken-docsadditional_dependencies:- black==24.10.0+- repo: https://github.com/pre-commit/pre-commit-hooks+ rev: v5.0.0+ hooks:+ - id: trailing-whitespacediff --git a/README.rst b/README.rstindex 3f468953e..cf7c6043c 100644--- a/README.rst+++ b/README.rst@@ -1,6 +1,6 @@.. image:: https://scrapy.org/img/scrapylogo.png:target: https://scrapy.org/-+======Scrapy======diff --git a/docs/_tests/quotes.html b/docs/_tests/quotes.htmlindex f4002ecd1..d1cfd9020 100644--- a/docs/_tests/quotes.html+++ b/docs/_tests/quotes.html@@ -16,13 +16,13 @@</div><div class="col-md-4"><p>-+<a href="/login">Login</a>-+</p></div></div>-+<div class="row"><div class="col-md-8">@@ -34,16 +34,16 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world" / >-+ <meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world" / >+<a class="tag" href="/tag/change/page/1/">change</a>-+<a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>-+<a class="tag" href="/tag/thinking/page/1/">thinking</a>-+<a class="tag" href="/tag/world/page/1/">world</a>-+</div></div>@@ -54,12 +54,12 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="abilities,choices" / >-+ <meta class="keywords" itemprop="keywords" content="abilities,choices" / >+<a class="tag" href="/tag/abilities/page/1/">abilities</a>-+<a class="tag" href="/tag/choices/page/1/">choices</a>-+</div></div>@@ -70,18 +70,18 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="inspirational,life,live,miracle,miracles" / >-+ <meta class="keywords" itemprop="keywords" content="inspirational,life,live,miracle,miracles" / >+<a class="tag" href="/tag/inspirational/page/1/">inspirational</a>-+<a class="tag" href="/tag/life/page/1/">life</a>-+<a class="tag" href="/tag/live/page/1/">live</a>-+<a class="tag" href="/tag/miracle/page/1/">miracle</a>-+<a class="tag" href="/tag/miracles/page/1/">miracles</a>-+</div></div>@@ -92,16 +92,16 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="aliteracy,books,classic,humor" / >-+ <meta class="keywords" itemprop="keywords" content="aliteracy,books,classic,humor" / >+<a class="tag" href="/tag/aliteracy/page/1/">aliteracy</a>-+<a class="tag" href="/tag/books/page/1/">books</a>-+<a class="tag" href="/tag/classic/page/1/">classic</a>-+<a class="tag" href="/tag/humor/page/1/">humor</a>-+</div></div>@@ -112,12 +112,12 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="be-yourself,inspirational" / >-+ <meta class="keywords" itemprop="keywords" content="be-yourself,inspirational" / >+<a class="tag" href="/tag/be-yourself/page/1/">be-yourself</a>-+<a class="tag" href="/tag/inspirational/page/1/">inspirational</a>-+</div></div>@@ -128,14 +128,14 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="adulthood,success,value" / >-+ <meta class="keywords" itemprop="keywords" content="adulthood,success,value" / >+<a class="tag" href="/tag/adulthood/page/1/">adulthood</a>-+<a class="tag" href="/tag/success/page/1/">success</a>-+<a class="tag" href="/tag/value/page/1/">value</a>-+</div></div>@@ -146,12 +146,12 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="life,love" / >-+ <meta class="keywords" itemprop="keywords" content="life,love" / >+<a class="tag" href="/tag/life/page/1/">life</a>-+<a class="tag" href="/tag/love/page/1/">love</a>-+</div></div>@@ -162,16 +162,16 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="edison,failure,inspirational,paraphrased" / >-+ <meta class="keywords" itemprop="keywords" content="edison,failure,inspirational,paraphrased" / >+<a class="tag" href="/tag/edison/page/1/">edison</a>-+<a class="tag" href="/tag/failure/page/1/">failure</a>-+<a class="tag" href="/tag/inspirational/page/1/">inspirational</a>-+<a class="tag" href="/tag/paraphrased/page/1/">paraphrased</a>-+</div></div>@@ -182,10 +182,10 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="misattributed-eleanor-roosevelt" / >-+ <meta class="keywords" itemprop="keywords" content="misattributed-eleanor-roosevelt" / >+<a class="tag" href="/tag/misattributed-eleanor-roosevelt/page/1/">misattributed-eleanor-roosevelt</a>-+</div></div>@@ -196,73 +196,73 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="humor,obvious,simile" / >-+ <meta class="keywords" itemprop="keywords" content="humor,obvious,simile" / >+<a class="tag" href="/tag/humor/page/1/">humor</a>-+<a class="tag" href="/tag/obvious/page/1/">obvious</a>-+<a class="tag" href="/tag/simile/page/1/">simile</a>-+</div></div><nav><ul class="pager">--++<li class="next"><a href="/page/2/">Next <span aria-hidden="true">→</span></a></li>-+</ul></nav></div><div class="col-md-4 tags-box">-+<h2>Top Ten tags</h2>-+<span class="tag-item"><a class="tag" style="font-size: 28px" href="/tag/love/">love</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 26px" href="/tag/inspirational/">inspirational</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 26px" href="/tag/life/">life</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 24px" href="/tag/humor/">humor</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 22px" href="/tag/books/">books</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 14px" href="/tag/reading/">reading</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 10px" href="/tag/friendship/">friendship</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 8px" href="/tag/friends/">friends</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 8px" href="/tag/truth/">truth</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 6px" href="/tag/simile/">simile</a></span>--++</div></div>diff --git a/docs/_tests/quotes1.html b/docs/_tests/quotes1.htmlindex f4002ecd1..d1cfd9020 100644--- a/docs/_tests/quotes1.html+++ b/docs/_tests/quotes1.html@@ -16,13 +16,13 @@</div><div class="col-md-4"><p>-+<a href="/login">Login</a>-+</p></div></div>-+<div class="row"><div class="col-md-8">@@ -34,16 +34,16 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world" / >-+ <meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world" / >+<a class="tag" href="/tag/change/page/1/">change</a>-+<a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>-+<a class="tag" href="/tag/thinking/page/1/">thinking</a>-+<a class="tag" href="/tag/world/page/1/">world</a>-+</div></div>@@ -54,12 +54,12 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="abilities,choices" / >-+ <meta class="keywords" itemprop="keywords" content="abilities,choices" / >+<a class="tag" href="/tag/abilities/page/1/">abilities</a>-+<a class="tag" href="/tag/choices/page/1/">choices</a>-+</div></div>@@ -70,18 +70,18 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="inspirational,life,live,miracle,miracles" / >-+ <meta class="keywords" itemprop="keywords" content="inspirational,life,live,miracle,miracles" / >+<a class="tag" href="/tag/inspirational/page/1/">inspirational</a>-+<a class="tag" href="/tag/life/page/1/">life</a>-+<a class="tag" href="/tag/live/page/1/">live</a>-+<a class="tag" href="/tag/miracle/page/1/">miracle</a>-+<a class="tag" href="/tag/miracles/page/1/">miracles</a>-+</div></div>@@ -92,16 +92,16 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="aliteracy,books,classic,humor" / >-+ <meta class="keywords" itemprop="keywords" content="aliteracy,books,classic,humor" / >+<a class="tag" href="/tag/aliteracy/page/1/">aliteracy</a>-+<a class="tag" href="/tag/books/page/1/">books</a>-+<a class="tag" href="/tag/classic/page/1/">classic</a>-+<a class="tag" href="/tag/humor/page/1/">humor</a>-+</div></div>@@ -112,12 +112,12 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="be-yourself,inspirational" / >-+ <meta class="keywords" itemprop="keywords" content="be-yourself,inspirational" / >+<a class="tag" href="/tag/be-yourself/page/1/">be-yourself</a>-+<a class="tag" href="/tag/inspirational/page/1/">inspirational</a>-+</div></div>@@ -128,14 +128,14 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="adulthood,success,value" / >-+ <meta class="keywords" itemprop="keywords" content="adulthood,success,value" / >+<a class="tag" href="/tag/adulthood/page/1/">adulthood</a>-+<a class="tag" href="/tag/success/page/1/">success</a>-+<a class="tag" href="/tag/value/page/1/">value</a>-+</div></div>@@ -146,12 +146,12 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="life,love" / >-+ <meta class="keywords" itemprop="keywords" content="life,love" / >+<a class="tag" href="/tag/life/page/1/">life</a>-+<a class="tag" href="/tag/love/page/1/">love</a>-+</div></div>@@ -162,16 +162,16 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="edison,failure,inspirational,paraphrased" / >-+ <meta class="keywords" itemprop="keywords" content="edison,failure,inspirational,paraphrased" / >+<a class="tag" href="/tag/edison/page/1/">edison</a>-+<a class="tag" href="/tag/failure/page/1/">failure</a>-+<a class="tag" href="/tag/inspirational/page/1/">inspirational</a>-+<a class="tag" href="/tag/paraphrased/page/1/">paraphrased</a>-+</div></div>@@ -182,10 +182,10 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="misattributed-eleanor-roosevelt" / >-+ <meta class="keywords" itemprop="keywords" content="misattributed-eleanor-roosevelt" / >+<a class="tag" href="/tag/misattributed-eleanor-roosevelt/page/1/">misattributed-eleanor-roosevelt</a>-+</div></div>@@ -196,73 +196,73 @@</span><div class="tags">Tags:- <meta class="keywords" itemprop="keywords" content="humor,obvious,simile" / >-+ <meta class="keywords" itemprop="keywords" content="humor,obvious,simile" / >+<a class="tag" href="/tag/humor/page/1/">humor</a>-+<a class="tag" href="/tag/obvious/page/1/">obvious</a>-+<a class="tag" href="/tag/simile/page/1/">simile</a>-+</div></div><nav><ul class="pager">--++<li class="next"><a href="/page/2/">Next <span aria-hidden="true">→</span></a></li>-+</ul></nav></div><div class="col-md-4 tags-box">-+<h2>Top Ten tags</h2>-+<span class="tag-item"><a class="tag" style="font-size: 28px" href="/tag/love/">love</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 26px" href="/tag/inspirational/">inspirational</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 26px" href="/tag/life/">life</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 24px" href="/tag/humor/">humor</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 22px" href="/tag/books/">books</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 14px" href="/tag/reading/">reading</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 10px" href="/tag/friendship/">friendship</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 8px" href="/tag/friends/">friends</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 8px" href="/tag/truth/">truth</a></span>-+<span class="tag-item"><a class="tag" style="font-size: 6px" href="/tag/simile/">simile</a></span>--++</div></div>diff --git a/docs/faq.rst b/docs/faq.rstindex f81ec3601..cef3e69f3 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -410,7 +410,7 @@ How can I make a blank request?-------------------------------.. code-block:: python-+from scrapy import Requestdiff --git a/docs/intro/install.rst b/docs/intro/install.rstindex 82a0e18c5..488a66f36 100644--- a/docs/intro/install.rst+++ b/docs/intro/install.rst@@ -111,7 +111,7 @@ Once you've installed `Anaconda`_ or `Miniconda`_, install Scrapy with::To install Scrapy on Windows using ``pip``:.. warning::- This installation method requires “Microsoft Visual C++” for installing some+ This installation method requires “Microsoft Visual C++” for installing someScrapy dependencies, which demands significantly more disk space than Anaconda.#. Download and execute `Microsoft C++ Build Tools`_ to install the Visual Studio Installer.@@ -123,7 +123,7 @@ To install Scrapy on Windows using ``pip``:#. Check the installation details and make sure following packages are selected as optional components:* **MSVC** (e.g MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) )-+* **Windows SDK** (e.g Windows 10 SDK (10.0.18362.0))#. Install the Visual Studio Build Tools.diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex 6e6caebf1..5041b49ea 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -292,7 +292,7 @@ As an alternative, you could've written:>>> response.css("title::text")[0].get()'Quotes to Scrape'-Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will+Accessing an index on a :class:`~scrapy.selector.SelectorList` instance willraise an :exc:`IndexError` exception if there are no results:.. code-block:: pycon@@ -302,8 +302,8 @@ raise an :exc:`IndexError` exception if there are no results:...IndexError: list index out of range-You might want to use ``.get()`` directly on the-:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``+You might want to use ``.get()`` directly on the+:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``if there are no results:.. code-block:: pycondiff --git a/docs/news.rst b/docs/news.rstindex 8230c3aef..9a68f8852 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -934,10 +934,10 @@ Modified requirementsBackward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-- The value of the :setting:`FEED_STORE_EMPTY` setting is now ``True``- instead of ``False``. In earlier Scrapy versions empty files were created- even when this setting was ``False`` (which was a bug that is now fixed),- so the new default should keep the old behavior. (:issue:`872`,+- The value of the :setting:`FEED_STORE_EMPTY` setting is now ``True``+ instead of ``False``. In earlier Scrapy versions empty files were created+ even when this setting was ``False`` (which was a bug that is now fixed),+ so the new default should keep the old behavior. (:issue:`872`,:issue:`5847`)Deprecation removalsdiff --git a/docs/topics/api.rst b/docs/topics/api.rstindex f7cffb61b..edc625be8 100644--- a/docs/topics/api.rst+++ b/docs/topics/api.rst@@ -88,7 +88,7 @@ how you :ref:`configure the downloader middlewaresThe execution engine, which coordinates the core crawling logicbetween the scheduler, downloader and spiders.- Some extension may want to access the Scrapy engine, to inspect or+ Some extension may want to access the Scrapy engine, to inspect ormodify the downloader and scheduler behaviour, although this is anadvanced use and this API is not yet stable.diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rstindex 0370dc538..4e53b6e3d 100644--- a/docs/topics/architecture.rst+++ b/docs/topics/architecture.rst@@ -87,8 +87,8 @@ of the system, and triggering events when certain actions occur. See theScheduler----------The :ref:`scheduler <topics-scheduler>` receives requests from the engine and-enqueues them for feeding them later (also to the engine) when the engine+The :ref:`scheduler <topics-scheduler>` receives requests from the engine and+enqueues them for feeding them later (also to the engine) when the enginerequests them... _component-downloader:diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rstindex 7a85c099b..5c078568b 100644--- a/docs/topics/exporters.rst+++ b/docs/topics/exporters.rst@@ -224,7 +224,7 @@ BaseItemExporter.. [1] Not all exporters respect the specified field order... [2] When using :ref:`item objects <item-types>` that do not exposeall their possible fields, exporters that do not support exporting- a different subset of fields per item will only export the fields+ a different subset of fields per item will only export the fieldsfound in the first item exported... attribute:: export_empty_fieldsdiff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex c47a3226a..23bbcfcb5 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -256,14 +256,14 @@ Spider state extensionManages spider state data by loading it before a crawl and saving it after.Give a value to the :setting:`JOBDIR` setting to enable this extension.-When enabled, this extension manages the :attr:`~scrapy.Spider.state`+When enabled, this extension manages the :attr:`~scrapy.Spider.state`attribute of your :class:`~scrapy.Spider` instance:--- When your spider closes (:signal:`spider_closed`), the contents of its- :attr:`~scrapy.Spider.state` attribute are serialized into a file named++- When your spider closes (:signal:`spider_closed`), the contents of its+ :attr:`~scrapy.Spider.state` attribute are serialized into a file named``spider.state`` in the :setting:`JOBDIR` folder.-- When your spider opens (:signal:`spider_opened`), if a previously-generated- ``spider.state`` file exists in the :setting:`JOBDIR` folder, it is loaded+- When your spider opens (:signal:`spider_opened`), if a previously-generated+ ``spider.state`` file exists in the :setting:`JOBDIR` folder, it is loadedinto the :attr:`~scrapy.Spider.state` attribute.@@ -291,8 +291,8 @@ settings:.. note::- When a certain closing condition is met, requests which are- currently in the downloader queue (up to :setting:`CONCURRENT_REQUESTS`+ When a certain closing condition is met, requests which are+ currently in the downloader queue (up to :setting:`CONCURRENT_REQUESTS`requests) are still processed... setting:: CLOSESPIDER_TIMEOUTdiff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex 07a3f3678..7f401f0c7 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -180,7 +180,7 @@ FTP supports two different connection modes: `active or passivemode by default. To use the active connection mode instead, set the:setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``.-The default value for the ``overwrite`` key in the :setting:`FEEDS` for this+The default value for the ``overwrite`` key in the :setting:`FEEDS` for thisstorage backend is: ``True``... caution:: The value ``True`` in ``overwrite`` will cause you to lose the@@ -222,7 +222,7 @@ feeds using these settings:- :setting:`AWS_ENDPOINT_URL`- :setting:`AWS_REGION_NAME`-The default value for the ``overwrite`` key in the :setting:`FEEDS` for this+The default value for the ``overwrite`` key in the :setting:`FEEDS` for thisstorage backend is: ``True``... caution:: The value ``True`` in ``overwrite`` will cause you to lose the@@ -255,7 +255,7 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following- :setting:`FEED_STORAGE_GCS_ACL`- :setting:`GCS_PROJECT_ID`-The default value for the ``overwrite`` key in the :setting:`FEEDS` for this+The default value for the ``overwrite`` key in the :setting:`FEEDS` for thisstorage backend is: ``True``... caution:: The value ``True`` in ``overwrite`` will cause you to lose the@@ -587,8 +587,8 @@ FEED_STORE_EMPTYDefault: ``True``Whether to export empty feeds (i.e. feeds with no items).-If ``False``, and there are no items to export, no new files are created and-existing files are not modified, even if the :ref:`overwrite feed option+If ``False``, and there are no items to export, no new files are created and+existing files are not modified, even if the :ref:`overwrite feed option<feed-options>` is enabled... setting:: FEED_STORAGESdiff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex fe1c4d162..a398d6c83 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -266,9 +266,9 @@ e.g. in the spider's ``__init__`` method:If you run this spider again then INFO messages from``scrapy.spidermiddlewares.httperror`` logger will be gone.-You can also filter log records by :class:`~logging.LogRecord` data. For+You can also filter log records by :class:`~logging.LogRecord` data. Forexample, you can filter log records by message content using a substring or-a regular expression. Create a :class:`logging.Filter` subclass+a regular expression. Create a :class:`logging.Filter` subclassand equip it with a regular expression pattern tofilter out unwanted messages:@@ -284,8 +284,8 @@ filter out unwanted messages:if match:return False-A project-level filter may be attached to the root-handler created by Scrapy, this is a wieldy way to+A project-level filter may be attached to the root+handler created by Scrapy, this is a wieldy way tofilter all loggers in different parts of the project(middlewares, spider, etc.):@@ -301,7 +301,7 @@ filter all loggers in different parts of the projectfor handler in logging.root.handlers:handler.addFilter(ContentFilter())-Alternatively, you may choose a specific logger+Alternatively, you may choose a specific loggerand hide it without affecting other loggers:.. code-block:: pythondiff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex f086a943e..cc1fe8703 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -414,7 +414,7 @@ class name. E.g. given pipeline class called MyPipeline you can set setting key:and pipeline class MyPipeline will have expiration time set to 180.-The last modified time from the file is used to determine the age of the file in days,+The last modified time from the file is used to determine the age of the file in days,which is then compared to the set expiration time to determine if the file is expired... _topics-images-thumbnails:@@ -519,7 +519,7 @@ See here the methods that you can override in your custom Files Pipeline:In addition to ``response``, this method receives the original:class:`request <scrapy.Request>`,- :class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and+ :class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and:class:`item <scrapy.Item>`You can override this method to customize the download path of each file.@@ -541,9 +541,9 @@ See here the methods that you can override in your custom Files Pipeline:def file_path(self, request, response=None, info=None, *, item=None):return "files/" + PurePosixPath(urlparse_cached(request).path).name- Similarly, you can use the ``item`` to determine the file path based on some item+ Similarly, you can use the ``item`` to determine the file path based on some itemproperty.-+By default the :meth:`file_path` method returns``full/<request URL hash>.<extension>``.@@ -677,7 +677,7 @@ See here the methods that you can override in your custom Images Pipeline:In addition to ``response``, this method receives the original:class:`request <scrapy.Request>`,- :class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and+ :class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and:class:`item <scrapy.Item>`You can override this method to customize the download path of each file.@@ -699,9 +699,9 @@ See here the methods that you can override in your custom Images Pipeline:def file_path(self, request, response=None, info=None, *, item=None):return "files/" + PurePosixPath(urlparse_cached(request).path).name- Similarly, you can use the ``item`` to determine the file path based on some item+ Similarly, you can use the ``item`` to determine the file path based on some itemproperty.-+By default the :meth:`file_path` method returns``full/<request URL hash>.<extension>``.diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex 5f6798601..db91cd073 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -309,7 +309,7 @@ Here are some tips to keep in mind when dealing with these kinds of sites:services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, asuper proxy that you can attach your own proxies to.* use a ban avoidance service, such as `Zyte API`_, which provides a `Scrapy- plugin <https://github.com/scrapy-plugins/scrapy-zyte-api>`__ and additional+ plugin <https://github.com/scrapy-plugins/scrapy-zyte-api>`__ and additionalfeatures, like `AI web scraping <https://www.zyte.com/ai-web-scraping/>`__If you are still unable to prevent your bot getting banned, consider contactingdiff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex b187f3aaf..55e8518a5 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -1309,7 +1309,7 @@ JsonResponse objects.. class:: JsonResponse(url[, ...])- The :class:`JsonResponse` class is a subclass of :class:`TextResponse`- that is used when the response has a `JSON MIME type- <https://mimesniff.spec.whatwg.org/#json-mime-type>`_ in its `Content-Type`+ The :class:`JsonResponse` class is a subclass of :class:`TextResponse`+ that is used when the response has a `JSON MIME type+ <https://mimesniff.spec.whatwg.org/#json-mime-type>`_ in its `Content-Type`header.diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rstindex b95e6eab3..dbef07b73 100644--- a/docs/topics/selectors.rst+++ b/docs/topics/selectors.rst@@ -559,7 +559,7 @@ For example, suppose you want to extract all ``<p>`` elements inside ``<div>``elements. First, you would get all ``<div>`` elements:.. code-block:: pycon-+>>> divs = response.xpath("//div")At first, you may be tempted to use the following approach, which is wrong, as@@ -610,7 +610,7 @@ As it turns out, Scrapy selectors allow you to chain selectors, so most of the tyou can just select by class using CSS and then switch to XPath when needed:.. code-block:: pycon-+>>> from scrapy import Selector>>> sel = Selector(... text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>'@@ -1032,7 +1032,7 @@ whereas the CSS lookup is translated into XPath and thus runs more efficiently,so performance-wise its uses are limited to situations that are not easilydescribed with CSS selectors.-Parsel also simplifies adding your own XPath extensions with+Parsel also simplifies adding your own XPath extensions with:func:`~parsel.xpathfuncs.set_xpathfunc`... _topics-selectors-ref:diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex e1b1c5ad6..0a67240d6 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -379,8 +379,8 @@ The above example can also be written as follows:def start_requests(self):yield scrapy.Request(f"http://www.example.com/categories/{self.category}")-If you are :ref:`running Scrapy from a script <run-from-script>`, you can-specify spider arguments when calling+If you are :ref:`running Scrapy from a script <run-from-script>`, you can+specify spider arguments when calling:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:diff --git a/docs/topics/stats.rst b/docs/topics/stats.rstindex be8ecb7a5..9572a3785 100644--- a/docs/topics/stats.rst+++ b/docs/topics/stats.rst@@ -86,7 +86,7 @@ Available Stats CollectorsBesides the basic :class:`StatsCollector` there are other Stats Collectorsavailable in Scrapy which extend the basic Stats Collector. You can selectwhich Stats Collector to use through the :setting:`STATS_CLASS` setting. The-default Stats Collector used is the :class:`MemoryStatsCollector`.+default Stats Collector used is the :class:`MemoryStatsCollector`... currentmodule:: scrapy.statscollectorsdiff --git a/sep/sep-004.rst b/sep/sep-004.rstindex b1cef2600..7a4ebe886 100644--- a/sep/sep-004.rst+++ b/sep/sep-004.rst@@ -11,7 +11,7 @@ SEP-004: Library API====================.. note:: the library API has been implemented, but slightly different fromproposed in this SEP. You can run a Scrapy crawler inside a Twisted- reactor, but not outside it.+ reactor, but not outside it.Introduction============diff --git a/sep/sep-007.rst b/sep/sep-007.rstindex 0ca2036ce..73ce0d338 100644--- a/sep/sep-007.rst+++ b/sep/sep-007.rst@@ -96,7 +96,7 @@ specified, else utf-8 is used) and returns a new unicode object. E.g:``clean_spaces``-----------------+Converts multispaces into single spaces for the given string. E.g:::diff --git a/sep/sep-008.rst b/sep/sep-008.rstindex be5987e39..1c38b1c40 100644--- a/sep/sep-008.rst+++ b/sep/sep-008.rst@@ -73,8 +73,8 @@ Alternative Public API Proposal- ``ItemLoader.get_stored_values()`` or ``ItemLoader.get_values()`` *(returns the ``ItemLoader values)*- ``ItemLoader.get_output_value()``-- ``ItemLoader.get_input_processor()`` or ``ItemLoader.get_in_processor()`` *(short version)*-- ``ItemLoader.get_output_processor()`` or ``ItemLoader.get_out_processor()`` *(short version)*+- ``ItemLoader.get_input_processor()`` or ``ItemLoader.get_in_processor()`` *(short version)*+- ``ItemLoader.get_output_processor()`` or ``ItemLoader.get_out_processor()`` *(short version)*- ``ItemLoader.context``diff --git a/sep/sep-014.rst b/sep/sep-014.rstindex e03a2b0f6..0a2e6b51e 100644--- a/sep/sep-014.rst+++ b/sep/sep-014.rst@@ -21,7 +21,7 @@ Current flaws and inconsistencies2. Link extractors are inflexible and hard to maintain, linkprocessing/filtering is tightly coupled. (e.g. canonicalize)3. Isn't possible to crawl an url directly from command line because the Spider- does not know which callback use.+ does not know which callback use.These flaws will be corrected by the changes proposed in this SEP.@@ -55,7 +55,7 @@ Request ExtractorsRequest Extractors takes response object and determines which requests follow.This is an enhancement to ``LinkExtractors`` which returns urls (links),-Request Extractors return Request objects.+Request Extractors return Request objects.Request Processors------------------diff --git a/sep/sep-018.rst b/sep/sep-018.rstindex 13ab501ed..e6d601fe1 100644--- a/sep/sep-018.rst+++ b/sep/sep-018.rst@@ -200,7 +200,7 @@ the same spider:# extract item from responsereturn item-The Spider Middleware that implements spider code+The Spider Middleware that implements spider code=================================================There's gonna be one middleware that will take care of calling the proper@@ -625,7 +625,7 @@ Resolved:not the original one (think of redirections), but it does carry the ``meta``of the original one. The original one may not be available anymore (inmemory) if we're using a persistent scheduler., but in that case it would be- the deserialized request from the persistent scheduler queue.+ the deserialized request from the persistent scheduler queue.- No - this would make implementation more complex and we're not sure it'sreally needed