Scrapy, a fast high-level web crawling & scraping framework for Python.
[docs] refactor python docs links using intersphinx
Aditya80c69d68adde
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex ab6fd4829..5f35dc936 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -287,8 +287,8 @@ to be scraped, you can at least get **some** data.Besides the :meth:`~scrapy.selector.SelectorList.getall` and:meth:`~scrapy.selector.SelectorList.get` methods, you can also use-the :meth:`~scrapy.selector.SelectorList.re` method to extract using `regular-expressions`_:+the :meth:`~scrapy.selector.SelectorList.re` method to extract using+:doc:`regular expressions <library/re>`:>>> response.css('title::text').re(r'Quotes.*')['Quotes to Scrape']@@ -305,7 +305,6 @@ with a selector (see :ref:`topics-developer-tools`).`Selector Gadget`_ is also a nice tool to quickly find CSS selector forvisually selected elements, which works in many browsers.-.. _regular expressions: https://docs.python.org/3/library/re.html.. _Selector Gadget: https://selectorgadget.com/diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 487cf4c6c..5f61d6796 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -76,8 +76,8 @@ becomes::Coroutines may be used to call asynchronous code. This includes othercoroutines, functions that return Deferreds and functions that return-`awaitable objects`_ such as :class:`~asyncio.Future`. This means you can use-many useful Python libraries providing such code::+:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.+This means you can use many useful Python libraries providing such code::class MySpider(Spider):# ...@@ -107,4 +107,3 @@ Common use cases for asynchronous code include::ref:`the screenshot pipeline example<ScreenshotPipeline>`)... _aio-libs: https://github.com/aio-libs-.. _awaitable objects: https://docs.python.org/3/glossary.html#term-awaitablediff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex d7ec53bfa..d309bbc49 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -980,7 +980,7 @@ RobotsTxtMiddlewareScrapy ships with support for the following robots.txt_ parsers:* :ref:`Protego <protego-parser>` (default)- * :class:`~urllib.robotparser.RobotFileParser`+ * :ref:`RobotFileParser <python-robotfileparser>`* :ref:`Reppy <reppy-parser>`* :ref:`Robotexclusionrulesparser <rerp-parser>`@@ -1028,6 +1028,8 @@ Based on `Protego <https://github.com/scrapy/protego>`_:Scrapy uses this parser by default.+.. _python-robotfileparser:+RobotFileParser~~~~~~~~~~~~~~~diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rstindex 22bcac268..a3f0d6ebb 100644--- a/docs/topics/dynamic-content.rst+++ b/docs/topics/dynamic-content.rst@@ -130,8 +130,9 @@ data from it depends on the type of response:- If the response is JavaScript, or HTML with a ``<script/>`` elementcontaining the desired data, see :ref:`topics-parsing-javascript`.-- If the response is CSS, use :mod:`re` to extract the desired- data from :attr:`response.text <scrapy.http.TextResponse.text>`.+- If the response is CSS, use a :doc:`regular expression <library/re>` to+ extract the desired data from+ :attr:`response.text <scrapy.http.TextResponse.text>`... _topics-parsing-images:@@ -168,8 +169,9 @@ JavaScript code:Once you have a string with the JavaScript code, you can extract the desireddata from it:-- You might be able to use :mod:`re` to extract the desired- data in JSON format, which you can then parse with :func:`json.loads`.+- You might be able to use a :doc:`regular expression <library/re>` to+ extract the desired data in JSON format, which you can then parse with+ :func:`json.loads`.For example, if the JavaScript code contains a separate line like``var data = {"field": "value"};`` you can extract that data as follows:diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rstindex f73c6728d..de8b51195 100644--- a/docs/topics/exporters.rst+++ b/docs/topics/exporters.rst@@ -311,7 +311,7 @@ CsvItemExporterThe additional keyword arguments of this ``__init__`` method are passed to the:class:`BaseItemExporter` ``__init__`` method, and the leftover arguments to the- :func:`csv.writer` ``__init__`` method, so you can use any ``csv.writer`` ``__init__`` method+ :func:`csv.writer` function, so you can use any :func:`csv.writer` functionargument to customize this exporter.A typical output of this exporter would be::@@ -333,7 +333,7 @@ PickleItemExporter:param protocol: The pickle protocol to use.:type protocol: int- For more information, refer :mod:`pickle`.+ For more information, see :mod:`pickle`.The additional keyword arguments of this ``__init__`` method are passed to the:class:`BaseItemExporter` ``__init__`` method.diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex 1b8413abf..0fc83e645 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -364,7 +364,7 @@ Debugger extension.. class:: Debugger-Invokes a :mod:`Python debugger <pdb>`: inside a running Scrapy process when a `SIGUSR2`_+Invokes a :doc:`Python debugger <library/pdb>` inside a running Scrapy process when a `SIGUSR2`_signal is received. After the debugger is exited, the Scrapy process continuesrunning normally.diff --git a/docs/topics/items.rst b/docs/topics/items.rstindex 2e5c88054..78612f524 100644--- a/docs/topics/items.rst+++ b/docs/topics/items.rst@@ -15,8 +15,8 @@ especially in a larger project with many spiders.To define common output data format Scrapy provides the :class:`Item` class.:class:`Item` objects are simple containers used to collect the scraped data.-They provide a :class:`dict` like API with a convenient syntax for declaring-their available fields.+They provide an API similar to :class:`dict` API with a convenient syntax+for declaring their available fields.Various Scrapy components use extra information provided by Items:exporters look at declared fields to figure out columns to export,@@ -143,7 +143,7 @@ KeyError: 'Product does not support field: lala'Accessing all populated values-------------------------------To access all populated values, just use the typical :class:`dict`:+To access all populated values, just use the typical :class:`dict` API:>>> product.keys()['price', 'name']@@ -160,11 +160,9 @@ Copying itemsTo copy an item, you must first decide whether you want a shallow copy or adeep copy.-If your item contains mutable_ values like lists or dictionaries, a shallow-copy will keep references to the same mutable values across all different-copies.--.. _mutable: https://docs.python.org/3/glossary.html#term-mutable+If your item contains :term:`mutable` values like lists or dictionaries,+a shallow copy will keep references to the same mutable values across all+different copies.For example, if you have an item with a list of tags, and you create a shallowcopy of that item, both the original item and the copy have the same list of@@ -231,8 +229,8 @@ Item objectsReturn a new Item optionally initialized from the given argument.- Items replicate the standard :class:`dict`, including its ``__init__`` method, and- also provide the following additional API members:+ Items replicate the standard :class:`dict` API, including its ``__init__``+ method, and also provide the following additional API members:.. automethod:: copydiff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex df631b3dc..675e65ef1 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -82,10 +82,10 @@ path::.. seealso::- Module logging, `HowTo <https://docs.python.org/3/howto/logging.html>`_+ Module logging, :doc:`HowTo <howto/logging>`Basic Logging Tutorial- Module logging, :class:`~logging.Logger`+ Module logging, :ref:`Loggers <logger>`Further documentation on loggers.. _topics-logging-from-spiders:@@ -164,10 +164,8 @@ possible levels listed in :ref:`topics-logging-levels`.:setting:`LOG_FORMAT` and :setting:`LOG_DATEFORMAT` specify formatting stringsused as layouts for all messages. Those strings can contain any placeholders-listed in `logging's logrecord attributes docs-<https://docs.python.org/3/library/logging.html#logrecord-attributes>`_ and-`datetime's strftime and strptime directives-<https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`_+listed in :ref:`logging's logrecord attributes docs <logrecord-attributes>` and+:ref:`datetime's strftime and strptime directives <strftime-strptime-behavior>`respectively.If :setting:`LOG_SHORT_NAMES` is set, then the logs will not display the Scrapydiff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 7260141e9..69a51a17c 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -174,9 +174,9 @@ Request objectsSee :ref:`topics-request-meta` for a list of special meta keysrecognized by Scrapy.- This dict is `shallow copied`_ when the request is cloned using the- ``copy()`` or ``replace()`` methods, and can also be accessed, in your- spider, from the ``response.meta`` attribute.+ This dict is :mod:`shallow copied <copy>` when the request is+ cloned using the ``copy()`` or ``replace()`` methods, and can also be+ accessed, in your spider, from the ``response.meta`` attribute... attribute:: Request.cb_kwargs@@ -185,11 +185,9 @@ Request objectsfor new Requests, which means by default callbacks only get a :class:`Response`object as argument.- This dict is `shallow copied`_ when the request is cloned using the- ``copy()`` or ``replace()`` methods, and can also be accessed, in your- spider, from the ``response.cb_kwargs`` attribute.-- .. _shallow copied: https://docs.python.org/3/library/copy.html+ This dict is :mod:`shallow copied <copy>` when the request is+ cloned using the ``copy()`` or ``replace()`` methods, and can also be+ accessed, in your spider, from the ``response.cb_kwargs`` attribute... method:: Request.copy()diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex d78a6253e..0049dbfca 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -26,9 +26,7 @@ do this by using an environment variable, ``SCRAPY_SETTINGS_MODULE``.The value of ``SCRAPY_SETTINGS_MODULE`` should be in Python path syntax, e.g.``myproject.settings``. Note that the settings module should be on the-Python `import search path`_.--.. _import search path: https://docs.python.org/3/tutorial/modules.html#the-module-search-path+Python :ref:`import search path <tut-searchpath>`... _populating-settings:@@ -899,10 +897,9 @@ LOG_FORMATDefault: ``'%(asctime)s [%(name)s] %(levelname)s: %(message)s'``-String for formatting log messages. Refer to the `Python logging documentation`_ for the whole list of available-placeholders.--.. _Python logging documentation: https://docs.python.org/3/library/logging.html#logrecord-attributes+String for formatting log messages. Refer to the+:ref:`Python logging documentation <logrecord-attributes>` for the qwhole+list of available placeholders... setting:: LOG_DATEFORMAT@@ -912,10 +909,9 @@ LOG_DATEFORMATDefault: ``'%Y-%m-%d %H:%M:%S'``String for formatting date/time, expansion of the ``%(asctime)s`` placeholder-in :setting:`LOG_FORMAT`. Refer to the `Python datetime documentation`_ for the whole list of available-directives.--.. _Python datetime documentation: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior+in :setting:`LOG_FORMAT`. Refer to the+:ref:`Python datetime documentation <strftime-strptime-behavior>` for the+whole list of available directives... setting:: LOG_FORMATTER