Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

[docs] refactor python docs links using intersphinx

Aditya80c69d68adde
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index ab6fd4829..5f35dc936 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -287,8 +287,8 @@ to be scraped, you can at least get **some** data.
Besides the :meth:`~scrapy.selector.SelectorList.getall` and
:meth:`~scrapy.selector.SelectorList.get` methods, you can also use
-the :meth:`~scrapy.selector.SelectorList.re` method to extract using `regular
-expressions`_:
+the :meth:`~scrapy.selector.SelectorList.re` method to extract using
+:doc:`regular expressions <library/re>`:
>>> response.css('title::text').re(r'Quotes.*')
['Quotes to Scrape']
@@ -305,7 +305,6 @@ with a selector (see :ref:`topics-developer-tools`).
`Selector Gadget`_ is also a nice tool to quickly find CSS selector for
visually selected elements, which works in many browsers.
-.. _regular expressions: https://docs.python.org/3/library/re.html
.. _Selector Gadget: https://selectorgadget.com/
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 487cf4c6c..5f61d6796 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -76,8 +76,8 @@ becomes::
Coroutines may be used to call asynchronous code. This includes other
coroutines, functions that return Deferreds and functions that return
-`awaitable objects`_ such as :class:`~asyncio.Future`. This means you can use
-many useful Python libraries providing such code::
+:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.
+This means you can use many useful Python libraries providing such code::
class MySpider(Spider):
# ...
@@ -107,4 +107,3 @@ Common use cases for asynchronous code include:
:ref:`the screenshot pipeline example<ScreenshotPipeline>`).
.. _aio-libs: https://github.com/aio-libs
-.. _awaitable objects: https://docs.python.org/3/glossary.html#term-awaitable
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index d7ec53bfa..d309bbc49 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -980,7 +980,7 @@ RobotsTxtMiddleware
Scrapy ships with support for the following robots.txt_ parsers:
* :ref:`Protego <protego-parser>` (default)
- * :class:`~urllib.robotparser.RobotFileParser`
+ * :ref:`RobotFileParser <python-robotfileparser>`
* :ref:`Reppy <reppy-parser>`
* :ref:`Robotexclusionrulesparser <rerp-parser>`
@@ -1028,6 +1028,8 @@ Based on `Protego <https://github.com/scrapy/protego>`_:
Scrapy uses this parser by default.
+.. _python-robotfileparser:
+
RobotFileParser
~~~~~~~~~~~~~~~
diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst
index 22bcac268..a3f0d6ebb 100644
--- a/docs/topics/dynamic-content.rst
+++ b/docs/topics/dynamic-content.rst
@@ -130,8 +130,9 @@ data from it depends on the type of response:
- If the response is JavaScript, or HTML with a ``<script/>`` element
containing the desired data, see :ref:`topics-parsing-javascript`.
-- If the response is CSS, use :mod:`re` to extract the desired
- data from :attr:`response.text <scrapy.http.TextResponse.text>`.
+- If the response is CSS, use a :doc:`regular expression <library/re>` to
+ extract the desired data from
+ :attr:`response.text <scrapy.http.TextResponse.text>`.
.. _topics-parsing-images:
@@ -168,8 +169,9 @@ JavaScript code:
Once you have a string with the JavaScript code, you can extract the desired
data from it:
-- You might be able to use :mod:`re` to extract the desired
- data in JSON format, which you can then parse with :func:`json.loads`.
+- You might be able to use a :doc:`regular expression <library/re>` to
+ extract the desired data in JSON format, which you can then parse with
+ :func:`json.loads`.
For example, if the JavaScript code contains a separate line like
``var data = {"field": "value"};`` you can extract that data as follows:
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index f73c6728d..de8b51195 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -311,7 +311,7 @@ CsvItemExporter
The additional keyword arguments of this ``__init__`` method are passed to the
:class:`BaseItemExporter` ``__init__`` method, and the leftover arguments to the
- :func:`csv.writer` ``__init__`` method, so you can use any ``csv.writer`` ``__init__`` method
+ :func:`csv.writer` function, so you can use any :func:`csv.writer` function
argument to customize this exporter.
A typical output of this exporter would be::
@@ -333,7 +333,7 @@ PickleItemExporter
:param protocol: The pickle protocol to use.
:type protocol: int
- For more information, refer :mod:`pickle`.
+ For more information, see :mod:`pickle`.
The additional keyword arguments of this ``__init__`` method are passed to the
:class:`BaseItemExporter` ``__init__`` method.
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 1b8413abf..0fc83e645 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -364,7 +364,7 @@ Debugger extension
.. class:: Debugger
-Invokes a :mod:`Python debugger <pdb>`: inside a running Scrapy process when a `SIGUSR2`_
+Invokes a :doc:`Python debugger <library/pdb>` inside a running Scrapy process when a `SIGUSR2`_
signal is received. After the debugger is exited, the Scrapy process continues
running normally.
diff --git a/docs/topics/items.rst b/docs/topics/items.rst
index 2e5c88054..78612f524 100644
--- a/docs/topics/items.rst
+++ b/docs/topics/items.rst
@@ -15,8 +15,8 @@ especially in a larger project with many spiders.
To define common output data format Scrapy provides the :class:`Item` class.
:class:`Item` objects are simple containers used to collect the scraped data.
-They provide a :class:`dict` like API with a convenient syntax for declaring
-their available fields.
+They provide an API similar to :class:`dict` API with a convenient syntax
+for declaring their available fields.
Various Scrapy components use extra information provided by Items:
exporters look at declared fields to figure out columns to export,
@@ -143,7 +143,7 @@ KeyError: 'Product does not support field: lala'
Accessing all populated values
------------------------------
-To access all populated values, just use the typical :class:`dict`:
+To access all populated values, just use the typical :class:`dict` API:
>>> product.keys()
['price', 'name']
@@ -160,11 +160,9 @@ Copying items
To copy an item, you must first decide whether you want a shallow copy or a
deep copy.
-If your item contains mutable_ values like lists or dictionaries, a shallow
-copy will keep references to the same mutable values across all different
-copies.
-
-.. _mutable: https://docs.python.org/3/glossary.html#term-mutable
+If your item contains :term:`mutable` values like lists or dictionaries,
+a shallow copy will keep references to the same mutable values across all
+different copies.
For example, if you have an item with a list of tags, and you create a shallow
copy of that item, both the original item and the copy have the same list of
@@ -231,8 +229,8 @@ Item objects
Return a new Item optionally initialized from the given argument.
- Items replicate the standard :class:`dict`, including its ``__init__`` method, and
- also provide the following additional API members:
+ Items replicate the standard :class:`dict` API, including its ``__init__``
+ method, and also provide the following additional API members:
.. automethod:: copy
diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst
index df631b3dc..675e65ef1 100644
--- a/docs/topics/logging.rst
+++ b/docs/topics/logging.rst
@@ -82,10 +82,10 @@ path::
.. seealso::
- Module logging, `HowTo <https://docs.python.org/3/howto/logging.html>`_
+ Module logging, :doc:`HowTo <howto/logging>`
Basic Logging Tutorial
- Module logging, :class:`~logging.Logger`
+ Module logging, :ref:`Loggers <logger>`
Further documentation on loggers
.. _topics-logging-from-spiders:
@@ -164,10 +164,8 @@ possible levels listed in :ref:`topics-logging-levels`.
:setting:`LOG_FORMAT` and :setting:`LOG_DATEFORMAT` specify formatting strings
used as layouts for all messages. Those strings can contain any placeholders
-listed in `logging's logrecord attributes docs
-<https://docs.python.org/3/library/logging.html#logrecord-attributes>`_ and
-`datetime's strftime and strptime directives
-<https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`_
+listed in :ref:`logging's logrecord attributes docs <logrecord-attributes>` and
+:ref:`datetime's strftime and strptime directives <strftime-strptime-behavior>`
respectively.
If :setting:`LOG_SHORT_NAMES` is set, then the logs will not display the Scrapy
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 7260141e9..69a51a17c 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -174,9 +174,9 @@ Request objects
See :ref:`topics-request-meta` for a list of special meta keys
recognized by Scrapy.
- This dict is `shallow copied`_ when the request is cloned using the
- ``copy()`` or ``replace()`` methods, and can also be accessed, in your
- spider, from the ``response.meta`` attribute.
+ This dict is :mod:`shallow copied <copy>` when the request is
+ cloned using the ``copy()`` or ``replace()`` methods, and can also be
+ accessed, in your spider, from the ``response.meta`` attribute.
.. attribute:: Request.cb_kwargs
@@ -185,11 +185,9 @@ Request objects
for new Requests, which means by default callbacks only get a :class:`Response`
object as argument.
- This dict is `shallow copied`_ when the request is cloned using the
- ``copy()`` or ``replace()`` methods, and can also be accessed, in your
- spider, from the ``response.cb_kwargs`` attribute.
-
- .. _shallow copied: https://docs.python.org/3/library/copy.html
+ This dict is :mod:`shallow copied <copy>` when the request is
+ cloned using the ``copy()`` or ``replace()`` methods, and can also be
+ accessed, in your spider, from the ``response.cb_kwargs`` attribute.
.. method:: Request.copy()
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index d78a6253e..0049dbfca 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -26,9 +26,7 @@ do this by using an environment variable, ``SCRAPY_SETTINGS_MODULE``.
The value of ``SCRAPY_SETTINGS_MODULE`` should be in Python path syntax, e.g.
``myproject.settings``. Note that the settings module should be on the
-Python `import search path`_.
-
-.. _import search path: https://docs.python.org/3/tutorial/modules.html#the-module-search-path
+Python :ref:`import search path <tut-searchpath>`.
.. _populating-settings:
@@ -899,10 +897,9 @@ LOG_FORMAT
Default: ``'%(asctime)s [%(name)s] %(levelname)s: %(message)s'``
-String for formatting log messages. Refer to the `Python logging documentation`_ for the whole list of available
-placeholders.
-
-.. _Python logging documentation: https://docs.python.org/3/library/logging.html#logrecord-attributes
+String for formatting log messages. Refer to the
+:ref:`Python logging documentation <logrecord-attributes>` for the qwhole
+list of available placeholders.
.. setting:: LOG_DATEFORMAT
@@ -912,10 +909,9 @@ LOG_DATEFORMAT
Default: ``'%Y-%m-%d %H:%M:%S'``
String for formatting date/time, expansion of the ``%(asctime)s`` placeholder
-in :setting:`LOG_FORMAT`. Refer to the `Python datetime documentation`_ for the whole list of available
-directives.
-
-.. _Python datetime documentation: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
+in :setting:`LOG_FORMAT`. Refer to the
+:ref:`Python datetime documentation <strftime-strptime-behavior>` for the
+whole list of available directives.
.. setting:: LOG_FORMATTER