Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Documentation of link extractor usage (#4775)

Habeeb Shopejuf47b120e2b67
diff --git a/docs/index.rst b/docs/index.rst
index 11aa5c9be..da264fb34 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -78,7 +78,6 @@ Basic concepts
topics/settings
topics/exceptions
-
:doc:`topics/commands`
Learn about the command-line tool used to manage your Scrapy project.
diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rst
index ed32411b0..e12ad45e0 100644
--- a/docs/topics/link-extractors.rst
+++ b/docs/topics/link-extractors.rst
@@ -10,12 +10,19 @@ The ``__init__`` method of
:class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` takes settings that
determine which links may be extracted. :class:`LxmlLinkExtractor.extract_links
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` returns a
-list of matching :class:`scrapy.link.Link` objects from a
+list of matching :class:`~scrapy.link.Link` objects from a
:class:`~scrapy.http.Response` object.
Link extractors are used in :class:`~scrapy.spiders.CrawlSpider` spiders
-through a set of :class:`~scrapy.spiders.Rule` objects. You can also use link
-extractors in regular spiders.
+through a set of :class:`~scrapy.spiders.Rule` objects.
+
+You can also use link extractors in regular spiders. For example, you can instantiate
+:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class
+variable in your spider, and use it from your spider callbacks::
+
+ def parse(self, response):
+ for link in self.link_extractor.extract_links(response):
+ yield Request(link.url, callback=self.parse)
.. _topics-link-extractors-ref:
@@ -145,4 +152,12 @@ LxmlLinkExtractor
.. automethod:: extract_links
+Link
+----
+
+.. module:: scrapy.link
+ :synopsis: Link from link extractors
+
+.. autoclass:: Link
+
.. _scrapy.linkextractors: https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractors/__init__.py
diff --git a/scrapy/link.py b/scrapy/link.py
index 684735f6e..e70667361 100644
--- a/scrapy/link.py
+++ b/scrapy/link.py
@@ -7,7 +7,22 @@ its documentation in: docs/topics/link-extractors.rst
class Link:
- """Link objects represent an extracted link by the LinkExtractor."""
+ """Link objects represent an extracted link by the LinkExtractor.
+
+ Using the anchor tag sample below to illustrate the parameters::
+
+ <a href="https://example.com/nofollow.html#foo" rel="nofollow">Dont follow this one</a>
+
+ :param url: the absolute url being linked to in the anchor tag.
+ From the sample, this is ``https://example.com/nofollow.html``.
+
+ :param text: the text in the anchor tag. From the sample, this is ``Dont follow this one``.
+
+ :param fragment: the part of the url after the hash symbol. From the sample, this is ``foo``.
+
+ :param nofollow: an indication of the presence or absence of a nofollow value in the ``rel`` attribute
+ of the anchor tag.
+ """
__slots__ = ['url', 'text', 'fragment', 'nofollow']