Scrapy, a fast high-level web crawling & scraping framework for Python.
Documentation of link extractor usage (#4775)
Habeeb Shopejuf47b120e2b67
diff --git a/docs/index.rst b/docs/index.rstindex 11aa5c9be..da264fb34 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -78,7 +78,6 @@ Basic conceptstopics/settingstopics/exceptions-:doc:`topics/commands`Learn about the command-line tool used to manage your Scrapy project.diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rstindex ed32411b0..e12ad45e0 100644--- a/docs/topics/link-extractors.rst+++ b/docs/topics/link-extractors.rst@@ -10,12 +10,19 @@ The ``__init__`` method of:class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` takes settings thatdetermine which links may be extracted. :class:`LxmlLinkExtractor.extract_links<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` returns a-list of matching :class:`scrapy.link.Link` objects from a+list of matching :class:`~scrapy.link.Link` objects from a:class:`~scrapy.http.Response` object.Link extractors are used in :class:`~scrapy.spiders.CrawlSpider` spiders-through a set of :class:`~scrapy.spiders.Rule` objects. You can also use link-extractors in regular spiders.+through a set of :class:`~scrapy.spiders.Rule` objects.++You can also use link extractors in regular spiders. For example, you can instantiate+:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` into a class+variable in your spider, and use it from your spider callbacks::++ def parse(self, response):+ for link in self.link_extractor.extract_links(response):+ yield Request(link.url, callback=self.parse).. _topics-link-extractors-ref:@@ -145,4 +152,12 @@ LxmlLinkExtractor.. automethod:: extract_links+Link+----++.. module:: scrapy.link+ :synopsis: Link from link extractors++.. autoclass:: Link+.. _scrapy.linkextractors: https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractors/__init__.pydiff --git a/scrapy/link.py b/scrapy/link.pyindex 684735f6e..e70667361 100644--- a/scrapy/link.py+++ b/scrapy/link.py@@ -7,7 +7,22 @@ its documentation in: docs/topics/link-extractors.rstclass Link:- """Link objects represent an extracted link by the LinkExtractor."""+ """Link objects represent an extracted link by the LinkExtractor.++ Using the anchor tag sample below to illustrate the parameters::++ <a href="https://example.com/nofollow.html#foo" rel="nofollow">Dont follow this one</a>++ :param url: the absolute url being linked to in the anchor tag.+ From the sample, this is ``https://example.com/nofollow.html``.++ :param text: the text in the anchor tag. From the sample, this is ``Dont follow this one``.++ :param fragment: the part of the url after the hash symbol. From the sample, this is ``foo``.++ :param nofollow: an indication of the presence or absence of a nofollow value in the ``rel`` attribute+ of the anchor tag.+ """__slots__ = ['url', 'text', 'fragment', 'nofollow']