Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Use autodoc for XMLFeedSpider and CSVFeedSpider (#8239)

Adrian4526e1967591
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 728b08d1c..a0e7eeef2 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -621,101 +621,9 @@ a dictionary will be filled with it.
XMLFeedSpider
-------------
-.. class:: XMLFeedSpider
-
- XMLFeedSpider is designed for parsing XML feeds by iterating through them by a
- certain node name. The iterator can be chosen from: ``iternodes``, ``xml``,
- and ``html``. It's recommended to use the ``iternodes`` iterator for
- performance reasons, since the ``xml`` and ``html`` iterators generate the
- whole DOM at once in order to parse it. However, using ``html`` as the
- iterator may be useful when parsing XML with bad markup.
-
- To set the iterator and the tag name, you must define the following class
- attributes:
-
- .. attribute:: iterator
-
- A string which defines the iterator to use. It can be either:
-
- - ``'iternodes'`` - a fast iterator based on ``lxml``
-
- - ``'html'`` - an iterator which uses :class:`~scrapy.Selector`.
- Keep in mind this uses DOM parsing and must load all DOM in memory
- which could be a problem for big feeds. It also parses the feed
- with an HTML parser, which can silently mangle tags that HTML
- treats as void elements, such as ``<link>``, dropping their
- content and closing tag. Use ``xml`` or ``iternodes`` instead
- for feeds affected by this.
-
- - ``'xml'`` - an iterator which uses :class:`~scrapy.Selector`.
- Keep in mind this uses DOM parsing and must load all DOM in memory
- which could be a problem for big feeds
-
- It defaults to: ``'iternodes'``.
-
- .. attribute:: itertag
-
- A string with the name of the node (or element) to iterate in. Example:
-
- .. code-block:: python
-
- itertag = "product"
-
- .. attribute:: namespaces
-
- A list of ``(prefix, uri)`` tuples which define the namespaces
- available in that document that will be processed with this spider. The
- ``prefix`` and ``uri`` will be used to automatically register
- namespaces using the
- :meth:`~scrapy.Selector.register_namespace` method.
-
- You can then specify nodes with namespaces in the :attr:`itertag`
- attribute.
-
- Example:
-
- .. code-block:: python
-
- from scrapy.spiders import XMLFeedSpider
-
-
- class YourSpider(XMLFeedSpider):
-
- namespaces = [("n", "http://www.sitemaps.org/schemas/sitemap/0.9")]
- itertag = "n:url"
- # ...
-
- Apart from these new attributes, this spider has the following overridable
- methods too:
-
- .. method:: adapt_response(response)
-
- A method that receives the response as soon as it arrives from the spider
- middleware, before the spider starts parsing it. It can be used to modify
- the response body before parsing it. This method receives a response and
- also returns a response (it could be the same or another one).
-
- .. method:: parse_node(response, selector)
-
- This method is called for the nodes matching the provided tag name
- (``itertag``). Receives the response and an
- :class:`~scrapy.Selector` for each node. Overriding this
- method is mandatory. Otherwise, your spider won't work. This method
- must return an :ref:`item object <topics-items>`, a
- :class:`~scrapy.Request` object, or an iterable containing any of
- them.
-
- .. method:: process_results(response, results)
-
- This method is called for each result (item or request) returned by the
- spider, and it's intended to perform any last time processing required
- before returning the results to the framework core, for example setting the
- item IDs. It receives a list of results and the response which originated
- those results. It must return a list of results (items or requests).
-
- .. warning:: Because of its internal implementation, you must explicitly set
- callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;
- unexpected behaviour can occur otherwise.
+.. autoclass:: XMLFeedSpider
+ :members:
+ :exclude-members: parse_nodes
XMLFeedSpider example
@@ -755,32 +663,8 @@ prints them out, and stores some random data in an :class:`~scrapy.Item`.
CSVFeedSpider
-------------
-.. class:: CSVFeedSpider
-
- This spider is very similar to the XMLFeedSpider, except that it iterates
- over rows, instead of nodes. The method that gets called in each iteration
- is :meth:`parse_row`.
-
- .. attribute:: delimiter
-
- A string with the separator character for each field in the CSV file
- Defaults to ``','`` (comma).
-
- .. attribute:: quotechar
-
- A string with the enclosure character for each field in the CSV file
- Defaults to ``'"'`` (quotation mark).
-
- .. attribute:: headers
-
- A list of the column names in the CSV file.
-
- .. method:: parse_row(response, row)
-
- Receives a response and a dict (representing each row) with a key for each
- provided (or detected) header of the CSV file. This spider also gives the
- opportunity to override ``adapt_response`` and ``process_results`` methods
- for pre- and post-processing purposes.
+.. autoclass:: CSVFeedSpider
+ :members:
CSVFeedSpider example
~~~~~~~~~~~~~~~~~~~~~
diff --git a/scrapy/spiders/feed.py b/scrapy/spiders/feed.py
index ad5015227..e2caa5dfb 100644
--- a/scrapy/spiders/feed.py
+++ b/scrapy/spiders/feed.py
@@ -22,46 +22,103 @@ if TYPE_CHECKING:
class XMLFeedSpider(Spider):
- """
- This class intends to be the base class for spiders that scrape
- from XML feeds.
+ """Spider for parsing XML feeds by iterating through them by a certain node
+ name.
+
+ The iterator can be chosen from: ``iternodes``, ``xml``, and ``html``. It's
+ recommended to use the ``iternodes`` iterator for performance reasons, since
+ the ``xml`` and ``html`` iterators generate the whole DOM at once in order to
+ parse it. However, using ``html`` as the iterator may be useful when parsing
+ XML with bad markup.
+
+ To set the iterator and the tag name, you must define the :attr:`iterator`
+ and :attr:`itertag` class attributes.
+
+ .. warning:: Unlike in other spiders, a request without a callback is not
+ handled by :meth:`Spider.parse <scrapy.Spider.parse>`. Its response is
+ parsed as an XML feed, calling :meth:`parse_node` for each matching node.
- You can choose whether to parse the file using the 'iternodes' iterator, an
- 'xml' selector, or an 'html' selector. In most cases, it's convenient to
- use iternodes, since it's a faster and cleaner.
+ That is the way to have an additional feed parsed as such. If you want
+ your own parsing code to run for it instead, set its callback explicitly;
+ defining :meth:`~scrapy.Spider.parse` is not enough.
"""
iterator: str = "iternodes"
+ """The iterator to use. It can be either:
+
+ - ``'iternodes'`` - a fast iterator based on ``lxml``
+
+ - ``'html'`` - an iterator which uses :class:`~scrapy.Selector`. Keep in
+ mind this uses DOM parsing and must load all DOM in memory which could
+ be a problem for big feeds. It also parses the feed with an HTML
+ parser, which can silently mangle tags that HTML treats as void
+ elements, such as ``<link>``, dropping their content and closing tag.
+ Use ``xml`` or ``iternodes`` instead for feeds affected by this.
+
+ - ``'xml'`` - an iterator which uses :class:`~scrapy.Selector`. Keep in
+ mind this uses DOM parsing and must load all DOM in memory which could
+ be a problem for big feeds
+ """
+
itertag: str = "item"
+ """Name of the node (or element) to iterate in. Example:
+
+ .. code-block:: python
+
+ itertag = "product"
+ """
+
namespaces: Sequence[tuple[str, str]] = ()
+ """``(prefix, uri)`` tuples defining the namespaces available in that
+ document that will be processed with this spider. The ``prefix`` and
+ ``uri`` will be used to automatically register namespaces using the
+ :meth:`~scrapy.Selector.register_namespace` method.
+
+ You can then specify nodes with namespaces in the :attr:`itertag`
+ attribute.
+
+ Example:
+
+ .. code-block:: python
+
+ from scrapy.spiders import XMLFeedSpider
+
+
+ class YourSpider(XMLFeedSpider):
+
+ namespaces = [("n", "http://www.sitemaps.org/schemas/sitemap/0.9")]
+ itertag = "n:url"
+ # ...
+ """
def process_results(
self, response: Response, results: Iterable[Any]
) -> Iterable[Any]:
- """This overridable method is called for each result (item or request)
- returned by the spider, and it's intended to perform any last time
- processing required before returning the results to the framework core,
- for example setting the item GUIDs. It receives a list of results and
- the response which originated that results. It must return a list of
- results (items or requests).
+ """Handle *results*, the items and requests returned by the spider for
+ *response*, and return them, either unmodified or with changes.
+
+ Override it to perform any last-time processing required before
+ returning the results to the framework core, for example setting the
+ item IDs.
"""
return results
def adapt_response(self, response: Response) -> Response:
- """You can override this function in order to make any changes you want
- to into the feed before parsing it. This function must return a
- response.
+ """Handle *response* as soon as it arrives from the spider middleware,
+ before the spider starts parsing it, and return a response, which can be
+ the same one or a different one.
+
+ Override it to make any changes you want to the feed before parsing it,
+ e.g. to its body.
"""
return response
def parse_node(self, response: Response, selector: Selector) -> Any:
- """This method is called for the nodes matching the provided tag name
- (itertag). Receives the response and an Selector for each node.
+ """Handle a node of *response* matching :attr:`itertag`, for which
+ *selector* is a :class:`~scrapy.Selector`.
- This method must return either an item, a request, or a list
- containing any of them.
-
- This method must be overridden with your custom spider functionality.
+ This is a :meth:`spider callback <scrapy.Spider.parse>`. Overriding it
+ is mandatory. Otherwise, your spider won't work.
"""
if hasattr(self, "parse_item"):
warnings.warn(
@@ -116,39 +173,52 @@ class XMLFeedSpider(Spider):
class CSVFeedSpider(Spider):
"""Spider for parsing CSV feeds.
- It receives a CSV file in a response; iterates through each of its rows,
- and calls parse_row with a dict containing each field's data.
- This spider also gives the opportunity to override adapt_response and
- process_results methods for pre and post-processing purposes.
+ This spider is very similar to :class:`XMLFeedSpider`, except that it
+ iterates over rows, instead of nodes. The method that gets called in each
+ iteration is :meth:`parse_row`.
+
+ .. warning:: Unlike in other spiders, a request without a callback is not
+ handled by :meth:`Spider.parse <scrapy.Spider.parse>`. Its response is
+ parsed as a CSV feed, calling :meth:`parse_row` for each row.
+
+ That is the way to have an additional feed parsed as such. If you want
+ your own parsing code to run for it instead, set its callback explicitly;
+ defining :meth:`~scrapy.Spider.parse` is not enough.
+ """
+
+ delimiter: str | None = None
+ """Separator character for each field in the CSV file.
+
+ ``None`` means using the default delimiter of the :mod:`csv` module,
+ ``','`` (comma).
+ """
+
+ quotechar: str | None = None
+ """Enclosure character for each field in the CSV file.
- You can set some options regarding the CSV file, such as the delimiter, quotechar
- and the file's headers.
+ ``None`` means using the default quote character of the :mod:`csv` module,
+ ``'"'`` (quotation mark).
"""
- delimiter: str | None = (
- None # When this is None, python's csv module's default delimiter is used
- )
- quotechar: str | None = (
- None # When this is None, python's csv module's default quotechar is used
- )
headers: list[str] | None = None
+ """Column names in the CSV file."""
def process_results(
self, response: Response, results: Iterable[Any]
) -> Iterable[Any]:
- """This method has the same purpose as the one in XMLFeedSpider"""
+ """Same as :meth:`XMLFeedSpider.process_results`."""
return results
def adapt_response(self, response: Response) -> Response:
- """This method has the same purpose as the one in XMLFeedSpider"""
+ """Same as :meth:`XMLFeedSpider.adapt_response`."""
return response
def parse_row(self, response: Response, row: dict[str, str]) -> Any:
- """Receives a response and a dict (representing each row) with a key for
- each provided (or detected) header of the CSV file.
+ """Handle *row*, a row of *response* as a dict with a key for each
+ provided (or detected) header of the CSV file.
- This method must be overridden with your custom spider functionality.
+ Overriding this method is mandatory. Otherwise, your spider won't work.
"""
raise NotImplementedError