Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

DOC use autodocs for selectors; document more methods and attributes; suggest get/getall

Mikhail Korobovdc95ecbe25a5
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 2db982b37..95d6a1a04 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -891,127 +891,67 @@ Built-in Selectors reference
Selector objects
----------------
-.. class:: Selector(response=None, text=None, type=None)
+.. autoclass:: Selector
- An instance of :class:`Selector` is a wrapper over response to select
- certain parts of its content.
-
- ``response`` is an :class:`~scrapy.http.HtmlResponse` or an
- :class:`~scrapy.http.XmlResponse` object that will be used for selecting and
- extracting data.
-
- ``text`` is a unicode string or utf-8 encoded text for cases when a
- ``response`` isn't available. Using ``text`` and ``response`` together is
- undefined behavior.
-
- ``type`` defines the selector type, it can be ``"html"``, ``"xml"`` or ``None`` (default).
-
- If ``type`` is ``None``, the selector automatically chooses the best type
- based on ``response`` type (see below), or defaults to ``"html"`` in case it
- is used together with ``text``.
-
- If ``type`` is ``None`` and a ``response`` is passed, the selector type is
- inferred from the response type as follows:
-
- * ``"html"`` for :class:`~scrapy.http.HtmlResponse` type
- * ``"xml"`` for :class:`~scrapy.http.XmlResponse` type
- * ``"html"`` for anything else
-
- Otherwise, if ``type`` is set, the selector type will be forced and no
- detection will occur.
-
- .. method:: xpath(query)
-
- Find nodes matching the xpath ``query`` and return the result as a
- :class:`SelectorList` instance with all elements flattened. List
- elements implement :class:`Selector` interface too.
-
- ``query`` is a string containing the XPATH query to apply.
+ .. automethod:: xpath
.. note::
For convenience, this method can be called as ``response.xpath()``
- .. method:: css(query)
-
- Apply the given CSS selector and return a :class:`SelectorList` instance.
-
- ``query`` is a string containing the CSS selector to apply.
-
- In the background, CSS queries are translated into XPath queries using
- `cssselect`_ library and run ``.xpath()`` method.
+ .. automethod:: css
.. note::
- For convenience this method can be called as ``response.css()``
-
- .. method:: extract()
+ For convenience, this method can be called as ``response.css()``
- Serialize and return the matched nodes as a list of unicode strings.
- Percent encoded content is unquoted.
+ .. automethod:: get
- .. method:: re(regex)
+ See also: :ref:`old-extraction-api`
- Apply the given regex and return a list of unicode strings with the
- matches.
+ .. autoattribute:: attrib
- ``regex`` can be either a compiled regular expression or a string which
- will be compiled to a regular expression using ``re.compile(regex)``
+ See also: :ref:`selecting-attributes`.
- .. note::
+ .. automethod:: re
- Note that ``re()`` and ``re_first()`` both decode HTML entities (except ``<`` and ``&``).
+ .. automethod:: re_first
- .. method:: register_namespace(prefix, uri)
+ .. automethod:: register_namespace
- Register the given namespace to be used in this :class:`Selector`.
- Without registering namespaces you can't select or extract data from
- non-standard namespaces. See examples below.
+ .. automethod:: remove_namespaces
- .. method:: remove_namespaces()
+ .. automethod:: __bool__
- Remove all namespaces, allowing to traverse the document using
- namespace-less xpaths. See example below.
-
- .. method:: __nonzero__()
-
- Returns ``True`` if there is any real content selected or ``False``
- otherwise. In other words, the boolean value of a :class:`Selector` is
- given by the contents it selects.
+ .. automethod:: getall
+ This method is added to Selector for consistency; it is more useful
+ with SelectorList. See also: :ref:`old-extraction-api`
SelectorList objects
--------------------
-.. class:: SelectorList
-
- The :class:`SelectorList` class is a subclass of the builtin ``list``
- class, which provides a few additional methods.
-
- .. method:: xpath(query)
+.. autoclass:: SelectorList
- Call the ``.xpath()`` method for each element in this list and return
- their results flattened as another :class:`SelectorList`.
+ .. automethod:: xpath
- ``query`` is the same argument as the one in :meth:`Selector.xpath`
+ .. automethod:: css
- .. method:: css(query)
+ .. automethod:: getall
- Call the ``.css()`` method for each element in this list and return
- their results flattened as another :class:`SelectorList`.
+ See also: :ref:`old-extraction-api`
- ``query`` is the same argument as the one in :meth:`Selector.css`
+ .. automethod:: get
- .. method:: extract()
+ See also: :ref:`old-extraction-api`
- Call the ``.extract()`` method for each element in this list and return
- their results flattened, as a list of unicode strings.
+ .. automethod:: re
- .. method:: re()
+ .. automethod:: re_first
- Call the ``.re()`` method for each element in this list and return
- their results flattened, as a list of unicode strings.
+ .. autoattribute:: attrib
+ See also: :ref:`selecting-attributes`.
.. _selector-examples:
diff --git a/scrapy/selector/unified.py b/scrapy/selector/unified.py
index 64cb0232c..8f6cb1d79 100644
--- a/scrapy/selector/unified.py
+++ b/scrapy/selector/unified.py
@@ -27,6 +27,10 @@ def _response_from_text(text, st):
class SelectorList(_ParselSelector.selectorlist_cls, object_ref):
+ """
+ The :class:`SelectorList` class is a subclass of the builtin ``list``
+ class, which provides a few additional methods.
+ """
@deprecated(use_instead='.extract()')
def extract_unquoted(self):
return [x.extract_unquoted() for x in self]
@@ -41,6 +45,35 @@ class SelectorList(_ParselSelector.selectorlist_cls, object_ref):
class Selector(_ParselSelector, object_ref):
+ """
+ An instance of :class:`Selector` is a wrapper over response to select
+ certain parts of its content.
+
+ ``response`` is an :class:`~scrapy.http.HtmlResponse` or an
+ :class:`~scrapy.http.XmlResponse` object that will be used for selecting
+ and extracting data.
+
+ ``text`` is a unicode string or utf-8 encoded text for cases when a
+ ``response`` isn't available. Using ``text`` and ``response`` together is
+ undefined behavior.
+
+ ``type`` defines the selector type, it can be ``"html"``, ``"xml"``
+ or ``None`` (default).
+
+ If ``type`` is ``None``, the selector automatically chooses the best type
+ based on ``response`` type (see below), or defaults to ``"html"`` in case it
+ is used together with ``text``.
+
+ If ``type`` is ``None`` and a ``response`` is passed, the selector type is
+ inferred from the response type as follows:
+
+ * ``"html"`` for :class:`~scrapy.http.HtmlResponse` type
+ * ``"xml"`` for :class:`~scrapy.http.XmlResponse` type
+ * ``"html"`` for anything else
+
+ Otherwise, if ``type`` is set, the selector type will be forced and no
+ detection will occur.
+ """
__slots__ = ['response']
selectorlist_cls = SelectorList