Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Provide complete API documentation coverage of scrapy.exporters

Adrián Chaves0fa384e80def
diff --git a/docs/conf.py b/docs/conf.py
index f49f79cd5..eba416cd6 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -252,6 +252,10 @@ coverage_ignore_pyobjects = [
# Private exception used by the command-line interface implementation.
r'^scrapy\.exceptions\.UsageError',
+
+ # Methods of BaseItemExporter subclasses are only documented in
+ # BaseItemExporter.
+ r'^scrapy\.exporters\.(?!BaseItemExporter\b)\w*?\.',
]
diff --git a/docs/news.rst b/docs/news.rst
index ce5b8b406..5915ba742 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -1269,8 +1269,8 @@ This 1.1 release brings a lot of interesting features and bug fixes:
this behavior, update :setting:`ROBOTSTXT_OBEY` in ``settings.py`` file
after creating a new project.
- Exporters now work on unicode, instead of bytes by default (:issue:`1080`).
- If you use ``PythonItemExporter``, you may want to update your code to
- disable binary mode which is now deprecated.
+ If you use :class:`~scrapy.exporters.PythonItemExporter`, you may want to
+ update your code to disable binary mode which is now deprecated.
- Accept XML node names containing dots as valid (:issue:`1533`).
- When uploading files or images to S3 (with ``FilesPipeline`` or
``ImagesPipeline``), the default ACL policy is now "private" instead
@@ -1408,8 +1408,8 @@ Bugfixes
- Fixed bug on ``XMLItemExporter`` with non-string fields in
items (:issue:`1738`).
- Fixed startproject command in OS X (:issue:`1635`).
-- Fixed PythonItemExporter and CSVExporter for non-string item
- types (:issue:`1737`).
+- Fixed :class:`~scrapy.exporters.PythonItemExporter` and CSVExporter for
+ non-string item types (:issue:`1737`).
- Various logging related fixes (:issue:`1294`, :issue:`1419`, :issue:`1263`,
:issue:`1624`, :issue:`1654`, :issue:`1722`, :issue:`1726` and :issue:`1303`).
- Fixed bug in ``utils.template.render_templatefile()`` (:issue:`1212`).
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index f5048d2da..11b3045ec 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -165,9 +165,9 @@ BaseItemExporter
value unchanged except for ``unicode`` values which are encoded to
``str`` using the encoding declared in the :attr:`encoding` attribute.
- :param field: the field being serialized. If a raw dict is being
+ :param field: the field being serialized. If a raw dict is being
exported (not :class:`~.Item`) *field* value is an empty dict.
- :type field: :class:`~scrapy.item.Field` object or an empty dict
+ :type field: :class:`~scrapy.item.Field` object or an empty dict
:param name: the name of the field being serialized
:type name: str
@@ -223,6 +223,12 @@ BaseItemExporter
* ``indent<=0`` each item on its own line, no indentation
* ``indent>0`` each item on its own line, indented with the provided numeric value
+PythonItemExporter
+------------------
+
+.. autoclass:: PythonItemExporter
+
+
.. highlight:: none
XmlItemExporter
@@ -410,3 +416,8 @@ JsonLinesItemExporter
this exporter is well suited for serializing large amounts of data.
.. _JSONEncoder: https://docs.python.org/2/library/json.html#json.JSONEncoder
+
+MarshalItemExporter
+-------------------
+
+.. autoclass:: MarshalItemExporter
diff --git a/scrapy/exporters.py b/scrapy/exporters.py
index 695c74fec..6fc87ed18 100644
--- a/scrapy/exporters.py
+++ b/scrapy/exporters.py
@@ -276,6 +276,13 @@ class PickleItemExporter(BaseItemExporter):
class MarshalItemExporter(BaseItemExporter):
+ """Exports items in a Python-specific binary format (see
+ :mod:`marshal`).
+
+ :param file: The file-like object to use for exporting the data. Its
+ ``write`` method should accept :class:`bytes` (a disk file
+ opened in binary mode, a :class:`~io.BytesIO` object, etc)
+ """
def __init__(self, file, **kwargs):
self._configure(kwargs)
@@ -297,10 +304,13 @@ class PprintItemExporter(BaseItemExporter):
class PythonItemExporter(BaseItemExporter):
- """The idea behind this exporter is to have a mechanism to serialize items
- to built-in python types so any serialization library (like
- json, msgpack, binc, etc) can be used on top of it. Its main goal is to
- seamless support what BaseItemExporter does plus nested items.
+ """This is a base class for item exporters that extends
+ :class:`BaseItemExporter` with support for nested items.
+
+ It serializes items to built-in Python types, so that any serialization
+ library (e.g. :mod:`json` or msgpack_) can be used on top of it.
+
+ .. _msgpack: https://pypi.org/project/msgpack/
"""
def _configure(self, options, dont_fail=False):
self.binary = options.pop('binary', True)