Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Drop the SEP as we decided we don't want to update it.

Andrey Rakhmatullin282fe3dd4fd4
diff --git a/sep/sep-021.rst b/sep/sep-021.rst
deleted file mode 100644
index 47cba004c..000000000
--- a/sep/sep-021.rst
+++ /dev/null
@@ -1,324 +0,0 @@
-======= ===================
-SEP 21
-Title Add-ons
-Author Pablo Hoffman
-Created 2014-02-14
-Status Draft
-======= ===================
-
-================
-SEP-021: Add-ons
-================
-
-This proposal introduces add-ons, a unified way to manage Scrapy extensions,
-middlewares and pipelines.
-
-Scrapy currently supports many hooks and mechanisms for extending its
-functionality, but no single entry point for enabling and configuring them.
-Instead, the hooks are spread over:
-
-* Spider middlewares (``SPIDER_MIDDLEWARES``)
-* Downloader middlewares (``DOWNLOADER_MIDDLEWARES``)
-* Downloader handlers (``DOWNLOADER_HANDLERS``)
-* Item pipelines (``ITEM_PIPELINES``)
-* Feed exporters and storages (``FEED_EXPORTERS``, ``FEED_STORAGES``)
-* Overridable components (``DUPEFILTER_CLASS``, ``STATS_CLASS``,
- ``SCHEDULER``, ``SPIDER_MANAGER_CLASS``, ``ITEM_PROCESSOR``, etc.)
-* Generic extensions (``EXTENSIONS``)
-* CLI commands (``COMMANDS_MODULE``)
-
-This approach has several shortfalls:
-
-* Enabling an extension often requires modifying many settings, often in a
- coordinated way, which is complex and error prone.
-* Extension developers have little control over ensuring their library
- dependencies and configuration requirements are met, especially since most
- extensions never 'see' a fully-configured crawler before it starts running.
-* The user is burdened with supervising potential interplay of extensions,
- especially non-included ones, ranging from setting name clashes to mutually
- excluding dependencies/configuration requirements.
-
-*Add-ons* search to remedy these shortcomings by enhancing Scrapy's extension
-management, making it easy-to-use and transparent for users while giving more
-configuration control to developers.
-
-
-Design goals and non-goals
-==========================
-
-Goals:
-
-* simple to manage: adding or removing extensions should be just a matter of
- adding or removing lines in a configuration file
-* backward compatibility with enabling extension the "old way" (i.e. modifying
- settings directly)
-
-Non-goals:
-
-* a way to publish, distribute or discover extensions (use pypi for that)
-
-
-User experience: managing add-ons
-=================================
-
-Add-ons are enabled and configured either via Scrapy's settings, or (for add-ons
-not bound to any project) in ``scrapy.cfg``.
-
-In the settings, add-ons can be enabled by adding either their name (for
-built-in add-ons), their Python path, or their file path, to a
-``INSTALLED_ADDONS`` setting. If necessary, each add-on can be configured by
-providing a dictionary-valued setting with the uppercase add-on name. For
-example, to enable and configure the built-in ``httpcache`` add-on and enable
-(without configuring) two custom add-ons, one via Python path and one via file
-path, add these entries to your settings module::
-
- INSTALLED_ADDONS = (
- 'httpcache',
- 'mymodule.filters.myfilter',
- 'mymodule/filters/otherfilter.py',
- )
-
- HTTPCACHE = {
- 'ignore_http_codes': [404, 503],
- }
-
-In ``scrapy.cfg``, add-ons are enabled and configured with one section per
-add-on. The section names correspond to the entries of ``INSTALLED_ADDONS``.
-The configuration from above could look like this::
-
- [addon:httpcache]
- ignore_http_codes = 404,503
-
- [addon:mymodule.filters.myfilter]
-
- [addon:mymodule/filters/otherfilter.py]
-
-
-Developer experience: writing add-ons
-=====================================
-
-Add-ons are (any) Python *objects* that implement Scrapy's *add-on interface*.
-The interface is enforced through ``zope.interface``. This leaves the choice of
-Python object up the developer. Examples:
-
-* for a small pipeline, the add-on interface could be implemented in the same
- class that also implements the ``open/close_spider`` and ``process_item``
- callbacks
-* for larger add-ons, or for clearer structure, the interface could be provided
- by a stand-alone module
-
-The absolute minimum interface consists of just two attributes:
-
-* ``NAME``: string with add-on name
-* ``VERSION``: PEP-440 style version string
-
-To be any useful, an add-on should implement at least one of the following
-callback methods:
-
-* ``update_addons()``: adds and configures other add-ons
-* ``update_settings()``: sets configuration (such as default values for this
- add-on and required settings for other extensions) and enables needed
- components.
-* ``check_configuration()``: receives the fully-initialized ``Crawler``
- instance before it starts running, performs additional dependency and
- configuration requirement checks
-
-Additionally, an add-on may (and should, where appropriate) provide one or more
-variables that can be used for automated detection of possible dependency
-clashes:
-
-* ``REQUIRES``: list of built-in or custom components required by this add-on,
- as PEP-440 strings
-* ``MODIFIES``: list of components whose functionality is affected or replaced
- by this add-on (a custom HTTP cache should list ``httpcache`` here)
-* ``PROVIDES``: list of components provided by this add-on (e.g. ``mongodb``
- for an extension that provides generic read/write access to a MongoDB
- database, releasing other components from having to provide their own
- database access methods)
-
-update_addons()
------------------
-
-Called:
-~~~~~~~
-
-Shortly after initialisation of the ``Crawler`` object.
-
-Arguments:
-~~~~~~~~~~
-
-* ``config``: configuration of this add-on
-* ``addons``: the add-on manager, providing methods to add and configure add-ons
-
-Purpose:
-~~~~~~~~
-
-* Configure and enable related add-ons, useful for 'umbrella add-ons' which
- chain-load other add-ons based on the configuration
-
-Examples:
-~~~~~~~~~
-
-.. code-block:: python
-
- def update_addons(config, addons):
- if "httpcache" not in addons.enabled:
- addons.add("httpcache", {"expiration_secs": 60})
-
-or:
-
-.. code-block:: python
-
- def update_addons(config, addons):
- if "otheraddon" in addons.enabled:
- addons.configs["otheraddon"]["some_config_name"] = True
-
-update_settings()
------------------
-
-Called:
-~~~~~~~
-
-Directly after the ``update_addons()`` callback of all add-ons has been called.
-
-Arguments:
-~~~~~~~~~~
-
-* ``config``: configuration of this add-on
-* ``settings``: the crawler's ``Settings`` instance containing all project
- settings
-
-Purpose:
-~~~~~~~~
-
-* Modify ``settings`` to enable required components
-* Expose some add-on specific configuration (``config``) into the global
- settings namespace (``settings``) if necessary
-* Raise exception if components can not be properly configured (e.g. on missing
- dependencies); Scrapy will print this exception *and exit* (making users
- explicitly acknowledge that the add-on does not work by forcing them to
- disable it).
-
-Side note:
-~~~~~~~~~~
-
-The ``MiddlewareManager.from_settings()`` method will receive a slight
-modification to allow directly placing Python objects instead of class paths
-in the middleware dict settings. This way, add-ons can place already
-instantiated components into the settings. This allows keeping configuration
-as local to components as possible and avoids cluttering up the global
-settings namespace. Furthermore, it allows reusing components (e.g. using
-two instances of the same mongodb pipeline to write to different locations).
-
-Examples:
-~~~~~~~~~
-
-::
-
- def update_settings(config, settings):
- # Don't care where this module is located
- settings.set['DOWNLADER_MIDDLEWARES']({
- __name__ + '.downloadermw.coolmw': 900,
- })
-
- # Instantiate components to not expose settings into
- # the global namespace
- from .pipelines import MySQLPipeline
- mysqlpl = MySQLPipeline(password = config['password'])
- settings.set['ITEM_PIPELINES']({
- mysqlpl: 200,
- })
-
-or::
-
- def update_settings(config, settings):
- # Assuming this class also has a process_item() method
- settings.set['ITEM_PIPELINES']({
- self: 200,
- })
-
-or::
-
- def update_settings(config, settings):
- try:
- import boto
- except ImportError:
- raise RuntimeError("boto library is required")
-
-check_configuration()
----------------------
-
-Called:
-~~~~~~~
-
-Shortly before the crawler starts crawling.
-
-Arguments:
-~~~~~~~~~~
-
-* ``config``: configuration of this add-on
-* ``crawler``: fully-initialized ``Crawler`` object, ready to start crawling
-
-Purpose:
-~~~~~~~~
-
-* Perform post-initialization checks like making sure the extension and its
- dependencies were configured properly.
-* Raise exception if a critical check failed; Scrapy will print this exception
- *and exit* (see ``update_settings()`` purpose for rationale on this).
-
-Examples:
-~~~~~~~~~
-
-::
-
- def check_configuration(config, crawler):
- if 'some.other.addon' not in crawler.addons.enabled:
- raise RuntimeError("Some other add-on required to use this add-on")
-
-
-Implementation
-==============
-
-A new core component, the *add-on manager*, is introduced to Scrapy. It
-facilitates loading add-ons, gathering and providing information on them,
-calling their callbacks at appropriate times, and performing basic checks for
-dependency and configuration clashes.
-
-Layout
-------
-
-A new ``AddonManager`` class is introduced, providing methods to
-
-* add and remove add-ons,
-* search for add-ons by name
-* read enabled add-ons and their configurations from the settings module and
- from ``settings.py``,
-* enable and disable add-ons
-* check for possible dependency incompatibilites by inspecting the collected
- ``REQUIRES``, ``MODIFIES`` and ``PROVIDES`` add-on variables
-* call the add-on callbacks
-
-Integration into start-up process
----------------------------------
-
-The settings used to crawl are not complete until the spider-specific settings
-have been loaded in ``Crawler.__init__()``. Add-on management follows this
-approach and only starts loading add-ons when the crawler is initialised.
-
-Instantiation and the calls ``update_addons()`` and ``update_settings()`` happen
-in ``Crawler.__init__()``. The final checks (i.e. the callback to
-``check_configuration()``) is coded into the ``Crawler.crawl()`` method after
-creating the engine.
-
-Finding add-ons
----------------
-
-Add-on localisation is governed by the add-on paths given in
-``INSTALLED_ADDONS`` (or by the section names if using ``scrapy.cfg``). If
-nothing is found at the given path, it is tried again with ``addons.``
-prepended (i.e. pointing to the project's ``addons`` folder or module), then
-with ``scrapy.addons.`` prepended (i.e. pointing to Scrapy's ``addons``
-submodule). If the object found has an ``_addon`` attribute, that attribute
-will be treated as the found add-on. This allows, for example, to change the
-add-on based on the Python version.