Scrapy, a fast high-level web crawling & scraping framework for Python.
Drop the SEP as we decided we don't want to update it.
Andrey Rakhmatullin282fe3dd4fd4
diff --git a/sep/sep-021.rst b/sep/sep-021.rstdeleted file mode 100644index 47cba004c..000000000--- a/sep/sep-021.rst+++ /dev/null@@ -1,324 +0,0 @@-======= ===================-SEP 21-Title Add-ons-Author Pablo Hoffman-Created 2014-02-14-Status Draft-======= ===================--================-SEP-021: Add-ons-================--This proposal introduces add-ons, a unified way to manage Scrapy extensions,-middlewares and pipelines.--Scrapy currently supports many hooks and mechanisms for extending its-functionality, but no single entry point for enabling and configuring them.-Instead, the hooks are spread over:--* Spider middlewares (``SPIDER_MIDDLEWARES``)-* Downloader middlewares (``DOWNLOADER_MIDDLEWARES``)-* Downloader handlers (``DOWNLOADER_HANDLERS``)-* Item pipelines (``ITEM_PIPELINES``)-* Feed exporters and storages (``FEED_EXPORTERS``, ``FEED_STORAGES``)-* Overridable components (``DUPEFILTER_CLASS``, ``STATS_CLASS``,- ``SCHEDULER``, ``SPIDER_MANAGER_CLASS``, ``ITEM_PROCESSOR``, etc.)-* Generic extensions (``EXTENSIONS``)-* CLI commands (``COMMANDS_MODULE``)--This approach has several shortfalls:--* Enabling an extension often requires modifying many settings, often in a- coordinated way, which is complex and error prone.-* Extension developers have little control over ensuring their library- dependencies and configuration requirements are met, especially since most- extensions never 'see' a fully-configured crawler before it starts running.-* The user is burdened with supervising potential interplay of extensions,- especially non-included ones, ranging from setting name clashes to mutually- excluding dependencies/configuration requirements.--*Add-ons* search to remedy these shortcomings by enhancing Scrapy's extension-management, making it easy-to-use and transparent for users while giving more-configuration control to developers.---Design goals and non-goals-==========================--Goals:--* simple to manage: adding or removing extensions should be just a matter of- adding or removing lines in a configuration file-* backward compatibility with enabling extension the "old way" (i.e. modifying- settings directly)--Non-goals:--* a way to publish, distribute or discover extensions (use pypi for that)---User experience: managing add-ons-=================================--Add-ons are enabled and configured either via Scrapy's settings, or (for add-ons-not bound to any project) in ``scrapy.cfg``.--In the settings, add-ons can be enabled by adding either their name (for-built-in add-ons), their Python path, or their file path, to a-``INSTALLED_ADDONS`` setting. If necessary, each add-on can be configured by-providing a dictionary-valued setting with the uppercase add-on name. For-example, to enable and configure the built-in ``httpcache`` add-on and enable-(without configuring) two custom add-ons, one via Python path and one via file-path, add these entries to your settings module::-- INSTALLED_ADDONS = (- 'httpcache',- 'mymodule.filters.myfilter',- 'mymodule/filters/otherfilter.py',- )-- HTTPCACHE = {- 'ignore_http_codes': [404, 503],- }--In ``scrapy.cfg``, add-ons are enabled and configured with one section per-add-on. The section names correspond to the entries of ``INSTALLED_ADDONS``.-The configuration from above could look like this::-- [addon:httpcache]- ignore_http_codes = 404,503-- [addon:mymodule.filters.myfilter]-- [addon:mymodule/filters/otherfilter.py]---Developer experience: writing add-ons-=====================================--Add-ons are (any) Python *objects* that implement Scrapy's *add-on interface*.-The interface is enforced through ``zope.interface``. This leaves the choice of-Python object up the developer. Examples:--* for a small pipeline, the add-on interface could be implemented in the same- class that also implements the ``open/close_spider`` and ``process_item``- callbacks-* for larger add-ons, or for clearer structure, the interface could be provided- by a stand-alone module--The absolute minimum interface consists of just two attributes:--* ``NAME``: string with add-on name-* ``VERSION``: PEP-440 style version string--To be any useful, an add-on should implement at least one of the following-callback methods:--* ``update_addons()``: adds and configures other add-ons-* ``update_settings()``: sets configuration (such as default values for this- add-on and required settings for other extensions) and enables needed- components.-* ``check_configuration()``: receives the fully-initialized ``Crawler``- instance before it starts running, performs additional dependency and- configuration requirement checks--Additionally, an add-on may (and should, where appropriate) provide one or more-variables that can be used for automated detection of possible dependency-clashes:--* ``REQUIRES``: list of built-in or custom components required by this add-on,- as PEP-440 strings-* ``MODIFIES``: list of components whose functionality is affected or replaced- by this add-on (a custom HTTP cache should list ``httpcache`` here)-* ``PROVIDES``: list of components provided by this add-on (e.g. ``mongodb``- for an extension that provides generic read/write access to a MongoDB- database, releasing other components from having to provide their own- database access methods)--update_addons()--------------------Called:-~~~~~~~--Shortly after initialisation of the ``Crawler`` object.--Arguments:-~~~~~~~~~~--* ``config``: configuration of this add-on-* ``addons``: the add-on manager, providing methods to add and configure add-ons--Purpose:-~~~~~~~~--* Configure and enable related add-ons, useful for 'umbrella add-ons' which- chain-load other add-ons based on the configuration--Examples:-~~~~~~~~~--.. code-block:: python-- def update_addons(config, addons):- if "httpcache" not in addons.enabled:- addons.add("httpcache", {"expiration_secs": 60})--or:--.. code-block:: python-- def update_addons(config, addons):- if "otheraddon" in addons.enabled:- addons.configs["otheraddon"]["some_config_name"] = True--update_settings()--------------------Called:-~~~~~~~--Directly after the ``update_addons()`` callback of all add-ons has been called.--Arguments:-~~~~~~~~~~--* ``config``: configuration of this add-on-* ``settings``: the crawler's ``Settings`` instance containing all project- settings--Purpose:-~~~~~~~~--* Modify ``settings`` to enable required components-* Expose some add-on specific configuration (``config``) into the global- settings namespace (``settings``) if necessary-* Raise exception if components can not be properly configured (e.g. on missing- dependencies); Scrapy will print this exception *and exit* (making users- explicitly acknowledge that the add-on does not work by forcing them to- disable it).--Side note:-~~~~~~~~~~--The ``MiddlewareManager.from_settings()`` method will receive a slight-modification to allow directly placing Python objects instead of class paths-in the middleware dict settings. This way, add-ons can place already-instantiated components into the settings. This allows keeping configuration-as local to components as possible and avoids cluttering up the global-settings namespace. Furthermore, it allows reusing components (e.g. using-two instances of the same mongodb pipeline to write to different locations).--Examples:-~~~~~~~~~--::-- def update_settings(config, settings):- # Don't care where this module is located- settings.set['DOWNLADER_MIDDLEWARES']({- __name__ + '.downloadermw.coolmw': 900,- })-- # Instantiate components to not expose settings into- # the global namespace- from .pipelines import MySQLPipeline- mysqlpl = MySQLPipeline(password = config['password'])- settings.set['ITEM_PIPELINES']({- mysqlpl: 200,- })--or::-- def update_settings(config, settings):- # Assuming this class also has a process_item() method- settings.set['ITEM_PIPELINES']({- self: 200,- })--or::-- def update_settings(config, settings):- try:- import boto- except ImportError:- raise RuntimeError("boto library is required")--check_configuration()------------------------Called:-~~~~~~~--Shortly before the crawler starts crawling.--Arguments:-~~~~~~~~~~--* ``config``: configuration of this add-on-* ``crawler``: fully-initialized ``Crawler`` object, ready to start crawling--Purpose:-~~~~~~~~--* Perform post-initialization checks like making sure the extension and its- dependencies were configured properly.-* Raise exception if a critical check failed; Scrapy will print this exception- *and exit* (see ``update_settings()`` purpose for rationale on this).--Examples:-~~~~~~~~~--::-- def check_configuration(config, crawler):- if 'some.other.addon' not in crawler.addons.enabled:- raise RuntimeError("Some other add-on required to use this add-on")---Implementation-==============--A new core component, the *add-on manager*, is introduced to Scrapy. It-facilitates loading add-ons, gathering and providing information on them,-calling their callbacks at appropriate times, and performing basic checks for-dependency and configuration clashes.--Layout---------A new ``AddonManager`` class is introduced, providing methods to--* add and remove add-ons,-* search for add-ons by name-* read enabled add-ons and their configurations from the settings module and- from ``settings.py``,-* enable and disable add-ons-* check for possible dependency incompatibilites by inspecting the collected- ``REQUIRES``, ``MODIFIES`` and ``PROVIDES`` add-on variables-* call the add-on callbacks--Integration into start-up process------------------------------------The settings used to crawl are not complete until the spider-specific settings-have been loaded in ``Crawler.__init__()``. Add-on management follows this-approach and only starts loading add-ons when the crawler is initialised.--Instantiation and the calls ``update_addons()`` and ``update_settings()`` happen-in ``Crawler.__init__()``. The final checks (i.e. the callback to-``check_configuration()``) is coded into the ``Crawler.crawl()`` method after-creating the engine.--Finding add-ons------------------Add-on localisation is governed by the add-on paths given in-``INSTALLED_ADDONS`` (or by the section names if using ``scrapy.cfg``). If-nothing is found at the given path, it is tried again with ``addons.``-prepended (i.e. pointing to the project's ``addons`` folder or module), then-with ``scrapy.addons.`` prepended (i.e. pointing to Scrapy's ``addons``-submodule). If the object found has an ``_addon`` attribute, that attribute-will be treated as the found add-on. This allows, for example, to change the-add-on based on the Python version.