Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Remove unneeded code.

Andrey Rakhmatulline7124447f7e8
diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst
index 5d1a4f753..6a9590f33 100644
--- a/docs/topics/addons.rst
+++ b/docs/topics/addons.rst
@@ -15,12 +15,13 @@ Activating and configuring add-ons
==================================
Add-ons and their configuration live in Scrapy's
-:class:`~scrapy.addons.AddonManager`. During Scrapy's start-up process, and
-only then, the add-on manager will read a list of enabled add-ons and their
-configurations from your ``ADDONS`` setting.
+:class:`~scrapy.addons.AddonManager`. During a :class:`~scrapy.crawler.Crawler`
+initialization the add-on manager will read a list of enabled add-ons from your
+``ADDONS`` setting and their optional configuration from the respective
+settings.
The ``ADDONS`` setting is a dict in which every key is an addon class or its
-import path and the vaoue is its priority.
+import path and the value is its priority.
The configuration of an add-on, if necessary at all, is stored as a dictionary
setting whose name is the uppercase add-on name.
@@ -37,66 +38,17 @@ configuration) are enabled/configured in a project's ``settings.py``::
'some_config': True,
}
-Enabling and configuring add-ons within Python code
----------------------------------------------------
-
-The :class:`~scrapy.addons.AddonManager` will only read from Scrapy's settings
-*at the beginning* of Scrapy's start-up process.
-Afterwards, i.e. as soon as the :class:`~scrapy.addons.AddonManager` is
-populated, changing the ``ADDONS`` setting or any of the add-on
-configuration dictionary settings will have no effect.
-
-If you want to enable, disable, or configure add-ons in Python code, for example
-when writing your own add-on, you will have to use the
-:class:`~scrapy.addons.AddonManager`. You can access the add-on manager through
-either ``crawler.addons`` or, if you are writing an add-on, through the
-``addons`` argument of the :meth:`update_addons` callback. The add-on manager
-provides many useful methods and attributes to facilitate interacting with the
-add-ons framework, e.g.:
-
-* an :meth:`~scrapy.addons.AddonManager.add` method to load add-ons,
-* the :attr:`~scrapy.addons.AddonManager.enabled` list of enabled add-ons,
-* :meth:`~scrapy.addons.AddonManager.enable` and
- :meth:`~scrapy.addons.AddonManager.disable` methods,
-* the :attr:`~scrapy.addons.AddonManager.configs` dictionary which holds the
- configuration of all add-ons.
-
-In this example, we ensure that the ``httpcache`` add-on is loaded, and that
-its ``expiration_secs`` configuration is set to ``60``::
-
- # addons is an instance of AddonManager
- if 'httpcache' not in addons:
- addons.add('httpcache', {'expiration_secs': 60})
- else:
- addons.configs['httpcache']['expiration_secs'] = 60
-
Writing your own add-ons
========================
-Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*.
-The interface is enforced through ``zope.interface``. This leaves the choice of
-Python object up the developer. Examples:
-
-* for a small pipeline, the add-on interface could be implemented in the same
- class that also implements the ``open/close_spider`` and ``process_item``
- callbacks,
-* for larger add-ons, or for clearer structure, the interface could be provided
- by a stand-alone module.
-
-The absolute minimum interface consists of two attributes:
+Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*:
.. attribute:: name
string with add-on name
-.. attribute:: version
-
- version string (PEP-404, e.g. ``'1.0.1'``)
-
-Of course, stating just these two attributes will not get you very far. Add-ons
-can provide three callback methods that are called at various stages before the
-crawling process:
+ :type: ``str``
.. method:: update_settings(config, settings)
@@ -124,75 +76,18 @@ crawling process:
:param crawler: Fully initialized Scrapy crawler
:type crawler: :class:`~scrapy.crawler.Crawler`
-.. method:: update_addons(config, addons)
-
- This method is called immediately before :meth:`update_settings`, and should
- be used to enable and configure other *add-ons* only.
-
- Add-ons that are added to the :class:`~scrapy.addons.AddonManager` during
- this callback will also have their :meth:`update_addons` method called.
-
- :param config: Configuration of this add-on
- :type config: ``dict``
-
- :param addons: Add-on manager holding all loaded add-ons
- :type addons: :class:`~scrapy.addons.AddonManager`
-
-Additionally, add-ons may (and should, where appropriate) provide one or more
-attributes that can be used for limited automated detection of possible
-dependency clashes:
-
-.. attribute:: requires
-
- list of built-in or custom components needed by this add-on, as strings.
-
-.. attribute:: modifies
-
- list of built-in or custom components whose functionality is affected or
- replaced by this add-on (a custom HTTP cache should list ``httpcache`` here)
-
-.. attribute:: provides
-
- list of components provided by this add-on (e.g. ``mongodb`` for an
- extension that provides generic read/write access to a MongoDB database)
-
-The entries in the :attr:`requires` and :attr:`modifies` attributes can be add-on
-names or components from other add-ons' :attr:`provides` attribute. You can
-specify :pep:`440`-style information about required versions. Examples::
-
- requires = ['httpcache']
- requires = ['otheraddon >= 2.0', 'yetanotheraddon']
-
-The Python object or module that is pointed to by an add-on path (e.g. given in
-the ``ADDONS`` setting, or given to
-:meth:`~scrapy.addons.AddonManager.add`) does not necessarily have to be an
-add-on. Instead, it can provide an ``_addon`` attribute. This attribute can be
-either an add-on or another add-on path.
-
Add-on base class
=================
Scrapy comes with a built-in base class for add-ons which provides some
-convenience functionality:
-
-* basic settings can be exported via :meth:`~scrapy.addons.Addon.export_basics`,
- configurable via :attr:`~scrapy.addons.Addon.basic_settings`.
-* a single component (e.g. an item pipeline or a downloader middleware) can be
- inserted into Scrapy's settings via
- :meth:`~scrapy.addons.Addon.export_component`, configurable via
- :attr:`~scrapy.addons.Addon.component_type`,
- :attr:`~scrapy.addons.Addon.component_key`,
- :attr:`~scrapy.addons.Addon.component`, and the ``order`` key in
- :attr:`~scrapy.addons.Addon.default_config`.
-* the add-on configuration can be exposed into Scrapy's settings via
- :meth:`~scrapy.addons.Addon.export_config`, configurable via
- :attr:`~scrapy.addons.Addon.default_config`,
- :attr:`~scrapy.addons.Addon.config_mapping`, and
- :attr:`~scrapy.addons.Addon.settings_prefix`.
+convenience functionality: the add-on configuration can be exposed into
+Scrapy's settings via :meth:`~scrapy.addons.Addon.export_config`, configurable
+via :attr:`~scrapy.addons.Addon.default_config` and
+:attr:`~scrapy.addons.Addon.config_mapping`.
By default, the base add-on class will expose the add-on configuration into
-Scrapy's settings namespace, in caps and with the add-on name prepended. It is
+Scrapy's settings namespace, in upper case. It is
easy to write your own functionality while still being able to use the
convenience functions by overwriting
:meth:`~scrapy.addons.Addon.update_settings`.
@@ -213,13 +108,11 @@ Set some basic configuration using the :class:`Addon` base class::
class MyAddon(Addon):
name = 'myaddon'
- version = '1.0'
- component = 'path.to.mypipeline'
- component_type = 'ITEM_PIPELINES'
- component_order = 200
- basic_settings = {
- 'DNSCACHE_ENABLED': False,
- }
+
+ def update_settings(self, config, settings):
+ super().update_settings(settings)
+ settings["ITEM_PIPELINES"]["path.to.mypipeline"] = 200
+ settings["DNSCACHE_ENABLED"] = True
Check dependencies::
@@ -227,78 +120,22 @@ Check dependencies::
class MyAddon(Addon):
name = 'myaddon'
- version = '1.0'
def update_settings(self, config, settings):
try:
import boto
except ImportError:
raise RuntimeError("myaddon requires the boto library")
- self.export_config(config, settings)
-
-Enable a component that lives relative to the add-on (see
-:ref:`topics-api-settings`)::
-
- from scrapy.addons import Addon
-
- class MyAddon(Addon):
- name = 'myaddon'
- version = '1.0'
- component = __name__ + '.downloadermw.coolmw'
- component_type = 'DOWNLOADER_MIDDLEWARES'
- component_order = 900
-
-Instantiate components ad hoc::
-
- from path.to.my.pipelines import MySQLPipeline
-
- class MyAddon(object):
- name = 'myaddon'
- version = '1.0'
-
- def update_settings(self, config, settings):
- mysqlpl = MySQLPipeline(password=config['password'])
- settings.set(
- 'ITEM_PIPELINES',
- {mysqlpl: 200},
- priority='addon',
- )
-
-Provide add-on interface along component interface::
-
- class MyPipeline(object):
- name = 'mypipeline'
- version = '1.0'
-
- def process_item(self, item, spider):
- # Do some processing here
- return item
-
- def update_settings(self, config, settings):
- settings.set(
- 'ITEM_PIPELINES',
- {self: 200},
- priority='addon',
- )
-
-Enable another addon (see :ref:`topics-api-addonmanager`)::
-
- class MyAddon(object):
- name = 'myaddon'
- version = '1.0'
-
- def update_addons(self, config, addons):
- if 'httpcache' not in addons.enabled:
- addons.add('httpcache', {'expiration_secs': 60})
+ super().update_settings(settings)
Check configuration of fully initialized crawler (see
:ref:`topics-api-crawler`)::
class MyAddon(object):
name = 'myaddon'
- version = '1.0'
def update_settings(self, config, settings):
+ super().update_settings(settings)
settings.set('DNSCACHE_ENABLED', False, priority='addon')
def check_configuration(self, config, crawler):
@@ -306,43 +143,3 @@ Check configuration of fully initialized crawler (see
# The spider, some other add-on, or the user messed with the
# DNS cache setting
raise ValueError("myaddon is incompatible with DNS cache")
-
-Provide add-on interface through a module:
-
-.. code-block:: python
-
- name = "AddonModule"
- version = "1.0"
-
-
- class MyPipeline(object):
- ...
-
-
- class MyDownloaderMiddleware(object):
- ...
-
-
- def update_settings(config, settings):
- settings.set(
- "ITEM_PIPELINES",
- {MyPipeline(): 200},
- priority="addon",
- )
- settings.set(
- "DOWNLOADER_MIDDLEWARES",
- {MyDownloaderMiddleware(): 800},
- priority="addon",
- )
-
-Forward to other add-ons depending on Python version::
-
- # This could be a Python module, say project/pipelines/mypipeline.py, but
- # could also be done inside a class, etc.
- import six
-
- if six.PY3:
- # We're running Python 3
- _addon = 'path.to.addon'
- else:
- _addon = 'path.to.other.addon'
diff --git a/scrapy/addons.py b/scrapy/addons.py
index aced6092a..a54086fda 100644
--- a/scrapy/addons.py
+++ b/scrapy/addons.py
@@ -1,108 +1,27 @@
-import warnings
-from collections import OrderedDict, defaultdict
from collections.abc import Mapping
-from inspect import isclass
-from typing import Dict
+from typing import Any, Dict, Iterator, Optional, OrderedDict
-import zope.interface
-from packaging.requirements import Requirement
-from packaging.version import Version
-from zope.interface.verify import verifyObject
-
-from scrapy.interfaces import IAddon
from scrapy.utils.conf import build_component_list
-from scrapy.utils.misc import load_module_or_object
+from scrapy.utils.misc import load_object
class Addon(object):
- basic_settings = None
- """``dict`` of settings that will be exported via :meth:`export_basics`."""
+ name: str
default_config = None
"""``dict`` with default configuration."""
config_mapping = None
"""``dict`` with mappings from config names to setting names. The given
- setting names will be taken as given, i.e. they will be neither prefixed
- nor uppercased.
- """
-
- component_type = None
- """Component setting into which to export via :meth:`export_component`. Can
- be any of the dictionary-like component setting names. If ``None``,
- :meth:`export_component` will do nothing.
- """
-
- component_key = None
- """Key to be used in the component dictionary setting when exporting via
- :meth:`export_component`. This is only useful for the settings that have
- no order, e.g. ``DOWNLOAD_HANDLERS`` or ``FEED_EXPORTERS``.
- """
-
- component_order = 0
- """Component order to use when not given in the add-on configuration. Has
- no effect for component types that use :attr:`component_key`.
+ setting names will be taken as given, not uppercased.
"""
- component = None
- """Component to be inserted via :meth:`export_component`. This can be
- anything that can be used in the dictionary-like component settings, i.e.
- a class path or a class. If ``None``, it is assumed that the add-on itself
- also provides the component interface, and ``self`` will be used.
- """
-
- settings_prefix = None
- """Prefix with which the add-on configuration will be exported into the
- global settings namespace via :meth:`export_config`. If ``None``,
- :attr:`name` will be used. If ``False``, no configuration will be exported.
- """
-
- def export_component(self, config, settings):
- """Export the component in :attr:`component` into the dictionary-like
- component setting derived from :attr:`component_type`.
-
- Where applicable, the order parameter of the component (i.e. the
- dictionary value) will be retrieved from the ``order`` add-on
- configuration value.
-
- :param config: Add-on configuration from which to read component order
- :type config: ``dict``
-
- :param settings: Settings object into which to export component
- :type settings: :class:`~scrapy.settings.Settings`
- """
- if self.component_type:
- comp = self.component or self
- if self.component_key:
- # e.g. for DOWNLOAD_HANDLERS: {'http': 'myclass'}
- k = self.component_key
- v = comp
- else:
- # e.g. for DOWNLOADER_MIDDLEWARES: {'myclass': 100}
- k = comp
- v = config.get("order", self.component_order)
- settings[self.component_type].update({k: v}, "addon")
-
- def export_basics(self, settings):
- """Export the :attr:`basic_settings` attribute into the settings object.
-
- All settings will be exported with ``addon`` priority (see
- :ref:`topics-api-settings`).
-
- :param settings: Settings object into which to expose the basic settings
- :type settings: :class:`~scrapy.settings.Settings`
- """
- for setting, value in (self.basic_settings or {}).items():
- settings.set(setting, value, "addon")
-
def export_config(self, config, settings):
- """Export the add-on configuration, all keys in caps and with
- :attr:`settings_prefix` or :attr:`name` prepended, into the settings
+ """Export the add-on configuration, all keys in caps, into the settings
object.
For example, the add-on configuration ``{'key': 'value'}`` will export
- the setting ``ADDONNAME_KEY`` with a value of ``value``. All settings
+ the setting ``KEY`` with a value of ``value``. All settings
will be exported with ``addon`` priority (see
:ref:`topics-api-settings`).
@@ -112,11 +31,8 @@ class Addon(object):
:param settings: Settings object into which to export the configuration
:type settings: :class:`~scrapy.settings.Settings`
"""
- if self.settings_prefix is False:
- return
conf = self.default_config or {}
conf.update(config)
- prefix = self.settings_prefix or self.name
# Since default exported config is case-insensitive (everything will be
# uppercased), make mapped config case-insensitive as well
conf_mapping = {k.lower(): v for k, v in (self.config_mapping or {}).items()}
@@ -124,14 +40,11 @@ class Addon(object):
if key.lower() in conf_mapping:
key = conf_mapping[key.lower()]
else:
- key = (prefix + "_" + key).upper()
+ key = key.upper()
settings.set(key, val, "addon")
def update_settings(self, config, settings):
- """Export both the basic settings and the add-on configuration. I.e.,
- call :meth:`export_basics` and :meth:`export_config`.
-
- For more advanced add-ons, you may want to override this callback.
+ """Modifiy `settings` to enable and configure required components.
:param config: Add-on configuration
:type config: ``dict``
@@ -139,12 +52,21 @@ class Addon(object):
:param settings: Crawler settings object
:type settings: :class:`~scrapy.settings.Settings`
"""
- self.export_component(config, settings)
- self.export_basics(settings)
self.export_config(config, settings)
+ def check_configuration(self, config, crawler):
+ """Perform post-initialization checks on fully configured `crawler`.
+
+ :param config: Add-on configuration
+ :type config: ``dict``
+
+ :param crawler: the fully-initialized crawler
+ :type crawler: :class:`~scrapy.crawler.Crawler`
+ """
+ pass
+
-class AddonManager(Mapping):
+class AddonManager(Mapping[str, Addon]):
"""This class facilitates loading and storing :ref:`topics-addons`.
You can treat it like a read-only dictionary in which keys correspond to
@@ -160,108 +82,42 @@ class AddonManager(Mapping):
"""
- def __init__(self):
- self._addons = OrderedDict()
- self.configs = {}
- self._disable_on_add = []
+ def __init__(self) -> None:
+ self._addons: OrderedDict[str, Addon] = OrderedDict[str, Addon]()
+ self.configs: Dict[str, Dict[str, Any]] = {}
- def __getitem__(self, name):
+ def __getitem__(self, name: str) -> Addon:
return self._addons[name]
- def __delitem__(self, name):
- del self._addons[name]
- del self.configs[name]
-
- def __iter__(self):
+ def __iter__(self) -> Iterator[str]:
return iter(self._addons)
- def __len__(self):
+ def __len__(self) -> int:
return len(self._addons)
- def add(self, addon, config=None):
+ def add(self, addon: Any, config: Optional[Dict[str, Any]] = None):
"""Store an add-on.
- If ``addon`` is a string, it will be treated as add-on path and passed
- to :meth:`get_addon`. Otherwise, ``addon`` must be a Python object
- implementing or providing Scrapy's add-on interface. The interface
- will be enforced through ``zope.interface``'s ``verifyObject()``.
-
- If ``addon`` is a class, it will be instantiated. You can avoid this
- (for example if you have implemented the add-on callbacks as class
- methods) by declaring -- via ``zope.interface`` -- that your class
- directly *provides* ``scrapy.interfaces.IAddon``.
-
:param addon: The add-on object (or path) to be stored
- :type addon: Any Python object providing the add-on interface or ``str``
+ :type addon: Python object, class or ``str``
:param config: The add-on configuration dictionary
:type config: ``dict``
"""
- addon = self.get_addon(addon)
- if isclass(addon) and not IAddon.providedBy(addon):
+ if isinstance(addon, (type, str)):
+ addon = load_object(addon)
+ if isinstance(addon, type):
addon = addon()
- if not IAddon.providedBy(addon):
- zope.interface.alsoProvides(addon, IAddon)
- # zope.interface's exceptions are already quite helpful. Still, should
- # we catch them and log an error message?
- verifyObject(IAddon, addon)
name = addon.name
if name in self:
raise ValueError(f"Addon '{name}' already loaded")
self._addons[name] = addon
self.configs[name] = config or {}
- if name in self._disable_on_add:
- self.configs[name]["_enabled"] = False
- self._disable_on_add.remove(name)
-
- def remove(self, addon):
- """Remove an add-on.
-
- If ``addon`` is the name of a stored add-on, that add-on will be
- removed. Otherwise, you can use the argument in the same fashion as
- in :meth:`add`.
-
- :param addon: The add-on name, object, or path to be removed
- :type addon: Any Python object providing the add-on interface or ``str``
- """
- if addon in self:
- del self[addon]
- elif hasattr(addon, "name") and addon.name in self:
- del self[addon.name]
- else:
- try:
- del self[self.get_addon(addon).name]
- except NameError:
- raise KeyError
-
- @staticmethod
- def get_addon(path):
- """Get an add-on object by its Python or file path.
-
- ``path`` is assumed to be an import path of an add-on. If the object or
- module pointed to by ``path`` has an attribute named ``_addon`` that
- attribute will be assumed to be the add-on. :meth:`get_addon` will keep
- following ``_addon`` attributes until it finds an object that does not
- have an attribute named ``_addon``.
-
- :param path: Import path of an add-on
- :type path: ``str``
- """
- if isinstance(path, str):
- try:
- obj = load_module_or_object(path)
- except NameError:
- raise NameError(f"Could not find add-on '{path}'")
- else:
- obj = path
- if hasattr(obj, "_addon"):
- obj = AddonManager.get_addon(obj._addon)
- return obj
def load_settings(self, settings):
"""Load add-ons and configurations from settings object.
- This will invoke :meth:`get_addon` for every add-on path in the
+ This will load the addon for every add-on path in the
``ADDONS`` setting. For each of these add-ons, the configuration will be
read from the dictionary setting whose name matches the uppercase add-on
name.
@@ -271,141 +127,12 @@ class AddonManager(Mapping):
:type settings: :class:`~scrapy.settings.Settings`
"""
paths = build_component_list(settings["ADDONS"])
- addons = [self.get_addon(path) for path in paths]
+ addons = [load_object(path) for path in paths]
configs = [settings.getdict(addon.name.upper()) for addon in addons]
for a, c in zip(addons, configs):
self.add(a, c)
- def check_dependency_clashes(self) -> None:
- """Check for incompatibilities in add-on dependencies.
-
- Add-ons can provide information about their dependencies in their
- ``provides``, ``modifies`` and ``requires`` attributes. This method will
- raise an ``ImportError`` if
-
- * a component required by an add-on is not provided by any other add-on,
- or
- * a component modified by an add-on is not provided by any other add-on,
- or
- * the same component is provided by more than one add-on,
-
- and warn when a component required by an add-on is modified by any other
- add-on.
- """
- # Collect all active add-ons and the components they provide
- versions: Dict[str, Version] = {}
-
- def add_version(project_name, version):
- if project_name in versions:
- raise ImportError(
- f"Component {project_name} provided by multiple add-ons"
- )
- versions[project_name] = Version(version)
-
- for name in self:
- ver = self[name].version
- add_version(name, ver)
- for provides_name in getattr(self[name], "provides", []):
- add_version(provides_name, ver)
-
- # Collect all required and modified components
- def compile_attribute_dict(attribute_name):
- attrs = defaultdict(list)
- for name in self:
- for entry in getattr(self[name], attribute_name, []):
- attrs[entry].append(name)
- return attrs
-
- modified = compile_attribute_dict("modifies")
- required = compile_attribute_dict("requires")
-
- req_or_mod = set(required.keys()).union(modified.keys())
- for reqstr in req_or_mod:
- req = Requirement(reqstr)
- if req.name not in versions or versions[req.name] not in req.specifier:
- raise ImportError(
- f"Add-ons {required[reqstr] + modified[reqstr]} require"
- f" or modify missing component {reqstr}"
- )
-
- mod_and_req = set(required.keys()).intersection(modified.keys())
- for conflict in mod_and_req:
- warnings.warn(
- f"Component '{conflict}', required by add-ons {required[conflict]},"
- f" is modified by add-ons {modified[conflict]}"
- )
-
- def disable(self, addon):
- """Disable an add-on, i.e. prevent its callbacks from being called.
-
- If you disable an add-on before it is loaded, it will be disabled as
- soon as it is added to the :class:`AddonManager`.
-
- :param addon: Name of the add-on to be disabled
- :type addon: ``str``
- """
- if addon in self:
- self.configs[addon]["_enabled"] = False
- else:
- self._disable_on_add.append(addon)
-
- def enable(self, addon):
- """Re-enable a disabled add-on.
-
- Will raise ``ValueError`` if the add-on is neither already loaded nor
- marked for being disabled on adding.
-
- :param addon: Name of the add-on to be enabled
- :type addon: ``str``
- """
- if addon in self:
- self.configs[addon]["_enabled"] = True
- elif addon in self._disable_on_add:
- self._disable_on_add.remove(addon)
- else:
- raise ValueError("Add-ons need to be added before they can be enabled")
-
- @property
- def disabled(self):
- """Names of disabled add-ons"""
- return [
- a for a in self if not self.configs[a].get("_enabled", True)
- ] + self._disable_on_add
-
- @property
- def enabled(self):
- """Names of enabled add-ons"""
- return [a for a in self if self.configs[a].get("_enabled", True)]
-
- def _call_if_exists(self, obj, cbname, *args, **kwargs):
- if obj is None:
- return
- try:
- cb = getattr(obj, cbname)
- except AttributeError:
- return
- else:
- cb(*args, **kwargs)
-
- def _call_addon(self, addonname, cbname, *args, **kwargs):
- if self.configs[addonname].get("_enabled", True):
- self._call_if_exists(
- self[addonname], cbname, self.configs[addonname], *args, **kwargs
- )
-
- def update_addons(self):
- """Call ``update_addons()`` of all held add-ons.
-
- This will also call ``update_addons()`` of all add-ons that are added
- last minute during the ``update_addons()`` routine of other add-ons.
- """
- called_addons = set()
- while called_addons != set(self):
- for name in set(self).difference(called_addons):
- called_addons.add(name)
- self._call_addon(name, "update_addons", self)
-
- def update_settings(self, settings):
+ def update_settings(self, settings) -> None:
"""Call ``update_settings()`` of all held add-ons.
:param settings: The :class:`~scrapy.settings.Settings` object to be \
@@ -413,13 +140,13 @@ class AddonManager(Mapping):
:type settings: :class:`~scrapy.settings.Settings`
"""
for name in self:
- self._call_addon(name, "update_settings", settings)
+ self[name].update_settings(self.configs[name], settings)
- def check_configuration(self, crawler):
+ def check_configuration(self, crawler) -> None:
"""Call ``check_configuration()`` of all held add-ons.
:param crawler: the fully-initialized crawler
:type crawler: :class:`~scrapy.crawler.Crawler`
"""
for name in self:
- self._call_addon(name, "check_configuration", crawler)
+ self[name].check_configuration(self.configs[name], crawler)
diff --git a/scrapy/cmdline.py b/scrapy/cmdline.py
index 1b579f10e..efc9b36ea 100644
--- a/scrapy/cmdline.py
+++ b/scrapy/cmdline.py
@@ -6,7 +6,6 @@ import sys
from importlib.metadata import entry_points
import scrapy
-from scrapy.addons import AddonManager
from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
from scrapy.crawler import CrawlerProcess
from scrapy.exceptions import UsageError
@@ -131,8 +130,6 @@ def execute(argv=None, settings=None):
else:
settings["EDITOR"] = editor
- addons = AddonManager()
-
inproject = inside_project()
cmds = _get_commands_dict(settings, inproject)
cmdname = _pop_command_name(argv)
@@ -156,7 +153,7 @@ def execute(argv=None, settings=None):
opts, args = parser.parse_known_args(args=argv[1:])
_run_print_help(parser, cmd.process_options, args, opts)
- cmd.crawler_process = CrawlerProcess(settings, addons=addons)
+ cmd.crawler_process = CrawlerProcess(settings)
_run_print_help(parser, _run_command, cmd, args, opts)
sys.exit(cmd.exitcode)
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index 0c9861dbd..a0cb368ed 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -58,7 +58,6 @@ class Crawler:
spidercls: Type[Spider],
settings: Union[None, dict, Settings] = None,
init_reactor: bool = False,
- addons: Optional[AddonManager] = None,
):
if isinstance(spidercls, Spider):
raise ValueError("The spidercls argument must be a class, not an object")
@@ -70,10 +69,8 @@ class Crawler:
self.settings: Settings = settings.copy()
self.spidercls.update_settings(self.settings)
- self.addons: AddonManager = addons if addons is not None else AddonManager()
+ self.addons: AddonManager = AddonManager()
self.addons.load_settings(self.settings)
- self.addons.update_addons()
- self.addons.check_dependency_clashes()
self.addons.update_settings(self.settings)
self.signals: SignalManager = SignalManager(self)
@@ -200,15 +197,10 @@ class CrawlerRunner:
)
return loader_cls.from_settings(settings.frozencopy())
- def __init__(
- self,
- settings: Union[Dict[str, Any], Settings, None] = None,
- addons: Optional[AddonManager] = None,
- ):
+ def __init__(self, settings: Union[Dict[str, Any], Settings, None] = None):
if isinstance(settings, dict) or settings is None:
settings = Settings(settings)
self.settings = settings
- self.addons: Optional[AddonManager] = addons
self.spider_loader = self._get_spider_loader(settings)
self._crawlers: Set[Crawler] = set()
self._active: Set[defer.Deferred] = set()
@@ -292,7 +284,7 @@ class CrawlerRunner:
def _create_crawler(self, spidercls: Union[str, Type[Spider]]) -> Crawler:
if isinstance(spidercls, str):
spidercls = self.spider_loader.load(spidercls)
- return Crawler(spidercls, self.settings, addons=self.addons)
+ return Crawler(spidercls, self.settings)
def stop(self):
"""
@@ -338,13 +330,8 @@ class CrawlerProcess(CrawlerRunner):
process. See :ref:`run-from-script` for an example.
"""
- def __init__(
- self,
- settings=None,
- install_root_handler: bool = True,
- addons: Optional[AddonManager] = None,
- ):
- super().__init__(settings, addons)
+ def __init__(self, settings=None, install_root_handler=True):
+ super().__init__(settings)
configure_logging(self.settings, install_root_handler)
log_scrapy_info(self.settings)
self._initialized_reactor = False
diff --git a/scrapy/interfaces.py b/scrapy/interfaces.py
index b8aa77ced..9a2c5f170 100644
--- a/scrapy/interfaces.py
+++ b/scrapy/interfaces.py
@@ -1,4 +1,4 @@
-from zope.interface import Attribute, Interface
+from zope.interface import Interface
class ISpiderLoader(Interface):
@@ -15,22 +15,3 @@ class ISpiderLoader(Interface):
def find_by_request(request):
"""Return the list of spiders names that can handle the given request"""
-
-
-class IAddon(Interface):
- """Scrapy add-on"""
-
- name = Attribute("""Add-on name""")
- version = Attribute("""Add-on version string (PEP440)""")
-
- # XXX: Can methods be declared optional? I.e., can I enforce the signature
- # but not the existence of a method?
-
- # def update_addons(config, addons):
- # """Enables and configures other add-ons"""
-
- # def update_settings(config, settings):
- # """Modifies `settings` to enable and configure required components"""
-
- # def check_configuration(config, crawler):
- # """Performs post-initialization checks on fully configured `crawler`"""
diff --git a/scrapy/utils/conf.py b/scrapy/utils/conf.py
index 8d1544c68..3ade1d105 100644
--- a/scrapy/utils/conf.py
+++ b/scrapy/utils/conf.py
@@ -99,18 +99,12 @@ def init_env(project="default", set_syspath=True):
sys.path.append(projdir)
-def config_from_filepath(sources):
- """Create a ConfigParser and read in the given `sources`, which can be
- either a filename or a list of filenames."""
- cfg = ConfigParser()
- cfg.read(sources)
- return cfg
-
-
def get_config(use_closest=True):
"""Get Scrapy config file as a ConfigParser"""
sources = get_sources(use_closest)
- return config_from_filepath(sources)
+ cfg = ConfigParser()
+ cfg.read(sources)
+ return cfg
def get_sources(use_closest=True) -> List[str]:
diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
index 4e23b01c3..b3c28da92 100644
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -67,7 +67,7 @@ def load_object(path: Union[str, Callable]) -> Any:
if callable(path):
return path
raise TypeError(
- "Unexpected argument type, expected string " f"or object, got: {type(path)}"
+ f"Unexpected argument type, expected string or object, got: {type(path)}"
)
try:
@@ -86,22 +86,6 @@ def load_object(path: Union[str, Callable]) -> Any:
return obj
-def load_module_or_object(path):
- """Load python module or (non-module) object from given path.
-
- Path can be both a Python or a file path.
- """
- try:
- return import_module(path)
- except ImportError:
- pass
- try:
- return load_object(path)
- except (ValueError, NameError, ImportError):
- pass
- raise NameError(f"Could not load '{path}'")
-
-
def walk_modules(path: str) -> List[ModuleType]:
"""Loads a module and all its submodules from the given module path and
returns them. If *any* module throws an exception while importing, that
diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py
index fe26e1708..97de8d25a 100644
--- a/scrapy/utils/test.py
+++ b/scrapy/utils/test.py
@@ -73,7 +73,6 @@ def get_crawler(
spidercls: Optional[Type[Spider]] = None,
settings_dict: Optional[Dict[str, Any]] = None,
prevent_warnings: bool = True,
- addons=None,
) -> Crawler:
"""Return an unconfigured Crawler object. If settings_dict is given, it
will be used to populate the crawler settings with a project level
@@ -87,7 +86,7 @@ def get_crawler(
if prevent_warnings:
settings["REQUEST_FINGERPRINTER_IMPLEMENTATION"] = "2.7"
settings.update(settings_dict or {})
- runner = CrawlerRunner(settings, addons)
+ runner = CrawlerRunner(settings)
return runner.create_crawler(spidercls or Spider)
diff --git a/tests/test_addons.py b/tests/test_addons.py
new file mode 100644
index 000000000..ecdc0426c
--- /dev/null
+++ b/tests/test_addons.py
@@ -0,0 +1,72 @@
+import unittest
+
+from scrapy.addons import Addon, AddonManager
+from scrapy.settings import BaseSettings
+
+
+class GoodAddon(object):
+ name = "GoodAddon"
+
+ def update_settings(self, config, settings):
+ pass
+
+ def check_configuration(self, config, crawler):
+ pass
+
+
+class AddonTest(unittest.TestCase):
+ def setUp(self):
+ class AddonWithAttributes(Addon):
+ name = "Test"
+
+ self.testaddon = AddonWithAttributes()
+
+ def test_export_config(self):
+ settings = BaseSettings()
+ self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}
+ self.testaddon.default_config = {"key": 55, "defaultkey": 100}
+ self.testaddon.export_config(
+ {"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings
+ )
+ self.assertEqual(settings["KEY"], 313)
+ self.assertEqual(settings["DEFAULTKEY"], 100)
+ self.assertEqual(settings["OTHERKEY"], True)
+ self.assertNotIn("MAPPED_key", settings)
+ self.assertNotIn("MAPPED_KEY", settings)
+ self.assertEqual(settings["MAPPING_WORKED"], 99)
+ self.assertEqual(settings.getpriority("KEY"), 15)
+
+ def test_update_settings(self):
+ settings = BaseSettings()
+ settings.set("KEY1", "default", priority="default")
+ settings.set("KEY2", "project", priority="project")
+ addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}
+ self.testaddon.update_settings(addon_config, settings)
+ self.assertEqual(settings["KEY1"], "addon")
+ self.assertEqual(settings["KEY2"], "project")
+ self.assertEqual(settings["KEY3"], "addon")
+
+
+class AddonManagerTest(unittest.TestCase):
+ def setUp(self):
+ self.manager = AddonManager()
+
+ def test_add(self):
+ manager = AddonManager()
+ manager.add("tests.test_addons.GoodAddon")
+ self.assertCountEqual(manager, ["GoodAddon"])
+ self.assertIsInstance(manager["GoodAddon"], GoodAddon)
+
+ def test_load_settings(self):
+ settings = BaseSettings()
+ settings.set(
+ "ADDONS",
+ {"tests.test_addons.GoodAddon": 0},
+ )
+ settings.set("GOODADDON", {"key": "val2"})
+ manager = AddonManager()
+ manager.load_settings(settings)
+ self.assertCountEqual(manager, ["GoodAddon"])
+ self.assertIsInstance(manager["GoodAddon"], GoodAddon)
+ self.assertCountEqual(manager.configs["GoodAddon"], ["key"])
+ self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")
diff --git a/tests/test_addons/__init__.py b/tests/test_addons/__init__.py
deleted file mode 100644
index fa3f706e5..000000000
--- a/tests/test_addons/__init__.py
+++ /dev/null
@@ -1,342 +0,0 @@
-import itertools
-import unittest
-import warnings
-from unittest import mock
-
-from zope.interface import directlyProvides
-from zope.interface.exceptions import BrokenImplementation, MultipleInvalid
-from zope.interface.verify import verifyObject
-
-from scrapy.addons import Addon, AddonManager
-from scrapy.crawler import Crawler
-from scrapy.interfaces import IAddon
-from scrapy.settings import BaseSettings
-
-from . import addonmod, addons
-
-
-class AddonTest(unittest.TestCase):
- def setUp(self):
- self.rawaddon = Addon()
-
- class AddonWithAttributes(Addon):
- name = "Test"
- version = "1.0"
-
- self.testaddon = AddonWithAttributes()
-
- def test_interface(self):
- # Raw Addon should fail exactly b/c name and version are not given
- self.assertFalse(hasattr(self.rawaddon, "name"))
- self.assertFalse(hasattr(self.rawaddon, "version"))
- self.assertRaises(MultipleInvalid, verifyObject, IAddon, self.rawaddon)
- verifyObject(IAddon, self.testaddon)
-
- def test_export_component(self):
- settings = BaseSettings(
- {"ITEM_PIPELINES": BaseSettings(), "DOWNLOAD_HANDLERS": BaseSettings()},
- "default",
- )
- self.testaddon.component_type = None
- self.testaddon.export_component({}, settings)
- self.assertEqual(len(settings["ITEM_PIPELINES"]), 0)
- self.testaddon.component_type = "ITEM_PIPELINES"
- self.testaddon.component = "test.component"
- self.testaddon.export_component({}, settings)
- self.assertCountEqual(settings["ITEM_PIPELINES"], ["test.component"])
- self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 0)
- self.testaddon.component_order = 313
- self.testaddon.export_component({}, settings)
- self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 313)
- self.testaddon.component_type = "DOWNLOAD_HANDLERS"
- self.testaddon.component_key = "http"
- self.testaddon.export_component({}, settings)
- self.assertEqual(settings["DOWNLOAD_HANDLERS"]["http"], "test.component")
-
- def test_export_basics(self):
- settings = BaseSettings()
- self.testaddon.basic_settings = {"TESTKEY": 313, "OTHERKEY": True}
- self.testaddon.export_basics(settings)
- self.assertEqual(settings["TESTKEY"], 313)
- self.assertEqual(settings["OTHERKEY"], True)
- self.assertEqual(settings.getpriority("TESTKEY"), 15)
-
- def test_export_config(self):
- settings = BaseSettings()
- self.testaddon.settings_prefix = None
- self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}
- self.testaddon.default_config = {"key": 55, "defaultkey": 100}
- self.testaddon.export_config(
- {"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings
- )
- self.assertEqual(settings["TEST_KEY"], 313)
- self.assertEqual(settings["TEST_DEFAULTKEY"], 100)
- self.assertEqual(settings["TEST_OTHERKEY"], True)
- self.assertNotIn("MAPPED_key", settings)
- self.assertNotIn("MAPPED_KEY", settings)
- self.assertEqual(settings["MAPPING_WORKED"], 99)
- self.assertEqual(settings.getpriority("TEST_KEY"), 15)
-
- self.testaddon.settings_prefix = "PREF"
- self.testaddon.export_config({"newkey": 99}, settings)
- self.assertEqual(settings["PREF_NEWKEY"], 99)
-
- with mock.patch.object(settings, "set") as mock_set:
- self.testaddon.settings_prefix = False
- self.testaddon.export_config({"thirdnewkey": 99}, settings)
- self.assertEqual(mock_set.call_count, 0)
-
- def test_update_settings(self):
- settings = BaseSettings()
- settings.set("TEST_KEY1", "default", priority="default")
- settings.set("TEST_KEY2", "project", priority="project")
- self.testaddon.settings_prefix = None
- self.testaddon.basic_settings = {"OTHERTEST_KEY": "addon"}
- addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}
- self.testaddon.update_settings(addon_config, settings)
- self.assertEqual(settings["OTHERTEST_KEY"], "addon")
- self.assertEqual(settings["TEST_KEY1"], "addon")
- self.assertEqual(settings["TEST_KEY2"], "project")
- self.assertEqual(settings["TEST_KEY3"], "addon")
-
-
-class AddonManagerTest(unittest.TestCase):
- def setUp(self):
- self.manager = AddonManager()
-
- def test_add(self):
- manager = AddonManager()
- manager.add(addonmod, {"key": "val1"})
- manager.add("tests.test_addons.addons.GoodAddon")
- self.assertCountEqual(manager, ["AddonModule", "GoodAddon"])
- self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)
- self.assertCountEqual(manager.configs["AddonModule"], ["key"])
- self.assertEqual(manager.configs["AddonModule"]["key"], "val1")
- self.assertRaises(ValueError, manager.add, addonmod)
-
- def test_add_dont_instantiate_providing_classes(self):
- class ProviderGoodAddon(addons.GoodAddon):
- pass
-
- directlyProvides(ProviderGoodAddon, IAddon)
- manager = AddonManager()
- manager.add(ProviderGoodAddon)
- self.assertIs(manager["GoodAddon"], ProviderGoodAddon)
-
- def test_add_verifies(self):
- brokenaddon = self.manager.get_addon("tests.test_addons.addons.BrokenAddon")
- self.assertRaises(
- BrokenImplementation,
- self.manager.add,
- brokenaddon,
- )
-
- def test_add_adds_missing_interface_declaration(self):
- class GoodAddonWithoutDeclaration(object):
- name = "GoodAddonWithoutDeclaration"
- version = "1.0"
-
- self.manager.add(GoodAddonWithoutDeclaration)
-
- def test_remove(self):
- manager = AddonManager()
-
- def test_gets_removed(removearg):
- manager.add(addonmod)
- self.assertIn("AddonModule", manager)
- manager.remove(removearg)
- self.assertNotIn("AddonModule", manager)
-
- test_gets_removed("AddonModule")
- test_gets_removed(addonmod)
- test_gets_removed("tests.test_addons.addonmod")
- self.assertRaises(KeyError, manager.remove, "nonexistent")
- self.assertRaises(KeyError, manager.remove, addons.GoodAddon())
-
- def test_get_addon(self):
- goodaddon = self.manager.get_addon("tests.test_addons.addons.GoodAddon")
- self.assertIs(goodaddon, addons.GoodAddon)
-
- loaded_addonmod = self.manager.get_addon("tests.test_addons.addonmod")
- self.assertIs(loaded_addonmod, addonmod)
-
- goodaddon = self.manager.get_addon("tests.test_addons.addons")
- self.assertIsInstance(goodaddon, addons.GoodAddon)
-
- self.assertRaises(NameError, self.manager.get_addon, "xy.n_onexistent")
-
- def test_get_addon_forward(self):
- class SomeCls(object):
- _addon = "tests.test_addons.addons.GoodAddon"
-
- self.assertIs(self.manager.get_addon(SomeCls()), addons.GoodAddon)
-
- def test_get_addon_nested(self):
- x = addons.GoodAddon("outer")
- x._addon = addons.GoodAddon("middle")
- x._addon._addon = addons.GoodAddon("inner")
- self.assertIs(self.manager.get_addon(x), x._addon._addon)
-
- def test_load_settings(self):
- settings = BaseSettings()
- settings.set(
- "ADDONS",
- {"tests.test_addons.addonmod": 0, "tests.test_addons.addons.GoodAddon": 0},
- )
- settings.set("ADDONMODULE", {"key": "val1"})
- settings.set("GOODADDON", {"key": "val2"})
- manager = AddonManager()
- manager.load_settings(settings)
- self.assertCountEqual(manager, ["GoodAddon", "AddonModule"])
- self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)
- self.assertCountEqual(manager.configs["GoodAddon"], ["key"])
- self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")
- self.assertEqual(manager["AddonModule"], addonmod)
- self.assertIn("key", manager.configs["AddonModule"])
- self.assertEqual(manager.configs["AddonModule"]["key"], "val1")
-
- def test_load_settings_order(self):
- # Get three addons named 0, 1, 2
- addonlist = [addons.GoodAddon(str(x)) for x in range(3)]
- # Test for every possible ordering
- for ordered_addons in itertools.permutations(addonlist):
- expected_order = [a.name for a in ordered_addons]
- settings = BaseSettings(
- {"ADDONS": {a: i for i, a in enumerate(ordered_addons)}}
- )
- manager = AddonManager()
- manager.load_settings(settings)
- self.assertEqual(list(manager.keys()), expected_order)
-
- def test_enabled_disabled(self):
- manager = AddonManager()
- manager.add(addons.GoodAddon("FirstAddon"))
- manager.add(addons.GoodAddon("SecondAddon"))
- self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))
- self.assertEqual(manager.disabled, [])
- manager.disable("FirstAddon")
- self.assertEqual(manager.enabled, ["SecondAddon"])
- self.assertEqual(manager.disabled, ["FirstAddon"])
- manager.enable("FirstAddon")
- self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))
- self.assertEqual(manager.disabled, [])
-
- def test_enable_before_add(self):
- manager = AddonManager()
- self.assertRaises(ValueError, manager.enable, "FirstAddon")
- manager.disable("FirstAddon")
- manager.enable("FirstAddon")
- manager.add(addons.GoodAddon("FirstAddon"))
- self.assertIn("FirstAddon", manager.enabled)
-
- def test_disable_before_add(self):
- manager = AddonManager()
- manager.disable("FirstAddon")
- manager.add(addons.GoodAddon("FirstAddon"))
- self.assertEqual(manager.disabled, ["FirstAddon"])
-
- def test_callbacks(self):
- first_addon = addons.GoodAddon("FirstAddon")
- second_addon = addons.GoodAddon("SecondAddon")
-
- manager = AddonManager()
- manager.add(first_addon, {"test": "first"})
- manager.add(second_addon, {"test": "second"})
- crawler = mock.create_autospec(Crawler)
- settings = BaseSettings()
-
- with mock.patch.object(
- first_addon, "update_addons"
- ) as ua_first, mock.patch.object(
- second_addon, "update_addons"
- ) as ua_second, mock.patch.object(
- first_addon, "update_settings"
- ) as us_first, mock.patch.object(
- second_addon, "update_settings"
- ) as us_second, mock.patch.object(
- first_addon, "check_configuration"
- ) as cc_first, mock.patch.object(
- second_addon, "check_configuration"
- ) as cc_second:
- manager.update_addons()
- ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)
- ua_second.assert_called_once_with(manager.configs["SecondAddon"], manager)
- manager.update_settings(settings)
- us_first.assert_called_once_with(manager.configs["FirstAddon"], settings)
- us_second.assert_called_once_with(manager.configs["SecondAddon"], settings)
- manager.check_configuration(crawler)
- cc_first.assert_called_once_with(manager.configs["FirstAddon"], crawler)
- cc_second.assert_called_once_with(manager.configs["SecondAddon"], crawler)
- self.assertEqual(ua_first.call_count, 1)
- self.assertEqual(ua_second.call_count, 1)
- self.assertEqual(us_first.call_count, 1)
- self.assertEqual(us_second.call_count, 1)
-
- us_first.reset_mock()
- us_second.reset_mock()
- manager.disable("FirstAddon")
- manager.update_settings(settings)
- self.assertEqual(us_first.call_count, 0)
- manager.enable("FirstAddon")
- manager.update_settings(settings)
- self.assertEqual(us_first.call_count, 1)
- self.assertEqual(us_second.call_count, 2)
-
- # This will become relevant when we let spiders implement the add-on
- # interface and should be replaced with a test where
- # AddonManager.spidercls = None then.
- manager._call_if_exists(None, "irrelevant")
-
- def test_update_addons_last_minute_add(self):
- class AddedAddon(addons.GoodAddon):
- name = "AddedAddon"
-
- class FirstAddon(addons.GoodAddon):
- name = "FirstAddon"
-
- def update_addons(self, config, addons):
- addons.add(AddedAddon())
-
- manager = AddonManager()
- first_addon = FirstAddon()
- with mock.patch.object(
- first_addon, "update_addons", wraps=first_addon.update_addons
- ) as ua_first, mock.patch.object(AddedAddon, "update_addons") as ua_added:
- manager.add(first_addon, {"non-empty": "dict"})
- manager.update_addons()
- self.assertCountEqual(manager, ["FirstAddon", "AddedAddon"])
- ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)
- ua_added.assert_called_once_with(manager.configs["AddedAddon"], manager)
-
- def test_check_dependency_clashes_attributes(self):
- provides = addons.GoodAddon("ProvidesAddon")
- provides.provides = ("test",)
- provides2 = addons.GoodAddon("ProvidesAddon2")
- provides2.provides = ("test",)
- requires = addons.GoodAddon("RequiresAddon")
- requires.requires = ("test",)
- requires_name = addons.GoodAddon("RequiresNameAddon")
- requires_name.requires = ("ProvidesAddon",)
- requires_newer = addons.GoodAddon("RequiresNewerAddon")
- requires_newer.requires = ("test>=2.0",)
- modifies = addons.GoodAddon("ModifiesAddon")
- modifies.modifies = ("test",)
-
- def check_with(*addons):
- manager = AddonManager()
- for a in addons:
- manager.add(a)
- return manager.check_dependency_clashes()
-
- self.assertRaises(ImportError, check_with, requires)
- self.assertRaises(ImportError, check_with, modifies)
- self.assertRaises(ImportError, check_with, provides, provides2)
- self.assertRaises(ImportError, check_with, provides, requires_newer)
- with warnings.catch_warnings(record=True) as w:
- check_with(provides, modifies)
- check_with(provides)
- check_with(provides, requires)
- check_with(provides, requires_name)
- self.assertEqual(len(w), 0)
- check_with(requires, provides, modifies)
- self.assertEqual(len(w), 1)
diff --git a/tests/test_addons/addonmod.py b/tests/test_addons/addonmod.py
deleted file mode 100644
index 092c3c0eb..000000000
--- a/tests/test_addons/addonmod.py
+++ /dev/null
@@ -1,16 +0,0 @@
-from zope.interface import moduleProvides
-
-from scrapy.interfaces import IAddon
-
-moduleProvides(IAddon)
-
-name = "AddonModule"
-version = "1.0"
-
-
-def update_settings(config, settings):
- pass
-
-
-def check_configuration(config, crawler):
- pass
diff --git a/tests/test_addons/addons.py b/tests/test_addons/addons.py
deleted file mode 100644
index d878f37ea..000000000
--- a/tests/test_addons/addons.py
+++ /dev/null
@@ -1,33 +0,0 @@
-from zope.interface import implementer
-
-from scrapy.interfaces import IAddon
-
-
-@implementer(IAddon)
-class GoodAddon(object):
- name = "GoodAddon"
- version = "1.0"
-
- def __init__(self, name=None, version=None):
- if name is not None:
- self.name = name
- if version is not None:
- self.version = version
-
- def update_addons(self, config, addons):
- pass
-
- def update_settings(self, config, settings):
- pass
-
- def check_configuration(self, config, crawler):
- pass
-
-
-@implementer(IAddon)
-class BrokenAddon(object):
- name = "BrokenAddon"
- # No version
-
-
-_addon = GoodAddon()
diff --git a/tests/test_crawl.py b/tests/test_crawl.py
index 920e5f4ae..d844a645f 100644
--- a/tests/test_crawl.py
+++ b/tests/test_crawl.py
@@ -412,7 +412,6 @@ with multiples lines
def test_abort_on_addon_failed_check(self):
class FailedCheckAddon(Addon):
name = "FailedCheckAddon"
- version = "1.0"
def check_configuration(self, config, crawler):
raise ValueError
diff --git a/tests/test_crawler.py b/tests/test_crawler.py
index ed34d9e58..d54a2cb7e 100644
--- a/tests/test_crawler.py
+++ b/tests/test_crawler.py
@@ -15,7 +15,6 @@ from twisted.trial import unittest
from w3lib import __version__ as w3lib_version
import scrapy
-from scrapy.addons import Addon, AddonManager
from scrapy.crawler import Crawler, CrawlerProcess, CrawlerRunner
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.extensions import telnet
@@ -56,32 +55,6 @@ class CrawlerTestCase(BaseCrawlerTest):
self.assertFalse(settings.frozen)
self.assertTrue(crawler.settings.frozen)
- def test_populate_addons_settings(self):
- class TestAddon(Addon):
- name = "TestAddon"
- version = "1.0"
-
- addonconfig = {"TEST1": "addon", "TEST2": "addon", "TEST3": "addon"}
-
- class TestAddon2(Addon):
- name = "testAddon2"
- version = "1.0"
-
- addonconfig2 = {"TEST": "addon2"}
-
- settings = Settings()
- settings.set("TESTADDON_TEST1", "project", priority="project")
- settings.set("TESTADDON_TEST2", "default", priority="default")
- addonmgr = AddonManager()
- addonmgr.add(TestAddon(), addonconfig)
- addonmgr.add(TestAddon2(), addonconfig2)
- crawler = Crawler(DefaultSpider, settings, addons=addonmgr)
-
- self.assertEqual(crawler.settings["TESTADDON_TEST1"], "project")
- self.assertEqual(crawler.settings["TESTADDON_TEST2"], "addon")
- self.assertEqual(crawler.settings["TESTADDON_TEST3"], "addon")
- self.assertEqual(crawler.settings["TESTADDON2_TEST"], "addon2")
-
def test_crawler_accepts_dict(self):
crawler = get_crawler(DefaultSpider, {"foo": "bar"})
self.assertEqual(crawler.settings["foo"], "bar")
diff --git a/tests/test_utils_misc/__init__.py b/tests/test_utils_misc/__init__.py
index 7932ca04c..69793ee75 100644
--- a/tests/test_utils_misc/__init__.py
+++ b/tests/test_utils_misc/__init__.py
@@ -8,7 +8,6 @@ from scrapy.item import Field, Item
from scrapy.utils.misc import (
arg_to_iter,
create_instance,
- load_module_or_object,
load_object,
rel_has_nofollow,
set_environ,
@@ -36,12 +35,6 @@ class UtilsMiscTestCase(unittest.TestCase):
self.assertRaises(NameError, load_object, "scrapy.utils.misc.load_object999")
self.assertRaises(TypeError, load_object, {})
- def test_load_module_or_object(self):
- testmod = load_module_or_object(__name__ + ".testmod")
- self.assertTrue(hasattr(testmod, "TESTVAR"))
- obj = load_object("scrapy.utils.misc.load_object")
- self.assertIs(obj, load_object)
-
def test_walk_modules(self):
mods = walk_modules("tests.test_utils_misc.test_walk_modules")
expected = [
diff --git a/tests/test_utils_misc/testmod.py b/tests/test_utils_misc/testmod.py
deleted file mode 100644
index eb540335f..000000000
--- a/tests/test_utils_misc/testmod.py
+++ /dev/null
@@ -1 +0,0 @@
-TESTVAR = True