Scrapy, a fast high-level web crawling & scraping framework for Python.
Remove unneeded code.
Andrey Rakhmatulline7124447f7e8
diff --git a/docs/topics/addons.rst b/docs/topics/addons.rstindex 5d1a4f753..6a9590f33 100644--- a/docs/topics/addons.rst+++ b/docs/topics/addons.rst@@ -15,12 +15,13 @@ Activating and configuring add-ons==================================Add-ons and their configuration live in Scrapy's-:class:`~scrapy.addons.AddonManager`. During Scrapy's start-up process, and-only then, the add-on manager will read a list of enabled add-ons and their-configurations from your ``ADDONS`` setting.+:class:`~scrapy.addons.AddonManager`. During a :class:`~scrapy.crawler.Crawler`+initialization the add-on manager will read a list of enabled add-ons from your+``ADDONS`` setting and their optional configuration from the respective+settings.The ``ADDONS`` setting is a dict in which every key is an addon class or its-import path and the vaoue is its priority.+import path and the value is its priority.The configuration of an add-on, if necessary at all, is stored as a dictionarysetting whose name is the uppercase add-on name.@@ -37,66 +38,17 @@ configuration) are enabled/configured in a project's ``settings.py``::'some_config': True,}-Enabling and configuring add-ons within Python code------------------------------------------------------The :class:`~scrapy.addons.AddonManager` will only read from Scrapy's settings-*at the beginning* of Scrapy's start-up process.-Afterwards, i.e. as soon as the :class:`~scrapy.addons.AddonManager` is-populated, changing the ``ADDONS`` setting or any of the add-on-configuration dictionary settings will have no effect.--If you want to enable, disable, or configure add-ons in Python code, for example-when writing your own add-on, you will have to use the-:class:`~scrapy.addons.AddonManager`. You can access the add-on manager through-either ``crawler.addons`` or, if you are writing an add-on, through the-``addons`` argument of the :meth:`update_addons` callback. The add-on manager-provides many useful methods and attributes to facilitate interacting with the-add-ons framework, e.g.:--* an :meth:`~scrapy.addons.AddonManager.add` method to load add-ons,-* the :attr:`~scrapy.addons.AddonManager.enabled` list of enabled add-ons,-* :meth:`~scrapy.addons.AddonManager.enable` and- :meth:`~scrapy.addons.AddonManager.disable` methods,-* the :attr:`~scrapy.addons.AddonManager.configs` dictionary which holds the- configuration of all add-ons.--In this example, we ensure that the ``httpcache`` add-on is loaded, and that-its ``expiration_secs`` configuration is set to ``60``::-- # addons is an instance of AddonManager- if 'httpcache' not in addons:- addons.add('httpcache', {'expiration_secs': 60})- else:- addons.configs['httpcache']['expiration_secs'] = 60-Writing your own add-ons========================-Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*.-The interface is enforced through ``zope.interface``. This leaves the choice of-Python object up the developer. Examples:--* for a small pipeline, the add-on interface could be implemented in the same- class that also implements the ``open/close_spider`` and ``process_item``- callbacks,-* for larger add-ons, or for clearer structure, the interface could be provided- by a stand-alone module.--The absolute minimum interface consists of two attributes:+Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*:.. attribute:: namestring with add-on name-.. attribute:: version-- version string (PEP-404, e.g. ``'1.0.1'``)--Of course, stating just these two attributes will not get you very far. Add-ons-can provide three callback methods that are called at various stages before the-crawling process:+ :type: ``str``.. method:: update_settings(config, settings)@@ -124,75 +76,18 @@ crawling process::param crawler: Fully initialized Scrapy crawler:type crawler: :class:`~scrapy.crawler.Crawler`-.. method:: update_addons(config, addons)-- This method is called immediately before :meth:`update_settings`, and should- be used to enable and configure other *add-ons* only.-- Add-ons that are added to the :class:`~scrapy.addons.AddonManager` during- this callback will also have their :meth:`update_addons` method called.-- :param config: Configuration of this add-on- :type config: ``dict``-- :param addons: Add-on manager holding all loaded add-ons- :type addons: :class:`~scrapy.addons.AddonManager`--Additionally, add-ons may (and should, where appropriate) provide one or more-attributes that can be used for limited automated detection of possible-dependency clashes:--.. attribute:: requires-- list of built-in or custom components needed by this add-on, as strings.--.. attribute:: modifies-- list of built-in or custom components whose functionality is affected or- replaced by this add-on (a custom HTTP cache should list ``httpcache`` here)--.. attribute:: provides-- list of components provided by this add-on (e.g. ``mongodb`` for an- extension that provides generic read/write access to a MongoDB database)--The entries in the :attr:`requires` and :attr:`modifies` attributes can be add-on-names or components from other add-ons' :attr:`provides` attribute. You can-specify :pep:`440`-style information about required versions. Examples::-- requires = ['httpcache']- requires = ['otheraddon >= 2.0', 'yetanotheraddon']--The Python object or module that is pointed to by an add-on path (e.g. given in-the ``ADDONS`` setting, or given to-:meth:`~scrapy.addons.AddonManager.add`) does not necessarily have to be an-add-on. Instead, it can provide an ``_addon`` attribute. This attribute can be-either an add-on or another add-on path.-Add-on base class=================Scrapy comes with a built-in base class for add-ons which provides some-convenience functionality:--* basic settings can be exported via :meth:`~scrapy.addons.Addon.export_basics`,- configurable via :attr:`~scrapy.addons.Addon.basic_settings`.-* a single component (e.g. an item pipeline or a downloader middleware) can be- inserted into Scrapy's settings via- :meth:`~scrapy.addons.Addon.export_component`, configurable via- :attr:`~scrapy.addons.Addon.component_type`,- :attr:`~scrapy.addons.Addon.component_key`,- :attr:`~scrapy.addons.Addon.component`, and the ``order`` key in- :attr:`~scrapy.addons.Addon.default_config`.-* the add-on configuration can be exposed into Scrapy's settings via- :meth:`~scrapy.addons.Addon.export_config`, configurable via- :attr:`~scrapy.addons.Addon.default_config`,- :attr:`~scrapy.addons.Addon.config_mapping`, and- :attr:`~scrapy.addons.Addon.settings_prefix`.+convenience functionality: the add-on configuration can be exposed into+Scrapy's settings via :meth:`~scrapy.addons.Addon.export_config`, configurable+via :attr:`~scrapy.addons.Addon.default_config` and+:attr:`~scrapy.addons.Addon.config_mapping`.By default, the base add-on class will expose the add-on configuration into-Scrapy's settings namespace, in caps and with the add-on name prepended. It is+Scrapy's settings namespace, in upper case. It iseasy to write your own functionality while still being able to use theconvenience functions by overwriting:meth:`~scrapy.addons.Addon.update_settings`.@@ -213,13 +108,11 @@ Set some basic configuration using the :class:`Addon` base class::class MyAddon(Addon):name = 'myaddon'- version = '1.0'- component = 'path.to.mypipeline'- component_type = 'ITEM_PIPELINES'- component_order = 200- basic_settings = {- 'DNSCACHE_ENABLED': False,- }++ def update_settings(self, config, settings):+ super().update_settings(settings)+ settings["ITEM_PIPELINES"]["path.to.mypipeline"] = 200+ settings["DNSCACHE_ENABLED"] = TrueCheck dependencies::@@ -227,78 +120,22 @@ Check dependencies::class MyAddon(Addon):name = 'myaddon'- version = '1.0'def update_settings(self, config, settings):try:import botoexcept ImportError:raise RuntimeError("myaddon requires the boto library")- self.export_config(config, settings)--Enable a component that lives relative to the add-on (see-:ref:`topics-api-settings`)::-- from scrapy.addons import Addon-- class MyAddon(Addon):- name = 'myaddon'- version = '1.0'- component = __name__ + '.downloadermw.coolmw'- component_type = 'DOWNLOADER_MIDDLEWARES'- component_order = 900--Instantiate components ad hoc::-- from path.to.my.pipelines import MySQLPipeline-- class MyAddon(object):- name = 'myaddon'- version = '1.0'-- def update_settings(self, config, settings):- mysqlpl = MySQLPipeline(password=config['password'])- settings.set(- 'ITEM_PIPELINES',- {mysqlpl: 200},- priority='addon',- )--Provide add-on interface along component interface::-- class MyPipeline(object):- name = 'mypipeline'- version = '1.0'-- def process_item(self, item, spider):- # Do some processing here- return item-- def update_settings(self, config, settings):- settings.set(- 'ITEM_PIPELINES',- {self: 200},- priority='addon',- )--Enable another addon (see :ref:`topics-api-addonmanager`)::-- class MyAddon(object):- name = 'myaddon'- version = '1.0'-- def update_addons(self, config, addons):- if 'httpcache' not in addons.enabled:- addons.add('httpcache', {'expiration_secs': 60})+ super().update_settings(settings)Check configuration of fully initialized crawler (see:ref:`topics-api-crawler`)::class MyAddon(object):name = 'myaddon'- version = '1.0'def update_settings(self, config, settings):+ super().update_settings(settings)settings.set('DNSCACHE_ENABLED', False, priority='addon')def check_configuration(self, config, crawler):@@ -306,43 +143,3 @@ Check configuration of fully initialized crawler (see# The spider, some other add-on, or the user messed with the# DNS cache settingraise ValueError("myaddon is incompatible with DNS cache")--Provide add-on interface through a module:--.. code-block:: python-- name = "AddonModule"- version = "1.0"--- class MyPipeline(object):- ...--- class MyDownloaderMiddleware(object):- ...--- def update_settings(config, settings):- settings.set(- "ITEM_PIPELINES",- {MyPipeline(): 200},- priority="addon",- )- settings.set(- "DOWNLOADER_MIDDLEWARES",- {MyDownloaderMiddleware(): 800},- priority="addon",- )--Forward to other add-ons depending on Python version::-- # This could be a Python module, say project/pipelines/mypipeline.py, but- # could also be done inside a class, etc.- import six-- if six.PY3:- # We're running Python 3- _addon = 'path.to.addon'- else:- _addon = 'path.to.other.addon'diff --git a/scrapy/addons.py b/scrapy/addons.pyindex aced6092a..a54086fda 100644--- a/scrapy/addons.py+++ b/scrapy/addons.py@@ -1,108 +1,27 @@-import warnings-from collections import OrderedDict, defaultdictfrom collections.abc import Mapping-from inspect import isclass-from typing import Dict+from typing import Any, Dict, Iterator, Optional, OrderedDict-import zope.interface-from packaging.requirements import Requirement-from packaging.version import Version-from zope.interface.verify import verifyObject--from scrapy.interfaces import IAddonfrom scrapy.utils.conf import build_component_list-from scrapy.utils.misc import load_module_or_object+from scrapy.utils.misc import load_object[email protected](IAddon)class Addon(object):- basic_settings = None- """``dict`` of settings that will be exported via :meth:`export_basics`."""+ name: strdefault_config = None"""``dict`` with default configuration."""config_mapping = None"""``dict`` with mappings from config names to setting names. The given- setting names will be taken as given, i.e. they will be neither prefixed- nor uppercased.- """-- component_type = None- """Component setting into which to export via :meth:`export_component`. Can- be any of the dictionary-like component setting names. If ``None``,- :meth:`export_component` will do nothing.- """-- component_key = None- """Key to be used in the component dictionary setting when exporting via- :meth:`export_component`. This is only useful for the settings that have- no order, e.g. ``DOWNLOAD_HANDLERS`` or ``FEED_EXPORTERS``.- """-- component_order = 0- """Component order to use when not given in the add-on configuration. Has- no effect for component types that use :attr:`component_key`.+ setting names will be taken as given, not uppercased."""- component = None- """Component to be inserted via :meth:`export_component`. This can be- anything that can be used in the dictionary-like component settings, i.e.- a class path or a class. If ``None``, it is assumed that the add-on itself- also provides the component interface, and ``self`` will be used.- """-- settings_prefix = None- """Prefix with which the add-on configuration will be exported into the- global settings namespace via :meth:`export_config`. If ``None``,- :attr:`name` will be used. If ``False``, no configuration will be exported.- """-- def export_component(self, config, settings):- """Export the component in :attr:`component` into the dictionary-like- component setting derived from :attr:`component_type`.-- Where applicable, the order parameter of the component (i.e. the- dictionary value) will be retrieved from the ``order`` add-on- configuration value.-- :param config: Add-on configuration from which to read component order- :type config: ``dict``-- :param settings: Settings object into which to export component- :type settings: :class:`~scrapy.settings.Settings`- """- if self.component_type:- comp = self.component or self- if self.component_key:- # e.g. for DOWNLOAD_HANDLERS: {'http': 'myclass'}- k = self.component_key- v = comp- else:- # e.g. for DOWNLOADER_MIDDLEWARES: {'myclass': 100}- k = comp- v = config.get("order", self.component_order)- settings[self.component_type].update({k: v}, "addon")-- def export_basics(self, settings):- """Export the :attr:`basic_settings` attribute into the settings object.-- All settings will be exported with ``addon`` priority (see- :ref:`topics-api-settings`).-- :param settings: Settings object into which to expose the basic settings- :type settings: :class:`~scrapy.settings.Settings`- """- for setting, value in (self.basic_settings or {}).items():- settings.set(setting, value, "addon")-def export_config(self, config, settings):- """Export the add-on configuration, all keys in caps and with- :attr:`settings_prefix` or :attr:`name` prepended, into the settings+ """Export the add-on configuration, all keys in caps, into the settingsobject.For example, the add-on configuration ``{'key': 'value'}`` will export- the setting ``ADDONNAME_KEY`` with a value of ``value``. All settings+ the setting ``KEY`` with a value of ``value``. All settingswill be exported with ``addon`` priority (see:ref:`topics-api-settings`).@@ -112,11 +31,8 @@ class Addon(object)::param settings: Settings object into which to export the configuration:type settings: :class:`~scrapy.settings.Settings`"""- if self.settings_prefix is False:- returnconf = self.default_config or {}conf.update(config)- prefix = self.settings_prefix or self.name# Since default exported config is case-insensitive (everything will be# uppercased), make mapped config case-insensitive as wellconf_mapping = {k.lower(): v for k, v in (self.config_mapping or {}).items()}@@ -124,14 +40,11 @@ class Addon(object):if key.lower() in conf_mapping:key = conf_mapping[key.lower()]else:- key = (prefix + "_" + key).upper()+ key = key.upper()settings.set(key, val, "addon")def update_settings(self, config, settings):- """Export both the basic settings and the add-on configuration. I.e.,- call :meth:`export_basics` and :meth:`export_config`.-- For more advanced add-ons, you may want to override this callback.+ """Modifiy `settings` to enable and configure required components.:param config: Add-on configuration:type config: ``dict``@@ -139,12 +52,21 @@ class Addon(object)::param settings: Crawler settings object:type settings: :class:`~scrapy.settings.Settings`"""- self.export_component(config, settings)- self.export_basics(settings)self.export_config(config, settings)+ def check_configuration(self, config, crawler):+ """Perform post-initialization checks on fully configured `crawler`.++ :param config: Add-on configuration+ :type config: ``dict``++ :param crawler: the fully-initialized crawler+ :type crawler: :class:`~scrapy.crawler.Crawler`+ """+ pass+-class AddonManager(Mapping):+class AddonManager(Mapping[str, Addon]):"""This class facilitates loading and storing :ref:`topics-addons`.You can treat it like a read-only dictionary in which keys correspond to@@ -160,108 +82,42 @@ class AddonManager(Mapping):"""- def __init__(self):- self._addons = OrderedDict()- self.configs = {}- self._disable_on_add = []+ def __init__(self) -> None:+ self._addons: OrderedDict[str, Addon] = OrderedDict[str, Addon]()+ self.configs: Dict[str, Dict[str, Any]] = {}- def __getitem__(self, name):+ def __getitem__(self, name: str) -> Addon:return self._addons[name]- def __delitem__(self, name):- del self._addons[name]- del self.configs[name]-- def __iter__(self):+ def __iter__(self) -> Iterator[str]:return iter(self._addons)- def __len__(self):+ def __len__(self) -> int:return len(self._addons)- def add(self, addon, config=None):+ def add(self, addon: Any, config: Optional[Dict[str, Any]] = None):"""Store an add-on.- If ``addon`` is a string, it will be treated as add-on path and passed- to :meth:`get_addon`. Otherwise, ``addon`` must be a Python object- implementing or providing Scrapy's add-on interface. The interface- will be enforced through ``zope.interface``'s ``verifyObject()``.-- If ``addon`` is a class, it will be instantiated. You can avoid this- (for example if you have implemented the add-on callbacks as class- methods) by declaring -- via ``zope.interface`` -- that your class- directly *provides* ``scrapy.interfaces.IAddon``.-:param addon: The add-on object (or path) to be stored- :type addon: Any Python object providing the add-on interface or ``str``+ :type addon: Python object, class or ``str``:param config: The add-on configuration dictionary:type config: ``dict``"""- addon = self.get_addon(addon)- if isclass(addon) and not IAddon.providedBy(addon):+ if isinstance(addon, (type, str)):+ addon = load_object(addon)+ if isinstance(addon, type):addon = addon()- if not IAddon.providedBy(addon):- zope.interface.alsoProvides(addon, IAddon)- # zope.interface's exceptions are already quite helpful. Still, should- # we catch them and log an error message?- verifyObject(IAddon, addon)name = addon.nameif name in self:raise ValueError(f"Addon '{name}' already loaded")self._addons[name] = addonself.configs[name] = config or {}- if name in self._disable_on_add:- self.configs[name]["_enabled"] = False- self._disable_on_add.remove(name)-- def remove(self, addon):- """Remove an add-on.-- If ``addon`` is the name of a stored add-on, that add-on will be- removed. Otherwise, you can use the argument in the same fashion as- in :meth:`add`.-- :param addon: The add-on name, object, or path to be removed- :type addon: Any Python object providing the add-on interface or ``str``- """- if addon in self:- del self[addon]- elif hasattr(addon, "name") and addon.name in self:- del self[addon.name]- else:- try:- del self[self.get_addon(addon).name]- except NameError:- raise KeyError-- @staticmethod- def get_addon(path):- """Get an add-on object by its Python or file path.-- ``path`` is assumed to be an import path of an add-on. If the object or- module pointed to by ``path`` has an attribute named ``_addon`` that- attribute will be assumed to be the add-on. :meth:`get_addon` will keep- following ``_addon`` attributes until it finds an object that does not- have an attribute named ``_addon``.-- :param path: Import path of an add-on- :type path: ``str``- """- if isinstance(path, str):- try:- obj = load_module_or_object(path)- except NameError:- raise NameError(f"Could not find add-on '{path}'")- else:- obj = path- if hasattr(obj, "_addon"):- obj = AddonManager.get_addon(obj._addon)- return objdef load_settings(self, settings):"""Load add-ons and configurations from settings object.- This will invoke :meth:`get_addon` for every add-on path in the+ This will load the addon for every add-on path in the``ADDONS`` setting. For each of these add-ons, the configuration will beread from the dictionary setting whose name matches the uppercase add-onname.@@ -271,141 +127,12 @@ class AddonManager(Mapping)::type settings: :class:`~scrapy.settings.Settings`"""paths = build_component_list(settings["ADDONS"])- addons = [self.get_addon(path) for path in paths]+ addons = [load_object(path) for path in paths]configs = [settings.getdict(addon.name.upper()) for addon in addons]for a, c in zip(addons, configs):self.add(a, c)- def check_dependency_clashes(self) -> None:- """Check for incompatibilities in add-on dependencies.-- Add-ons can provide information about their dependencies in their- ``provides``, ``modifies`` and ``requires`` attributes. This method will- raise an ``ImportError`` if-- * a component required by an add-on is not provided by any other add-on,- or- * a component modified by an add-on is not provided by any other add-on,- or- * the same component is provided by more than one add-on,-- and warn when a component required by an add-on is modified by any other- add-on.- """- # Collect all active add-ons and the components they provide- versions: Dict[str, Version] = {}-- def add_version(project_name, version):- if project_name in versions:- raise ImportError(- f"Component {project_name} provided by multiple add-ons"- )- versions[project_name] = Version(version)-- for name in self:- ver = self[name].version- add_version(name, ver)- for provides_name in getattr(self[name], "provides", []):- add_version(provides_name, ver)-- # Collect all required and modified components- def compile_attribute_dict(attribute_name):- attrs = defaultdict(list)- for name in self:- for entry in getattr(self[name], attribute_name, []):- attrs[entry].append(name)- return attrs-- modified = compile_attribute_dict("modifies")- required = compile_attribute_dict("requires")-- req_or_mod = set(required.keys()).union(modified.keys())- for reqstr in req_or_mod:- req = Requirement(reqstr)- if req.name not in versions or versions[req.name] not in req.specifier:- raise ImportError(- f"Add-ons {required[reqstr] + modified[reqstr]} require"- f" or modify missing component {reqstr}"- )-- mod_and_req = set(required.keys()).intersection(modified.keys())- for conflict in mod_and_req:- warnings.warn(- f"Component '{conflict}', required by add-ons {required[conflict]},"- f" is modified by add-ons {modified[conflict]}"- )-- def disable(self, addon):- """Disable an add-on, i.e. prevent its callbacks from being called.-- If you disable an add-on before it is loaded, it will be disabled as- soon as it is added to the :class:`AddonManager`.-- :param addon: Name of the add-on to be disabled- :type addon: ``str``- """- if addon in self:- self.configs[addon]["_enabled"] = False- else:- self._disable_on_add.append(addon)-- def enable(self, addon):- """Re-enable a disabled add-on.-- Will raise ``ValueError`` if the add-on is neither already loaded nor- marked for being disabled on adding.-- :param addon: Name of the add-on to be enabled- :type addon: ``str``- """- if addon in self:- self.configs[addon]["_enabled"] = True- elif addon in self._disable_on_add:- self._disable_on_add.remove(addon)- else:- raise ValueError("Add-ons need to be added before they can be enabled")-- @property- def disabled(self):- """Names of disabled add-ons"""- return [- a for a in self if not self.configs[a].get("_enabled", True)- ] + self._disable_on_add-- @property- def enabled(self):- """Names of enabled add-ons"""- return [a for a in self if self.configs[a].get("_enabled", True)]-- def _call_if_exists(self, obj, cbname, *args, **kwargs):- if obj is None:- return- try:- cb = getattr(obj, cbname)- except AttributeError:- return- else:- cb(*args, **kwargs)-- def _call_addon(self, addonname, cbname, *args, **kwargs):- if self.configs[addonname].get("_enabled", True):- self._call_if_exists(- self[addonname], cbname, self.configs[addonname], *args, **kwargs- )-- def update_addons(self):- """Call ``update_addons()`` of all held add-ons.-- This will also call ``update_addons()`` of all add-ons that are added- last minute during the ``update_addons()`` routine of other add-ons.- """- called_addons = set()- while called_addons != set(self):- for name in set(self).difference(called_addons):- called_addons.add(name)- self._call_addon(name, "update_addons", self)-- def update_settings(self, settings):+ def update_settings(self, settings) -> None:"""Call ``update_settings()`` of all held add-ons.:param settings: The :class:`~scrapy.settings.Settings` object to be \@@ -413,13 +140,13 @@ class AddonManager(Mapping)::type settings: :class:`~scrapy.settings.Settings`"""for name in self:- self._call_addon(name, "update_settings", settings)+ self[name].update_settings(self.configs[name], settings)- def check_configuration(self, crawler):+ def check_configuration(self, crawler) -> None:"""Call ``check_configuration()`` of all held add-ons.:param crawler: the fully-initialized crawler:type crawler: :class:`~scrapy.crawler.Crawler`"""for name in self:- self._call_addon(name, "check_configuration", crawler)+ self[name].check_configuration(self.configs[name], crawler)diff --git a/scrapy/cmdline.py b/scrapy/cmdline.pyindex 1b579f10e..efc9b36ea 100644--- a/scrapy/cmdline.py+++ b/scrapy/cmdline.py@@ -6,7 +6,6 @@ import sysfrom importlib.metadata import entry_pointsimport scrapy-from scrapy.addons import AddonManagerfrom scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatterfrom scrapy.crawler import CrawlerProcessfrom scrapy.exceptions import UsageError@@ -131,8 +130,6 @@ def execute(argv=None, settings=None):else:settings["EDITOR"] = editor- addons = AddonManager()-inproject = inside_project()cmds = _get_commands_dict(settings, inproject)cmdname = _pop_command_name(argv)@@ -156,7 +153,7 @@ def execute(argv=None, settings=None):opts, args = parser.parse_known_args(args=argv[1:])_run_print_help(parser, cmd.process_options, args, opts)- cmd.crawler_process = CrawlerProcess(settings, addons=addons)+ cmd.crawler_process = CrawlerProcess(settings)_run_print_help(parser, _run_command, cmd, args, opts)sys.exit(cmd.exitcode)diff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex 0c9861dbd..a0cb368ed 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -58,7 +58,6 @@ class Crawler:spidercls: Type[Spider],settings: Union[None, dict, Settings] = None,init_reactor: bool = False,- addons: Optional[AddonManager] = None,):if isinstance(spidercls, Spider):raise ValueError("The spidercls argument must be a class, not an object")@@ -70,10 +69,8 @@ class Crawler:self.settings: Settings = settings.copy()self.spidercls.update_settings(self.settings)- self.addons: AddonManager = addons if addons is not None else AddonManager()+ self.addons: AddonManager = AddonManager()self.addons.load_settings(self.settings)- self.addons.update_addons()- self.addons.check_dependency_clashes()self.addons.update_settings(self.settings)self.signals: SignalManager = SignalManager(self)@@ -200,15 +197,10 @@ class CrawlerRunner:)return loader_cls.from_settings(settings.frozencopy())- def __init__(- self,- settings: Union[Dict[str, Any], Settings, None] = None,- addons: Optional[AddonManager] = None,- ):+ def __init__(self, settings: Union[Dict[str, Any], Settings, None] = None):if isinstance(settings, dict) or settings is None:settings = Settings(settings)self.settings = settings- self.addons: Optional[AddonManager] = addonsself.spider_loader = self._get_spider_loader(settings)self._crawlers: Set[Crawler] = set()self._active: Set[defer.Deferred] = set()@@ -292,7 +284,7 @@ class CrawlerRunner:def _create_crawler(self, spidercls: Union[str, Type[Spider]]) -> Crawler:if isinstance(spidercls, str):spidercls = self.spider_loader.load(spidercls)- return Crawler(spidercls, self.settings, addons=self.addons)+ return Crawler(spidercls, self.settings)def stop(self):"""@@ -338,13 +330,8 @@ class CrawlerProcess(CrawlerRunner):process. See :ref:`run-from-script` for an example."""- def __init__(- self,- settings=None,- install_root_handler: bool = True,- addons: Optional[AddonManager] = None,- ):- super().__init__(settings, addons)+ def __init__(self, settings=None, install_root_handler=True):+ super().__init__(settings)configure_logging(self.settings, install_root_handler)log_scrapy_info(self.settings)self._initialized_reactor = Falsediff --git a/scrapy/interfaces.py b/scrapy/interfaces.pyindex b8aa77ced..9a2c5f170 100644--- a/scrapy/interfaces.py+++ b/scrapy/interfaces.py@@ -1,4 +1,4 @@-from zope.interface import Attribute, Interface+from zope.interface import Interfaceclass ISpiderLoader(Interface):@@ -15,22 +15,3 @@ class ISpiderLoader(Interface):def find_by_request(request):"""Return the list of spiders names that can handle the given request"""---class IAddon(Interface):- """Scrapy add-on"""-- name = Attribute("""Add-on name""")- version = Attribute("""Add-on version string (PEP440)""")-- # XXX: Can methods be declared optional? I.e., can I enforce the signature- # but not the existence of a method?-- # def update_addons(config, addons):- # """Enables and configures other add-ons"""-- # def update_settings(config, settings):- # """Modifies `settings` to enable and configure required components"""-- # def check_configuration(config, crawler):- # """Performs post-initialization checks on fully configured `crawler`"""diff --git a/scrapy/utils/conf.py b/scrapy/utils/conf.pyindex 8d1544c68..3ade1d105 100644--- a/scrapy/utils/conf.py+++ b/scrapy/utils/conf.py@@ -99,18 +99,12 @@ def init_env(project="default", set_syspath=True):sys.path.append(projdir)-def config_from_filepath(sources):- """Create a ConfigParser and read in the given `sources`, which can be- either a filename or a list of filenames."""- cfg = ConfigParser()- cfg.read(sources)- return cfg--def get_config(use_closest=True):"""Get Scrapy config file as a ConfigParser"""sources = get_sources(use_closest)- return config_from_filepath(sources)+ cfg = ConfigParser()+ cfg.read(sources)+ return cfgdef get_sources(use_closest=True) -> List[str]:diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.pyindex 4e23b01c3..b3c28da92 100644--- a/scrapy/utils/misc.py+++ b/scrapy/utils/misc.py@@ -67,7 +67,7 @@ def load_object(path: Union[str, Callable]) -> Any:if callable(path):return pathraise TypeError(- "Unexpected argument type, expected string " f"or object, got: {type(path)}"+ f"Unexpected argument type, expected string or object, got: {type(path)}")try:@@ -86,22 +86,6 @@ def load_object(path: Union[str, Callable]) -> Any:return obj-def load_module_or_object(path):- """Load python module or (non-module) object from given path.-- Path can be both a Python or a file path.- """- try:- return import_module(path)- except ImportError:- pass- try:- return load_object(path)- except (ValueError, NameError, ImportError):- pass- raise NameError(f"Could not load '{path}'")--def walk_modules(path: str) -> List[ModuleType]:"""Loads a module and all its submodules from the given module path andreturns them. If *any* module throws an exception while importing, thatdiff --git a/scrapy/utils/test.py b/scrapy/utils/test.pyindex fe26e1708..97de8d25a 100644--- a/scrapy/utils/test.py+++ b/scrapy/utils/test.py@@ -73,7 +73,6 @@ def get_crawler(spidercls: Optional[Type[Spider]] = None,settings_dict: Optional[Dict[str, Any]] = None,prevent_warnings: bool = True,- addons=None,) -> Crawler:"""Return an unconfigured Crawler object. If settings_dict is given, itwill be used to populate the crawler settings with a project level@@ -87,7 +86,7 @@ def get_crawler(if prevent_warnings:settings["REQUEST_FINGERPRINTER_IMPLEMENTATION"] = "2.7"settings.update(settings_dict or {})- runner = CrawlerRunner(settings, addons)+ runner = CrawlerRunner(settings)return runner.create_crawler(spidercls or Spider)diff --git a/tests/test_addons.py b/tests/test_addons.pynew file mode 100644index 000000000..ecdc0426c--- /dev/null+++ b/tests/test_addons.py@@ -0,0 +1,72 @@+import unittest++from scrapy.addons import Addon, AddonManager+from scrapy.settings import BaseSettings+++class GoodAddon(object):+ name = "GoodAddon"++ def update_settings(self, config, settings):+ pass++ def check_configuration(self, config, crawler):+ pass+++class AddonTest(unittest.TestCase):+ def setUp(self):+ class AddonWithAttributes(Addon):+ name = "Test"++ self.testaddon = AddonWithAttributes()++ def test_export_config(self):+ settings = BaseSettings()+ self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}+ self.testaddon.default_config = {"key": 55, "defaultkey": 100}+ self.testaddon.export_config(+ {"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings+ )+ self.assertEqual(settings["KEY"], 313)+ self.assertEqual(settings["DEFAULTKEY"], 100)+ self.assertEqual(settings["OTHERKEY"], True)+ self.assertNotIn("MAPPED_key", settings)+ self.assertNotIn("MAPPED_KEY", settings)+ self.assertEqual(settings["MAPPING_WORKED"], 99)+ self.assertEqual(settings.getpriority("KEY"), 15)++ def test_update_settings(self):+ settings = BaseSettings()+ settings.set("KEY1", "default", priority="default")+ settings.set("KEY2", "project", priority="project")+ addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}+ self.testaddon.update_settings(addon_config, settings)+ self.assertEqual(settings["KEY1"], "addon")+ self.assertEqual(settings["KEY2"], "project")+ self.assertEqual(settings["KEY3"], "addon")+++class AddonManagerTest(unittest.TestCase):+ def setUp(self):+ self.manager = AddonManager()++ def test_add(self):+ manager = AddonManager()+ manager.add("tests.test_addons.GoodAddon")+ self.assertCountEqual(manager, ["GoodAddon"])+ self.assertIsInstance(manager["GoodAddon"], GoodAddon)++ def test_load_settings(self):+ settings = BaseSettings()+ settings.set(+ "ADDONS",+ {"tests.test_addons.GoodAddon": 0},+ )+ settings.set("GOODADDON", {"key": "val2"})+ manager = AddonManager()+ manager.load_settings(settings)+ self.assertCountEqual(manager, ["GoodAddon"])+ self.assertIsInstance(manager["GoodAddon"], GoodAddon)+ self.assertCountEqual(manager.configs["GoodAddon"], ["key"])+ self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")diff --git a/tests/test_addons/__init__.py b/tests/test_addons/__init__.pydeleted file mode 100644index fa3f706e5..000000000--- a/tests/test_addons/__init__.py+++ /dev/null@@ -1,342 +0,0 @@-import itertools-import unittest-import warnings-from unittest import mock--from zope.interface import directlyProvides-from zope.interface.exceptions import BrokenImplementation, MultipleInvalid-from zope.interface.verify import verifyObject--from scrapy.addons import Addon, AddonManager-from scrapy.crawler import Crawler-from scrapy.interfaces import IAddon-from scrapy.settings import BaseSettings--from . import addonmod, addons---class AddonTest(unittest.TestCase):- def setUp(self):- self.rawaddon = Addon()-- class AddonWithAttributes(Addon):- name = "Test"- version = "1.0"-- self.testaddon = AddonWithAttributes()-- def test_interface(self):- # Raw Addon should fail exactly b/c name and version are not given- self.assertFalse(hasattr(self.rawaddon, "name"))- self.assertFalse(hasattr(self.rawaddon, "version"))- self.assertRaises(MultipleInvalid, verifyObject, IAddon, self.rawaddon)- verifyObject(IAddon, self.testaddon)-- def test_export_component(self):- settings = BaseSettings(- {"ITEM_PIPELINES": BaseSettings(), "DOWNLOAD_HANDLERS": BaseSettings()},- "default",- )- self.testaddon.component_type = None- self.testaddon.export_component({}, settings)- self.assertEqual(len(settings["ITEM_PIPELINES"]), 0)- self.testaddon.component_type = "ITEM_PIPELINES"- self.testaddon.component = "test.component"- self.testaddon.export_component({}, settings)- self.assertCountEqual(settings["ITEM_PIPELINES"], ["test.component"])- self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 0)- self.testaddon.component_order = 313- self.testaddon.export_component({}, settings)- self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 313)- self.testaddon.component_type = "DOWNLOAD_HANDLERS"- self.testaddon.component_key = "http"- self.testaddon.export_component({}, settings)- self.assertEqual(settings["DOWNLOAD_HANDLERS"]["http"], "test.component")-- def test_export_basics(self):- settings = BaseSettings()- self.testaddon.basic_settings = {"TESTKEY": 313, "OTHERKEY": True}- self.testaddon.export_basics(settings)- self.assertEqual(settings["TESTKEY"], 313)- self.assertEqual(settings["OTHERKEY"], True)- self.assertEqual(settings.getpriority("TESTKEY"), 15)-- def test_export_config(self):- settings = BaseSettings()- self.testaddon.settings_prefix = None- self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}- self.testaddon.default_config = {"key": 55, "defaultkey": 100}- self.testaddon.export_config(- {"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings- )- self.assertEqual(settings["TEST_KEY"], 313)- self.assertEqual(settings["TEST_DEFAULTKEY"], 100)- self.assertEqual(settings["TEST_OTHERKEY"], True)- self.assertNotIn("MAPPED_key", settings)- self.assertNotIn("MAPPED_KEY", settings)- self.assertEqual(settings["MAPPING_WORKED"], 99)- self.assertEqual(settings.getpriority("TEST_KEY"), 15)-- self.testaddon.settings_prefix = "PREF"- self.testaddon.export_config({"newkey": 99}, settings)- self.assertEqual(settings["PREF_NEWKEY"], 99)-- with mock.patch.object(settings, "set") as mock_set:- self.testaddon.settings_prefix = False- self.testaddon.export_config({"thirdnewkey": 99}, settings)- self.assertEqual(mock_set.call_count, 0)-- def test_update_settings(self):- settings = BaseSettings()- settings.set("TEST_KEY1", "default", priority="default")- settings.set("TEST_KEY2", "project", priority="project")- self.testaddon.settings_prefix = None- self.testaddon.basic_settings = {"OTHERTEST_KEY": "addon"}- addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}- self.testaddon.update_settings(addon_config, settings)- self.assertEqual(settings["OTHERTEST_KEY"], "addon")- self.assertEqual(settings["TEST_KEY1"], "addon")- self.assertEqual(settings["TEST_KEY2"], "project")- self.assertEqual(settings["TEST_KEY3"], "addon")---class AddonManagerTest(unittest.TestCase):- def setUp(self):- self.manager = AddonManager()-- def test_add(self):- manager = AddonManager()- manager.add(addonmod, {"key": "val1"})- manager.add("tests.test_addons.addons.GoodAddon")- self.assertCountEqual(manager, ["AddonModule", "GoodAddon"])- self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)- self.assertCountEqual(manager.configs["AddonModule"], ["key"])- self.assertEqual(manager.configs["AddonModule"]["key"], "val1")- self.assertRaises(ValueError, manager.add, addonmod)-- def test_add_dont_instantiate_providing_classes(self):- class ProviderGoodAddon(addons.GoodAddon):- pass-- directlyProvides(ProviderGoodAddon, IAddon)- manager = AddonManager()- manager.add(ProviderGoodAddon)- self.assertIs(manager["GoodAddon"], ProviderGoodAddon)-- def test_add_verifies(self):- brokenaddon = self.manager.get_addon("tests.test_addons.addons.BrokenAddon")- self.assertRaises(- BrokenImplementation,- self.manager.add,- brokenaddon,- )-- def test_add_adds_missing_interface_declaration(self):- class GoodAddonWithoutDeclaration(object):- name = "GoodAddonWithoutDeclaration"- version = "1.0"-- self.manager.add(GoodAddonWithoutDeclaration)-- def test_remove(self):- manager = AddonManager()-- def test_gets_removed(removearg):- manager.add(addonmod)- self.assertIn("AddonModule", manager)- manager.remove(removearg)- self.assertNotIn("AddonModule", manager)-- test_gets_removed("AddonModule")- test_gets_removed(addonmod)- test_gets_removed("tests.test_addons.addonmod")- self.assertRaises(KeyError, manager.remove, "nonexistent")- self.assertRaises(KeyError, manager.remove, addons.GoodAddon())-- def test_get_addon(self):- goodaddon = self.manager.get_addon("tests.test_addons.addons.GoodAddon")- self.assertIs(goodaddon, addons.GoodAddon)-- loaded_addonmod = self.manager.get_addon("tests.test_addons.addonmod")- self.assertIs(loaded_addonmod, addonmod)-- goodaddon = self.manager.get_addon("tests.test_addons.addons")- self.assertIsInstance(goodaddon, addons.GoodAddon)-- self.assertRaises(NameError, self.manager.get_addon, "xy.n_onexistent")-- def test_get_addon_forward(self):- class SomeCls(object):- _addon = "tests.test_addons.addons.GoodAddon"-- self.assertIs(self.manager.get_addon(SomeCls()), addons.GoodAddon)-- def test_get_addon_nested(self):- x = addons.GoodAddon("outer")- x._addon = addons.GoodAddon("middle")- x._addon._addon = addons.GoodAddon("inner")- self.assertIs(self.manager.get_addon(x), x._addon._addon)-- def test_load_settings(self):- settings = BaseSettings()- settings.set(- "ADDONS",- {"tests.test_addons.addonmod": 0, "tests.test_addons.addons.GoodAddon": 0},- )- settings.set("ADDONMODULE", {"key": "val1"})- settings.set("GOODADDON", {"key": "val2"})- manager = AddonManager()- manager.load_settings(settings)- self.assertCountEqual(manager, ["GoodAddon", "AddonModule"])- self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)- self.assertCountEqual(manager.configs["GoodAddon"], ["key"])- self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")- self.assertEqual(manager["AddonModule"], addonmod)- self.assertIn("key", manager.configs["AddonModule"])- self.assertEqual(manager.configs["AddonModule"]["key"], "val1")-- def test_load_settings_order(self):- # Get three addons named 0, 1, 2- addonlist = [addons.GoodAddon(str(x)) for x in range(3)]- # Test for every possible ordering- for ordered_addons in itertools.permutations(addonlist):- expected_order = [a.name for a in ordered_addons]- settings = BaseSettings(- {"ADDONS": {a: i for i, a in enumerate(ordered_addons)}}- )- manager = AddonManager()- manager.load_settings(settings)- self.assertEqual(list(manager.keys()), expected_order)-- def test_enabled_disabled(self):- manager = AddonManager()- manager.add(addons.GoodAddon("FirstAddon"))- manager.add(addons.GoodAddon("SecondAddon"))- self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))- self.assertEqual(manager.disabled, [])- manager.disable("FirstAddon")- self.assertEqual(manager.enabled, ["SecondAddon"])- self.assertEqual(manager.disabled, ["FirstAddon"])- manager.enable("FirstAddon")- self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))- self.assertEqual(manager.disabled, [])-- def test_enable_before_add(self):- manager = AddonManager()- self.assertRaises(ValueError, manager.enable, "FirstAddon")- manager.disable("FirstAddon")- manager.enable("FirstAddon")- manager.add(addons.GoodAddon("FirstAddon"))- self.assertIn("FirstAddon", manager.enabled)-- def test_disable_before_add(self):- manager = AddonManager()- manager.disable("FirstAddon")- manager.add(addons.GoodAddon("FirstAddon"))- self.assertEqual(manager.disabled, ["FirstAddon"])-- def test_callbacks(self):- first_addon = addons.GoodAddon("FirstAddon")- second_addon = addons.GoodAddon("SecondAddon")-- manager = AddonManager()- manager.add(first_addon, {"test": "first"})- manager.add(second_addon, {"test": "second"})- crawler = mock.create_autospec(Crawler)- settings = BaseSettings()-- with mock.patch.object(- first_addon, "update_addons"- ) as ua_first, mock.patch.object(- second_addon, "update_addons"- ) as ua_second, mock.patch.object(- first_addon, "update_settings"- ) as us_first, mock.patch.object(- second_addon, "update_settings"- ) as us_second, mock.patch.object(- first_addon, "check_configuration"- ) as cc_first, mock.patch.object(- second_addon, "check_configuration"- ) as cc_second:- manager.update_addons()- ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)- ua_second.assert_called_once_with(manager.configs["SecondAddon"], manager)- manager.update_settings(settings)- us_first.assert_called_once_with(manager.configs["FirstAddon"], settings)- us_second.assert_called_once_with(manager.configs["SecondAddon"], settings)- manager.check_configuration(crawler)- cc_first.assert_called_once_with(manager.configs["FirstAddon"], crawler)- cc_second.assert_called_once_with(manager.configs["SecondAddon"], crawler)- self.assertEqual(ua_first.call_count, 1)- self.assertEqual(ua_second.call_count, 1)- self.assertEqual(us_first.call_count, 1)- self.assertEqual(us_second.call_count, 1)-- us_first.reset_mock()- us_second.reset_mock()- manager.disable("FirstAddon")- manager.update_settings(settings)- self.assertEqual(us_first.call_count, 0)- manager.enable("FirstAddon")- manager.update_settings(settings)- self.assertEqual(us_first.call_count, 1)- self.assertEqual(us_second.call_count, 2)-- # This will become relevant when we let spiders implement the add-on- # interface and should be replaced with a test where- # AddonManager.spidercls = None then.- manager._call_if_exists(None, "irrelevant")-- def test_update_addons_last_minute_add(self):- class AddedAddon(addons.GoodAddon):- name = "AddedAddon"-- class FirstAddon(addons.GoodAddon):- name = "FirstAddon"-- def update_addons(self, config, addons):- addons.add(AddedAddon())-- manager = AddonManager()- first_addon = FirstAddon()- with mock.patch.object(- first_addon, "update_addons", wraps=first_addon.update_addons- ) as ua_first, mock.patch.object(AddedAddon, "update_addons") as ua_added:- manager.add(first_addon, {"non-empty": "dict"})- manager.update_addons()- self.assertCountEqual(manager, ["FirstAddon", "AddedAddon"])- ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)- ua_added.assert_called_once_with(manager.configs["AddedAddon"], manager)-- def test_check_dependency_clashes_attributes(self):- provides = addons.GoodAddon("ProvidesAddon")- provides.provides = ("test",)- provides2 = addons.GoodAddon("ProvidesAddon2")- provides2.provides = ("test",)- requires = addons.GoodAddon("RequiresAddon")- requires.requires = ("test",)- requires_name = addons.GoodAddon("RequiresNameAddon")- requires_name.requires = ("ProvidesAddon",)- requires_newer = addons.GoodAddon("RequiresNewerAddon")- requires_newer.requires = ("test>=2.0",)- modifies = addons.GoodAddon("ModifiesAddon")- modifies.modifies = ("test",)-- def check_with(*addons):- manager = AddonManager()- for a in addons:- manager.add(a)- return manager.check_dependency_clashes()-- self.assertRaises(ImportError, check_with, requires)- self.assertRaises(ImportError, check_with, modifies)- self.assertRaises(ImportError, check_with, provides, provides2)- self.assertRaises(ImportError, check_with, provides, requires_newer)- with warnings.catch_warnings(record=True) as w:- check_with(provides, modifies)- check_with(provides)- check_with(provides, requires)- check_with(provides, requires_name)- self.assertEqual(len(w), 0)- check_with(requires, provides, modifies)- self.assertEqual(len(w), 1)diff --git a/tests/test_addons/addonmod.py b/tests/test_addons/addonmod.pydeleted file mode 100644index 092c3c0eb..000000000--- a/tests/test_addons/addonmod.py+++ /dev/null@@ -1,16 +0,0 @@-from zope.interface import moduleProvides--from scrapy.interfaces import IAddon--moduleProvides(IAddon)--name = "AddonModule"-version = "1.0"---def update_settings(config, settings):- pass---def check_configuration(config, crawler):- passdiff --git a/tests/test_addons/addons.py b/tests/test_addons/addons.pydeleted file mode 100644index d878f37ea..000000000--- a/tests/test_addons/addons.py+++ /dev/null@@ -1,33 +0,0 @@-from zope.interface import implementer--from scrapy.interfaces import IAddon---@implementer(IAddon)-class GoodAddon(object):- name = "GoodAddon"- version = "1.0"-- def __init__(self, name=None, version=None):- if name is not None:- self.name = name- if version is not None:- self.version = version-- def update_addons(self, config, addons):- pass-- def update_settings(self, config, settings):- pass-- def check_configuration(self, config, crawler):- pass---@implementer(IAddon)-class BrokenAddon(object):- name = "BrokenAddon"- # No version---_addon = GoodAddon()diff --git a/tests/test_crawl.py b/tests/test_crawl.pyindex 920e5f4ae..d844a645f 100644--- a/tests/test_crawl.py+++ b/tests/test_crawl.py@@ -412,7 +412,6 @@ with multiples linesdef test_abort_on_addon_failed_check(self):class FailedCheckAddon(Addon):name = "FailedCheckAddon"- version = "1.0"def check_configuration(self, config, crawler):raise ValueErrordiff --git a/tests/test_crawler.py b/tests/test_crawler.pyindex ed34d9e58..d54a2cb7e 100644--- a/tests/test_crawler.py+++ b/tests/test_crawler.py@@ -15,7 +15,6 @@ from twisted.trial import unittestfrom w3lib import __version__ as w3lib_versionimport scrapy-from scrapy.addons import Addon, AddonManagerfrom scrapy.crawler import Crawler, CrawlerProcess, CrawlerRunnerfrom scrapy.exceptions import ScrapyDeprecationWarningfrom scrapy.extensions import telnet@@ -56,32 +55,6 @@ class CrawlerTestCase(BaseCrawlerTest):self.assertFalse(settings.frozen)self.assertTrue(crawler.settings.frozen)- def test_populate_addons_settings(self):- class TestAddon(Addon):- name = "TestAddon"- version = "1.0"-- addonconfig = {"TEST1": "addon", "TEST2": "addon", "TEST3": "addon"}-- class TestAddon2(Addon):- name = "testAddon2"- version = "1.0"-- addonconfig2 = {"TEST": "addon2"}-- settings = Settings()- settings.set("TESTADDON_TEST1", "project", priority="project")- settings.set("TESTADDON_TEST2", "default", priority="default")- addonmgr = AddonManager()- addonmgr.add(TestAddon(), addonconfig)- addonmgr.add(TestAddon2(), addonconfig2)- crawler = Crawler(DefaultSpider, settings, addons=addonmgr)-- self.assertEqual(crawler.settings["TESTADDON_TEST1"], "project")- self.assertEqual(crawler.settings["TESTADDON_TEST2"], "addon")- self.assertEqual(crawler.settings["TESTADDON_TEST3"], "addon")- self.assertEqual(crawler.settings["TESTADDON2_TEST"], "addon2")-def test_crawler_accepts_dict(self):crawler = get_crawler(DefaultSpider, {"foo": "bar"})self.assertEqual(crawler.settings["foo"], "bar")diff --git a/tests/test_utils_misc/__init__.py b/tests/test_utils_misc/__init__.pyindex 7932ca04c..69793ee75 100644--- a/tests/test_utils_misc/__init__.py+++ b/tests/test_utils_misc/__init__.py@@ -8,7 +8,6 @@ from scrapy.item import Field, Itemfrom scrapy.utils.misc import (arg_to_iter,create_instance,- load_module_or_object,load_object,rel_has_nofollow,set_environ,@@ -36,12 +35,6 @@ class UtilsMiscTestCase(unittest.TestCase):self.assertRaises(NameError, load_object, "scrapy.utils.misc.load_object999")self.assertRaises(TypeError, load_object, {})- def test_load_module_or_object(self):- testmod = load_module_or_object(__name__ + ".testmod")- self.assertTrue(hasattr(testmod, "TESTVAR"))- obj = load_object("scrapy.utils.misc.load_object")- self.assertIs(obj, load_object)-def test_walk_modules(self):mods = walk_modules("tests.test_utils_misc.test_walk_modules")expected = [diff --git a/tests/test_utils_misc/testmod.py b/tests/test_utils_misc/testmod.pydeleted file mode 100644index eb540335f..000000000--- a/tests/test_utils_misc/testmod.py+++ /dev/null@@ -1 +0,0 @@-TESTVAR = True