Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document how to enforce Scrapy versions on Scrapy components

Adrián Chavesb95c634b861b
diff --git a/docs/index.rst b/docs/index.rst
index 75e08f537..6e22db884 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -229,10 +229,11 @@ Extending Scrapy
topics/downloader-middleware
topics/spider-middleware
topics/extensions
- topics/api
topics/signals
topics/scheduler
topics/exporters
+ topics/components
+ topics/api
:doc:`topics/architecture`
@@ -247,9 +248,6 @@ Extending Scrapy
:doc:`topics/extensions`
Extend Scrapy with your custom functionality
-:doc:`topics/api`
- Use it on extensions and middlewares to extend Scrapy functionality
-
:doc:`topics/signals`
See all available signals and how to work with them.
@@ -259,6 +257,13 @@ Extending Scrapy
:doc:`topics/exporters`
Quickly export your scraped items to a file (XML, CSV, etc).
+:doc:`topics/components`
+ Learn the common API and some good practices when building custom Scrapy
+ components.
+
+:doc:`topics/api`
+ Use it on extensions and middlewares to extend Scrapy functionality
+
All the rest
============
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index 3a6941a2c..dbee7146d 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -96,3 +96,27 @@ Futures. Scrapy provides two helpers for this:
down to Scrapy 2.0 (earlier versions do not support
:mod:`asyncio`), you can copy the implementation of these functions
into your own code.
+
+
+.. _enforce-asyncio-requirement:
+
+Enforcing asyncio as a requirement
+==================================
+
+If you are writing a :ref:`component <topics-components>` that requires asyncio
+to work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to
+:ref:`enforce it as a requirement <enforce-component-requirements>`. For
+example::
+
+ from scrapy.utils.reactor import is_asyncio_reactor_installed
+
+ class MyComponent:
+
+ def __init__(self):
+ if not is_asyncio_reactor_installed():
+ raise ValueError(
+ f"{MyComponent.__qualname__} requires the asyncio Twisted "
+ f"reactor. Make sure you have it configured in the "
+ f"TWISTED_REACTOR setting. See the asyncio documentation "
+ f"of Scrapy for more information."
+ )
diff --git a/docs/topics/components.rst b/docs/topics/components.rst
new file mode 100644
index 000000000..1fff2d61a
--- /dev/null
+++ b/docs/topics/components.rst
@@ -0,0 +1,84 @@
+.. _topics-components:
+
+==========
+Components
+==========
+
+A Scrapy component is any class whose objects are created using
+:func:`scrapy.utils.misc.create_instance`.
+
+That includes the classes that you may assign to the following settings:
+
+- :setting:`DNS_RESOLVER`
+
+- :setting:`DOWNLOAD_HANDLERS`
+
+- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY`
+
+- :setting:`DOWNLOADER_MIDDLEWARES`
+
+- :setting:`DUPEFILTER_CLASS`
+
+- :setting:`EXTENSIONS`
+
+- :setting:`FEED_EXPORTERS`
+
+- :setting:`FEED_STORAGES`
+
+- :setting:`ITEM_PIPELINES`
+
+- :setting:`SCHEDULER`
+
+- :setting:`SCHEDULER_DISK_QUEUE`
+
+- :setting:`SCHEDULER_MEMORY_QUEUE`
+
+- :setting:`SCHEDULER_PRIORITY_QUEUE`
+
+- :setting:`SPIDER_MIDDLEWARES`
+
+Third-party Scrapy components may also let you define additional Scrapy
+components, usually configurable through :ref:`settings <topics-settings>`, to
+modify their behavior.
+
+.. _enforce-component-requirements:
+
+Enforcing component requirements
+================================
+
+Sometimes, your components may only be intended to work under certain
+conditions. For example, the may require a minimum version of Scrapy to work as
+intended, or they may require certain settings to have specific values.
+
+In addition to describing those conditions in the documentation of your
+component, it is a good practice to raise an exception from the ``__init__``
+method of your component if those conditions are not met at run time.
+
+In the case of :ref:`downloader middlewares <topics-downloader-middleware>`,
+:ref:`extensions <topics-extensions>`, :ref:`item pipelines
+<topics-item-pipeline>`, and :ref:`spider middlewares
+<topics-spider-middleware>`, you should raise
+:exc:`scrapy.exceptions.NotConfigured`, passing a description of the issue as a
+parameter to the exception so that it is printed in the logs, for the user to
+see. For other components, feel free to raise whatever other exception feels
+right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy
+version mismatch, while :exc:`ValueError` may be better if the issue is the
+value of a setting.
+
+If your requirement is a minimum Scrapy version, you may use
+:attr:`scrapy.__version__` to enforce your requirement. For example::
+
+ from pkg_resources import parse_version
+
+ import scrapy
+
+ class MyComponent:
+
+ def __init__(self):
+ if parse_version(scrapy.__version__) < parse_version('VERSION'):
+ raise RuntimeError(
+ f"{MyComponent.__qualname__} requires Scrapy VERSION or "
+ f"later, which allow defining the process_spider_output "
+ f"method of spider middlewares as an asynchronous "
+ f"generator."
+ )
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index efc4566a0..55d013c06 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -162,7 +162,7 @@ Asynchronous-to-synchronous conversions are supported for backward
compatibility, but they are deprecated and will stop working in a future
version of Scrapy.
-To avoid asynchronous-to-synchronous conversion, when defining ``Request``
+To avoid asynchronous-to-synchronous conversions, when defining ``Request``
callbacks as coroutine methods or when using spider middlewares whose
``process_spider_output`` method is an :term:`asynchronous generator`, all
active spider middlewares must either have their ``process_spider_output``
@@ -177,8 +177,8 @@ process_spider_output_async method <universal-spider-middleware>`.
.. _universal-spider-middleware:
-Universal spider middleware
-===========================
+Universal spider middlewares
+============================
.. versionadded:: VERSION
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 787545ed2..816cb5e03 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -111,9 +111,11 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
Consider defining this method as an :term:`asynchronous generator`,
which will be a requirement in a future version of Scrapy. However, if
- you wish your spider middleware to work with Scrapy versions earlier
- than Scrapy VERSION, :ref:`make your spider middleware universal
- <universal-spider-middleware>` instead.
+ you plan on sharing your spider middleware with other people, consider
+ either :ref:`enforcing Scrapy VERSION <enforce-component-requirements>`
+ as a minimum requirement of your spider middleware, or :ref:`making
+ your spider middleware universal <universal-spider-middleware>` so that
+ it works with Scrapy versions earlier than Scrapy VERSION.
:param response: the response which generated this output from the
spider