Scrapy, a fast high-level web crawling & scraping framework for Python.
Document how to enforce Scrapy versions on Scrapy components
Adrián Chavesb95c634b861b
diff --git a/docs/index.rst b/docs/index.rstindex 75e08f537..6e22db884 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -229,10 +229,11 @@ Extending Scrapytopics/downloader-middlewaretopics/spider-middlewaretopics/extensions- topics/apitopics/signalstopics/schedulertopics/exporters+ topics/components+ topics/api:doc:`topics/architecture`@@ -247,9 +248,6 @@ Extending Scrapy:doc:`topics/extensions`Extend Scrapy with your custom functionality-:doc:`topics/api`- Use it on extensions and middlewares to extend Scrapy functionality-:doc:`topics/signals`See all available signals and how to work with them.@@ -259,6 +257,13 @@ Extending Scrapy:doc:`topics/exporters`Quickly export your scraped items to a file (XML, CSV, etc).+:doc:`topics/components`+ Learn the common API and some good practices when building custom Scrapy+ components.++:doc:`topics/api`+ Use it on extensions and middlewares to extend Scrapy functionality+All the rest============diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex 3a6941a2c..dbee7146d 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -96,3 +96,27 @@ Futures. Scrapy provides two helpers for this:down to Scrapy 2.0 (earlier versions do not support:mod:`asyncio`), you can copy the implementation of these functionsinto your own code.+++.. _enforce-asyncio-requirement:++Enforcing asyncio as a requirement+==================================++If you are writing a :ref:`component <topics-components>` that requires asyncio+to work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to+:ref:`enforce it as a requirement <enforce-component-requirements>`. For+example::++ from scrapy.utils.reactor import is_asyncio_reactor_installed++ class MyComponent:++ def __init__(self):+ if not is_asyncio_reactor_installed():+ raise ValueError(+ f"{MyComponent.__qualname__} requires the asyncio Twisted "+ f"reactor. Make sure you have it configured in the "+ f"TWISTED_REACTOR setting. See the asyncio documentation "+ f"of Scrapy for more information."+ )diff --git a/docs/topics/components.rst b/docs/topics/components.rstnew file mode 100644index 000000000..1fff2d61a--- /dev/null+++ b/docs/topics/components.rst@@ -0,0 +1,84 @@+.. _topics-components:++==========+Components+==========++A Scrapy component is any class whose objects are created using+:func:`scrapy.utils.misc.create_instance`.++That includes the classes that you may assign to the following settings:++- :setting:`DNS_RESOLVER`++- :setting:`DOWNLOAD_HANDLERS`++- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY`++- :setting:`DOWNLOADER_MIDDLEWARES`++- :setting:`DUPEFILTER_CLASS`++- :setting:`EXTENSIONS`++- :setting:`FEED_EXPORTERS`++- :setting:`FEED_STORAGES`++- :setting:`ITEM_PIPELINES`++- :setting:`SCHEDULER`++- :setting:`SCHEDULER_DISK_QUEUE`++- :setting:`SCHEDULER_MEMORY_QUEUE`++- :setting:`SCHEDULER_PRIORITY_QUEUE`++- :setting:`SPIDER_MIDDLEWARES`++Third-party Scrapy components may also let you define additional Scrapy+components, usually configurable through :ref:`settings <topics-settings>`, to+modify their behavior.++.. _enforce-component-requirements:++Enforcing component requirements+================================++Sometimes, your components may only be intended to work under certain+conditions. For example, the may require a minimum version of Scrapy to work as+intended, or they may require certain settings to have specific values.++In addition to describing those conditions in the documentation of your+component, it is a good practice to raise an exception from the ``__init__``+method of your component if those conditions are not met at run time.++In the case of :ref:`downloader middlewares <topics-downloader-middleware>`,+:ref:`extensions <topics-extensions>`, :ref:`item pipelines+<topics-item-pipeline>`, and :ref:`spider middlewares+<topics-spider-middleware>`, you should raise+:exc:`scrapy.exceptions.NotConfigured`, passing a description of the issue as a+parameter to the exception so that it is printed in the logs, for the user to+see. For other components, feel free to raise whatever other exception feels+right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy+version mismatch, while :exc:`ValueError` may be better if the issue is the+value of a setting.++If your requirement is a minimum Scrapy version, you may use+:attr:`scrapy.__version__` to enforce your requirement. For example::++ from pkg_resources import parse_version++ import scrapy++ class MyComponent:++ def __init__(self):+ if parse_version(scrapy.__version__) < parse_version('VERSION'):+ raise RuntimeError(+ f"{MyComponent.__qualname__} requires Scrapy VERSION or "+ f"later, which allow defining the process_spider_output "+ f"method of spider middlewares as an asynchronous "+ f"generator."+ )diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex efc4566a0..55d013c06 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -162,7 +162,7 @@ Asynchronous-to-synchronous conversions are supported for backwardcompatibility, but they are deprecated and will stop working in a futureversion of Scrapy.-To avoid asynchronous-to-synchronous conversion, when defining ``Request``+To avoid asynchronous-to-synchronous conversions, when defining ``Request``callbacks as coroutine methods or when using spider middlewares whose``process_spider_output`` method is an :term:`asynchronous generator`, allactive spider middlewares must either have their ``process_spider_output``@@ -177,8 +177,8 @@ process_spider_output_async method <universal-spider-middleware>`... _universal-spider-middleware:-Universal spider middleware-===========================+Universal spider middlewares+============================.. versionadded:: VERSIONdiff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 787545ed2..816cb5e03 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -111,9 +111,11 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.Consider defining this method as an :term:`asynchronous generator`,which will be a requirement in a future version of Scrapy. However, if- you wish your spider middleware to work with Scrapy versions earlier- than Scrapy VERSION, :ref:`make your spider middleware universal- <universal-spider-middleware>` instead.+ you plan on sharing your spider middleware with other people, consider+ either :ref:`enforcing Scrapy VERSION <enforce-component-requirements>`+ as a minimum requirement of your spider middleware, or :ref:`making+ your spider middleware universal <universal-spider-middleware>` so that+ it works with Scrapy versions earlier than Scrapy VERSION.:param response: the response which generated this output from thespider