Scrapy, a fast high-level web crawling & scraping framework for Python.
Add a list of Deferred-only APIs.
Andrey Rakhmatullinb93290f28aff
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 1c80857f6..439474310 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -62,18 +62,103 @@ In addition to native coroutine APIs Scrapy has some APIs that return a:class:`~twisted.internet.defer.Deferred` object or take a user-suppliedfunction that returns a :class:`~twisted.internet.defer.Deferred` object. TheseAPIs are also asynchronous but don't yet support native ``async def`` syntax.-For example:+In the future we plan to add support for the ``async def`` syntax to these APIs+or replace them with other APIs where changing the existing ones is+possible.++The following Scrapy methods return :class:`~twisted.internet.defer.Deferred`+objects (this list is not complete as it only includes methods that we think+may be useful for user code):++- :class:`scrapy.crawler.Crawler`:++ - :meth:`~scrapy.crawler.Crawler.crawl`++ - :meth:`~scrapy.crawler.Crawler.stop`++- :class:`scrapy.crawler.CrawlerRunner` (also inherited by+ :class:`scrapy.crawler.CrawlerProcess`):++ - :meth:`~scrapy.crawler.CrawlerRunner.crawl`++ - :meth:`~scrapy.crawler.CrawlerRunner.stop`++ - :meth:`~scrapy.crawler.CrawlerRunner.join`++- :class:`scrapy.core.engine.ExecutionEngine`:++ - :meth:`~scrapy.core.engine.ExecutionEngine.download`++- :class:`scrapy.signalmanager.SignalManager`:++ - :meth:`~scrapy.signalmanager.SignalManager.send_catch_log_deferred`++- :class:`~scrapy.mail.MailSender`++ - :meth:`~scrapy.mail.MailSender.send`++The following user-supplied methods can return+:class:`~twisted.internet.defer.Deferred` objects (the methods that can also+return coroutines are listed in :ref:`coroutine-support`):++- Custom download handlers (see :setting:`DOWNLOAD_HANDLERS`):++ - ``download_request()``++ - ``close()``-- The :meth:`ExecutionEngine.download` method returns a- :class:`~twisted.internet.defer.Deferred` object.-- A custom download handler needs to define a ``download_request()`` method that- returns a :class:`~twisted.internet.defer.Deferred` object.+- Custom downloader implementations (see :setting:`DOWNLOADER`):++ - ``fetch()``++- Custom scheduler implementations (see :setting:`SCHEDULER`):++ - :meth:`~scrapy.core.scheduler.BaseScheduler.open`++ - :meth:`~scrapy.core.scheduler.BaseScheduler.close`++- Custom dupefilters (see :setting:`DUPEFILTER_CLASS`):++ - ``open()``++ - ``close()``++- Custom feed storages (see :setting:`FEED_STORAGES`):++ - ``store()``++- Subclasses of :class:`scrapy.pipelines.media.MediaPipeline`:++ - ``media_to_download()``++ - ``item_completed()``++- Custom storages used by subclasses of+ :class:`scrapy.pipelines.files.FilesPipeline`:++ - ``persist_file()``++ - ``stat_file()``In most cases you can use these APIs in code that otherwise uses coroutines, bywrapping a :class:`~twisted.internet.defer.Deferred` object into a:class:`~asyncio.Future` object or vice versa. See :ref:`asyncio-await-dfd` formore information about this.+For example:++- The :meth:`ExecutionEngine.download()+ <scrapy.core.engine.ExecutionEngine.download>` method returns a+ :class:`~twisted.internet.defer.Deferred` object that fires with the+ downloaded response. You can use this object directly in Deferred-based+ code or convert it into a :class:`~asyncio.Future` object with+ :func:`~scrapy.utils.defer.maybe_deferred_to_future`.+- A custom download handler needs to define a ``download_request()`` method+ that returns a :class:`~twisted.internet.defer.Deferred` object. You can+ write a method that works with Deferreds and returns one directly, or you+ can write a coroutine and convert it into a functions that returns a+ Deferred with :func:`~scrapy.utils.defer.deferred_f_from_coro_f`.+General usage=============