Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document @inthread (#7956)

Kelly Su11fba8b4cc57
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 93a5fe239..4bc1831f9 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -223,6 +223,32 @@ means you can use many useful Python libraries that provide such code:
.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor,
you need to :ref:`wrap them<asyncio-await-dfd>`.
+Sometimes you need to call code that is not async-friendly at all, such as a
+third-party client library with no asynchronous support. Calling such code
+directly from a coroutine would block the underlying event loop, preventing
+Scrapy from doing any other work while it runs. To avoid this, you can wrap
+the blocking call with :func:`~scrapy.utils.decorators.inthread`, which runs
+it in a separate thread and returns an awaitable with the result:
+
+.. skip: next
+.. code-block:: python
+
+ from scrapy.utils.decorators import inthread
+
+
+ @inthread
+ def blocking_call(value):
+ # e.g. a call into a library with no async support
+ return some_sync_client.get(value)
+
+
+ class MySpider(Spider):
+ async def parse(self, response):
+ result = await blocking_call(response.url)
+ # ... use result to yield items and requests
+
+.. autofunction:: scrapy.utils.decorators.inthread
+
Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in