Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Fix some async docs examples. (#8404)

Andrey Rakhmatullin41a2ded027c6
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 4c488d66e..df07c7dae 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -208,8 +208,10 @@ means you can use many useful Python libraries that provide such code:
class MySpiderDeferred(Spider):
# ...
async def parse(self, response):
- additional_response = await treq.get("https://additional.url")
- additional_data = await treq.content(additional_response)
+ additional_response = await deferred_to_future(
+ treq.get("https://additional.url")
+ )
+ additional_data = await deferred_to_future(treq.content(additional_response))
# ... use response and additional_data to yield items and requests
@@ -249,7 +251,7 @@ it in a separate thread and returns an awaitable with the result:
class MySpider(Spider):
async def parse(self, response):
- result = await blocking_call(response.url)
+ result = await deferred_to_future(blocking_call(response.url))
# ... use result to yield items and requests
.. autofunction:: scrapy.utils.decorators.inthread
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index cc1b12d9f..41e6fc193 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -69,10 +69,8 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
.. skip: next
.. code-block:: python
- import json
-
+ import aiohttp
import scrapy
- import treq
class SignalSpider(scrapy.Spider):
@@ -87,13 +85,8 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
async def item_scraped(self, item):
# Send the scraped item to the server
- response = await treq.post(
- "http://example.com/post",
- json.dumps(item).encode("ascii"),
- headers={b"Content-Type": [b"application/json"]},
- )
-
- return response
+ async with aiohttp.ClientSession() as session:
+ await session.post("http://example.com/post", json=item)
def parse(self, response):
for quote in response.css("div.quote"):