Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Limit the queue size in _parallel_asyncio(). (#7159)

Andrey Rakhmatullin7ed20ee7f32e
diff --git a/scrapy/utils/asyncio.py b/scrapy/utils/asyncio.py
index 15e7736ec..25237d0ba 100644
--- a/scrapy/utils/asyncio.py
+++ b/scrapy/utils/asyncio.py
@@ -80,7 +80,7 @@ async def _parallel_asyncio(
assumes that neither *callable* nor iterating *iterable* will raise an
exception.
"""
- queue: asyncio.Queue[_T | None] = asyncio.Queue()
+ queue: asyncio.Queue[_T | None] = asyncio.Queue(count * 2)
async def worker() -> None:
while True: