Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix request queue throttling at high concurrency
Ondra Urban7cc0dfeb3a54
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex d83e1b8d1..4733bea0d 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -191,9 +191,10 @@ For example, this lets you store the number of retries for the request.## `requestQueue.isEmpty()` ⇒ <code>Promise<Boolean></code>Resolves to `true` if the next call to [`fetchNextRequest`](#RequestQueue+fetchNextRequest) would return `null`, otherwise it resolves to `false`.Note that even if the queue is empty, there might be some pending requests currently being processed.+Due to the nature of distributed storage systems, the function might occasionally return a false negative,+but it will never return a false positive.-Due to the nature of distributed storage systems,-the function might occasionally return a false negative, but it should never return a false positive!+If you need to ensure that there is no activity in the queue, use [`isFinished`](#RequestQueue+isFinished).<a name="RequestQueue+isFinished"></a>diff --git a/src/request_queue.js b/src/request_queue.jsindex c0ba8156c..1dac54864 100644--- a/src/request_queue.js+++ b/src/request_queue.js@@ -6,6 +6,7 @@ import LruCache from 'apify-shared/lru_cache';import ListDictionary from 'apify-shared/list_dictionary';import { ENV_VARS, LOCAL_STORAGE_SUBDIRS, REQUEST_QUEUE_HEAD_MAX_LIMIT } from 'apify-shared/consts';import { delayPromise, checkParamPrototypeOrThrow } from 'apify-shared/utilities';+import log from 'apify-shared/log';import Promise from 'bluebird';import crypto from 'crypto';import Request from './request';@@ -384,16 +385,17 @@ export class RequestQueue {/*** Resolves to `true` if the next call to {@link RequestQueue#fetchNextRequest} would return `null`, otherwise it resolves to `false`.* Note that even if the queue is empty, there might be some pending requests currently being processed.+ * Due to the nature of distributed storage systems, the function might occasionally return a false negative,+ * but it will never return a false positive.*- * Due to the nature of distributed storage systems,- * the function might occasionally return a false negative, but it should never return a false positive!+ * If you need to ensure that there is no activity in the queue, use {@link RequestQueue#isFinished}.** @returns {Promise<Boolean>}*/isEmpty() {return this- ._ensureHeadIsNonEmpty(true)- .then(isHeadConsistent => isHeadConsistent && this.queueHeadDict.length() === 0);+ ._ensureHeadIsNonEmpty()+ .then(() => this.queueHeadDict.length() === 0);}/**@@ -502,6 +504,9 @@ export class RequestQueue {//// If limit was not reached in the call then there are no more requests to be returned.const shouldRepeatWithHigherLimit = !this.queueHeadDict.length() && limitReached && prevLimit < REQUEST_QUEUE_HEAD_MAX_LIMIT;+ if (prevLimit >= REQUEST_QUEUE_HEAD_MAX_LIMIT) {+ log.warning(`'RequestQueue: We've reached the maximum number of requests in progress: ${REQUEST_QUEUE_HEAD_MAX_LIMIT}.'`);+ }// If checkModifiedAt=true then we must ensure that queueModifiedAt is older than// queryStartedAt for at least API_PROCESSED_REQUESTS_DELAY_MILLIS.