Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: notify autoscaled pool about newly added requests (#2400)

Martin Adámeka90177d52077
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 2a00b012a..57a6ec1a9 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -950,7 +950,14 @@ export class BasicCrawler<Context extends CrawlingContext = BasicCrawlingContext
*/
async addRequests(requests: (string | Source)[], options: CrawlerAddRequestsOptions = {}): Promise<CrawlerAddRequestsResult> {
const requestQueue = await this.getRequestQueue();
- return requestQueue.addRequestsBatched(requests, options);
+ const res = await requestQueue.addRequestsBatched(requests, options);
+
+ // Notify AutoscaledPool, if it exists, the method is called after the crawler started, and we don't want to wait
+ // this is important for crawlers in `keepAlive` mode that might have an empty queue and we don't want to wait till
+ // the next scheduled check (which might come in as much as 500ms by default).
+ await this.autoscaledPool?.notify();
+
+ return res;
}
/**
diff --git a/packages/core/src/autoscaling/autoscaled_pool.ts b/packages/core/src/autoscaling/autoscaled_pool.ts
index 77702e3e7..058e97d6b 100644
--- a/packages/core/src/autoscaling/autoscaled_pool.ts
+++ b/packages/core/src/autoscaling/autoscaled_pool.ts
@@ -448,6 +448,14 @@ export class AutoscaledPool {
this.isStopped = false;
}
+ /**
+ * Explicitly check the queue for new tasks. The AutoscaledPool checks the queue for new tasks periodically,
+ * every `maybeRunIntervalSecs` seconds. If you want to trigger the processing immediately, use this method.
+ */
+ async notify(): Promise<void> {
+ await this._maybeRunTask();
+ }
+
/**
* Starts a new task
* if the number of running tasks (current concurrency) is lower than desired concurrency