Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Bugfix in BasicCrawler - async calls in `isFinishedFunction` were not awaited

Jan Curn6e515362efa8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 878a6eb21..aa166c70f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,7 @@
+xxx
+====================
+- Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited
+
0.14.15 / 2019-05-31
====================
- Fixes in `RequestQueue` to deal with inconsistencies in the underlying data storage
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index b11984070..e08a40254 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -146,8 +146,9 @@ const DEFAULT_OPTIONS = {
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
* @param {Object} [options.autoscaledPoolOptions]
* Custom options passed to the underlying {@link AutoscaledPool} constructor.
- * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
* are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
* @param {Object} [options.minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
@@ -225,8 +226,8 @@ class BasicCrawler {
}
const isFinished = isFinishedFunction
- ? isFinishedFunction()
- : this._defaultIsFinishedFunction();
+ ? await isFinishedFunction()
+ : await this._defaultIsFinishedFunction();
if (isFinished) {
const reason = isFinishedFunction
diff --git a/src/request_list.js b/src/request_list.js
index c8a67d555..99b675a72 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -154,11 +154,12 @@ export class RequestList {
// Dictionary of requests that were returned by fetchNextRequest().
// The key is uniqueKey, value is true.
+ // TODO: Change this to Set
this.inProgress = {};
// Dictionary of requests for which reclaimRequest() was called.
- // The key is uniqueKey, value is true.
- // Note that reclaimedRequests is always a subset of inProgressRequests!
+ // The key is uniqueKey, value is true. TODO: Change this to Set
+ // Note that reclaimedRequests is always a subset of inProgress!
this.reclaimed = {};
this.persistStateKey = persistStateKey;