Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: `isTaskReadyFunction` can be overridden (#3022)
Jindřich Bäre1b58ddc478c
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.tsindex e6a79f6f5..f0058d5d0 100644--- a/packages/basic-crawler/src/internals/basic-crawler.ts+++ b/packages/basic-crawler/src/internals/basic-crawler.ts@@ -270,9 +270,9 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw/*** Custom options passed to the underlying {@apilink AutoscaledPool} constructor.* > *NOTE:* The {@apilink AutoscaledPoolOptions.runTaskFunction|`runTaskFunction`}- * and {@apilink AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`} options- * are provided by the crawler and cannot be overridden.- * However, we can provide a custom implementation of {@apilink AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`}.+ * option is provided by the crawler and cannot be overridden.+ * However, we can provide custom implementations of {@apilink AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`}+ * and {@apilink AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`}.*/autoscaledPoolOptions?: AutoscaledPoolOptions;diff --git a/website/versioned_docs/version-3.13/api-typedoc.json b/website/versioned_docs/version-3.13/api-typedoc.jsonindex 4685fc260..24826c8b9 100644--- a/website/versioned_docs/version-3.13/api-typedoc.json+++ b/website/versioned_docs/version-3.13/api-typedoc.json@@ -4182,21 +4182,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -14824,21 +14824,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -116784,21 +116784,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -213691,21 +213691,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -224497,21 +224497,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -235857,21 +235857,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -276360,21 +276360,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -285996,21 +285996,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",@@ -313680,21 +313680,21 @@},{"kind": "text",- "text": "\nand "+ "text": "\noption is provided by the crawler and cannot be overridden.\nHowever, we can provide custom implementations of "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"+ "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"},{"kind": "text",- "text": " options\nare provided by the crawler and cannot be overridden.\nHowever, we can provide a custom implementation of "+ "text": "\nand "},{"kind": "inline-tag","tag": "@apilink",- "text": "AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`"+ "text": "AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`"},{"kind": "text",