Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix CheerioC requestFunction; Add autoscaledPoolOptions to PuppeteerC
Ondra Urban034415c10651
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2c26f137f..74164b950 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,7 +13,6 @@ xxx0.7.4 / 2018-09-18==================- Bug fixes and documentation improvements.-- Updated to [email protected]0.7.1 / 2018-09-14==================diff --git a/src/basic_crawler.js b/src/basic_crawler.jsindex b3a256082..daaacfd28 100644--- a/src/basic_crawler.js+++ b/src/basic_crawler.js@@ -68,7 +68,7 @@ const DEFAULT_OPTIONS = {* ```** @param {Object} options- * @param {Function} [options.handleRequestFunction]+ * @param {Function} options.handleRequestFunction* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.** The function that receives an object as argument, with the following field:@@ -78,10 +78,12 @@ const DEFAULT_OPTIONS = {* </ul>** The function must return a promise.- * @param {RequestList} [options.requestList]+ * @param {RequestList} options.requestList* Static list of URLs to be processed.- * @param {RequestQueue} [options.requestQueue]+ * Either RequestList or RequestQueue must be provided.+ * @param {RequestQueue} options.requestQueue* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.+ * Either RequestList or RequestQueue must be provided.* @param {Function} [options.handleFailedRequestFunction]* Function that handles requests that failed more then `option.maxRequestRetries` times.* See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11">GitHub</a> for default behavior.@@ -95,10 +97,10 @@ const DEFAULT_OPTIONS = {* Custom options passed to the underlying {@link AutoscaledPool|`AutoscaledPool`} instance constructor.* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options* are provided by `BasicCrawler` and cannot be overridden.- * @param {Object} [options.minConcurrency]- * Sets the minimum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.- * @param {Object} [options.maxConcurrency]- * Sets the maximum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.+ * @param {Object} [options.minConcurrency=1]+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding `AutoscaledPool` option.+ * @param {Object} [options.maxConcurrency=1000]+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding `AutoscaledPool` option.** @see {@link CheerioCrawler}* @see {@link PuppeteerCrawler}diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.jsindex 81d605860..0929e428c 100644--- a/src/cheerio_crawler.js+++ b/src/cheerio_crawler.js@@ -82,7 +82,7 @@ const DEFAULT_OPTIONS = {* ```** @param {Object} options- * @param {Function} [options.handlePageFunction]+ * @param {Function} options.handlePageFunction* User-provided function that performs the logic of the crawler. It is called for each page* loaded and parsed by the crawler.*@@ -96,13 +96,15 @@ const DEFAULT_OPTIONS = {** If the function returns a promise, it is awaited.*- * @param {RequestList} [options.requestList]+ * @param {RequestList} options.requestList* Static list of URLs to be processed.- * @param {RequestQueue} [options.requestQueue]+ * Either RequestList or RequestQueue must be provided.+ * @param {RequestQueue} options.requestQueue* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.+ * Either RequestList or RequestQueue must be provided.* @param {Function} [options.requestFunction]* Overrides the function that performs the HTTP request to get the raw HTML needed for Cheerio.- * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L246">GitHub</a> for default behavior.+ * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L264">GitHub</a> for default behavior.* @param {Number} [options.handlePageTimeoutSecs=300]* Timeout in which the function passed as `options.handlePageFunction` needs to finish, given in seconds.* @param {Number} [options.requestTimeoutSecs=30]@@ -112,7 +114,7 @@ const DEFAULT_OPTIONS = {* request function. If using a custom request function, user needs to implement this functionality.* @param {Function} [options.handleFailedRequestFunction]* Function that handles requests that failed more then `option.maxRequestRetries` times.- * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L12">GitHub</a> for default behavior.+ * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">GitHub</a> for default behavior.* @param {Number} [options.maxRequestRetries=3]* How many times the request is retried if either `requestFunction` or `handlePageFunction` failed.* @param {Number} [options.maxRequestsPerCrawl]@@ -123,10 +125,10 @@ const DEFAULT_OPTIONS = {* Custom options passed to the underlying {@link AutoscaledPool|`AutoscaledPool`} instance constructor.* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options* are provided by `CheerioCrawler` and cannot be overridden.- * @param {Object} [options.minConcurrency]- * Sets the minimum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.- * @param {Object} [options.maxConcurrency]- * Sets the maximum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.+ * @param {Object} [options.minConcurrency=1]+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding `AutoscaledPool` option.+ * @param {Object} [options.maxConcurrency=1000]+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding `AutoscaledPool` option.** @see {@link BasicCrawler}* @see {@link PuppeteerCrawler}@@ -163,10 +165,15 @@ export default class CheerioCrawler {this.requestFunction = async ({ request }) => {if (!this.isRunning) throw new Error('CheerioCrawler is stopped.');-- return requestFunction+ const rfPromise = requestFunction? requestFunction({ request }): this._defaultRequestFunction({ request });++ // Return the response of requestFunction or throw.+ return Promise.race([+ rfPromise,+ this.rejectOnAbortPromise,+ ]);};this.handlePageFunction = handlePageFunction;this.handlePageTimeoutMillis = handlePageTimeoutSecs * 1000;@@ -227,16 +234,23 @@ export default class CheerioCrawler {async _handleRequestFunction({ request }) {if (!this.isRunning) throw new Error('CheerioCrawler is stopped.');+ const rfPromise = this.requestFunction({ request });// rejectOnAbortPromise rejects when .abort() is called or BasicCrawler throws.// All running pages are therefore terminated with an error to be reclaimed and retried.- const html = await Promise.race([- this.requestFunction({ request }),+ const response = await Promise.race([+ rfPromise,createTimeoutPromise(this.requestTimeoutMillis, 'CheerioCrawler: requestFunction timed out.'),this.rejectOnAbortPromise,]);++ let html;+ if (typeof response === 'string') html = response;+ else if (typeof response === 'object' && typeof response.body === 'string') html = response.body;+ else throw new Error('CheerioCrawler: requestFunction returned neither string, nor an object with a body property of type string.');+const $ = cheerio.load(html);await Promise.race([- this.handlePageFunction({ $, html, request }),+ this.handlePageFunction({ $, html, request, response }),createTimeoutPromise(this.handlePageTimeoutMillis, 'CheerioCrawler: handlePageFunction timed out.'),this.rejectOnAbortPromise,]);@@ -247,16 +261,13 @@ export default class CheerioCrawler {* @ignore*/async _defaultRequestFunction({ request }) {- if (!this.isRunning) throw new Error('CheerioCrawler is stopped.');-- return Promise.race([- rp({- url: request.url,- method: request.method,- headers: request.headers,- strictSSL: !this.ignoreSslErrors,- }),- this.rejectOnAbortPromise,- ]);+ return rp({+ url: request.url,+ method: request.method,+ headers: request.headers,+ strictSSL: !this.ignoreSslErrors,+ resolveWithFullResponse: true,+ simple: false,+ });}}diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex b0a493a91..539a687b3 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -8,6 +8,11 @@ import { createTimeoutPromise } from './utils';const DEFAULT_OPTIONS = {gotoFunction: async ({ request, page }) => page.goto(request.url, { timeout: 60000 }),handlePageTimeoutSecs: 300,+ handleFailedRequestFunction: ({ request }) => {+ const details = _.pick(request, 'id', 'url', 'method', 'uniqueKey');++ log.error('PuppeteerCrawler: Request failed and reached maximum retries', details);+ },};const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;@@ -50,33 +55,37 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;** await crawler.run();* ```- *- * @param {RequestList} [options.requestList]- * List of the requests to be processed.- * See the `requestList` parameter of `BasicCrawler` for more details.- * @param {RequestQueue} [options.requestQueue]- * Queue of the requests to be processed.- * See the `requestQueue` parameter of `BasicCrawler` for more details.- * @param {Function} [options.handlePageFunction]+ * @param {Object} options+ * @param {Function} options.handlePageFunction* Function that is called to process each request.* It is passed an object with the following fields:* `request` is an instance of the `Request` object with details about the URL to open, HTTP method etc.* `page` is an instance of the `Puppeteer.Page` class with `page.goto(request.url)` already called.+ * @param {RequestList} options.requestList+ * List of the requests to be processed.+ * Either RequestList or RequestQueue must be provided.+ * See the `requestList` parameter of `BasicCrawler` for more details.+ * @param {RequestQueue} options.requestQueue+ * Queue of the requests to be processed.+ * Either RequestList or RequestQueue must be provided.+ * See the `requestQueue` parameter of `BasicCrawler` for more details.* @param {Number} [options.handlePageTimeoutSecs=300]* Timeout in which the function passed as `options.handlePageFunction` needs to finish, in seconds.- * @param {Function} [options.gotoFunction=({ request, page }) => page.goto(request.url, { timeout: 60000 })]- * Overrides the function that opens the request in Puppeteer.- * The function should return a result of Puppeteer's+ * @param {Function} [options.gotoFunction]+ * Overrides the function that opens the request in Puppeteer. The function should return a result of Puppeteer's* <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagegotourl-options">page.goto()</a> function,* i.e. a promise resolving to the <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-response">Response</a> object.*- * For example, this is useful if you need to extend the page load timeout or select a different criteria+ * For example, this is useful if you need to extend the page load timeout or select different criteria* to determine that the navigation succeeded.** Note that a single page object is only used to process a single request and it is closed afterwards.- * @param {Function} [options.handleFailedRequestFunction=({ request }) => log.error('Request failed', _.pick(request, 'url', 'uniqueKey'))]+ *+ * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L9">GitHub</a> for default behavior.+ * @param {Function} [options.handleFailedRequestFunction]* Function to handle requests that failed more than `option.maxRequestRetries` times. See the `handleFailedRequestFunction`* parameter of `Apify.BasicCrawler` for details.+ * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L13">GitHub</a> for default behavior.* @param {Number} [options.maxRequestRetries=3]* Indicates how many times each request is retried if `handleRequestFunction` failed.* See `maxRequestRetries` parameter of `BasicCrawler`.@@ -85,20 +94,6 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;* Always set this value in order to prevent infinite loops in misconfigured crawlers.* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.* See `maxRequestsPerCrawl` parameter of `BasicCrawler`.- * @param {Number} [options.maxMemoryMbytes]- * Maximum memory available for crawling. See `maxMemoryMbytes` parameter of `AutoscaledPool`.- * @param {Number} [options.maxConcurrency=1000]- * Maximum concurrency of request processing. See `maxConcurrency` parameter of `AutoscaledPool`.- * @param {Number} [options.minConcurrency=1]- * Minimum concurrency of requests processing. See `minConcurrency` parameter of `AutoscaledPool`.- * @param {Number} [options.minFreeMemoryRatio=0.2]- * Minimum ratio of free memory kept in the system. See `minFreeMemoryRatio` parameter of `AutoscaledPool`.- * @param {Function} [opts.isFinishedFunction]- * By default PuppeteerCrawler finishes when all the requests have been processed.- * You can override this behaviour by providing custom `isFinishedFunction`.- * This function that is called every time there are no requests being processed.- * If it resolves to `true` then the crawler's run finishes.- * See `isFinishedFunction` parameter of `AutoscaledPool`.* @param {Number} [options.maxOpenPagesPerInstance=50]* Maximum number of opened tabs per browser. If this limit is reached then a new* browser instance is started. See `maxOpenPagesPerInstance` parameter of `PuppeteerPool`.@@ -116,14 +111,21 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;* the whole browser is closed too. This parameter defines a time limit for inactivity* after which the browser is closed even if there are pending tabs. See* `killInstanceAfterMillis` parameter of `PuppeteerPool`.- * @param {Object} [options.puppeteerConfig={ dumpio: process.env.NODE_ENV !== 'production', slowMo: 0, args: []}]- * Default options for each new `Puppeteer` instance. See `puppeteerConfig` parameter of `PuppeteerPool`.- * @param {Function} [options.launchPuppeteerFunction=launchPuppeteerOptions => Apify.launchPuppeteer(launchPuppeteerOptions)]+ * @param {Function} [options.launchPuppeteerFunction]* Overrides the default function to launch a new Puppeteer instance.* See `launchPuppeteerFunction` parameter of `PuppeteerPool`.+ * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L9">GitHub</a> for default behavior.* @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]* Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.* See `launchPuppeteerOptions` parameter of `PuppeteerPool`.+ * @param {Object} [options.autoscaledPoolOptions]+ * Custom options passed to the underlying {@link AutoscaledPool|`AutoscaledPool`} instance constructor.+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options+ * are provided by `PuppeteerCrawler` and should not be overridden.+ * @param {Object} [options.minConcurrency=1]+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding `AutoscaledPool` option.+ * @param {Object} [options.maxConcurrency=1000]+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding `AutoscaledPool` option.** @see {@link CheerioCrawler}* @see {@link BasicCrawler}@@ -139,15 +141,12 @@ export default class PuppeteerCrawler {const {handlePageFunction,gotoFunction,- pageOpsTimeoutMillis,+ pageOpsTimeoutMillis, // Deprecated, remove in the future.handlePageTimeoutSecs,- // Autoscaled pool options- maxMemoryMbytes,+ // Autoscaled pool shorthandsmaxConcurrency,minConcurrency,- minFreeMemoryRatio,- isFinishedFunction,// Basic crawler optionsrequestList,@@ -155,6 +154,7 @@ export default class PuppeteerCrawler {maxRequestRetries,maxRequestsPerCrawl,handleFailedRequestFunction,+ autoscaledPoolOptions,// Puppeteer Pool optionsmaxOpenPagesPerInstance,@@ -196,12 +196,9 @@ export default class PuppeteerCrawler {handleFailedRequestFunction,// Autoscaled pool options.- maxMemoryMbytes,maxConcurrency,minConcurrency,- minFreeMemoryRatio,- isFinishedFunction,- ignoreMainProcess: true,+ autoscaledPoolOptions,});}