Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improved PuppeteerCrawler and PuppeteerPool docs
Jan Curnc19bcdca8896
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex 95d18e98d..88c13bf49 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -13,14 +13,14 @@ const DEFAULT_OPTIONS = {/*** Provides a simple framework for parallel crawling of web pages- * using the headless Chrome with Puppeteer.- * The URLs of pages to visit are given by `Request` objects that are provided by the `RequestList` class- * or a dynamically enqueued requests provided by the `RequestQueue` class.+ * using headless Chrome with [Puppeteer](https://github.com/GoogleChrome/puppeteer).+ * The URLs of pages to visit are given by `Request` objects that are fed from a list (see `RequestList` class)+ * or from a dynamic queue (see `RequestQueue` class).*- * For each `Request` object to crawl the class opens a Chrome tab and then calls- * the function provided by user in the `handlePageFunction` option. New tasks are only- * started if there is enough free CPU capacity and memory available,- * internally using the `AutoscaledPool` instance.+ * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each `Request` object to crawl+ * and then calls the function provided by user as the `handlePageFunction` option.+ * New tasks are only started if there is enough free CPU and memory available,+ * using the `AutoscaledPool` class internally.** Basic usage:*@@ -28,6 +28,7 @@ const DEFAULT_OPTIONS = {* const crawler = new Apify.PuppeteerCrawler({* requestList,* handlePageFunction: async ({ page, request }) => {+ * // This function is called to extract data from a single web page* // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called* // 'request' is an instance of Request class with information about the page to load* await Apify.pushData({@@ -37,6 +38,7 @@ const DEFAULT_OPTIONS = {* })* },* handleFailedRequestFunction: async ({ request }) => {+ * // This function is called when crawling of a request failed too many time* await Apify.pushData({* url: request.url,* succeeded: false,@@ -48,52 +50,68 @@ const DEFAULT_OPTIONS = {* await crawler.run();* ```*- * @param {RequestList} [options.requestList] List of the requests to be processed.- * See the `requestList` parameter of `BasicCrawler` for more details.- * @param {RequestList} [options.requestQueue] Queue of the requests to be processed.- * See the `requestQueue` parameter of `BasicCrawler` for more details.- * @param {Function} [options.handlePageFunction] Function that is called to process each request.- * It is passed an object with the following fields:- * `request` is an instance of the `Request` object with details about the URL to open, HTTP method etc.- * `page` is an instance of the `Puppeteer.Page` class with `page.goto(request.url)` already called.- * @param {Number} [options.pageOpsTimeoutMillis=300000] Timeout in which the fuction passed as `options.handlePageFunction` needs to finish.- * @param {Function} [options.gotoFunction=({ request, page }) => page.goto(request.url)] Overrides the function that opens the request in Puppeteer.- * This function should return a result of `page.goto()`, i.e. the Puppeteer's `Response` object.+ * @param {RequestList} [options.requestList]+ * List of the requests to be processed.+ * See the `requestList` parameter of `BasicCrawler` for more details.+ * @param {RequestQueue} [options.requestQueue]+ * Queue of the requests to be processed.+ * See the `requestQueue` parameter of `BasicCrawler` for more details.+ * @param {Function} [options.handlePageFunction]+ * Function that is called to process each request.+ * It is passed an object with the following fields:+ * `request` is an instance of the `Request` object with details about the URL to open, HTTP method etc.+ * `page` is an instance of the `Puppeteer.Page` class with `page.goto(request.url)` already called.+ * @param {Number} [options.pageOpsTimeoutMillis=300000]+ * Timeout in which the function passed as `options.handlePageFunction` needs to finish.+ * @param {Function} [options.gotoFunction=({ request, page }) => page.goto(request.url)]+ * Overrides the function that opens the request in Puppeteer.+ * This function should return a result of `page.goto()`, i.e. the Puppeteer's `Response` object.+ * Note that one page is only used to process one request, and it is closed afterwards.* @param {Function} [options.handleFailedRequestFunction=({ request }) => log.error('Request failed', _.pick(request, 'url', 'uniqueKey'))]- * Function to handle requests that failed more than `option.maxRequestRetries` times. See the `handleFailedRequestFunction`- * parameter of `Apify.BasicCrawler` for details.- * @param {Number} [options.maxRequestRetries=3] How many times each request is retried if `handleRequestFunction` failed. See `maxRequestRetries`- * parameter of `BasicCrawler`.- * @param {Number} [options.maxMemoryMbytes] Maximum memory available for crawling. See `maxMemoryMbytes` parameter of `AutoscaledPool`.- * @param {Number} [options.maxConcurrency=1000] Maximum concurrency of request processing. See `maxConcurrency` parameter of `AutoscaledPool`.- * @param {Number} [options.minConcurrency=1] Minimum concurrency of requests processing. See `minConcurrency` parameter of `AutoscaledPool`.- * @param {Number} [options.minFreeMemoryRatio=0.2] Minimum ratio of free memory kept in the system. See `minFreeMemoryRatio` parameter of- * `AutoscaledPool`.- * @param {Function} [opts.isFinishedFunction] By default PuppeteerCrawler finishes when all the requests have been processed.- * You can override this behaviour by providing custom `isFinishedFunction`.- * This function that is called every time there are no requests being processed.- * If it resolves to `true` then the crawler's run finishes.- * See `isFinishedFunction` parameter of `AutoscaledPool`.- * @param {Number} [options.maxOpenPagesPerInstance=50] Maximum number of opened tabs per browser. If this limit is reached then a new- * browser instance is started. See `maxOpenPagesPerInstance` parameter of `PuppeteerPool`.- * @param {Number} [options.retireInstanceAfterRequestCount=100] Maximum number of requests that can be processed by a single browser instance.- * After the limit is reached the browser will be retired and new requests will- * be handled by a new browser instance.- * See `retireInstanceAfterRequestCount` parameter of `PuppeteerPool`.- * @param {Number} [options.instanceKillerIntervalMillis=60000] How often the launched Puppeteer instances are checked whether they can be- * closed. See `instanceKillerIntervalMillis` parameter of `PuppeteerPool`.- * @param {Number} [options.killInstanceAfterMillis=300000] If Puppeteer instance reaches the `options.retireInstanceAfterRequestCount` limit then- * it is considered retired and no more tabs will be opened. After the last tab is closed- * the whole browser is closed too. This parameter defines a time limit for inactivity- * after which the browser is closed even if there are pending tabs. See- * `killInstanceAfterMillis` parameter of `PuppeteerPool`.- * @param {Object} [options.puppeteerConfig={ dumpio: process.env.NODE_ENV !== 'production', slowMo: 0, args: []}] Default options for each- * new Puppeteer instance. See `puppeteerConfig` parameter of `PuppeteerPool`.+ * Function to handle requests that failed more than `option.maxRequestRetries` times. See the `handleFailedRequestFunction`+ * parameter of `Apify.BasicCrawler` for details.+ * @param {Number} [options.maxRequestRetries=3]+ * Indicates how many times each request is retried if `handleRequestFunction` failed.+ * See `maxRequestRetries` parameter of `BasicCrawler`.+ * @param {Number} [options.maxMemoryMbytes]+ * Maximum memory available for crawling. See `maxMemoryMbytes` parameter of `AutoscaledPool`.+ * @param {Number} [options.maxConcurrency=1000]+ * Maximum concurrency of request processing. See `maxConcurrency` parameter of `AutoscaledPool`.+ * @param {Number} [options.minConcurrency=1]+ * Minimum concurrency of requests processing. See `minConcurrency` parameter of `AutoscaledPool`.+ * @param {Number} [options.minFreeMemoryRatio=0.2]+ * Minimum ratio of free memory kept in the system. See `minFreeMemoryRatio` parameter of `AutoscaledPool`.+ * @param {Function} [opts.isFinishedFunction]+ * By default PuppeteerCrawler finishes when all the requests have been processed.+ * You can override this behaviour by providing custom `isFinishedFunction`.+ * This function that is called every time there are no requests being processed.+ * If it resolves to `true` then the crawler's run finishes.+ * See `isFinishedFunction` parameter of `AutoscaledPool`.+ * @param {Number} [options.maxOpenPagesPerInstance=50]+ * Maximum number of opened tabs per browser. If this limit is reached then a new+ * browser instance is started. See `maxOpenPagesPerInstance` parameter of `PuppeteerPool`.+ * @param {Number} [options.retireInstanceAfterRequestCount=100]+ * Maximum number of requests that can be processed by a single browser instance.+ * After the limit is reached the browser will be retired and new requests will+ * be handled by a new browser instance.+ * See `retireInstanceAfterRequestCount` parameter of `PuppeteerPool`.+ * @param {Number} [options.instanceKillerIntervalMillis=60000]+ * How often the launched Puppeteer instances are checked whether they can be+ * closed. See `instanceKillerIntervalMillis` parameter of `PuppeteerPool`.+ * @param {Number} [options.killInstanceAfterMillis=300000]+ * If Puppeteer instance reaches the `options.retireInstanceAfterRequestCount` limit then+ * it is considered retired and no more tabs will be opened. After the last tab is closed+ * the whole browser is closed too. This parameter defines a time limit for inactivity+ * after which the browser is closed even if there are pending tabs. See+ * `killInstanceAfterMillis` parameter of `PuppeteerPool`.+ * @param {Object} [options.puppeteerConfig={ dumpio: process.env.NODE_ENV !== 'production', slowMo: 0, args: []}]+ * Default options for each new `Puppeteer` instance. See `puppeteerConfig` parameter of `PuppeteerPool`.* @param {Function} [options.launchPuppeteerFunction=launchPuppeteerOptions => Apify.launchPuppeteer(launchPuppeteerOptions)]- * Overrides the default function to launch a new Puppeteer instance.- * See `launchPuppeteerFunction` parameter of `PuppeteerPool`.- * @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions] Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.- * See `launchPuppeteerOptions` parameter of `PuppeteerPool`.+ * Overrides the default function to launch a new Puppeteer instance.+ * See `launchPuppeteerFunction` parameter of `PuppeteerPool`.+ * @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]+ * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.+ * See `launchPuppeteerOptions` parameter of `PuppeteerPool`.*/export default class PuppeteerCrawler {constructor(opts) {diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.jsindex 6363496a1..5920f48af 100644--- a/src/puppeteer_pool.js+++ b/src/puppeteer_pool.js@@ -38,8 +38,10 @@ class PuppeteerInstance {}/**- * Provides a pool of Puppeteer (Chrome browser) instances.- * The class rotates the instances based on its configuration in order to change proxies.+ * Manages a pool of Chrome browser instances controlled by [Puppeteer](https://github.com/GoogleChrome/puppeteer).+ * `PuppeteerPool` rotates Chrome instances to change proxies+ * and other settings, in order to prevent detection of your web scraping bot,+ * access web pages from various countries etc.** Example usage:*@@ -56,20 +58,23 @@ class PuppeteerInstance {* await puppeteerPool.destroy();* ```*- * @param {Number} [options.maxOpenPagesPerInstance=50] Maximum number of open tabs per browser. If this limit is reached then a new- * browser will be started.- * @param {Number} [options.retireInstanceAfterRequestCount=100] Maximum number of requests that can be processed by a single browser instance.- * After the limit is reached the browser will be retired and new requests will- * be handled by a new browser instance.- * @param {Number} [options.instanceKillerIntervalMillis=60000] How often opened Puppeteer instances are checked wheter they can be- * closed.- * @param {Number} [options.killInstanceAfterMillis=300000] If Puppeteer instance reaches the `options.retireInstanceAfterRequestCount` limit then- * it is considered retired and no more tabs will be opened. After the last tab is closed the- * whole browser is closed too. This parameter defines a time limit for inactivity after- * which the browser is closed even if there are pending open tabs.+ * @param {Number} [options.maxOpenPagesPerInstance=50]+ * Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.+ * @param {Number} [options.retireInstanceAfterRequestCount=100]+ * Maximum number of requests that can be processed by a single browser instance.+ * After the limit is reached, the browser is retired and new requests are+ * be handled by a new browser instance.+ * @param {Number} [options.instanceKillerIntervalMillis=60000]+ * Indicates how often opened Puppeteer instances are checked whether they can be closed.+ * @param {Number} [options.killInstanceAfterMillis=300000]+ * When Puppeteer instance reaches the `options.retireInstanceAfterRequestCount` limit then+ * it is considered retired and no more tabs will be opened. After the last tab is closed the+ * whole browser is closed too. This parameter defines a time limit for inactivity after+ * which the browser is closed even if there are pending open tabs.* @param {Function} [options.launchPuppeteerFunction=launchPuppeteerOptions => Apify.launchPuppeteer(launchPuppeteerOptions)]- * Overrides the default function to launch a new Puppeteer instance.- * @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions] Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.+ * Overrides the default function to launch a new `Puppeteer` instance.+ * @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]+ * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.*/export default class PuppeteerPool {constructor(opts = {}) {