Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fixed comments
Jan Curn146d84f13dd2
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex 72f67308d..b4c8c5cd7 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -74,9 +74,9 @@ const DEFAULT_OPTIONS = {* This function that is called every time there are no requests being processed.* If it resolves to `true` then the crawler's run finishes.* See `isFinishedFunction` parameter of `AutoscaledPool`.- * @param {Number} [options.maxOpenPagesPerInstance=100] Maximum number of opened tabs per browser. If this limit is reached then a new+ * @param {Number} [options.maxOpenPagesPerInstance=50] Maximum number of opened tabs per browser. If this limit is reached then a new* browser instance is started. See `maxOpenPagesPerInstance` parameter of `PuppeteerPool`.- * @param {Number} [options.retireInstanceAfterRequestCount=150] Maximum number of requests that can be processed by a single browser instance.+ * @param {Number} [options.retireInstanceAfterRequestCount=100] Maximum number of requests that can be processed by a single browser instance.* After the limit is reached the browser will be retired and new requests will* be handled by a new browser instance.* See `retireInstanceAfterRequestCount` parameter of `PuppeteerPool`.diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.jsindex 0c186865e..6363496a1 100644--- a/src/puppeteer_pool.js+++ b/src/puppeteer_pool.js@@ -56,9 +56,9 @@ class PuppeteerInstance {* await puppeteerPool.destroy();* ```*- * @param {Number} [options.maxOpenPagesPerInstance=100] Maximum number of open tabs per browser. If this limit is reached then a new+ * @param {Number} [options.maxOpenPagesPerInstance=50] Maximum number of open tabs per browser. If this limit is reached then a new* browser will be started.- * @param {Number} [options.retireInstanceAfterRequestCount=150] Maximum number of requests that can be processed by a single browser instance.+ * @param {Number} [options.retireInstanceAfterRequestCount=100] Maximum number of requests that can be processed by a single browser instance.* After the limit is reached the browser will be retired and new requests will* be handled by a new browser instance.* @param {Number} [options.instanceKillerIntervalMillis=60000] How often opened Puppeteer instances are checked wheter they can be