Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update comments maxConcurrency<->minConcurrency
JakubDrobnike3d371faf1a9
diff --git a/src/basic_crawler.js b/src/basic_crawler.jsindex 6c9f93766..4e87d18b2 100644--- a/src/basic_crawler.js+++ b/src/basic_crawler.js@@ -43,8 +43,8 @@ const DEFAULT_OPTIONS = {* Function to handle requests that failed more then option.maxRequestRetries times.* @param {Number} [options.maxRequestRetries=3] How many times request is retried if handleRequestFunction failed.* @param {Number} [options.maxMemoryMbytes] Maximal memory available in the system (see `maxMemoryMbytes` parameter of `Apify.AutoscaledPool`).- * @param {Number} [options.maxConcurrency=1] Minimal concurrency of requests processing (see `maxConcurrency` parameter of `Apify.AutoscaledPool`).- * @param {Number} [options.minConcurrency=1000] Maximal concurrency of request processing (see `minConcurrency` parameter of `Apify.AutoscaledPool`).+ * @param {Number} [options.maxConcurrency=1000] Maximal concurrency of request processing (see `maxConcurrency` parameter of `Apify.AutoscaledPool`).+ * @param {Number} [options.minConcurrency=1] Minimal concurrency of requests processing (see `minConcurrency` parameter of `Apify.AutoscaledPool`).* @param {Number} [options.minFreeMemoryRatio=0.2] Minumum ratio of free memory kept in the system.*/export default class BasicCrawler {diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex cc6837c35..97b656d75 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -56,8 +56,8 @@ const DEFAULT_OPTIONS = {* parameter of `Apify.BasicCrawler`)** @param {Number} [options.maxMemoryMbytes] Maximal memory available in the system (See `maxMemoryMbytes` parameter of `Apify.AutoscaledPool`).- * @param {Number} [options.maxConcurrency=1] Minimal concurrency of requests processing (See `maxConcurrency` parameter of `Apify.AutoscaledPool`).- * @param {Number} [options.minConcurrency=1000] Maximal concurrency of request processing (See `minConcurrency` parameter of `Apify.AutoscaledPool`).+ * @param {Number} [options.maxConcurrency=1000] Maximal concurrency of request processing (See `maxConcurrency` parameter of `Apify.AutoscaledPool`).+ * @param {Number} [options.minConcurrency=1] Minimal concurrency of requests processing (See `minConcurrency` parameter of `Apify.AutoscaledPool`).* @param {Number} [options.minFreeMemoryRatio=0.2] Minumum ratio of free memory kept in the system.** @param {Number} [options.maxOpenPagesPerInstance=100] Maximal number of opened tabs per browser. If limit is reached then the new