Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: Mention initialConcurrency in v4 upgrading guide (#4245)
Jan Buchar5ca137fae89c
diff --git a/docs/upgrading/upgrading-v4/autoscaling.md b/docs/upgrading/upgrading-v4/autoscaling.mdindex 4f9fa05cb..d4acf6ee2 100644--- a/docs/upgrading/upgrading-v4/autoscaling.md+++ b/docs/upgrading/upgrading-v4/autoscaling.md@@ -127,7 +127,7 @@ try {### `autoscaledPoolOptions` is now `taskLoopOptions`, and no longer carries concurrency config-The crawler option was renamed — it was named after a class that is now internal — and narrowed to **only** the task-loop predicates `isFinishedFunction` and `isTaskReadyFunction`. Its type changed from `AutoscaledPoolOptions` to `TaskLoopPredicates`. Concurrency configuration goes through either the `minConcurrency` / `maxConcurrency` / `maxRequestsPerMinute` shortcuts (which configure the crawler's default `ConcurrencySystem`), or — for anything finer — a supplied `concurrencySystem`.+The crawler option was renamed — it was named after a class that is now internal — and narrowed to **only** the task-loop predicates `isFinishedFunction` and `isTaskReadyFunction`. Its type changed from `AutoscaledPoolOptions` to `TaskLoopPredicates`. Concurrency configuration goes through either the `minConcurrency` / `maxConcurrency` / `initialConcurrency` / `maxRequestsPerMinute` shortcuts (which configure the crawler's default `ConcurrencySystem`), or — for anything finer — a supplied `concurrencySystem`. The new `initialConcurrency` shortcut replaces `autoscaledPoolOptions.desiredConcurrency` and defaults to `minConcurrency`.Three options that used to live here — `maybeRunIntervalSecs`, `taskTimeoutSecs` and `log` — did *not* move to the `ConcurrencySystem` and have no replacement: the crawler's task-loop cadence is no longer configurable.@@ -160,6 +160,7 @@ The shortcuts cannot be combined with a supplied `concurrencySystem` — they coconst crawler = new CheerioCrawler({minConcurrency: 5,maxConcurrency: 50,+ initialConcurrency: 10,maxRequestsPerMinute: 120,requestHandler,});