Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat: log desired concurrency in the default status message
Martin Adámek9f0b79684d9e
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.tsindex a8acc75b1..1cd5cefa6 100644--- a/packages/basic-crawler/src/internals/basic-crawler.ts+++ b/packages/basic-crawler/src/internals/basic-crawler.ts@@ -824,7 +824,7 @@ export class BasicCrawler<Context extends CrawlingContext = BasicCrawlingContextconst total = this.requestQueue?.getTotalCount() || this.requestList?.length();message = `Crawled ${this.stats.state.requestsFinished}${total ? `/${total}` : ''} pages, ${this.stats.state.requestsFailed- } failed requests.`;+ } failed requests, desired concurrency ${this.autoscaledPool?.desiredConcurrency ?? 0}.`;}if (this.statusMessageCallback) {