Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Refactor _ensureCorrectMaxMemory
Ondra Urban913be97b565b
diff --git a/src/autoscaling/snapshotter.js b/src/autoscaling/snapshotter.jsindex 96fd96723..7c689433e 100644--- a/src/autoscaling/snapshotter.js+++ b/src/autoscaling/snapshotter.js@@ -437,15 +437,14 @@ class Snapshotter {* @ignore*/async _ensureCorrectMaxMemory() {- if (!this.maxMemoryBytes) {- const { totalBytes } = await getMemoryInfo();- if (isAtHome()) {- this.maxMemoryBytes = totalBytes;- } else {- this.maxMemoryBytes = Math.ceil(totalBytes / 4);- // NOTE: Log as AutoscaledPool, so that users are not confused what "Snapshotter" is- log.info(`AutoscaledPool: Setting max memory of this run to ${Math.round(this.maxMemoryBytes / 1024 / 1024)} MB. Use the ${ENV_VARS.MEMORY_MBYTES} environment variable to override it.`); // eslint-disable-line max-len- }+ if (this.maxMemoryBytes) return;+ const { totalBytes } = await getMemoryInfo();+ if (isAtHome()) {+ this.maxMemoryBytes = totalBytes;+ } else {+ this.maxMemoryBytes = Math.ceil(totalBytes / 4);+ // NOTE: Log as AutoscaledPool, so that users are not confused what "Snapshotter" is+ log.info(`AutoscaledPool: Setting max memory of this run to ${Math.round(this.maxMemoryBytes / 1024 / 1024)} MB. Use the ${ENV_VARS.MEMORY_MBYTES} environment variable to override it.`); // eslint-disable-line max-len}}}