Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Add throttling to critical memory overload warnings

Ondra Urbancc02840a1ddb
diff --git a/src/autoscaling/snapshotter.js b/src/autoscaling/snapshotter.js
index 75866e7c1..ebe1a7f81 100644
--- a/src/autoscaling/snapshotter.js
+++ b/src/autoscaling/snapshotter.js
@@ -284,7 +284,8 @@ class Snapshotter {
* @param {Object} systemInfo
*/
_memoryOverloadWarning({ memCurrentBytes }) {
- if (Date.now() < this.lastLoggedCriticalMemoryOverloadAt + CRITICAL_OVERLOAD_RATE_LIMIT_MILLIS) return;
+ const now = new Date();
+ if (now < this.lastLoggedCriticalMemoryOverloadAt + CRITICAL_OVERLOAD_RATE_LIMIT_MILLIS) return;
const maxDesiredMemoryBytes = this.maxUsedMemoryRatio * this.maxMemoryBytes;
const reserveMemory = this.maxMemoryBytes * (1 - this.maxUsedMemoryRatio) * RESERVE_MEMORY_RATIO;
@@ -295,6 +296,7 @@ class Snapshotter {
const toMb = bytes => Math.round(bytes / (1024 ** 2));
log.warning('Memory is critically overloaded. '
+ `Used ${toMb(memCurrentBytes)} MB of ${toMb(this.maxMemoryBytes)} MB (${usedPercentage}%)`);
+ this.lastLoggedCriticalMemoryOverloadAt = now;
}
}