Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Disabled squid cache in proxy_chain.js, bumped package.json version to 0.4.11

Jan Curn542e2507f8e4
diff --git a/package.json b/package.json
index 007a7feb8..58dc25893 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apify",
- "version": "0.4.10",
+ "version": "0.4.11",
"description": "Apify runtime for JavaScript",
"main": "build/index.js",
"keywords": [
diff --git a/src/proxy_chain.js b/src/proxy_chain.js
index d6253d4d0..a5f53829c 100644
--- a/src/proxy_chain.js
+++ b/src/proxy_chain.js
@@ -229,6 +229,10 @@ via off
forwarded_for transparent
access_log none
#access_log daemon:${path.join(this.tmpDir, 'access.log')} squid
+# An attempt to reduce memory consumption (proxy shouldn't store anything)
+cache deny all
+memory_pools off
+cache_mem 16 MB
cache_store_log none
cache_log /dev/null
#cache_log ${path.join(this.tmpDir, 'cache.log')}