Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Disabled squid cache in proxy_chain.js, bumped package.json version to 0.4.11
Jan Curn542e2507f8e4
diff --git a/package.json b/package.jsonindex 007a7feb8..58dc25893 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "apify",- "version": "0.4.10",+ "version": "0.4.11","description": "Apify runtime for JavaScript","main": "build/index.js","keywords": [diff --git a/src/proxy_chain.js b/src/proxy_chain.jsindex d6253d4d0..a5f53829c 100644--- a/src/proxy_chain.js+++ b/src/proxy_chain.js@@ -229,6 +229,10 @@ via offforwarded_for transparentaccess_log none#access_log daemon:${path.join(this.tmpDir, 'access.log')} squid+# An attempt to reduce memory consumption (proxy shouldn't store anything)+cache deny all+memory_pools off+cache_mem 16 MBcache_store_log nonecache_log /dev/null#cache_log ${path.join(this.tmpDir, 'cache.log')}