Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Added visible_hostname to Squid configuration to avoid warnings
Jan Curne64486284bbb
diff --git a/src/proxy_chain.js b/src/proxy_chain.jsindex 7da068ac7..6ee54088a 100644--- a/src/proxy_chain.js+++ b/src/proxy_chain.js@@ -216,6 +216,7 @@ export class ProxyChain {// NOTE: need to set pid_filename to isolate our squid instance from others possibly running in the systemconst conf = `+visible_hostname apify-actor# debug_options 44,9 28,9http_access allow allnever_direct allow all