Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated comments
Jan Curn8f91ab19cab4
diff --git a/README-new.md b/README-new.mdindex 4253d989c..1d7d9a351 100644--- a/README-new.md+++ b/README-new.md@@ -108,7 +108,7 @@ The Apify SDK package provides the following tools:<li><a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerPool">PuppeteerPool</a>- Provides web browser tabs for user jobs- from an automatically-managed pool of Chrome browser instances, with configurable browser reuse and retirement policies.+ from an automatically-managed pool of Chrome browser instances, with configurable browser recycling and retirement policies.Supports reuse of the disk cache to speed up crawling of websites and reduce proxy bandwidth.</li><li>diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.jsindex 719393790..2a7e76703 100644--- a/src/puppeteer_pool.js+++ b/src/puppeteer_pool.js@@ -119,6 +119,8 @@ class PuppeteerInstance {* To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `options.launchPuppeteerOptions.args`,* where `X` is the approximate maximum number of bytes for disk cache.*+ * *IMPORTANT:* Currently this feature only works in headful mode, because of a bug in Chromium.+ ** The `options.recycleDiskCache` setting should not be used together with `--disk-cache-dir` argument in `options.launchPuppeteerOptions.args`.*/export default class PuppeteerPool {