Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fix reusePages memory leak and config

Ondra Urban2ec3f277ce26
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 29cf6d5ca..a1f3ff31a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,8 @@
+xxx
+===================
+- Fix missing `reusePages` configuration parameter in `PuppeteerCrawler`.
+- Fix a memory leak where `reusePages` would prevent browsers from closing.
+
0.11.1 / 2018-01-07
===================
- Fix missing `autoscaledPool` parameter in `handlePageFunction` of `PuppeteerCrawler`.
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index ab3dbb975..e03bb4b81 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -147,6 +147,10 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @param {boolean} [options.reusePages=false]
+ * Individual browser tabs will be reused after their task is complete instead
+ * of closing them and spawning a new tab. This saves CPU resources.
+ * See `reusePages` parameter of {@link PuppeteerPool}.
* @param {Number} [options.maxOpenPagesPerInstance=50]
* Maximum number of opened tabs per browser. If this limit is reached then a new
* browser instance is started. See `maxOpenPagesPerInstance` parameter of {@link PuppeteerPool}.
@@ -218,6 +222,7 @@ class PuppeteerCrawler {
// PuppeteerPool options
// TODO: We should put these into a single object, similarly to autoscaledPoolOptions
+ reusePages,
maxOpenPagesPerInstance,
retireInstanceAfterRequestCount,
instanceKillerIntervalMillis,
@@ -238,6 +243,7 @@ class PuppeteerCrawler {
this.handlePageTimeoutMillis = handlePageTimeoutSecs * 1000 || pageOpsTimeoutMillis;
this.puppeteerPoolOptions = {
+ reusePages,
maxOpenPagesPerInstance,
retireInstanceAfterRequestCount,
instanceKillerIntervalMillis,
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index 1d6a1fe30..dd1c7c016 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -440,6 +440,10 @@ class PuppeteerPool {
this._incrementPageCount(instance);
return idlePage;
}
+ // Close pages of retired instances so they don't keep hanging there forever.
+ if (pageIsNotClosed && !instanceIsActive) {
+ await idlePage.close();
+ }
}
// If there are no live pages to be reused, we spawn a new tab.
return this._openNewTab();
diff --git a/test/puppeteer_pool.js b/test/puppeteer_pool.js
index 3385e63f4..109082610 100644
--- a/test/puppeteer_pool.js
+++ b/test/puppeteer_pool.js
@@ -419,6 +419,40 @@ describe('PuppeteerPool', () => {
await pool.destroy();
});
+ it('should close pages in retired instances', async () => {
+ const pool = new Apify.PuppeteerPool({
+ retireInstanceAfterRequestCount: 1,
+ reusePages: true,
+ });
+
+ // Open 3 pages, should have 3 retired instances later.
+ const pages = [
+ await pool.newPage(),
+ await pool.newPage(),
+ await pool.newPage(),
+ ];
+
+ let closeCounter = 0;
+
+ // Recycle all pages, this should just make them idle.
+ for (const p of pages) {
+ const { close } = p;
+ p.close = (...args) => { // eslint-disable-line no-loop-func
+ closeCounter++;
+ close.apply(p, ...args);
+ };
+ await pool.recyclePage(p);
+ }
+
+ // Get one new page. This should flush all the idle ones because
+ // they are in retired instances.
+ await pool.newPage();
+
+ expect(closeCounter).to.be.eql(3);
+
+ await pool.destroy();
+ });
+
it('should skip closed pages', async () => {
const pool = new Apify.PuppeteerPool({ reusePages: true });