Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fix reusePages with request interception

Ondra Urban70bbb4e437f3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a36054387..60e347f6b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,7 @@
+0.11.3 / 2018-01-10
+===================
+- Fix `reusePages` causing Puppeteer to fail when used together with request interception.
+
0.11.2 / 2018-01-10
===================
- Fix missing `reusePages` configuration parameter in `PuppeteerCrawler`.
diff --git a/package.json b/package.json
index 4cc3bc197..7d7d2ff5f 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apify",
- "version": "0.11.2",
+ "version": "0.11.3",
"description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=8.14.0"
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index dd1c7c016..744d0f57e 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -594,8 +594,12 @@ class PuppeteerPool {
* @return {Promise}
*/
async recyclePage(page) {
- if (this.reusePages) this.idlePages.push(page);
- else await page.close();
+ if (this.reusePages) {
+ page.removeAllListeners();
+ this.idlePages.push(page);
+ } else {
+ await page.close();
+ }
}
}
diff --git a/test/puppeteer_pool.js b/test/puppeteer_pool.js
index bda946175..7f31e40c7 100644
--- a/test/puppeteer_pool.js
+++ b/test/puppeteer_pool.js
@@ -483,6 +483,10 @@ describe('PuppeteerPool', () => {
await pool.destroy();
});
+
+ xit('should work together with request interception', async () => {
+ // TODO
+ });
});
describe('the proxyUrls parameter', () => {