Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Merge branch 'develop' into feature/reuse-tabs

Ondra Urban1c8b6c1dc4e3
diff --cc CHANGELOG.md
index d78195c7e,5eedc6f3c..3f8449ffc
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -3,8 -6,8 +6,10 @@@ xx
- Bugfix: PuppeteerPool was incorrectly overriding `proxyUrls` even if they were not defined.
- Fixed an issue where an error would be thrown when `datasetLocal.getData()` was invoked
with an overflowing offset. It now correctly returns an empty `Array`.
+- Added the `reusePages` option to `PuppeteerPool`. It will now reuse existing tabs by default
+ instead of opening new ones for each page.
+ - `BasicCrawler` (and therefore all Crawlers) now logs a message explaining why it finished.
+ - Fixed an issue where `maxRequestsPerCrawl` option would not be honored after restart or migration.
0.9.15 / 2018-11-30
===================
diff --cc src/puppeteer_crawler.js
index ada7dcb63,c430e0a4e..0bc6155f9
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@@ -323,7 -322,7 +322,7 @@@ class PuppeteerCrawler
return await Promise.race([pageOperationsPromise, this.rejectOnAbortPromise]);
} finally {
try {
- await Promise.race([this.puppeteerPool.recyclePage(page), createTimeoutPromise(PAGE_CLOSE_TIMEOUT_MILLIS, 'Operation timed out.')]);
- await addTimeoutToPromise(page.close(), PAGE_CLOSE_TIMEOUT_MILLIS, 'Operation timed out.');
++ await addTimeoutToPromise(this.puppeteerPool.recyclePage(page), PAGE_CLOSE_TIMEOUT_MILLIS, 'Operation timed out.');
} catch (err) {
log.debug('PuppeteerCrawler: Page.close() failed.', { reason: err && err.message });
}