Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge branch 'develop' into feature/reuse-tabs
Ondra Urban1c8b6c1dc4e3
diff --cc CHANGELOG.mdindex d78195c7e,5eedc6f3c..3f8449ffc--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -3,8 -6,8 +6,10 @@@ xx- Bugfix: PuppeteerPool was incorrectly overriding `proxyUrls` even if they were not defined.- Fixed an issue where an error would be thrown when `datasetLocal.getData()` was invokedwith an overflowing offset. It now correctly returns an empty `Array`.+- Added the `reusePages` option to `PuppeteerPool`. It will now reuse existing tabs by default+ instead of opening new ones for each page.+ - `BasicCrawler` (and therefore all Crawlers) now logs a message explaining why it finished.+ - Fixed an issue where `maxRequestsPerCrawl` option would not be honored after restart or migration.0.9.15 / 2018-11-30===================diff --cc src/puppeteer_crawler.jsindex ada7dcb63,c430e0a4e..0bc6155f9--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@@ -323,7 -322,7 +322,7 @@@ class PuppeteerCrawlerreturn await Promise.race([pageOperationsPromise, this.rejectOnAbortPromise]);} finally {try {- await Promise.race([this.puppeteerPool.recyclePage(page), createTimeoutPromise(PAGE_CLOSE_TIMEOUT_MILLIS, 'Operation timed out.')]);- await addTimeoutToPromise(page.close(), PAGE_CLOSE_TIMEOUT_MILLIS, 'Operation timed out.');++ await addTimeoutToPromise(this.puppeteerPool.recyclePage(page), PAGE_CLOSE_TIMEOUT_MILLIS, 'Operation timed out.');} catch (err) {log.debug('PuppeteerCrawler: Page.close() failed.', { reason: err && err.message });}