Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge branch 'develop' into feature/recycle-disk-cache
Ondra Urban28b0225f1dbc
diff --cc CHANGELOG.mdindex ae1159352,9707511f7..1cd5fc047--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,10 -1,30 +1,35 @@@+next+===================+- Added back the `recycleDiskCache` option to `PuppeteerPool` now that it is supported+ even in headless mode ([read more](https://bugs.chromium.org/p/chromium/issues/detail?id=882431))+xxx===================+ - **BREAKING CHANGE:** Added `puppeteerOperationTimeoutSecs` option to `PuppeteerPool`.+ It defaults to 15 seconds and all Puppeteer operations such as `browser.newPage()`+ or `puppeteer.launch()` will now time out. This is to prevent hanging requests.+ - **BREAKING CHANGE:** Added `handleRequestTimeoutSecs` option to `BasicCrawler` with a 60 second default.+ - **DEPRECATED:** `PuppeteerPool` options in the `PuppeteerCrawler` constructor are now deprecated.+ Please use the new `puppeteerPoolOptions` argument of type `Object` to pass them. `launchPuppeteerFunction`+ and `launchPuppeteerOptions` are still available as shortcuts for convenience.+ - `CheerioCrawler` and `PuppeteerCrawler` now automatically set `handleRequestTimeoutSecs` to double of+ `handlePageTimeoutSecs`.++ 0.12.4 / 2019-03-05+ ===================+ - Parameters `input` and `options` added to `Apify.callTask()`.++ 0.12.2 / 2019-02-27+ ===================+ - Added oldest active tab focusing to `PuppeteerPool` to combat resource throttling in Chromium.++ 0.12.1 / 2019-02-27+ ===================+ - Added `Apify.metamorph()`, see documentation for more information.+ - Added `Apify.getInput()`++ 0.12.0 / 2019-02-25+ ===================- **BREAKING CHANGE:** Reduced default `handlePageTimeoutSecs` for both `CheerioCrawler` and `PuppeteerCrawler` from 300 to 60 seconds,in order to prevent stalling crawlers.- **BREAKING CHANGE:** `PseudoUrl` now performs case-insensitive matching, even for the query string part of the URLs.diff --cc src/puppeteer_pool.jsindex f5b6050c4,f32f411b0..3b520a0f8--- a/src/puppeteer_pool.js+++ b/src/puppeteer_pool.js@@@ -146,10 -143,13 +143,13 @@@ class PuppeteerPoolmaxOpenPagesPerInstance,retireInstanceAfterRequestCount,launchPuppeteerFunction,+ puppeteerOperationTimeoutSecs,instanceKillerIntervalMillis,+ instanceKillerIntervalSecs,killInstanceAfterMillis,+ killInstanceAfterSecs,launchPuppeteerOptions,- // recycleDiskCache,+ recycleDiskCache,proxyUrls,} = _.defaults({}, options, DEFAULT_OPTIONS);