Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog
Ondra Urban1d6bd403e11f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8e91d3fdf..a3957865d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,13 @@+xxx+===================+- **BREAKING CHANGE:** Added `puppeteerOperationTimeoutSecs` option to `PuppeteerPool`.+ It defaults to 15 seconds and all Puppeteer operations such as `browser.newPage()`+ or `puppeteer.launch()` will now time out. This is to prevent hanging requests.+- **BREAKING CHANGE:** Added `handleRequestTimeoutSecs` option to `BasicCrawler` with a 60 second default.+- `CheerioCrawler` and `PuppeteerCrawler` now automatically set `handleRequestTimeoutSecs` to double of+ `handlePageTimeoutSecs`.+- `puppeteerPool.newPage()` now retries twice before throwing an error.+0.12.2 / 2019-02-27===================- Added oldest active tab focusing to `PuppeteerPool` to combat resource throttling in Chromium.