Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update Puppeteer to 1.13
Ondra Urban5d30adb9219b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9707511f7..c87a91c88 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,8 +7,10 @@ xxx- **DEPRECATED:** `PuppeteerPool` options in the `PuppeteerCrawler` constructor are now deprecated.Please use the new `puppeteerPoolOptions` argument of type `Object` to pass them. `launchPuppeteerFunction`and `launchPuppeteerOptions` are still available as shortcuts for convenience.-- `CheerioCrawler` and `PuppeteerCrawler` now automatically set `handleRequestTimeoutSecs` to double of- `handlePageTimeoutSecs`.+- `CheerioCrawler` and `PuppeteerCrawler` now automatically set `handleRequestTimeoutSecs` to 10 times+ their `handlePageTimeoutSecs`. This is a precaution that should keep requests from hanging forever.+- Added `options.prepareRequestFunction()` to `CheerioCrawler` constructor to enable modification+ of `Request` before the HTTP request is made to the target URL.0.12.4 / 2019-03-05===================diff --git a/package.json b/package.jsonindex fed3aa653..7c095a5e2 100644--- a/package.json+++ b/package.json@@ -94,7 +94,7 @@"why-is-node-running": "^2.0.3"},"optionalDependencies": {- "puppeteer": "1.12.2",+ "puppeteer": "1.13.0","selenium-webdriver": "^3.6.0"}}