Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix merge conflicts
Ondra Urban2748ffd3516d
diff --cc CHANGELOG.mdindex d9fe6fa57,f3e5f3d8c..bc8bc8b1d--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,9 -1,13 +1,15 @@@xxx===================+- Added `loadedUrl` property to `Request` that contains the `request.url` after redirects.+- Reduced default `handlePageTimeoutSecs` for both `CheerioCrawler` and `PuppeteerCrawler` to `60` seconds.+ - Added memory overload warning log message.- 0.11.7 / 2018/01/30+ 0.11.8 / 2019-02-05+ ===================+ - Fix the `keyValueStore.forEachKey()` method.+ - Fix version of `puppeteer` to prevent errors with automatic updates.++ 0.11.7 / 2019-01-30===================- Apify SDK now logs basic system info when `required`.- Added `utils.createRequestDebugInfo()` function to create a standardized debug info from request and response.