Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge branch 'master' into feature/desired-conc
Jan Curn44ef7a168f0d
diff --cc CHANGELOG.mdindex 844062c88,6a5775ad3..f8fe199c3--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,11 -1,33 +1,38 @@@- xxx+ NEXT====================+- Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited+- Better logging of memory overload errors+- Added `desiredConcurrency` option to `AutoscaledPool` constructor, removed+ unnecessary bound check from the setter property+- Upgraded `apify-client` to 0.5.22+ 0.15.2 / 2019-07-11+ ====================+ - Fix error where Puppeteer would fail to launch when pipes are turned off.+ - Switch back to default Web Socket transport for Puppeteer due to upstream issues.++ 0.15.1 / 2019-07-09+ ====================+ - **BREAKING CHANGE** Removed support for Web Driver (Selenium) since no further updates are planned.+ If you wish to continue using Web Driver, please stay on Apify SDK version ^0.14.15+ - **BREAKING CHANGE**: `Dataset.getData()` throws an error if user provides an unsupported option+ when using local disk storage.+ - **DEPRECATED**: `options.userData` of `Apify.utils.enqueueLinks()` is deprecated.+ Use `options.transformRequestFunction` instead.+ - Improve logging of memory overload errors.+ - Improve error message in `Apify.call()`.+ - Fix multiple log lines appearing when a crawler was about to finish.+ - Add `Apify.utils.puppeteer.enqueueLinksByClickingElements()` function which enables you+ to add requests to the queue from pure JavaScript navigations, form submissions etc.+ - Add `Apify.utils.puppeteer.infiniteScroll()` function which helps you with scrolling to the bottom+ of websites that auto-load new content.+ - The `RequestQueue.handledCount()` function has been resurrected from deprecation,+ in order to have compatible interface with `RequestList`.+ - Add `useExtendedUniqueKey` option to `Request` constructor to include `method` and `payload`+ in the `Request`'s computed `uniqueKey`.+ - Updated Puppeteer to 1.18.1+ - Updated `apify-client` to 0.5.22+0.14.15 / 2019-05-31====================- Fixes in `RequestQueue` to deal with inconsistencies in the underlying data storage