Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge master, resolve conflicts
Ondra Urbanb785fec40ced
diff --cc CHANGELOG.mdindex 57edcc808,fb3c964b9..c50a5779f--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,9 -1,11 +1,13 @@@- xxx+ NEXT====================- - Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited- - Better logging of memory overload errors+ - **BREAKING CHANGE** Removed support for Web Driver (Selenium) since no further updates are planned.+ If you wish to continue using Web Driver, please stay on Apify SDK version ^0.14.15+ - Improve logging of memory overload errors.+ - Fix multiple log lines appearing when a crawler was about to finish.+ - Add `Apify.utils.puppeteer.enqueueLinksByClickingElements()` function which enables you+ to add requests to the queue from pure JavaScript navigations, form submissions etc.+- The `RequestQueue.handledCount()` function has been resurrected from deprecation,+ in order to have compatible interface with `RequestList`.0.14.15 / 2019-05-31====================