Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merging develop
Marek Trunkat77fcdf58d36a
diff --cc CHANGELOG.mdindex 4cb00b6bd,44a1101c7..3be15cb3e--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,6 -1,25 +1,26 @@@NEXT====================+- Bump Puppeteer to 1.19.0 and use `{ pipe: true }` again because upstream bug has been fixed.+ - Add `useIncognitoPages` option to `PuppeteerPool` to enable opening new pages in incognito+ browser contexts. This is useful to keep cookies and cache unique for each page.++ 0.16.0 / 2019-08-15+ ====================+ - This release updates `@apify/http-request` to version 1.1.2.+ - Update `CheerioCrawler` to use `requestAsBrowser()` to better disguise as a real browser.+++ 0.15.5 / 2019-08-15+ ====================+ - This release just updates some dependencies (not Puppeteer).++ 0.15.4 / 2019-08-02+ ====================+ - **DEPRECATED**: `dataset.delete()`, `keyValueStore.delete()` and `requestQueue.delete()` methods have been+ deprecated in favor of `*.drop()` methods, because the `drop` name more clearly communicates the fact+ that those methods drop / delete the storage itself, not individual elements in the storage.+ - Added `Apify.utils.requestAsBrowser()` helper function that enables you to make HTTP(S) requests disguising+ as a browser (Firefox). This may help in overcoming certain anti-scraping and anti-bot protections.- Added `options.gotoTimeoutSecs` to `PuppeteerCrawler` to enable easier setting of navigation timeouts.- `PuppeteerPool` options that were deprecated from the `PuppeteerCrawler` constructor were finally removed.Please use `maxOpenPagesPerInstance`, `retireInstanceAfterRequestCount`, `instanceKillerIntervalSecs`,