Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog [skip ci]
Ondra Urbanbbeeb266326d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f28c7dc7..3d266f094 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,7 @@NEXT====================+- **DEPRECATED:** `CheerioCrawlerOptions.requestOptions` is now deprecated. Please use+ `CheerioCrawlerOptions.prepareRequestFunction` instead.- Add `limit` option to `Apify.utils.enqueueLinks()` for situations when full crawls are not needed.- Add `responseEncoding` option to `CheerioCrawler` to allow users to provide a fall-back(or forced) encoding of responses in situations where websites serve invalid encoding info.diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex 1fea91cdb..114578888 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -371,6 +371,7 @@ class CheerioCrawler {if (requestOptions) {+ // DEPRECATED 2020-03-22this.requestOptions = requestOptions;log.deprecated('CheerioCrawler: options.requestOptions is deprecated. Use options.prepareRequestFunction instead.');}