Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
CheerioCrawler: Added doc for option json in handlePageFunction
drobnikj1e6f2e8bbbcb
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 075ec1e49..5f4c393c0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ NEXT- Fix `Apify.utils.puppeteer.blockRequests()` always including default patterns.- Fix parsing bad Content-Type headers for `CheerioCrawler`.- Added regexps for Youtube videos (`YOUTUBE_REGEX`, `YOUTUBE_REGEX_GLOBAL`) to `utils.social`+- Added documentation for option `json` in handlePageFunction of `CheerioCrawler`0.16.1 / 2019-10-31====================diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 51e80ec57..5979f83e6 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -102,6 +102,7 @@ await crawler.run();<pre><code>{$: Cheerio, // the Cheerio object with parsed HTMLbody: String|Buffer // the request body of the web page+ json: Object, // the parsed object from JSON string if the response contains the content type application/jsonrequest: Request,contentType: Object, // Parsed `Content-Type` headerresponse: Object // An instance of Node's http.IncomingMessage object,diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex 4e6ca9cca..8e3f3e472 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -123,6 +123,7 @@ const DEFAULT_OPTIONS = {* {* $: Cheerio, // the Cheerio object with parsed HTML* body: String|Buffer // the request body of the web page+ * json: Object, // the parsed object from JSON string if the response contains the content type application/json* request: Request,* contentType: Object, // Parsed `Content-Type` header* response: Object // An instance of Node's http.IncomingMessage object,