Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Merge branch 'master' into feature/session-pool-cheerio

Petr Patekfaebf6c86da1
diff --cc docs/api/CheerioCrawler.md
index 68d6ac696,0fe49ae90..fe455ad3b
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@@ -102,11 -102,9 +102,11 @@@ await crawler.run()
<pre><code>{
$: Cheerio, // the Cheerio object with parsed HTML
body: String|Buffer // the request body of the web page
- json: Object, // parsed JSON when Content-Type: application/json
+ // the parsed object from JSON string
+ // if the response contains the content type application/json
+ json: Object,
request: Request,
- contentType: Object, // Parsed `Content-Type` header
+ contentType: Object, // Parsed Content-Type header: { type, encoding }
response: Object // An instance of Node&#39;s http.IncomingMessage object,
autoscaledPool: AutoscaledPool
}</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p>
diff --cc src/crawlers/cheerio_crawler.js
index ef3f87078,9e879b63c..3816a837e
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@@ -127,11 -124,9 +127,11 @@@ const DEFAULT_OPTIONS =
* {
* $: Cheerio, // the Cheerio object with parsed HTML
* body: String|Buffer // the request body of the web page
- * json: Object, // parsed JSON when Content-Type: application/json
+ * // the parsed object from JSON string
+ * // if the response contains the content type application/json
+ * json: Object,
* request: Request,
- * contentType: Object, // Parsed `Content-Type` header
+ * contentType: Object, // Parsed Content-Type header: { type, encoding }
* response: Object // An instance of Node's http.IncomingMessage object,
* autoscaledPool: AutoscaledPool
* }