Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Better docs for CheerioCrawler's parsing behavior (#579)
Jan Čurn15df545d75bb
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 072e484f0..594ae2f1b 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -27,8 +27,13 @@ that a single URL is not crawled multiple times.The crawler finishes when there are no more [`Request`](request) objects to crawl.-By default, `CheerioCrawler` downloads HTML using the <a href="https://www.npmjs.com/package/request" target="_blank">request</a> NPM package. You can-use the `requestOptions` parameter to pass additional options to `request`.+`CheerioCrawler` downloads the web pages using the [`requestAsBrowser`](requestasbrowser) utility function. You can use the `requestOptions` parameter+to pass additional options to this function.++By default, `CheerioCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the+`Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the+[`additionalMimeTypes`](#new_CheerioCrawler_new) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of+content. For details, see [`CheerioCrawlerOptions#handlePageFunction`](cheeriocrawleroptions#handlepagefunction).New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the[`AutoscaledPool`](autoscaledpool) class. All [`AutoscaledPool`](autoscaledpool) configuration options can be passed to the `autoscaledPoolOptions`diff --git a/docs/typedefs/CheerioCrawlerOptions.md b/docs/typedefs/CheerioCrawlerOptions.mdindex 102c046c7..06683f7bf 100644--- a/docs/typedefs/CheerioCrawlerOptions.md+++ b/docs/typedefs/CheerioCrawlerOptions.md@@ -21,22 +21,30 @@ title: CheerioCrawlerOptions<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each pageloaded and parsed by the crawler.</p><p> The function receives the following object as an argument:</p>-<pre><code>{- $: Cheerio, // the Cheerio object with parsed HTML- body: String|Buffer // the request body of the web page- // the parsed object from JSON string- // if the response contains the content type application/json- json: Object,- request: Request,- contentType: Object, // Parsed Content-Type header: { type, encoding }- response: Object // An instance of Node's http.IncomingMessage object,- autoscaledPool: AutoscaledPool,- session: Session-}</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p>+<pre><code class="language-javascript">{+ // The Cheerio object's function with the parsed HTML.+ $: Cheerio,++// The request body of the web page, whose type depends on the content type. body: String|Buffer,++// The parsed object from JSON for responses with the "application/json" content types. // For other content types it's null. json:+Object,++// Apify.Request object with details of the requested web page request: Request,++// Parsed Content-Type HTTP header: { type, encoding } contentType: Object,++// An instance of Node's http.IncomingMessage object, response: Object,++// Underlying AutoscaledPool instance used to manage the concurrency of crawler autoscaledPool: AutoscaledPool,++// Session object, useful to work around anti-scraping protections session: Session }</code></pre>++<p> Type of <code>body</code> depends on the <code>Content-Type</code> header of the web page:</p><ul>-<li><p>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> mime types</p>+<li><p>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> MIME content types</p></li>-<li><p>Buffer for others mime types</p>+<li><p>Buffer for others MIME content types</p><p>Parsed <code>Content-Type</code> header using<a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>is stored in <code>contentType</code>.</p>@@ -69,8 +77,7 @@ The exceptions are logged to the request using the<td><code>[requestOptions]</code></td><td><code><a href="../typedefs/requestasbrowseroptions">RequestAsBrowserOptions</a></code></td><td></td></tr><tr>-<td colspan="3"><p>Represents the options passed to- <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.+<td colspan="3"><p>Represents the options passed to the <a href="requestasbrowser"><code>requestAsBrowser</code></a> function that makes the HTTP requests to fetch the web pages.Provided <code>requestOptions</code> are added to internal defaults that cannot be overridden to ensurethe operation of <code>CheerioCrawler</code> and all its options. Headers will not be merged,use <a href="requestlist"><code>RequestList</code></a> and/or <a href="requestqueue"><code>RequestQueue</code></a> to initialize your <a href="request"><code>Request</code></a> with thediff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex d66d484ca..c7cb78b16 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -66,23 +66,37 @@ const DEFAULT_OPTIONS = {* loaded and parsed by the crawler.** The function receives the following object as an argument:- * ```+ * ```javascript* {- * $: Cheerio, // the Cheerio object with parsed HTML- * body: String|Buffer // the request body of the web page- * // the parsed object from JSON string- * // if the response contains the content type application/json+ * // The Cheerio object's function with the parsed HTML.+ * $: Cheerio,+ *+ * // The request body of the web page, whose type depends on the content type.+ * body: String|Buffer,+ *+ * // The parsed object from JSON for responses with the "application/json" content types.+ * // For other content types it's null.* json: Object,+ *+ * // Apify.Request object with details of the requested web page* request: Request,- * contentType: Object, // Parsed Content-Type header: { type, encoding }- * response: Object // An instance of Node's http.IncomingMessage object,+ *+ * // Parsed Content-Type HTTP header: { type, encoding }+ * contentType: Object,+ *+ * // An instance of Node's http.IncomingMessage object,+ * response: Object,+ *+ * // Underlying AutoscaledPool instance used to manage the concurrency of crawler* autoscaledPool: AutoscaledPool,+ *+ * // Session object, useful to work around anti-scraping protections* session: Session* }* ```- * Type of `body` depends on web page `Content-Type` header.- * - String for `text/html`, `application/xhtml+xml`, `application/xml` mime types- * - Buffer for others mime types+ * Type of `body` depends on the `Content-Type` header of the web page:+ * - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types+ * - Buffer for others MIME content types** Parsed `Content-Type` header using* <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>@@ -109,8 +123,7 @@ const DEFAULT_OPTIONS = {* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.* Either `requestList` or `requestQueue` option must be provided (or both).* @property {RequestAsBrowserOptions} [requestOptions]- * Represents the options passed to- * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.+ * Represents the options passed to the {@link requestAsBrowser} function that makes the HTTP requests to fetch the web pages.* Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure* the operation of `CheerioCrawler` and all its options. Headers will not be merged,* use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the@@ -248,9 +261,15 @@ const DEFAULT_OPTIONS = {** The crawler finishes when there are no more {@link Request} objects to crawl.*- * By default, `CheerioCrawler` downloads HTML using the- * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> NPM package.- * You can use the `requestOptions` parameter to pass additional options to `request`.+ * `CheerioCrawler` downloads the web pages using the {@link requestAsBrowser} utility function.+ * You can use the `requestOptions` parameter to pass additional options to this function.+ *+ * By default, `CheerioCrawler` only processes web pages with the `text/html`+ * and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),+ * and skips pages with other content types. If you want the crawler to process other content types,+ * use the [`additionalMimeTypes`](#new_CheerioCrawler_new) constructor option.+ * Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.+ * For details, see {@link CheerioCrawlerOptions#handlePageFunction}.** New requests are only dispatched when there is enough free CPU and memory available,* using the functionality provided by the {@link AutoscaledPool} class.@@ -441,11 +460,17 @@ class CheerioCrawler {return dom && !isXml && $.html({ decodeEntities: false });},get json() {+ // TODO (JC): The naming here is quite unfortunate. The returned thing here is not a JSON,+ // but an object parsed from a JSON... It would make more sense to return the object from+ // `body` property instead. We could have a separate property for original buffer (e.g. called `content`)if (contentType.type !== 'application/json') return null;const jsonString = body.toString(contentType.encoding);return JSON.parse(jsonString);},get body() {+ // NOTE: For XML/HTML documents, we don't store the original body and only reconstruct it from Cheerio's DOM.+ // This is to save memory for high-concurrency crawls. The downside is that changes+ // made to DOM are reflected in the HTML, but we can live with that...if (dom) {return isXml ? $.xml() : $.html({ decodeEntities: false });}diff --git a/types/crawlers/cheerio_crawler.d.ts b/types/crawlers/cheerio_crawler.d.tsindex 0331793b6..6ee5c4de3 100644--- a/types/crawlers/cheerio_crawler.d.ts+++ b/types/crawlers/cheerio_crawler.d.ts@@ -5,23 +5,37 @@ export type CheerioCrawlerOptions = {* loaded and parsed by the crawler.** The function receives the following object as an argument:- * ```+ * ```javascript* {- * $: Cheerio, // the Cheerio object with parsed HTML- * body: String|Buffer // the request body of the web page- * // the parsed object from JSON string- * // if the response contains the content type application/json+ * // The Cheerio object's function with the parsed HTML.+ * $: Cheerio,+ *+ * // The request body of the web page, whose type depends on the content type.+ * body: String|Buffer,+ *+ * // The parsed object from JSON for responses with the "application/json" content types.+ * // For other content types it's null.* json: Object,+ *+ * // Apify.Request object with details of the requested web page* request: Request,- * contentType: Object, // Parsed Content-Type header: { type, encoding }- * response: Object // An instance of Node's http.IncomingMessage object,+ *+ * // Parsed Content-Type HTTP header: { type, encoding }+ * contentType: Object,+ *+ * // An instance of Node's http.IncomingMessage object,+ * response: Object,+ *+ * // Underlying AutoscaledPool instance used to manage the concurrency of crawler* autoscaledPool: AutoscaledPool,+ *+ * // Session object, useful to work around anti-scraping protections* session: Session* }* ```- * Type of `body` depends on web page `Content-Type` header.- * - String for `text/html`, `application/xhtml+xml`, `application/xml` mime types- * - Buffer for others mime types+ * Type of `body` depends on the `Content-Type` header of the web page:+ * - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types+ * - Buffer for others MIME content types** Parsed `Content-Type` header using* <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>@@ -54,8 +68,7 @@ export type CheerioCrawlerOptions = {*/requestQueue?: RequestQueue;/**- * Represents the options passed to- * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.+ * Represents the options passed to the {@link requestAsBrowser} function that makes the HTTP requests to fetch the web pages.* Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure* the operation of `CheerioCrawler` and all its options. Headers will not be merged,* use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the@@ -249,23 +262,37 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* loaded and parsed by the crawler.** The function receives the following object as an argument:- * ```+ * ```javascript* {- * $: Cheerio, // the Cheerio object with parsed HTML- * body: String|Buffer // the request body of the web page- * // the parsed object from JSON string- * // if the response contains the content type application/json+ * // The Cheerio object's function with the parsed HTML.+ * $: Cheerio,+ *+ * // The request body of the web page, whose type depends on the content type.+ * body: String|Buffer,+ *+ * // The parsed object from JSON for responses with the "application/json" content types.+ * // For other content types it's null.* json: Object,+ *+ * // Apify.Request object with details of the requested web page* request: Request,- * contentType: Object, // Parsed Content-Type header: { type, encoding }- * response: Object // An instance of Node's http.IncomingMessage object,+ *+ * // Parsed Content-Type HTTP header: { type, encoding }+ * contentType: Object,+ *+ * // An instance of Node's http.IncomingMessage object,+ * response: Object,+ *+ * // Underlying AutoscaledPool instance used to manage the concurrency of crawler* autoscaledPool: AutoscaledPool,+ *+ * // Session object, useful to work around anti-scraping protections* session: Session* }* ```- * Type of `body` depends on web page `Content-Type` header.- * - String for `text/html`, `application/xhtml+xml`, `application/xml` mime types- * - Buffer for others mime types+ * Type of `body` depends on the `Content-Type` header of the web page:+ * - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types+ * - Buffer for others MIME content types** Parsed `Content-Type` header using* <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>@@ -292,8 +319,7 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.* Either `requestList` or `requestQueue` option must be provided (or both).* @property {RequestAsBrowserOptions} [requestOptions]- * Represents the options passed to- * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.+ * Represents the options passed to the {@link requestAsBrowser} function that makes the HTTP requests to fetch the web pages.* Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure* the operation of `CheerioCrawler` and all its options. Headers will not be merged,* use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the@@ -430,9 +456,15 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi** The crawler finishes when there are no more {@link Request} objects to crawl.*- * By default, `CheerioCrawler` downloads HTML using the- * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> NPM package.- * You can use the `requestOptions` parameter to pass additional options to `request`.+ * `CheerioCrawler` downloads the web pages using the {@link requestAsBrowser} utility function.+ * You can use the `requestOptions` parameter to pass additional options to this function.+ *+ * By default, `CheerioCrawler` only processes web pages with the `text/html`+ * and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),+ * and skips pages with other content types. If you want the crawler to process other content types,+ * use the [`additionalMimeTypes`](#new_CheerioCrawler_new) constructor option.+ * Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.+ * For details, see {@link CheerioCrawlerOptions#handlePageFunction}.** New requests are only dispatched when there is enough free CPU and memory available,* using the functionality provided by the {@link AutoscaledPool} class.