Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Improve docs in CheerioCrawler requestOptions.

Ondra Urbanc65b85385f72
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index e9afcda34..3efd61bcd 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -143,16 +143,18 @@ const DEFAULT_OPTIONS = {
* Represents the options passed to
* <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.
* Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure
- * the operation of `CheerioCrawler` and all its options. If you need more granular control over
- * your requests, use {@link BasicCrawler}.
+ * the operation of `CheerioCrawler` and all its options. Headers will not be merged,
+ * use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the
+ * correct headers or use `options.prepareRequestFunction` to modify your {@link Request} dynamically.
+ * If you need more granular control over your requests, use {@link BasicCrawler}.
*
- * The mandatory internal defaults:
+ * The mandatory internal defaults that **CANNOT BE OVERRIDDEN** by `requestOptions`:
* ```
* {
- * url,
- * method,
- * headers,
- * payload, // Are provided by RequestList and/or RequestQueue
+ * url, // Provided by RequestList and/or RequestQueue
+ * method, // Provided by RequestList and/or RequestQueue
+ * headers, // Provided by RequestList and/or RequestQueue
+ * payload, // Provided by RequestList and/or RequestQueue
* strictSSL, // Use options.ignoreSslErrors
* proxy, // Use options.useApifyProxy or options.proxyUrls
* }