Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Build docs

Ondra Urbanf1bc61045e9a
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index cad848968..b3e1834f4 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -97,7 +97,7 @@ await crawler.run();
<p> The function receives the following object as an argument:</p>
<pre><code>{
$: Cheerio, // the Cheerio object with parsed HTML
- html: String // the raw HTML of the page
+ html: String // the raw HTML of the page, lazy loaded only when used
request: Request,
response: Object // An instance of Node&#39;s http.IncomingMessage object,
autoscaledPool: AutoscaledPool
diff --git a/docs/api/Request.md b/docs/api/Request.md
index fbe29475f..867ad3da0 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -74,10 +74,10 @@ const foo = request.userData.foo;
<tr>
<td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p>
</td></tr><tr>
-<td><code>retry</code></td><td><code>Boolean</code></td>
+<td><code>noRetry</code></td><td><code>Boolean</code></td>
</tr>
<tr>
-<td colspan="3"><p>Indicates whether the request will be automatically retried or not.</p>
+<td colspan="3"><p>The <code>true</code> value indicates that the request will not be automatically retried on error.</p>
</td></tr><tr>
<td><code>retryCount</code></td><td><code>Number</code></td>
</tr>
@@ -221,7 +221,7 @@ as possible, since just throwing a bad type error makes any debugging rather dif
## `request.doNotRetry([message])`
Flags the request with no retry which prevents [`BasicCrawler`](basiccrawler)
-(and {@PuppeteerCrawler} + {@CheerioCrawler}, since they use {@BasicCrawler} internally)
+(as well as {@PuppeteerCrawler} and {@CheerioCrawler}, since they use {@BasicCrawler} internally)
from retrying the request after an error occurs.
Optionally accepts a message that will be used to construct
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.js
index 43d43ab08..81483395d 100644
--- a/src/cheerio_crawler.js
+++ b/src/cheerio_crawler.js
@@ -109,7 +109,7 @@ const DEFAULT_OPTIONS = {
* ```
* {
* $: Cheerio, // the Cheerio object with parsed HTML
- * html: String // the raw HTML of the page
+ * html: String // the raw HTML of the page, lazy loaded only when used
* request: Request,
* response: Object // An instance of Node's http.IncomingMessage object,
* autoscaledPool: AutoscaledPool