Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urbanf1bc61045e9a
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex cad848968..b3e1834f4 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -97,7 +97,7 @@ await crawler.run();<p> The function receives the following object as an argument:</p><pre><code>{$: Cheerio, // the Cheerio object with parsed HTML- html: String // the raw HTML of the page+ html: String // the raw HTML of the page, lazy loaded only when usedrequest: Request,response: Object // An instance of Node's http.IncomingMessage object,autoscaledPool: AutoscaledPooldiff --git a/docs/api/Request.md b/docs/api/Request.mdindex fbe29475f..867ad3da0 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -74,10 +74,10 @@ const foo = request.userData.foo;<tr><td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p></td></tr><tr>-<td><code>retry</code></td><td><code>Boolean</code></td>+<td><code>noRetry</code></td><td><code>Boolean</code></td></tr><tr>-<td colspan="3"><p>Indicates whether the request will be automatically retried or not.</p>+<td colspan="3"><p>The <code>true</code> value indicates that the request will not be automatically retried on error.</p></td></tr><tr><td><code>retryCount</code></td><td><code>Number</code></td></tr>@@ -221,7 +221,7 @@ as possible, since just throwing a bad type error makes any debugging rather dif## `request.doNotRetry([message])`Flags the request with no retry which prevents [`BasicCrawler`](basiccrawler)-(and {@PuppeteerCrawler} + {@CheerioCrawler}, since they use {@BasicCrawler} internally)+(as well as {@PuppeteerCrawler} and {@CheerioCrawler}, since they use {@BasicCrawler} internally)from retrying the request after an error occurs.Optionally accepts a message that will be used to constructdiff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.jsindex 43d43ab08..81483395d 100644--- a/src/cheerio_crawler.js+++ b/src/cheerio_crawler.js@@ -109,7 +109,7 @@ const DEFAULT_OPTIONS = {* ```* {* $: Cheerio, // the Cheerio object with parsed HTML- * html: String // the raw HTML of the page+ * html: String // the raw HTML of the page, lazy loaded only when used* request: Request,* response: Object // An instance of Node's http.IncomingMessage object,* autoscaledPool: AutoscaledPool