Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update docs
Ondra Urbanb2d152a4e6a4
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex dcd9850fa..59507545d 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -122,7 +122,7 @@ await crawler.run();}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See- <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>+ <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>for the default implementation of this function.</p></td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 93d4b1179..0d3b66684 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -217,7 +217,7 @@ await crawler.run();error: Error,}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p>-<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>+<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>for the default implementation of this function.</p></td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex f40bf12ab..6683e5a95 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -151,7 +151,7 @@ await crawler.run();}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>+ <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L11" target="_blank">source code</a>for the default implementation of this function.</p></td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.jsindex 76887e5e4..435a8fba3 100644--- a/src/crawlers/basic_crawler.js+++ b/src/crawlers/basic_crawler.js@@ -136,7 +136,7 @@ const DEFAULT_OPTIONS = {* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>* for the default implementation of this function.* @param {Number} [options.maxRequestRetries=3]* Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex 40d81b06e..4aafe3609 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -215,7 +215,7 @@ const DEFAULT_OPTIONS = {* where the {@link Request} instance corresponds to the failed request, and the `Error` instance* represents the last error thrown during processing of the request.*- * See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>+ * See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>* for the default implementation of this function.* @param {Number} [options.maxRequestRetries=3]* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.jsindex 1cf19f8bf..0561d77f9 100644--- a/src/crawlers/puppeteer_crawler.js+++ b/src/crawlers/puppeteer_crawler.js@@ -146,7 +146,7 @@ const DEFAULT_OPTIONS = {* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L11" target="_blank">source code</a>* for the default implementation of this function.* @param {Number} [options.maxRequestRetries=3]* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.