Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated docs

Jan Curn20c2836b94e1
diff --git a/src/basic_crawler.js b/src/basic_crawler.js
index 86814e87e..88d012b94 100644
--- a/src/basic_crawler.js
+++ b/src/basic_crawler.js
@@ -74,12 +74,12 @@ const DEFAULT_OPTIONS = {
* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
*
* The function receives the following object as an argument:
- * ```
- * {
- * request: Request
- * }
- * ```
- * With the {@link Request} object representing the URL to crawl.
+ * ```
+ * {
+ * request: Request
+ * }
+ * ```
+ * where the {@link Request} instance represents the URL to crawl.
* The function must return a promise.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
@@ -88,10 +88,21 @@ const DEFAULT_OPTIONS = {
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
* Either `requestList` or `requestQueue` option must be provided (or both).
* @param {Function} [options.handleFailedRequestFunction]
- * Function that handles requests that failed more then `options.maxRequestRetries` times.
- * See source code on
- * <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">GitHub</a>
- * for default behavior.
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>
+ * for the default implementation of this function.
* @param {Number} [options.maxRequestRetries=3]
* Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.
* @param {Number} [options.maxRequestsPerCrawl]
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.js
index 3ff3b8d8b..99721adc0 100644
--- a/src/cheerio_crawler.js
+++ b/src/cheerio_crawler.js
@@ -93,14 +93,14 @@ const DEFAULT_OPTIONS = {
* loaded and parsed by the crawler.
*
* The function receives the following object as an argument:
- * ```
- * {
- * $: Cheerio, // the Cheerio object with parsed HTML
- * html: String // the raw HTML of the page
- * request: Request,
- * response: Object // a http.IncomingMessage object with properties such as the `statusCode`
- * }
- * ```
+ * ```
+ * {
+ * $: Cheerio, // the Cheerio object with parsed HTML
+ * html: String // the raw HTML of the page
+ * request: Request,
+ * response: Object // a http.IncomingMessage object with properties such as the `statusCode`
+ * }
+ * ```
* With the {@link Request} object representing the URL to crawl.
* If the function returns a promise, it is awaited.
* @param {RequestList} options.requestList
@@ -148,8 +148,20 @@ const DEFAULT_OPTIONS = {
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
* configuration options will cause an error to be thrown on startup.
* @param {Function} [options.handleFailedRequestFunction]
- * Function that handles requests that failed more then `option.maxRequestRetries` times.
- * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">GitHub</a> for default behavior.
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>
+ * for the default implementation of this function.
* @param {Number} [options.maxRequestRetries=3]
* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
* @param {Number} [options.maxRequestsPerCrawl]
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index aa2deb9cf..bf59c5901 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -77,14 +77,14 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;
* Function that is called to process each request.
* It is passed an object with the following fields:
*
- * ```
- * {
- * request: Request,
- * response: Response,
- * page: Page,
- * puppeteerPool: PuppeteerPool
- * }
- * ```
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * puppeteerPool: PuppeteerPool
+ * }
+ * ```
*
* `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
* `response` is an instance of the `Puppeteer`
@@ -115,10 +115,21 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;
* <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L9" target="_blank">GitHub</a>
* for default behavior.
* @param {Function} [options.handleFailedRequestFunction]
- * Function to handle requests that failed more than `option.maxRequestRetries` times.
- * See source code on
- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">GitHub</a>
- * for default behavior.
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * Where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>
+ * for the default implementation of this function.
* @param {Number} [options.maxRequestRetries=3]
* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
* @param {Number} [options.maxRequestsPerCrawl]