Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Build docs

Ondra Urban6e7116ef24bd
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index 59507545d..9b642a394 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -92,7 +92,8 @@ await crawler.run();
provided to the <code>options.handleFailedRequestFunction</code> parameter.
To make this work, you should <strong>always</strong>
let your function throw exceptions rather than catch them.
- The exceptions are logged to the request using the <a href="request.pusherrormessage"><code>Request.pushErrorMessage</code></a> function.</p>
+ The exceptions are logged to the request using the
+ <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
</td></tr><tr>
<td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
</tr>
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 0d3b66684..54ee3c72f 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -113,7 +113,8 @@ await crawler.run();
provided to the <code>options.handleFailedRequestFunction</code> parameter.
To make this work, you should <strong>always</strong>
let your function throw exceptions rather than catch them.
- The exceptions are logged to the request using the <a href="request.pusherrormessage"><code>Request.pushErrorMessage</code></a> function.</p>
+ The exceptions are logged to the request using the
+ <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
</td></tr><tr>
<td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
</tr>
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index d37341214..14059d499 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -108,7 +108,8 @@ await crawler.run();
provided to the <code>options.handleFailedRequestFunction</code> parameter.
To make this work, you should <strong>always</strong>
let your function throw exceptions rather than catch them.
- The exceptions are logged to the request using the <a href="request.pusherrormessage"><code>Request.pushErrorMessage</code></a> function.</p>
+ The exceptions are logged to the request using the
+ <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
</td></tr><tr>
<td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
</tr>
diff --git a/docs/api/Request.md b/docs/api/Request.md
index 3d397d0e6..808308a20 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -111,7 +111,6 @@ const foo = request.userData.foo;
- [Request](request)
- [`new Request(options)`](#new_Request_new)
- [`.pushErrorMessage(errorOrMessage, [options])`](#Request+pushErrorMessage)
- - ~~[`.doNotRetry([message])`](#Request+doNotRetry)~~
<a name="new_Request_new"></a>
@@ -225,27 +224,3 @@ debugging rather difficult.
<td colspan="3"><p>Only push the error message without stack trace when true.</p>
</td></tr></tbody>
</table>
-<a name="Request+doNotRetry"></a>
-
-## ~~`request.doNotRetry([message])`~~
-
-**_Deprecated_**
-
-Flags the request with no retry which prevents [`BasicCrawler`](basiccrawler) (as well as {@PuppeteerCrawler} and {@CheerioCrawler}, since they use
-{@BasicCrawler} internally) from retrying the request after an error occurs.
-
-Optionally accepts a message that will be used to construct and throw an Error.
-
-<table>
-<thead>
-<tr>
-<th>Param</th><th>Type</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td><code>[message]</code></td><td><code>String</code></td>
-</tr>
-<tr>
-</tr></tbody>
-</table>
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index 55bdb228e..e8f1888be 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -136,20 +136,20 @@ Gets the request from the queue specified by ID.
Returns a next request in the queue to be processed, or `null` if there are no more pending requests.
-Once you successfully finish processing of the request, you need to call [`RequestQueue.markRequestHandled`](requestqueue.markrequesthandled) to mark
-the request as handled in the queue. If there was some error in processing the request, call
-[`RequestQueue.reclaimRequest`](requestqueue.reclaimrequest) instead, so that the queue will give the request to some other consumer in another call
-to the `fetchNextRequest` function.
+Once you successfully finish processing of the request, you need to call [`requestQueue.markRequestHandled()`](#RequestQueue+markRequestHandled) to
+mark the request as handled in the queue. If there was some error in processing the request, call
+[`requestQueue.reclaimRequest()`](#RequestQueue+reclaimRequest) instead, so that the queue will give the request to some other consumer in another
+call to the `fetchNextRequest` function.
Note that the `null` return value doesn't mean the queue processing finished, it means there are currently no pending requests. To check whether all
-requests in queue were finished, use [`RequestQueue.isFinished`](requestqueue.isfinished) instead.
+requests in queue were finished, use [`requestQueue.isFinished()`](#RequestQueue+isFinished) instead.
**Returns**: [`Promise<Request>`](request) - Returns the request object or `null` if there are no more pending requests.
<a name="RequestQueue+markRequestHandled"></a>
## `requestQueue.markRequestHandled(request)` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)
-Marks a request that was previously returned by the [`RequestQueue.fetchNextRequest`](requestqueue.fetchnextrequest) function as handled after
+Marks a request that was previously returned by the [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest) function as handled after
successful processing. Handled requests will never again be returned by the `fetchNextRequest` function.
<table>
@@ -170,8 +170,8 @@ successful processing. Handled requests will never again be returned by the `fet
## `requestQueue.reclaimRequest(request, [options])` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)
Reclaims a failed request back to the queue, so that it can be returned for processed later again by another call to
-[`RequestQueue.fetchNextRequest`](requestqueue.fetchnextrequest). The request record in the queue is updated using the provided `request` parameter.
-For example, this lets you store the number of retries or error messages for the request.
+[`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest). The request record in the queue is updated using the provided `request`
+parameter. For example, this lets you store the number of retries or error messages for the request.
<table>
<thead>
@@ -193,16 +193,17 @@ For example, this lets you store the number of retries or error messages for the
</tr>
<tr>
<td colspan="3"><p>If <code>true</code> then the request it placed to the beginning of the queue, so that it&#39;s returned
-in the next call to <a href="requestqueue.fetchnextrequest"><code>RequestQueue.fetchNextRequest</code></a>. By default, it&#39;s put to the end of the queue.</p>
+in the next call to <a href="#RequestQueue+fetchNextRequest"><code>requestQueue.fetchNextRequest()</code></a>.
+By default, it&#39;s put to the end of the queue.</p>
</td></tr></tbody>
</table>
<a name="RequestQueue+isEmpty"></a>
## `requestQueue.isEmpty()` ⇒ `Promise<Boolean>`
-Resolves to `true` if the next call to [`RequestQueue.fetchNextRequest`](requestqueue.fetchnextrequest) would return `null`, otherwise it resolves to
-`false`. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that there is no
-activity in the queue, use [`RequestQueue.isFinished`](requestqueue.isfinished).
+Resolves to `true` if the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest) would return `null`, otherwise it resolves
+to `false`. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that there is
+no activity in the queue, use [`requestQueue.isFinished()`](#RequestQueue+isFinished).
<a name="RequestQueue+isFinished"></a>
diff --git a/docs/api/social.md b/docs/api/social.md
index e5e274d6c..5f576b038 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -388,8 +388,8 @@ The result of the function is an object with the following structure:
```
Note that the `phones` field contains phone numbers extracted from the special phone links such as `<a href="tel:+1234556789">call us</a>` (see
-[`phonesFromUrls`](phonesfromurls)) and potentially other sources with high certainty, while `phonesUncertain` contains phone numbers extracted from
-the plain text, which might be very inaccurate.
+[`social.phonesFromUrls()`](#social.phonesFromUrls)]) and potentially other sources with high certainty, while `phonesUncertain` contains phone
+numbers extracted from the plain text, which might be very inaccurate.
**Example usage:**
diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.md
index 3fd35f4cd..5e9396750 100644
--- a/docs/typedefs/QueueOperationInfo.md
+++ b/docs/typedefs/QueueOperationInfo.md
@@ -33,9 +33,9 @@ A helper class that is used to report results from various [`RequestQueue`](../a
<tr>
<td colspan="3"><p>The ID of the added request</p>
</td></tr><tr>
-<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td>
+<td><code>request</code></td><td><code>Object</code></td>
</tr>
<tr>
-<td colspan="3"><p>The original <code>Request</code> object passed to the <code>RequestQueue</code> function.</p>
+<td colspan="3"><p>The original <a href="../api/request"><code>Request</code></a> object passed to the <code>RequestQueue</code> function.</p>
</td></tr></tbody>
</table>