Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Build docs

Ondra Urban5eef3fca95fe
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index 3509280ee..b9a483a8a 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -215,7 +215,7 @@ For more information, see
the <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the app
or the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>.
-**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
+**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
<table>
<thead>
<tr>
@@ -338,7 +338,7 @@ To store the value to the default-key value store, you can use the [`Apify.setVa
For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)
and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).
-**Returns**: <code>Promise&lt;Object&gt;</code> - Returns a promise that resolves once the record is stored.
+**Returns**: <code>Promise&lt;Object&gt;</code> - Returns a promise that resolves once the record is stored.
<table>
<thead>
<tr>
@@ -404,9 +404,10 @@ by using the `apify/actor-node-chrome` base Docker image for your actor - see
<a href="https://www.apify.com/docs/actor#base-images" target="_blank">Apify Actor documentation</a>
for details.
-For an example of usage, see the [Synchronous run Example](../examples/synchronousrun) or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)
+For an example of usage, see the [Synchronous run Example](../examples/synchronousrun)
+or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)
-**Returns**: <code>Promise&lt;Browser&gt;</code> - Promise that resolves to Puppeteer's `Browser` instance.
+**Returns**: <code>Promise&lt;Browser&gt;</code> - Promise that resolves to Puppeteer's `Browser` instance.
<table>
<thead>
<tr>
diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.md
index abd4f761f..06a8b52fe 100644
--- a/docs/api/PseudoUrl.md
+++ b/docs/api/PseudoUrl.md
@@ -6,7 +6,7 @@ title: PseudoUrl
Represents a pseudo URL (PURL) - an URL pattern used by web crawlers
to specify which URLs should the crawler visit.
-This class is used by the [utils.puppeteer.enqueueLinks()](puppeteer#puppeteer.enqueueLinks) function.
+This class is used by the [`utils.puppeteer.enqueueLinks()`](puppeteer#puppeteer.enqueueLinks) function.
A PURL is simply a URL with special directives enclosed in `[]` brackets.
Currently, the only supported directive is `[RegExp]`,
@@ -33,7 +33,9 @@ http://www.example.com/search?do[load]=1
**Example usage:**
```javascript
-const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]');
+const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]', {
+ userData: { foo: 'bar' },
+});
if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');
```
@@ -63,7 +65,8 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M
<td><code>requestTemplate</code></td><td><code>Object</code></td>
</tr>
<tr>
-<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs.</p>
+<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs
+ by the <a href="puppeteer#puppeteer.enqueueLinks">utils.puppeteer.enqueueLinks()</a> function.</p>
</td></tr></tbody>
</table>
<a name="PseudoUrl+matches"></a>
@@ -89,7 +92,7 @@ Determines whether a URL matches this pseudo-URL pattern.
<a name="PseudoUrl+createRequest"></a>
## `pseudoUrl.createRequest(url)` ⇒ [<code>Request</code>](request)
-Creates a Request object from requestTemplate and given URL.
+Creates a Request object from a provided `requestTemplate` and given URL.
<table>
<thead>
diff --git a/docs/api/Request.md b/docs/api/Request.md
index c0038c977..af714ab38 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -102,6 +102,7 @@ const foo = request.userData.foo;
* [Request](request)
* [`new Request(options)`](#new_Request_new)
* [`.pushErrorMessage(errorOrMessage, [options])`](#Request+pushErrorMessage)
+ * [`.skip()`](#Request+skip)
<a name="new_Request_new"></a>
@@ -123,7 +124,7 @@ const foo = request.userData.foo;
<td><code>options.url</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>URL of the web page to crawl.</p>
+<td colspan="3"><p>URL of the web page to crawl. It must be a non-empty string.</p>
</td></tr><tr>
<td><code>[options.uniqueKey]</code></td><td><code>String</code></td><td></td>
</tr>
@@ -214,3 +215,10 @@ as possible, since just throwing a bad type error makes any debugging rather dif
<td colspan="3"><p>Only push the error message without stack trace when true.</p>
</td></tr></tbody>
</table>
+<a name="Request+skip"></a>
+
+## `request.skip()`
+Marks the request as skipped which in turn prevents [`BasicCrawler`](basiccrawler)
+(and {@PuppeteerCrawler} + {@CheerioCrawler}, since they use {@BasicCrawler} internally)
+from retrying the request after an error occurs.
+
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index b21523c48..2c2d19618 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -95,7 +95,7 @@ it will not be updated. You can find out whether this happened from the resultin
<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>Request object, or an Object to construct a Request from.</p>
+<td colspan="3"><p><a href="request"><code>Request</code></a> object, or an object to construct a <code>Request</code> instance from.</p>
</td></tr><tr>
<td><code>[options]</code></td><td><code>Object</code></td><td></td>
</tr>
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 236dde1c5..4169ca66a 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -133,7 +133,7 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse
*WARNING*: This is work in progress. Currently the function doesn't click elements and only takes their `href` attribute and so
is working only for link (`a`) elements, but not for buttons or JavaScript links.
-**Returns**: <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.
+**Returns**: <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.
<table>
<thead>
<tr>
@@ -262,7 +262,7 @@ Custom context may also be provided using the `context` parameter. To improve se
make sure to only pass the really necessary objects to the context. Preferably making
secured copies beforehand.
-**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`
+**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`
<table>
<thead>
<tr>
diff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.md
index 81c6c7413..9ba1bcedf 100644
--- a/docs/examples/Screenshots.md
+++ b/docs/examples/Screenshots.md
@@ -62,4 +62,4 @@ Apify.main(async () => {
console.log('Crawler finished.');
});
-```
+```
\ No newline at end of file
diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.md
index 1af3f2a52..09dd26c14 100644
--- a/docs/typedefs/QueueOperationInfo.md
+++ b/docs/typedefs/QueueOperationInfo.md
@@ -4,8 +4,9 @@ title: QueueOperationInfo
---
<a name="QueueOperationInfo"></a>
-A helper class that is used to report results from the
-[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks) function.
+A helper class that is used to report results from various
+[`RequestQueue`](../api/requestqueue) functions as well as
+[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks).
**Properties**
<table>
@@ -30,5 +31,10 @@ A helper class that is used to report results from the
</tr>
<tr>
<td colspan="3"><p>The ID of the added request</p>
+</td></tr><tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The original <code>Request</code> object passed to the <code>RequestQueue</code> function.</p>
</td></tr></tbody>
</table>