Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Regenerated docs

Jan Curn7a279b835d9a
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 54ee3c72f..a2743798d 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -105,7 +105,8 @@ await crawler.run();
request: Request,
response: Object // An instance of Node's http.IncomingMessage object,
autoscaledPool: AutoscaledPool
-}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
+}
+</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
<p> If the function returns a promise, it is awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
@@ -146,7 +147,8 @@ await crawler.run();
payload, // Provided by RequestList and/or RequestQueue
strictSSL, // Use options.ignoreSslErrors
proxy, // Use options.useApifyProxy or options.proxyUrls
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
@@ -155,7 +157,8 @@ await crawler.run();
<p> The function receives the following object as an argument:</p>
<pre><code>{
request: Request
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance
in place because there are already earlier references to it. Making a copy and returning it from
this function is therefore not supported, because it would create inconsistencies where
@@ -216,7 +219,8 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
for the default implementation of this function.</p>
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.md
index 5bf9c75ff..31d5f7294 100644
--- a/docs/api/Dataset.md
+++ b/docs/api/Dataset.md
@@ -190,15 +190,36 @@ Returns items in the dataset based on the provided parameters. The returned obje
<tr>
<td colspan="3"><p>If set to <code>true</code> then header row in CSV format is skipped.</p>
</td></tr><tr>
+<td><code>[options.clean]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>true</code> then the function returns only non-empty items and skips hidden fields (i.e. fields starting with <code>#</code> character).
+ Note that the <code>clean</code> parameter is a shortcut for <code>skipHidden: true</code> and <code>skipEmpty: true</code> options.</p>
+</td></tr><tr>
+<td><code>[options.skipHidden]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>true</code> then the function doesn&#39;t return hidden fields (fields starting with &quot;#&quot; character).</p>
+</td></tr><tr>
+<td><code>[options.skipEmpty]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>true</code> then the function doesn&#39;t return empty items.
+ Note that in this case the returned number of items might be lower than limit parameter and pagination must be done using the <code>limit</code> value.</p>
+</td></tr><tr>
<td><code>[options.simplified]</code></td><td><code>Boolean</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>If set to <code>true</code> then function applies the <code>fields: [&#39;url&#39;,&#39;pageFunctionResult&#39;,&#39;errorInfo&#39;]</code> and <code>unwind: &#39;pageFunctionResult&#39;</code> options.</p>
+<td colspan="3"><p>If <code>true</code> then function applies the <code>fields: [&#39;url&#39;,&#39;pageFunctionResult&#39;,&#39;errorInfo&#39;]</code> and <code>unwind: &#39;pageFunctionResult&#39;</code> options.
+ This feature is used to emulate simplified results provided by Apify API version 1 used for
+ the legacy Apify Crawler and it&#39;s not recommended to use it in new integrations.</p>
</td></tr><tr>
<td><code>[options.skipFailedPages]</code></td><td><code>Boolean</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>If set to <code>true</code> then all the items with errorInfo property will be skipped from the output.</p>
+<td colspan="3"><p>If <code>true</code> then, the all the items with errorInfo property will be skipped from the output.
+ This feature is here to emulate functionality of Apify API version 1 used for
+ the legacy Apify Crawler product and it&#39;s not recommended to use it in new integrations.</p>
</td></tr></tbody>
</table>
<a name="Dataset+getInfo"></a>
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index 14059d499..1bb368f22 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -94,7 +94,8 @@ await crawler.run();
page: Page,
puppeteerPool: PuppeteerPool,
autoscaledPool: AutoscaledPool
-}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
+}
+</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
<code>response</code> is an instance of the <code>Puppeteer</code>
<a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
<code>page</code> is an instance of the <code>Puppeteer</code>
@@ -155,7 +156,8 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}
+</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
diff --git a/docs/api/Request.md b/docs/api/Request.md
index 808308a20..dc9b4c48c 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -166,7 +166,8 @@ to override the default behavior and specify which URLs shall be considered equa
<pre><code> {
Accept: &#39;text/html&#39;,
&#39;Content-Type&#39;: &#39;application/json&#39;
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>
</tr>
<tr>
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index c98df283f..2b19689ec 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -104,7 +104,8 @@ await requestList.reclaimRequest(request2);
{ method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
// Batch import of URLs from a file hosted on the web
{ method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
-]</code></pre></td></tr><tr>
+]
+</code></pre></td></tr><tr>
<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
@@ -132,7 +133,8 @@ await requestList.reclaimRequest(request2);
&#39;unique-key-1&#39;: true,
&#39;unique-key-4&#39;: true,
},
-}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
+}
+</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
is to use the <code>stateKeyPrefix</code> parameter instead.</p>
</td></tr><tr>
<td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 668b15278..5c17b96cb 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -135,13 +135,14 @@ objects.
a combination of <code>url</code>, <code>method</code> and <code>payload</code> which enables crawling of websites that navigate using form submits
(POST requests).</p>
<p> <strong>Example:</strong></p>
-<pre><code class="language-javascript"> {
+<pre><code class="lang-javascript"> {
transformRequestFunction: (request) =&gt; {
request.userData.foo = &#39;bar&#39;;
request.useExtendedUniqueKey = true;
return request;
}
- }</code></pre>
+ }
+</code></pre>
</td></tr><tr>
<td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td>
</tr>
diff --git a/docs/api/utils.md b/docs/api/utils.md
index 8f62ca0ee..6163a4f48 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -126,13 +126,14 @@ objects.
<p> For example: by adding <code>keepUrlFragment: true</code> to the <code>request</code> object, URL fragments will not be removed
when <code>uniqueKey</code> is computed.</p>
<p> <strong>Example:</strong></p>
-<pre><code class="language-javascript"> {
+<pre><code class="lang-javascript"> {
transformRequestFunction: (request) =&gt; {
request.userData.foo = &#39;bar&#39;;
request.keepUrlFragment = true;
return request;
}
- }</code></pre>
+ }
+</code></pre>
</td></tr></tbody>
</table>
<a name="utils.requestAsBrowser"></a>
@@ -196,7 +197,7 @@ request. All `options` not recognized by this function are passed to it, so see
<tr>
<td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.
If function returns true request gets aborted. This function is passed to the
- (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>
+ (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request]">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>
</td></tr></tbody>
</table>
<a name="utils.sleep"></a>
diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.md
index 571bffacc..d3cee5406 100644
--- a/docs/typedefs/ActorRun.md
+++ b/docs/typedefs/ActorRun.md
@@ -55,7 +55,8 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;origin&quot;: &quot;API&quot;,
&quot;clientIp&quot;: &quot;1.2.3.4&quot;,
&quot;userAgent&quot;: &quot;ApifyClient/0.2.13 (Linux; Node/v8.11.3)&quot;
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>stats</code></td><td><code>Object</code></td>
</tr>
<tr>
@@ -64,7 +65,8 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;inputBodyLen&quot;: 22,
&quot;restartCount&quot;: 0,
&quot;workersUsed&quot;: 1,
- }</code></pre><p> Beware that object fields might change in future releases.</p>
+ }
+</code></pre><p> Beware that object fields might change in future releases.</p>
</td></tr><tr>
<td><code>options</code></td><td><code>Object</code></td>
</tr>
@@ -75,7 +77,8 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;waitSecs&quot;: 0,
&quot;memoryMbytes&quot;: 256,
&quot;diskMbytes&quot;: 512
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>buildId</code></td><td><code>String</code></td>
</tr>
<tr>
@@ -127,5 +130,6 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run"
&quot;body&quot;: {
&quot;message&quot;: &quot;Hello world!&quot;
}
- }</code></pre></td></tr></tbody>
+ }
+</code></pre></td></tr></tbody>
</table>