Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Build docs

Ondra Urbandfb346a66fee
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index 6f54b77fd..86c5ee368 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -91,8 +91,7 @@ await crawler.run();
<pre><code>{
request: Request,
autoscaledPool: AutoscaledPool
-}
-</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
<p> The function must return a promise, which is then awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
@@ -127,8 +126,7 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}
-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 6926febe8..0d4160221 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -108,8 +108,7 @@ await crawler.run();
request: Request,
response: Object // An instance of Node&#39;s http.IncomingMessage object,
autoscaledPool: AutoscaledPool
-}
-</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
+}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
<p> If the function returns a promise, it is awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
@@ -147,8 +146,7 @@ await crawler.run();
payload, // Are provided by RequestList and/or RequestQueue
strictSSL, // Use options.ignoreSslErrors
proxy, // Use options.useApifyProxy or options.proxyUrls
- }
-</code></pre></td></tr><tr>
+ }</code></pre></td></tr><tr>
<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
@@ -157,8 +155,7 @@ await crawler.run();
<p> The function receives the following object as an argument:</p>
<pre><code>{
request: Request
-}
-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance
in place because there are already earlier references to it. Making a copy and returning it from
this function is therefore not supported, because it would create inconsistencies where
@@ -219,8 +216,7 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}
-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>
for the default implementation of this function.</p>
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index d9c2a5221..059741515 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -99,8 +99,7 @@ await crawler.run();
page: Page,
puppeteerPool: PuppeteerPool,
autoscaledPool: AutoscaledPool
-}
-</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
+}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
<code>response</code> is an instance of the <code>Puppeteer</code>
<a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
<code>page</code> is an instance of the <code>Puppeteer</code>
@@ -154,8 +153,7 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}
-</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>
diff --git a/docs/api/Request.md b/docs/api/Request.md
index a3edbc7a0..dd98f518a 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -165,8 +165,7 @@ to override the default behavior and specify which URLs shall be considered equa
<pre><code> {
Accept: &#39;text/html&#39;,
&#39;Content-Type&#39;: &#39;application/json&#39;
- }
-</code></pre></td></tr><tr>
+ }</code></pre></td></tr><tr>
<td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>
</tr>
<tr>
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index 1ab1e2a6b..c69017d52 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -110,8 +110,7 @@ await requestList.reclaimRequest(request2);
{ method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
// Batch import of URLs from a file hosted on the web
{ method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
-]
-</code></pre></td></tr><tr>
+]</code></pre></td></tr><tr>
<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
@@ -139,8 +138,7 @@ await requestList.reclaimRequest(request2);
&#39;unique-key-1&#39;: true,
&#39;unique-key-4&#39;: true,
},
-}
-</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
+}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
is to use the <code>stateKeyPrefix</code> parameter instead.</p>
</td></tr><tr>
<td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
diff --git a/docs/api/utils.md b/docs/api/utils.md
index 9a74c6e43..a8e2cc911 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -127,19 +127,16 @@ await Apify.utils.enqueueLinks({
{
name: &#39;John&#39;,
surname: &#39;Doe&#39;,
-}
-</code></pre><pre><code>// userData
+}</code></pre><pre><code>// userData
{
name: &#39;Albert&#39;,
age: 31
-}
-</code></pre><pre><code>// Enqueued request.userData
+}</code></pre><pre><code>// Enqueued request.userData
{
name: &#39;Albert&#39;,
surname: &#39;Doe&#39;,
age: 31,
-}
-</code></pre></td></tr></tbody>
+}</code></pre></td></tr></tbody>
</table>
<a name="utils.sleep"></a>
diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.md
index 9bf68d855..3e79ac0fa 100644
--- a/docs/typedefs/ActorRun.md
+++ b/docs/typedefs/ActorRun.md
@@ -57,8 +57,7 @@ For more details, see
&quot;origin&quot;: &quot;API&quot;,
&quot;clientIp&quot;: &quot;1.2.3.4&quot;,
&quot;userAgent&quot;: &quot;ApifyClient/0.2.13 (Linux; Node/v8.11.3)&quot;
- }
-</code></pre></td></tr><tr>
+ }</code></pre></td></tr><tr>
<td><code>stats</code></td><td><code>Object</code></td>
</tr>
<tr>
@@ -67,8 +66,7 @@ For more details, see
&quot;inputBodyLen&quot;: 22,
&quot;restartCount&quot;: 0,
&quot;workersUsed&quot;: 1,
- }
-</code></pre><p> Beware that object fields might change in future releases.</p>
+ }</code></pre><p> Beware that object fields might change in future releases.</p>
</td></tr><tr>
<td><code>options</code></td><td><code>Object</code></td>
</tr>
@@ -79,8 +77,7 @@ For more details, see
&quot;waitSecs&quot;: 0,
&quot;memoryMbytes&quot;: 256,
&quot;diskMbytes&quot;: 512
- }
-</code></pre></td></tr><tr>
+ }</code></pre></td></tr><tr>
<td><code>buildId</code></td><td><code>String</code></td>
</tr>
<tr>
@@ -132,6 +129,5 @@ For more details, see
&quot;body&quot;: {
&quot;message&quot;: &quot;Hello world!&quot;
}
- }
-</code></pre></td></tr></tbody>
+ }</code></pre></td></tr></tbody>
</table>