Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

builded docs

Petr Patekb410cf30c265
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index ddfeb6ee5..26772e6f9 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -92,7 +92,8 @@ await crawler.run();
<pre><code>{
request: Request,
autoscaledPool: AutoscaledPool
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
<p> The function must return a promise, which is then awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
@@ -127,7 +128,8 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 3a08441bd..f32f69a4d 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -113,7 +113,8 @@ await crawler.run();
request: Request,
response: Object // An instance of Node&#39;s http.IncomingMessage object,
autoscaledPool: AutoscaledPool
-}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
+}
+</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
<p> If the function returns a promise, it is awaited by the crawler.</p>
<p> If the function throws an exception, the crawler will try to re-crawl the
request later, up to <code>option.maxRequestRetries</code> times.
@@ -153,7 +154,8 @@ await crawler.run();
payload, // Provided by RequestList and/or RequestQueue
strictSSL, // Use options.ignoreSslErrors
proxy, // Use options.useApifyProxy or options.proxyUrls
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
@@ -162,7 +164,8 @@ await crawler.run();
<p> The function receives the following object as an argument:</p>
<pre><code>{
request: Request
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance
in place because there are already earlier references to it. Making a copy and returning it from
this function is therefore not supported, because it would create inconsistencies where
@@ -223,7 +226,8 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>
for the default implementation of this function.</p>
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index 07206c8b6..9747a7a59 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -100,7 +100,8 @@ await crawler.run();
page: Page,
puppeteerPool: PuppeteerPool,
autoscaledPool: AutoscaledPool
-}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
+}
+</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
<code>response</code> is an instance of the <code>Puppeteer</code>
<a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
<code>page</code> is an instance of the <code>Puppeteer</code>
@@ -154,7 +155,8 @@ await crawler.run();
<pre><code>{
request: Request,
error: Error,
-}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+}
+</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
represents the last error thrown during processing of the request.</p>
<p> See
<a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>
diff --git a/docs/api/Request.md b/docs/api/Request.md
index 8547a8768..4642e3b1b 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -168,7 +168,8 @@ to override the default behavior and specify which URLs shall be considered equa
<pre><code> {
Accept: &#39;text/html&#39;,
&#39;Content-Type&#39;: &#39;application/json&#39;
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>
</tr>
<tr>
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index dff501924..faa9ebad1 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -114,7 +114,8 @@ await requestList.reclaimRequest(request2);
{ method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
// Batch import of URLs from a file hosted on the web
{ method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
-]</code></pre></td></tr><tr>
+]
+</code></pre></td></tr><tr>
<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
@@ -142,7 +143,8 @@ await requestList.reclaimRequest(request2);
&#39;unique-key-1&#39;: true,
&#39;unique-key-4&#39;: true,
},
-}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
+}
+</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
is to use the <code>stateKeyPrefix</code> parameter instead.</p>
</td></tr><tr>
<td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 90f49e81f..5a8c8a4f8 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -139,11 +139,12 @@ await Apify.utils.enqueueLinksByClickingElements({
a combination of <code>url</code>, <code>method</code> and <code>payload</code> which enables crawling of websites that navigate using form submits
(POST requests).</p>
<p> <strong>Example:</strong></p>
-<pre><code class="language-javascript"> function transformRequestFunction(request) {
+<pre><code class="lang-javascript"> function transformRequestFunction(request) {
request.userData.foo = &#39;bar&#39;;
request.useExtendedUniqueKey = true;
return request;
- }</code></pre>
+ }
+</code></pre>
</td></tr><tr>
<td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td>
</tr>
diff --git a/docs/api/utils.md b/docs/api/utils.md
index 5bbf7cda3..3e446aea0 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -20,6 +20,7 @@ await Apify.utils.sleep(1500);
- [`utils`](#utils) : `object`
- [`.enqueueLinks`](#utils.enqueueLinks) ⇒ `Promise<Array<QueueOperationInfo>>`
+ - [`.requestAsBrowser`](#utils.requestAsBrowser) ⇒ `http.IncomingMessage`
- [`.sleep`](#utils.sleep) ⇒ `Promise`
- [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)
- [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)
@@ -129,6 +130,67 @@ await Apify.utils.enqueueLinks({
when <code>uniqueKey</code> is computed.</p>
</td></tr></tbody>
</table>
+<a name="utils.requestAsBrowser"></a>
+
+## `utils.requestAsBrowser` ⇒ `http.IncomingMessage`
+
+Sends a HTTP request that looks like a request sent by a web browser,
+fully emulating browser's HTTP headers.
+
+This function is useful for web scraping of websites that send the full HTML in the first response.
+Thanks to this function, the target web server has no simple way to find out the request
+hasn't been sent by a full web browser. Using a headless browser for such requests
+is an order of magnitude more resource-intensive than this function.
+
+Currently, the function sends requests the same way as Firefox web browser does.
+In the future, it might add support for other browsers too.
+
+Internally, the function uses httpRequest function from the [@apify/httpRequest](https://github.com/apifytech/http-request)
+NPM package to perform the request.
+All `options` not recognized by this function are passed to it,
+so see it for more details.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>options.url</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>URL of the target endpoint. Supports both HTTP and HTTPS schemes.</p>
+</td></tr><tr>
+<td><code>[options.method]</code></td><td><code>GET</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>HTTP method.</p>
+</td></tr><tr>
+<td><code>[options.headers]</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Additional HTTP headers to add. It&#39;s only recommended to use this option,
+ with headers that are typically added by websites, such as cookies. Overriding
+ default browser headers will remove the masking this function provides.</p>
+</td></tr><tr>
+<td><code>[options.languageCode]</code></td><td><code>en</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Two-letter ISO 639 language code.</p>
+</td></tr><tr>
+<td><code>[options.countryCode]</code></td><td><code>US</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Two-letter ISO 3166 country code.</p>
+</td></tr><tr>
+<td><code>[options.isMobile]</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>true</code>, the function uses User-Agent of a mobile browser.</p>
+</td></tr></tbody>
+</table>
<a name="utils.sleep"></a>
## `utils.sleep` ⇒ `Promise`
diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.md
index e97b3c189..73261628f 100644
--- a/docs/typedefs/ActorRun.md
+++ b/docs/typedefs/ActorRun.md
@@ -59,7 +59,8 @@ For more details, see
&quot;origin&quot;: &quot;API&quot;,
&quot;clientIp&quot;: &quot;1.2.3.4&quot;,
&quot;userAgent&quot;: &quot;ApifyClient/0.2.13 (Linux; Node/v8.11.3)&quot;
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>stats</code></td><td><code>Object</code></td>
</tr>
<tr>
@@ -68,7 +69,8 @@ For more details, see
&quot;inputBodyLen&quot;: 22,
&quot;restartCount&quot;: 0,
&quot;workersUsed&quot;: 1,
- }</code></pre><p> Beware that object fields might change in future releases.</p>
+ }
+</code></pre><p> Beware that object fields might change in future releases.</p>
</td></tr><tr>
<td><code>options</code></td><td><code>Object</code></td>
</tr>
@@ -79,7 +81,8 @@ For more details, see
&quot;waitSecs&quot;: 0,
&quot;memoryMbytes&quot;: 256,
&quot;diskMbytes&quot;: 512
- }</code></pre></td></tr><tr>
+ }
+</code></pre></td></tr><tr>
<td><code>buildId</code></td><td><code>String</code></td>
</tr>
<tr>
@@ -131,5 +134,6 @@ For more details, see
&quot;body&quot;: {
&quot;message&quot;: &quot;Hello world!&quot;
}
- }</code></pre></td></tr></tbody>
+ }
+</code></pre></td></tr></tbody>
</table>