Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
builded docs
Petr Patekb410cf30c265
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex ddfeb6ee5..26772e6f9 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -92,7 +92,8 @@ await crawler.run();<pre><code>{request: Request,autoscaledPool: AutoscaledPool-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p><p> The function must return a promise, which is then awaited by the crawler.</p><p> If the function throws an exception, the crawler will try to re-crawl therequest later, up to <code>option.maxRequestRetries</code> times.@@ -127,7 +128,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 3a08441bd..f32f69a4d 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -113,7 +113,8 @@ await crawler.run();request: Request,response: Object // An instance of Node's http.IncomingMessage object,autoscaledPool: AutoscaledPool-}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>+}+</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p><p> If the function returns a promise, it is awaited by the crawler.</p><p> If the function throws an exception, the crawler will try to re-crawl therequest later, up to <code>option.maxRequestRetries</code> times.@@ -153,7 +154,8 @@ await crawler.run();payload, // Provided by RequestList and/or RequestQueuestrictSSL, // Use options.ignoreSslErrorsproxy, // Use options.useApifyProxy or options.proxyUrls- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>@@ -162,7 +164,8 @@ await crawler.run();<p> The function receives the following object as an argument:</p><pre><code>{request: Request-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p><p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instancein place because there are already earlier references to it. Making a copy and returning it fromthis function is therefore not supported, because it would create inconsistencies where@@ -223,7 +226,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>for the default implementation of this function.</p>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 07206c8b6..9747a7a59 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -100,7 +100,8 @@ await crawler.run();page: Page,puppeteerPool: PuppeteerPool,autoscaledPool: AutoscaledPool-}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.+}+</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.<code>response</code> is an instance of the <code>Puppeteer</code><a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a><code>page</code> is an instance of the <code>Puppeteer</code>@@ -154,7 +155,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>diff --git a/docs/api/Request.md b/docs/api/Request.mdindex 8547a8768..4642e3b1b 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -168,7 +168,8 @@ to override the default behavior and specify which URLs shall be considered equa<pre><code> {Accept: 'text/html','Content-Type': 'application/json'- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td></tr><tr>diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex dff501924..faa9ebad1 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -114,7 +114,8 @@ await requestList.reclaimRequest(request2);{ method: 'GET', url: 'http://example.com/a/b' },// Batch import of URLs from a file hosted on the web{ method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },-]</code></pre></td></tr><tr>+]+</code></pre></td></tr><tr><td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td></tr><tr>@@ -142,7 +143,8 @@ await requestList.reclaimRequest(request2);'unique-key-1': true,'unique-key-4': true,},-}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>+}+</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>is to use the <code>stateKeyPrefix</code> parameter instead.</p></td></tr><tr><td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 90f49e81f..5a8c8a4f8 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -139,11 +139,12 @@ await Apify.utils.enqueueLinksByClickingElements({a combination of <code>url</code>, <code>method</code> and <code>payload</code> which enables crawling of websites that navigate using form submits(POST requests).</p><p> <strong>Example:</strong></p>-<pre><code class="language-javascript"> function transformRequestFunction(request) {+<pre><code class="lang-javascript"> function transformRequestFunction(request) {request.userData.foo = 'bar';request.useExtendedUniqueKey = true;return request;- }</code></pre>+ }+</code></pre></td></tr><tr><td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td></tr>diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 5bbf7cda3..3e446aea0 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -20,6 +20,7 @@ await Apify.utils.sleep(1500);- [`utils`](#utils) : `object`- [`.enqueueLinks`](#utils.enqueueLinks) ⇒ `Promise<Array<QueueOperationInfo>>`+ - [`.requestAsBrowser`](#utils.requestAsBrowser) ⇒ `http.IncomingMessage`- [`.sleep`](#utils.sleep) ⇒ `Promise`- [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)- [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)@@ -129,6 +130,67 @@ await Apify.utils.enqueueLinks({when <code>uniqueKey</code> is computed.</p></td></tr></tbody></table>+<a name="utils.requestAsBrowser"></a>++## `utils.requestAsBrowser` ⇒ `http.IncomingMessage`++Sends a HTTP request that looks like a request sent by a web browser,+fully emulating browser's HTTP headers.++This function is useful for web scraping of websites that send the full HTML in the first response.+Thanks to this function, the target web server has no simple way to find out the request+hasn't been sent by a full web browser. Using a headless browser for such requests+is an order of magnitude more resource-intensive than this function.++Currently, the function sends requests the same way as Firefox web browser does.+In the future, it might add support for other browsers too.++Internally, the function uses httpRequest function from the [@apify/httpRequest](https://github.com/apifytech/http-request)+NPM package to perform the request.+All `options` not recognized by this function are passed to it,+so see it for more details.++<table>+<thead>+<tr>+<th>Param</th><th>Default</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>options.url</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>URL of the target endpoint. Supports both HTTP and HTTPS schemes.</p>+</td></tr><tr>+<td><code>[options.method]</code></td><td><code>GET</code></td>+</tr>+<tr>+<td colspan="3"><p>HTTP method.</p>+</td></tr><tr>+<td><code>[options.headers]</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Additional HTTP headers to add. It's only recommended to use this option,+ with headers that are typically added by websites, such as cookies. Overriding+ default browser headers will remove the masking this function provides.</p>+</td></tr><tr>+<td><code>[options.languageCode]</code></td><td><code>en</code></td>+</tr>+<tr>+<td colspan="3"><p>Two-letter ISO 639 language code.</p>+</td></tr><tr>+<td><code>[options.countryCode]</code></td><td><code>US</code></td>+</tr>+<tr>+<td colspan="3"><p>Two-letter ISO 3166 country code.</p>+</td></tr><tr>+<td><code>[options.isMobile]</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>If <code>true</code>, the function uses User-Agent of a mobile browser.</p>+</td></tr></tbody>+</table><a name="utils.sleep"></a>## `utils.sleep` ⇒ `Promise`diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.mdindex e97b3c189..73261628f 100644--- a/docs/typedefs/ActorRun.md+++ b/docs/typedefs/ActorRun.md@@ -59,7 +59,8 @@ For more details, see"origin": "API","clientIp": "1.2.3.4","userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>stats</code></td><td><code>Object</code></td></tr><tr>@@ -68,7 +69,8 @@ For more details, see"inputBodyLen": 22,"restartCount": 0,"workersUsed": 1,- }</code></pre><p> Beware that object fields might change in future releases.</p>+ }+</code></pre><p> Beware that object fields might change in future releases.</p></td></tr><tr><td><code>options</code></td><td><code>Object</code></td></tr>@@ -79,7 +81,8 @@ For more details, see"waitSecs": 0,"memoryMbytes": 256,"diskMbytes": 512- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>buildId</code></td><td><code>String</code></td></tr><tr>@@ -131,5 +134,6 @@ For more details, see"body": {"message": "Hello world!"}- }</code></pre></td></tr></tbody>+ }+</code></pre></td></tr></tbody></table>