Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix docs [skip ci]
Ondra Urban89aff8d51172
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 3b7346801..06b9c9ce5 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -83,7 +83,7 @@ effect.<td colspan="3"><p>Idempotency key enables you to ensure that a webhook will not be added multiple times in case ofan actor restart or other situation that would cause the <code>addWebhook()</code> function to be called again.We suggest using the actor run ID as the idempotency key. You can get the run ID by calling- <a href="apify#module_Apify.getEnv">`Apify.getEnv()</a> function.</p>+ <a href="apify#module_Apify.getEnv"><code>Apify.getEnv()</code></a> function.</p></td></tr></tbody></table><a name="module_Apify.call"></a>@@ -533,14 +533,14 @@ Gets a value from the default [`KeyValueStore`](keyvaluestore) associated with tThis is just a convenient shortcut for [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue). For example, calling the following code:```javascript-const input = await Apify.getValue('my-key');+const value = await Apify.getValue('my-key');```is equivalent to:```javascriptconst store = await Apify.openKeyValueStore();-await store.getValue('my-key');+const value = await store.getValue('my-key');```To store the value to the default-key value store, you can use the [`Apify.setValue()`](#module_Apify.setValue) function.diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 022050446..9b642a394 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -84,8 +84,7 @@ await crawler.run();<pre><code>{request: Request,autoscaledPool: AutoscaledPool-}-</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p><p> The function must return a promise, which is then awaited by the crawler.</p><p> If the function throws an exception, the crawler will try to re-crawl therequest later, up to <code>option.maxRequestRetries</code> times.@@ -121,8 +120,7 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 6168f3d80..51e80ec57 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -106,10 +106,10 @@ await crawler.run();contentType: Object, // Parsed `Content-Type` headerresponse: Object // An instance of Node's http.IncomingMessage object,autoscaledPool: AutoscaledPool-}-</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p>+}</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p><ul>-<li>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> mime types</li>+<li><p>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> mime types</p>+</li><li><p>Buffer for others mime types</p><p>Parsed <code>Content-Type</code> header using<a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>@@ -158,8 +158,7 @@ The exceptions are logged to the request using thepayload, // Provided by RequestList and/or RequestQueuestrictSSL, // Use options.ignoreSslErrorsproxy, // Use options.useApifyProxy or options.proxyUrls- }-</code></pre></td></tr><tr>+ }</code></pre></td></tr><tr><td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>@@ -168,8 +167,7 @@ The exceptions are logged to the request using the<p> The function receives the following object as an argument:</p><pre><code>{request: Request-}-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p><p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instancein place because there are already earlier references to it. Making a copy and returning it fromthis function is therefore not supported, because it would create inconsistencies where@@ -229,8 +227,7 @@ The exceptions are logged to the request using the<pre><code>{request: Request,error: Error,-}-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>for the default implementation of this function.</p>diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex 14e6e5724..94cb88ece 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -105,7 +105,7 @@ const buffer = await store.getValue('screenshot1.png');</tr><tr><td colspan="3"><p>Unique key of the record. It can be at most 256 characters long and only consist- of the following characters: <code>[a-zA-Z0-9!-_.'()</code></p>+ of the following characters: <code>a</code>-<code>z</code>, <code>A</code>-<code>Z</code>, <code>0</code>-<code>9</code> and <code>!-_.'()</code></p></td></tr></tbody></table><a name="KeyValueStore+setValue"></a>@@ -153,7 +153,7 @@ stored!</tr><tr><td colspan="3"><p>Unique key of the record. It can be at most 256 characters long and only consist- of the following characters: <code>[a-zA-Z0-9!-_.'()</code></p>+ of the following characters: <code>a</code>-<code>z</code>, <code>A</code>-<code>Z</code>, <code>0</code>-<code>9</code> and <code>!-_.'()</code></p></td></tr><tr><td><code>value</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td></tr>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 1bb368f22..14059d499 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -94,8 +94,7 @@ await crawler.run();page: Page,puppeteerPool: PuppeteerPool,autoscaledPool: AutoscaledPool-}-</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.+}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.<code>response</code> is an instance of the <code>Puppeteer</code><a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a><code>page</code> is an instance of the <code>Puppeteer</code>@@ -156,8 +155,7 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}-</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>diff --git a/docs/api/Request.md b/docs/api/Request.mdindex dc9b4c48c..808308a20 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -166,8 +166,7 @@ to override the default behavior and specify which URLs shall be considered equa<pre><code> {Accept: 'text/html','Content-Type': 'application/json'- }-</code></pre></td></tr><tr>+ }</code></pre></td></tr><tr><td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td></tr><tr>diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex d2e1de1ef..02fe12f6c 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -107,8 +107,7 @@ await requestList.reclaimRequest(request2);{ method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },// Batch import combined with regex.{ method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },-]-</code></pre></td></tr><tr>+]</code></pre></td></tr><tr><td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td></tr><tr>@@ -136,8 +135,7 @@ await requestList.reclaimRequest(request2);'unique-key-1': true,'unique-key-4': true,},-}-</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>+}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>is to use the <code>stateKeyPrefix</code> parameter instead.</p></td></tr><tr><td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 4372453a4..39244527a 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -135,14 +135,13 @@ objects.a combination of <code>url</code>, <code>method</code> and <code>payload</code> which enables crawling of websites that navigate using form submits(POST requests).</p><p> <strong>Example:</strong></p>-<pre><code class="lang-javascript"> {+<pre><code class="language-javascript"> {transformRequestFunction: (request) => {request.userData.foo = 'bar';request.useExtendedUniqueKey = true;return request;}- }-</code></pre>+ }</code></pre></td></tr><tr><td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td></tr>diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 6163a4f48..8f62ca0ee 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -126,14 +126,13 @@ objects.<p> For example: by adding <code>keepUrlFragment: true</code> to the <code>request</code> object, URL fragments will not be removedwhen <code>uniqueKey</code> is computed.</p><p> <strong>Example:</strong></p>-<pre><code class="lang-javascript"> {+<pre><code class="language-javascript"> {transformRequestFunction: (request) => {request.userData.foo = 'bar';request.keepUrlFragment = true;return request;}- }-</code></pre>+ }</code></pre></td></tr></tbody></table><a name="utils.requestAsBrowser"></a>@@ -197,7 +196,7 @@ request. All `options` not recognized by this function are passed to it, so see<tr><td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.If function returns true request gets aborted. This function is passed to the- (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request]">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>+ (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p></td></tr></tbody></table><a name="utils.sleep"></a>diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.mdindex d3cee5406..571bffacc 100644--- a/docs/typedefs/ActorRun.md+++ b/docs/typedefs/ActorRun.md@@ -55,8 +55,7 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run""origin": "API","clientIp": "1.2.3.4","userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"- }-</code></pre></td></tr><tr>+ }</code></pre></td></tr><tr><td><code>stats</code></td><td><code>Object</code></td></tr><tr>@@ -65,8 +64,7 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run""inputBodyLen": 22,"restartCount": 0,"workersUsed": 1,- }-</code></pre><p> Beware that object fields might change in future releases.</p>+ }</code></pre><p> Beware that object fields might change in future releases.</p></td></tr><tr><td><code>options</code></td><td><code>Object</code></td></tr>@@ -77,8 +75,7 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run""waitSecs": 0,"memoryMbytes": 256,"diskMbytes": 512- }-</code></pre></td></tr><tr>+ }</code></pre></td></tr><tr><td><code>buildId</code></td><td><code>String</code></td></tr><tr>@@ -130,6 +127,5 @@ certain fields. For more details, see <a href="https://apify.com/docs/actor#run""body": {"message": "Hello world!"}- }-</code></pre></td></tr></tbody>+ }</code></pre></td></tr></tbody></table>diff --git a/src/actor.js b/src/actor.jsindex 0f880ffc6..ec33c582b 100644--- a/src/actor.js+++ b/src/actor.js@@ -802,7 +802,7 @@ export const getApifyProxyUrl = (options = {}) => {* Idempotency key enables you to ensure that a webhook will not be added multiple times in case of* an actor restart or other situation that would cause the `addWebhook()` function to be called again.* We suggest using the actor run ID as the idempotency key. You can get the run ID by calling- * [`Apify.getEnv()](apify#module_Apify.getEnv) function.+ * [`Apify.getEnv()`](apify#module_Apify.getEnv) function.* @return {Promise<Object>} The return value is the Webhook object.* For more information, see the [Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.*