Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Docs update
Marek Trunkat4219fe892a56
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 63f13fb95..da6630706 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -36,7 +36,7 @@ accessible from the left sidebar.## `Apify.addWebhook(options)` ⇒ `Promise<Object>`Creates an ad-hoc webhook for the current actor run, which lets you receive a notification when the actor run finished or failed.-For more information about Apify actor webhooks, please see the <a href="https://apify.com/docs/webhook" target="_blank">documentation</a>.+For more information about Apify actor webhooks, please see the <a href="https://apify.com/docs/webhooks" target="_blank">documentation</a>.Note that webhooks are only supported for actors running on the Apify platform.In local environment, the function will print a warning and have no effect.diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 86c5ee368..6f54b77fd 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -91,7 +91,8 @@ await crawler.run();<pre><code>{request: Request,autoscaledPool: AutoscaledPool-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p><p> The function must return a promise, which is then awaited by the crawler.</p><p> If the function throws an exception, the crawler will try to re-crawl therequest later, up to <code>option.maxRequestRetries</code> times.@@ -126,7 +127,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 0d4160221..6926febe8 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -108,7 +108,8 @@ await crawler.run();request: Request,response: Object // An instance of Node's http.IncomingMessage object,autoscaledPool: AutoscaledPool-}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>+}+</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p><p> If the function returns a promise, it is awaited by the crawler.</p><p> If the function throws an exception, the crawler will try to re-crawl therequest later, up to <code>option.maxRequestRetries</code> times.@@ -146,7 +147,8 @@ await crawler.run();payload, // Are provided by RequestList and/or RequestQueuestrictSSL, // Use options.ignoreSslErrorsproxy, // Use options.useApifyProxy or options.proxyUrls- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>@@ -155,7 +157,8 @@ await crawler.run();<p> The function receives the following object as an argument:</p><pre><code>{request: Request-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p><p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instancein place because there are already earlier references to it. Making a copy and returning it fromthis function is therefore not supported, because it would create inconsistencies where@@ -216,7 +219,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>for the default implementation of this function.</p>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 059741515..d9c2a5221 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -99,7 +99,8 @@ await crawler.run();page: Page,puppeteerPool: PuppeteerPool,autoscaledPool: AutoscaledPool-}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.+}+</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.<code>response</code> is an instance of the <code>Puppeteer</code><a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a><code>page</code> is an instance of the <code>Puppeteer</code>@@ -153,7 +154,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>diff --git a/docs/api/Request.md b/docs/api/Request.mdindex dd98f518a..a3edbc7a0 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -165,7 +165,8 @@ to override the default behavior and specify which URLs shall be considered equa<pre><code> {Accept: 'text/html','Content-Type': 'application/json'- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td></tr><tr>diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex c69017d52..1ab1e2a6b 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -110,7 +110,8 @@ await requestList.reclaimRequest(request2);{ method: 'GET', url: 'http://example.com/a/b' },// Batch import of URLs from a file hosted on the web{ method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },-]</code></pre></td></tr><tr>+]+</code></pre></td></tr><tr><td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td></tr><tr>@@ -138,7 +139,8 @@ await requestList.reclaimRequest(request2);'unique-key-1': true,'unique-key-4': true,},-}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>+}+</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>is to use the <code>stateKeyPrefix</code> parameter instead.</p></td></tr><tr><td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex 47066d856..87ae1d9f4 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -77,7 +77,7 @@ await queue.reclaimRequest(request2);* [`.isEmpty()`](#RequestQueue+isEmpty) ⇒ `Promise<Boolean>`* [`.isFinished()`](#RequestQueue+isFinished) ⇒ `Promise<Boolean>`* [`.delete()`](#RequestQueue+delete) ⇒ `Promise`- * [`.handledCount()`](#RequestQueue+handledCount) ⇒ `Promise<number>`+ * ~~[`.handledCount()`](#RequestQueue+handledCount) ⇒ `Promise<number>`~~* [`.getInfo()`](#RequestQueue+getInfo) ⇒ `Promise<Object>`<a name="RequestQueue+addRequest"></a>@@ -100,7 +100,8 @@ it will not be updated. You can find out whether this happened from the resultin<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td><td></td></tr><tr>-<td colspan="3"><p><a href="request"><code>Request</code></a> object, or an object to construct a <code>Request</code> instance from.</p>+<td colspan="3"><p><a href="request"><code>Request</code></a> object or vanilla object with request data.+Note that the function sets the <code>uniqueKey</code> and <code>id</code> fields to the passed object.</p></td></tr><tr><td><code>[options]</code></td><td><code>Object</code></td><td></td></tr>@@ -129,19 +130,30 @@ Gets the request from the queue specified by ID.<td><code>requestId</code></td><td><code>String</code></td></tr><tr>-<td colspan="3"><p>Request ID</p>+<td colspan="3"><p>ID of the request.</p></td></tr></tbody></table><a name="RequestQueue+fetchNextRequest"></a>## `requestQueue.fetchNextRequest()` ⇒ [`Promise<Request>`](request)-Returns next request in the queue to be processed.+Returns a next request in the queue to be processed, or `null` if there are no more pending requests.-**Returns**: [`Promise<Request>`](request) - Returns the request object, or `null` if there are no more pending requests.+Once you successfully finish processing of the request, you need to call [`RequestQueue.markRequestHandled`](requestqueue.markrequesthandled)+to mark the request as handled in the queue. If there was some error in processing the request,+call [`RequestQueue.reclaimRequest`](requestqueue.reclaimrequest) instead, so that the queue will give the request to some other consumer+in another call to the `fetchNextRequest` function.++Note that the `null` return value doesn't mean the queue processing finished,+it means there are currently no pending requests.+To check whether all requests in queue were finished, use [`RequestQueue.isFinished`](requestqueue.isfinished) instead.++**Returns**: [`Promise<Request>`](request) - Returns the request object or `null` if there are no more pending requests.<a name="RequestQueue+markRequestHandled"></a>## `requestQueue.markRequestHandled(request)` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)-Marks request handled after successful processing.+Marks a request that was previously returned by the [`RequestQueue.fetchNextRequest`](requestqueue.fetchnextrequest) function+as handled after successful processing.+Handled requests will never again be returned by the `fetchNextRequest` function.<table><thead>@@ -159,9 +171,10 @@ Marks request handled after successful processing.<a name="RequestQueue+reclaimRequest"></a>## `requestQueue.reclaimRequest(request, [options])` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)-Reclaims failed request back to the queue, so that it can be processed later again.+Reclaims a failed request back to the queue, so that it can be returned for processed later again+by another call to [`RequestQueue.fetchNextRequest`](requestqueue.fetchnextrequest).The request record in the queue is updated using the provided `request` parameter.-For example, this lets you store the number of retries for the request.+For example, this lets you store the number of retries or error messages for the request.<table><thead>@@ -182,23 +195,24 @@ For example, this lets you store the number of retries for the request.<td><code>[options.forefront]</code></td><td><code>Boolean</code></td><td><code>false</code></td></tr><tr>-<td colspan="3"><p>If <code>true</code> then requests get returned to the start of the queue- and to the back of the queue otherwise.</p>+<td colspan="3"><p>If <code>true</code> then the request it placed to the beginning of the queue, so that it's returned+in the next call to <a href="requestqueue.fetchnextrequest"><code>RequestQueue.fetchNextRequest</code></a>. By default, it's put to the end of the queue.</p></td></tr></tbody></table><a name="RequestQueue+isEmpty"></a>## `requestQueue.isEmpty()` ⇒ `Promise<Boolean>`-Resolves to `true` if the next call to [`fetchNextRequest`](#RequestQueue+fetchNextRequest) would return `null`, otherwise it resolves to `false`.+Resolves to `true` if the next call to [`RequestQueue.fetchNextRequest`](requestqueue.fetchnextrequest) would return `null`, otherwise it resolves to `false`.Note that even if the queue is empty, there might be some pending requests currently being processed.-If you need to ensure that there is no activity in the queue, use [`isFinished`](#RequestQueue+isFinished).+If you need to ensure that there is no activity in the queue, use [`RequestQueue.isFinished`](requestqueue.isfinished).<a name="RequestQueue+isFinished"></a>## `requestQueue.isFinished()` ⇒ `Promise<Boolean>`Resolves to `true` if all requests were already handled and there are no more left.-Due to the nature of distributed storage systems,-the function might occasionally return a false negative, but it will never return a false positive.+Due to the nature of distributed storage used by the queue,+the function might occasionally return a false negative,+but it will never return a false positive.<a name="RequestQueue+delete"></a>@@ -208,7 +222,9 @@ depending on the mode of operation.<a name="RequestQueue+handledCount"></a>-## `requestQueue.handledCount()` ⇒ `Promise<number>`+## ~~`requestQueue.handledCount()` ⇒ `Promise<number>`~~+***Deprecated***+Returns the number of handled requests.<a name="RequestQueue+getInfo"></a>diff --git a/docs/api/utils.md b/docs/api/utils.mdindex a8e2cc911..9a74c6e43 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -127,16 +127,19 @@ await Apify.utils.enqueueLinks({{name: 'John',surname: 'Doe',-}</code></pre><pre><code>// userData+}+</code></pre><pre><code>// userData{name: 'Albert',age: 31-}</code></pre><pre><code>// Enqueued request.userData+}+</code></pre><pre><code>// Enqueued request.userData{name: 'Albert',surname: 'Doe',age: 31,-}</code></pre></td></tr></tbody>+}+</code></pre></td></tr></tbody></table><a name="utils.sleep"></a>diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.mdindex 3e79ac0fa..9bf68d855 100644--- a/docs/typedefs/ActorRun.md+++ b/docs/typedefs/ActorRun.md@@ -57,7 +57,8 @@ For more details, see"origin": "API","clientIp": "1.2.3.4","userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>stats</code></td><td><code>Object</code></td></tr><tr>@@ -66,7 +67,8 @@ For more details, see"inputBodyLen": 22,"restartCount": 0,"workersUsed": 1,- }</code></pre><p> Beware that object fields might change in future releases.</p>+ }+</code></pre><p> Beware that object fields might change in future releases.</p></td></tr><tr><td><code>options</code></td><td><code>Object</code></td></tr>@@ -77,7 +79,8 @@ For more details, see"waitSecs": 0,"memoryMbytes": 256,"diskMbytes": 512- }</code></pre></td></tr><tr>+ }+</code></pre></td></tr><tr><td><code>buildId</code></td><td><code>String</code></td></tr><tr>@@ -129,5 +132,6 @@ For more details, see"body": {"message": "Hello world!"}- }</code></pre></td></tr></tbody>+ }+</code></pre></td></tr></tbody></table>diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.mdindex 7c47130d0..b97676b45 100644--- a/docs/typedefs/QueueOperationInfo.md+++ b/docs/typedefs/QueueOperationInfo.md@@ -32,7 +32,7 @@ A helper class that is used to report results from various<tr><td colspan="3"><p>The ID of the added request</p></td></tr><tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>+<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td></tr><tr><td colspan="3"><p>The original <code>Request</code> object passed to the <code>RequestQueue</code> function.</p>