Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urban9c7cb9fde4af
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex cd51e535f..26ac3a158 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -11,6 +11,7 @@ accessible from the left sidebar.* [Apify](#module_Apify)* [`.call(actId, [input], [options])`](#module_Apify.call) ⇒ [<code>Promise<ActorRun></code>](../typedefs/actorrun)+ * [`.callTask(taskId, [input], [options])`](#module_Apify.callTask) ⇒ [<code>Promise<ActorRun></code>](../typedefs/actorrun)* [`.client`](#module_Apify.client)* [`.events`](#module_Apify.events)* [`.getApifyProxyUrl(options)`](#module_Apify.getApifyProxyUrl) ⇒ <code>String</code>@@ -22,8 +23,8 @@ accessible from the left sidebar.* [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ <code>Promise<Browser></code>* [`.launchWebDriver([options])`](#module_Apify.launchWebDriver) ⇒ <code>Promise</code>* [`.main(userFunc)`](#module_Apify.main)- * [`.openDataset([datasetIdOrName])`](#module_Apify.openDataset) ⇒ [<code>Promise<Dataset></code>](dataset)- * [`.openKeyValueStore([storeIdOrName])`](#module_Apify.openKeyValueStore) ⇒ [<code>Promise<KeyValueStore></code>](keyvaluestore)+ * [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [<code>Promise<Dataset></code>](dataset)+ * [`.openKeyValueStore([storeIdOrName], [options])`](#module_Apify.openKeyValueStore) ⇒ [<code>Promise<KeyValueStore></code>](keyvaluestore)* [`.openRequestQueue`](#module_Apify.openRequestQueue) ⇒ [<code>Promise<RequestQueue></code>](requestqueue)* [`.pushData(item)`](#module_Apify.pushData) ⇒ <code>Promise</code>* [`.setValue(key, value, [options])`](#module_Apify.setValue) ⇒ <code>Promise</code>@@ -94,12 +95,18 @@ Apify API endpoint and few others to obtain the output.<tr><td colspan="3"><p>User API token that is used to run the actor. By default, it is taken from the <code>APIFY_TOKEN</code> environment variable.</p></td></tr><tr>-<td><code>[options.memory]</code></td><td><code>Number</code></td><td></td>+<td><code>[options.memoryMbytes]</code></td><td><code>Number</code></td><td></td></tr><tr><td colspan="3"><p>Memory in megabytes which will be allocated for the new actor run.If not provided, the run uses memory of the default actor run configuration.</p></td></tr><tr>+<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Timeout for the act run in seconds. Zero value means there is no timeout.+ If not provided, the run uses timeout of the default actor run configuration.</p>+</td></tr><tr><td><code>[options.build]</code></td><td><code>String</code></td><td></td></tr><tr>@@ -126,6 +133,71 @@ Apify API endpoint and few others to obtain the output.actor's output and will return it in a raw <code>Buffer</code>.</p></td></tr></tbody></table>+<a name="module_Apify.callTask"></a>++## `Apify.callTask(taskId, [input], [options])` ⇒ [<code>Promise<ActorRun></code>](../typedefs/actorrun)+Runs an actor actor on the Apify platform using the current user account (determined by the `APIFY_TOKEN` environment variable),+waits for the actor to finish and fetches its output.++By passing the `waitSecs` option you can reduce the maximum amount of time to wait for the run to finish.+If the value is less than or equal to zero, the function returns immediately after the run is started.++The result of the function is an [`ActorRun`](../typedefs/actorrun) object+that contains details about the actor run and its output (if any).+If the actor run failed, the function fails with [`ApifyCallError`](../typedefs/apifycallerror) exception.++**Example usage:**++```javascript+const run = await Apify.call('apify/hello-world-task');+console.log(`Received message: ${run.output.body.message}`);+```++Internally, the `call()` function calls the+<a href="https://www.apify.com/docs/api/v2#/reference/actor-tasks/runs-collection/run-task-asynchronously" target="_blank">Run actor</a>+Apify API endpoint and few others to obtain the output.++**Throws**:++- [<code>ApifyCallError</code>](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>taskId</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Either <code>username/task-name</code> or task ID.</p>+</td></tr><tr>+<td><code>[input]</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td>+</tr>+<tr>+<td colspan="3"><p>This parameter is not supported yet. You must pass either <code>null</code> or <code>undefined</code> value!</p>+</td></tr><tr>+<td><code>[options]</code></td><td><code>Object</code></td>+</tr>+<tr>+<td colspan="3"><p>Object with the settings below:</p>+</td></tr><tr>+<td><code>[options.token]</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>User API token that is used to run the actor. By default, it is taken from the <code>APIFY_TOKEN</code> environment variable.</p>+</td></tr><tr>+<td><code>[options.waitSecs]</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Maximum time to wait for the actor run to finish, in seconds.+ If the limit is reached, the returned promise is resolved to a run object that will have+ status <code>READY</code> or <code>RUNNING</code> and it will not contain the actor run output.+ If <code>waitSecs</code> is null or undefined, the function waits for the actor to finish (default behavior).</p>+</td></tr></tbody>+</table><a name="module_Apify.client"></a>## `Apify.client`@@ -552,7 +624,7 @@ the promise will be awaited. The user function is called with no arguments.</p></table><a name="module_Apify.openDataset"></a>-## `Apify.openDataset([datasetIdOrName])` ⇒ [<code>Promise<Dataset></code>](dataset)+## `Apify.openDataset([datasetIdOrName], [options])` ⇒ [<code>Promise<Dataset></code>](dataset)Opens a dataset and returns a promise resolving to an instance of the [`Dataset`](dataset) class.Datasets are used to store structured data where each object stored has the same attributes,@@ -564,21 +636,31 @@ For more details and code examples, see the [`Dataset`](dataset) class.<table><thead><tr>-<th>Param</th><th>Type</th>+<th>Param</th><th>Type</th><th>Default</th></tr></thead><tbody><tr>-<td><code>[datasetIdOrName]</code></td><td><code>string</code></td>+<td><code>[datasetIdOrName]</code></td><td><code>string</code></td><td></td></tr><tr><td colspan="3"><p>ID or name of the dataset to be opened. If <code>null</code> or <code>undefined</code>,the function returns the default dataset associated with the actor run.</p>+</td></tr><tr>+<td><code>[options]</code></td><td><code>object</code></td><td></td>+</tr>+<tr>+<td colspan="3"></td></tr><tr>+<td><code>[options.forceCloud]</code></td><td><code>boolean</code></td><td><code>false</code></td>+</tr>+<tr>+<td colspan="3"><p>If set to <code>true</code> then enforces cloud storage usage. This way is possible to combine local and+ cloud storages when devloping locally.</p></td></tr></tbody></table><a name="module_Apify.openKeyValueStore"></a>-## `Apify.openKeyValueStore([storeIdOrName])` ⇒ [<code>Promise<KeyValueStore></code>](keyvaluestore)+## `Apify.openKeyValueStore([storeIdOrName], [options])` ⇒ [<code>Promise<KeyValueStore></code>](keyvaluestore)Opens a key-value store and returns a promise resolving to an instance of the [`KeyValueStore`](keyvaluestore) class.Key-value stores are used to store records or files, along with their MIME content type.@@ -590,16 +672,26 @@ For more details and code examples, see the [`KeyValueStore`](keyvaluestore) cla<table><thead><tr>-<th>Param</th><th>Type</th>+<th>Param</th><th>Type</th><th>Default</th></tr></thead><tbody><tr>-<td><code>[storeIdOrName]</code></td><td><code>string</code></td>+<td><code>[storeIdOrName]</code></td><td><code>string</code></td><td></td></tr><tr><td colspan="3"><p>ID or name of the key-value store to be opened. If <code>null</code> or <code>undefined</code>,the function returns the default key-value store associated with the actor run.</p>+</td></tr><tr>+<td><code>[options]</code></td><td><code>object</code></td><td></td>+</tr>+<tr>+<td colspan="3"></td></tr><tr>+<td><code>[options.forceCloud]</code></td><td><code>boolean</code></td><td><code>false</code></td>+</tr>+<tr>+<td colspan="3"><p>If set to <code>true</code> then enforces cloud storage usage. This way is possible to combine local and+ cloud storages when devloping locally.</p></td></tr></tbody></table><a name="module_Apify.openRequestQueue"></a>@@ -618,16 +710,26 @@ For more details and code examples, see the [`RequestQueue`](requestqueue) class<table><thead><tr>-<th>Param</th><th>Type</th>+<th>Param</th><th>Type</th><th>Default</th></tr></thead><tbody><tr>-<td><code>[queueIdOrName]</code></td><td><code>string</code></td>+<td><code>[queueIdOrName]</code></td><td><code>string</code></td><td></td></tr><tr><td colspan="3"><p>ID or name of the request queue to be opened. If <code>null</code> or <code>undefined</code>,the function returns the default request queue associated with the actor run.</p>+</td></tr><tr>+<td><code>[options]</code></td><td><code>object</code></td><td></td>+</tr>+<tr>+<td colspan="3"></td></tr><tr>+<td><code>[options.forceCloud]</code></td><td><code>boolean</code></td><td><code>false</code></td>+</tr>+<tr>+<td colspan="3"><p>If set to <code>true</code> then enforces cloud storage usage. This way is possible to combine local and+ cloud storages when devloping locally.</p></td></tr></tbody></table><a name="module_Apify.pushData"></a>diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 14cfb889a..7ac8f8b55 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -94,13 +94,13 @@ await crawler.run();</tr><tr><td colspan="3"><p>Static list of URLs to be processed.- Either <a href="requestlist"><code>RequestList</code></a> or <a href="requestqueue"><code>RequestQueue</code></a> must be provided.</p>+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p></td></tr><tr><td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td></tr><tr><td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.- Either <a href="requestlist"><code>RequestList</code></a> or <a href="requestqueue"><code>RequestQueue</code></a> must be provided.</p>+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p></td></tr><tr><td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td></tr>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 75f280a7a..a03aaec3a 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -111,13 +111,13 @@ await crawler.run();</tr><tr><td colspan="3"><p>Static list of URLs to be processed.- Either <a href="requestlist"><code>RequestList</code></a> or <a href="requestqueue"><code>RequestQueue</code></a> must be provided.</p>+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p></td></tr><tr><td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td></tr><tr><td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.- Either <a href="requestlist"><code>RequestList</code></a> or <a href="requestqueue"><code>RequestQueue</code></a> must be provided.</p>+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p></td></tr><tr><td><code>[options.requestFunction]</code></td><td><code>function</code></td><td></td></tr>diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.mdindex 06a8b52fe..f3f80938f 100644--- a/docs/api/PseudoUrl.md+++ b/docs/api/PseudoUrl.md@@ -66,7 +66,7 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M</tr><tr><td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs- by the <a href="puppeteer#puppeteer.enqueueLinks">utils.puppeteer.enqueueLinks()</a> function.</p>+ by the <a href="puppeteer#puppeteer.enqueueLinks"><code>utils.puppeteer.enqueueLinks()</code></a> function.</p></td></tr></tbody></table><a name="PseudoUrl+matches"></a>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 7e5bab942..42bd74f8e 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -89,26 +89,29 @@ await crawler.run();It is passed an object with the following fields:</p><pre><code> {request: Request,+ response: Response,page: Page,puppeteerPool: PuppeteerPool}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.- <code>page</code> is an instance of the <code>Puppeteer</code>+ <code>response</code> is an instance of the <code>Puppeteer</code><a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>- class with <code>page.goto(request.url)</code> already called.+ <code>page</code> is an instance of the <code>Puppeteer</code>+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ which is the main resource response as returned by <code>page.goto(request.url)</code>.<code>puppeteerPool</code> is an instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p></td></tr><tr><td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td></tr><tr><td colspan="3"><p>Static list of URLs to be processed.- Either <a href="requestlist"><code>RequestList</code></a> or <a href="requestqueue"><code>RequestQueue</code></a> must be provided.</p>+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p></td></tr><tr><td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td></tr><tr><td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.- Either <a href="requestlist"><code>RequestList</code></a> or <a href="requestqueue"><code>RequestQueue</code></a> must be provided.</p>+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p></td></tr><tr><td><code>[options.handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>300</code></td></tr>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 4169ca66a..74422219d 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -163,7 +163,9 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,or an array of Strings or Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s should be constructedThe Objects must include at least a <code>purl</code> property, which holds a pseudoUrl string.- All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.</p>+ All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.+ If <code>pseudoUrls</code> is an empty array, null or undefined, then the function+ enqueues all links found on the page.</p></td></tr></tbody></table><a name="puppeteer.blockResources"></a>diff --git a/docs/api/utils.md b/docs/api/utils.mdindex b55f4b4d8..bd1cec175 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -28,13 +28,13 @@ await Apify.utils.sleep(1500);<a name="utils.URL_NO_COMMAS_REGEX"></a>-## `utils.URL\_NO\_COMMAS\_REGEX`+## `utils.URL_NO_COMMAS_REGEX`Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL charactersand does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).<a name="utils.URL_WITH_COMMAS_REGEX"></a>-## `utils.URL\_WITH\_COMMAS\_REGEX`+## `utils.URL_WITH_COMMAS_REGEX`Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.mdindex 91044c6fe..3458035f3 100644--- a/docs/typedefs/ActorRun.md+++ b/docs/typedefs/ActorRun.md@@ -76,7 +76,7 @@ For more details, see<td colspan="3"><p>Actor run options. For example:</p><pre><code> {"build": "latest",- "timeoutSecs": 0,+ "waitSecs": 0,"memoryMbytes": 256,"diskMbytes": 512}diff --git a/src/actor.js b/src/actor.jsindex ae72df0ab..e83c0c5eb 100644--- a/src/actor.js+++ b/src/actor.js@@ -400,7 +400,7 @@ export const call = async (actId, input, options = {}) => {** @memberof module:Apify* @function- * @name call+ * @name callTask*/export const callTask = async (taskId, input, options = {}) => {const { tasks, keyValueStores } = apifyClient;