Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Marek Trunkat1b212144386b
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 1c533be0d..31cf35bcb 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -16,12 +16,14 @@ accessible from the left sidebar.* [`.events`](#module_Apify.events)* [`.getApifyProxyUrl(options)`](#module_Apify.getApifyProxyUrl) ⇒ <code>String</code>* [`.getEnv()`](#module_Apify.getEnv) ⇒ <code>Object</code>+ * [`.getInput`](#module_Apify.getInput) ⇒ <code>Promise<Object></code>* [`.getMemoryInfo()`](#module_Apify.getMemoryInfo) ⇒ <code>Promise<Object></code>* [`.getValue(key)`](#module_Apify.getValue) ⇒ <code>Promise<Object></code>* [`.isAtHome()`](#module_Apify.isAtHome) ⇒ <code>Boolean</code>* [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ <code>Promise<Browser></code>* [`.launchWebDriver([options])`](#module_Apify.launchWebDriver) ⇒ <code>Promise</code>* [`.main(userFunc)`](#module_Apify.main)+ * [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ <code>Promise<undefined></code>* [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [<code>Promise<Dataset></code>](dataset)* [`.openKeyValueStore([storeIdOrName], [options])`](#module_Apify.openKeyValueStore) ⇒ [<code>Promise<KeyValueStore></code>](keyvaluestore)* [`.openRequestList`](#module_Apify.openRequestList) ⇒ [<code>Promise<RequestList></code>](requestlist)@@ -377,6 +379,27 @@ For the list of the `APIFY_XXX` environment variables, see<a href="https://www.apify.com/docs/actor#run-env-vars" target="_blank">Actor documentation</a>.If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.+<a name="module_Apify.getInput"></a>++## `Apify.getInput` ⇒ <code>Promise<Object></code>+Gets the actor input value from the default [`KeyValueStore`](keyvaluestore) associated with the current actor run.++This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](keyvaluestore#KeyValueStore+getValue).+For example, calling the following code:+```javascript+const input = await Apify.getInput();+```++is equivalent to:+```javascript+const store = await Apify.openKeyValueStore();+await store.getValue('INPUT');+```++For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)+and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).++**Returns**: <code>Promise<Object></code> - Returns a promise that resolves once the record is stored.<a name="module_Apify.getMemoryInfo"></a>## `Apify.getMemoryInfo()` ⇒ <code>Promise<Object></code>@@ -411,13 +434,13 @@ Gets a value from the default [`KeyValueStore`](keyvaluestore) associated with tThis is just a convenient shortcut for [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).For example, calling the following code:```javascript-const input = await Apify.getValue('INPUT');+const input = await Apify.getValue('my-key');```is equivalent to:```javascriptconst store = await Apify.openKeyValueStore();-await store.getValue('INPUT');+await store.getValue('my-key');```To store the value to the default-key value store, you can use the [`Apify.setValue()`](#module_Apify.setValue) function.@@ -640,6 +663,52 @@ Apify.main(async () => {the promise will be awaited. The user function is called with no arguments.</p></td></tr></tbody></table>+<a name="module_Apify.metamorph"></a>++## `Apify.metamorph(targetActorId, [input], [options])` ⇒ <code>Promise<undefined></code>+Transforms this actor run to an actor run of a given actor. The system stops the current container and starts the new container+instead. All the default storages are preserved and the new input is stored under the `INPUT-METAMORPH-1` key in the same default key-value store.++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>targetActorId</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Either <code>username/actor-name</code> or actor ID of an actor to which we want to metamorph.</p>+</td></tr><tr>+<td><code>[input]</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td>+</tr>+<tr>+<td colspan="3"><p>Input for the actor. If it is an object, it will be stringified to+ JSON and its content type set to <code>application/json; charset=utf-8</code>.+ Otherwise the <code>options.contentType</code> parameter must be provided.</p>+</td></tr><tr>+<td><code>[options]</code></td><td><code>Object</code></td>+</tr>+<tr>+<td colspan="3"><p>Object with the settings below:</p>+</td></tr><tr>+<td><code>[options.contentType]</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Content type for the <code>input</code>. If not specified,+ <code>input</code> is expected to be an object that will be stringified to JSON and content type set to+ <code>application/json; charset=utf-8</code>. If <code>options.contentType</code> is specified, then <code>input</code> must be a+ <code>String</code> or <code>Buffer</code>.</p>+</td></tr><tr>+<td><code>[options.build]</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Tag or number of the target actor build to metamorph into (e.g. <code>beta</code> or <code>1.2.345</code>).+ If not provided, the run uses build tag or number from the default actor run configuration (typically <code>latest</code>).</p>+</td></tr></tbody>+</table><a name="module_Apify.openDataset"></a>## `Apify.openDataset([datasetIdOrName], [options])` ⇒ [<code>Promise<Dataset></code>](dataset)diff --git a/docs/api/AutoscaledPool.md b/docs/api/AutoscaledPool.mdindex 0e95ac5e6..9dde7d47c 100644--- a/docs/api/AutoscaledPool.md+++ b/docs/api/AutoscaledPool.md@@ -53,13 +53,13 @@ await pool.run();* [AutoscaledPool](autoscaledpool)* [`new AutoscaledPool(options)`](#new_AutoscaledPool_new)+ * [`.minConcurrency`](#AutoscaledPool+minConcurrency) ⇒ <code>number</code>* [`.minConcurrency`](#AutoscaledPool+minConcurrency)- * [`.minConcurrency`](#AutoscaledPool+minConcurrency)- * [`.maxConcurrency`](#AutoscaledPool+maxConcurrency)+ * [`.maxConcurrency`](#AutoscaledPool+maxConcurrency) ⇒ <code>number</code>* [`.maxConcurrency`](#AutoscaledPool+maxConcurrency)+ * [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency) ⇒ <code>number</code>* [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency)- * [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency)- * [`.currentConcurrency`](#AutoscaledPool+currentConcurrency)+ * [`.currentConcurrency`](#AutoscaledPool+currentConcurrency) ⇒ <code>number</code>* [`.run()`](#AutoscaledPool+run) ⇒ <code>Promise</code>* [`.abort()`](#AutoscaledPool+abort) ⇒ <code>Promise</code>* [`.pause([timeoutSecs])`](#AutoscaledPool+pause) ⇒ <code>Promise</code>@@ -169,7 +169,7 @@ await pool.run();</table><a name="AutoscaledPool+minConcurrency"></a>-## `autoscaledPool.minConcurrency`+## `autoscaledPool.minConcurrency` ⇒ <code>number</code>Gets the minimum number of tasks running in parallel.<a name="AutoscaledPool+minConcurrency"></a>@@ -180,9 +180,22 @@ Sets the minimum number of tasks running in parallel.*WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.If you're not sure, just keep the default value and the concurrency will scale up automatically.+<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>value</code></td><td><code>number</code></td>+</tr>+<tr>+</tr></tbody>+</table><a name="AutoscaledPool+maxConcurrency"></a>-## `autoscaledPool.maxConcurrency`+## `autoscaledPool.maxConcurrency` ⇒ <code>number</code>Gets the maximum number of tasks running in parallel.<a name="AutoscaledPool+maxConcurrency"></a>@@ -190,9 +203,22 @@ Gets the maximum number of tasks running in parallel.## `autoscaledPool.maxConcurrency`Sets the maximum number of tasks running in parallel.+<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>value</code></td><td><code>number</code></td>+</tr>+<tr>+</tr></tbody>+</table><a name="AutoscaledPool+desiredConcurrency"></a>-## `autoscaledPool.desiredConcurrency`+## `autoscaledPool.desiredConcurrency` ⇒ <code>number</code>Gets the desired concurrency for the pool,which is an estimated number of parallel tasks that the system can currently support.@@ -202,9 +228,22 @@ which is an estimated number of parallel tasks that the system can currently supSets the desired concurrency for the pool, i.e. the number of tasks that should be runningin parallel if there's large enough supply of tasks.+<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>value</code></td><td><code>number</code></td>+</tr>+<tr>+</tr></tbody>+</table><a name="AutoscaledPool+currentConcurrency"></a>-## `autoscaledPool.currentConcurrency`+## `autoscaledPool.currentConcurrency` ⇒ <code>number</code>Gets the the number of parallel tasks currently running in the pool.<a name="AutoscaledPool+run"></a>diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex d44a51536..1b7275b24 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -20,6 +20,8 @@ To access the default key-value store directly, you can use the[`Apify.getValue()`](apify#module_Apify.getValue)and [`Apify.setValue()`](apify#module_Apify.setValue) convenience functions.+To access the input, you can also use the [`Apify.getInput()`](apify#module_Apify.getInput) convenience function.+`KeyValueStore` stores its data either on local disk or in the Apify cloud,depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.@@ -42,7 +44,8 @@ even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.```javascript// Get actor input from the default key-value store-const input = await Apify.getValue('INPUT');+const input = await Apify.getInput();+const otherValue = Apify.getValue('my-key');// Write actor output to the default key-value store.await Apify.setValue('OUTPUT', { myResult: 123 });diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 656bc8138..fb2080b87 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -22,8 +22,8 @@ await Apify.utils.sleep(1500);* [`.enqueueLinks`](#utils.enqueueLinks) ⇒ <code>Promise<Array<QueueOperationInfo>></code>* [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)+ * [`.exports.sleep`](#utils.exports.sleep) ⇒ <code>Promise</code>* [`.isDocker()`](#utils.isDocker) ⇒ <code>Promise</code>- * [`.sleep(millis)`](#utils.sleep) ⇒ <code>Promise</code>* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ <code>Promise<Array<String>></code>* [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ <code>Array<String></code>* [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ <code>String</code>@@ -153,14 +153,9 @@ and does not support URLs containing commas or spaces. The URLs also may containRegular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.-<a name="utils.isDocker"></a>--## `utils.isDocker()` ⇒ <code>Promise</code>-Returns a `Promise` that resolves to true if the code is running in a Docker container.--<a name="utils.sleep"></a>+<a name="utils.exports.sleep"></a>-## `utils.sleep(millis)` ⇒ <code>Promise</code>+## `utils.exports.sleep` ⇒ <code>Promise</code>Returns a `Promise` that resolves after a specific period of time. This is useful to implement waitingin your code, e.g. to prevent overloading of target website or to avoid bot detection.@@ -189,6 +184,11 @@ await Apify.utils.sleep(1500);<td colspan="3"><p>Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.</p></td></tr></tbody></table>+<a name="utils.isDocker"></a>++## `utils.isDocker()` ⇒ <code>Promise</code>+Returns a `Promise` that resolves to true if the code is running in a Docker container.+<a name="utils.downloadListOfUrls"></a>## `utils.downloadListOfUrls(options)` ⇒ <code>Promise<Array<String>></code>diff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.mdindex e4a68b207..cfc58dcfd 100644--- a/docs/examples/Screenshots.md+++ b/docs/examples/Screenshots.md@@ -33,7 +33,7 @@ const Apify = require('apify');Apify.main(async () => {// Read the actor input configuration containing the URLs for the screenshot.// By convention, the input is present in the actor's default key-value store under the "INPUT" key.- const input = await Apify.getValue('INPUT');+ const input = await Apify.getInput();if (!input) throw new Error('Have you passed the correct INPUT ?');const { sources } = input;