Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Docs update
Marek Trunkatc2c24487b28f
diff --git a/README.md b/README.mdindex 3158e4b4c..20f4e01db 100644--- a/README.md+++ b/README.md@@ -10,8 +10,8 @@ to maintain queues of URLs to crawl, store crawling results to a local filesysterotate proxies and much more.The SDK is available as the <a href="https://www.npmjs.com/package/apify" target="_blank"><code>apify</code></a> NPM package.It can be used either stand-alone in your own applications-or in <a href="https://apify.com/docs/actor" target="_blank">actors</a>-running on the <a href="https://apify.com/" target="_blank">Apify Cloud</a>.+or in <a href="https://www.apify.com/docs/actor" target="_blank">actors</a>+running on the <a href="https://www.apify.com/" target="_blank">Apify Cloud</a>.**View full documentation, guides and examples on the dedicated <a href="https://sdk.apify.com" target="_blank">Apify SDK project website</a>**@@ -77,7 +77,7 @@ The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify"></li><li><a href="https://sdk.apify.com/docs/api/requestqueue"><code>RequestQueue</code></a>- - Represents a queue of URLs to crawl, which is stored either on a local filesystem or in the <a href="https://apify.com" target="_blank">Apify Cloud</a>.+ - Represents a queue of URLs to crawl, which is stored either on a local filesystem or in the <a href="https://www.apify.com" target="_blank">Apify Cloud</a>.The queue is used for deep crawling of websites, where you start withseveral URLs and then recursively follow links to other pages.The data structure supports both breadth-first and depth-first crawling orders.@@ -128,7 +128,7 @@ npm install apify --save```Run the following example to perform a recursive crawl of a website using Puppeteer.-For more examples showcasing various features of the Apify SDK,+For more examples showcasing various features of the Apify SDK,[see the Examples section of the documentation](https://sdk.apify.com/docs/examples/basiccrawler).```javascriptdiff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 31cf35bcb..a3dda26af 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -48,7 +48,7 @@ If you want to run an actor task rather than an actor, please use the[`Apify.callTask()`](../api/apify#module_Apify.callTask) function instead.For more information about actors, read the-<a href="https://www.apify.com/docs/actor" target="_blank">documentation</a>.+<a href="https://apify.com/docs/actor" target="_blank">documentation</a>.**Example usage:**@@ -58,7 +58,7 @@ console.log(`Received message: ${run.output.body.message}`);```Internally, the `call()` function invokes the-<a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor</a>+<a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor</a>and several other API endpoints to obtain the output.**Throws**:@@ -158,7 +158,7 @@ Note that an actor task is a saved input configuration and options for an actor.If you want to run an actor directly rather than an actor task, please use the[`Apify.call()`](../api/apify#module_Apify.call) function instead.-For more information about actor tasks, read the [`documentation`](https://www.apify.com/docs/tasks).+For more information about actor tasks, read the [`documentation`](https://apify.com/docs/tasks).**Example usage:**@@ -168,7 +168,7 @@ console.log(`Received message: ${run.output.body.message}`);```Internally, the `callTask()` function calls the-<a href="https://www.apify.com/docs/api/v2#/reference/actor-tasks/runs-collection/run-task-asynchronously" target="_blank">Run task</a>+<a href="https://apify.com/docs/api/v2#/reference/actor-tasks/runs-collection/run-task-asynchronously" target="_blank">Run task</a>and several other API endpoints to obtain the output.**Throws**:@@ -216,7 +216,7 @@ and several other API endpoints to obtain the output.## `Apify.client`Gets the default instance of the `ApifyClient` class provided-<a href="https://www.apify.com/docs/sdk/apify-client-js/latest"+<a href="https://apify.com/docs/sdk/apify-client-js/latest"target="_blank">apify-client</a> by the NPM package.The instance is created automatically by the Apify SDK and it is configured using the`APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN` environment variables.@@ -224,7 +224,7 @@ The instance is created automatically by the Apify SDK and it is configured usinThe instance is used for all underlying calls to the Apify API in functions such as[`Apify.getValue()`](#module_Apify.getValue) or [`Apify.call()`](#module_Apify.call).The settings of the client can be globally altered by calling the-<a href="https://www.apify.com/docs/sdk/apify-client-js/latest#ApifyClient-setOptions"+<a href="https://apify.com/docs/sdk/apify-client-js/latest#ApifyClient-setOptions"target="_blank">`Apify.client.setOptions()`</a> function.Beware that altering these settings might have unintended effects on the entire Apify SDK package.@@ -302,7 +302,7 @@ proxy-enabled applications.For more information, seethe <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the app-or the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>.+or the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a>.**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.<table>@@ -376,7 +376,7 @@ It has the following properties:}```For the list of the `APIFY_XXX` environment variables, see-<a href="https://www.apify.com/docs/actor#run-env-vars" target="_blank">Actor documentation</a>.+<a href="https://apify.com/docs/actor#run-env-vars" target="_blank">Actor documentation</a>.If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.<a name="module_Apify.getInput"></a>@@ -493,7 +493,7 @@ The `launchPuppeteer()` function alters the following Puppeteer options:</li><li>If <code>options.useApifyProxy</code> is <code>true</code> then the function generates a URL of- <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>+ <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>based on <code>options.apifyProxyGroups</code> and <code>options.apifyProxySession</code> and passes it as <code>options.proxyUrl</code>.</li><li>@@ -514,7 +514,7 @@ To use this function, you need to have the <a href="https://www.npmjs.com/packagNPM package installed in your project.When running on the Apify cloud, you can achieve that simplyby using the `apify/actor-node-chrome` base Docker image for your actor - see-<a href="https://www.apify.com/docs/actor#base-images" target="_blank">Apify Actor documentation</a>+<a href="https://apify.com/docs/actor#base-images" target="_blank">Apify Actor documentation</a>for details.For an example of usage, see the [Synchronous run Example](../examples/synchronousrun)@@ -550,10 +550,10 @@ class.To use this function, you need to have Google Chrome and<a href="https://sites.google.com/a/chromium.org/chromedriver/" target="_blank">ChromeDriver</a> installed in your environment.For example, you can use the `apify/actor-node-chrome` base Docker image for your actor - see-<a href="https://www.apify.com/docs/actor#base-images" target="_blank">documentation</a>+<a href="https://apify.com/docs/actor#base-images" target="_blank">documentation</a>for more details.-For an example of usage, see the <a href="https://www.apify.com/apify/example-selenium" target="_blank">apify/example-selenium</a> actor.+For an example of usage, see the <a href="https://apify.com/apify/example-selenium" target="_blank">apify/example-selenium</a> actor.<table><thead>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 9bbfb52d3..ec4af5cb5 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -158,13 +158,13 @@ await crawler.run();<tr><td colspan="3"><p>If set to <code>true</code>, <code>CheerioCrawler</code> will be configured to use<a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- For more information, see the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a></p>+ For more information, see the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a></p></td></tr><tr><td><code>[options.apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td></tr><tr><td colspan="3"><p>An array of proxy groups to be used- by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ by the <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>.Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p></td></tr><tr><td><code>[options.apifyProxySession]</code></td><td><code>String</code></td><td></td>diff --git a/docs/api/Dataset.md b/docs/api/Dataset.mdindex a914d0f67..00fea4df8 100644--- a/docs/api/Dataset.md+++ b/docs/api/Dataset.md@@ -26,7 +26,7 @@ unless you override it by setting the `APIFY_DEFAULT_DATASET_ID` environment varEach dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the-<a href="https://www.apify.com/docs/storage#dataset" target="_blank">Apify Dataset</a>+<a href="https://apify.com/docs/storage#dataset" target="_blank">Apify Dataset</a>cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`option to [`Apify.openDataset()`](apify#module_Apify.openDataset) function,even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.@@ -196,9 +196,9 @@ Returns items in the dataset based on the provided parameters.Returns an object containing general information about the dataset.The function returns the same object as the Apify API Client's-[getDataset](https://www.apify.com/docs/api/apify-client-js/latest#ApifyClient-datasets-getDataset)+[getDataset](https://apify.com/docs/api/apify-client-js/latest#ApifyClient-datasets-getDataset)function, which in turn calls the-[Get dataset](https://www.apify.com/docs/api/v2#/reference/datasets/dataset/get-dataset)+[Get dataset](https://apify.com/docs/api/v2#/reference/datasets/dataset/get-dataset)API endpoint.**Example:**diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex 1b7275b24..87e65e910 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -35,7 +35,7 @@ unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environThe `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the-<a href="https://www.apify.com/docs/storage#key-value-store" target="_blank">Apify Key-value store</a>+<a href="https://apify.com/docs/storage#key-value-store" target="_blank">Apify Key-value store</a>cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`option to [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function,even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex 59129a7c1..09ca151d4 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -36,7 +36,7 @@ Each request in the queue is stored as a separate JSON file, where `{STATE}` isand `{NUMBER}` is an integer indicating the position of the request in the queue.If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the-<a href="https://www.apify.com/docs/storage#queue" target="_blank">Apify Request Queue</a>+<a href="https://apify.com/docs/storage#queue" target="_blank">Apify Request Queue</a>cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`option to [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function,even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.@@ -219,9 +219,9 @@ Returns the number of handled requests.Returns an object containing general information about the request queue.The function returns the same object as the Apify API Client's-[getQueue](https://www.apify.com/docs/api/apify-client-js/latest#ApifyClient-requestQueues-getQueue)+[getQueue](https://apify.com/docs/api/apify-client-js/latest#ApifyClient-requestQueues-getQueue)function, which in turn calls the-[Get request queue](https://www.apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue)+[Get request queue](https://apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue)API endpoint.**Example:**diff --git a/docs/examples/CallActor.md b/docs/examples/CallActor.mdindex 2ae39042b..734ff1852 100644--- a/docs/examples/CallActor.md+++ b/docs/examples/CallActor.md@@ -8,7 +8,7 @@ This example demonstrates how to start an Apify actor usingand how to call Apify API using[`Apify.client`](../api/apify#module_Apify.client).The script extracts the current Bitcoin prices from <a href="https://www.kraken.com/" target="_blank">Kraken.com</a>-and sends them to your email using the <a href="https://www.apify.com/apify/send-mail" target="_blank">apify/send-mail</a> actor.+and sends them to your email using the <a href="https://apify.com/apify/send-mail" target="_blank">apify/send-mail</a> actor.To make the example work, you'll need an <a href="https://my.apify.com/" target="_blank">Apify Account</a>.Go to <a href="https://my.apify.com/account#/integrations" target="_blank">Account - Integrations</a> page to obtain your API token@@ -38,7 +38,7 @@ Apify.main(async () => {// have available on the platform under the name: apify/send-mail.// The second parameter to the Apify.call() invocation is the actor's// desired input. You can find the required input parameters by checking- // the actor's documentation page: https://www.apify.com/apify/send-mail+ // the actor's documentation page: https://apify.com/apify/send-mailconsole.log(`Sending email to ${user.email}...`);await Apify.call('apify/send-mail', {to: user.email,diff --git a/docs/examples/PuppeteerWithProxy.md b/docs/examples/PuppeteerWithProxy.mdindex d70baa2fb..ae38887b5 100644--- a/docs/examples/PuppeteerWithProxy.md+++ b/docs/examples/PuppeteerWithProxy.md@@ -4,7 +4,7 @@ title: Puppeteer With Proxy---This example demonstrates how to load pages in headless Chrome / Puppeteer-over <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+over <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>.To make it work, you'll need an Apify Accountthat has access to the proxy.The proxy password is available on the <a href="https://my.apify.com/proxy" target="_blank">Proxy</a> page in the app.diff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.mdindex cfc58dcfd..3087abd24 100644--- a/docs/examples/Screenshots.md+++ b/docs/examples/Screenshots.md@@ -21,8 +21,8 @@ In local configuration, the input is stored in the default key-value store's dirOn the Apify cloud, the input can be either set manuallyin the UI app or passed as the POST payload to the-<a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>.-For more details, see <a href="https://www.apify.com/docs/actor#input-output" target="_blank">Input and output</a>+<a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>.+For more details, see <a href="https://apify.com/docs/actor#input-output" target="_blank">Input and output</a>in the Apify Actor documentation.To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base imagediff --git a/docs/examples/SynchronousRun.md b/docs/examples/SynchronousRun.mdindex 610ecd3dd..e26813292 100644--- a/docs/examples/SynchronousRun.md+++ b/docs/examples/SynchronousRun.md@@ -9,9 +9,9 @@ that contains a webcam stream from the Golden Gate Bridge, takes a screenshot ofThis actorcan be invoked synchronously using a single HTTP request to directly obtain its output as a reponse, using the-<a href="https://www.apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a>+<a href="https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a>Apify API endpoint. The example is also shared as the-<a href="https://www.apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a>+<a href="https://apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a>actor in the Apify library, so you can test it directly there simply by sending a POST request to```httphttps://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.mdindex 56e574ac8..9bf68d855 100644--- a/docs/typedefs/ActorRun.md+++ b/docs/typedefs/ActorRun.md@@ -8,10 +8,10 @@ Represents information about an actor run, as returned by the[`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.The object is almost equivalent to the JSON responseof the-<a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>+<a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>Apify API endpoint and extended with certain fields.For more details, see-<a href="https://www.apify.com/docs/actor#run" target="_blank">Runs.</a>+<a href="https://apify.com/docs/actor#run" target="_blank">Runs.</a>**Properties**<table>@@ -46,7 +46,7 @@ For more details, see</tr><tr><td colspan="3"><p>Status of the run. For possible values, see- <a href="https://www.apify.com/docs/actor#run-lifecycle" target="_blank">Run lifecycle</a>+ <a href="https://apify.com/docs/actor#run-lifecycle" target="_blank">Run lifecycle</a>in Apify actor documentation.</p></td></tr><tr><td><code>meta</code></td><td><code>Object</code></td>@@ -85,7 +85,7 @@ For more details, see</tr><tr><td colspan="3"><p>ID of the actor build used for the run. For details, see- <a href="https://www.apify.com/docs/actor#build" target="_blank">Builds</a>+ <a href="https://apify.com/docs/actor#build" target="_blank">Builds</a>in Apify actor documentation.</p></td></tr><tr><td><code>buildNumber</code></td><td><code>String</code></td>@@ -118,7 +118,7 @@ For more details, see<tr><td colspan="3"><p>URL on which the web server running inside actor run's Docker container can be accessed.For more details, see- <a href="https://www.apify.com/docs/actor#container-web-server" target="_blank">Container web server</a>+ <a href="https://apify.com/docs/actor#container-web-server" target="_blank">Container web server</a>in Apify actor documentation.</p></td></tr><tr><td><code>output</code></td><td><code>Object</code></td>diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.mdindex 27100a4f1..344c06e17 100644--- a/docs/typedefs/LaunchPuppeteerOptions.md+++ b/docs/typedefs/LaunchPuppeteerOptions.md@@ -55,13 +55,13 @@ function and in addition, all the options available below.<tr><td colspan="3"><p>If set to <code>true</code>, Puppeteer will be configured to use<a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- For more information, see the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a></p>+ For more information, see the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a></p></td></tr><tr><td><code>[apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td></tr><tr><td colspan="3"><p>An array of proxy groups to be used- by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ by the <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>.Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p></td></tr><tr><td><code>[apifyProxySession]</code></td><td><code>String</code></td><td></td>