Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Tuning up the docs (#589)
Ondra Urban28f5795ebe66
diff --git a/README.md b/README.mdindex cb6900fba..e163c7674 100644--- a/README.md+++ b/README.md@@ -9,17 +9,17 @@ Apify SDK simplifies the development of web crawlers, scrapers, data extractorsIt provides tools to manage and automatically scale a pool of headless Chrome / Puppeteer instances,to maintain queues of URLs to crawl, store crawling results to a local filesystem or into the cloud,rotate proxies and much more.-The SDK is available as the <a href="https://www.npmjs.com/package/apify" target="_blank"><code>apify</code></a> NPM package.+The SDK is available as the [`apify`](https://www.npmjs.com/package/apify) NPM package.It can be used either stand-alone in your own applications-or in <a href="https://docs.apify.com/actor" target="_blank">actors</a>-running on the <a href="https://apify.com/" target="_blank">Apify Cloud</a>.+or in [actors](https://docs.apify.com/actor)+running on the [Apify Cloud](https://apify.com/).-**View full documentation, guides and examples on the dedicated <a href="https://sdk.apify.com" target="_blank">Apify SDK project website</a>**+**View full documentation, guides and examples on the dedicated [Apify SDK project website](https://sdk.apify.com)**## Motivation-Thanks to tools like <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> or-<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>, it is easy to write Node.js code to extract data from web pages. But+Thanks to tools like [Puppeteer](https://github.com/GoogleChrome/puppeteer) or+[Cheerio](https://www.npmjs.com/package/cheerio), it is easy to write Node.js code to extract data from web pages. Buteventually things will get complicated. For example, when you try to:- Perform a deep crawl of an entire website using a persistent queue of URLs.@@ -28,7 +28,7 @@ eventually things will get complicated. For example, when you try to:- Schedule the code to run periodically and send notification on errors.- Disable browser fingerprinting protections used by websites.-Python has <a href="https://scrapy.org/" target="_blank">Scrapy</a> for these tasks, but there was no such library for **JavaScript, the language of+Python has [Scrapy](https://scrapy.org/) for these tasks, but there was no such library for **JavaScript, the language ofthe web**. The use of JavaScript is natural, since the same language is used to write the scripts as well as the data extraction code running in abrowser.@@ -38,87 +38,59 @@ commonalities.## Overview-The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify"><code>apify</code></a> NPM package and it provides the following tools:--<ul>- <li>- <a href="https://sdk.apify.com/docs/api/basiccrawler"><code>BasicCrawler</code></a>- - Provides a simple framework for the parallel crawling of web pages- whose URLs are fed either from a static list or from a dynamic queue of URLs.- This class serves as a base for more complex crawlers (see below).- </li>- <li>- <a href="https://sdk.apify.com/docs/api/cheeriocrawler"><code>CheerioCrawler</code></a>- - Enables the parallel crawling of a large number of web pages- using the <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a>- HTML parser.- This is the most efficient web crawler, but it does not work on websites that require JavaScript.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/puppeteercrawler"><code>PuppeteerCrawler</code></a>- - Enables the parallel crawling of a large number of web pages using the headless Chrome browser- and <a href="https://github.com/GoogleChrome/puppeteer">Puppeteer</a>.- The pool of Chrome browsers is automatically scaled up and down based on available system resources.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/puppeteerpool"><code>PuppeteerPool</code></a>- - Provides web browser tabs for user jobs- from an automatically-managed pool of Chrome browser instances, with configurable browser recycling and retirement policies.- Supports reuse of the disk cache to speed up the crawling of websites and reduce proxy bandwidth.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/requestlist"><code>RequestList</code></a>- - Represents a list of URLs to crawl. The URLs can be passed in code or in a text file hosted on the web.- The list persists its state so that crawling can resume- when the Node.js process restarts.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/requestqueue"><code>RequestQueue</code></a>- - Represents a queue of URLs to crawl, which is stored either on a local filesystem or in the <a href="https://apify.com" target="_blank">Apify Cloud</a>.- The queue is used for deep crawling of websites, where you start with- several URLs and then recursively follow links to other pages.- The data structure supports both breadth-first and depth-first crawling orders.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/dataset"><code>Dataset</code></a>- - Provides a store for structured data and enables their- export to formats like JSON, JSONL, CSV, XML, Excel or HTML.- The data is stored on a local filesystem or in the Apify Cloud.- Datasets are useful for storing and sharing large tabular crawling results,- such as a list of products or real estate offers.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/keyvaluestore"><code>KeyValueStore</code></a>- - A simple key-value store for arbitrary data records or files, along with their MIME content type.- It is ideal for saving screenshots of web pages, PDFs or to persist the state of your crawlers.- The data is stored on a local filesystem or in the Apify Cloud.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/autoscaledpool"><code>AutoscaledPool</code></a>- - Runs asynchronous background tasks, while automatically adjusting the concurrency- based on free system memory and CPU usage. This is useful for running web scraping tasks- at the maximum capacity of the system.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/puppeteer"><code>Puppeteer Utils</code></a>- - Provides several helper functions useful for web scraping. For example, to inject jQuery into web pages- or to hide browser origin.- </li>- <li>- Additionally, the package provides various helper functions to simplify- running your code on the Apify Cloud and thus- take advantage of its pool of proxies, job scheduler, data storage, etc.- For more information,- see the <a href="https://sdk.apify.com">Apify SDK Programmer's Reference</a>.- </li>-</ul>+The Apify SDK is available as the [`apify`](https://www.npmjs.com/package/apify) NPM package and it provides the following tools:++- [`BasicCrawler`](https://sdk.apify.com/docs/api/basic-crawler) - Provides a simple framework for the parallel+crawling of web pages whose URLs are fed either from a static list or from a dynamic queue of URLs. This class+serves as a base for more complex crawlers (see below).++- [`CheerioCrawler`](https://sdk.apify.com/docs/api/cheerio-crawler) - Enables the parallel crawling of a large+number of web pages using the [cheerio](https://www.npmjs.com/package/cheerio) HTML parser. This is the most+efficient web crawler, but it does not work on websites that require JavaScript.++- [`PuppeteerCrawler`](https://sdk.apify.com/docs/api/puppeteer-crawler) - Enables the parallel crawling of+a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/GoogleChrome/puppeteer).+The pool of Chrome browsers is automatically scaled up and down based on available system resources.++- [`PuppeteerPool`](https://sdk.apify.com/docs/api/puppeteer-pool) - Provides web browser tabs for user jobs+from an automatically-managed pool of Chrome browser instances, with configurable browser recycling+and retirement policies. Supports reuse of the disk cache to speed up the crawling of websites and reduce proxy bandwidth.++- [`RequestList`](https://sdk.apify.com/docs/api/request-list) - Represents a list of URLs to crawl.+The URLs can be passed in code or in a text file hosted on the web. The list persists its state so that crawling+can resume when the Node.js process restarts.++- [`RequestQueue`](https://sdk.apify.com/docs/api/request-queue) - Represents a queue of URLs to crawl,+which is stored either on a local filesystem or in the [Apify Cloud](https://apify.com). The queue is used+for deep crawling of websites, where you start with several URLs and then recursively follow links to other pages.+The data structure supports both breadth-first and depth-first crawling orders.++- [`Dataset`](https://sdk.apify.com/docs/api/dataset) - Provides a store for structured data and enables their export+to formats like JSON, JSONL, CSV, XML, Excel or HTML. The data is stored on a local filesystem or in the Apify Cloud.+Datasets are useful for storing and sharing large tabular crawling results, such as a list of products or real estate offers.++- [`KeyValueStore`](https://sdk.apify.com/docs/api/key-value-store) - A simple key-value store for arbitrary data+records or files, along with their MIME content type. It is ideal for saving screenshots of web pages, PDFs+or to persist the state of your crawlers. The data is stored on a local filesystem or in the Apify Cloud.++- [`AutoscaledPool`](https://sdk.apify.com/docs/api/autoscaled-pool) - Runs asynchronous background tasks,+while automatically adjusting the concurrency based on free system memory and CPU usage. This is useful for running+web scraping tasks at the maximum capacity of the system.++- [`Puppeteer Utils`](https://sdk.apify.com/docs/api/puppeteer) - Provides several helper functions useful+for web scraping. For example, to inject jQuery into web pages or to hide browser origin. Additionally,+the package provides various helper functions to simplify running your code on the Apify Cloud and thus+take advantage of its pool of proxies, job scheduler, data storage, etc. For more information,+see the [Apify SDK Programmer's Reference](https://sdk.apify.com).## Quick Start-The Apify SDK requires <a href="https://nodejs.org/en/" target="_blank">Node.js</a> 10 or later.+This short tutorial will set you up to start using Apify SDK in a minute or two.+If you want to learn more, proceed to the [Getting Started](https://sdk.apify.com/docs/guides/getting-started)+tutorial that will take you step by step through creating your first scraper.### Local stand-alone usage-+Apify SDK requires [Node.js](https://nodejs.org/en/) 10 or later.Add Apify SDK to any Node.js project by running:```bash@@ -126,7 +98,7 @@ npm install apify --save```Run the following example to perform a recursive crawl of a website using Puppeteer. For more examples showcasing various features of the Apify SDK,-[see the Examples section of the documentation](https://sdk.apify.com/docs/examples/basiccrawler).+[see the Examples section of the documentation](https://sdk.apify.com/docs/examples/basic-crawler).```javascriptconst Apify = require('apify');@@ -161,14 +133,13 @@ When you run the example, you should see Apify SDK automating a Chrome browser.By default, Apify SDK stores data to `./apify_storage` in the current working directory. You can override this behavior by setting either the-`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](https://sdk.apify.com/docs/guides/environmentvariables) and-[Data storage](https://sdk.apify.com/docs/guides/datastorage).+`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](https://sdk.apify.com/docs/guides/environment-variables) and+[Data storage](https://sdk.apify.com/docs/guides/data-storage).### Local usage with Apify command-line interface (CLI)To avoid the need to set the environment variables manually, to create a boilerplate of your project, and to enable pushing and running your code on-the <a href="https://apify.com" target="_blank">Apify Cloud</a>, you can use the-<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify command-line interface</a> (CLI) tool.+the [Apify Cloud](https://apify.com), you can use the [Apify command-line interface](https://github.com/apifytech/apify-cli)%20(CLI) tool.Install the CLI by running:@@ -206,16 +177,15 @@ apify push```Your script will be uploaded to the Apify Cloud and built there so that it can be run. For more information, view the-<a href="https://docs.apify.com/cli" target="_blank">Apify CLI</a> and <a href="https://docs.apify.com/actor" target="_blank">Apify Actor</a>-documentation.+[Apify Actor](https://docs.apify.com/cli) documentation.### Usage on the Apify Cloud-You can also develop your web scraping project in an online code editor directly on the <a href="https://apify.com" target="_blank">Apify Cloud</a>.-You'll need to have an Apify Account. Go to <a href="https://my.apify.com/actors" target="_blank">Actors</a>, page in the app, click <i>Create new</i>+You can also develop your web scraping project in an online code editor directly on the [Apify Cloud](https://apify.com/).+You'll need to have an Apify Account. Go to [Actors](https://my.apify.com/actors), page in the app, click <i>Create new</i>and then go to the <i>Source</i> tab and start writing your code or paste one of the examples from the Examples section.-For more information, view the <a href="https://docs.apify.com/actor/quick-start" target="_blank">Apify actors quick start guide</a>.+For more information, view the [Apify actors quick start guide](https://docs.apify.com/actor/quick-start).## Supportdiff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 2b48c4c48..1d5dd7ada 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -3,92 +3,52 @@ id: apifytitle: Apify----<a name="module_Apify"></a>+<a name="apify"></a>The following section describes all functions and properties provided by the `apify` package, except individual classes and namespaces that have their-separate, detailed, documentation pages accessible from the left sidebar.--- [Apify](#module_Apify)- - [`.addWebhook(options)`](#module_Apify.addWebhook) ⇒ `Promise<Object>`- - [`.call(actId, [input], [options])`](#module_Apify.call) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)- - [`.callTask(taskId, [input], [options])`](#module_Apify.callTask) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)- - [`.client`](#module_Apify.client) : `*`- - [`.events`](#module_Apify.events)- - [`.getApifyProxyUrl(options)`](#module_Apify.getApifyProxyUrl) ⇒ `String`- - [`.getEnv()`](#module_Apify.getEnv) ⇒ [`ApifyEnv`](../typedefs/apifyenv)- - [`.getInput`](#module_Apify.getInput) ⇒ `Promise<Object>`- - [`.getMemoryInfo()`](#module_Apify.getMemoryInfo) ⇒ [`Promise<MemoryInfo>`](../typedefs/memoryinfo)- - [`.getValue(key)`](#module_Apify.getValue) ⇒ `Promise<Object>`- - [`.isAtHome()`](#module_Apify.isAtHome) ⇒ `Boolean`- - [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ `Promise<Browser>`- - [`.main(userFunc)`](#module_Apify.main)- - [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ `Promise<void>`- - [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [`Promise<Dataset>`](dataset)- - [`.openKeyValueStore([storeIdOrName], [options])`](#module_Apify.openKeyValueStore) ⇒ [`Promise<KeyValueStore>`](keyvaluestore)- - [`.openRequestList`](#module_Apify.openRequestList) ⇒ [`Promise<RequestList>`](requestlist)- - [`.openRequestQueue`](#module_Apify.openRequestQueue) ⇒ [`Promise<RequestQueue>`](requestqueue)- - [`.openSessionPool`](#module_Apify.openSessionPool) ⇒ [`Promise<SessionPool>`](sessionpool)- - [`.pushData(item)`](#module_Apify.pushData) ⇒ `Promise`- - [`.setValue(key, value, [options])`](#module_Apify.setValue) ⇒ `Promise`--<a name="module_Apify.addWebhook"></a>--## `Apify.addWebhook(options)` ⇒ `Promise<Object>`+separate, detailed, documentation pages accessible from the left sidebar. To learn how Apify SDK works, we suggest following the+[Getting Started](/docs/guides/getting-started) tutorial.++---++<a name="addwebhook"></a>++## `Apify.addWebhook(options)`++**Returns**: `Promise<Object>` - The return value is the Webhook object. For more information, see the+[Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.Creates an ad-hoc webhook for the current actor run, which lets you receive a notification when the actor run finished or failed. For more information-about Apify actor webhooks, please see the <a href="https://docs.apify.com/webhooks" target="_blank">documentation</a>.+about Apify actor webhooks, please see the [documentation](https://docs.apify.com/webhooks).Note that webhooks are only supported for actors running on the Apify platform. In local environment, the function will print a warning and have noeffect.-**Returns**: `Promise<Object>` - The return value is the Webhook object. For more information, see the-[Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>options.eventTypes</code></td><td><code>Array<string></code></td>-</tr>-<tr>-<td colspan="3"><p>Array of event types, which you can set for actor run, see- the <a href="https://docs.apify.com/webhooks/events#actor-run" target="_blank">actor run events</a> in the Apify doc.</p>-</td></tr><tr>-<td><code>options.requestUrl</code></td><td><code>string</code></td>-</tr>-<tr>-<td colspan="3"><p>URL which will be requested using HTTP POST request, when actor run will reach the set event type.</p>-</td></tr><tr>-<td><code>[options.payloadTemplate]</code></td><td><code>string</code></td>-</tr>-<tr>-<td colspan="3"><p>Payload template is a JSON-like string that describes the structure of the webhook POST request payload.- It uses JSON syntax, extended with a double curly braces syntax for injecting variables <code>{{variable}}</code>.- Those variables are resolved at the time of the webhook's dispatch, and a list of available variables with their descriptions- is available in the <a href="https://docs.apify.com/webhooks" target="_blank">Apify webhook documentation</a>.</p>-<p> When omitted, the default payload template will be used.- <a href="https://docs.apify.com/webhooks" target="_blank">See the docs for the default payload template</a>.</p>-</td></tr><tr>-<td><code>[options.idempotencyKey]</code></td><td><code>string</code></td>-</tr>-<tr>-<td colspan="3"><p>Idempotency key enables you to ensure that a webhook will not be added multiple times in case of- an actor restart or other situation that would cause the <code>addWebhook()</code> function to be called again.- We suggest using the actor run ID as the idempotency key. You can get the run ID by calling- <a href="apify#module_Apify.getEnv"><code>Apify.getEnv()</code></a> function.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.call"></a>+- **`options`**: `Object`++ - **`.eventTypes`**: `Array<string>` - Array of event types, which you can set for actor run, see the+ [actor run events](https://docs.apify.com/webhooks/events#actor-run) in the Apify doc.+ - **`.requestUrl`**: `string` - URL which will be requested using HTTP POST request, when actor run will reach the set event type.+ - **`[.payloadTemplate]`**: `string` - Payload template is a JSON-like string that describes the structure of the webhook POST request payload.+ It uses JSON syntax, extended with a double curly braces syntax for injecting variables `{{variable}}`. Those variables are resolved at the+ time of the webhook's dispatch, and a list of available variables with their descriptions is available in the+ [Apify webhook documentation](https://docs.apify.com/webhooks).++ When omitted, the default payload template will be used. [See the docs for the default payload template](https://docs.apify.com/webhooks).++ - **`[.idempotencyKey]`**: `string` - Idempotency key enables you to ensure that a webhook will not be added multiple times in case of an actor+ restart or other situation that would cause the `addWebhook()` function to be called again. We suggest using the actor run ID as the+ idempotency key. You can get the run ID by calling [`Apify.getEnv()`](/docs/api/apify#getenv) function.++---++<a name="call"></a>-## `Apify.call(actId, [input], [options])` ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)+## `Apify.call(actId, [input], [options])`++**Returns**: [`Promise<ActorRun>`](/docs/typedefs/actor-run)Runs an actor on the Apify platform using the current user account (determined by the `APIFY_TOKEN` environment variable), waits for the actor tofinish and fetches its output.@@ -96,12 +56,12 @@ finish and fetches its output.By passing the `waitSecs` option you can reduce the maximum amount of time to wait for the run to finish. If the value is less than or equal to zero,the function returns immediately after the run is started.-The result of the function is an [`ActorRun`](../typedefs/actorrun) object that contains details about the actor run and its output (if any). If the-actor run fails, the function throws the [`ApifyCallError`](../typedefs/apifycallerror) exception.+The result of the function is an [`ActorRun`](/docs/typedefs/actor-run) object that contains details about the actor run and its output (if any). If+the actor run fails, the function throws the [`ApifyCallError`](/docs/typedefs/apify-call-error) exception.-If you want to run an actor task rather than an actor, please use the [`Apify.callTask()`](../api/apify#module_Apify.callTask) function instead.+If you want to run an actor task rather than an actor, please use the [`Apify.callTask()`](/docs/api/apify#calltask) function instead.-For more information about actors, read the <a href="https://docs.apify.com/actor" target="_blank">documentation</a>.+For more information about actors, read the [documentation](https://docs.apify.com/actor).**Example usage:**@@ -110,99 +70,45 @@ const run = await Apify.call('apify/hello-world', { myInput: 123 });console.log(`Received message: ${run.output.body.message}`);```-Internally, the `call()` function invokes the <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run-actor</a> and several other API endpoints to obtain the output.+Internally, the `call()` function invokes the [Run actor](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor) and several other+API endpoints to obtain the output.**Throws**:-- [`ApifyCallError`](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.+- [`ApifyCallError`](/docs/typedefs/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.++**Params**++- **`actId`**: `String` - Either `username/actor-name` or actor ID.+- **`[input]`**: `Object` | `String` | `Buffer` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set+ to `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.+- **`[options]`**: `Object` - Object with the settings below:+ - **`[.contentType]`**: `String` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified+ to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or+ `Buffer`.+ - **`[.token]`**: `String` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.+ - **`[.memoryMbytes]`**: `Number` - Memory in megabytes which will be allocated for the new actor run. If not provided, the run uses memory of+ the default actor run configuration.+ - **`[.timeoutSecs]`**: `Number` - Timeout for the actor run in seconds. Zero value means there is no timeout. If not provided, the run uses+ timeout of the default actor run configuration.+ - **`[.build]`**: `String` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or+ number from the default actor run configuration (typically `latest`).+ - **`[.waitSecs]`**: `String` - Maximum time to wait for the actor run to finish, in seconds. If the limit is reached, the returned promise is+ resolved to a run object that will have status `READY` or `RUNNING` and it will not contain the actor run output. If `waitSecs` is null or+ undefined, the function waits for the actor to finish (default behavior).+ - **`[.fetchOutput]`**: `Boolean` <code> = true</code> - If `false` then the function does not fetch output of the actor.+ - **`[.disableBodyParser]`**: `Boolean` <code> = false</code> - If `true` then the function will not attempt to parse the actor's output and+ will return it in a raw `Buffer`.+ - **`[.webhooks]`**: `Array` - Specifies optional webhooks associated with the actor run, which can be used to receive a notification e.g. when+ the actor finished or failed, see [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>actId</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Either <code>username/actor-name</code> or actor ID.</p>-</td></tr><tr>-<td><code>[input]</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Input for the actor. If it is an object, it will be stringified to- JSON and its content type set to <code>application/json; charset=utf-8</code>.- Otherwise the <code>options.contentType</code> parameter must be provided.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Object with the settings below:</p>-</td></tr><tr>-<td><code>[options.contentType]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Content type for the <code>input</code>. If not specified,- <code>input</code> is expected to be an object that will be stringified to JSON and content type set to- <code>application/json; charset=utf-8</code>. If <code>options.contentType</code> is specified, then <code>input</code> must be a- <code>String</code> or <code>Buffer</code>.</p>-</td></tr><tr>-<td><code>[options.token]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>User API token that is used to run the actor. By default, it is taken from the <code>APIFY_TOKEN</code> environment variable.</p>-</td></tr><tr>-<td><code>[options.memoryMbytes]</code></td><td><code>Number</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Memory in megabytes which will be allocated for the new actor run.- If not provided, the run uses memory of the default actor run configuration.</p>-</td></tr><tr>-<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Timeout for the actor run in seconds. Zero value means there is no timeout.- If not provided, the run uses timeout of the default actor run configuration.</p>-</td></tr><tr>-<td><code>[options.build]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Tag or number of the actor build to run (e.g. <code>beta</code> or <code>1.2.345</code>).- If not provided, the run uses build tag or number from the default actor run configuration (typically <code>latest</code>).</p>-</td></tr><tr>-<td><code>[options.waitSecs]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Maximum time to wait for the actor run to finish, in seconds.- If the limit is reached, the returned promise is resolved to a run object that will have- status <code>READY</code> or <code>RUNNING</code> and it will not contain the actor run output.- If <code>waitSecs</code> is null or undefined, the function waits for the actor to finish (default behavior).</p>-</td></tr><tr>-<td><code>[options.fetchOutput]</code></td><td><code>Boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>false</code> then the function does not fetch output of the actor.</p>-</td></tr><tr>-<td><code>[options.disableBodyParser]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the function will not attempt to parse the- actor's output and will return it in a raw <code>Buffer</code>.</p>-</td></tr><tr>-<td><code>[options.webhooks]</code></td><td><code>Array</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Specifies optional webhooks associated with the actor run, which can be used- to receive a notification e.g. when the actor finished or failed, see- <a href="https://docs.apify.com/webhooks/ad-hoc-webhooks">ad hook webhooks documentation</a> for detailed description.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.callTask"></a>+---++<a name="calltask"></a>-## `Apify.callTask(taskId, [input], [options])` ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)+## `Apify.callTask(taskId, [input], [options])`++**Returns**: [`Promise<ActorRun>`](/docs/typedefs/actor-run)Runs an actor task on the Apify platform using the current user account (determined by the `APIFY_TOKEN` environment variable), waits for the task tofinish and fetches its output.@@ -210,13 +116,13 @@ finish and fetches its output.By passing the `waitSecs` option you can reduce the maximum amount of time to wait for the run to finish. If the value is less than or equal to zero,the function returns immediately after the run is started.-The result of the function is an [`ActorRun`](../typedefs/actorrun) object that contains details about the actor run and its output (if any). If the-actor run failed, the function fails with [`ApifyCallError`](../typedefs/apifycallerror) exception.+The result of the function is an [`ActorRun`](/docs/typedefs/actor-run) object that contains details about the actor run and its output (if any). If+the actor run failed, the function fails with [`ApifyCallError`](/docs/typedefs/apify-call-error) exception.Note that an actor task is a saved input configuration and options for an actor. If you want to run an actor directly rather than an actor task,-please use the [`Apify.call()`](../api/apify#module_Apify.call) function instead.+please use the [`Apify.call()`](/docs/api/apify#call) function instead.-For more information about actor tasks, read the [`documentation`](https://docs.apify.com/tasks).+For more information about actor tasks, read the [documentation](https://docs.apify.com/tasks).**Example usage:**@@ -225,108 +131,60 @@ const run = await Apify.callTask('bob/some-task');console.log(`Received message: ${run.output.body.message}`);```-Internally, the `callTask()` function calls the-<a href="https://apify.com/docs/api/v2#/reference/actor-tasks/run-collection/run-task" target="_blank">Run task</a> and several other API endpoints to-obtain the output.+Internally, the `callTask()` function calls the [Run task](https://apify.com/docs/api/v2#/reference/actor-tasks/run-collection/run-task) and several+other API endpoints to obtain the output.**Throws**:-- [`ApifyCallError`](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.+- [`ApifyCallError`](/docs/typedefs/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.++**Params**++- **`taskId`**: `String` - Either `username/task-name` or task ID.+- **`[input]`**: `Object` | `String` | `Buffer` - Input overrides for the actor task. If it is an object, it will be stringified to JSON and its+ content type set to `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided. Provided input will be+ merged with actor task input.+- **`[options]`**: `Object` - Object with the settings below:+ - **`[.contentType]`**: `String` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified+ to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or+ `Buffer`.+ - **`[.token]`**: `String` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.+ - **`[.memoryMbytes]`**: `Number` - Memory in megabytes which will be allocated for the new actor task run. If not provided, the run uses memory+ of the default actor run configuration.+ - **`[.timeoutSecs]`**: `Number` - Timeout for the actor task run in seconds. Zero value means there is no timeout. If not provided, the run+ uses timeout of the default actor run configuration.+ - **`[.build]`**: `String` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or+ number from the default actor run configuration (typically `latest`).+ - **`[.waitSecs]`**: `String` - Maximum time to wait for the actor task run to finish, in seconds. If the limit is reached, the returned promise+ is resolved to a run object that will have status `READY` or `RUNNING` and it will not contain the actor run output. If `waitSecs` is null or+ undefined, the function waits for the actor task to finish (default behavior).+ - **`[.webhooks]`**: `Array` - Specifies optional webhooks associated with the actor run, which can be used to receive a notification e.g. when+ the actor finished or failed, see [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>taskId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Either <code>username/task-name</code> or task ID.</p>-</td></tr><tr>-<td><code>[input]</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td>-</tr>-<tr>-<td colspan="3"><p>Input overrides for the actor task. If it is an object, it will be stringified to- JSON and its content type set to <code>application/json; charset=utf-8</code>.- Otherwise the <code>options.contentType</code> parameter must be provided.- Provided input will be merged with actor task input.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Object with the settings below:</p>-</td></tr><tr>-<td><code>[options.contentType]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Content type for the <code>input</code>. If not specified,- <code>input</code> is expected to be an object that will be stringified to JSON and content type set to- <code>application/json; charset=utf-8</code>. If <code>options.contentType</code> is specified, then <code>input</code> must be a- <code>String</code> or <code>Buffer</code>.</p>-</td></tr><tr>-<td><code>[options.token]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>User API token that is used to run the actor. By default, it is taken from the <code>APIFY_TOKEN</code> environment variable.</p>-</td></tr><tr>-<td><code>[options.memoryMbytes]</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Memory in megabytes which will be allocated for the new actor task run.- If not provided, the run uses memory of the default actor run configuration.</p>-</td></tr><tr>-<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Timeout for the actor task run in seconds. Zero value means there is no timeout.- If not provided, the run uses timeout of the default actor run configuration.</p>-</td></tr><tr>-<td><code>[options.build]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Tag or number of the actor build to run (e.g. <code>beta</code> or <code>1.2.345</code>).- If not provided, the run uses build tag or number from the default actor run configuration (typically <code>latest</code>).</p>-</td></tr><tr>-<td><code>[options.waitSecs]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum time to wait for the actor task run to finish, in seconds.- If the limit is reached, the returned promise is resolved to a run object that will have- status <code>READY</code> or <code>RUNNING</code> and it will not contain the actor run output.- If <code>waitSecs</code> is null or undefined, the function waits for the actor task to finish (default behavior).</p>-</td></tr><tr>-<td><code>[options.webhooks]</code></td><td><code>Array</code></td>-</tr>-<tr>-<td colspan="3"><p>Specifies optional webhooks associated with the actor run, which can be used- to receive a notification e.g. when the actor finished or failed, see- <a href="https://docs.apify.com/webhooks/ad-hoc-webhooks">ad hook webhooks documentation</a> for detailed description.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.client"></a>+----## `Apify.client` : `*`+<a name="client"></a>++## `Apify.client`Gets the default instance of the `ApifyClient` class provided <a href="https://docs.apify.com/api/apify-client-js/latest"target="_blank">apify-client</a> by the NPM package. The instance is created automatically by the Apify SDK and it is configured using the`APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN` environment variables.-The instance is used for all underlying calls to the Apify API in functions such as [`Apify.getValue()`](#module_Apify.getValue) or-[`Apify.call()`](#module_Apify.call). The settings of the client can be globally altered by calling the+The instance is used for all underlying calls to the Apify API in functions such as [`Apify.getValue()`](/docs/api/apify#getvalue) or+[`Apify.call()`](/docs/api/apify#call). The settings of the client can be globally altered by calling the<a href="https://docs.apify.com/api/apify-client-js/latest#ApifyClient-setOptions"target="_blank">`Apify.client.setOptions()`</a> function. Beware that altering these settings might have unintended effects on the entire Apify SDKpackage.-<a name="module_Apify.events"></a>+---++<a name="events"></a>## `Apify.events`-Gets an instance of a Node.js' <a href="https://nodejs.org/api/events.html#events_class_eventemitter" target="_blank">EventEmitter</a> class that-emits various events from the SDK or the Apify platform. The event emitter is initialized by calling the [`Apify.main()`](#module_Apify.main)-function.+Gets an instance of a Node.js' [EventEmitter](https://nodejs.org/api/events.html#events_class_eventemitter) class that emits various events from the+SDK or the Apify platform. The event emitter is initialized by calling the [`Apify.main()`](/docs/api/apify#main) function.**Example usage:**@@ -354,7 +212,7 @@ The following table shows all currently emitted events:The event is emitted approximately every secondand it indicates whether the actor is using the maximum of available CPU resources.If that's the case, the actor should not add more workload.- For example, this event is used by the <a href="autoscaledpool"><code>AutoscaledPool</code></a> class.+ For example, this event is used by the [`AutoscaledPool`](/docs/api/autoscaled-pool) class.</td></tr><tr>@@ -365,7 +223,7 @@ The following table shows all currently emitted events:<td colspan="2">Emitted when the actor running on the Apify platform is going to be migrated to another worker server soon.You can use it to persist the state of the actor and abort the run, to speed up migration.- For example, this is used by the <a href="requestlist"><code>RequestList</code></a> class.+ For example, this is used by the [`RequestList`](/docs/api/request-list) class.</td></tr><tr>@@ -386,85 +244,60 @@ The following table shows all currently emitted events:</tbody></table>-<a name="module_Apify.getApifyProxyUrl"></a>+---++<a name="getapifyproxyurl"></a>++## `Apify.getApifyProxyUrl(options)`-## `Apify.getApifyProxyUrl(options)` ⇒ `String`+**Returns**: `String` - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.Constructs an Apify Proxy URL using the specified settings. The proxy URL can be used from Apify actors, web browsers or any other HTTP proxy-enabledapplications.-For more information, see the <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the app or the-<a href="https://docs.apify.com/proxy" target="_blank">documentation</a>.+For more information, see the [Apify Proxy](https://my.apify.com/proxy) page in the app or the [documentation](https://docs.apify.com/proxy).-**Returns**: `String` - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Object with the settings below:</p>-</td></tr><tr>-<td><code>[options.password]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>User's password for the proxy.- By default, it is taken from the <code>APIFY_PROXY_PASSWORD</code> environment variable,- which is automatically set by the system when running the actors on the Apify cloud,- or when using the <a href="https://github.com/apifytech/apify-cli" target="_blank">Apify CLI</a>- package and the user previously logged in (called <code>apify login</code>).</p>-</td></tr><tr>-<td><code>[options.groups]</code></td><td><code>Array<String></code></td>-</tr>-<tr>-<td colspan="3"><p>Array of Apify Proxy groups to be used.- If not provided, the proxy will select the groups automatically.</p>-</td></tr><tr>-<td><code>[options.session]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Apify Proxy session identifier to be used by the Chrome browser.- All HTTP requests going through the proxy with the same session identifier- will use the same target proxy server (i.e. the same IP address), unless using Residential proxies.- The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>"."</code>, <code>"_"</code> and <code>"~"</code>.</p>-</td></tr><tr>-<td><code>[options.country]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>If specified, all proxied requests will use IP addresses that are geolocated to the specified country.-For example <code>GB</code> for IPs from Great Britain. Note that online services often have their own rules for handling geolocation and thus-the country selection is a best attempt at geolocation, rather than a guaranteed hit.-This parameter is optional, by default, each proxied request is assigned an IP address from a random country.-The country code needs to be a two letter ISO country code-- see the <a href="https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements" target="_blank">-full list of available country codes-</a>.</p>-<p>This parameter is optional, by default, the proxy uses all available proxy servers from all countries.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.getEnv"></a>+- **`options`**: `Object` - Object with the settings below: - **`[.password]`**: `String` - User's password for the proxy. By default, it is taken+ from the `APIFY_PROXY_PASSWORD` environment variable, which is automatically set by the system when running the actors on the Apify cloud, or when+ using the [Apify CLI](https://github.com/apifytech/apify-cli) package and the user previously logged in (called `apify login`). - **`[.groups]`**:+ `Array<String>` - Array of Apify Proxy groups to be used. If not provided, the proxy will select the groups automatically. - **`[.session]`**:+ `String` - Apify Proxy session identifier to be used by the Chrome browser. All HTTP requests going through the proxy with the same session+ identifier will use the same target proxy server (i.e. the same IP address), unless using Residential proxies. The identifier can only contain the+ following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`. - **`[.country]`**: `String` - If specified, all proxied requests will use IP+ addresses that are geolocated to the specified country. For example `GB` for IPs from Great Britain. Note that online services often have their+ own rules for handling geolocation and thus the country selection is a best attempt at geolocation, rather than a guaranteed hit. This parameter+ is optional, by default, each proxied request is assigned an IP address from a random country. The country code needs to be a two letter ISO+ country code \- see the <a href="https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements" target="_blank"> full list of+ available country codes </a>.-## `Apify.getEnv()` ⇒ [`ApifyEnv`](../typedefs/apifyenv)+This parameter is optional, by default, the proxy uses all available proxy servers from all countries.-Returns a new [`ApifyEnv`](../typedefs/apifyenv) object which contains information parsed from all the `APIFY_XXX` environment variables.+----For the list of the `APIFY_XXX` environment variables, see <a href="https://docs.apify.com/actor/run#environment-variables" target="_blank">Actor-documentation</a>. If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.+<a name="getenv"></a>-<a name="module_Apify.getInput"></a>+## `Apify.getEnv()`-## `Apify.getInput` ⇒ `Promise<Object>`+**Returns**: [`ApifyEnv`](/docs/typedefs/apify-env)-Gets the actor input value from the default [`KeyValueStore`](keyvaluestore) associated with the current actor run.+Returns a new [`ApifyEnv`](/docs/typedefs/apify-env) object which contains information parsed from all the `APIFY_XXX` environment variables.-This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](keyvaluestore#KeyValueStore+getValue). For example, calling the following-code:+For the list of the `APIFY_XXX` environment variables, see [Actor documentation](https://docs.apify.com/actor/run#environment-variables). If some of+the variables are not defined or are invalid, the corresponding value in the resulting object will be null.++---++<a name="getinput"></a>++## `Apify.getInput()`++**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.++Gets the actor input value from the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.++This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](keyvaluestore#getvalue). For example, calling the following code:```javascriptconst input = await Apify.getInput();@@ -477,28 +310,35 @@ const store = await Apify.openKeyValueStore();await store.getValue('INPUT');```-For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore) and-[`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyvaluestore) and+[`KeyValueStore.getValue()`](/docs/api/key-value-store#getvalue).++---++<a name="getmemoryinfo"></a>-**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.-<a name="module_Apify.getMemoryInfo"></a>+## `Apify.getMemoryInfo()`-## `Apify.getMemoryInfo()` ⇒ [`Promise<MemoryInfo>`](../typedefs/memoryinfo)+**Returns**: [`Promise<MemoryInfo>`](/docs/typedefs/memory-info)-Returns memory statistics of the process and the system, see [`MemoryInfo`](../typedefs/memoryinfo).+Returns memory statistics of the process and the system, see [`MemoryInfo`](/docs/typedefs/memory-info).If the process runs inside of Docker, the `getMemoryInfo` gets container memory limits, otherwise it gets system memory limits.Beware that the function is quite inefficient because it spawns a new process. Therefore you shouldn't call it too often, like more than once persecond.-<a name="module_Apify.getValue"></a>+---++<a name="getvalue"></a>++## `Apify.getValue(key)`-## `Apify.getValue(key)` ⇒ `Promise<Object>`+**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.-Gets a value from the default [`KeyValueStore`](keyvaluestore) associated with the current actor run.+Gets a value from the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.-This is just a convenient shortcut for [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue). For example, calling the following code:+This is just a convenient shortcut for [`KeyValueStore.getValue()`](/docs/api/key-value-store#getvalue). For example, calling the following code:```javascriptconst value = await Apify.getValue('my-key');@@ -511,36 +351,32 @@ const store = await Apify.openKeyValueStore();const value = await store.getValue('my-key');```-To store the value to the default-key value store, you can use the [`Apify.setValue()`](#module_Apify.setValue) function.+To store the value to the default-key value store, you can use the [`Apify.setValue()`](/docs/api/apify#setvalue) function.-For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore) and-[`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyvaluestore) and+[`KeyValueStore.getValue()`](/docs/api/key-value-store#getvalue).-**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>key</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Unique record key.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.isAtHome"></a>+- **`key`**: `String` - Unique record key.-## `Apify.isAtHome()` ⇒ `Boolean`+---++<a name="isathome"></a>++## `Apify.isAtHome()`++**Returns**: `Boolean`Returns `true` when code is running on Apify platform and `false` otherwise (for example locally).-<a name="module_Apify.launchPuppeteer"></a>+----## `Apify.launchPuppeteer([options])` ⇒ `Promise<Browser>`+<a name="launchpuppeteer"></a>++## `Apify.launchPuppeteer([options])`++**Returns**: `Promise<Browser>` - Promise that resolves to Puppeteer's `Browser` instance.Launches headless Chrome using Puppeteer pre-configured to work within the Apify platform. The function has the same argument and the return value as`puppeteer.launch()`. See <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions" target="_blank">@@ -566,7 +402,7 @@ The `launchPuppeteer()` function alters the following Puppeteer options:</li><li>If <code>options.useApifyProxy</code> is <code>true</code> then the function generates a URL of- <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>+ [Apify Proxy](https://docs.apify.com/proxy)based on <code>options.apifyProxyGroups</code> and <code>options.apifyProxySession</code> and passes it as <code>options.proxyUrl</code>.</li><li>@@ -581,40 +417,31 @@ The `launchPuppeteer()` function alters the following Puppeteer options:</li></ul>-To use this function, you need to have the <a href="https://www.npmjs.com/package/puppeteer" target="_blank">puppeteer</a> NPM package installed in-your project. When running on the Apify cloud, you can achieve that simply by using the `apify/actor-node-chrome` base Docker image for your actor --see <a href="https://docs.apify.com/actor/build#base-images" target="_blank">Apify Actor documentation</a> for details.+To use this function, you need to have the [puppeteer](https://www.npmjs.com/package/puppeteer) NPM package installed in your project. When running on+the Apify cloud, you can achieve that simply by using the `apify/actor-node-chrome` base Docker image for your actor - see+[Apify Actor documentation](https://docs.apify.com/actor/build#base-images) for details.For an example of usage, see the [Synchronous run Example](../examples/synchronousrun) or the[Puppeteer proxy Example](../examples/puppeteerwithproxy)-**Returns**: `Promise<Browser>` - Promise that resolves to Puppeteer's `Browser` instance.+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Optional settings passed to <code>puppeteer.launch()</code>. In addition to- <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank">Puppeteer's options</a>- the object may contain our own <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a> that enable additional features.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.main"></a>+- **`[options]`**: [`LaunchPuppeteerOptions`](/docs/typedefs/launch-puppeteer-options) - Optional settings passed to `puppeteer.launch()`. In+ addition to [Puppeteer's options](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions) the object may contain our own+ [`LaunchPuppeteerOptions`](/docs/typedefs/launch-puppeteer-options) that enable additional features.++---++<a name="main"></a>## `Apify.main(userFunc)`Runs the main user function that performs the job of the actor and terminates the process when the user function finishes.-_The `Apify.main()` function is optional_ and is provided merely for your convenience. It is especially useful when you're running your code as an-actor on the [Apify platform](https://apify.com/actors). However, if you want to use Apify SDK tools directly inside your existing projects or outside-of Apify platform, it's probably better to avoid it since it terminates the main process.+**The `Apify.main()` function is optional** and is provided merely for your convenience. It is mainly useful when you're running your code as an actor+on the [Apify platform](https://apify.com/actors). However, if you want to use Apify SDK tools directly inside your existing projects, e.g. running in+an [Express](https://expressjs.com/) server, on [Google Cloud functions](https://cloud.google.com/functions) or+[AWS Lambda](https://aws.amazon.com/lambda/), it's better to avoid it since the function terminates the main process when it finishes!The `Apify.main()` function performs the following actions:@@ -622,7 +449,7 @@ The `Apify.main()` function performs the following actions:<li>When running on the Apify platform (i.e. <code>APIFY_IS_AT_HOME</code> environment variable is set),it sets up a connection to listen for platform events.For example, to get a notification about an imminent migration to another server.- See <a href="apify#apifyevents"><code>Apify.events</code></a> for details.+ See [](apify#apifyevents) for details.</li><li>It checks that either <code>APIFY_TOKEN</code> or <code>APIFY_LOCAL_STORAGE_DIR</code> environment variableis defined. If not, the functions sets <code>APIFY_LOCAL_STORAGE_DIR</code> to <code>./apify_storage</code>@@ -669,151 +496,95 @@ Apify.main(async () => {});```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>userFunc</code></td><td><code>function</code></td>-</tr>-<tr>-<td colspan="3"><p>User function to be executed. If it returns a promise,-the promise will be awaited. The user function is called with no arguments.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.metamorph"></a>+**Params**-## `Apify.metamorph(targetActorId, [input], [options])` ⇒ `Promise<void>`+- **`userFunc`**: `function` - User function to be executed. If it returns a promise, the promise will be awaited. The user function is called with+ no arguments.++---++<a name="metamorph"></a>++## `Apify.metamorph(targetActorId, [input], [options])`++**Returns**: `Promise<void>`Transforms this actor run to an actor run of a given actor. The system stops the current container and starts the new container instead. All thedefault storages are preserved and the new input is stored under the `INPUT-METAMORPH-1` key in the same default key-value store.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>targetActorId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Either <code>username/actor-name</code> or actor ID of an actor to which we want to metamorph.</p>-</td></tr><tr>-<td><code>[input]</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td>-</tr>-<tr>-<td colspan="3"><p>Input for the actor. If it is an object, it will be stringified to- JSON and its content type set to <code>application/json; charset=utf-8</code>.- Otherwise the <code>options.contentType</code> parameter must be provided.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Object with the settings below:</p>-</td></tr><tr>-<td><code>[options.contentType]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Content type for the <code>input</code>. If not specified,- <code>input</code> is expected to be an object that will be stringified to JSON and content type set to- <code>application/json; charset=utf-8</code>. If <code>options.contentType</code> is specified, then <code>input</code> must be a- <code>String</code> or <code>Buffer</code>.</p>-</td></tr><tr>-<td><code>[options.build]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Tag or number of the target actor build to metamorph into (e.g. <code>beta</code> or <code>1.2.345</code>).- If not provided, the run uses build tag or number from the default actor run configuration (typically <code>latest</code>).</p>-</td></tr></tbody>-</table>-<a name="module_Apify.openDataset"></a>+**Params**++- **`targetActorId`**: `String` - Either `username/actor-name` or actor ID of an actor to which we want to metamorph.+- **`[input]`**: `Object` | `String` | `Buffer` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set+ to `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.+- **`[options]`**: `Object` - Object with the settings below:+ - **`[.contentType]`**: `String` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified+ to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or+ `Buffer`.+ - **`[.build]`**: `String` - Tag or number of the target actor build to metamorph into (e.g. `beta` or `1.2.345`). If not provided, the run uses+ build tag or number from the default actor run configuration (typically `latest`).++----## `Apify.openDataset([datasetIdOrName], [options])` ⇒ [`Promise<Dataset>`](dataset)+<a name="opendataset"></a>-Opens a dataset and returns a promise resolving to an instance of the [`Dataset`](dataset) class.+## `Apify.openDataset([datasetIdOrName], [options])`++**Returns**: [`Promise<Dataset>`](/docs/api/dataset)++Opens a dataset and returns a promise resolving to an instance of the [`Dataset`](/docs/api/dataset) class.Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. Theactual data is stored either on the local filesystem or in the cloud.-For more details and code examples, see the [`Dataset`](dataset) class.+For more details and code examples, see the [`Dataset`](/docs/api/dataset) class.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[datasetIdOrName]</code></td><td><code>string</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>ID or name of the dataset to be opened. If <code>null</code> or <code>undefined</code>,- the function returns the default dataset associated with the actor run.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.forceCloud]</code></td><td><code>boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to <code>true</code> then the function uses cloud storage usage even if the <code>APIFY_LOCAL_STORAGE_DIR</code>- environment variable is set. This way it is possible to combine local and cloud storage.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.openKeyValueStore"></a>+**Params**-## `Apify.openKeyValueStore([storeIdOrName], [options])` ⇒ [`Promise<KeyValueStore>`](keyvaluestore)+- **`[datasetIdOrName]`**: `string` - ID or name of the dataset to be opened. If `null` or `undefined`, the function returns the default dataset+ associated with the actor run.+- **`[options]`**: `object`+ - **`[.forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the+ `APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.-Opens a key-value store and returns a promise resolving to an instance of the [`KeyValueStore`](keyvaluestore) class.+---++<a name="openkeyvaluestore"></a>++## `Apify.openKeyValueStore([storeIdOrName], [options])`++**Returns**: [`Promise<KeyValueStore>`](/docs/api/key-value-store)++Opens a key-value store and returns a promise resolving to an instance of the [`KeyValueStore`](/docs/api/key-value-store) class.Key-value stores are used to store records or files, along with their MIME content type. The records are stored and retrieved using a unique key. Theactual data is stored either on a local filesystem or in the Apify cloud.-For more details and code examples, see the [`KeyValueStore`](keyvaluestore) class.+For more details and code examples, see the [`KeyValueStore`](/docs/api/key-value-store) class.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[storeIdOrName]</code></td><td><code>string</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>ID or name of the key-value store to be opened. If <code>null</code> or <code>undefined</code>,- the function returns the default key-value store associated with the actor run.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.forceCloud]</code></td><td><code>boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to <code>true</code> then the function uses cloud storage usage even if the <code>APIFY_LOCAL_STORAGE_DIR</code>- environment variable is set. This way it is possible to combine local and cloud storage.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.openRequestList"></a>+**Params**-## `Apify.openRequestList` ⇒ [`Promise<RequestList>`](requestlist)+- **`[storeIdOrName]`**: `string` - ID or name of the key-value store to be opened. If `null` or `undefined`, the function returns the default+ key-value store associated with the actor run.+- **`[options]`**: `object`+ - **`[.forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the+ `APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.++----Opens a request list and returns a promise resolving to an instance of the [`RequestList`](requestlist) class that is already initialized.+<a name="openrequestlist"></a>-[`RequestList`](requestlist) represents a list of URLs to crawl, which is always stored in memory. To enable picking up where left off after a process-restart, the request list sources are persisted to the key value store at initialization of the list. Then, while crawling, a small state object is-regularly persisted to keep track of the crawling status.+## `Apify.openRequestList(listName, sources, [options])`-For more details and code examples, see the [`RequestList`](requestlist) class.+**Returns**: [`Promise<RequestList>`](/docs/api/request-list)++Opens a request list and returns a promise resolving to an instance of the [`RequestList`](/docs/api/request-list) class that is already initialized.++[`RequestList`](/docs/api/request-list) represents a list of URLs to crawl, which is always stored in memory. To enable picking up where left off+after a process restart, the request list sources are persisted to the key value store at initialization of the list. Then, while crawling, a small+state object is regularly persisted to keep track of the crawling status.++For more details and code examples, see the [`RequestList`](/docs/api/request-list) class.**Example usage:**@@ -823,112 +594,83 @@ const sources = ['https://www.example.com', 'https://www.google.com', 'https://wconst requestList = await Apify.openRequestList('my-name', sources);```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>listName</code></td><td><code>string</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>Name of the request list to be opened. Setting a name enables the <code>RequestList</code>'s state to be persisted- in the key value store. This is useful in case of a restart or migration. Since <code>RequestList</code> is only- stored in memory, a restart or migration wipes it clean. Setting a name will enable the <code>RequestList</code>'s- state to survive those situations and continue where it left off.</p>-<p> The name will be used as a prefix in key value store, producing keys such as <code>NAME-REQUEST_LIST_STATE</code>- and <code>NAME-REQUEST_LIST_SOURCES</code>.</p>-<p> If <code>null</code>, the list will not be persisted and will only be stored in memory. Process restart- will then cause the list to be crawled again from the beginning. We suggest always using a name.</p>-</td></tr><tr>-<td><code>sources</code></td><td><code>Array<(Request|RequestOptions|string)></code></td>-</tr>-<tr>-<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>.- It can be either an array of plain objects that- define the <code>url</code> property, or an array of instances of the <a href="request"><code>Request</code></a> class.</p>-<p> Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,- which will instruct <code>RequestList</code> to download the source URLs from a given remote location.- The URLs will be parsed from the received response. In this case you can limit the URLs-using <code>regex</code> parameter containing regular expression pattern for URLs to be included.</p>-<p> For details, see the <a href="requestlist#new_RequestList_new"><code>RequestList</code></a>- constructor options.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code><a href="../typedefs/requestlistoptions">RequestListOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>The <a href="requestlist#new_RequestList_new"><code>new RequestList</code></a> options. Note that the listName parameter supersedes- the <code>persistStateKey</code> and <code>persistSourcesKey</code> options and the sources parameter supersedes the <code>sources</code> option.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.openRequestQueue"></a>+**Params**-## `Apify.openRequestQueue` ⇒ [`Promise<RequestQueue>`](requestqueue)+- **`listName`**: `string` | `null` - Name of the request list to be opened. Setting a name enables the `RequestList`'s state to be persisted in the+ key value store. This is useful in case of a restart or migration. Since `RequestList` is only stored in memory, a restart or migration wipes it+ clean. Setting a name will enable the `RequestList`'s state to survive those situations and continue where it left off.-Opens a request queue and returns a promise resolving to an instance of the [`RequestQueue`](requestqueue) class.+ The name will be used as a prefix in key value store, producing keys such as `NAME-REQUEST_LIST_STATE` and `NAME-REQUEST_LIST_SOURCES`.-[`RequestQueue`](requestqueue) represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud. The queue is used for-deep crawling of websites, where you start with several URLs and then recursively follow links to other pages. The data structure supports both-breadth-first and depth-first crawling orders.+ If `null`, the list will not be persisted and will only be stored in memory. Process restart will then cause the list to be crawled again from the+ beginning. We suggest always using a name.-For more details and code examples, see the [`RequestQueue`](requestqueue) class.+- **`sources`**: [`Array<(Request|RequestOptions|string)>`](/docs/api/request) - An array of sources of URLs for the+ [`RequestList`](/docs/api/request-list). It can be either an array of plain objects that define at least the `url` property, or an array of+ instances of the [`Request`](/docs/api/request) class.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[queueIdOrName]</code></td><td><code>string</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>ID or name of the request queue to be opened. If <code>null</code> or <code>undefined</code>,- the function returns the default request queue associated with the actor run.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.forceCloud]</code></td><td><code>boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to <code>true</code> then the function uses cloud storage usage even if the <code>APIFY_LOCAL_STORAGE_DIR</code>- environment variable is set. This way it is possible to combine local and cloud storage.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.openSessionPool"></a>+Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct [`RequestList`](/docs/api/request-list) to download the+source URLs from a given remote location. The URLs will be parsed from the received response. In this case you can limit the URLs using `regex`+parameter containing regular expression pattern for URLs to be included.++For details, see the [`RequestListOptions.sources`](/docs/typedefs/request-list-options#sources)-## `Apify.openSessionPool` ⇒ [`Promise<SessionPool>`](sessionpool)+- **`[options]`**: [`RequestListOptions`](/docs/typedefs/request-list-options) - The [`RequestList`](/docs/api/request-list) options. Note that the+ `listName` parameter supersedes the [`RequestListOptions.persistStateKey`](/docs/typedefs/request-list-options#persiststatekey) and+ [`RequestListOptions.persistSourcesKey`](/docs/typedefs/request-list-options#persistsourceskey) options and the `sources` parameter supersedes the+ [`RequestListOptions.sources`](/docs/typedefs/request-list-options#sources) option.-Opens a SessionPool and returns a promise resolving to an instance of the [`SessionPool`](sessionpool) class that is already initialized.+----For more details and code examples, see the [`SessionPool`](sessionpool) class.+<a name="openrequestqueue"></a>-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>sessionPoolOptions</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>The <a href="sessionpool#new_SessionPool_new"><code>new SessionPool</code></a> options</p>-</td></tr></tbody>-</table>-<a name="module_Apify.pushData"></a>+## `Apify.openRequestQueue([queueIdOrName], [options])`++**Returns**: [`Promise<RequestQueue>`](/docs/api/request-queue)++Opens a request queue and returns a promise resolving to an instance of the [`RequestQueue`](/docs/api/request-queue) class.++[`RequestQueue`](/docs/api/request-queue) represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud. The queue+is used for deep crawling of websites, where you start with several URLs and then recursively follow links to other pages. The data structure supports+both breadth-first and depth-first crawling orders.++For more details and code examples, see the [`RequestQueue`](/docs/api/request-queue) class.++**Params**++- **`[queueIdOrName]`**: `string` - ID or name of the request queue to be opened. If `null` or `undefined`, the function returns the default request+ queue associated with the actor run.+- **`[options]`**: `object`+ - **`[.forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the+ `APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.++---++<a name="opensessionpool"></a>++## `Apify.openSessionPool(sessionPoolOptions)`++**Returns**: [`Promise<SessionPool>`](/docs/api/session-pool)++Opens a SessionPool and returns a promise resolving to an instance of the [`SessionPool`](/docs/api/session-pool) class that is already initialized.++For more details and code examples, see the [`SessionPool`](/docs/api/session-pool) class.++**Params**-## `Apify.pushData(item)` ⇒ `Promise`+- **`sessionPoolOptions`**: [`SessionPoolOptions`](/docs/typedefs/session-pool-options)-Stores an object or an array of objects to the default [`Dataset`](dataset) of the current actor run.+---++<a name="pushdata"></a>++## `Apify.pushData(item)`++**Returns**: `Promise`++Stores an object or an array of objects to the default [`Dataset`](/docs/api/dataset) of the current actor run.-This is just a convenient shortcut for [`dataset.pushData()`](dataset#Dataset+pushData). For example, calling the following code:+This is just a convenient shortcut for [`Dataset.pushData()`](/docs/api/dataset#pushdata). For example, calling the following code:```javascriptawait Apify.pushData({ myValue: 123 });@@ -941,32 +683,26 @@ const dataset = await Apify.openDataset();await dataset.pushData({ myValue: 123 });```-For more information, see [`Apify.openDataset()`](apify#module_Apify.openDataset) and [`dataset.pushData()`](dataset#Dataset+pushData)+For more information, see [`Apify.openDataset()`](/docs/api/apify#opendataset) and [`Dataset.pushData()`](/docs/api/dataset#pushdata)**IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`, otherwise the actor process might finish before the data are stored!-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>item</code></td><td><code>Object</code> | <code>Array</code></td>-</tr>-<tr>-<td colspan="3"><p>Object or array of objects containing data to be stored in the default dataset.-The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.</p>-</td></tr></tbody>-</table>-<a name="module_Apify.setValue"></a>+**Params**-## `Apify.setValue(key, value, [options])` ⇒ `Promise`+- **`item`**: `Object` | `Array` - Object or array of objects containing data to be stored in the default dataset. The objects must be serializable+ to JSON and the JSON representation of each object must be smaller than 9MB.++----Stores or deletes a value in the default [`KeyValueStore`](keyvaluestore) associated with the current actor run.+<a name="setvalue"></a>-This is just a convenient shortcut for [`keyValueStore.setValue()`](keyvaluestore#KeyValueStore+setValue). For example, calling the following code:+## `Apify.setValue(key, value, [options])`++**Returns**: `Promise`++Stores or deletes a value in the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.++This is just a convenient shortcut for [`KeyValueStore.setValue()`](/docs/api/key-value-store#setvalue). For example, calling the following code:```javascriptawait Apify.setValue('OUTPUT', { foo: 'bar' });@@ -979,43 +715,23 @@ const store = await Apify.openKeyValueStore();await store.setValue('OUTPUT', { foo: 'bar' });```-To get a value from the default-key value store, you can use the [`Apify.getValue()`](#module_Apify.getValue) function.+To get a value from the default-key value store, you can use the [`Apify.getValue()`](/docs/api/apify#getvalue) function.-For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore) and-[`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyvaluestore) and+[`KeyValueStore.getValue()`](/docs/api/key-value-store#getvalue).-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>key</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Unique record key.</p>-</td></tr><tr>-<td><code>value</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td>-</tr>-<tr>-<td colspan="3"><p>Record data, which can be one of the following values:- <ul>- <li>If <code>null</code>, the record in the key-value store is deleted.</li>- <li>If no <code>options.contentType</code> is specified, <code>value</code> can be any JavaScript object and it will be stringified to JSON.</li>- <li>If <code>options.contentType</code> is specified, <code>value</code> is considered raw data and it must be a <code>String</code>- or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>.</li>- </ul>- For any other value an error will be thrown.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.contentType]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Specifies a custom MIME content type of the record.</p>-</td></tr></tbody>-</table>+**Params**++- **`key`**: `String` - Unique record key.+- **`value`**: `Object` | `String` | `Buffer` - Record data, which can be one of the following values:+ <ul>+ <li>If `null`, the record in the key-value store is deleted.</li>+ <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>+ <li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`+ or [](https://nodejs.org/api/buffer.html).</li>+ </ul>+ For any other value an error will be thrown.+- **`[options]`**: `Object`+ - **`[.contentType]`**: `String` - Specifies a custom MIME content type of the record.++---diff --git a/docs/api/AutoscaledPool.md b/docs/api/AutoscaledPool.mdindex 0fda68935..cd8b64236 100644--- a/docs/api/AutoscaledPool.md+++ b/docs/api/AutoscaledPool.md@@ -1,25 +1,28 @@----id: autoscaledpool+id: autoscaled-pooltitle: AutoscaledPool----<a name="AutoscaledPool"></a>+<a name="autoscaledpool"></a>Manages a pool of asynchronous resource-intensive tasks that are executed in parallel. The pool only starts new tasks if there is enough free CPU andmemory available and the Javascript event loop is not blocked.-The information about the CPU and memory usage is obtained by the [`Snapshotter`](snapshotter) class, which makes regular snapshots of system-resources that may be either local or from the Apify cloud infrastructure in case the process is running on the Apify platform. Meaningful data-gathered from these snapshots is provided to `AutoscaledPool` by the [`SystemStatus`](systemstatus) class.+The information about the CPU and memory usage is obtained by the [`Snapshotter`](/docs/api/snapshotter) class, which makes regular snapshots of+system resources that may be either local or from the Apify cloud infrastructure in case the process is running on the Apify platform. Meaningful data+gathered from these snapshots is provided to `AutoscaledPool` by the [`SystemStatus`](/docs/api/system-status) class.-Before running the pool, you need to implement the following three functions: [`runTaskFunction()`](#new_AutoscaledPool_new),-[`isTaskReadyFunction()`](#new_AutoscaledPool_new) and [`isFinishedFunction()`](#new_AutoscaledPool_new).+Before running the pool, you need to implement the following three functions:+[`AutoscaledPoolOptions.runTaskFunction()`](/docs/typedefs/autoscaled-pool-options#runtaskfunction),+[`AutoscaledPoolOptions.isTaskReadyFunction()`](/docs/typedefs/autoscaled-pool-options#istaskreadyfunction) and+[`AutoscaledPoolOptions.isFinishedFunction()`](/docs/typedefs/autoscaled-pool-options#isfinishedfunction).-The auto-scaled pool is started by calling the [`run()`](#AutoscaledPool+run) function. The pool periodically queries the-[`isTaskReadyFunction()`](#new_AutoscaledPool_new) function for more tasks, managing optimal concurrency, until the function resolves to `false`. The-pool then queries the [`isFinishedFunction()`](#new_AutoscaledPool_new). If it resolves to `true`, the run finishes after all running tasks complete.-If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for tasks. If any of the tasks throws-then the [`run()`](#AutoscaledPool+run) function rejects the promise with an error.+The auto-scaled pool is started by calling the [`AutoscaledPool.run()`](/docs/api/autoscaled-pool#run) function. The pool periodically queries the+[`AutoscaledPoolOptions.isTaskReadyFunction()`](/docs/typedefs/autoscaled-pool-options#istaskreadyfunction) function for more tasks, managing optimal+concurrency, until the function resolves to `false`. The pool then queries the+[`AutoscaledPoolOptions.isFinishedFunction()`](/docs/typedefs/autoscaled-pool-options#isfinishedfunction). If it resolves to `true`, the run finishes+after all running tasks complete. If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for+tasks. If any of the tasks throws then the [`AutoscaledPool.run()`](/docs/api/autoscaled-pool#run) function rejects the promise with an error.The pool evaluates whether it should start a new task every time one of the tasks finishes and also in the interval set by the`options.maybeRunIntervalSecs` parameter.@@ -46,45 +49,29 @@ const pool = new Apify.AutoscaledPool({await pool.run();```-- [AutoscaledPool](autoscaledpool)- - [`new AutoscaledPool(options)`](#new_AutoscaledPool_new)- - [`.minConcurrency`](#AutoscaledPool+minConcurrency) ⇒ `number`- - [`.minConcurrency`](#AutoscaledPool+minConcurrency)- - [`.maxConcurrency`](#AutoscaledPool+maxConcurrency) ⇒ `number`- - [`.maxConcurrency`](#AutoscaledPool+maxConcurrency)- - [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency) ⇒ `number`- - [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency)- - [`.currentConcurrency`](#AutoscaledPool+currentConcurrency) ⇒ `number`- - [`.run()`](#AutoscaledPool+run) ⇒ `Promise`- - [`.abort()`](#AutoscaledPool+abort) ⇒ `Promise`- - [`.pause([timeoutSecs])`](#AutoscaledPool+pause) ⇒ `Promise`- - [`.resume()`](#AutoscaledPool+resume)--<a name="new_AutoscaledPool_new"></a>+---++<a name="autoscaledpool"></a>## `new AutoscaledPool(options)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>AutoscaledPool</code> configuration options.</p>-</td></tr></tbody>-</table>-<a name="AutoscaledPool+minConcurrency"></a>--## `autoscaledPool.minConcurrency` ⇒ `number`+**Params**++- **`options`**: [`AutoscaledPoolOptions`](/docs/typedefs/autoscaled-pool-options) - All `AutoscaledPool` configuration options.++---++<a name="minconcurrency"></a>++## `autoscaledPool.minConcurrency`++**Returns**: `number`Gets the minimum number of tasks running in parallel.-<a name="AutoscaledPool+minConcurrency"></a>+---++<a name="minconcurrency"></a>## `autoscaledPool.minConcurrency`@@ -93,121 +80,119 @@ Sets the minimum number of tasks running in parallel._WARNING:_ If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash. If you'renot sure, just keep the default value and the concurrency will scale up automatically.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>value</code></td><td><code>number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="AutoscaledPool+maxConcurrency"></a>--## `autoscaledPool.maxConcurrency` ⇒ `number`+**Params**++- **`value`**: `number`++---++<a name="maxconcurrency"></a>++## `autoscaledPool.maxConcurrency`++**Returns**: `number`Gets the maximum number of tasks running in parallel.-<a name="AutoscaledPool+maxConcurrency"></a>+---++<a name="maxconcurrency"></a>## `autoscaledPool.maxConcurrency`Sets the maximum number of tasks running in parallel.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>value</code></td><td><code>number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="AutoscaledPool+desiredConcurrency"></a>--## `autoscaledPool.desiredConcurrency` ⇒ `number`+**Params**++- **`value`**: `number`++---++<a name="desiredconcurrency"></a>++## `autoscaledPool.desiredConcurrency`++**Returns**: `number`Gets the desired concurrency for the pool, which is an estimated number of parallel tasks that the system can currently support.-<a name="AutoscaledPool+desiredConcurrency"></a>+---++<a name="desiredconcurrency"></a>## `autoscaledPool.desiredConcurrency`Sets the desired concurrency for the pool, i.e. the number of tasks that should be running in parallel if there's large enough supply of tasks.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>value</code></td><td><code>number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="AutoscaledPool+currentConcurrency"></a>--## `autoscaledPool.currentConcurrency` ⇒ `number`+**Params**++- **`value`**: `number`++---++<a name="currentconcurrency"></a>++## `autoscaledPool.currentConcurrency`++**Returns**: `number`Gets the the number of parallel tasks currently running in the pool.-<a name="AutoscaledPool+run"></a>+---++<a name="run"></a>++## `autoscaledPool.run()`-## `autoscaledPool.run()` ⇒ `Promise`+**Returns**: `Promise`Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once all the tasks are finished or one of them fails.-<a name="AutoscaledPool+abort"></a>+----## `autoscaledPool.abort()` ⇒ `Promise`+<a name="abort"></a>-Aborts the run of the auto-scaled pool and destroys it. The promise returned from the [`run()`](#AutoscaledPool+run) function will immediately-resolve, no more new tasks will be spawned and all running tasks will be left in their current state.+## `autoscaledPool.abort()`++**Returns**: `Promise`++Aborts the run of the auto-scaled pool and destroys it. The promise returned from the [`AutoscaledPool.run()`](/docs/api/autoscaled-pool#run) function+will immediately resolve, no more new tasks will be spawned and all running tasks will be left in their current state.Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion of all the running tasks, therefore, no abortion is attempted andsome of the tasks may finish, while others may not. Essentially, auto-scaled pool doesn't care about their state after the invocation of `.abort()`,but that does not mean that some parts of their asynchronous chains of commands will not execute.-<a name="AutoscaledPool+pause"></a>+---++<a name="pause"></a>++## `autoscaledPool.pause([timeoutSecs])`-## `autoscaledPool.pause([timeoutSecs])` ⇒ `Promise`+**Returns**: `Promise`Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish (unlike abort, which terminates them). Used together with-[`resume()`](#AutoscaledPool+resume)+[`AutoscaledPool.resume()`](/docs/api/autoscaled-pool#resume)The function's promise will resolve once all running tasks have completed and the pool is effectively idle. If the `timeoutSecs` argument is provided,the promise will reject with a timeout error after the `timeoutSecs` seconds.-The promise returned from the [`run()`](#AutoscaledPool+run) function will not resolve when `.pause()` is invoked (unlike abort, which resolves it).--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[timeoutSecs]</code></td><td><code>number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="AutoscaledPool+resume"></a>+The promise returned from the [`AutoscaledPool.run()`](/docs/api/autoscaled-pool#run) function will not resolve when `.pause()` is invoked (unlike+abort, which resolves it).++**Params**++- **`[timeoutSecs]`**: `number`++---++<a name="resume"></a>## `autoscaledPool.resume()`-Resumes the operation of the autoscaled-pool by allowing more tasks to be run. Used together with [`pause()`](#AutoscaledPool+pause)+Resumes the operation of the autoscaled-pool by allowing more tasks to be run. Used together with+[`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause)Tasks will automatically start running again in `options.maybeRunIntervalSecs`.++---diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex d7b339ce0..9a55cd956 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -1,31 +1,34 @@----id: basiccrawler+id: basic-crawlertitle: BasicCrawler----<a name="BasicCrawler"></a>+<a name="basiccrawler"></a>Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue ofURLs enabling recursive crawling of websites.`BasicCrawler` is a low-level tool that requires the user to implement the page download and data extraction functionality themselves. If you want a-crawler that already facilitates this functionality, please consider using [`PuppeteerCrawler`](puppeteercrawler) or-[`CheerioCrawler`](cheeriocrawler).+crawler that already facilitates this functionality, please consider using [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) or+[`CheerioCrawler`](/docs/api/cheerio-crawler).-`BasicCrawler` invokes the user-provided [`handleRequestFunction()`](#new_BasicCrawler_new) for each [`Request`](request) object, which represents a-single URL to crawl. The [`Request`](request) objects are fed from the [`RequestList`](requestlist) or the [`RequestQueue`](requestqueue) instances-provided by the [`requestList`](#new_BasicCrawler_new) or [`requestQueue`](#new_BasicCrawler_new) constructor options, respectively.+`BasicCrawler` invokes the user-provided [`BasicCrawlerOptions.handleRequestFunction`](/docs/typedefs/basic-crawler-options#handlerequestfunction) for+each [`Request`](/docs/api/request) object, which represents a single URL to crawl. The [`Request`](/docs/api/request) objects are fed from the+[`RequestList`](/docs/api/request-list) or the [`RequestQueue`](/docs/api/request-queue) instances provided by the+[`BasicCrawlerOptions.requestList`](/docs/typedefs/basic-crawler-options#requestlist) or+[`BasicCrawlerOptions.requestQueue`](/docs/typedefs/basic-crawler-options#requestqueue) constructor options, respectively.-If both [`requestList`](#new_BasicCrawler_new) and [`requestQueue`](#new_BasicCrawler_new) options are used, the instance first processes URLs from-the [`RequestList`](requestlist) and automatically enqueues all of them to [`RequestQueue`](requestqueue) before it starts their processing. This-ensures that a single URL is not crawled multiple times.+If both [`BasicCrawlerOptions.requestList`](/docs/typedefs/basic-crawler-options#requestlist) and+[`BasicCrawlerOptions.requestQueue`](/docs/typedefs/basic-crawler-options#requestqueue) options are used, the instance first processes URLs from the+[`RequestList`](/docs/api/request-list) and automatically enqueues all of them to [`RequestQueue`](/docs/api/request-queue) before it starts their+processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes if there are no more [`Request`](request) objects to crawl.+The crawler finishes if there are no more [`Request`](/docs/api/request) objects to crawl.New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the-[`AutoscaledPool`](autoscaledpool) class. All [`AutoscaledPool`](autoscaledpool) configuration options can be passed to the `autoscaledPoolOptions`-parameter of the `BasicCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [`AutoscaledPool`](autoscaledpool)-options are available directly in the `BasicCrawler` constructor.+[`AutoscaledPool`](/docs/api/autoscaled-pool) class. All [`AutoscaledPool`](/docs/api/autoscaled-pool) configuration options can be passed to the+`autoscaledPoolOptions` parameter of the `BasicCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`+[`AutoscaledPool`](/docs/api/autoscaled-pool) options are available directly in the `BasicCrawler` constructor.**Example usage:**@@ -54,50 +57,24 @@ const crawler = new Apify.BasicCrawler({await crawler.run();```-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.- Note that this property is only initialized after calling the <a href="#BasicCrawler+run"><code>run</code></a> function.- You can use it to change the concurrency settings on the fly,- to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>- or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>-</td></tr></tbody>-</table>--- [BasicCrawler](basiccrawler)- - [`new BasicCrawler(options)`](#new_BasicCrawler_new)- - [`.run()`](#BasicCrawler+run) ⇒ `Promise<void>`--<a name="new_BasicCrawler_new"></a>+---++<a name="basiccrawler"></a>## `new BasicCrawler(options)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/basiccrawleroptions">BasicCrawlerOptions</a></code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="BasicCrawler+run"></a>--## `basicCrawler.run()` ⇒ `Promise<void>`+**Params**++- **`options`**: [`BasicCrawlerOptions`](/docs/typedefs/basic-crawler-options) - All `BasicCrawler` parameters are passed via an options object.++---++<a name="run"></a>++## `basicCrawler.run()`++**Returns**: `Promise<void>`Runs the crawler. Returns a promise that gets resolved once all the requests are processed.++---diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex b842a3a2e..aeac05560 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -1,44 +1,46 @@----id: cheeriocrawler+id: cheerio-crawlertitle: CheerioCrawler----<a name="CheerioCrawler"></a>+<a name="cheeriocrawler"></a>-Provides a framework for the parallel crawling of web pages using plain HTTP requests and-<a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser. The URLs to crawl are fed either from a static list of URLs-or from a dynamic queue of URLs enabling recursive crawling of websites.+Provides a framework for the parallel crawling of web pages using plain HTTP requests and [cheerio](https://www.npmjs.com/package/cheerio) HTML+parser. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.Since `CheerioCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website-requires JavaScript to display the content, you might need to use [`PuppeteerCrawler`](puppeteercrawler) instead, because it loads the pages using-full-featured headless Chrome browser.+requires JavaScript to display the content, you might need to use [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) instead, because it loads the+pages using full-featured headless Chrome browser.-`CheerioCrawler` downloads each URL using a plain HTTP request, parses the HTML content using-<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> and then invokes the user-provided-[`handlePageFunction()`](#new_CheerioCrawler_new) to extract page data using a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface-to the parsed HTML DOM.+`CheerioCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio) and+then invokes the user-provided [`CheerioCrawlerOptions.handlePageFunction`](/docs/typedefs/cheerio-crawler-options#handlepagefunction) to extract page+data using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.-The source URLs are represented using [`Request`](request) objects that are fed from [`RequestList`](requestlist) or [`RequestQueue`](requestqueue)-instances provided by the [`requestList`](#new_CheerioCrawler_new) or [`requestQueue`](#new_CheerioCrawler_new) constructor options, respectively.+The source URLs are represented using [`Request`](/docs/api/request) objects that are fed from [`RequestList`](/docs/api/request-list) or+[`RequestQueue`](/docs/api/request-queue) instances provided by the+[`CheerioCrawlerOptions.requestList`](/docs/typedefs/cheerio-crawler-options#requestlist) or+[`CheerioCrawlerOptions.requestQueue`](/docs/typedefs/cheerio-crawler-options#requestqueue) constructor options, respectively.-If both [`requestList`](#new_CheerioCrawler_new) and [`requestQueue`](#new_CheerioCrawler_new) are used, the instance first processes URLs from the-[`RequestList`](requestlist) and automatically enqueues all of them to [`RequestQueue`](requestqueue) before it starts their processing. This ensures-that a single URL is not crawled multiple times.+If both [`CheerioCrawlerOptions.requestList`](/docs/typedefs/cheerio-crawler-options#requestlist) and+[`CheerioCrawlerOptions.requestQueue`](/docs/typedefs/cheerio-crawler-options#requestqueue) are used, the instance first processes URLs from the+[`RequestList`](/docs/api/request-list) and automatically enqueues all of them to [`RequestQueue`](/docs/api/request-queue) before it starts their+processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [`Request`](request) objects to crawl.+The crawler finishes when there are no more [`Request`](/docs/api/request) objects to crawl.-`CheerioCrawler` downloads the web pages using the [`requestAsBrowser`](requestasbrowser) utility function. You can use the `requestOptions` parameter-to pass additional options to this function.+`CheerioCrawler` downloads the web pages using the [`utils.requestAsBrowser()`](/docs/api/utils#requestasbrowser) utility function. You can use the+`requestOptions` parameter to pass additional options to this function.By default, `CheerioCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the`Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the-[`additionalMimeTypes`](#new_CheerioCrawler_new) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of-content. For details, see [`CheerioCrawlerOptions#handlePageFunction`](cheeriocrawleroptions#handlepagefunction).+[`CheerioCrawlerOptions.additionalMimeTypes`](/docs/typedefs/cheerio-crawler-options#additionalmimetypes) constructor option. Beware that the parsing+behavior differs for HTML, XML, JSON and other types of content. For details, see+[`CheerioCrawlerOptions.handlePageFunction`](/docs/typedefs/cheerio-crawler-options#handlepagefunction).New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the-[`AutoscaledPool`](autoscaledpool) class. All [`AutoscaledPool`](autoscaledpool) configuration options can be passed to the `autoscaledPoolOptions`-parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [`AutoscaledPool`](autoscaledpool)-options are available directly in the `CheerioCrawler` constructor.+[`AutoscaledPool`](/docs/api/autoscaled-pool) class. All [`AutoscaledPool`](/docs/api/autoscaled-pool) configuration options can be passed to the+`autoscaledPoolOptions` parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`+[`AutoscaledPool`](/docs/api/autoscaled-pool) options are available directly in the `CheerioCrawler` constructor.**Example usage:**@@ -76,50 +78,25 @@ const crawler = new Apify.CheerioCrawler({await crawler.run();```-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.- Note that this property is only initialized after calling the <a href="#CheerioCrawler+run"><code>run</code></a> function.- You can use it to change the concurrency settings on the fly,- to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>- or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>-</td></tr></tbody>-</table>--- [CheerioCrawler](cheeriocrawler)- - [`new CheerioCrawler(options)`](#new_CheerioCrawler_new)- - [`.run()`](#CheerioCrawler+run) ⇒ `Promise`--<a name="new_CheerioCrawler_new"></a>+---++<a name="cheeriocrawler"></a>## `new CheerioCrawler(options)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/cheeriocrawleroptions">CheerioCrawlerOptions</a></code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="CheerioCrawler+run"></a>--## `cheerioCrawler.run()` ⇒ `Promise`+**Params**++- **`options`**: [`CheerioCrawlerOptions`](/docs/typedefs/cheerio-crawler-options) - All `CheerioCrawler` parameters are passed via an options+ object.++---++<a name="run"></a>++## `cheerioCrawler.run()`++**Returns**: `Promise`Runs the crawler. Returns promise that gets resolved once all the requests got processed.++---diff --git a/docs/api/Dataset.md b/docs/api/Dataset.mdindex 6a1195d90..8cceb2097 100644--- a/docs/api/Dataset.md+++ b/docs/api/Dataset.md@@ -3,13 +3,13 @@ id: datasettitle: Dataset----<a name="Dataset"></a>+<a name="dataset"></a>The `Dataset` class represents a store for structured data where each object stored has the same attributes, such as online store products or realestate offers. You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you canonly add new records to it but you cannot modify or remove existing records. Typically it is used to store crawling results.-Do not instantiate this class directly, use the [`Apify.openDataset()`](apify#module_Apify.openDataset) function instead.+Do not instantiate this class directly, use the [`Apify.openDataset()`](/docs/api/apify#opendataset) function instead.`Dataset` stores its data either on local disk or in the Apify cloud, depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environmentvariables are set.@@ -25,9 +25,8 @@ Note that `{DATASET_ID}` is the name or ID of the dataset. The default dataset hitem in the dataset.If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the-<a href="https://docs.apify.com/storage/dataset" target="_blank">Apify Dataset</a> cloud storage. Note that you can force usage of the cloud storage-also by passing the `forceCloud` option to [`Apify.openDataset()`](apify#module_Apify.openDataset) function, even if the `APIFY_LOCAL_STORAGE_DIR`-variable is set.+[Apify Dataset](https://docs.apify.com/storage/dataset) cloud storage. Note that you can force usage of the cloud storage also by passing the+`forceCloud` option to [`Apify.openDataset()`](/docs/api/apify#opendataset) function, even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.**Example usage:**@@ -45,40 +44,24 @@ await dataset.pushData({ foo: 'bar' });await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);```-- [Dataset](dataset)- - [`new exports.Dataset(datasetId, datasetName)`](#new_Dataset_new)- - [`.pushData(data)`](#Dataset+pushData) ⇒ `Promise<void>`- - [`.getData([options])`](#Dataset+getData) ⇒ [`Promise<DatasetContent>`](../typedefs/datasetcontent)- - [`.getInfo()`](#Dataset+getInfo) ⇒ `Promise<Object>`- - [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ `Promise<void>`- - [`.map(iteratee, options)`](#Dataset+map) ⇒ `Promise<Array<T>>`- - [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ `Promise<T>`- - [`.drop()`](#Dataset+drop) ⇒ `Promise<void>`--<a name="new_Dataset_new"></a>--## `new exports.Dataset(datasetId, datasetName)`--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>datasetId</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>datasetName</code></td><td><code>string</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Dataset+pushData"></a>--## `dataset.pushData(data)` ⇒ `Promise<void>`+---++<a name="exports.dataset"></a>++## `new Dataset(datasetId, datasetName)`++**Params**++- **`datasetId`**: `string`+- **`datasetName`**: `string`++---++<a name="pushdata"></a>++## `dataset.pushData(data)`++**Returns**: `Promise<void>`Stores an object or an array of objects to the dataset. The function returns a promise that resolves when the operation finishes. It has no result,but throws on invalid args or other errors.@@ -94,150 +77,68 @@ Promise will reject and the dataset will be left in a state where some of the itsource array were not. To overcome this limitation, the developer may, for example, read the last item saved in the dataset and re-attempt the save ofthe data from this item onwards to prevent duplicates.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>data</code></td><td><code>Object</code> | <code>Array</code></td>-</tr>-<tr>-<td colspan="3"><p>Object or array of objects containing data to be stored in the default dataset.-The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.</p>-</td></tr></tbody>-</table>-<a name="Dataset+getData"></a>--## `dataset.getData([options])` ⇒ [`Promise<DatasetContent>`](../typedefs/datasetcontent)+**Params**++- **`data`**: `Object` | `Array` - Object or array of objects containing data to be stored in the default dataset. The objects must be serializable+ to JSON and the JSON representation of each object must be smaller than 9MB.++---++<a name="getdata"></a>++## `dataset.getData([options])`++**Returns**: [`Promise<DatasetContent>`](/docs/typedefs/dataset-content)Returns {DatasetContent} object holding the items in the dataset based on the provided parameters.**NOTE**: If using dataset with local disk storage, the `format` option must be `json` and the following options are not supported: `unwind`,`disableBodyParser`, `attachment`, `bom` and `simplified`. If you try to use them, you will receive an error.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>getData()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>[options.format]</code></td><td><code>String</code></td><td><code>'json'</code></td>-</tr>-<tr>-<td colspan="3"><p>Format of the <code>items</code> property, possible values are: <code>json</code>, <code>csv</code>, <code>xlsx</code>, <code>html</code>, <code>xml</code> and <code>rss</code>.</p>-</td></tr><tr>-<td><code>[options.offset]</code></td><td><code>Number</code></td><td><code>0</code></td>-</tr>-<tr>-<td colspan="3"><p>Number of array elements that should be skipped at the start.</p>-</td></tr><tr>-<td><code>[options.limit]</code></td><td><code>Number</code></td><td><code>250000</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of array elements to return.</p>-</td></tr><tr>-<td><code>[options.desc]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the objects are sorted by <code>createdAt</code> in descending order.- Otherwise they are sorted in ascending order.</p>-</td></tr><tr>-<td><code>[options.fields]</code></td><td><code>Array</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of field names that will be included in the result. If omitted, all fields are included in the results.</p>-</td></tr><tr>-<td><code>[options.unwind]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Specifies a name of the field in the result objects that will be used to unwind the resulting objects.- By default, the results are returned as they are.</p>-</td></tr><tr>-<td><code>[options.disableBodyParser]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then response from API will not be parsed.</p>-</td></tr><tr>-<td><code>[options.attachment]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the response will define the <code>Content-Disposition: attachment</code> HTTP header, forcing a web- browser to download the file rather than to display it. By default, this header is not present.</p>-</td></tr><tr>-<td><code>[options.delimiter]</code></td><td><code>String</code></td><td><code>','</code></td>-</tr>-<tr>-<td colspan="3"><p>A delimiter character for CSV files, only used if <code>format</code> is <code>csv</code>.</p>-</td></tr><tr>-<td><code>[options.bom]</code></td><td><code>Boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte- Order Mark (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default- behavior, set <code>bom</code> option to <code>true</code> to include the BOM, or set <code>bom</code> to <code>false</code> to skip it.</p>-</td></tr><tr>-<td><code>[options.xmlRoot]</code></td><td><code>String</code></td><td><code>'results'</code></td>-</tr>-<tr>-<td colspan="3"><p>Overrides the default root element name of the XML output. By default, the root element is <code>results</code>.</p>-</td></tr><tr>-<td><code>[options.xmlRow]</code></td><td><code>String</code></td><td><code>'page'</code></td>-</tr>-<tr>-<td colspan="3"><p>Overrides the default element name that wraps each page or page function result object in XML output.- By default, the element name is <code>page</code> or <code>result</code>, depending on the value of the <code>simplified</code> option.</p>-</td></tr><tr>-<td><code>[options.skipHeaderRow]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to <code>true</code> then header row in CSV format is skipped.</p>-</td></tr><tr>-<td><code>[options.clean]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the function returns only non-empty items and skips hidden fields (i.e. fields starting with <code>#</code> character).- Note that the <code>clean</code> parameter is a shortcut for <code>skipHidden: true</code> and <code>skipEmpty: true</code> options.</p>-</td></tr><tr>-<td><code>[options.skipHidden]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the function doesn't return hidden fields (fields starting with "#" character).</p>-</td></tr><tr>-<td><code>[options.skipEmpty]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the function doesn't return empty items.- Note that in this case the returned number of items might be lower than limit parameter and pagination must be done using the <code>limit</code> value.</p>-</td></tr><tr>-<td><code>[options.simplified]</code></td><td><code>Boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then function applies the <code>fields: ['url','pageFunctionResult','errorInfo']</code> and <code>unwind: 'pageFunctionResult'</code> options.- This feature is used to emulate simplified results provided by Apify API version 1 used for- the legacy Apify Crawler and it's not recommended to use it in new integrations.</p>-</td></tr><tr>-<td><code>[options.skipFailedPages]</code></td><td><code>Boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then, the all the items with errorInfo property will be skipped from the output.- This feature is here to emulate functionality of Apify API version 1 used for- the legacy Apify Crawler product and it's not recommended to use it in new integrations.</p>-</td></tr></tbody>-</table>-<a name="Dataset+getInfo"></a>--## `dataset.getInfo()` ⇒ `Promise<Object>`+**Params**++- **`[options]`**: `Object` - All `getData()` parameters are passed via an options object with the following keys:+ - **`[.format]`**: `String` <code> = 'json'</code> - Format of the `items` property, possible values are: `json`, `csv`, `xlsx`,+ `html`, `xml` and `rss`.+ - **`[.offset]`**: `Number` <code> = 0</code> - Number of array elements that should be skipped at the start.+ - **`[.limit]`**: `Number` <code> = 250000</code> - Maximum number of array elements to return.+ - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by `createdAt` in descending order. Otherwise they are+ sorted in ascending order.+ - **`[.fields]`**: `Array` - An array of field names that will be included in the result. If omitted, all fields are included in the results.+ - **`[.unwind]`**: `String` - Specifies a name of the field in the result objects that will be used to unwind the resulting objects. By default,+ the results are returned as they are.+ - **`[.disableBodyParser]`**: `Boolean` <code> = false</code> - If `true` then response from API will not be parsed.+ - **`[.attachment]`**: `Boolean` <code> = false</code> - If `true` then the response will define the `Content-Disposition: attachment` HTTP+ header, forcing a web browser to download the file rather than to display it. By default, this header is not present.+ - **`[.delimiter]`**: `String` <code> = ','</code> - A delimiter character for CSV files, only used if `format` is `csv`.+ - **`[.bom]`**: `Boolean` - All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte Order Mark+ (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default behavior, set `bom` option to `true` to+ include the BOM, or set `bom` to `false` to skip it.+ - **`[.xmlRoot]`**: `String` <code> = 'results'</code> - Overrides the default root element name of the XML output. By default, the+ root element is `results`.+ - **`[.xmlRow]`**: `String` <code> = 'page'</code> - Overrides the default element name that wraps each page or page function result+ object in XML output. By default, the element name is `page` or `result`, depending on the value of the `simplified` option.+ - **`[.skipHeaderRow]`**: `Boolean` <code> = false</code> - If set to `true` then header row in CSV format is skipped.+ - **`[.clean]`**: `Boolean` <code> = false</code> - If `true` then the function returns only non-empty items and skips hidden fields (i.e.+ fields starting with `#` character). Note that the `clean` parameter is a shortcut for `skipHidden: true` and `skipEmpty: true` options.+ - **`[.skipHidden]`**: `Boolean` <code> = false</code> - If `true` then the function doesn't return hidden fields (fields starting with "#"+ character).+ - **`[.skipEmpty]`**: `Boolean` <code> = false</code> - If `true` then the function doesn't return empty items. Note that in this case the+ returned number of items might be lower than limit parameter and pagination must be done using the `limit` value.+ - **`[.simplified]`**: `Boolean` - If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and+ `unwind: 'pageFunctionResult'` options. This feature is used to emulate simplified results provided by Apify API version 1 used for the legacy+ Apify Crawler and it's not recommended to use it in new integrations.+ - **`[.skipFailedPages]`**: `Boolean` - If `true` then, the all the items with errorInfo property will be skipped from the output. This feature+ is here to emulate functionality of Apify API version 1 used for the legacy Apify Crawler product and it's not recommended to use it in new+ integrations.++---++<a name="getinfo"></a>++## `dataset.getInfo()`++**Returns**: `Promise<Object>`Returns an object containing general information about the dataset.@@ -260,9 +161,13 @@ The function returns the same object as the Apify API Client's}```-<a name="Dataset+forEach"></a>+---++<a name="foreach"></a>++## `dataset.forEach(iteratee, [options], [index])`-## `dataset.forEach(iteratee, [options], [index])` ⇒ `Promise<void>`+**Returns**: `Promise<void>`Iterates over dataset items, yielding each in turn to an `iteratee` function. Each invocation of `iteratee` is called with two arguments:`(item, index)`.@@ -279,92 +184,43 @@ await dataset.forEach(async (item, index) => {});```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>iteratee</code></td><td><code><a href="../typedefs/datasetconsumer">DatasetConsumer</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function that is called for every item in the dataset.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>forEach()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>[options.desc]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the objects are sorted by <code>createdAt</code> in descending order.</p>-</td></tr><tr>-<td><code>[options.fields]</code></td><td><code>Array</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If provided then returned objects will only contain specified keys.</p>-</td></tr><tr>-<td><code>[options.unwind]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If provided then objects will be unwound based on provided field.</p>-</td></tr><tr>-<td><code>[index]</code></td><td><code>Number</code></td><td><code>0</code></td>-</tr>-<tr>-<td colspan="3"><p>Specifies the initial index number passed to the <code>iteratee</code> function.</p>-</td></tr></tbody>-</table>-<a name="Dataset+map"></a>--## `dataset.map(iteratee, options)` ⇒ `Promise<Array<T>>`+**Params**++- **`iteratee`**: [`DatasetConsumer`](/docs/typedefs/dataset-consumer) - A function that is called for every item in the dataset.+- **`[options]`**: `Object` - All `forEach()` parameters are passed via an options object with the following keys:+ - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by `createdAt` in descending order.+ - **`[.fields]`**: `Array` - If provided then returned objects will only contain specified keys.+ - **`[.unwind]`**: `String` - If provided then objects will be unwound based on provided field.+- **`[index]`**: `Number` <code> = 0</code> - Specifies the initial index number passed to the `iteratee` function.++---++<a name="map"></a>++## `dataset.map(iteratee, options)`++**Returns**: `Promise<Array<T>>`Produces a new array of values by mapping each value in list through a transformation function `iteratee()`. Each invocation of `iteratee()` is calledwith two arguments: `(element, index)`.If `iteratee` returns a `Promise` then it's awaited before a next call.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>iteratee</code></td><td><code><a href="../typedefs/datasetmapper">DatasetMapper</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>map()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>[options.desc]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the objects are sorted by createdAt in descending order.</p>-</td></tr><tr>-<td><code>[options.fields]</code></td><td><code>Array</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If provided then returned objects will only contain specified keys</p>-</td></tr><tr>-<td><code>[options.unwind]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If provided then objects will be unwound based on provided field.</p>-</td></tr></tbody>-</table>-<a name="Dataset+reduce"></a>--## `dataset.reduce(iteratee, memo, options)` ⇒ `Promise<T>`+**Params**++- **`iteratee`**: [`DatasetMapper`](/docs/typedefs/dataset-mapper)+- **`options`**: `Object` - All `map()` parameters are passed via an options object with the following keys:+ - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by createdAt in descending order.+ - **`[.fields]`**: `Array` - If provided then returned objects will only contain specified keys+ - **`[.unwind]`**: `String` - If provided then objects will be unwound based on provided field.++---++<a name="reduce"></a>++## `dataset.reduce(iteratee, memo, options)`++**Returns**: `Promise<T>`Reduces a list of values down to a single value.@@ -376,47 +232,23 @@ instead passed as the memo in the invocation of the `iteratee()` on the next eleIf `iteratee()` returns a `Promise` then it's awaited before a next call.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>iteratee</code></td><td><code><a href="../typedefs/datasetreducer">DatasetReducer</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>memo</code></td><td><code>T</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Initial state of the reduction.</p>-</td></tr><tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>reduce()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>[options.desc]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the objects are sorted by createdAt in descending order.</p>-</td></tr><tr>-<td><code>[options.fields]</code></td><td><code>Array</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If provided then returned objects will only contain specified keys</p>-</td></tr><tr>-<td><code>[options.unwind]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If provided then objects will be unwound based on provided field.</p>-</td></tr></tbody>-</table>-<a name="Dataset+drop"></a>--## `dataset.drop()` ⇒ `Promise<void>`+**Params**++- **`iteratee`**: [`DatasetReducer`](/docs/typedefs/dataset-reducer)+- **`memo`**: `T` - Initial state of the reduction.+- **`options`**: `Object` - All `reduce()` parameters are passed via an options object with the following keys:+ - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by createdAt in descending order.+ - **`[.fields]`**: `Array` - If provided then returned objects will only contain specified keys+ - **`[.unwind]`**: `String` - If provided then objects will be unwound based on provided field.++---++<a name="drop"></a>++## `dataset.drop()`++**Returns**: `Promise<void>`Removes the dataset either from the Apify cloud storage or from the local directory, depending on the mode of operation.++---diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex a528f447d..f0702dec7 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -1,25 +1,26 @@----id: keyvaluestore+id: key-value-storetitle: KeyValueStore----<a name="KeyValueStore"></a>+<a name="keyvaluestore"></a>The `KeyValueStore` class represents a key-value store, a simple data storage that is used for saving and reading data records or files. Each datarecord is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots, actor inputs andoutputs, web pages, PDFs or to persist the state of crawlers.-Do not instantiate this class directly, use the [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function instead.+Do not instantiate this class directly, use the [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyvaluestore) function instead.Each actor run is associated with a default key-value store, which is created exclusively for the run. By convention, the actor input and output arestored into the default key-value store under the `INPUT` and `OUTPUT` key, respectively. Typically, input and output are JSON files, although it can-be any other format. To access the default key-value store directly, you can use the [`Apify.getValue()`](apify#module_Apify.getValue) and-[`Apify.setValue()`](apify#module_Apify.setValue) convenience functions.+be any other format. To access the default key-value store directly, you can use the [`Apify.getValue()`](/docs/api/apify#getvalue) and+[`Apify.setValue()`](/docs/api/apify#setvalue) convenience functions.-To access the input, you can also use the [`Apify.getInput()`](apify#module_Apify.getInput) convenience function.+To access the input, you can also use the [`Apify.getInput()`](/docs/api/apify#getinput) convenience function.-`KeyValueStore` stores its data either on local disk or in the Apify cloud, depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN`-environment variables are set.+`KeyValueStore` stores its data either on local disk or in the Apify cloud, depending on whether the+[`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) or+[`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variables are set.If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in the local directory in the following files:@@ -31,10 +32,11 @@ Note that `{STORE_ID}` is the name or ID of the key-value store. The default key`APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of thedata value.-If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the-<a href="https://docs.apify.com/storage/key-value-store" target="_blank">Apify Key-value store</a> cloud storage. Note that you can force usage of the-cloud storage also by passing the `forceCloud` option to [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function, even if the-`APIFY_LOCAL_STORAGE_DIR` variable is set.+If the [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable is set but+[`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) not, the data is stored in the+[Apify Key-value store](https://docs.apify.com/storage/key-value-store) cloud storage. Note that you can force usage of the cloud storage also by+passing the `forceCloud` option to [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyvaluestore) function, even if the+[`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) variable is set.**Example usage:**@@ -61,86 +63,63 @@ const value = await store.getValue('some-key');await store.drop();```-- [KeyValueStore](keyvaluestore)- - [`new exports.KeyValueStore(storeId, storeName)`](#new_KeyValueStore_new)- - [`.getValue(key)`](#KeyValueStore+getValue) ⇒ `Promise<(Object|String|Buffer)>`- - [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ `Promise<void>`- - [`.drop()`](#KeyValueStore+drop) ⇒ `Promise<void>`- - [`.getPublicUrl(key)`](#KeyValueStore+getPublicUrl) ⇒ `string`- - [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ `Promise<void>`--<a name="new_KeyValueStore_new"></a>--## `new exports.KeyValueStore(storeId, storeName)`--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>storeId</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>storeName</code></td><td><code>string</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="KeyValueStore+getValue"></a>--## `keyValueStore.getValue(key)` ⇒ `Promise<(Object|String|Buffer)>`+---++<a name="exports.keyvaluestore"></a>++## `new KeyValueStore(storeId, storeName)`++**Params**++- **`storeId`**: `string`+- **`storeName`**: `string`++---++<a name="getvalue"></a>++## `keyValueStore.getValue(key)`++**Returns**: `Promise<(Object|String|Buffer)>` - Returns a promise that resolves to an object, string or [](https://nodejs.org/api/buffer.html),+depending on the MIME content type of the record.Gets a value from the key-value store.The function returns a `Promise` that resolves to the record value, whose JavaScript type depends on the MIME content type of the record. Records withthe `application/json` content type are automatically parsed and returned as a JavaScript object. Similarly, records with `text/plain` content types-are returned as a string. For all other content types, the value is returned as a raw-<a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a> instance.+are returned as a string. For all other content types, the value is returned as a raw [](https://nodejs.org/api/buffer.html) instance.If the record does not exist, the function resolves to `null`.-To save or delete a value in the key-value store, use the [`setValue`](#KeyValueStore+setValue) function.+To save or delete a value in the key-value store, use the [`KeyValueStore.setValue()`](/docs/api/key-value-store#setvalue) function.**Example usage:**```javascript-const store = await Apify.openKeyValueStore('my-screenshots');+const store = await Apify.openKeyValueStore();const buffer = await store.getValue('screenshot1.png');```-**Returns**: `Promise<(Object|String|Buffer)>` - Returns a promise that resolves to an object, string or-<a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, depending on the MIME content type of the record.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>key</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Unique key of the record. It can be at most 256 characters long and only consist- of the following characters: <code>a</code>-<code>z</code>, <code>A</code>-<code>Z</code>, <code>0</code>-<code>9</code> and <code>!-_.'()</code></p>-</td></tr></tbody>-</table>-<a name="KeyValueStore+setValue"></a>--## `keyValueStore.setValue(key, value, [options])` ⇒ `Promise<void>`+**Params**++- **`key`**: `String` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,+ `A`-`Z`, `0`-`9` and `!-_.'()`++---++<a name="setvalue"></a>++## `keyValueStore.setValue(key, value, [options])`++**Returns**: `Promise<void>`Saves or deletes a record in the key-value store. The function returns a promise that resolves once the record has been saved or deleted.**Example usage:**```javascript-const store = await Apify.openKeyValueStore('my-store');-await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });+const store = await Apify.openKeyValueStore();+await store.setValue('OUTPUT', { foo: 'bar' });```Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`@@ -149,84 +128,65 @@ By default, `value` is converted to JSON and stored with the `application/json;content type, pass it in the options as follows:```javascript-const store = await Apify.openKeyValueStore('my-store');+const store = await Apify.openKeyValueStore('my-text-store');await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });```-If you set custom content type, `value` must be either a string or-<a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, otherwise an error will be thrown.+If you set custom content type, `value` must be either a string or [](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds regardless whether the record existed or not.-To retrieve a value from the key-value store, use the [`getValue`](#KeyValueStore+getValue) function.+To retrieve a value from the key-value store, use the [`KeyValueStore.getValue()`](/docs/api/key-value-store#getvalue) function.**IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`, otherwise the actor process might finish before the value isstored!-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>key</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Unique key of the record. It can be at most 256 characters long and only consist- of the following characters: <code>a</code>-<code>z</code>, <code>A</code>-<code>Z</code>, <code>0</code>-<code>9</code> and <code>!-_.'()</code></p>-</td></tr><tr>-<td><code>value</code></td><td><code>Object</code> | <code>String</code> | <code>Buffer</code></td>-</tr>-<tr>-<td colspan="3"><p>Record data, which can be one of the following values:- <ul>- <li>If <code>null</code>, the record in the key-value store is deleted.</li>- <li>If no <code>options.contentType</code> is specified, <code>value</code> can be any JavaScript object and it will be stringified to JSON.</li>- <li>If <code>options.contentType</code> is specified, <code>value</code> is considered raw data and it must be either a <code>String</code>- or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>.</li>- </ul>- For any other value an error will be thrown.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.contentType]</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Specifies a custom MIME content type of the record.</p>-</td></tr></tbody>-</table>-<a name="KeyValueStore+drop"></a>--## `keyValueStore.drop()` ⇒ `Promise<void>`+**Params**++- **`key`**: `String` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,+ `A`-`Z`, `0`-`9` and `!-_.'()`+- **`value`**: `Object` | `String` | `Buffer` - Record data, which can be one of the following values:+ <ul>+ <li>If `null`, the record in the key-value store is deleted.</li>+ <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>+ <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`+ or [](https://nodejs.org/api/buffer.html).</li>+ </ul>+ For any other value an error will be thrown.+- **`[options]`**: `Object`+ - **`[.contentType]`**: `String` - Specifies a custom MIME content type of the record.++---++<a name="drop"></a>++## `keyValueStore.drop()`++**Returns**: `Promise<void>`Removes the key-value store either from the Apify cloud storage or from the local directory, depending on the mode of operation.-<a name="KeyValueStore+getPublicUrl"></a>+---++<a name="getpublicurl"></a>-## `keyValueStore.getPublicUrl(key)` ⇒ `string`+## `keyValueStore.getPublicUrl(key)`++**Returns**: `string`Returns a URL for the given key that may be used to publicly access the value in the remote key value store.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>key</code></td><td><code>string</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="KeyValueStore+forEachKey"></a>--## `keyValueStore.forEachKey(iteratee, [options])` ⇒ `Promise<void>`+**Params**++- **`key`**: `string`++---++<a name="foreachkey"></a>++## `keyValueStore.forEachKey(iteratee, [options])`++**Returns**: `Promise<void>`Iterates over key value store keys, yielding each in turn to an `iteratee` function. Each invocation of `iteratee` is called with three arguments:`(key, index, info)`, where `key` is the record key, `index` is a zero-based index of the key in the current iteration (regardless of@@ -244,28 +204,10 @@ await keyValueStore.forEachKey(async (key, index, info) => {});```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>iteratee</code></td><td><code><a href="../typedefs/keyconsumer">KeyConsumer</a></code></td>-</tr>-<tr>-<td colspan="3"><p>A function that is called for every key in the key value store.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>forEachKey()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>[options.exclusiveStartKey]</code></td><td><code>string</code></td>-</tr>-<tr>-<td colspan="3"><p>All keys up to this one (including) are skipped from the result.</p>-</td></tr></tbody>-</table>+**Params**++- **`iteratee`**: [`KeyConsumer`](/docs/typedefs/key-consumer) - A function that is called for every key in the key value store.+- **`[options]`**: `Object` - All `forEachKey()` parameters are passed via an options object with the following keys:+ - **`[.exclusiveStartKey]`**: `string` - All keys up to this one (including) are skipped from the result.++---diff --git a/docs/api/LiveViewServer.md b/docs/api/LiveViewServer.mdindex b49b9d184..26d19551a 100644--- a/docs/api/LiveViewServer.md+++ b/docs/api/LiveViewServer.md@@ -1,9 +1,9 @@----id: liveviewserver+id: live-view-servertitle: LiveViewServer----<a name="LiveViewServer"></a>+<a name="liveviewserver"></a>`LiveViewServer` enables serving of browser snapshots via web sockets. It includes its own client that provides a simple frontend to viewing thecaptured snapshots. A snapshot consists of three pieces of information, the currently opened URL, the content of the page (HTML) and its screenshot.@@ -24,8 +24,9 @@ snapshots. Once no longer needed, it can be disabled again in the client to remoNOTE: Screenshot taking in browser typically takes around 300ms. So having the `LiveViewServer` always serve snapshots will have a significant impacton performance.-When using [`PuppeteerPool`](puppeteerpool), the `LiveViewServer` can be easily used just by providing the `useLiveView = true` option to the-[`PuppeteerPool`](puppeteerpool). It can also be initiated via [`PuppeteerCrawler`](puppeteercrawler) `puppeteerPoolOptions`.+When using [`PuppeteerPool`](/docs/api/puppeteer-pool), the `LiveViewServer` can be easily used just by providing the `useLiveView = true` option to+the [`PuppeteerPool`](/docs/api/puppeteer-pool). It can also be initiated via [`PuppeteerCrawler`](/docs/api/puppeteer-crawler)+`puppeteerPoolOptions`.It will take snapshots of the first page of the latest browser. Taking snapshots of only a single page improves performance and stability dramaticallyin high concurrency situations.@@ -33,98 +34,75 @@ in high concurrency situations.When running locally, it is often best to use a headful browser for debugging, since it provides a better view into the browser, including DevTools,but `LiveViewServer` works too.-- [LiveViewServer](liveviewserver)- - [`new LiveViewServer([options])`](#new_LiveViewServer_new)- - [`.start()`](#LiveViewServer+start) ⇒ `Promise`- - [`.stop()`](#LiveViewServer+stop) ⇒ `Promise`- - [`.serve(page)`](#LiveViewServer+serve) ⇒ `Promise`- - [`.isRunning()`](#LiveViewServer+isRunning) ⇒ `boolean`- - [`.hasClients()`](#LiveViewServer+hasClients) ⇒ `boolean`+----<a name="new_LiveViewServer_new"></a>+<a name="liveviewserver"></a>## `new LiveViewServer([options])`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>LiveViewServer</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>[options.screenshotDirectoryPath]</code></td><td><code>string</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>By default, the screenshots are saved to- the <code>live_view</code> directory in the Apify local storage directory.- Provide a different absolute path to change the settings.</p>-</td></tr><tr>-<td><code>[options.maxScreenshotFiles]</code></td><td><code>number</code></td><td><code>10</code></td>-</tr>-<tr>-<td colspan="3"><p>Limits the number of screenshots stored- by the server. This is to prevent using up too much disk space.</p>-</td></tr><tr>-<td><code>[options.snapshotTimeoutSecs]</code></td><td><code>number</code></td><td><code>3</code></td>-</tr>-<tr>-<td colspan="3"><p>If a snapshot is not made within the timeout,- its creation will be aborted. This is to prevent- pages from being hung up by a stalled screenshot.</p>-</td></tr><tr>-<td><code>[options.maxSnapshotFrequencySecs]</code></td><td><code>number</code></td><td><code>2</code></td>-</tr>-<tr>-<td colspan="3"><p>Use this parameter to further decrease the resource consumption- of <code>LiveViewServer</code> by limiting the frequency at which it'll- serve snapshots.</p>-</td></tr></tbody>-</table>-<a name="LiveViewServer+start"></a>--## `liveViewServer.start()` ⇒ `Promise`+**Params**++- **`[options]`**: `Object` - All `LiveViewServer` parameters are passed via an options object with the following keys:+ - **`[.screenshotDirectoryPath]`**: `string` - By default, the screenshots are saved to the `live_view` directory in the Apify local storage+ directory. Provide a different absolute path to change the settings.+ - **`[.maxScreenshotFiles]`**: `number` <code> = 10</code> - Limits the number of screenshots stored by the server. This is to prevent using up+ too much disk space.+ - **`[.snapshotTimeoutSecs]`**: `number` <code> = 3</code> - If a snapshot is not made within the timeout, its creation will be aborted. This is+ to prevent pages from being hung up by a stalled screenshot.+ - **`[.maxSnapshotFrequencySecs]`**: `number` <code> = 2</code> - Use this parameter to further decrease the resource consumption of+ `LiveViewServer` by limiting the frequency at which it'll serve snapshots.++---++<a name="start"></a>++## `liveViewServer.start()`++**Returns**: `Promise`Starts the HTTP server with web socket connections enabled. Snapshots will not be created until a client has connected.-<a name="LiveViewServer+stop"></a>+---++<a name="stop"></a>++## `liveViewServer.stop()`-## `liveViewServer.stop()` ⇒ `Promise`+**Returns**: `Promise`Prevents the server from receiving more connections. Existing connections will not be terminated, but the server will not prevent a process exit.-<a name="LiveViewServer+serve"></a>+---++<a name="serve"></a>++## `liveViewServer.serve(page)`-## `liveViewServer.serve(page)` ⇒ `Promise`+**Returns**: `Promise`Serves a snapshot to all connected clients. Screenshots are not served directly, only their index number which is used by client to retrieve thescreenshot.Will time out and throw in `options.snapshotTimeoutSecs`.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="LiveViewServer+isRunning"></a>--## `liveViewServer.isRunning()` ⇒ `boolean`--<a name="LiveViewServer+hasClients"></a>--## `liveViewServer.hasClients()` ⇒ `boolean`+**Params**++- **`page`**: `Page`++---++<a name="isrunning"></a>++## `liveViewServer.isRunning()`++**Returns**: `boolean`++---++<a name="hasclients"></a>++## `liveViewServer.hasClients()`++**Returns**: `boolean`++---diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.mdindex d065d6660..5592be50c 100644--- a/docs/api/PseudoUrl.md+++ b/docs/api/PseudoUrl.md@@ -1,12 +1,12 @@----id: pseudourl+id: pseudo-urltitle: PseudoUrl----<a name="PseudoUrl"></a>+<a name="pseudourl"></a>Represents a pseudo-URL (PURL) - an URL pattern used by web crawlers to specify which URLs should the crawler visit. This class is used by the-[`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.+[`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks) function.A PURL is simply a URL with special directives enclosed in `[]` brackets. Currently, the only supported directive is `[RegExp]`, which defines aJavaScript-style regular expression to match against the URL.@@ -52,77 +52,47 @@ const purl2 = new Apify.PseudoUrl(/http:\/\/www\.example\.com\/pages\/(\w|-)+/);if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');```-- [PseudoUrl](pseudourl)- - [`new PseudoUrl(purl, requestTemplate)`](#new_PseudoUrl_new)- - [`.matches(url)`](#PseudoUrl+matches) ⇒ `Boolean`- - [`.createRequest(urlOrProps)`](#PseudoUrl+createRequest) ⇒ [`Request`](request)+----<a name="new_PseudoUrl_new"></a>+<a name="pseudourl"></a>## `new PseudoUrl(purl, requestTemplate)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>purl</code></td><td><code>String</code> | <code>RegExp</code></td>-</tr>-<tr>-<td colspan="3"><p>A pseudo-URL string or a regular expression object.- Using a <code>RegExp</code> instance enables more granular control,- such as making the matching case sensitive.</p>-</td></tr><tr>-<td><code>requestTemplate</code></td><td><code><a href="../typedefs/requestoptions">RequestOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs- by the <a href="utils#utils.enqueueLinks"><code>utils.enqueueLinks()</code></a> function.</p>-</td></tr></tbody>-</table>-<a name="PseudoUrl+matches"></a>--## `pseudoUrl.matches(url)` ⇒ `Boolean`+**Params**-Determines whether a URL matches this pseudo-URL pattern.+- **`purl`**: `String` | `RegExp` - A pseudo-URL string or a regular expression object. Using a `RegExp` instance enables more granular control,+ such as making the matching case sensitive.+- **`requestTemplate`**: [`RequestOptions`](/docs/typedefs/request-options) - Options for the new [`Request`](/docs/api/request) instances created+ for matching URLs by the [`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks) function.++---++<a name="matches"></a>++## `pseudoUrl.matches(url)`**Returns**: `Boolean` - Returns `true` if given URL matches pseudo-URL.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>url</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>URL to be matched.</p>-</td></tr></tbody>-</table>-<a name="PseudoUrl+createRequest"></a>--## `pseudoUrl.createRequest(urlOrProps)` ⇒ [`Request`](request)--Creates a Request object from a provided `requestTemplate` and a given URL or an object that specifies \$[`Request`](request) properties. In case of a-collision the properties will override the template, except for `userData`, which will be merged together, with the `userData` property having-preference over the template. This enables dynamic overriding of the template.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>urlOrProps</code></td><td><code>string</code> | <code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>+Determines whether a URL matches this pseudo-URL pattern.++**Params**++- **`url`**: `String` - URL to be matched.++---++<a name="createrequest"></a>++## `pseudoUrl.createRequest(urlOrProps)`++**Returns**: [`Request`](/docs/api/request)++Creates a Request object from a provided `requestTemplate` and a given URL or an object that specifies \$[`Request`](/docs/api/request) properties. In+case of a collision the properties will override the template, except for `userData`, which will be merged together, with the `userData` property+having preference over the template. This enables dynamic overriding of the template.++**Params**++- **`urlOrProps`**: `string` | `Object`++---diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex e3aca8226..88ee0f50e 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -1,37 +1,42 @@----id: puppeteercrawler+id: puppeteer-crawlertitle: PuppeteerCrawler----<a name="PuppeteerCrawler"></a>+<a name="puppeteercrawler"></a>-Provides a simple framework for parallel crawling of web pages using headless Chrome with-<a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>. The URLs to crawl are fed either from a static list of URLs or from-a dynamic queue of URLs enabling recursive crawling of websites.+Provides a simple framework for parallel crawling of web pages using headless Chrome with [Puppeteer](https://github.com/GoogleChrome/puppeteer). The+URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data, it is useful for crawling of websites that require to execute-JavaScript. If the target website doesn't need JavaScript, consider using [`CheerioCrawler`](cheeriocrawler), which downloads the pages using raw HTTP-requests and is about 10x faster.+JavaScript. If the target website doesn't need JavaScript, consider using [`CheerioCrawler`](/docs/api/cheerio-crawler), which downloads the pages+using raw HTTP requests and is about 10x faster.-The source URLs are represented using [`Request`](request) objects that are fed from [`RequestList`](requestlist) or [`RequestQueue`](requestqueue)-instances provided by the [`requestList`](#new_PuppeteerCrawler_new) or [`requestQueue`](#new_PuppeteerCrawler_new) constructor options, respectively.+The source URLs are represented using [`Request`](/docs/api/request) objects that are fed from [`RequestList`](/docs/api/request-list) or+[`RequestQueue`](/docs/api/request-queue) instances provided by the+[`PuppeteerCrawlerOptions.requestList`](/docs/typedefs/puppeteer-crawler-options#requestlist) or+[`PuppeteerCrawlerOptions.requestQueue`](/docs/typedefs/puppeteer-crawler-options#requestqueue) constructor options, respectively.-If both [`requestList`](#new_PuppeteerCrawler_new) and [`requestQueue`](#new_PuppeteerCrawler_new) are used, the instance first processes URLs from-the [`RequestList`](requestlist) and automatically enqueues all of them to [`RequestQueue`](requestqueue) before it starts their processing. This-ensures that a single URL is not crawled multiple times.+If both [`PuppeteerCrawlerOptions.requestList`](/docs/typedefs/puppeteer-crawler-options#requestlist) and+[`PuppeteerCrawlerOptions.requestQueue`](/docs/typedefs/puppeteer-crawler-options#requestqueue) are used, the instance first processes URLs from the+[`RequestList`](/docs/api/request-list) and automatically enqueues all of them to [`RequestQueue`](/docs/api/request-queue) before it starts their+processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [`Request`](request) objects to crawl.+The crawler finishes when there are no more [`Request`](/docs/api/request) objects to crawl.-`PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each [`Request`](request) object to crawl and then calls the function provided by user as-the [`handlePageFunction()`](#new_PuppeteerCrawler_new) option.+`PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each [`Request`](/docs/api/request) object to crawl and then calls the function provided by+user as the [`PuppeteerCrawlerOptions.handlePageFunction`](/docs/typedefs/puppeteer-crawler-options#handlepagefunction) option.New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the-[`AutoscaledPool`](autoscaledpool) class. All [`AutoscaledPool`](autoscaledpool) configuration options can be passed to the `autoscaledPoolOptions`-parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [`AutoscaledPool`](autoscaledpool)-options are available directly in the `PuppeteerCrawler` constructor.+[`AutoscaledPool`](/docs/api/autoscaled-pool) class. All [`AutoscaledPool`](/docs/api/autoscaled-pool) configuration options can be passed to the+[`PuppeteerCrawlerOptions.autoscaledPoolOptions`](/docs/typedefs/puppeteer-crawler-options#autoscaledpooloptions) parameter of the `PuppeteerCrawler`+constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [`AutoscaledPoolOptions`](/docs/typedefs/autoscaled-pool-options) are+available directly in the `PuppeteerCrawler` constructor.-Note that the pool of Puppeteer instances is internally managed by the [`PuppeteerPool`](puppeteerpool) class. Many constructor options such as-`maxOpenPagesPerInstance` or `launchPuppeteerFunction` are passed directly to [`PuppeteerPool`](puppeteerpool) constructor.+Note that the pool of Puppeteer instances is internally managed by the [`PuppeteerPool`](/docs/api/puppeteer-pool) class. Many constructor options+such as [`PuppeteerPoolOptions.maxOpenPagesPerInstance`](/docs/typedefs/puppeteer-pool-options#maxopenpagesperinstance) or+[`PuppeteerPoolOptions.launchPuppeteerFunction`](/docs/typedefs/puppeteer-pool-options#launchpuppeteerfunction) are passed directly to the+[`PuppeteerPool`](/docs/api/puppeteer-pool) constructor.**Example usage:**@@ -61,52 +66,25 @@ const crawler = new Apify.PuppeteerCrawler({await crawler.run();```-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.- Note that this property is only initialized after calling the <a href="#PuppeteerCrawler+run"><code>run</code></a> function.- You can use it to change the concurrency settings on the fly,- to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>- or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>-</td></tr></tbody>-</table>--- [PuppeteerCrawler](puppeteercrawler)- - [`new PuppeteerCrawler(options)`](#new_PuppeteerCrawler_new)- - [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise<void>`--<a name="new_PuppeteerCrawler_new"></a>+---++<a name="puppeteercrawler"></a>## `new PuppeteerCrawler(options)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/puppeteercrawleroptions">PuppeteerCrawlerOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>PuppeteerCrawler</code> parameters are passed- via an options object with the following keys:</p>-</td></tr></tbody>-</table>-<a name="PuppeteerCrawler+run"></a>--## `puppeteerCrawler.run()` ⇒ `Promise<void>`+**Params**++- **`options`**: [`PuppeteerCrawlerOptions`](/docs/typedefs/puppeteer-crawler-options) - All `PuppeteerCrawler` parameters are passed via an options+ object.++---++<a name="run"></a>++## `puppeteerCrawler.run()`++**Returns**: `Promise<void>`Runs the crawler. Returns promise that gets resolved once all the requests got processed.++---diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.mdindex 86cfe2825..45f6d01d8 100644--- a/docs/api/PuppeteerPool.md+++ b/docs/api/PuppeteerPool.md@@ -1,11 +1,11 @@----id: puppeteerpool+id: puppeteer-pooltitle: PuppeteerPool----<a name="PuppeteerPool"></a>+<a name="puppeteerpool"></a>-Manages a pool of Chrome browser instances controlled using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.+Manages a pool of Chrome browser instances controlled using [Puppeteer](https://github.com/GoogleChrome/puppeteer).`PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies. This is useful in order to facilitaterotation of proxies, cookies or other settings in order to prevent detection of your web scraping bot, access web pages from various countries etc.@@ -13,7 +13,7 @@ rotation of proxies, cookies or other settings in order to prevent detection ofAdditionally, the reuse of browser instances instances speeds up crawling, and the retirement of instances helps mitigate effects of memory leaks inChrome.-`PuppeteerPool` is internally used by the [`PuppeteerCrawler`](puppeteercrawler) class.+`PuppeteerPool` is internally used by the [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) class.**Example usage:**@@ -38,107 +38,85 @@ const page3 = await puppeteerPool.newPage();await puppeteerPool.destroy();```-- [PuppeteerPool](puppeteerpool)- - [`new PuppeteerPool([options])`](#new_PuppeteerPool_new)- - [`.newPage()`](#PuppeteerPool+newPage) ⇒ `Promise<Page>`- - [`.destroy()`](#PuppeteerPool+destroy) ⇒ `Promise<void>`- - [`.retire(browser)`](#PuppeteerPool+retire) ⇒ `Promise<void>`- - [`.recyclePage(page)`](#PuppeteerPool+recyclePage) ⇒ `Promise<void>`- - [`.serveLiveViewSnapshot(page)`](#PuppeteerPool+serveLiveViewSnapshot) ⇒ `Promise<void>`+----<a name="new_PuppeteerPool_new"></a>+<a name="puppeteerpool"></a>## `new PuppeteerPool([options])`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code><a href="../typedefs/puppeteerpooloptions">PuppeteerPoolOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>PuppeteerPool</code> parameters are passed- via an options object with the following keys:</p>-</td></tr></tbody>-</table>-<a name="PuppeteerPool+newPage"></a>--## `puppeteerPool.newPage()` ⇒ `Promise<Page>`+**Params**++- **`[options]`**: [`PuppeteerPoolOptions`](/docs/typedefs/puppeteer-pool-options) - All `PuppeteerPool` parameters are passed via an options+ object.++---++<a name="newpage"></a>++## `puppeteerPool.newPage()`++**Returns**: `Promise<Page>`Produces a new page instance either by reusing an idle page that currently isn't processing any request or by spawning a new page (new browser tab) inone of the available browsers when no idle pages are available.To spawn a new browser tab for each page, set the `reusePages` constructor option to false.-<a name="PuppeteerPool+destroy"></a>+---++<a name="destroy"></a>-## `puppeteerPool.destroy()` ⇒ `Promise<void>`+## `puppeteerPool.destroy()`++**Returns**: `Promise<void>`Closes all open browsers.-<a name="PuppeteerPool+retire"></a>+----## `puppeteerPool.retire(browser)` ⇒ `Promise<void>`+<a name="retire"></a>-Manually retires a Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-browser" target="_blank"><code>Browser</code></a> instance-from the pool. The browser will continue to process open pages so that they may gracefully finish. This is unlike `browser.close()` which will-forcibly terminate the browser and all open pages will be closed.+## `puppeteerPool.retire(browser)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>browser</code></td><td><code>Browser</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="PuppeteerPool+recyclePage"></a>+**Returns**: `Promise<void>`-## `puppeteerPool.recyclePage(page)` ⇒ `Promise<void>`+Manually retires a Puppeteer [`Browser`](https://pptr.dev/#?product=Puppeteer&show=api-class-browser) instance from the pool. The browser will+continue to process open pages so that they may gracefully finish. This is unlike `browser.close()` which will forcibly terminate the browser and all+open pages will be closed.++**Params**++- **`browser`**: `Browser`++---++<a name="recyclepage"></a>++## `puppeteerPool.recyclePage(page)`++**Returns**: `Promise<void>`Closes the page, unless the `reuseTabs` option is set to true. Then it would only flag the page for a future reuse, without actually closing it.NOTE: LiveView snapshotting is tied to this function. When `useLiveView` option is set to true, a snapshot of the page will be taken just beforeclosing the page or flagging it for reuse.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="PuppeteerPool+serveLiveViewSnapshot"></a>--## `puppeteerPool.serveLiveViewSnapshot(page)` ⇒ `Promise<void>`+**Params**++- **`page`**: `Page`++---++<a name="serveliveviewsnapshot"></a>++## `puppeteerPool.serveLiveViewSnapshot(page)`++**Returns**: `Promise<void>`Tells the connected LiveViewServer to serve a snapshot when available.-<table>-<thead>-<tr>-<th>Param</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td>-</tr>-<tr>-</tr></tbody>-</table>+**Params**++- **`page`**:++---diff --git a/docs/api/Request.md b/docs/api/Request.mdindex 6dd5765a8..445833842 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -3,14 +3,14 @@ id: requesttitle: Request----<a name="Request"></a>+<a name="request"></a>Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata. The `Request` object also stores informationabout errors that occurred during processing of the request.Each `Request` instance has the `uniqueKey` property, which can be either specified manually in the constructor or generated automatically from theURL. Two requests with the same `uniqueKey` are considered as pointing to the same web resource. This behavior applies to all Apify SDK classes, such-as [`RequestList`](requestlist), [`RequestQueue`](requestqueue) or [`PuppeteerCrawler`](puppeteercrawler).+as [`RequestList`](/docs/api/request-list), [`RequestQueue`](/docs/api/request-queue) or [`PuppeteerCrawler`](/docs/api/puppeteer-crawler).Example use:@@ -30,107 +30,118 @@ request.pushErrorMessage(new Error('Request failed!'));const foo = request.userData.foo;```-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>id</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Request ID</p>-</td></tr><tr>-<td><code>url</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>URL of the web page to crawl.</p>-</td></tr><tr>-<td><code>loadedUrl</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>An actually loaded URL after redirects, if present. HTTP redirects are guaranteed- to be included.</p>-<p> When using <a href="puppeteercrawler"><code>PuppeteerCrawler</code></a>, meta tag and JavaScript redirects may,- or may not be included, depending on their nature. This generally means that redirects,- which happen immediately will most likely be included, but delayed redirects will not.</p>-</td></tr><tr>-<td><code>uniqueKey</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>A unique key identifying the request.- Two requests with the same <code>uniqueKey</code> are considered as pointing to the same URL.</p>-</td></tr><tr>-<td><code>method</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>HTTP method, e.g. <code>GET</code> or <code>POST</code>.</p>-</td></tr><tr>-<td><code>payload</code></td><td><code>String</code> | <code>Buffer</code></td>-</tr>-<tr>-<td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p>-</td></tr><tr>-<td><code>noRetry</code></td><td><code>Boolean</code></td>-</tr>-<tr>-<td colspan="3"><p>The <code>true</code> value indicates that the request will not be automatically retried on error.</p>-</td></tr><tr>-<td><code>retryCount</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates the number of times the crawling of the request has been retried on error.</p>-</td></tr><tr>-<td><code>errorMessages</code></td><td><code>Array<String></code></td>-</tr>-<tr>-<td colspan="3"><p>An array of error messages from request processing.</p>-</td></tr><tr>-<td><code>headers</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Object with HTTP headers. Key is header name, value is the value.</p>-</td></tr><tr>-<td><code>userData</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Custom user data assigned to the request.</p>-</td></tr><tr>-<td><code>handledAt</code></td><td><code>Date</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates the time when the request has been processed.- Is <code>null</code> if the request has not been crawled yet.</p>-</td></tr></tbody>-</table>--- [Request](request)- - [`new Request(options)`](#new_Request_new)- - [`.pushErrorMessage(errorOrMessage, [options])`](#Request+pushErrorMessage)--<a name="new_Request_new"></a>+## Properties++### `id`++**Type**: `String`++Request ID++---++### `url`++**Type**: `String`++URL of the web page to crawl.++---++### `loadedUrl`++**Type**: `String`++An actually loaded URL after redirects, if present. HTTP redirects are guaranteed to be included.++When using [`PuppeteerCrawler`](/docs/api/puppeteer-crawler), meta tag and JavaScript redirects may, or may not be included, depending on their+nature. This generally means that redirects, which happen immediately will most likely be included, but delayed redirects will not.++---++### `uniqueKey`++**Type**: `String`++A unique key identifying the request. Two requests with the same `uniqueKey` are considered as pointing to the same URL.++---++### `method`++**Type**: `String`++HTTP method, e.g. `GET` or `POST`.++---++### `payload`++**Type**: `String` | `Buffer`++HTTP request payload, e.g. for POST requests.++---++### `noRetry`++**Type**: `Boolean`++The `true` value indicates that the request will not be automatically retried on error.++---++### `retryCount`++**Type**: `Number`++Indicates the number of times the crawling of the request has been retried on error.++---++### `errorMessages`++**Type**: `Array<String>`++An array of error messages from request processing.++---++### `headers`++**Type**: `Object`++Object with HTTP headers. Key is header name, value is the value.++---++### `userData`++**Type**: `Object`++Custom user data assigned to the request.++---++### `handledAt`++**Type**: `Date`++Indicates the time when the request has been processed. Is `null` if the request has not been crawled yet.++---++<a name="request"></a>## `new Request(options)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/requestoptions">RequestOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p><code>Request</code> parameters including the URL, HTTP method and headers, and others.</p>-</td></tr></tbody>-</table>-<a name="Request+pushErrorMessage"></a>+**Params**++- **`options`**: [`RequestOptions`](/docs/typedefs/request-options) - `Request` parameters including the URL, HTTP method and headers, and others.++---++<a name="pusherrormessage"></a>## `request.pushErrorMessage(errorOrMessage, [options])`@@ -142,26 +153,10 @@ Nevertheless, to improve the debugging experience when using third party libraria type inspection of the passed argument and attempts to extract as much information as possible, since just throwing a bad type error makes anydebugging rather difficult.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>errorOrMessage</code></td><td><code>Error</code> | <code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Error object or error message to be stored in the request.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.omitStack]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>Only push the error message without stack trace when true.</p>-</td></tr></tbody>-</table>+**Params**++- **`errorOrMessage`**: `Error` | `String` - Error object or error message to be stored in the request.+- **`[options]`**: `Object`+ - **`[.omitStack]`**: `Boolean` <code> = false</code> - Only push the error message without stack trace when true.++---diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex 2c811590a..c14c6f55e 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -1,27 +1,28 @@----id: requestlist+id: request-listtitle: RequestList----<a name="RequestList"></a>+<a name="requestlist"></a>Represents a static list of URLs to crawl. The URLs can be provided either in code or parsed from a text file hosted on the web.-Each URL is represented using an instance of the [`Request`](request) class. The list can only contain unique URLs. More precisely, it can only-contain `Request` instances with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden. To-add a single URL to the list multiple times, corresponding [`Request`](request) objects will need to have different `uniqueKey` properties. You can-use the `keepDuplicateUrls` option to do this for you when initializing the `RequestList` from sources.+Each URL is represented using an instance of the [`Request`](/docs/api/request) class. The list can only contain unique URLs. More precisely, it can+only contain `Request` instances with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be+overridden. To add a single URL to the list multiple times, corresponding [`Request`](/docs/api/request) objects will need to have different+`uniqueKey` properties. You can use the `keepDuplicateUrls` option to do this for you when initializing the `RequestList` from sources.-Once you create an instance of `RequestList`, you need to call the [`initialize`](#RequestList+initialize) function before the instance can be used.-After that, no more URLs can be added to the list.+Once you create an instance of `RequestList`, you need to call the [`RequestList.initialize()`](/docs/api/request-list#initialize) function before the+instance can be used. After that, no more URLs can be added to the list.-`RequestList` is used by [`BasicCrawler`](basiccrawler), [`CheerioCrawler`](cheeriocrawler) and [`PuppeteerCrawler`](puppeteercrawler) as a source of-URLs to crawl. Unlike [`RequestQueue`](requestqueue), `RequestList` is static but it can contain even millions of URLs.+`RequestList` is used by [`BasicCrawler`](/docs/api/basic-crawler), [`CheerioCrawler`](/docs/api/cheerio-crawler) and+[`PuppeteerCrawler`](/docs/api/puppeteer-crawler) as a source of URLs to crawl. Unlike [`RequestQueue`](/docs/api/request-queue), `RequestList` is+static but it can contain even millions of URLs.`RequestList` has an internal state where it stores information about which requests were already handled, which are in progress and which were-reclaimed. The state may be automatically persisted to the default [`KeyValueStore`](keyvaluestore) by setting the `persistStateKey` option so that if-the Node.js process is restarted, the crawling can continue where it left off. The automated persisting is launched upon receiving the `persistState`-event that is periodically emitted by [`Apify.events`](events).+reclaimed. The state may be automatically persisted to the default [`KeyValueStore`](/docs/api/key-value-store) by setting the `persistStateKey`+option so that if the Node.js process is restarted, the crawling can continue where it left off. The automated persisting is launched upon receiving+the `persistState` event that is periodically emitted by [`Apify.events`](/docs/api/apify#events).The internal state is closely tied to the provided sources (URLs). If the sources change on actor restart, the state will become corrupted and`RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web. In such case, use the@@ -63,126 +64,129 @@ await requestList.markRequestHandled(request1);await requestList.reclaimRequest(request2);```-- [RequestList](requestlist)- - [`new exports.RequestList(options)`](#new_RequestList_new)- - [`.initialize()`](#RequestList+initialize) ⇒ `Promise<void>`- - [`.persistState()`](#RequestList+persistState) ⇒ `Promise<void>`- - [`.getState()`](#RequestList+getState) ⇒ [`RequestListState`](../typedefs/requestliststate)- - [`.isEmpty()`](#RequestList+isEmpty) ⇒ `Promise<Boolean>`- - [`.isFinished()`](#RequestList+isFinished) ⇒ `Promise<Boolean>`- - [`.fetchNextRequest()`](#RequestList+fetchNextRequest) ⇒ [`Promise<Request>`](request)- - [`.markRequestHandled(request)`](#RequestList+markRequestHandled) ⇒ `Promise<void>`- - [`.reclaimRequest(request)`](#RequestList+reclaimRequest) ⇒ `Promise<void>`- - [`.length()`](#RequestList+length) ⇒ `Number`- - [`.handledCount()`](#RequestList+handledCount) ⇒ `Number`--<a name="new_RequestList_new"></a>--## `new exports.RequestList(options)`--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/requestlistoptions">RequestListOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>RequestList</code> configuration options</p>-</td></tr></tbody>-</table>-<a name="RequestList+initialize"></a>--## `requestList.initialize()` ⇒ `Promise<void>`+---++<a name="exports.requestlist"></a>++## `new RequestList(options)`++**Params**++- **`options`**: [`RequestListOptions`](/docs/typedefs/request-list-options) - All `RequestList` configuration options++---++<a name="initialize"></a>++## `requestList.initialize()`++**Returns**: `Promise<void>`Loads all remote sources of URLs and potentially starts periodic state persistence. This function must be called before you can start using theinstance in a meaningful way.-<a name="RequestList+persistState"></a>+---++<a name="persiststate"></a>-## `requestList.persistState()` ⇒ `Promise<void>`+## `requestList.persistState()`-Persists the current state of the `RequestList` into the default [`KeyValueStore`](keyvaluestore). The state is persisted automatically in regular-intervals, but calling this method manually is useful in cases where you want to have the most current state available after you pause or stop+**Returns**: `Promise<void>`++Persists the current state of the `RequestList` into the default [`KeyValueStore`](/docs/api/key-value-store). The state is persisted automatically in+regular intervals, but calling this method manually is useful in cases where you want to have the most current state available after you pause or stopfetching its requests. For example after you pause or abort a crawl. Or just before a server migration.-<a name="RequestList+getState"></a>+---++<a name="getstate"></a>-## `requestList.getState()` ⇒ [`RequestListState`](../typedefs/requestliststate)+## `requestList.getState()`++**Returns**: [`RequestListState`](/docs/typedefs/request-list-state)Returns an object representing the internal state of the `RequestList` instance. Note that the object's fields can change in future releases.-<a name="RequestList+isEmpty"></a>+----## `requestList.isEmpty()` ⇒ `Promise<Boolean>`+<a name="isempty"></a>-Resolves to `true` if the next call to [`fetchNextRequest`](#RequestList+fetchNextRequest) function would return `null`, otherwise it resolves to-`false`. Note that even if the list is empty, there might be some pending requests currently being processed.+## `requestList.isEmpty()`-<a name="RequestList+isFinished"></a>+**Returns**: `Promise<Boolean>`-## `requestList.isFinished()` ⇒ `Promise<Boolean>`+Resolves to `true` if the next call to [`RequestList.fetchNextRequest()`](/docs/api/request-list#fetchnextrequest) function would return `null`,+otherwise it resolves to `false`. Note that even if the list is empty, there might be some pending requests currently being processed.++---++<a name="isfinished"></a>++## `requestList.isFinished()`++**Returns**: `Promise<Boolean>`Returns `true` if all requests were already handled and there are no more left.-<a name="RequestList+fetchNextRequest"></a>+----## `requestList.fetchNextRequest()` ⇒ [`Promise<Request>`](request)+<a name="fetchnextrequest"></a>-Gets the next [`Request`](request) to process. First, the function gets a request previously reclaimed using the-[`reclaimRequest`](#RequestList+reclaimRequest) function, if there is any. Otherwise it gets the next request from sources.+## `requestList.fetchNextRequest()`++**Returns**: [`Promise<Request>`](/docs/api/request)++Gets the next [`Request`](/docs/api/request) to process. First, the function gets a request previously reclaimed using the+[`RequestList.reclaimRequest()`](/docs/api/request-list#reclaimrequest) function, if there is any. Otherwise it gets the next request from sources.The function's `Promise` resolves to `null` if there are no more requests to process.-<a name="RequestList+markRequestHandled"></a>+---++<a name="markrequesthandled"></a>++## `requestList.markRequestHandled(request)`-## `requestList.markRequestHandled(request)` ⇒ `Promise<void>`+**Returns**: `Promise<void>`Marks request as handled after successful processing.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="RequestList+reclaimRequest"></a>--## `requestList.reclaimRequest(request)` ⇒ `Promise<void>`+**Params**++- **`request`**: [`Request`](/docs/api/request)++---++<a name="reclaimrequest"></a>++## `requestList.reclaimRequest(request)`++**Returns**: `Promise<void>`Reclaims request to the list if its processing failed. The request will become available in the next `this.fetchNextRequest()`.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="RequestList+length"></a>--## `requestList.length()` ⇒ `Number`+**Params**++- **`request`**: [`Request`](/docs/api/request)++---++<a name="length"></a>++## `requestList.length()`++**Returns**: `Number`Returns the total number of unique requests present in the `RequestList`.-<a name="RequestList+handledCount"></a>+---++<a name="handledcount"></a>++## `requestList.handledCount()`-## `requestList.handledCount()` ⇒ `Number`+**Returns**: `Number`Returns number of handled requests.++---diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex bbd3f6f78..90ba812b0 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -1,23 +1,24 @@----id: requestqueue+id: request-queuetitle: RequestQueue----<a name="RequestQueue"></a>+<a name="requestqueue"></a>Represents a queue of URLs to crawl, which is used for deep crawling of websites where you start with several URLs and then recursively follow linksto other pages. The data structure supports both breadth-first and depth-first crawling orders.-Each URL is represented using an instance of the [`Request`](request) class. The queue can only contain unique URLs. More precisely, it can only-contain [`Request`](request) instances with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be-overridden. To add a single URL multiple times to the queue, corresponding [`Request`](request) objects will need to have different `uniqueKey`-properties.+Each URL is represented using an instance of the [`Request`](/docs/api/request) class. The queue can only contain unique URLs. More precisely, it can+only contain [`Request`](/docs/api/request) instances with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it+can also be overridden. To add a single URL multiple times to the queue, corresponding [`Request`](/docs/api/request) objects will need to have+different `uniqueKey` properties.-Do not instantiate this class directly, use the [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function instead.+Do not instantiate this class directly, use the [`Apify.openRequestQueue()`](/docs/api/apify#openrequestqueue) function instead.-`RequestQueue` is used by [`BasicCrawler`](basiccrawler), [`CheerioCrawler`](cheeriocrawler) and [`PuppeteerCrawler`](puppeteercrawler) as a source of-URLs to crawl. Unlike [`RequestList`](requestlist), `RequestQueue` supports dynamic adding and removing of requests. On the other hand, the queue is-not optimized for operations that add or remove a large number of URLs in a batch.+`RequestQueue` is used by [`BasicCrawler`](/docs/api/basic-crawler), [`CheerioCrawler`](/docs/api/cheerio-crawler) and+[`PuppeteerCrawler`](/docs/api/puppeteer-crawler) as a source of URLs to crawl. Unlike [`RequestList`](/docs/api/request-list), `RequestQueue`+supports dynamic adding and removing of requests. On the other hand, the queue is not optimized for operations that add or remove a large number of+URLs in a batch.`RequestQueue` stores its data either on local disk or in the Apify Cloud, depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN`environment variable is set.@@ -33,9 +34,9 @@ Note that `{QUEUE_ID}` is the name or ID of the request queue. The default queue`handled` or `pending`, and `{NUMBER}` is an integer indicating the position of the request in the queue.If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the-<a href="https://docs.apify.com/storage/request-queue" target="_blank">Apify Request Queue</a> cloud storage. Note that you can force usage of the-cloud storage also by passing the `forceCloud` option to [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function, even if the-`APIFY_LOCAL_STORAGE_DIR` variable is set.+[Apify Request Queue](https://docs.apify.com/storage/request-queue) cloud storage. Note that you can force usage of the cloud storage also by passing+the `forceCloud` option to [`Apify.openRequestQueue()`](/docs/api/apify#openrequestqueue) function, even if the `APIFY_LOCAL_STORAGE_DIR` variable is+set.**Example usage:**@@ -63,167 +64,135 @@ await queue.markRequestHandled(request1);await queue.reclaimRequest(request2);```-- [RequestQueue](requestqueue)- - [`.addRequest(request, [options])`](#RequestQueue+addRequest) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)- - [`.getRequest(requestId)`](#RequestQueue+getRequest) ⇒ [`Promise<Request>`](request)- - [`.fetchNextRequest()`](#RequestQueue+fetchNextRequest) ⇒ [`Promise<Request>`](request)- - [`.markRequestHandled(request)`](#RequestQueue+markRequestHandled) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)- - [`.reclaimRequest(request, [options])`](#RequestQueue+reclaimRequest) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)- - [`.isEmpty()`](#RequestQueue+isEmpty) ⇒ `Promise<Boolean>`- - [`.isFinished()`](#RequestQueue+isFinished) ⇒ `Promise<Boolean>`- - [`.drop()`](#RequestQueue+drop) ⇒ `Promise`- - [`.handledCount()`](#RequestQueue+handledCount) ⇒ `Promise<number>`- - [`.getInfo()`](#RequestQueue+getInfo) ⇒ `Promise<Object>`+---++<a name="addrequest"></a>-<a name="RequestQueue+addRequest"></a>+## `requestQueue.addRequest(request, [options])`-## `requestQueue.addRequest(request, [options])` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)+**Returns**: [`Promise<QueueOperationInfo>`](/docs/typedefs/queue-operation-info)Adds a request to the queue.If a request with the same `uniqueKey` property is already present in the queue, it will not be updated. You can find out whether this happened from-the resulting [`QueueOperationInfo`](../typedefs/queueoperationinfo) object.+the resulting [`QueueOperationInfo`](/docs/typedefs/queue-operation-info) object.-To add multiple requests to the queue by extracting links from a webpage, see the [`Apify.utils.enqueueLinks()`](utils#utils.enqueueLinks) helper+To add multiple requests to the queue by extracting links from a webpage, see the [`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks) helperfunction.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code><a href="../typedefs/requestoptions">RequestOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p><a href="request"><code>Request</code></a> object or vanilla object with request data.-Note that the function sets the <code>uniqueKey</code> and <code>id</code> fields to the passed object.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.forefront]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code>, the request will be added to the foremost position in the queue.</p>-</td></tr></tbody>-</table>-<a name="RequestQueue+getRequest"></a>--## `requestQueue.getRequest(requestId)` ⇒ [`Promise<Request>`](request)+**Params**++- **`request`**: [`Request`](/docs/api/request) | [`RequestOptions`](/docs/typedefs/request-options) - [`Request`](/docs/api/request) object or+ vanilla object with request data. Note that the function sets the `uniqueKey` and `id` fields to the passed object.+- **`[options]`**: `Object`+ - **`[.forefront]`**: `Boolean` <code> = false</code> - If `true`, the request will be added to the foremost position in the queue.++---++<a name="getrequest"></a>++## `requestQueue.getRequest(requestId)`++**Returns**: [`Promise<Request>`](/docs/api/request) - Returns the request object, or `null` if it was not found.Gets the request from the queue specified by ID.-**Returns**: [`Promise<Request>`](request) - Returns the request object, or `null` if it was not found.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>requestId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the request.</p>-</td></tr></tbody>-</table>-<a name="RequestQueue+fetchNextRequest"></a>--## `requestQueue.fetchNextRequest()` ⇒ [`Promise<Request>`](request)+**Params**++- **`requestId`**: `String` - ID of the request.++---++<a name="fetchnextrequest"></a>++## `requestQueue.fetchNextRequest()`++**Returns**: [`Promise<Request>`](/docs/api/request) - Returns the request object or `null` if there are no more pending requests.Returns a next request in the queue to be processed, or `null` if there are no more pending requests.-Once you successfully finish processing of the request, you need to call [`requestQueue.markRequestHandled()`](#RequestQueue+markRequestHandled) to-mark the request as handled in the queue. If there was some error in processing the request, call-[`requestQueue.reclaimRequest()`](#RequestQueue+reclaimRequest) instead, so that the queue will give the request to some other consumer in another-call to the `fetchNextRequest` function.+Once you successfully finish processing of the request, you need to call+[`RequestQueue.markRequestHandled()`](/docs/api/request-queue#markrequesthandled) to mark the request as handled in the queue. If there was some error+in processing the request, call [`RequestQueue.reclaimRequest()`](/docs/api/request-queue#reclaimrequest) instead, so that the queue will give the+request to some other consumer in another call to the `fetchNextRequest` function.Note that the `null` return value doesn't mean the queue processing finished, it means there are currently no pending requests. To check whether all-requests in queue were finished, use [`requestQueue.isFinished()`](#RequestQueue+isFinished) instead.+requests in queue were finished, use [`RequestQueue.isFinished()`](/docs/api/request-queue#isfinished) instead.++---++<a name="markrequesthandled"></a>-**Returns**: [`Promise<Request>`](request) - Returns the request object or `null` if there are no more pending requests.-<a name="RequestQueue+markRequestHandled"></a>+## `requestQueue.markRequestHandled(request)`-## `requestQueue.markRequestHandled(request)` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)+**Returns**: [`Promise<QueueOperationInfo>`](/docs/typedefs/queue-operation-info)-Marks a request that was previously returned by the [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest) function as handled after-successful processing. Handled requests will never again be returned by the `fetchNextRequest` function.+Marks a request that was previously returned by the [`RequestQueue.fetchNextRequest()`](/docs/api/request-queue#fetchnextrequest) function as handled+after successful processing. Handled requests will never again be returned by the `fetchNextRequest` function.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="RequestQueue+reclaimRequest"></a>+**Params**-## `requestQueue.reclaimRequest(request, [options])` ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)+- **`request`**: [`Request`](/docs/api/request)++---++<a name="reclaimrequest"></a>++## `requestQueue.reclaimRequest(request, [options])`++**Returns**: [`Promise<QueueOperationInfo>`](/docs/typedefs/queue-operation-info)Reclaims a failed request back to the queue, so that it can be returned for processed later again by another call to-[`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest). The request record in the queue is updated using the provided `request`+[`RequestQueue.fetchNextRequest()`](/docs/api/request-queue#fetchnextrequest). The request record in the queue is updated using the provided `request`parameter. For example, this lets you store the number of retries or error messages for the request.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.forefront]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the request it placed to the beginning of the queue, so that it's returned-in the next call to <a href="#RequestQueue+fetchNextRequest"><code>requestQueue.fetchNextRequest()</code></a>.-By default, it's put to the end of the queue.</p>-</td></tr></tbody>-</table>-<a name="RequestQueue+isEmpty"></a>--## `requestQueue.isEmpty()` ⇒ `Promise<Boolean>`--Resolves to `true` if the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest) would return `null`, otherwise it resolves-to `false`. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that there is-no activity in the queue, use [`requestQueue.isFinished()`](#RequestQueue+isFinished).--<a name="RequestQueue+isFinished"></a>--## `requestQueue.isFinished()` ⇒ `Promise<Boolean>`+**Params**++- **`request`**: [`Request`](/docs/api/request)+- **`[options]`**: `Object` - **`[.forefront]`**: `Boolean` <code> = false</code> - If `true` then the request it placed to the beginning of the+ queue, so that it's returned in the next call to [`RequestQueue.fetchNextRequest()`](/docs/api/request-queue#fetchnextrequest). By default, it's+ put to the end of the queue.++---++<a name="isempty"></a>++## `requestQueue.isEmpty()`++**Returns**: `Promise<Boolean>`++Resolves to `true` if the next call to [`RequestQueue.fetchNextRequest()`](/docs/api/request-queue#fetchnextrequest) would return `null`, otherwise it+resolves to `false`. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that+there is no activity in the queue, use [`RequestQueue.isFinished()`](/docs/api/request-queue#isfinished).++---++<a name="isfinished"></a>++## `requestQueue.isFinished()`++**Returns**: `Promise<Boolean>`Resolves to `true` if all requests were already handled and there are no more left. Due to the nature of distributed storage used by the queue, thefunction might occasionally return a false negative, but it will never return a false positive.-<a name="RequestQueue+drop"></a>+---++<a name="drop"></a>++## `requestQueue.drop()`-## `requestQueue.drop()` ⇒ `Promise`+**Returns**: `Promise`Removes the queue either from the Apify Cloud storage or from the local directory, depending on the mode of operation.-<a name="RequestQueue+handledCount"></a>+---++<a name="handledcount"></a>++## `requestQueue.handledCount()`-## `requestQueue.handledCount()` ⇒ `Promise<number>`+**Returns**: `Promise<number>`Returns the number of handled requests.@@ -233,9 +202,13 @@ This function is just a convenient shortcut for:const { handledRequestCount } = await queue.getInfo();```-<a name="RequestQueue+getInfo"></a>+---++<a name="getinfo"></a>++## `requestQueue.getInfo()`-## `requestQueue.getInfo()` ⇒ `Promise<Object>`+**Returns**: `Promise<Object>`Returns an object containing general information about the request queue.@@ -257,3 +230,5 @@ function, which in turn calls the [Get request queue](https://apify.com/docs/apipendingRequestCount: 20,}```++---diff --git a/docs/api/Session.md b/docs/api/Session.mdindex 6417a185b..d547689bc 100644--- a/docs/api/Session.md+++ b/docs/api/Session.md@@ -3,88 +3,88 @@ id: sessiontitle: Session----<a name="Session"></a>+<a name="session"></a>Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions. You can imagine each session asa specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint. Session internal state can be enriched with customuser data for example some authorization tokens and specific headers in general.-- [Session](session)- - [`new exports.Session(options)`](#new_Session_new)- - [`.isBlocked()`](#Session+isBlocked) ⇒ `boolean`- - [`.isExpired()`](#Session+isExpired) ⇒ `boolean`- - [`.isMaxUsageCountReached()`](#Session+isMaxUsageCountReached) ⇒ `boolean`- - [`.isUsable()`](#Session+isUsable) ⇒ `boolean`- - [`.markGood()`](#Session+markGood)- - [`.getState()`](#Session+getState) ⇒ [`SessionState`](../typedefs/sessionstate)- - [`.retire()`](#Session+retire)- - [`.markBad()`](#Session+markBad)- - [`.retireOnBlockedStatusCodes(statusCode, [blockedStatusCodes])`](#Session+retireOnBlockedStatusCodes) ⇒ `boolean`- - [`.setCookiesFromResponse(response)`](#Session+setCookiesFromResponse)- - [`.setPuppeteerCookies(cookies, url)`](#Session+setPuppeteerCookies)- - [`.getPuppeteerCookies(url)`](#Session+getPuppeteerCookies) ⇒ `Array<PuppeteerCookie>`- - [`.getCookieString(url)`](#Session+getCookieString) ⇒ `string`--<a name="new_Session_new"></a>--## `new exports.Session(options)`+---++<a name="exports.session"></a>++## `new Session(options)`Session configuration.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/sessionoptions">SessionOptions</a></code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Session+isBlocked"></a>--## `session.isBlocked()` ⇒ `boolean`+**Params**++- **`options`**: [`SessionOptions`](/docs/typedefs/session-options)++---++<a name="isblocked"></a>++## `session.isBlocked()`++**Returns**: `boolean`indicates whether the session is blocked. Session is blocked once it reaches the `maxErrorScore`.-<a name="Session+isExpired"></a>+---++<a name="isexpired"></a>-## `session.isExpired()` ⇒ `boolean`+## `session.isExpired()`++**Returns**: `boolean`Indicates whether the session is expired. Session expiration is determined by the `maxAgeSecs`. Once the session is older than`createdAt + maxAgeSecs` the session is considered expired.-<a name="Session+isMaxUsageCountReached"></a>+---++<a name="ismaxusagecountreached"></a>-## `session.isMaxUsageCountReached()` ⇒ `boolean`+## `session.isMaxUsageCountReached()`++**Returns**: `boolean`Indicates whether the session is used maximum number of times. Session maximum usage count can be changed by `maxUsageCount` parameter.-<a name="Session+isUsable"></a>+---++<a name="isusable"></a>++## `session.isUsable()`-## `session.isUsable()` ⇒ `boolean`+**Returns**: `boolean`Indicates whether the session can be used for next requests. Session is usable when it is not expired, not blocked and the maximum usage count has notbe reached.-<a name="Session+markGood"></a>+---++<a name="markgood"></a>## `session.markGood()`This method should be called after a successful session usage. It increases `usageCount` and potentially lowers the `errorScore` by the`errorScoreDecrement`.-<a name="Session+getState"></a>+---++<a name="getstate"></a>++## `session.getState()`-## `session.getState()` ⇒ [`SessionState`](../typedefs/sessionstate)+**Returns**: [`SessionState`](/docs/typedefs/session-state) - represents session internal state.Gets session state for persistence in KeyValueStore.-**Returns**: [`SessionState`](../typedefs/sessionstate) - represents session internal state.-<a name="Session+retire"></a>+---++<a name="retire"></a>## `session.retire()`@@ -92,41 +92,33 @@ Marks session as blocked and emits event on the `SessionPool` This method shouldit is because of the session configuration and not any external matters. For example when server returns 403 status code. If the session does not workdue to some external factors as server error such as 5XX you probably want to use `markBad` method.-<a name="Session+markBad"></a>+---++<a name="markbad"></a>## `session.markBad()`Increases usage and error count. Should be used when the session has been used unsuccessfully. For example because of timeouts.-<a name="Session+retireOnBlockedStatusCodes"></a>+----## `session.retireOnBlockedStatusCodes(statusCode, [blockedStatusCodes])` ⇒ `boolean`+<a name="retireonblockedstatuscodes"></a>++## `session.retireOnBlockedStatusCodes(statusCode, [blockedStatusCodes])`++**Returns**: `boolean` - whether the session was retired.With certain status codes: `401`, `403` or `429` we can be certain that the target website is blocking us. This function helps to do this convenientlyby retiring the session when such code is received. Optionally the default status codes can be extended in the second parameter.-**Returns**: `boolean` - whether the session was retired.+**Params**++- **`statusCode`**: `number` - HTTP status code+- **`[blockedStatusCodes]`**: `Array<number>` - Custom HTTP status codes that means blocking on particular website.++----<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>statusCode</code></td><td><code>number</code></td>-</tr>-<tr>-<td colspan="3"><p>HTTP status code</p>-</td></tr><tr>-<td><code>[blockedStatusCodes]</code></td><td><code>Array<number></code></td>-</tr>-<tr>-<td colspan="3"><p>Custom HTTP status codes that means blocking on particular website.</p>-</td></tr></tbody>-</table>-<a name="Session+setCookiesFromResponse"></a>+<a name="setcookiesfromresponse"></a>## `session.setCookiesFromResponse(response)`@@ -135,20 +127,13 @@ Node.js responses) or a `Function` (Puppeteer Response).It then parses and saves the cookies from the `set-cookie` header, if available.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>response</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Session+setPuppeteerCookies"></a>+**Params**++- **`response`**: `Object`++---++<a name="setpuppeteercookies"></a>## `session.setPuppeteerCookies(cookies, url)`@@ -163,61 +148,37 @@ manually:]```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>cookies</code></td><td><code>Array<PuppeteerCookie></code></td>-</tr>-<tr>-</tr><tr>-<td><code>url</code></td><td><code>string</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Session+getPuppeteerCookies"></a>--## `session.getPuppeteerCookies(url)` ⇒ `Array<PuppeteerCookie>`+**Params**++- **`cookies`**: `Array<PuppeteerCookie>`+- **`url`**: `string`++---++<a name="getpuppeteercookies"></a>++## `session.getPuppeteerCookies(url)`++**Returns**: `Array<PuppeteerCookie>`Returns cookies in a format compatible with puppeteer and ready to be used with `page.setCookie`.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>url</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>website url. Only cookies stored for this url will be returned</p>-</td></tr></tbody>-</table>-<a name="Session+getCookieString"></a>--## `session.getCookieString(url)` ⇒ `string`+**Params**-Returns cookies saved with the session in the typical key1=value1; key2=value2 format, ready to be used in a cookie header or elsewhere.+- **`url`**: `String` - website url. Only cookies stored for this url will be returned++---++<a name="getcookiestring"></a>++## `session.getCookieString(url)`**Returns**: `string` - - represents `Cookie` header.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>url</code></td><td><code>string</code></td>-</tr>-<tr>-</tr></tbody>-</table>+Returns cookies saved with the session in the typical key1=value1; key2=value2 format, ready to be used in a cookie header or elsewhere.++**Params**++- **`url`**: `string`++---diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.mdindex 726e1cb70..62b597286 100644--- a/docs/api/SessionPool.md+++ b/docs/api/SessionPool.md@@ -1,16 +1,16 @@----id: sessionpool+id: session-pooltitle: SessionPool----<a name="SessionPool"></a>+<a name="sessionpool"></a>-Handles the sessions rotation, creation and persistence. Creates a pool of [`Session`](session) instances, that are randomly rotated. When some-session is marked as blocked. It is removed and new one is created instead. Learn more in the-[`Session management guide`](../guides/sessionmanagement).+Handles the sessions rotation, creation and persistence. Creates a pool of [`Session`](/docs/api/session) instances, that are randomly rotated. When+some session is marked as blocked. It is removed and new one is created instead. Learn more in the+[`Session management guide`](/docs/guides/session-management).-Session pool is by default persisted in default [`KeyValueStore`](keyvaluestore). If you want to have one pool for all runs you have to specify-`persistStateKeyValueStoreId`.+Session pool is by default persisted in default [`KeyValueStore`](/docs/api/key-value-store). If you want to have one pool for all runs you have to+specify [`SessionPoolOptions.persistStateKeyValueStoreId`](/docs/typedefs/session-pool-options#persiststatekeyvaluestoreid).**Example usage:**@@ -48,77 +48,87 @@ session2.markGood();session3.retire();```-- [SessionPool](sessionpool)- - [`new exports.SessionPool([options])`](#new_SessionPool_new)- - [`.usableSessionsCount`](#SessionPool+usableSessionsCount) ⇒ `number`- - [`.retiredSessionsCount`](#SessionPool+retiredSessionsCount) ⇒ `number`- - [`.initialize()`](#SessionPool+initialize) ⇒ `Promise<void>`- - [`.getSession()`](#SessionPool+getSession) ⇒ [`Promise<Session>`](session)- - [`.getState()`](#SessionPool+getState) ⇒ `Object`- - [`.persistState()`](#SessionPool+persistState) ⇒ `Promise`- - [`.teardown()`](#SessionPool+teardown)+----<a name="new_SessionPool_new"></a>+<a name="exports.sessionpool"></a>-## `new exports.SessionPool([options])`+## `new SessionPool([options])`Session pool configuration.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>SessionPool</code> configuration options.</p>-</td></tr></tbody>-</table>-<a name="SessionPool+usableSessionsCount"></a>--## `sessionPool.usableSessionsCount` ⇒ `number`+**Params**++- **`[options]`**: [`SessionPoolOptions`](/docs/typedefs/session-pool-options) - All `SessionPool` configuration options.++---++<a name="usablesessionscount"></a>++## `sessionPool.usableSessionsCount`++**Returns**: `number`Gets count of usable sessions in the pool.-<a name="SessionPool+retiredSessionsCount"></a>+---++<a name="retiredsessionscount"></a>++## `sessionPool.retiredSessionsCount`-## `sessionPool.retiredSessionsCount` ⇒ `number`+**Returns**: `number`Gets count of retired sessions in the pool.-<a name="SessionPool+initialize"></a>+---++<a name="initialize"></a>++## `sessionPool.initialize()`-## `sessionPool.initialize()` ⇒ `Promise<void>`+**Returns**: `Promise<void>`-Starts periodic state persistence and potentially loads SessionPool state from [`KeyValueStore`](keyvaluestore). This function must be called before-you can start using the instance in a meaningful way.+Starts periodic state persistence and potentially loads SessionPool state from [`KeyValueStore`](/docs/api/key-value-store). This function must be+called before you can start using the instance in a meaningful way.-<a name="SessionPool+getSession"></a>+---++<a name="getsession"></a>-## `sessionPool.getSession()` ⇒ [`Promise<Session>`](session)+## `sessionPool.getSession()`++**Returns**: [`Promise<Session>`](/docs/api/session)Gets session. If there is space for new session, it creates and return new session. If the session pool is full, it picks a session from the pool, Ifthe picked session is usable it is returned, otherwise it creates and returns a new one.-<a name="SessionPool+getState"></a>+----## `sessionPool.getState()` ⇒ `Object`+<a name="getstate"></a>++## `sessionPool.getState()`++**Returns**: `Object`Returns an object representing the internal state of the `SessionPool` instance. Note that the object's fields can change in future releases.-<a name="SessionPool+persistState"></a>+----## `sessionPool.persistState()` ⇒ `Promise`+<a name="persiststate"></a>-Persists the current state of the `SessionPool` into the default [`KeyValueStore`](keyvaluestore). The state is persisted automatically in regular-intervals.+## `sessionPool.persistState()`-<a name="SessionPool+teardown"></a>+**Returns**: `Promise`++Persists the current state of the `SessionPool` into the default [`KeyValueStore`](/docs/api/key-value-store). The state is persisted automatically in+regular intervals.++---++<a name="teardown"></a>## `sessionPool.teardown()`Removes listener from `persistState` event. This function should be called after you are done with using the `SessionPool` instance.++---diff --git a/docs/api/Snapshotter.md b/docs/api/Snapshotter.mdindex ed69fc0bd..006442d8f 100644--- a/docs/api/Snapshotter.md+++ b/docs/api/Snapshotter.md@@ -3,11 +3,11 @@ id: snapshottertitle: Snapshotter----<a name="Snapshotter"></a>+<a name="snapshotter"></a>Creates snapshots of system resources at given intervals and marks the resource as either overloaded or not during the last interval. Keeps a history-of the snapshots. It tracks the following resources: Memory, EventLoop, API and CPU. The class is used by the [`AutoscaledPool`](autoscaledpool)-class.+of the snapshots. It tracks the following resources: Memory, EventLoop, API and CPU. The class is used by the+[`AutoscaledPool`](/docs/api/autoscaled-pool) class.When running on the Apify platform, the CPU and memory statistics are provided by the platform, as collected from the running Docker container. Whenrunning locally, `Snapshotter` makes its own statistics by querying the OS.@@ -23,122 +23,94 @@ Event loop becomes overloaded if it slows down by more than the `maxBlockedMilliClient becomes overloaded when rate limit errors (429 - Too Many Requests), typically received from the request queue, exceed the set limit within theset interval.-- [Snapshotter](snapshotter)- - [`new Snapshotter([options])`](#new_Snapshotter_new)- - [`.start()`](#Snapshotter+start) ⇒ `Promise`- - [`.stop()`](#Snapshotter+stop) ⇒ `Promise`- - [`.getMemorySample([sampleDurationMillis])`](#Snapshotter+getMemorySample) ⇒ `Array`- - [`.getEventLoopSample([sampleDurationMillis])`](#Snapshotter+getEventLoopSample) ⇒ `Array`- - [`.getCpuSample([sampleDurationMillis])`](#Snapshotter+getCpuSample) ⇒ `Array`- - [`.getClientSample(sampleDurationMillis)`](#Snapshotter+getClientSample) ⇒ `Array`+----<a name="new_Snapshotter_new"></a>+<a name="snapshotter"></a>## `new Snapshotter([options])`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code><a href="../typedefs/snapshotteroptions">SnapshotterOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>Snapshotter</code> configuration options.</p>-</td></tr></tbody>-</table>-<a name="Snapshotter+start"></a>--## `snapshotter.start()` ⇒ `Promise`+**Params**++- **`[options]`**: [`SnapshotterOptions`](/docs/typedefs/snapshotter-options) - All `Snapshotter` configuration options.++---++<a name="start"></a>++## `snapshotter.start()`++**Returns**: `Promise`Starts capturing snapshots at configured intervals.-<a name="Snapshotter+stop"></a>+---++<a name="stop"></a>-## `snapshotter.stop()` ⇒ `Promise`+## `snapshotter.stop()`++**Returns**: `Promise`Stops all resource capturing.-<a name="Snapshotter+getMemorySample"></a>+---++<a name="getmemorysample"></a>++## `snapshotter.getMemorySample([sampleDurationMillis])`-## `snapshotter.getMemorySample([sampleDurationMillis])` ⇒ `Array`+**Returns**: `Array`Returns a sample of latest memory snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a fullsnapshot history.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[sampleDurationMillis]</code></td><td><code>Number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Snapshotter+getEventLoopSample"></a>--## `snapshotter.getEventLoopSample([sampleDurationMillis])` ⇒ `Array`+**Params**++- **`[sampleDurationMillis]`**: `Number`++---++<a name="geteventloopsample"></a>++## `snapshotter.getEventLoopSample([sampleDurationMillis])`++**Returns**: `Array`Returns a sample of latest event loop snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns afull snapshot history.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[sampleDurationMillis]</code></td><td><code>Number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Snapshotter+getCpuSample"></a>--## `snapshotter.getCpuSample([sampleDurationMillis])` ⇒ `Array`+**Params**++- **`[sampleDurationMillis]`**: `Number`++---++<a name="getcpusample"></a>++## `snapshotter.getCpuSample([sampleDurationMillis])`++**Returns**: `Array`Returns a sample of latest CPU snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a fullsnapshot history.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[sampleDurationMillis]</code></td><td><code>Number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="Snapshotter+getClientSample"></a>--## `snapshotter.getClientSample(sampleDurationMillis)` ⇒ `Array`+**Params**++- **`[sampleDurationMillis]`**: `Number`++---++<a name="getclientsample"></a>++## `snapshotter.getClientSample(sampleDurationMillis)`++**Returns**: `Array`Returns a sample of latest Client snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a fullsnapshot history.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>sampleDurationMillis</code></td><td><code>Number</code></td>-</tr>-<tr>-</tr></tbody>-</table>+**Params**++- **`sampleDurationMillis`**: `Number`++---diff --git a/docs/api/SystemStatus.md b/docs/api/SystemStatus.mdindex 599de99d0..0a503d5ee 100644--- a/docs/api/SystemStatus.md+++ b/docs/api/SystemStatus.md@@ -1,50 +1,42 @@----id: systemstatus+id: system-statustitle: SystemStatus----<a name="SystemStatus"></a>+<a name="systemstatus"></a>-Provides a simple interface to reading system status from a [`Snapshotter`](snapshotter) instance. It only exposes two functions-[`getCurrentStatus()`](#SystemStatus+getCurrentStatus) and [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus). The system status is-calculated using a weighted average of overloaded messages in the snapshots, with the weights being the time intervals between the snapshots. Each-resource is calculated separately and the system is overloaded whenever at least one resource is overloaded. The class is used by the-[`AutoscaledPool`](autoscaledpool) class.+Provides a simple interface to reading system status from a [`Snapshotter`](/docs/api/snapshotter) instance. It only exposes two functions+[`SystemStatus.getCurrentStatus()`](/docs/api/system-status#getcurrentstatus) and+[`SystemStatus.getHistoricalStatus()`](/docs/api/system-status#gethistoricalstatus). The system status is calculated using a weighted average of+overloaded messages in the snapshots, with the weights being the time intervals between the snapshots. Each resource is calculated separately and the+system is overloaded whenever at least one resource is overloaded. The class is used by the [`AutoscaledPool`](/docs/api/autoscaled-pool) class.-[`getCurrentStatus()`](#SystemStatus+getCurrentStatus) returns a boolean that represents the current status of the system. The length of the current-timeframe in seconds is configurable by the `currentHistorySecs` option and represents the max age of snapshots to be considered for the calculation.+[`SystemStatus.getCurrentStatus()`](/docs/api/system-status#getcurrentstatus) returns a boolean that represents the current status of the system. The+length of the current timeframe in seconds is configurable by the `currentHistorySecs` option and represents the max age of snapshots to be considered+for the calculation.-[`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus) returns a boolean that represents the long-term status of the system. It considers the-full snapshot history available in the [`Snapshotter`](snapshotter) instance.+[`SystemStatus.getHistoricalStatus()`](/docs/api/system-status#gethistoricalstatus) returns a boolean that represents the long-term status of the+system. It considers the full snapshot history available in the [`Snapshotter`](/docs/api/snapshotter) instance.-- [SystemStatus](systemstatus)- - [`new SystemStatus([options])`](#new_SystemStatus_new)- - [`.getCurrentStatus()`](#SystemStatus+getCurrentStatus) ⇒ [`SystemInfo`](../typedefs/systeminfo)- - [`.getHistoricalStatus()`](#SystemStatus+getHistoricalStatus) ⇒ `Object`+----<a name="new_SystemStatus_new"></a>+<a name="systemstatus"></a>## `new SystemStatus([options])`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[options]</code></td><td><code><a href="../typedefs/systemstatusoptions">SystemStatusOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>SystemStatus</code> configuration options.</p>-</td></tr></tbody>-</table>-<a name="SystemStatus+getCurrentStatus"></a>--## `systemStatus.getCurrentStatus()` ⇒ [`SystemInfo`](../typedefs/systeminfo)--Returns an [`SystemInfo`](../typedefs/systeminfo) object with the following structure:+**Params**++- **`[options]`**: [`SystemStatusOptions`](/docs/typedefs/system-status-options) - All `SystemStatus` configuration options.++---++<a name="getcurrentstatus"></a>++## `systemStatus.getCurrentStatus()`++**Returns**: [`SystemInfo`](/docs/typedefs/system-info)++Returns an [`SystemInfo`](/docs/typedefs/system-info) object with the following structure:```javascript{@@ -58,11 +50,15 @@ Returns an [`SystemInfo`](../typedefs/systeminfo) object with the following struWhere the `isSystemIdle` property is set to `false` if the system has been overloaded in the last `options.currentHistorySecs` seconds, and `true`otherwise.-<a name="SystemStatus+getHistoricalStatus"></a>+---++<a name="gethistoricalstatus"></a>-## `systemStatus.getHistoricalStatus()` ⇒ `Object`+## `systemStatus.getHistoricalStatus()`-Returns an [`SystemInfo`](../typedefs/systeminfo) object with the following structure:+**Returns**: `Object`++Returns an [`SystemInfo`](/docs/typedefs/system-info) object with the following structure:```javascript{@@ -73,5 +69,7 @@ Returns an [`SystemInfo`](../typedefs/systeminfo) object with the following stru}```-Where the `isSystemIdle` property is set to `false` if the system has been overloaded in the full history of the [`Snapshotter`](snapshotter) (which-is configurable in the [`Snapshotter`](snapshotter)) and `true` otherwise.+Where the `isSystemIdle` property is set to `false` if the system has been overloaded in the full history of the+[`Snapshotter`](/docs/api/snapshotter) (which is configurable in the [`Snapshotter`](/docs/api/snapshotter)) and `true` otherwise.++---diff --git a/docs/api/log.md b/docs/api/log.mdindex fc6e3e2ae..a417c67f4 100644--- a/docs/api/log.md+++ b/docs/api/log.md@@ -46,24 +46,18 @@ try {Another very useful way of setting the log level is by setting the `APIFY_LOG_LEVEL` environment variable, such as `APIFY_LOG_LEVEL=DEBUG`. This way,no code changes are necessary to turn on your debug messages and start debugging right away.-- [`log`](#log) : `object`- - [`.LEVELS`](#log.LEVELS) : `Object`- - [`.setLevel(level)`](#log.setLevel)- - [`.getLevel()`](#log.getLevel)- - [`.debug(message, [data])`](#log.debug)- - [`.info(message, [data])`](#log.info)- - [`.warning(message, [data])`](#log.warning)- - [`.error(message, [data])`](#log.error)- - [`.exception(exception, [message], [data])`](#log.exception)+----<a name="log.LEVELS"></a>+<a name="levels"></a>-## `log.LEVELS` : `Object`+## `log.LEVELS`Map of available log levels that's useful for easy setting of appropriate log levels. Each log level is represented internally by a number. Eg.`log.LEVELS.DEBUG === 5`.-<a name="log.setLevel"></a>+---++<a name="setlevel"></a>## `log.setLevel(level)`@@ -76,20 +70,13 @@ log.setLevel(log.LEVELS.DEBUG);Default log level is INFO.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>level</code></td><td><code>number</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="log.getLevel"></a>+**Params**++- **`level`**: `number`++---++<a name="getlevel"></a>## `log.getLevel()`@@ -97,7 +84,9 @@ Returns the currently selected logging level. This is useful for checking whetheactually performs a resource intensive operation to construct the message, such as querying a DB for some metadata that need to be added. If the loglevel is not high enough at the moment, it doesn't make sense to execute the query.-<a name="log.debug"></a>+---++<a name="debug"></a>## `log.debug(message, [data])`@@ -105,119 +94,65 @@ Logs a `DEBUG` message. By default, it will not be written to the console. To seusing the `log.setLevel(log.LEVELS.DEBUG)` method or using the environment variable `APIFY_LOG_LEVEL=DEBUG`. Data are stringified and appended to themessage.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>message</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>[data]</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="log.info"></a>+**Params**++- **`message`**: `string`+- **`[data]`**: `Object`++---++<a name="info"></a>## `log.info(message, [data])`Logs an `INFO` message. `INFO` is the default log level so info messages will be always logged, unless the log level is changed. Data are stringifiedand appended to the message.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>message</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>[data]</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="log.warning"></a>+**Params**++- **`message`**: `string`+- **`[data]`**: `Object`++---++<a name="warning"></a>## `log.warning(message, [data])`Logs a `WARNING` level message. Data are stringified and appended to the message.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>message</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>[data]</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="log.error"></a>+**Params**++- **`message`**: `string`+- **`[data]`**: `Object`++---++<a name="error"></a>## `log.error(message, [data])`Logs an `ERROR` message. Use this method to log error messages that are not directly connected to an exception. For logging exceptions, use the`log.exception` method.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>message</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>[data]</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="log.exception"></a>+**Params**++- **`message`**: `string`+- **`[data]`**: `Object`++---++<a name="exception"></a>## `log.exception(exception, [message], [data])`Logs an `ERROR` level message with a nicely formatted exception. Note that the exception is the first parameter here and an additional message is onlyoptional.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>exception</code></td><td><code>Error</code></td>-</tr>-<tr>-</tr><tr>-<td><code>[message]</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>[data]</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>+**Params**++- **`exception`**: `Error`+- **`[message]`**: `string`+- **`[data]`**: `Object`++---diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 8209c096a..9dc326b30 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -22,152 +22,13 @@ await page.goto('https://www.example.com');await puppeteer.injectJQuery(page);```-- [`puppeteer`](#puppeteer) : `object`- - [`.enqueueLinksByClickingElements`](#puppeteer.enqueueLinksByClickingElements) ⇒ `Promise<Array<QueueOperationInfo>>`- - [`.addInterceptRequestHandler`](#puppeteer.addInterceptRequestHandler) ⇒ `Promise`- - [`.removeInterceptRequestHandler`](#puppeteer.removeInterceptRequestHandler) ⇒ `Promise`- - [`.gotoExtended`](#puppeteer.gotoExtended) ⇒ `Promise<Response>`- - [`.infiniteScroll`](#puppeteer.infiniteScroll) ⇒ `Promise`- - [`.saveSnapshot`](#puppeteer.saveSnapshot) ⇒ `Promise`- - [`.injectFile(page, filePath, [options])`](#puppeteer.injectFile) ⇒ `Promise`- - [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ `Promise`- - [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ `Promise`- - [`.blockRequests(page, [options])`](#puppeteer.blockRequests) ⇒ `Promise`- - ~~[`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ `Promise`~~- - [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ `function`--<a name="puppeteer.enqueueLinksByClickingElements"></a>--## `puppeteer.enqueueLinksByClickingElements` ⇒ `Promise<Array<QueueOperationInfo>>`--The function finds elements matching a specific CSS selector in a Puppeteer page, clicks all those elements using a mouse move and a left mouse button-click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods,-headers and payloads are then enqueued to a provided [`RequestQueue`](requestqueue). This is useful to crawl JavaScript heavy pages where links are-not available in `href` elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in `href` attributes of the-page, see [`enqueueLinks()`](utils#utils.enqueueLinks).--Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](pseudourl) objects and override settings of the-enqueued [`Request`](request) objects.--**IMPORTANT**: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and-their visibility. Therefore, it is recommended to only use this function as the last operation in the page.--**USING HEADFUL BROWSER**: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting-concurrency to 1. In headless mode, full concurrency can be achieved.--**PERFORMANCE**: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU-intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets-only the elements that you assume or know will produce a navigation. You can certainly click everything by using the `*` selector, but be prepared to-wait minutes to get results on a large and complex page.--**Example usage**--```javascript-const Apify = require('apify');+----const browser = await Apify.launchPuppeteer();-const page = await browser.goto('https://www.example.com');-const requestQueue = await Apify.openRequestQueue();+<a name="addinterceptrequesthandler"></a>-await Apify.utils.enqueueLinksByClickingElements({- page,- requestQueue,- selector: 'a.product-detail',- pseudoUrls: [- 'https://www.example.com/handbags/[.*]'- 'https://www.example.com/purses/[.*]'- ],-});-```+## `puppeteer.addInterceptRequestHandler`-**Returns**: `Promise<Array<QueueOperationInfo>>` - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo)-objects.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>enqueueLinksByClickingElements()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>options.page</code></td><td><code>Page</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A request queue to which the URLs will be enqueued.</p>-</td></tr><tr>-<td><code>options.selector</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A CSS selector matching elements to be clicked on. Unlike in <a href="utils#utils.enqueueLinks"><code>enqueueLinks()</code></a>, there is no default- value. This is to prevent suboptimal use of this function by using it too broadly.</p>-</td></tr><tr>-<td><code>[options.pseudoUrls]</code></td><td><code>Array<(String|RegExp|Object)></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,- or an array of strings or RegExps or plain Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.</p>-<p> The plain objects must include at least the <code>purl</code> property, which holds the pseudo-URL string or RegExp.- All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor,- which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p>-<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function- enqueues all links found on the page.</p>-</td></tr><tr>-<td><code>[options.transformRequestFunction]</code></td><td><code><a href="../typedefs/requesttransform">RequestTransform</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Just before a new <a href="request"><code>Request</code></a> is constructed and enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used- to remove it or modify its contents such as <code>userData</code>, <code>payload</code> or, most importantly <code>uniqueKey</code>. This is useful- when you need to enqueue multiple <code>Requests</code> to the queue that share the same URL, but differ in methods or payloads,- or to dynamically update or create <code>userData</code>.</p>-<p> For example: by adding <code>useExtendedUniqueKey: true</code> to the <code>request</code> object, <code>uniqueKey</code> will be computed from- a combination of <code>url</code>, <code>method</code> and <code>payload</code> which enables crawling of websites that navigate using form submits- (POST requests).</p>-<p> <strong>Example:</strong></p>-<pre><code class="language-javascript"> {- transformRequestFunction: (request) => {- request.userData.foo = 'bar';- request.useExtendedUniqueKey = true;- return request;- }- }</code></pre>-</td></tr><tr>-<td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Clicking in the page triggers various asynchronous operations that lead to new URLs being shown- by the browser. It could be a simple JavaScript redirect or opening of a new tab in the browser.- These events often happen only some time after the actual click. Requests typically take milliseconds- while new tabs open in hundreds of milliseconds.</p>-<p> To be able to capture all those events, the <code>enqueueLinksByClickingElements()</code> function repeatedly waits- for the <code>waitForPageIdleSecs</code>. By repeatedly we mean that whenever a relevant event is triggered, the timer- is restarted. As long as new events keep coming, the function will not return, unless- the below <code>maxWaitForPageIdleSecs</code> timeout is reached.</p>-<p> You may want to reduce this for example when you're sure that your clicks do not open new tabs,- or increase when you're not getting all the expected URLs.</p>-</td></tr><tr>-<td><code>[options.maxWaitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>5</code></td>-</tr>-<tr>-<td colspan="3"><p>This is the maximum period for which the function will keep tracking events, even if more events keep coming.- Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself.- See <code>waitForPageIdleSecs</code> above for an explanation.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.addInterceptRequestHandler"></a>--## `puppeteer.addInterceptRequestHandler` ⇒ `Promise`+**Returns**: `Promise`Adds request interception handler in similar to `page.on('request', handler);` but in addition to that supports multiple parallel handlers.@@ -209,53 +70,33 @@ await addInterceptRequestHandler(page, request => {await page.goto('http://example.com');```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>handler</code></td><td><code>function</code></td>-</tr>-<tr>-<td colspan="3"><p>Request interception handler.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.removeInterceptRequestHandler"></a>--## `puppeteer.removeInterceptRequestHandler` ⇒ `Promise`+**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`handler`**: `function` - Request interception handler.++---++<a name="removeinterceptrequesthandler"></a>++## `puppeteer.removeInterceptRequestHandler`++**Returns**: `Promise`Removes request interception handler for given page.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>handler</code></td><td><code>function</code></td>-</tr>-<tr>-<td colspan="3"><p>Request interception handler.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.gotoExtended"></a>--## `puppeteer.gotoExtended` ⇒ `Promise<Response>`+**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`handler`**: `function` - Request interception handler.++---++<a name="gotoextended"></a>++## `puppeteer.gotoExtended`++**Returns**: `Promise<Response>`Extended version of Puppeteer's `page.goto()` allowing to perform requests with HTTP method other than GET, with custom headers and POST payload. URL,method, headers and payload are taken from request parameter that must be an instance of Apify.Request class.@@ -263,157 +104,188 @@ method, headers and payload are taken from request parameter that must be an ins_NOTE:_ In recent versions of Puppeteer using requests other than GET, overriding headers and adding payloads disables browser cache which degradesperformance.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>gotoOptions</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Custom options for <code>page.goto()</code>.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.infiniteScroll"></a>--## `puppeteer.infiniteScroll` ⇒ `Promise`+**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`request`**: [`Request`](/docs/api/request)+- **`gotoOptions`**: `Object` - Custom options for `page.goto()`.++---++<a name="infinitescroll"></a>++## `puppeteer.infiniteScroll`++**Returns**: `Promise`Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td><td><code>0</code></td>-</tr>-<tr>-<td colspan="3"><p>How many seconds to scroll for. If 0, will scroll until bottom of page.</p>-</td></tr><tr>-<td><code>[options.waitForSecs]</code></td><td><code>Number</code></td><td><code>4</code></td>-</tr>-<tr>-<td colspan="3"><p>How many seconds to wait for no new content to load before exit.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.saveSnapshot"></a>--## `puppeteer.saveSnapshot` ⇒ `Promise`+**Params**++- **`page`**: `Object` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`[options]`**: `Object`+ - **`[.timeoutSecs]`**: `Number` <code> = 0</code> - How many seconds to scroll for. If 0, will scroll until bottom of page.+ - **`[.waitForSecs]`**: `Number` <code> = 4</code> - How many seconds to wait for no new content to load before exit.++---++<a name="savesnapshot"></a>++## `puppeteer.saveSnapshot`++**Returns**: `Promise`Saves a full screenshot and HTML of the current page into a Key-Value store.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.key]</code></td><td><code>String</code></td><td><code>SNAPSHOT</code></td>-</tr>-<tr>-<td colspan="3"><p>Key under which the screenshot and HTML will be saved. <code>.jpg</code> will be appended for screenshot and <code>.html</code> for HTML.</p>-</td></tr><tr>-<td><code>[options.screenshotQuality]</code></td><td><code>Number</code></td><td><code>50</code></td>-</tr>-<tr>-<td colspan="3"><p>The quality of the image, between 0-100. Higher quality images have bigger size and require more storage.</p>-</td></tr><tr>-<td><code>[options.saveScreenshot]</code></td><td><code>Boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>If true, it will save a full screenshot of the current page as a record with <code>key</code> appended by <code>.jpg</code>.</p>-</td></tr><tr>-<td><code>[options.saveHtml]</code></td><td><code>Boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>If true, it will save a full HTML of the current page as a record with <code>key</code> appended by <code>.html</code>.</p>-</td></tr><tr>-<td><code>[options.keyValueStoreName]</code></td><td><code>String</code></td><td><code></code></td>-</tr>-<tr>-<td colspan="3"><p>Name or id of the Key-Value store where snapshot is saved. By default it is saved to default Key-Value store.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.injectFile"></a>--## `puppeteer.injectFile(page, filePath, [options])` ⇒ `Promise`+**Params**++- **`page`**: `Object` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`[options]`**: `Object`+ - **`[.key]`**: `String` <code> = SNAPSHOT</code> - Key under which the screenshot and HTML will be saved. `.jpg` will be appended for+ screenshot and `.html` for HTML.+ - **`[.screenshotQuality]`**: `Number` <code> = 50</code> - The quality of the image, between 0-100. Higher quality images have bigger size and+ require more storage.+ - **`[.saveScreenshot]`**: `Boolean` <code> = true</code> - If true, it will save a full screenshot of the current page as a record with `key`+ appended by `.jpg`.+ - **`[.saveHtml]`**: `Boolean` <code> = true</code> - If true, it will save a full HTML of the current page as a record with `key` appended by+ `.html`.+ - **`[.keyValueStoreName]`**: `String` <code> = </code> - Name or id of the Key-Value store where snapshot is saved. By default it is saved to+ default Key-Value store.++---++<a name="enqueuelinksbyclickingelements"></a>++## `puppeteer.enqueueLinksByClickingElements(options)`++**Returns**: [`Promise<Array<QueueOperationInfo>>`](/docs/typedefs/queue-operation-info) - Promise that resolves to an array of+[`QueueOperationInfo`](/docs/typedefs/queue-operation-info) objects.++The function finds elements matching a specific CSS selector in a Puppeteer page, clicks all those elements using a mouse move and a left mouse button+click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods,+headers and payloads are then enqueued to a provided [`RequestQueue`](/docs/api/request-queue). This is useful to crawl JavaScript heavy pages where+links are not available in `href` elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in `href`+attributes of the page, see [`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks).++Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](/docs/api/pseudo-url) objects and override+settings of the enqueued [`Request`](/docs/api/request) objects.++**IMPORTANT**: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and+their visibility. Therefore, it is recommended to only use this function as the last operation in the page.++**USING HEADFUL BROWSER**: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting+concurrency to 1. In headless mode, full concurrency can be achieved.++**PERFORMANCE**: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU+intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets+only the elements that you assume or know will produce a navigation. You can certainly click everything by using the `*` selector, but be prepared to+wait minutes to get results on a large and complex page.++**Example usage**++```javascript+const Apify = require('apify');++const browser = await Apify.launchPuppeteer();+const page = await browser.goto('https://www.example.com');+const requestQueue = await Apify.openRequestQueue();++await Apify.utils.enqueueLinksByClickingElements({+ page,+ requestQueue,+ selector: 'a.product-detail',+ pseudoUrls: [+ 'https://www.example.com/handbags/[.*]'+ 'https://www.example.com/purses/[.*]'+ ],+});+```++**Params**++- **`options`**: `Object` - All `enqueueLinksByClickingElements()` parameters are passed via an options object with the following keys:++ - **`.page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ - **`.requestQueue`**: [`RequestQueue`](/docs/api/request-queue) - A request queue to which the URLs will be enqueued.+ - **`.selector`**: `String` - A CSS selector matching elements to be clicked on. Unlike in+ [`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks), there is no default value. This is to prevent suboptimal use of this function by using+ it too broadly.+ - **`[.pseudoUrls]`**: `Array<(String|RegExp|Object)>` - An array of [`PseudoUrl`](/docs/api/pseudo-url)s matching the URLs to be enqueued, or+ an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](/docs/api/pseudo-url)s can be constructed.++ The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the+ `requestTemplate` argument of the [`PseudoUrl`](/docs/api/pseudo-url) constructor, which lets you specify special properties for the enqueued+ [`Request`](/docs/api/request) objects.++ If `pseudoUrls` is an empty array, `null` or `undefined`, then the function enqueues all links found on the page.++ - **`[.transformRequestFunction]`**: [`RequestTransform`](/docs/typedefs/request-transform) - Just before a new [`Request`](/docs/api/request)+ is constructed and enqueued to the [`RequestQueue`](/docs/api/request-queue), this function can be used to remove it or modify its contents+ such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful when you need to enqueue multiple `Requests` to the queue that+ share the same URL, but differ in methods or payloads, or to dynamically update or create `userData`.++ For example: by adding `useExtendedUniqueKey: true` to the `request` object, `uniqueKey` will be computed from a combination of `url`, `method`+ and `payload` which enables crawling of websites that navigate using form submits (POST requests).++ **Example:**++ ```javascript+ {+ transformRequestFunction: request => {+ request.userData.foo = 'bar';+ request.useExtendedUniqueKey = true;+ return request;+ };+ }+ ```++ - **`[.waitForPageIdleSecs]`**: `number` <code> = 1</code> - Clicking in the page triggers various asynchronous operations that lead to new URLs+ being shown by the browser. It could be a simple JavaScript redirect or opening of a new tab in the browser. These events often happen only+ some time after the actual click. Requests typically take milliseconds while new tabs open in hundreds of milliseconds.++ To be able to capture all those events, the `enqueueLinksByClickingElements()` function repeatedly waits for the `waitForPageIdleSecs`. By+ repeatedly we mean that whenever a relevant event is triggered, the timer is restarted. As long as new events keep coming, the function will not+ return, unless the below `maxWaitForPageIdleSecs` timeout is reached.++ You may want to reduce this for example when you're sure that your clicks do not open new tabs, or increase when you're not getting all the+ expected URLs.++ - **`[.maxWaitForPageIdleSecs]`**: `number` <code> = 5</code> - This is the maximum period for which the function will keep tracking events,+ even if more events keep coming. Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself.+ See `waitForPageIdleSecs` above for an explanation.++---++<a name="injectfile"></a>++## `puppeteer.injectFile(page, filePath, [options])`++**Returns**: `Promise`Injects a JavaScript file into a Puppeteer page. Unlike Puppeteer's `addScriptTag` function, this function works on pages with arbitrary Cross-OriginResource Sharing (CORS) policies.File contents are cached for up to 10 files to limit file system access.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>filePath</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>File path</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.surviveNavigations]</code></td><td><code>boolean</code></td>-</tr>-<tr>-<td colspan="3"><p>Enables the injected script to survive page navigations and reloads without need to be re-injected manually.- This does not mean, however, that internal state will be preserved. Just that it will be automatically- re-injected on each navigation before any other scripts get the chance to execute.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.injectJQuery"></a>--## `puppeteer.injectJQuery(page)` ⇒ `Promise`--Injects the <a href="https://jquery.com/" target="_blank"><code>jQuery</code></a> library into a Puppeteer page. jQuery is often useful for various-web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors.+**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`filePath`**: `String` - File path+- **`[options]`**: `Object`+ - **`[.surviveNavigations]`**: `boolean` - Enables the injected script to survive page navigations and reloads without need to be re-injected+ manually. This does not mean, however, that internal state will be preserved. Just that it will be automatically re-injected on each+ navigation before any other scripts get the chance to execute.++---++<a name="injectjquery"></a>++## `puppeteer.injectJQuery(page)`++**Returns**: `Promise`++Injects the [](https://jquery.com/) library into a Puppeteer page. jQuery is often useful for various web scraping and crawling tasks. For example, it+can help extract text from HTML elements using CSS selectors.Beware that the injected jQuery object will be set to the `window.$` variable and thus it might cause conflicts with other libraries included by thepage that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.@@ -429,28 +301,21 @@ const title = await page.evaluate(() => {});```-Note that `injectJQuery()` does not affect the Puppeteer's-<a href="https://pptr.dev/#?product=Puppeteer&show=api-pageselector" target="_blank"><code>Page.\$()</code></a> function in any way.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.injectUnderscore"></a>--## `puppeteer.injectUnderscore(page)` ⇒ `Promise`--Injects the <a href="https://underscorejs.org/" target="_blank"><code>Underscore.js</code></a> library into a Puppeteer page.+Note that `injectJQuery()` does not affect the Puppeteer's [](https://pptr.dev/#?product=Puppeteer&show=api-pageselector) function in any way.++**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.++---++<a name="injectunderscore"></a>++## `puppeteer.injectUnderscore(page)`++**Returns**: `Promise`++Injects the [](https://underscorejs.org/) library into a Puppeteer page.Beware that the injected Underscore object will be set to the `window._` variable and thus it might cause conflicts with libraries included by thepage that use the same variable name. This can affect functionality of page's scripts.@@ -466,23 +331,17 @@ const escapedHtml = await page.evaluate(() => {});```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page">Page</a> object.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.blockRequests"></a>--## `puppeteer.blockRequests(page, [options])` ⇒ `Promise`+**Params**++- **`page`**: `Page` - Puppeteer [Page](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page) object.++---++<a name="blockrequests"></a>++## `puppeteer.blockRequests(page, [options])`++**Returns**: `Promise`Forces the Puppeteer browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since itreduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.@@ -519,43 +378,25 @@ await Apify.utils.puppeteer.blockRequests(page, {await page.goto('https://cnn.com');```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>[options]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[options.urlPatterns]</code></td><td><code>Array<string></code></td>-</tr>-<tr>-<td colspan="3"><p>The patterns of URLs to block from being loaded by the browser.- Only <code>*</code> can be used as a wildcard. It is also automatically added to the beginning- and end of the pattern. This limitation is enforced by the DevTools protocol.- <code>.png</code> is the same as <code>*.png*</code>.</p>-</td></tr><tr>-<td><code>[options.extraUrlPatterns]</code></td><td><code>Array<string></code></td>-</tr>-<tr>-<td colspan="3"><p>If you just want to append to the default blocked patterns, use this property.</p>-</td></tr></tbody>-</table>-<a name="puppeteer.cacheResponses"></a>--## ~~`puppeteer.cacheResponses(page, cache, responseUrlRules)` ⇒ `Promise`~~+**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`[options]`**: `Object`+ - **`[.urlPatterns]`**: `Array<string>` - The patterns of URLs to block from being loaded by the browser. Only `*` can be used as a wildcard. It+ is also automatically added to the beginning and end of the pattern. This limitation is enforced by the DevTools protocol. `.png` is the same+ as `*.png*`.+ - **`[.extraUrlPatterns]`**: `Array<string>` - If you just want to append to the default blocked patterns, use this property.++---++<a name="cacheresponses"></a>++## ~~`puppeteer.cacheResponses(page, cache, responseUrlRules)`~~**_Deprecated_**+**Returns**: `Promise`+_NOTE:_ In recent versions of Puppeteer using this function entirely disables browser cache which resolves in sub-optimal performance. Until thisresolves, we suggest just relying on the in-browser cache unless absolutely necessary.@@ -563,34 +404,20 @@ Enables caching of intercepted responses into a provided object. Automatically eresponses stores them to memory, so too loose rules could cause memory leaks for longer running crawlers. This issue should be resolved or atleastmitigated in future iterations of this feature.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>cache</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Object in which responses are stored</p>-</td></tr><tr>-<td><code>responseUrlRules</code></td><td><code>Array<(String|RegExp)></code></td>-</tr>-<tr>-<td colspan="3"><p>List of rules that are used to check if the response should be cached.- String rules are compared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).</p>-</td></tr></tbody>-</table>-<a name="puppeteer.compileScript"></a>--## `puppeteer.compileScript(scriptString, context)` ⇒ `function`+**Params**++- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`cache`**: `Object` - Object in which responses are stored+- **`responseUrlRules`**: `Array<(String|RegExp)>` - List of rules that are used to check if the response should be cached. String rules are+ compared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).++---++<a name="compilescript"></a>++## `puppeteer.compileScript(scriptString, context)`++**Returns**: `function` - `async ({ page, request }) => { scriptString }`Compiles a Puppeteer script into an async function that may be executed at any time by providing it with the following object:@@ -601,8 +428,7 @@ Compiles a Puppeteer script into an async function that may be executed at any t}```-Where `page` is a Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> and `request` is-a [`Request`](request).+Where `page` is a Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) and `request` is a [`Request`](/docs/api/request).The function is compiled by using the `scriptString` parameter as the function's body, so any limitations to function bodies apply. Return value ofthe compiled function is the return value of the function body = the `scriptString` parameter.@@ -614,22 +440,9 @@ Therefore you should only use this function to execute sanitized or safe code.Custom context may also be provided using the `context` parameter. To improve security, make sure to only pass the really necessary objects to thecontext. Preferably making secured copies beforehand.-**Returns**: `function` - `async ({ page, request }) => { scriptString }`+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>scriptString</code></td><td><code>String</code></td>-</tr>-<tr>-</tr><tr>-<td><code>context</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr></tbody>-</table>+- **`scriptString`**: `String`+- **`context`**: `Object`++---diff --git a/docs/api/social.md b/docs/api/social.mdindex a8e138981..4b2a8a658 100644--- a/docs/api/social.md+++ b/docs/api/social.md@@ -15,28 +15,11 @@ const Apify = require('apify');const emails = Apify.utils.social.emailsFromText('[email protected] [email protected]');```-- [`social`](#social) : `object`- - [`.LINKEDIN_REGEX`](#social.LINKEDIN_REGEX) : `RegExp`- - [`.LINKEDIN_REGEX_GLOBAL`](#social.LINKEDIN_REGEX_GLOBAL) : `RegExp`- - [`.INSTAGRAM_REGEX`](#social.INSTAGRAM_REGEX) : `RegExp`- - [`.INSTAGRAM_REGEX_GLOBAL`](#social.INSTAGRAM_REGEX_GLOBAL) : `RegExp`- - [`.TWITTER_REGEX`](#social.TWITTER_REGEX) : `RegExp`- - [`.TWITTER_REGEX_GLOBAL`](#social.TWITTER_REGEX_GLOBAL) : `RegExp`- - [`.FACEBOOK_REGEX`](#social.FACEBOOK_REGEX) : `RegExp`- - [`.FACEBOOK_REGEX_GLOBAL`](#social.FACEBOOK_REGEX_GLOBAL) : `RegExp`- - [`.YOUTUBE_REGEX`](#social.YOUTUBE_REGEX) : `RegExp`- - [`.YOUTUBE_REGEX_GLOBAL`](#social.YOUTUBE_REGEX_GLOBAL) : `RegExp`- - [`.EMAIL_REGEX`](#social.EMAIL_REGEX) : `RegExp`- - [`.EMAIL_REGEX_GLOBAL`](#social.EMAIL_REGEX_GLOBAL) : `RegExp`- - [`.emailsFromText(text)`](#social.emailsFromText) ⇒ `Array<String>`- - [`.emailsFromUrls(urls)`](#social.emailsFromUrls) ⇒ `Array<String>`- - [`.phonesFromText(text)`](#social.phonesFromText) ⇒ `Array<String>`- - [`.phonesFromUrls(urls)`](#social.phonesFromUrls) ⇒ `Array<String>`- - [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ [`SocialHandles`](../typedefs/socialhandles)--<a name="social.LINKEDIN_REGEX"></a>--## `social.LINKEDIN_REGEX` : `RegExp`+---++<a name="linkedin_regex"></a>++## `social.LINKEDIN_REGEX`Regular expression to exactly match a single LinkedIn profile URL. It has the following form: `/^...$/i` and matches URLs such as:@@ -60,9 +43,11 @@ if (Apify.utils.social.LINKEDIN_REGEX.test('https://www.linkedin.com/in/alan-tur}```-<a name="social.LINKEDIN_REGEX_GLOBAL"></a>+---++<a name="linkedin_regex_global"></a>-## `social.LINKEDIN_REGEX_GLOBAL` : `RegExp`+## `social.LINKEDIN_REGEX_GLOBAL`Regular expression to find multiple LinkedIn profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:@@ -92,9 +77,11 @@ const matches = text.match(Apify.utils.social.LINKEDIN_REGEX_GLOBAL);if (matches) console.log(`${matches.length} LinkedIn profiles found!`);```-<a name="social.INSTAGRAM_REGEX"></a>+---++<a name="instagram_regex"></a>-## `social.INSTAGRAM_REGEX` : `RegExp`+## `social.INSTAGRAM_REGEX`Regular expression to exactly match a single Instagram profile URL. It has the following form: `/^...$/i` and matches URLs such as:@@ -118,9 +105,11 @@ if (Apify.utils.social.INSTAGRAM_REGEX.test('https://www.instagram.com/old_pragu}```-<a name="social.INSTAGRAM_REGEX_GLOBAL"></a>+----## `social.INSTAGRAM_REGEX_GLOBAL` : `RegExp`+<a name="instagram_regex_global"></a>++## `social.INSTAGRAM_REGEX_GLOBAL`Regular expression to find multiple Instagram profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:@@ -150,9 +139,11 @@ const matches = text.match(Apify.utils.social.INSTAGRAM_REGEX_GLOBAL);if (matches) console.log(`${matches.length} Instagram profiles found!`);```-<a name="social.TWITTER_REGEX"></a>+---++<a name="twitter_regex"></a>-## `social.TWITTER_REGEX` : `RegExp`+## `social.TWITTER_REGEX`Regular expression to exactly match a single Twitter profile URL. It has the following form: `/^...$/i` and matches URLs such as:@@ -175,9 +166,11 @@ if (Apify.utils.social.TWITTER_REGEX.test('https://www.twitter.com/apify')) {}```-<a name="social.TWITTER_REGEX_GLOBAL"></a>+----## `social.TWITTER_REGEX_GLOBAL` : `RegExp`+<a name="twitter_regex_global"></a>++## `social.TWITTER_REGEX_GLOBAL`Regular expression to find multiple Twitter profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:@@ -206,9 +199,11 @@ const matches = text.match(Apify.utils.social.TWITTER_REGEX_STRING);if (matches) console.log(`${matches.length} Twitter profiles found!`);```-<a name="social.FACEBOOK_REGEX"></a>+---++<a name="facebook_regex"></a>-## `social.FACEBOOK_REGEX` : `RegExp`+## `social.FACEBOOK_REGEX`Regular expression to exactly match a single Facebook profile URL. It has the following form: `/^...$/i` and matches URLs such as:@@ -233,9 +228,11 @@ if (Apify.utils.social.FACEBOOK_REGEX.test('https://www.facebook.com/apifytech')}```-<a name="social.FACEBOOK_REGEX_GLOBAL"></a>+---++<a name="facebook_regex_global"></a>-## `social.FACEBOOK_REGEX_GLOBAL` : `RegExp`+## `social.FACEBOOK_REGEX_GLOBAL`Regular expression to find multiple Facebook profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:@@ -265,9 +262,11 @@ const matches = text.match(Apify.utils.social.FACEBOOK_REGEX_GLOBAL);if (matches) console.log(`${matches.length} Facebook profiles found!`);```-<a name="social.YOUTUBE_REGEX"></a>+---++<a name="youtube_regex"></a>-## `social.YOUTUBE_REGEX` : `RegExp`+## `social.YOUTUBE_REGEX`Regular expression to exactly match a single Youtube video URL. It has the following form: `/^...$/i` and matches URLs such as:@@ -284,9 +283,11 @@ if (Apify.utils.social.YOUTUBE_REGEX.test('https://www.youtube.com/watch?v=kM7Yf}```-<a name="social.YOUTUBE_REGEX_GLOBAL"></a>+---++<a name="youtube_regex_global"></a>-## `social.YOUTUBE_REGEX_GLOBAL` : `RegExp`+## `social.YOUTUBE_REGEX_GLOBAL`Regular expression to find multiple Youtube video URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:@@ -302,119 +303,95 @@ const matches = text.match(Apify.utils.social.YOUTUBE_REGEX_GLOBAL);if (matches) console.log(`${matches.length} Youtube videos found!`);```-<a name="social.EMAIL_REGEX"></a>+---++<a name="email_regex"></a>-## `social.EMAIL_REGEX` : `RegExp`+## `social.EMAIL_REGEX`Regular expression to exactly match a single email address. It has the following form: `/^...$/i`.-<a name="social.EMAIL_REGEX_GLOBAL"></a>+---++<a name="email_regex_global"></a>-## `social.EMAIL_REGEX_GLOBAL` : `RegExp`+## `social.EMAIL_REGEX_GLOBAL`Regular expression to find multiple email addresses in a text. It has the following form: `/.../ig`.-<a name="social.emailsFromText"></a>+----## `social.emailsFromText(text)` ⇒ `Array<String>`+<a name="emailsfromtext"></a>-The function extracts email addresses from a plain text. Note that the function preserves the order of emails and keep duplicates.+## `social.emailsFromText(text)`**Returns**: `Array<String>` - Array of emails addresses found. If no emails are found, the function returns an empty array.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>text</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Text to search in.</p>-</td></tr></tbody>-</table>-<a name="social.emailsFromUrls"></a>--## `social.emailsFromUrls(urls)` ⇒ `Array<String>`+The function extracts email addresses from a plain text. Note that the function preserves the order of emails and keep duplicates.++**Params**++- **`text`**: `String` - Text to search in.++---++<a name="emailsfromurls"></a>++## `social.emailsFromUrls(urls)`++**Returns**: `Array<String>` - Array of emails addresses found. If no emails are found, the function returns an empty array.The function extracts email addresses from a list of URLs. Basically it looks for all `mailto:` URLs and returns valid email addresses from them. Notethat the function preserves the order of emails and keep duplicates.-**Returns**: `Array<String>` - Array of emails addresses found. If no emails are found, the function returns an empty array.+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>urls</code></td><td><code>Array<String></code></td>-</tr>-<tr>-<td colspan="3"><p>Array of URLs.</p>-</td></tr></tbody>-</table>-<a name="social.phonesFromText"></a>--## `social.phonesFromText(text)` ⇒ `Array<String>`+- **`urls`**: `Array<String>` - Array of URLs.++---++<a name="phonesfromtext"></a>++## `social.phonesFromText(text)`++**Returns**: `Array<String>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.The function attempts to extract phone numbers from a text. Please note that the results might not be accurate, since phone numbers appear in a largevariety of formats and conventions. If you encounter some problems, please [file an issue](https://github.com/apifytech/apify-js/issues).-**Returns**: `Array<String>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>text</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Text to search the phone numbers in.</p>-</td></tr></tbody>-</table>-<a name="social.phonesFromUrls"></a>--## `social.phonesFromUrls(urls)` ⇒ `Array<String>`+- **`text`**: `String` - Text to search the phone numbers in.++---++<a name="phonesfromurls"></a>++## `social.phonesFromUrls(urls)`++**Returns**: `Array<String>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.Finds phone number links in an array of URLs and extracts the phone numbers from them. Note that the phone number links look like `tel://123456789`,`tel:/123456789` or `tel:123456789`.-**Returns**: `Array<String>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.+**Params**++- **`urls`**: `Array<String>` - Array of URLs.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>urls</code></td><td><code>Array<String></code></td>-</tr>-<tr>-<td colspan="3"><p>Array of URLs.</p>-</td></tr></tbody>-</table>-<a name="social.parseHandlesFromHtml"></a>--## `social.parseHandlesFromHtml(html, data)` ⇒ [`SocialHandles`](../typedefs/socialhandles)+---++<a name="parsehandlesfromhtml"></a>++## `social.parseHandlesFromHtml(html, data)`++**Returns**: [`SocialHandles`](/docs/typedefs/social-handles) - An object with the social handles.The function attempts to extract emails, phone numbers and social profile URLs from a HTML document, specifically LinkedIn, Twitter, Instagram andFacebook profile URLs. The function removes duplicates from the resulting arrays and sorts the items alphabetically.-Note that the `phones` field contains phone numbers extracted from the special phone links such as `<a href="tel:+1234556789">call us</a>` (see-[`social.phonesFromUrls()`](#social.phonesFromUrls)]) and potentially other sources with high certainty, while `phonesUncertain` contains phone-numbers extracted from the plain text, which might be very inaccurate.+Note that the `phones` field contains phone numbers extracted from the special phone links such as `[call us](tel:+1234556789)` (see+[`social.phonesFromUrls()`](/docs/api/social#phonesfromurls)) and potentially other sources with high certainty, while `phonesUncertain` contains+phone numbers extracted from the plain text, which might be very inaccurate.**Example usage:**@@ -431,26 +408,10 @@ console.log('Social handles:');console.dir(result);```-**Returns**: [`SocialHandles`](../typedefs/socialhandles) - An object with the social handles.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>html</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>HTML text</p>-</td></tr><tr>-<td><code>data</code></td><td><code>Object</code></td><td><code></code></td>-</tr>-<tr>-<td colspan="3"><p>Optional object which will receive the <code>text</code> and <code>$</code> properties- that contain text content of the HTML and <code>cheerio</code> object, respectively. This is an optimization- so that the caller doesn't need to parse the HTML document again, if needed.</p>-</td></tr></tbody>-</table>+**Params**++- **`html`**: `String` - HTML text+- **`data`**: `Object` <code> = </code> - Optional object which will receive the `text` and `$` properties that contain text content of the HTML and+ `cheerio` object, respectively. This is an optimization so that the caller doesn't need to parse the HTML document again, if needed.++---diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 6231e3d50..e3f436e22 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -18,30 +18,40 @@ const Apify = require('apify');await Apify.utils.sleep(1500);```-- [`utils`](#utils) : `object`- - [`.enqueueLinks`](#utils.enqueueLinks) ⇒ `Promise<Array<QueueOperationInfo>>`- - [`.requestAsBrowser`](#utils.requestAsBrowser) ⇒ `Promise<(http.IncomingMessage|stream.Readable)>`- - [`.sleep`](#utils.sleep) ⇒ `Promise<void>`- - [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)- - [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)- - [`.isDocker(forceReset)`](#utils.isDocker) ⇒ `Promise<boolean>`- - [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`- - [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`- - [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`- - [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`- - [`~$`](#utils.htmlToText..$) : `Cheerio`--<a name="utils.enqueueLinks"></a>--## `utils.enqueueLinks` ⇒ `Promise<Array<QueueOperationInfo>>`+---++<a name="url_no_commas_regex"></a>++## `utils.URL_NO_COMMAS_REGEX`++Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters and does not+support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).++---++<a name="url_with_commas_regex"></a>++## `utils.URL_WITH_COMMAS_REGEX`++Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query. Note,+however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.++---++<a name="enqueuelinks"></a>++## `utils.enqueueLinks(options)`++**Returns**: [`Promise<Array<QueueOperationInfo>>`](/docs/typedefs/queue-operation-info) - Promise that resolves to an array of+[`QueueOperationInfo`](/docs/typedefs/queue-operation-info) objects.The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default) either in a Puppeteer page, or in a Cheerio object-(parsed HTML), and enqueues the URLs in their `href` attributes to the provided [`RequestQueue`](requestqueue). If you're looking to find URLs in-JavaScript heavy pages where links are not available in `href` elements, but rather navigations are triggered in click handlers see-[`enqueueLinksByClickingElements()`](puppeteer#puppeteer.enqueueLinksByClickingElements).+(parsed HTML), and enqueues the URLs in their `href` attributes to the provided [`RequestQueue`](/docs/api/request-queue). If you're looking to find+URLs in JavaScript heavy pages where links are not available in `href` elements, but rather navigations are triggered in click handlers see+[`puppeteer.enqueueLinksByClickingElements()`](/docs/api/puppeteer#enqueuelinksbyclickingelements).-Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](pseudourl) objects and override settings of the-enqueued [`Request`](request) objects.+Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](/docs/api/pseudo-url) objects and override+settings of the enqueued [`Request`](/docs/api/request) objects.**Example usage**@@ -60,84 +70,55 @@ await Apify.utils.enqueueLinks({});```-**Returns**: `Promise<Array<QueueOperationInfo>>` - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo)-objects.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>All <code>enqueueLinks()</code> parameters are passed- via an options object with the following keys:</p>-</td></tr><tr>-<td><code>options.page</code></td><td><code>Page</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.- Either <code>page</code> or <code>$</code> option must be provided.</p>-</td></tr><tr>-<td><code>options.$</code></td><td><code>Cheerio</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p><a href="https://github.com/cheeriojs/cheerio" target="_blank"><code>Cheerio</code></a> object with loaded HTML.- Either <code>page</code> or <code>$</code> option must be provided.</p>-</td></tr><tr>-<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A request queue to which the URLs will be enqueued.</p>-</td></tr><tr>-<td><code>[options.selector]</code></td><td><code>String</code></td><td><code>'a'</code></td>-</tr>-<tr>-<td colspan="3"><p>A CSS selector matching links to be enqueued.</p>-</td></tr><tr>-<td><code>[options.baseUrl]</code></td><td><code>string</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer,- since the relative URL resolution is done inside the browser automatically.</p>-</td></tr><tr>-<td><code>[options.pseudoUrls]</code></td><td><code>Array<Object></code> | <code>Array.<String></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,- or an array of strings or RegExps or plain Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.</p>-<p> The plain objects must include at least the <code>purl</code> property, which holds the pseudo-URL string or RegExp.- All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor,- which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p>-<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function- enqueues all links found on the page.</p>-</td></tr><tr>-<td><code>[options.transformRequestFunction]</code></td><td><code><a href="../typedefs/requesttransform">RequestTransform</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Just before a new <a href="request"><code>Request</code></a> is constructed and enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used- to remove it or modify its contents such as <code>userData</code>, <code>payload</code> or, most importantly <code>uniqueKey</code>. This is useful- when you need to enqueue multiple <code>Requests</code> to the queue that share the same URL, but differ in methods or payloads,- or to dynamically update or create <code>userData</code>.</p>-<p> For example: by adding <code>keepUrlFragment: true</code> to the <code>request</code> object, URL fragments will not be removed- when <code>uniqueKey</code> is computed.</p>-<p> <strong>Example:</strong></p>-<pre><code class="language-javascript"> {- transformRequestFunction: (request) => {- request.userData.foo = 'bar';- request.keepUrlFragment = true;- return request;- }- }</code></pre>-</td></tr></tbody>-</table>-<a name="utils.requestAsBrowser"></a>--## `utils.requestAsBrowser` ⇒ `Promise<(http.IncomingMessage|stream.Readable)>`+**Params**++- **`options`**: `Object` - All `enqueueLinks()` parameters are passed via an options object with the following keys:++ - **`.page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object. Either `page` or `$` option must be+ provided.+ - **`.$`**: `Cheerio` - [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML. Either `page` or `$` option must be+ provided.+ - **`.requestQueue`**: [`RequestQueue`](/docs/api/request-queue) - A request queue to which the URLs will be enqueued.+ - **`[.selector]`**: `String` <code> = 'a'</code> - A CSS selector matching links to be enqueued.+ - **`[.baseUrl]`**: `string` - A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer, since the+ relative URL resolution is done inside the browser automatically.+ - **`[.pseudoUrls]`**: `Array<Object>` | `Array.<String>` - An array of [`PseudoUrl`](/docs/api/pseudo-url)s matching the URLs to be+ enqueued, or an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](/docs/api/pseudo-url)s can be constructed.++ The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the+ `requestTemplate` argument of the [`PseudoUrl`](/docs/api/pseudo-url) constructor, which lets you specify special properties for the enqueued+ [`Request`](/docs/api/request) objects.++ If `pseudoUrls` is an empty array, `null` or `undefined`, then the function enqueues all links found on the page.++ - **`[.transformRequestFunction]`**: [`RequestTransform`](/docs/typedefs/request-transform) - Just before a new [`Request`](/docs/api/request)+ is constructed and enqueued to the [`RequestQueue`](/docs/api/request-queue), this function can be used to remove it or modify its contents+ such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful when you need to enqueue multiple `Requests` to the queue that+ share the same URL, but differ in methods or payloads, or to dynamically update or create `userData`.++ For example: by adding `keepUrlFragment: true` to the `request` object, URL fragments will not be removed when `uniqueKey` is computed.++ **Example:**++ ```javascript+ {+ transformRequestFunction: request => {+ request.userData.foo = 'bar';+ request.keepUrlFragment = true;+ return request;+ };+ }+ ```++---++<a name="requestasbrowser"></a>++## `utils.requestAsBrowser(options)`++**Returns**: `Promise<(http.IncomingMessage|stream.Readable)>` - This will typically be a+[Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage), however, if returned from the cache it will be a+[response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.**IMPORTANT:** This function uses an insecure version of HTTP parser by default and also ignores SSL/TLS errors. This is very useful in scraping,because it allows bypassing certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping scenarios, please set@@ -155,27 +136,31 @@ Currently, the function sends requests the same way as Firefox web browser does.Internally, the function uses httpRequest function from the [@apify/httpRequest](https://github.com/apifytech/http-request) NPM package to perform therequest. All `options` not recognized by this function are passed to it, so see it for more details.-**Returns**: `Promise<(http.IncomingMessage|stream.Readable)>` - This will typically be a-[Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage), however, if returned from the cache it will be a-[response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.+**Params**++- **`options`**: [`RequestAsBrowserOptions`](/docs/typedefs/request-as-browser-options) - All `requestAsBrowser` configuration options.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code><a href="../typedefs/requestasbrowseroptions">RequestAsBrowserOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>All <code>requestAsBrowser</code> configuration options.</p>-</td></tr></tbody>-</table>-<a name="utils.sleep"></a>--## `utils.sleep` ⇒ `Promise<void>`+---++<a name="isdocker"></a>++## `utils.isDocker(forceReset)`++**Returns**: `Promise<boolean>`++Returns a `Promise` that resolves to true if the code is running in a Docker container.++**Params**++- **`forceReset`**: `boolean`++---++<a name="sleep"></a>++## `utils.sleep(millis)`++**Returns**: `Promise<void>`Returns a `Promise` that resolves after a specific period of time. This is useful to implement waiting in your code, e.g. to prevent overloading oftarget website or to avoid bot detection.@@ -191,125 +176,62 @@ const Apify = require('apify');await Apify.utils.sleep(1500);```-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>millis</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.</p>-</td></tr></tbody>-</table>-<a name="utils.URL_NO_COMMAS_REGEX"></a>+**Params**-## `utils.URL_NO_COMMAS_REGEX`+- **`millis`**: `Number` - Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.-Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters and does not-support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).+----<a name="utils.URL_WITH_COMMAS_REGEX"></a>+<a name="downloadlistofurls"></a>-## `utils.URL_WITH_COMMAS_REGEX`+## `utils.downloadListOfUrls(options)`-Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query. Note,-however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.+**Returns**: `Promise<Array<String>>`-<a name="utils.isDocker"></a>+Returns a promise that resolves to an array of urls parsed from the resource available at the provided url. Optionally, custom regular expression and+encoding may be provided.-## `utils.isDocker(forceReset)` ⇒ `Promise<boolean>`+**Params**-Returns a `Promise` that resolves to true if the code is running in a Docker container.+- **`options`**: `Object`+ - **`.url`**: `String` - URL to the file+ - **`[.encoding]`**: `String` <code> = 'utf8'</code> - The encoding of the file.+ - **`[.urlRegExp]`**: `RegExp` <code> = URL_NO_COMMAS_REGEX</code> - Custom regular expression to identify the URLs in the file to extract. The+ regular expression should be case-insensitive and have global flag set (i.e. `/something/gi`).-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>forceReset</code></td><td><code>boolean</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="utils.downloadListOfUrls"></a>--## `utils.downloadListOfUrls(options)` ⇒ `Promise<Array<String>>`+----Returns a promise that resolves to an array of urls parsed from the resource available at the provided url. Optionally, custom regular expression and-encoding may be provided.+<a name="extracturls"></a>++## `utils.extractUrls(options)`-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>options.url</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>URL to the file</p>-</td></tr><tr>-<td><code>[options.encoding]</code></td><td><code>String</code></td><td><code>'utf8'</code></td>-</tr>-<tr>-<td colspan="3"><p>The encoding of the file.</p>-</td></tr><tr>-<td><code>[options.urlRegExp]</code></td><td><code>RegExp</code></td><td><code>URL_NO_COMMAS_REGEX</code></td>-</tr>-<tr>-<td colspan="3"><p>Custom regular expression to identify the URLs in the file to extract.- The regular expression should be case-insensitive and have global flag set (i.e. <code>/something/gi</code>).</p>-</td></tr></tbody>-</table>-<a name="utils.extractUrls"></a>--## `utils.extractUrls(options)` ⇒ `Array<String>`+**Returns**: `Array<String>`Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-</tr><tr>-<td><code>options.string</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-</tr><tr>-<td><code>[options.urlRegExp]</code></td><td><code>RegExp</code></td><td><code>Apify.utils.URL_NO_COMMAS_REGEX</code></td>-</tr>-<tr>-</tr></tbody>-</table>-<a name="utils.getRandomUserAgent"></a>--## `utils.getRandomUserAgent()` ⇒ `String`+**Params**++- **`options`**: `Object`+ - **`.string`**: `String`+ - **`[.urlRegExp]`**: `RegExp` <code> = Apify.utils.URL_NO_COMMAS_REGEX</code>++---++<a name="getrandomuseragent"></a>++## `utils.getRandomUserAgent()`++**Returns**: `String`Returns a randomly selected User-Agent header out of a list of the most common headers.-<a name="utils.htmlToText"></a>+---++<a name="htmltotext"></a>-## `utils.htmlToText(html)` ⇒ `String`+## `utils.htmlToText(html)`++**Returns**: `String` - Plain textThe function converts a HTML document to a plain text.@@ -335,23 +257,8 @@ const html = '<html><body>Some text</body></html>';const text = htmlToText(cheerio.load(html, { decodeEntities: true }));```-**Returns**: `String` - Plain text+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>html</code></td><td><code>String</code> | <code>Cheerio</code></td>-</tr>-<tr>-<td colspan="3"><p>HTML text or parsed HTML represented using a-<a href="https://www.npmjs.com/package/cheerio">cheerio</a> function.</p>-</td></tr></tbody>-</table>-<a name="utils.htmlToText..$"></a>--### `htmlToText~$` : `Cheerio`+- **`html`**: `String` | `Cheerio` - HTML text or parsed HTML represented using a [cheerio](https://www.npmjs.com/package/cheerio) function.++---diff --git a/docs/examples/BasicCrawler.md b/docs/examples/BasicCrawler.mdindex ab644b4ad..c41d1b532 100644--- a/docs/examples/BasicCrawler.md+++ b/docs/examples/BasicCrawler.md@@ -1,14 +1,14 @@----id: basiccrawler+id: basic-crawlertitle: Basic Crawler----This is the most basic example of the Apify SDK, which demonstrates some of its elementary tools, such as the [`BasicCrawler`](../api/basiccrawler)-and [`RequestList`](../api/requestlist) classes. The script just downloads several web pages with plain HTTP requests (using the-<a href="https://www.npmjs.com/package/request-promise" target="_blank">request-promise</a> library) and stores their raw HTML and URL to the default+This is the most basic example of the Apify SDK, which demonstrates some of its elementary tools, such as the+[`BasicCrawler`](/docs/api/basic-crawler) and [`RequestList`](/docs/api/request-list) classes. The script just downloads several web pages with plain+HTTP requests (using the [`request-promise`](https://www.npmjs.com/package/request-promise) library) and stores their raw HTML and URL to the defaultdataset. In local configuration, the data will be stored as JSON files in `./apify_storage/datasets/default`.-To run this example on the Apify Platform, select the `Node.js 10 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actor+To run this example on the Apify Platform, select the `Node.js 12 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actorconfiguration.```javascriptdiff --git a/docs/examples/CallActor.md b/docs/examples/CallActor.mdindex 46f0dca3b..06f855248 100644--- a/docs/examples/CallActor.md+++ b/docs/examples/CallActor.md@@ -1,19 +1,18 @@----id: callactor+id: call-actortitle: Call Actor----This example demonstrates how to start an Apify actor using [`Apify.call()`](../api/apify#module_Apify.call) and how to call Apify API using-[`Apify.client`](../api/apify#module_Apify.client). The script extracts the current Bitcoin prices from-<a href="https://www.kraken.com/" target="_blank">Kraken.com</a> and sends them to your email using the-<a href="https://apify.com/apify/send-mail" target="_blank">apify/send-mail</a> actor.+This example demonstrates how to start an Apify actor using [`Apify.call()`](/docs/api/apify#call) and how to call Apify API using+[`Apify.client`](/docs/api/apify#client). The script extracts the current Bitcoin prices from [Kraken.com](https://www.kraken.com/) and sends them to+your email using the [`apify/send-mail`](https://apify.com/apify/send-mail) actor.-To make the example work, you'll need an <a href="https://my.apify.com/" target="_blank">Apify Account</a>. Go to-<a href="https://my.apify.com/account#/integrations" target="_blank">Account - Integrations</a> page to obtain your API token and set it to the-`APIFY_TOKEN` [environment variable](../guides/environmentvariables), or run the script using the CLI. If you deploy this actor to the Apify Cloud-then you can set up a scheduler for early morning.+To make the example work, you'll need an [Apify Account](https://my.apify.com/). Go to+[Account - Integrations](https://my.apify.com/account#/integrations) page to obtain your API token and set it to the+[`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable, or run the script using the CLI. If you deploy this actor to the+Apify Cloud then you can set up a scheduler for early morning.-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/CheerioCrawler.md b/docs/examples/CheerioCrawler.mdindex 49a1f4333..d87f34d82 100644--- a/docs/examples/CheerioCrawler.md+++ b/docs/examples/CheerioCrawler.md@@ -1,13 +1,13 @@----id: cheeriocrawler+id: cheerio-crawlertitle: Cheerio Crawler----This example demonstrates how to use [`CheerioCrawler`](../api/cheeriocrawler) to crawl a list of URLs from an external file, load each URL using a-plain HTTP request, parse the HTML using <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> and extract some data from it:-the page title and all H1 tags.+This example demonstrates how to use [`CheerioCrawler`](/docs/api/cheerio-crawler) to crawl a list of URLs from an external file, load each URL using+a plain HTTP request, parse the HTML using [cheerio](https://www.npmjs.com/package/cheerio) and extract some data from it: the page title and all H1+tags.-To run this example on the Apify Platform, select the `Node.js 10 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actor+To run this example on the Apify Platform, select the `Node.js 12 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actorconfiguration.```javascriptdiff --git a/docs/examples/Forms.md b/docs/examples/Forms.mdindex da96d0cd6..4b56b4f60 100644--- a/docs/examples/Forms.md+++ b/docs/examples/Forms.md@@ -3,13 +3,12 @@ id: formstitle: Forms----This example demonstrates how to use [`PuppeteerCrawler`](../api/puppeteercrawler) to automatically fill and submit a search form to look up-repositories on <a href="https://news.ycombinator.com" target="_blank">GitHub</a> using headless Chrome / Puppeteer. The actor first fills in the-search term, repository owner, start date and language of the repository, then submits the form and prints out the results. Finally, the results are-saved either on the Apify platform to the default <a href="https://sdk.apify.com/docs/api/dataset" target="_blank">dataset</a> or on the local machine-as JSON files in ./apify_storage/datasets/default.+This example demonstrates how to use [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) to automatically fill and submit a search form to look up+repositories on [GitHub](https://news.ycombinator.com) using headless Chrome / Puppeteer. The actor first fills in the search term, repository owner,+start date and language of the repository, then submits the form and prints out the results. Finally, the results are saved either on the Apify+platform to the default [`Dataset`](/docs/api/dataset) or on the local machine as JSON files in `./apify_storage/datasets/default`.-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/HelloWorld.md b/docs/examples/HelloWorld.mdindex f0a763222..abac63043 100644--- a/docs/examples/HelloWorld.md+++ b/docs/examples/HelloWorld.md@@ -1,11 +1,11 @@----id: helloworld+id: hello-worldtitle: Hello World---Run the following example to perform a recursive crawl of a website using Puppeteer.-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/PuppeteerCrawler.md b/docs/examples/PuppeteerCrawler.mdindex f8014e092..49de57e68 100644--- a/docs/examples/PuppeteerCrawler.md+++ b/docs/examples/PuppeteerCrawler.md@@ -1,14 +1,14 @@----id: puppeteercrawler+id: puppeteer-crawlertitle: Puppeteer Crawler----This example demonstrates how to use [`PuppeteerCrawler`](../api/puppeteercrawler) in combination with [`RequestQueue`](../api/requestqueue) to-recursively scrape the <a href="https://news.ycombinator.com" target="_blank">Hacker News website</a> using headless Chrome / Puppeteer. The crawler-starts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results are stored to-the default dataset. In local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`+This example demonstrates how to use [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) in combination with [`RequestQueue`](/docs/api/request-queue)+to recursively scrape the [Hacker News website](https://news.ycombinator.com) using headless Chrome / Puppeteer. The crawler starts with a single URL,+finds links to next pages, enqueues them and continues until no more desired links are available. The results are stored to the default dataset. In+local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/PuppeteerSitemap.md b/docs/examples/PuppeteerSitemap.mdindex 11acd5315..6805fca1b 100644--- a/docs/examples/PuppeteerSitemap.md+++ b/docs/examples/PuppeteerSitemap.md@@ -1,13 +1,13 @@----id: puppeteersitemap+id: puppeteer-sitemaptitle: Puppeteer Sitemap----This example demonstrates how to use [`PuppeteerCrawler`](../api/puppeteercrawler) to crawl a list of web pages specified in a sitemap. The crawler-extracts page title and URL from each page and stores them as a record to the default dataset. In local configuration, the results are stored as JSON-files in `./apify_storage/datasets/default`+This example demonstrates how to use [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) to crawl a list of web pages specified in a sitemap. The+crawler extracts page title and URL from each page and stores them as a record to the default dataset. In local configuration, the results are stored+as JSON files in `./apify_storage/datasets/default`-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/PuppeteerWithProxy.md b/docs/examples/PuppeteerWithProxy.mdindex 5a1e57a25..95bf403f8 100644--- a/docs/examples/PuppeteerWithProxy.md+++ b/docs/examples/PuppeteerWithProxy.md@@ -1,14 +1,13 @@----id: puppeteerwithproxy+id: puppeteer-with-proxytitle: Puppeteer With Proxy----This example demonstrates how to load pages in headless Chrome / Puppeteer over <a href="https://docs.apify.com/proxy" target="_blank">Apify-Proxy</a>. To make it work, you'll need an Apify Account that has access to the proxy. The proxy password is available on the-<a href="https://my.apify.com/proxy" target="_blank">Proxy</a> page in the app. Just set it to the `APIFY_PROXY_PASSWORD`-[environment variable](../guides/environmentvariables) or run the script using the CLI.+This example demonstrates how to load pages in headless Chrome / Puppeteer over [Apify Proxy](https://docs.apify.com/proxy). To make it work, you'll+need an Apify Account that has access to the proxy. The proxy password is available on the [Proxy](https://my.apify.com/proxy) page in the app. Just+set it to the [`APIFY_PROXY_PASSWORD`](../guides/environment-variables#APIFY_PROXY_PASSWORD) environment variable or run the script using the CLI.-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.mdindex 08eb91f16..6d3b13752 100644--- a/docs/examples/Screenshots.md+++ b/docs/examples/Screenshots.md@@ -3,9 +3,11 @@ id: screenshotstitle: Screenshots----This example demonstrates how to read and write data to the default key-value store using [`Apify.getValue()`](../api/apify#module_Apify.getValue) and-[`Apify.setValue()`](../api/apify#module_Apify.setValue). The script crawls a list of URLs using Puppeteer, captures a screenshot of each page and-saves it to the store. The list of URLs is provided as actor input that is also read from the store.+This example demonstrates how to read and write data to the default key-value store using [`Apify.getValue()`](/docs/api/apify#getValue) and+[`Apify.setValue()`](/docs/api/apify#setValue).++The script crawls a list of URLs using Puppeteer, captures a screenshot of each page and saves it to the store. The list of URLs is provided as actor+input that is also read from the store.In local configuration, the input is stored in the default key-value store's directory as a JSON file at`./apify_storage/key_value_stores/default/INPUT.json`. You need to create the file and set it with the following content:@@ -15,10 +17,10 @@ In local configuration, the input is stored in the default key-value store's dir```On the Apify cloud, the input can be either set manually in the UI app or passed as the POST payload to the-<a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>. For more details, see-<a href="https://docs.apify.com/actor/run#input-and-output" target="_blank">Input and output</a> in the Apify Actor documentation.+[Run actor API call](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor). For more details, see+[Input and output](https://docs.apify.com/actor/run#input-and-output) in the Apify Actor documentation.-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/SynchronousRun.md b/docs/examples/SynchronousRun.mdindex 083da1004..82322b2a2 100644--- a/docs/examples/SynchronousRun.md+++ b/docs/examples/SynchronousRun.md@@ -1,23 +1,21 @@----id: synchronousrun+id: synchronous-runtitle: Synchronous Run----This example shows a quick actor that has a run time of just a few seconds. It opens a-<a href="http://goldengatebridge75.org/news/webcam.html" target="_blank">web page</a> that contains a webcam stream from the Golden Gate Bridge, takes-a screenshot of the page and saves it as output.+This example shows a quick actor that has a run time of just a few seconds. It opens a [web page](http://goldengatebridge75.org/news/webcam.html) that+contains a webcam stream from the Golden Gate Bridge, takes a screenshot of the page and saves it as output.-This actor can be invoked synchronously using a single HTTP request to directly obtain its output as a reponse, using the-<a href="https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a> Apify API-endpoint. The example is also shared as the-<a href="https://apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a> actor in the Apify library, so you-can test it directly there simply by sending a POST request to+This actor can be invoked synchronously using a single HTTP request to directly obtain its output as a response, using the+[Run actor synchronously](https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input) Apify API endpoint. The example is+also shared as the [`apify/example-golden-gate-webcam`](https://apify.com/apify/example-golden-gate-webcam) actor in the Apify library, so you can+test it directly there simply by sending a POST request to```httphttps://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]```-To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascript@@ -25,20 +23,20 @@ const Apify = require('apify');Apify.main(async () => {// Launch web browser.- const browser = await Apify.launchPuppeteer();+ const browser = await Apify.launchPuppeteer({ headless: true });// Load http://goldengatebridge75.org/news/webcam.html and get an IFRAME with the webcam streamconsole.log('Opening web page...');const page = await browser.newPage();await page.goto('http://goldengatebridge75.org/news/webcam.html');- const iframe = (await page.frames()).pop();-- // Get webcam image element handle.- const imageElementHandle = await iframe.$('.VideoColm img');-- // Give the webcam image some time to load.- console.log('Waiting for page to load...');- await Apify.utils.sleep(3000);+ const frames = await page.frames();+ const iframeWithVideo = frames[1];++ // Wait for the webcam to load.+ console.log('Waiting for the webcam image.');+ const imageElementHandle = await iframeWithVideo.waitForSelector('.VideoColm img[src^="blob:"]');+ // Wait a little more to give the browser a chance to display the image.+ await page.waitFor(1000);// Get a screenshot of that image.const imageBuffer = await imageElementHandle.screenshot();diff --git a/docs/guides/data_storage.md b/docs/guides/data_storage.mdindex 544e2e31d..bd1ccb192 100644--- a/docs/guides/data_storage.md+++ b/docs/guides/data_storage.md@@ -1,10 +1,10 @@----id: datastorage+id: data-storagetitle: Data Storage---The Apify SDK has several data storage types that are useful for specific tasks. The data is stored either on local disk to a directory defined by the-`APIFY_LOCAL_STORAGE_DIR` environment variable, or on the <a href="https://apify.com" target="_blank">Apify Cloud</a> under the user account+`APIFY_LOCAL_STORAGE_DIR` environment variable, or on the [Apify Cloud](https://apify.com) under the user accountidentified by the API token defined by the `APIFY_TOKEN` environment variable. If neither of these variables is defined, by default Apify SDK sets`APIFY_LOCAL_STORAGE_DIR` to `./apify_storage` in the current working directory and prints a warning.@@ -14,9 +14,9 @@ code changes are needed.**Related links**-- <a href="https://docs.apify.com/storage" target="_blank">Apify cloud storage documentation</a>-- <a href="https://my.apify.com/storage" target="_blank">View storage in Apify app</a>-- <a href="https://apify.com/docs/api/v2#/reference/key-value-stores" target="_blank">API reference</a>+- [Apify cloud storage documentation](https://docs.apify.com/storage)+- [View storage in Apify app](https://my.apify.com/storage)+- [API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)## Key-value store@@ -27,9 +27,9 @@ Each actor run is associated with a **default key-value store**, which is createand output is stored in the default key-value store under the `INPUT` and `OUTPUT` key, respectively. Typically the input and output is a JSON file,although it can be any other format.-In the Apify SDK, the key-value store is represented by the [`KeyValueStore`](../api/keyvaluestore) class. In order to simplify access to the default-key-value store, the SDK also provides [`Apify.getValue()`](../api/apify#module_Apify.getValue) and-[`Apify.setValue()`](../api/apify#module_Apify.setValue) functions.+In the Apify SDK, the key-value store is represented by the [`KeyValueStore`](/docs/api/key-value-store) class. In order to simplify access to the default+key-value store, the SDK also provides [`Apify.getValue()`](/docs/api/apify#getValue) and+[`Apify.setValue()`](/docs/api/apify#setValue) functions.In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:@@ -65,7 +65,7 @@ const value = await store.getValue('some-key');await store.setValue('some-key', null);```-To see a real-world example of how to get the input from the key-value store, see the [Screenshots](../examples/screenshots) example.+To see a real-world example of how to get the input from the key-value store, see the [Screenshots](/docs/examples/screenshots) example.## Dataset@@ -73,17 +73,17 @@ Datasets are used to store structured data where each object stored has the samecan imagine a dataset as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add newrecords to it but you cannot modify or remove existing records.-When the dataset is stored in the <a href="https://apify.com" target="_blank">Apify Cloud</a>, you can export its data to the following formats: HTML,+When the dataset is stored in the [Apify Cloud](https://apify.com), you can export its data to the following formats: HTML,JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the-<a href="https://my.apify.com/storage" target="_blank">Storage</a> section in the Apify app. The actual data is exported using the-<a href="https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items" target="_blank">Get dataset items</a> Apify API endpoint. This+[Storage](https://my.apify.com/storage) section in the Apify app. The actual data is exported using the+[Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. Thisway you can easily share crawling results.Each actor run is associated with a **default dataset**, which is created exclusively for the actor run. Typically, it is used to store crawlingresults specific for the actor run. Its usage is optional.-In the Apify SDK, the dataset is represented by the [`Dataset`](../api/dataset) class. In order to simplify writes to the default dataset, the SDK-also provides the [`Apify.pushData()`](../api/apify#module_Apify.pushData) function.+In the Apify SDK, the dataset is represented by the [`Dataset`](/docs/api/dataset) class. In order to simplify writes to the default dataset, the SDK+also provides the [`Apify.pushData()`](/docs/api/apify#pushData) function.In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:@@ -111,7 +111,7 @@ await dataset.pushData({ foo: 'bar' });await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);```-To see how to use the dataset to store crawler results, see the [Cheerio Crawler](../examples/cheeriocrawler) example.+To see how to use the dataset to store crawler results, see the [Cheerio Crawler](/docs/examples/cheerio-crawler) example.## Request queue@@ -121,7 +121,7 @@ recursively follow links to other pages. The data structure supports both breadtEach actor run is associated with a **default request queue**, which is created exclusively for the actor run. Typically, it is used to store URLs tocrawl in the specific actor run. Its usage is optional.-In Apify SDK, the request queue is represented by the [`RequestQueue`](../api/requestqueue) class.+In Apify SDK, the request queue is represented by the [`RequestQueue`](/docs/api/request-queue) class.In local configuration, the request queue data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:@@ -159,4 +159,4 @@ await queue.markRequestHandled(request1);await queue.reclaimRequest(request2);```-To see how to use the request queue with a crawler, see the [Puppeteer Crawler](../examples/puppeteercrawler) example.+To see how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.diff --git a/docs/guides/environment_variables.md b/docs/guides/environment_variables.mdindex 72069b7cf..d3b1f9bb0 100644--- a/docs/guides/environment_variables.md+++ b/docs/guides/environment_variables.md@@ -1,78 +1,43 @@----id: environmentvariables+id: environment-variablestitle: Environment Variables----The following table shows the basic environment variables used by Apify SDK:+The following is a list of the environment variables used by Apify SDK that are available to the user:-<table>- <thead>- <tr>- <th>Environment variable</th>- <th>Description</th>- </tr>- </thead>- <tbody>- <tr>- <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>- <td>- Defines the path to a local directory where- <a href="../api/keyvaluestore">key-value stores</a>,- <a href="../api/requestlist">request lists</a>- and <a href="../api/requestqueue">request queues</a> store their data.- Typically it is set to <code>./apify_storage</code>.- If omitted, you should define- the <code>APIFY_TOKEN</code> environment variable instead.- </td>- </tr>- <tr>- <td><code>APIFY_TOKEN</code></td>- <td>- The API token for your Apify Account. It is used to access the Apify API, e.g. to access cloud storage or to run an actor in the Apify Cloud.- You can find your API token on the <a href="https://my.apify.com/account#integrations" target="_blank">Account - Integrations</a> page.- If omitted, you should define the <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.- </td>- </tr>- <tr>- <td><code>APIFY_PROXY_PASSWORD</code></td>- <td>- Optional password to <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a> for IP address rotation.- If you have have an Apify Account, you can find the password on the- <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.- This feature is optional. You can use your own proxies or no proxies at all.- </td>- </tr>- <tr>- <td><code>APIFY_HEADLESS</code></td>- <td>- If set to <code>1</code>, web browsers launched by Apify SDK will run in the headless- mode. You can still override this setting in the code, e.g. by- passing the <code>headless: true</code> option to the- <a href="../api/apify#module_Apify.launchPuppeteer"><code>Apify.launchPuppeteer()</code></a>- function. But having this setting in an environment variable allows you to develop- the crawler locally in headful mode to simplify the debugging, and only run the crawler in headless- mode once you deploy it to the Apify Cloud.- By default, the browsers are launched in headful mode, i.e. with windows.- </td>- </tr>- <tr>- <td><code>APIFY_LOG_LEVEL</code></td>- <td>- Specifies the minimum log level, which can be one of the following values (in order of severity):- <code>DEBUG</code>, <code>INFO</code>, <code>WARNING</code>, <code>SOFT_FAIL</code> and <code>ERROR</code>.- By default, the log level is set to <code>INFO</code>, which means that <code>DEBUG</code> messages- are not printed to console.- </td>- </tr>- <tr>- <td><code>APIFY_MEMORY_MBYTES</code></td>- <td>- Sets the amount of system memory in megabytes to be used by the- <a href="../api/autoscaledpool">autoscaled pool</a>.- It is used to limit the number of concurrently running tasks. By default, the max amount of memory- to be used is set to one quarter of total system memory, i. e. on a system with 8192 MB of memory,- the autoscaling feature will only use up to 2048 MB of memory.- </td>- </tr>- </tbody>-</table>+## `APIFY_HEADLESS`+If set to `1`, web browsers launched by Apify SDK will run in the headless mode. You can still override+this setting in the code, e.g. by passing the `headless: true` option to the+[`Apify.launchPuppeteer()`](/docs/api/apify#launchPuppeteer) function. But having this setting+in an environment variable allows you to develop the crawler locally in headful mode to simplify the debugging,+and only run the crawler in headless mode once you deploy it to the Apify Cloud. By default, the browsers+are launched in headful mode, i.e. with windows.++## `APIFY_LOCAL_STORAGE_DIR`+Defines the path to a local directory where [`KeyValueStore`](/docs/api/key-value-store),+[`Dataset`](/docs/api/dataset), and [`RequestQueue`](/docs/api/request-queue) store their data.+Typically it is set to `./apify_storage`. If omitted, you should define the [`APIFY_TOKEN`](#APIFY_TOKEN)+environment variable instead.++## `APIFY_LOG_LEVEL`+Specifies the minimum log level, which can be one of the following values (in order of severity):+`DEBUG`, `INFO`, `WARNING` and `ERROR`. By default, the log level is set to `INFO`,+which means that `DEBUG` messages are not printed to console. See the [`utils.log`](/docs/api/log)+namespace for logging utilities.++## `APIFY_MEMORY_MBYTES`+Sets the amount of system memory in megabytes to be used by the [`AutoscaledPool`](/docs/api/autoscaled-pool).+It is used to limit the number of concurrently running tasks. By default, the max amount of memory+to be used is set to one quarter of total system memory, i. e. on a system with 8192 MB of memory,+the autoscaling feature will only use up to 2048 MB of memory.++## `APIFY_PROXY_PASSWORD`+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.+If you have have an Apify Account, you can find the password on the [Proxy page](https://my.apify.com/proxy)+in the Apify app. This feature is optional. You can use your own proxies or no proxies at all.++## `APIFY_TOKEN`+The API token for your Apify Account. It is used to access the Apify API, e.g. to access cloud storage+or to run an actor in the Apify Cloud. You can find your API token on the+[Account - Integrations](https://my.apify.com/account#integrations) page. If omitted,+you should define the `APIFY_LOCAL_STORAGE_DIR` environment variable instead.diff --git a/docs/guides/getting_started.md b/docs/guides/getting_started.mdindex 3503c9502..671f18d60 100644--- a/docs/guides/getting_started.md+++ b/docs/guides/getting_started.md@@ -1,5 +1,5 @@----id: gettingstarted+id: getting-startedtitle: Getting Started---@@ -17,7 +17,7 @@ The goal of this getting started guide is to provide a step-by-step introductioncreating the simplest of crawlers that only print text to console, all the way up to complex systems that crawl pages, interact with them as if a realuser were sitting in front of a real browser and output structured data.-Since Apify SDK is usable both locally on any computer and on the <a href="https://my.apify.com" target="_blank">Apify Platform</a>, you will be able+Since Apify SDK is usable both locally on any computer and on the [Apify Platform](https://my.apify.com), you will be ableto use the source code in both environments interchangeably. Nevertheless, some initial setup is still required, so choose your preferred startingenvironment and let's get into it.@@ -26,8 +26,8 @@ environment and let's get into it.To run Apify SDK on your own computer, you need to meet the following pre-requisites first:1. Have Node.js version 10.17 or higher installed.- - Visit <a href="https://nodejs.org/en/download/" target="_blank">Node.js website</a> to download or use- <a href="https://github.com/creationix/nvm" target="_blank">nvm</a>+ - Visit [Node.js website](https://nodejs.org/en/download/) to download or use+ [nvm](https://github.com/creationix/nvm)2. Have NPM installed.- NPM comes bundled with Node.js so you should already have it. If not, reinstall Node.js.@@ -44,8 +44,8 @@ npm -v### Creating a new projectThe fastest and best way to create new projects with the Apify SDK is to use our own-<a href="https://www.npmjs.com/package/apify-cli" target="_blank">Apify CLI</a>. This command line tool allows you to create, run and manage Apify-projects with ease, including their deployment to the <a href="https://my.apify.com" target="_blank">Apify Platform</a> if you wish to run them in the+[Apify CLI](https://www.npmjs.com/package/apify-cli). This command line tool allows you to create, run and manage Apify+projects with ease, including their deployment to the [Apify Platform](https://my.apify.com) if you wish to run them in thecloud after developing them locally.Let's install the Apify CLI with the following command:@@ -96,16 +96,16 @@ Did you see all that? If you did, congratulations! You're ready to go!Maybe you don't have Node.js installed and don't want the hassle. Or you can't install anything on your computer because you're using a companyprovided one. Or perhaps you'd just prefer to start working in the cloud right away. Well, no worries, we've got you covered.-The <a href="https://my.apify.com" target="_blank">Apify Platform</a> is the foundational product of-<a href="https://apify.com" target="_blank">Apify</a>. It's a serverless cloud computing platform, specifically designed for any web automation jobs,+The [Apify Platform](https://my.apify.com) is the foundational product of+[Apify](https://apify.com). It's a serverless cloud computing platform, specifically designed for any web automation jobs,that may include crawling and scraping, but really works amazing for any batch jobs and long running tasks.-It comes with a free account, so let's go to our <a href="https://my.apify.com/sign-up" target="_blank">sign-up page</a> and create one, if you+It comes with a free account, so let's go to our [sign-up page](https://my.apify.com/sign-up) and create one, if youhaven't already. Don't forget to verify your email. Without it, you won't be able to run any projects.Once you're in, you might be prompted by our in-app help to walk through a step-by-step guide into some of our new features. Feel free to finish that,if you'd like, but once you're done, click on the **Actors** tab in the left menu. To read more about **Actors**, see:-[What is an Actor](./whatisanactor).+[What is an Actor](/docs/guides/what-is-an-actor).### Creating a new project@@ -119,7 +119,7 @@ the running actor. Feel free to check out the other **Run** tabs, such as **Info**Key-value-store**, where the actor's **INPUT** and **OUTPUT** are stored.Good job. You're now ready to run your own source code on the Apify Platform. For more information visit the-<a href="https://docs.apify.com/actor" target="_blank">Actor documentation page</a>, where you'll find everything about the platform's various+[Actor documentation page](https://docs.apify.com/actor), where you'll find everything about the platform's variousoptions.## First crawler@@ -129,8 +129,8 @@ introduce all the Apify SDK classes necessary to make it happen.### The general idea-There are 3 crawler classes available for use in the Apify SDK. [`BasicCrawler`](../api/basiccrawler), [`CheerioCrawler`](../api/cheeriocrawler) and-[`PuppeteerCrawler`](../api/puppeteercrawler). We'll talk about their differences later. Now, let's talk about what they have in common.+There are 3 crawler classes available for use in the Apify SDK. [`BasicCrawler`](/docs/api/basic-crawler), [`CheerioCrawler`](/docs/api/cheerio-crawler)+and [`PuppeteerCrawler`](/docs/api/puppeteer-crawler). We'll talk about their differences later. Now, let's talk about what they have in common.All the crawlers' general idea is to go to a web page, open it, do some stuff there, save some results and continue to the next page, until it's doneits job. So each time the crawler needs to find answers to two questions: **Where should I go?** and **What should I do there?**. Answering those two@@ -138,16 +138,16 @@ questions is the only setup mandatory to run the crawlers.### The Where - `Request`, `RequestList` and `RequestQueue`-All crawlers use instances of the [`Request`](../api/request) class to determine where they need to go. Each request may hold a lot of information,+All crawlers use instances of the [`Request`](/docs/api/request) class to determine where they need to go. Each request may hold a lot of information,but at the very least, it must hold a URL - a web page to open. But having only one URL would not make sense for crawling. We need to either have apre-existing list of our own URLs that we wish to visit, perhaps a thousand, or a million, or we need to build this list dynamically as we crawl,adding more and more URLs to the list as we progress.-A representation of the pre-existing list is an instance of the [`RequestList`](../api/requestlist) class. It is a static, immutable list of URLs and-other metadata (see the [`Request`](../api/request) object) that the crawler will visit, one by one, retrying whenever an error occurs, until there+A representation of the pre-existing list is an instance of the [`RequestList`](/docs/api/request-list) class. It is a static, immutable list of URLs and+other metadata (see the [`Request`](/docs/api/request) object) that the crawler will visit, one by one, retrying whenever an error occurs, until thereare no more `Requests` to process.-[`RequestQueue`](../api/requestqueue) on the other hand, represents a dynamic queue of `Requests`. One that can be updated at runtime by adding more+[`RequestQueue`](/docs/api/request-queue) on the other hand, represents a dynamic queue of `Requests`. One that can be updated at runtime by adding morepages - `Requests` to process. This allows the crawler to open one page, extract interesting URLs, such as links to other pages on the same domain,add them to the queue (called _enqueuing_) and repeat this process to build a queue of tens of thousands or more URLs while knowing only a single oneat the beginning.@@ -163,7 +163,7 @@ from the page, processing the data, saving it, calling APIs, doing calculationsThe `handlePageFunction` is provided by you, the user, and invoked automatically by the crawler for each `Request` from either the `RequestList` or`RequestQueue`. It always receives a single argument and that is a plain `Object`. Its properties change depending on the used crawler class, but italways includes at least the `request` property, which represents the currently crawled `Request` instance (i.e. the URL the crawler is visiting and-related metadata) and the `autoscaledPool` property, which is an instance of the [`AutoscaledPool`](../api/autoscaledpool) class and we'll talk about+related metadata) and the `autoscaledPool` property, which is an instance of the [`AutoscaledPool`](/docs/api/autoscaled-pool) class and we'll talk aboutit in detail later.```js@@ -177,7 +177,7 @@ it in detail later.### Putting it all togetherEnough theory! Let's put some of those hard learned facts into practice. We learned above that we need `Requests` and a `handlePageFunction` to setup-a crawler. We will also use the [`Apify.main()`](../api/apify#module_Apify.main) function. It's not mandatory, but it makes our life easier. We'll+a crawler. We will also use the [`Apify.main()`](/docs/api/apify#main) function. It's not mandatory, but it makes our life easier. We'lllearn about it in detail later on.Let's start super easy. Visit one page, get its title and close. First of all we need to require Apify, to make all of its features available to us:@@ -187,7 +187,7 @@ const Apify = require('apify');```Easy right? It doesn't get much more difficult, trust me. For the purposes of this tutorial, we'll be scraping our own webpage-<a href="https://apify.com" target="_blank">https://apify.com</a>. Now, to get there, we need a `Request` with the page's URL in one of our sources,+[https://apify.com](https://apify.com). Now, to get there, we need a `Request` with the page's URL in one of our sources,`RequestList` or `RequestQueue`. Let's go with `RequestQueue` for now.```js@@ -205,7 +205,7 @@ Apify.main(async () => {> If you're not familiar with the `async` and `await` keywords used in the example, trust that it is a native syntax in modern JavaScript and you can> [learn more about it here](https://nikgrozev.com/2017/10/01/async-await/).-The [`requestQueue.addRequest()`](../api/requestqueue#RequestQueue+addRequest) function automatically converts the plain object we passed to it to a+The [`requestQueue.addRequest()`](/docs/api/request-queue#addRequest) function automatically converts the plain object we passed to it to a`Request` instance, so now we have a `requestQueue` that holds one `request` which points to `https://apify.com`. Now we need the`handlePageFunction`.@@ -263,23 +263,23 @@ you learn more about the `CheerioCrawler`.## CheerioCrawler aka jQuery crawlerThis is the crawler that we used in our earlier example. Our simplest and also the fastest crawling solution. If you're familiar with `jQuery`, you'll-understand [`CheerioCrawler`](../api/cheeriocrawler) in minutes. <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> is+understand [`CheerioCrawler`](/docs/api/cheerio-crawler) in minutes. [Cheerio](https://www.npmjs.com/package/cheerio) isessentially `jQuery` for Node.js. It offers the same API, including the familiar `$` object. You can use it, as you would `jQuery`, for manipulatingthe DOM of a HTML page. In crawling, you'll mostly use it to select the right elements and extract their text values - the data you're interested in.But `jQuery` runs in a browser and attaches directly to the browser's DOM. Where does `cheerio` get its HTML? This is where the `Crawler` part of-[`CheerioCrawler`](../api/cheeriocrawler) comes in.+[`CheerioCrawler`](/docs/api/cheerio-crawler) comes in.### Overview-[`CheerioCrawler`](../api/cheeriocrawler) crawls by making plain HTTP requests to the provided URLs. As you remember from the previous section, the-URLs are fed to the crawler using either the [`RequestList`](../api/cheeriocrawler) or the [`RequestQueue`](../api/cheeriocrawler). The HTTP responses+[`CheerioCrawler`](/docs/api/cheerio-crawler) crawls by making plain HTTP requests to the provided URLs. As you remember from the previous section, the+URLs are fed to the crawler using either the [`RequestList`](/docs/api/cheerio-crawler) or the [`RequestQueue`](/docs/api/cheerio-crawler). The HTTP responsesit gets back are HTML pages, the same pages you would get in your browser when you first load a URL.> Note, however, that modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to> other resources such as CSS and JavaScript that get downloaded afterwards and together they create the final page. See our-> [`PuppeteerCrawler`](../api/puppeteercrawler) to crawl those.+> [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) to crawl those.-Once the page's HTML is retrieved, the crawler will pass it to <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> for+Once the page's HTML is retrieved, the crawler will pass it to [Cheerio](https://www.npmjs.com/package/cheerio) forparsing. The result is the typical `$` function, which should be familiar to `jQuery` users. You can use this `$` to do all sorts of lookups andmanipulations over the page's HTML, but in scraping, we will mostly use it to find specific HTML elements and extract their data.@@ -300,7 +300,7 @@ $('[href]')> This is not to show that Cheerio is better than plain browser JavaScript. Some might actually prefer the more expressive way plain JS provides.> Unfortunately, the browser JavaScript methods are not available in Node.js, so Cheerio is our best bet to do the parsing.-### When to use [`CheerioCrawler`](../api/cheeriocrawler)+### When to use [`CheerioCrawler`](/docs/api/cheerio-crawler)Even though using `CheerioCrawler` is extremely easy, it probably will not be your first choice for most kinds of crawling or scraping in productionenvironments. Since most websites nowadays use modern JavaScript to create rich, responsive and data driven user experiences, the plain HTTP requests@@ -308,7 +308,7 @@ the crawler uses may just fall short of your needs.But `CheerioCrawler` is far from useless! It really shines when you need to do extremely high workloads. With just 4 GBs of memory and a single CPUcore, you can scrape 500 or more pages a minute! _(assuming each page contains approximately 400KB of HTML)_ To get this high with a full browser-scraper, such as the [`PuppeteerCrawler`](../api/puppeteercrawler), you'd need significantly more computing power.+scraper, such as the [`PuppeteerCrawler`](/docs/api/puppeteer-crawler), you'd need significantly more computing power.**Advantages:**@@ -324,7 +324,7 @@ scraper, such as the [`PuppeteerCrawler`](../api/puppeteercrawler), you'd need s- May easily overload the target website with requests- Does not enable any manipulation of the website before scraping-### Basic use of [`CheerioCrawler`](../api/cheeriocrawler)+### Basic use of [`CheerioCrawler`](/docs/api/cheerio-crawler)Now that we have an idea of the crawler's inner workings, let's build one. We'll use the example from the previous section and improve on it byletting it truly crawl the page, finding new links as it goes, enqueuing them into the `RequestQueue` and then scraping them.@@ -543,7 +543,7 @@ It also allows you to modify the resulting `Requests` to match your crawling nee`enqueueLinks` is quite a powerful function so, like crawlers, it gets its arguments from an options object. This is useful, because you don't have toremember their order! But also because we can easily extend its API and add new features. You can-[find the full reference here](../api/utils#utils.enqueuelinks).+[find the full reference here](/docs/api/utils#enqueueLinks).We suggest using ES6 destructuring to grab the `enqueueLinks()` function off of the `utils` object, so you don't have to type `Apify.utils` all thetime.@@ -614,7 +614,7 @@ Pseudo-URLs help to use this structure to select only the relevant links from a#### Structure of a pseudo-URL-Pseudo-URL is a URL with <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">regular expressions</a>) enclosed+Pseudo-URL is a URL with [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions)) enclosedin `[]` brackets. Since we're running Node.js, the regular expressions should follow the JavaScript style.For example, a pseudo-URL@@ -672,7 +672,7 @@ That's been quite a lot of theory and examples. We might as well put it to practtwo arguments. The source, in our case the `$` object, and the destination - the `requestQueue`. To filter links, we need to add a third argument:`pseudoUrls`.-The `options.pseudoUrls` argument is always an `Array`, but its contents can take on many forms. [See the reference](../api/utils#utils.enqueueLinks)+The `options.pseudoUrls` argument is always an `Array`, but its contents can take on many forms. [See the reference](/docs/api/utils#enqueueLinks)for all of them. Since we just need to filter out same domain links, we'll keep it simple and use a pseudo-URL `string`.```js@@ -688,14 +688,14 @@ await enqueueLinks(options);> To break the pseudo-URL string down, we're looking for both `http` and `https` protocols and the links may only lead to `apify.com` domain. The> final brackets `[.*]` allow everything, so `apify.com/contact` as well as `apify.com/store` will match. If this is complex to you, we suggest-> <a href="https://www.regular-expressions.info/tutorial.html" target="_blank">reading a tutorial</a> or two on regular expression syntax.+> [reading a tutorial](https://www.regular-expressions.info/tutorial.html) or two on regular expression syntax.#### Resolving relative URLs with `enqueueLinks()`**TLDR;** Just use `baseUrl: request.loadedUrl` when working with `CheerioCrawler`.This is probably the weirdest and most complicated addition to the list. This is not the place to talk at length about-<a href="https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls" target="_blank">absolute and relative paths</a>, but in short, the+[absolute and relative paths](https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls), but in short, thelinks we encounter in a page can either be absolute, such as:```@@ -811,7 +811,7 @@ And that's it! No more parsing the links from HTML using Cheerio, filtering themgetting your data, while leaving the mundane crawling management to your tools.`Apify.utils.enqueueLinks()` has a lot more tricks up its sleeve. Make sure to check out the-[reference documentation](../api/utils#utils.enqueueLinks) to see what else it can do for you. Namely the feature to prepopulate the `Request`+[reference documentation](/docs/api/utils#enqueueLinks) to see what else it can do for you. Namely the feature to prepopulate the `Request`instances it creates with `userData` of your choice is extremely useful!## Getting some real world data@@ -821,16 +821,16 @@ instances it creates with `userData` of your choice is extremely useful!We hear you young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!-### Making a store crawler+### Making a store crawlerFortunately, we don't have to travel to a galaxy far far away to find a good candidate for learning how to scrape-structured data. The <a href="https://apify.com/store" target="_blank">Apify Store</a> is a store of public actors that anyone can grab and use. You-can find ready-made solutions for crawling <a href="https://apify.com/drobnikj/crawler-google-places" target="_blank">Google Places</a>,-<a href="https://apify.com/vaclavrut/amazon-crawler" target="_blank">Amazon</a>,-<a href="https://apify.com/apify/google-search-scraper" target="_blank">Google Search</a>,-<a href="https://apify.com/petr_cermak/booking-scraper" target="_blank">Booking</a>,-<a href="https://apify.com/jaroslavhejlek/instagram-scraper" target="_blank">Instagram</a>,-<a href="https://apify.com/petrpatek/tripadvisor-scraper" target="_blank">Tripadvisor</a> and many other websites. Feel free to check them out! It+structured data. The [Apify Store](https://apify.com/store) is a store of public actors that anyone can grab and use. You+can find ready-made solutions for crawling [Google Places](https://apify.com/drobnikj/crawler-google-places),+[Amazon](https://apify.com/vaclavrut/amazon-crawler),+[Google Search](https://apify.com/apify/google-search-scraper),+[Booking](https://apify.com/dtrungtin/booking-scraper),+[Instagram](https://apify.com/jaroslavhejlek/instagram-scraper),+[Tripadvisor](https://apify.com/maxcopell/tripadvisor) and many other websites. Feel free to check them out! Italso poses a great place to practice our jedi scraping skills since it has categories, lists and details. That's almost like our imaginary`online-store.com` from the previous chapter.@@ -838,10 +838,10 @@ also poses a great place to practice our jedi scraping skills since it has categSometimes scraping is really straightforward, but most of the times, it really pays out to do a little bit of research first. How is the websitestructured? Can I scrape it only with HTTP requests (read "with `CheerioCrawler`") or would I need a full browser solution? Are there any-anti-scraping protections in place? Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API. Jakub, one-of Apify's founders wrote a-<a href="https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c" target="_blank">great article-about all the different techniques</a> and tips and tricks so make sure to check that out!+anti-scraping protections in place? Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API. Jakub,+one of Apify's founders wrote a+[great article about all the different techniques](https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c)+and tips and tricks so make sure to check that out!For the purposes of this tutorial, let's just go ahead with HTTP requests and HTML parsing using `CheerioCrawler`. The number one reason being: Wealready know how to use it and we want to build on that knowledge to learn specific crawling and scraping techniques.@@ -964,7 +964,7 @@ const crawler = new Apify.CheerioCrawler({> code and perhaps quite a long time, if we were talking about tens of thousands or more `Requests`. The crawlers do it while running, so the time to> enqueue is spread out and you won't even notice it.-#### Sanity check+#### Sanity checkIt's always useful to create some simple boilerplate code to see that we've got everything set up correctly before we start to writethe scraping logic itself. We might realize that something in our previous analysis doesn't quite add up, or the website might not behave exactly the@@ -1479,7 +1479,7 @@ Apify.main(async () => {#### What's `Apify.pushData()`-[`Apify.pushData()`](../api/apify#apify.pushData) is a helper function that saves data to the default [`Dataset`](../api/dataset). `Dataset` is a+[`Apify.pushData()`](/docs/api/apify#pushData) is a helper function that saves data to the default [`Dataset`](/docs/api/dataset). `Dataset` is astorage designed to hold virtually unlimited amount of data in a format similar to a table. Each time you call `Apify.pushData()` a new row in thetable is created, with the property names serving as column titles.@@ -1524,7 +1524,7 @@ actor code.#### Meet the `INPUT``INPUT` is just a convention on how we call the actor's input. Because there's no magic in actors, just features, the `INPUT` is actually nothing more-than a key in the default [`KeyValueStore`](../api/keyvaluestore) that's, by convention, used as input on Apify Platform. Also by convention, the+than a key in the default [`KeyValueStore`](/docs/api/key-value-store) that's, by convention, used as input on Apify Platform. Also by convention, the`INPUT` is mostly expected to be of `Content-Type: application/json`.We will not go into `KeyValueStore` details here, but for the sake of `INPUT` you need to remember that there is a function that helps you get it.@@ -1534,7 +1534,7 @@ const input = await Apify.getInput();```On the Apify Platform, the actor's input that you can set in the Console is automatically saved to the default `KeyValueStore` under the key `INPUT`-and by calling [`Apify.getInput()`](../api/apify#apify.getValue) you retrieve the value from the `KeyValueStore`.+and by calling [`Apify.getInput()`](/docs/api/apify#getValue) you retrieve the value from the `KeyValueStore`.Running locally, you need to place an `INPUT.json` file in your default key value store for this to work.@@ -1720,7 +1720,7 @@ your logic separate. Less code in a single file means less code you need to thin#### Using `Apify.utils.log` instead of `console.log`-We wont go into great lengths here to talk about `utils.log` here, because you can read [it all in the documentation](../api/log), but there's just+We wont go into great lengths here to talk about `utils.log` here, because you can read [it all in the documentation](/docs/api/log), but there's justone thing that we need to stress: **log levels**.`utils.log` enables you to use different log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, butdiff --git a/docs/guides/motivation.md b/docs/guides/motivation.mdindex cb695544f..3d5932aab 100644--- a/docs/guides/motivation.md+++ b/docs/guides/motivation.md@@ -3,8 +3,8 @@ id: motivationtitle: Motivation----Thanks to tools like <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> or-<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>, it is easy to write Node.js code to extract data from web pages. But+Thanks to tools like [Puppeteer](https://github.com/GoogleChrome/puppeteer) or+[Cheerio](https://www.npmjs.com/package/cheerio), it is easy to write Node.js code to extract data from web pages. Buteventually things will get complicated. For example, when you try to:- Perform a deep crawl of an entire website using a persistent queue of URLs.@@ -13,7 +13,7 @@ eventually things will get complicated. For example, when you try to:- Schedule the code to run periodically and send notification on errors.- Disable browser fingerprinting protections used by websites.-Python has <a href="https://scrapy.org/" target="_blank">Scrapy</a> for these tasks, but there was no such library for **JavaScript, the language of+Python has [Scrapy](https://scrapy.org/) for these tasks, but there was no such library for **JavaScript, the language ofthe web**. The use of JavaScript is natural, since the same language is used to write the scripts as well as the data extraction code running in abrowser.diff --git a/docs/guides/puppeteer_live_view.md b/docs/guides/puppeteer_live_view.mdindex 65cd7c8a2..4847afcec 100644--- a/docs/guides/puppeteer_live_view.md+++ b/docs/guides/puppeteer_live_view.md@@ -1,5 +1,5 @@----id: puppeteerliveview+id: puppeteer-live-viewtitle: Puppeteer Live View---@@ -8,7 +8,7 @@ This is useful for debugging your crawlers that run in headless mode.The live view dashboard is run on a web server that is started on a port specified by the `APIFY_CONTAINER_PORT` environment variable (typically4321). To enable live view, pass the `useliveView: true` option to the `puppeteerPoolOptions` of-[`PuppeteerCrawler`](../api/puppeteercrawler#new_PuppeteerCrawler_new):+[`PuppeteerCrawler`](/docs/api/puppeteer-crawler#new_PuppeteerCrawler_new):```jsconst crawler = new Apify.PuppeteerCrawler({@@ -19,7 +19,7 @@ const crawler = new Apify.PuppeteerCrawler({});```-or directly to the [`PuppeteerPool`](../api/puppeteerpool) constructor, when using it standalone:+or directly to the [`PuppeteerPool`](/docs/api/puppeteer-pool) constructor, when using it standalone:```jsconst pool = new Apify.PuppeteerPool({@@ -29,7 +29,7 @@ const pool = new Apify.PuppeteerPool({```To simplify debugging, you may also want to add the `{ slowMo: 300 }` option to slow down all browser operations. See-<a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank">Puppeteer documentation</a> for details.+[Puppeteer documentation](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions) for details.Once live view is enabled, you can open `http://localhost:4321` and as the crawler runs, you should see screenshots and HTML of the pages it opensdisplayed in your browser.diff --git a/docs/guides/getting_started_quick.md b/docs/guides/quick_start.mdsimilarity index 72%rename from docs/guides/getting_started_quick.mdrename to docs/guides/quick_start.mdindex 7edd54d24..86bab0b33 100644--- a/docs/guides/getting_started_quick.md+++ b/docs/guides/quick_start.md@@ -1,12 +1,14 @@----id: quickstart+id: quick-starttitle: Quick Start----The Apify SDK requires <a href="https://nodejs.org/en/" target="_blank">Node.js</a> 10 or later.+This short tutorial will set you up to start using Apify SDK in a minute or two.+If you want to learn more, proceed to the [Getting Started](/docs/guides/getting-started)+tutorial that will take you step by step through creating your first scraper.## Local stand-alone usage-+Apify SDK requires [Node.js](https://nodejs.org/en/) 10 or later.Add Apify SDK to any Node.js project by running:```bash@@ -14,7 +16,7 @@ npm install apify --save```Run the following example to perform a recursive crawl of a website using Puppeteer. For more examples showcasing various features of the Apify SDK,-[see the Examples section of the documentation](../examples/basiccrawler).+[see the Examples section of the documentation](/docs/examples/basic-crawler).```javascriptconst Apify = require('apify');@@ -46,17 +48,16 @@ Apify.main(async () => {When you run the example, you should see Apify SDK automating a Chrome browser.-+By default, Apify SDK stores data to `./apify_storage` in the current working directory. You can override this behavior by setting either the-`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](environmentvariables) and-[Data storage](datastorage).+`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](/docs/guides/environment-variables) and+[Data storage](/docs/guides/data-storage).## Local usage with Apify command-line interface (CLI)To avoid the need to set the environment variables manually, to create a boilerplate of your project, and to enable pushing and running your code on-the <a href="https://apify.com" target="_blank">Apify Cloud</a>, you can use the-<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify command-line interface</a> (CLI) tool.+the [Apify Cloud](https://apify.com), you can use the [Apify command-line interface](https://github.com/apifytech/apify-cli) (CLI) tool.Install the CLI by running:@@ -94,13 +95,12 @@ apify push```Your script will be uploaded to the Apify Cloud and built there so that it can be run. For more information, view the-<a href="https://docs.apify.com/cli" target="_blank">Apify CLI</a> and <a href="https://docs.apify.com/actor" target="_blank">Apify Actor</a>-documentation.+[Apify Actor](https://docs.apify.com/cli) documentation.## Usage on the Apify Cloud-You can also develop your web scraping project in an online code editor directly on the <a href="https://apify.com" target="_blank">Apify Cloud</a>.-You'll need to have an Apify Account. Go to <a href="https://my.apify.com/actors" target="_blank">Actors</a>, page in the app, click <i>Create new</i>+You can also develop your web scraping project in an online code editor directly on the [Apify Cloud](https://apify.com).+You'll need to have an Apify Account. Go to [Actors](https://my.apify.com/actors), page in the app, click <i>Create new</i>and then go to the <i>Source</i> tab and start writing your code or paste one of the examples from the Examples section.-For more information, view the <a href="https://docs.apify.com/actor/quick-start" target="_blank">Apify actors quick start guide</a>.+For more information, view the [Apify actors quick start guide](https://docs.apify.com/actor/quick-start).diff --git a/docs/guides/session_management.md b/docs/guides/session_management.mdindex 0b94562cd..0f2951e79 100644--- a/docs/guides/session_management.md+++ b/docs/guides/session_management.md@@ -1,8 +1,8 @@----id: sessionmanagement+id: session-managementtitle: Session Management----[`SessionPool`](../api/sessionpool) is a class that allows you to handle the rotation of proxy IP addresses along with cookies and other custom settings in Apify SDK.+[`SessionPool`](/docs/api/sessionpool) is a class that allows you to handle the rotation of proxy IP addresses along with cookies and other custom settings in Apify SDK.The main benefit of a Session pool is that you can filter out blocked or non-working proxies,so your actor does not retry requests over known blocked/non-working proxies.@@ -13,7 +13,7 @@ which should prevent burning out a small pool of available IPs.Now let's take a look at how to use a Session pool.-**Example usage in [`PuppeteerCrawler`](../api/puppeteercrawler)**+**Example usage in [`PuppeteerCrawler`](/docs/api/puppeteer-crawler)**```javascriptconst crawler = new Apify.PuppeteerCrawler({@@ -46,7 +46,7 @@ const crawler = new Apify.PuppeteerCrawler({});```-**Example usage in [`CheerioCrawler`](../api/cheeriocrawler)**+**Example usage in [`CheerioCrawler`](/docs/api/cheerio-crawler)**```javascriptconst crawler = new Apify.CheerioCrawler({@@ -77,7 +77,7 @@ const crawler = new Apify.PuppeteerCrawler({});```-**Example usage in [`BasicCrawler`](../api/basiccrawler)**+**Example usage in [`BasicCrawler`](/docs/api/basic-crawler)**```javascriptconst crawler = new Apify.BasicCrawler({diff --git a/docs/guides/typescript_actor.md b/docs/guides/typescript_actor.mdindex b7d94bd6a..f47707556 100644--- a/docs/guides/typescript_actor.md+++ b/docs/guides/typescript_actor.md@@ -1,19 +1,19 @@----id: typescriptactor-title: Typescript Actors+id: type-script-actor+title: TypeScript Actors----Apify SDK supports Typescript by covering public APIs with type declarations. This-allows writing code with auto-completion for Typedcript and Javascript code alike.-Besides that, actors written in Typescript can take advantage of compile-time+Apify SDK supports TypeScript by covering public APIs with type declarations. This+allows writing code with auto-completion for TypeScript and JavaScript code alike.+Besides that, actors written in TypeScript can take advantage of compile-timetype-checking and avoid many mistakes.-Setting up Typescript project+Setting up a TypeScript project=============================-To use Typescript in your actors, you'll need the following prerequisities.+To use TypeScript in your actors, you'll need the following prerequisites.-1. Typescript compiler `tsc` installed somewhere:+1. TypeScript compiler `tsc` installed somewhere:```shell scriptnpm install --dev typescript@@ -21,7 +21,7 @@ To use Typescript in your actors, you'll need the following prerequisities.Type can be a development dependency in your project, as shown above. There's noneed to pollute your production environment or your system's package repository- with Typescript.+ with TypeScript.2. A build script invoking `tsc` and a correctly specified `main` entry point definedin your `package.json`:@@ -39,15 +39,13 @@ To use Typescript in your actors, you'll need the following prerequisities.you can take advantage of type-checking in all the features you'll use:```shell script- npm install --dev @types/node- npm install --dev @types/cheerio- npm install --dev @types/puppeteer+ npm install --dev @types/node @types/cheerio @types/puppeteer```- If you're using other Javascript packages in your actor, you'll want to include their+ If you're using other JavaScript packages in your actor, you'll want to include theirtype declarations too (if they are available).-4. Typescript configuration file allowing `tsc` to understand your project layout and+4. TypeScript configuration file allowing `tsc` to understand your project layout andthe features used in your project and a targeted language level:```json@@ -83,7 +81,7 @@ Auto-completion==============IDE auto-completion should work in most places. That's true even if you are writting-actors in pure Javascript. For time constraints, we left out the amendment of an+actors in pure JavaScript. For time constraints, we left out the amendment of aninternal API for the time being, and these need to be added as SDK developers writenew and enhance old code.@@ -105,11 +103,11 @@ details may be a potential priority.Problems========-Typescript sometimes generates invalid or incorrect declarations from JSDoc comments-until Typescript developers fix these problems. This needs to be handled in the future,+TypeScript sometimes generates invalid or incorrect declarations from JSDoc comments+until TypeScript developers fix these problems. This needs to be handled in the future,though all critical errors have been avoided or fixed already. Examples:-- Typescript does not handle `@extends` and `@typedef {BaseType}` as expected+- TypeScript does not handle `@extends` and `@typedef {BaseType}` as expected- In `types/session_pool/session_pool.d.ts`, it adds methods from the base-class`EventEmitter` and changes their return types, so they are not compatible with the`EventEmitter` (i.e., `addListener(): SessionPool`, instead of `addListener(): this`).diff --git a/docs/guides/what_is_an_actor.md b/docs/guides/what_is_an_actor.mdindex 82f01103e..2d7fa7bd5 100644--- a/docs/guides/what_is_an_actor.md+++ b/docs/guides/what_is_an_actor.md@@ -1,20 +1,20 @@----id: whatisanactor+id: what-is-an-actortitle: What is an Actor----When you deploy your script to the <a href="https://apify.com/" target="_blank">Apify Cloud</a>, it becomes an actor. An actor is a serverless+When you deploy your script to the [Apify Cloud](https://apify.com/), it becomes an actor. An actor is a serverlessmicroservice that accepts an input and produces an output. It can run for a few seconds, hours or even infinitely. An actor can perform anything froma simple action such as filling out a web form or sending an email, to complex operations such as crawling an entire website and removing duplicatesfrom a large dataset.-To run an actor, you need to have an <a href="https://my.apify.com/" target="_blank">Apify Account</a>. Actors can be shared in the-<a href="https://apify.com/store" target="_blank">Apify Store</a> so that other people can use them. But don't worry, if you share your actor in the+To run an actor, you need to have an [Apify Account](https://my.apify.com/). Actors can be shared in the+[Apify Store](https://apify.com/store) so that other people can use them. But don't worry, if you share your actor in thestore and somebody uses it, it runs under their account, not yours.**Related links**-- <a href="https://apify.com/store" target="_blank">Store of existing actors</a>-- <a href="https://docs.apify.com/actor" target="_blank">Documentation</a>-- <a href="https://my.apify.com/actors" target="_blank">View actors in Apify app</a>-- <a href="https://apify.com/docs/api/v2#/reference/actors" target="_blank">API reference</a>+- [Store of existing actors](https://apify.com/store)+- [Documentation](https://docs.apify.com/actor)+- [View actors in Apify app](https://my.apify.com/actors)+- [API reference](https://apify.com/docs/api/v2#/reference/actors)diff --git a/docs/readme/introduction.md b/docs/readme/introduction.mdindex 33e50003a..5156de2a1 100644--- a/docs/readme/introduction.md+++ b/docs/readme/introduction.md@@ -9,9 +9,9 @@ Apify SDK simplifies the development of web crawlers, scrapers, data extractorsIt provides tools to manage and automatically scale a pool of headless Chrome / Puppeteer instances,to maintain queues of URLs to crawl, store crawling results to a local filesystem or into the cloud,rotate proxies and much more.-The SDK is available as the <a href="https://www.npmjs.com/package/apify" target="_blank"><code>apify</code></a> NPM package.+The SDK is available as the [`apify`](https://www.npmjs.com/package/apify) NPM package.It can be used either stand-alone in your own applications-or in <a href="https://docs.apify.com/actor" target="_blank">actors</a>-running on the <a href="https://apify.com/" target="_blank">Apify Cloud</a>.+or in [actors](https://docs.apify.com/actor)+running on the [Apify Cloud](https://apify.com/).-**View full documentation, guides and examples on the dedicated <a href="https://sdk.apify.com" target="_blank">Apify SDK project website</a>**+**View full documentation, guides and examples on the dedicated [Apify SDK project website](https://sdk.apify.com)**diff --git a/docs/readme/overview.md b/docs/readme/overview.mdindex 16a42ccf1..79f7f0dbd 100644--- a/docs/readme/overview.md+++ b/docs/readme/overview.md@@ -1,76 +1,46 @@## Overview-The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify"><code>apify</code></a> NPM package and it provides the following tools:+The Apify SDK is available as the [`apify`](https://www.npmjs.com/package/apify) NPM package and it provides the following tools:-<ul>- <li>- <a href="https://sdk.apify.com/docs/api/basiccrawler"><code>BasicCrawler</code></a>- - Provides a simple framework for the parallel crawling of web pages- whose URLs are fed either from a static list or from a dynamic queue of URLs.- This class serves as a base for more complex crawlers (see below).- </li>- <li>- <a href="https://sdk.apify.com/docs/api/cheeriocrawler"><code>CheerioCrawler</code></a>- - Enables the parallel crawling of a large number of web pages- using the <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a>- HTML parser.- This is the most efficient web crawler, but it does not work on websites that require JavaScript.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/puppeteercrawler"><code>PuppeteerCrawler</code></a>- - Enables the parallel crawling of a large number of web pages using the headless Chrome browser- and <a href="https://github.com/GoogleChrome/puppeteer">Puppeteer</a>.- The pool of Chrome browsers is automatically scaled up and down based on available system resources.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/puppeteerpool"><code>PuppeteerPool</code></a>- - Provides web browser tabs for user jobs- from an automatically-managed pool of Chrome browser instances, with configurable browser recycling and retirement policies.- Supports reuse of the disk cache to speed up the crawling of websites and reduce proxy bandwidth.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/requestlist"><code>RequestList</code></a>- - Represents a list of URLs to crawl. The URLs can be passed in code or in a text file hosted on the web.- The list persists its state so that crawling can resume- when the Node.js process restarts.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/requestqueue"><code>RequestQueue</code></a>- - Represents a queue of URLs to crawl, which is stored either on a local filesystem or in the <a href="https://apify.com" target="_blank">Apify Cloud</a>.- The queue is used for deep crawling of websites, where you start with- several URLs and then recursively follow links to other pages.- The data structure supports both breadth-first and depth-first crawling orders.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/dataset"><code>Dataset</code></a>- - Provides a store for structured data and enables their- export to formats like JSON, JSONL, CSV, XML, Excel or HTML.- The data is stored on a local filesystem or in the Apify Cloud.- Datasets are useful for storing and sharing large tabular crawling results,- such as a list of products or real estate offers.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/keyvaluestore"><code>KeyValueStore</code></a>- - A simple key-value store for arbitrary data records or files, along with their MIME content type.- It is ideal for saving screenshots of web pages, PDFs or to persist the state of your crawlers.- The data is stored on a local filesystem or in the Apify Cloud.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/autoscaledpool"><code>AutoscaledPool</code></a>- - Runs asynchronous background tasks, while automatically adjusting the concurrency- based on free system memory and CPU usage. This is useful for running web scraping tasks- at the maximum capacity of the system.- </li>- <li>- <a href="https://sdk.apify.com/docs/api/puppeteer"><code>Puppeteer Utils</code></a>- - Provides several helper functions useful for web scraping. For example, to inject jQuery into web pages- or to hide browser origin.- </li>- <li>- Additionally, the package provides various helper functions to simplify- running your code on the Apify Cloud and thus- take advantage of its pool of proxies, job scheduler, data storage, etc.- For more information,- see the <a href="https://sdk.apify.com">Apify SDK Programmer's Reference</a>.- </li>-</ul>+- [`BasicCrawler`](https://sdk.apify.com/docs/api/basic-crawler) - Provides a simple framework for the parallel+crawling of web pages whose URLs are fed either from a static list or from a dynamic queue of URLs. This class+serves as a base for more complex crawlers (see below).++- [`CheerioCrawler`](https://sdk.apify.com/docs/api/cheerio-crawler) - Enables the parallel crawling of a large+number of web pages using the [cheerio](https://www.npmjs.com/package/cheerio) HTML parser. This is the most+efficient web crawler, but it does not work on websites that require JavaScript.++- [`PuppeteerCrawler`](https://sdk.apify.com/docs/api/puppeteer-crawler) - Enables the parallel crawling of+a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/GoogleChrome/puppeteer).+The pool of Chrome browsers is automatically scaled up and down based on available system resources.++- [`PuppeteerPool`](https://sdk.apify.com/docs/api/puppeteer-pool) - Provides web browser tabs for user jobs+from an automatically-managed pool of Chrome browser instances, with configurable browser recycling+and retirement policies. Supports reuse of the disk cache to speed up the crawling of websites and reduce proxy bandwidth.++- [`RequestList`](https://sdk.apify.com/docs/api/request-list) - Represents a list of URLs to crawl.+The URLs can be passed in code or in a text file hosted on the web. The list persists its state so that crawling+can resume when the Node.js process restarts.++- [`RequestQueue`](https://sdk.apify.com/docs/api/request-queue) - Represents a queue of URLs to crawl,+which is stored either on a local filesystem or in the [Apify Cloud](https://apify.com). The queue is used+for deep crawling of websites, where you start with several URLs and then recursively follow links to other pages.+The data structure supports both breadth-first and depth-first crawling orders.++- [`Dataset`](https://sdk.apify.com/docs/api/dataset) - Provides a store for structured data and enables their export+to formats like JSON, JSONL, CSV, XML, Excel or HTML. The data is stored on a local filesystem or in the Apify Cloud.+Datasets are useful for storing and sharing large tabular crawling results, such as a list of products or real estate offers.++- [`KeyValueStore`](https://sdk.apify.com/docs/api/key-value-store) - A simple key-value store for arbitrary data+records or files, along with their MIME content type. It is ideal for saving screenshots of web pages, PDFs+or to persist the state of your crawlers. The data is stored on a local filesystem or in the Apify Cloud.++- [`AutoscaledPool`](https://sdk.apify.com/docs/api/autoscaled-pool) - Runs asynchronous background tasks,+while automatically adjusting the concurrency based on free system memory and CPU usage. This is useful for running+web scraping tasks at the maximum capacity of the system.++- [`Puppeteer Utils`](https://sdk.apify.com/docs/api/puppeteer) - Provides several helper functions useful+for web scraping. For example, to inject jQuery into web pages or to hide browser origin. Additionally,+the package provides various helper functions to simplify running your code on the Apify Cloud and thus+take advantage of its pool of proxies, job scheduler, data storage, etc. For more information,+see the [Apify SDK Programmer's Reference](https://sdk.apify.com).diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.mdindex 4e8b4870e..a286cd18e 100644--- a/docs/typedefs/ActorRun.md+++ b/docs/typedefs/ActorRun.md@@ -1,131 +1,181 @@----id: actorrun+id: actor-runtitle: ActorRun----<a name="ActorRun"></a>--Represents information about an actor run, as returned by the [`Apify.call()`](../api/apify#module_Apify.call) or-[`Apify.callTask()`](../api/apify#module_Apify.callTask) function. The object is almost equivalent to the JSON response of the-<a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a> Apify API endpoint and extended with-certain fields. For more details, see <a href="https://docs.apify.com/actor/run" target="_blank">Runs.</a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>id</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Actor run ID</p>-</td></tr><tr>-<td><code>actId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Actor ID</p>-</td></tr><tr>-<td><code>startedAt</code></td><td><code>Date</code></td>-</tr>-<tr>-<td colspan="3"><p>Time when the actor run started</p>-</td></tr><tr>-<td><code>finishedAt</code></td><td><code>Date</code></td>-</tr>-<tr>-<td colspan="3"><p>Time when the actor run finished. Contains <code>null</code> for running actors.</p>-</td></tr><tr>-<td><code>status</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Status of the run. For possible values, see- <a href="https://docs.apify.com/actor/run#lifecycle" target="_blank">Run lifecycle</a>- in Apify actor documentation.</p>-</td></tr><tr>-<td><code>meta</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Actor run meta-data. For example:</p>-<pre><code> {- "origin": "API",- "clientIp": "1.2.3.4",- "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"- }</code></pre></td></tr><tr>-<td><code>stats</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>An object containing various actor run statistics. For example:</p>-<pre><code> {- "inputBodyLen": 22,- "restartCount": 0,- "workersUsed": 1,- }</code></pre><p> Beware that object fields might change in future releases.</p>-</td></tr><tr>-<td><code>options</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Actor run options. For example:</p>-<pre><code> {- "build": "latest",- "waitSecs": 0,- "memoryMbytes": 256,- "diskMbytes": 512- }</code></pre></td></tr><tr>-<td><code>buildId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the actor build used for the run. For details, see- <a href="https://docs.apify.com/actor/build" target="_blank">Builds</a>- in Apify actor documentation.</p>-</td></tr><tr>-<td><code>buildNumber</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Number of the actor build used for the run. For example, <code>0.0.10</code>.</p>-</td></tr><tr>-<td><code>exitCode</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Exit code of the actor run process. It's <code>null</code> if actor is still running.</p>-</td></tr><tr>-<td><code>defaultKeyValueStoreId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the default key-value store associated with the actor run. See <a href="../api/keyvaluestore"><code>KeyValueStore</code></a> for details.</p>-</td></tr><tr>-<td><code>defaultDatasetId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the default dataset associated with the actor run. See <a href="../api/dataset"><code>Dataset</code></a> for details.</p>-</td></tr><tr>-<td><code>defaultRequestQueueId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the default request queue associated with the actor run. See <a href="../api/requestqueue"><code>RequestQueue</code></a> for details.</p>-</td></tr><tr>-<td><code>containerUrl</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>URL on which the web server running inside actor run's Docker container can be accessed.- For more details, see- <a href="https://docs.apify.com/actor/run#container-web-server" target="_blank">Container web server</a>- in Apify actor documentation.</p>-</td></tr><tr>-<td><code>output</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Contains output of the actor run. The value is <code>null</code> or <code>undefined</code> in case the actor is still running,- or if you pass <code>false</code> to the <code>fetchOutput</code> option of <a href="../api/apify#module_Apify.call"><code>Apify.call()</code></a>.</p>-<p> For example:</p>-<pre><code> {- "contentType": "application/json; charset=utf-8",- "body": {- "message": "Hello world!"- }- }</code></pre></td></tr></tbody>-</table>+<a name="actorrun"></a>++Represents information about an actor run, as returned by the [`Apify.call()`](/docs/api/apify#call) or [`Apify.callTask()`](/docs/api/apify#calltask)+function. The object is almost equivalent to the JSON response of the+[Actor run](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor) Apify API endpoint and extended with certain fields. For more+details, see [Runs.](https://docs.apify.com/actor/run)++## Properties++### `id`++**Type**: `String`++Actor run ID++---++### `actId`++**Type**: `String`++Actor ID++---++### `startedAt`++**Type**: `Date`++Time when the actor run started++---++### `finishedAt`++**Type**: `Date`++Time when the actor run finished. Contains `null` for running actors.++---++### `status`++**Type**: `String`++Status of the run. For possible values, see [Run lifecycle](https://docs.apify.com/actor/run#lifecycle) in Apify actor documentation.++---++### `meta`++**Type**: `Object`++Actor run meta-data. For example:++```+{+ "origin": "API",+ "clientIp": "1.2.3.4",+ "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"+}+```++---++### `stats`++**Type**: `Object`++An object containing various actor run statistics. For example:++```+{+ "inputBodyLen": 22,+ "restartCount": 0,+ "workersUsed": 1,+}+```++Beware that object fields might change in future releases.++---++### `options`++**Type**: `Object`++Actor run options. For example:++```+{+ "build": "latest",+ "waitSecs": 0,+ "memoryMbytes": 256,+ "diskMbytes": 512+}+```++---++### `buildId`++**Type**: `String`++ID of the actor build used for the run. For details, see [Builds](https://docs.apify.com/actor/build) in Apify actor documentation.++---++### `buildNumber`++**Type**: `String`++Number of the actor build used for the run. For example, `0.0.10`.++---++### `exitCode`++**Type**: `Number`++Exit code of the actor run process. It's `null` if actor is still running.++---++### `defaultKeyValueStoreId`++**Type**: `String`++ID of the default key-value store associated with the actor run. See [`KeyValueStore`](/docs/api/key-value-store) for details.++---++### `defaultDatasetId`++**Type**: `String`++ID of the default dataset associated with the actor run. See [`Dataset`](/docs/api/dataset) for details.++---++### `defaultRequestQueueId`++**Type**: `String`++ID of the default request queue associated with the actor run. See [`RequestQueue`](/docs/api/request-queue) for details.++---++### `containerUrl`++**Type**: `String`++URL on which the web server running inside actor run's Docker container can be accessed. For more details, see+[Container web server](https://docs.apify.com/actor/run#container-web-server) in Apify actor documentation.++---++### `output`++**Type**: `Object`++Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running, or if you pass `false` to the `fetchOutput`+option of [`Apify.call()`](/docs/api/apify#call).++For example:++```+{+ "contentType": "application/json; charset=utf-8",+ "body": {+ "message": "Hello world!"+ }+}+```++---diff --git a/docs/typedefs/ApifyCallError.md b/docs/typedefs/ApifyCallError.mdindex f4154d2e8..d822042e3 100644--- a/docs/typedefs/ApifyCallError.md+++ b/docs/typedefs/ApifyCallError.md@@ -1,35 +1,34 @@----id: apifycallerror+id: apify-call-errortitle: ApifyCallError----<a name="ApifyCallError"></a>--The class represents exceptions thrown by the [`Apify.call()`](../api/apify#module_Apify.call) function.--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>message</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Error message</p>-</td></tr><tr>-<td><code>name</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Contains <code>"ApifyCallError"</code></p>-</td></tr><tr>-<td><code>run</code></td><td><code><a href="../typedefs/actorrun">ActorRun</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Object representing the failed actor run.</p>-</td></tr></tbody>-</table>+<a name="apifycallerror"></a>++The class represents exceptions thrown by the [`Apify.call()`](/docs/api/apify#call) function.++## Properties++### `message`++**Type**: `String`++Error message++---++### `name`++**Type**: `String`++Contains `"ApifyCallError"`++---++### `run`++**Type**: [`ActorRun`](/docs/typedefs/actor-run)++Object representing the failed actor run.++---diff --git a/docs/typedefs/ApifyEnv.md b/docs/typedefs/ApifyEnv.mdindex 52867afe0..ed5647609 100644--- a/docs/typedefs/ApifyEnv.md+++ b/docs/typedefs/ApifyEnv.md@@ -1,75 +1,90 @@----id: apifyenv+id: apify-envtitle: ApifyEnv----<a name="ApifyEnv"></a>+<a name="apifyenv"></a>Parsed representation of the `APIFY_XXX` environmental variables.-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>actorId</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the actor (APIFY_ACTOR_ID)</p>-</td></tr><tr>-<td><code>actorRunId</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the actor run (APIFY_ACTOR_RUN_ID)</p>-</td></tr><tr>-<td><code>actorTaskId</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the actor task (APIFY_ACTOR_TASK_ID)</p>-</td></tr><tr>-<td><code>userId</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the user who started the actor - note that it might be- different than the owner ofthe actor (APIFY_USER_ID)</p>-</td></tr><tr>-<td><code>token</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>Authentication token representing privileges given to the actor run,- it can be passed to various Apify APIs (APIFY_TOKEN)</p>-</td></tr><tr>-<td><code>startedAt</code></td><td><code>Date</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>Date when the actor was started (APIFY_STARTED_AT)</p>-</td></tr><tr>-<td><code>timeoutAt</code></td><td><code>Date</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>Date when the actor will time out (APIFY_TIMEOUT_AT)</p>-</td></tr><tr>-<td><code>defaultKeyValueStoreId</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the key-value store where input and output data of this- actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)</p>-</td></tr><tr>-<td><code>defaultDatasetId</code></td><td><code>String</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>ID of the dataset where input and output data of this- actor is stored (APIFY_DEFAULT_DATASET_ID)</p>-</td></tr><tr>-<td><code>memoryMbytes</code></td><td><code>Number</code> | <code>null</code></td>-</tr>-<tr>-<td colspan="3"><p>Amount of memory allocated for the actor,- in megabytes (APIFY_MEMORY_MBYTES)</p>-</td></tr></tbody>-</table>+## Properties++### `actorId`++**Type**: `String` | `null`++ID of the actor (APIFY_ACTOR_ID)++---++### `actorRunId`++**Type**: `String` | `null`++ID of the actor run (APIFY_ACTOR_RUN_ID)++---++### `actorTaskId`++**Type**: `String` | `null`++ID of the actor task (APIFY_ACTOR_TASK_ID)++---++### `userId`++**Type**: `String` | `null`++ID of the user who started the actor - note that it might be different than the owner ofthe actor (APIFY_USER_ID)++---++### `token`++**Type**: `String` | `null`++Authentication token representing privileges given to the actor run, it can be passed to various Apify APIs (APIFY_TOKEN)++---++### `startedAt`++**Type**: `Date` | `null`++Date when the actor was started (APIFY_STARTED_AT)++---++### `timeoutAt`++**Type**: `Date` | `null`++Date when the actor will time out (APIFY_TIMEOUT_AT)++---++### `defaultKeyValueStoreId`++**Type**: `String` | `null`++ID of the key-value store where input and output data of this actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)++---++### `defaultDatasetId`++**Type**: `String` | `null`++ID of the dataset where input and output data of this actor is stored (APIFY_DEFAULT_DATASET_ID)++---++### `memoryMbytes`++**Type**: `Number` | `null`++Amount of memory allocated for the actor, in megabytes (APIFY_MEMORY_MBYTES)++---diff --git a/docs/typedefs/AutoscaledPoolOptions.md b/docs/typedefs/AutoscaledPoolOptions.mdindex 597246e43..651e9424b 100644--- a/docs/typedefs/AutoscaledPoolOptions.md+++ b/docs/typedefs/AutoscaledPoolOptions.md@@ -1,109 +1,132 @@----id: autoscaledpooloptions+id: autoscaled-pool-optionstitle: AutoscaledPoolOptions----<a name="AutoscaledPoolOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>runTaskFunction</code></td><td><code>function</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function that performs an asynchronous resource-intensive task.- The function must either be labeled <code>async</code> or return a promise.</p>-</td></tr><tr>-<td><code>isTaskReadyFunction</code></td><td><code>function</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function that indicates whether <code>runTaskFunction</code> should be called.- This function is called every time there is free capacity for a new task and it should- indicate whether it should start a new task or not by resolving to either <code>true</code> or <code>false</code>.- Besides its obvious use, it is also useful for task throttling to save resources.</p>-</td></tr><tr>-<td><code>isFinishedFunction</code></td><td><code>function</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function that is called only when there are no tasks to be processed.- If it resolves to <code>true</code> then the pool's run finishes. Being called only- when there are no tasks being processed means that as long as <code>isTaskReadyFunction()</code>- keeps resolving to <code>true</code>, <code>isFinishedFunction()</code> will never be called.- To abort a run, use the <a href="#AutoscaledPool+abort"><code>abort()</code></a> method.</p>-</td></tr><tr>-<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>The minimum number of tasks running in parallel.</p>-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.- If you're not sure, just keep the default value and the concurrency will scale up automatically.</p>-</td></tr><tr>-<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>-</tr>-<tr>-<td colspan="3"><p>The maximum number of tasks running in parallel.</p>-</td></tr><tr>-<td><code>[desiredConcurrency]</code></td><td><code>Number</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>The desired number of tasks that should be running parallel on the start of the pool,- if there is a large enough supply of them.- By default, it is <code>minConcurrency</code>.</p>-</td></tr><tr>-<td><code>[desiredConcurrencyRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>-</tr>-<tr>-<td colspan="3"><p>Minimum level of desired concurrency to reach before more scaling up is allowed.</p>-</td></tr><tr>-<td><code>[scaleUpStepRatio]</code></td><td><code>Number</code></td><td><code>0.05</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the fractional amount of desired concurrency to be added with each scaling up.- The minimum scaling step is one.</p>-</td></tr><tr>-<td><code>[scaleDownStepRatio]</code></td><td><code>Number</code></td><td><code>0.05</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the amount of desired concurrency to be subtracted with each scaling down.- The minimum scaling step is one.</p>-</td></tr><tr>-<td><code>[maybeRunIntervalSecs]</code></td><td><code>Number</code></td><td><code>0.5</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how often the pool should call the <code>runTaskFunction()</code> to start a new task, in seconds.- This has no effect on starting new tasks immediately after a task completes.</p>-</td></tr><tr>-<td><code>[loggingIntervalSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Specifies a period in which the instance logs its state, in seconds.- Set to <code>null</code> to disable periodic logging.</p>-</td></tr><tr>-<td><code>[autoscaleIntervalSecs]</code></td><td><code>Number</code></td><td><code>10</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines in seconds how often the pool should attempt to adjust the desired concurrency- based on the latest system status. Setting it lower than 1 might have a severe impact on performance.- We suggest using a value from 5 to 20.</p>-</td></tr><tr>-<td><code>[snapshotterOptions]</code></td><td><code><a href="../typedefs/snapshotteroptions">SnapshotterOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Options to be passed down to the <a href="snapshotter"><code>Snapshotter</code></a> constructor. This is useful for fine-tuning- the snapshot intervals and history.</p>-</td></tr><tr>-<td><code>[systemStatusOptions]</code></td><td><code><a href="../typedefs/systemstatusoptions">SystemStatusOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Options to be passed down to the <a href="systemstatus"><code>SystemStatus</code></a> constructor. This is useful for fine-tuning- the system status reports. If a custom snapshotter is set in the options, it will be used- by the pool.</p>-</td></tr></tbody>-</table>+<a name="autoscaledpooloptions"></a>++## Properties++### `runTaskFunction`++**Type**: `function`++A function that performs an asynchronous resource-intensive task. The function must either be labeled `async` or return a promise.++---++### `isTaskReadyFunction`++**Type**: `function`++A function that indicates whether `runTaskFunction` should be called. This function is called every time there is free capacity for a new task and it+should indicate whether it should start a new task or not by resolving to either `true` or `false`. Besides its obvious use, it is also useful for+task throttling to save resources.++---++### `isFinishedFunction`++**Type**: `function`++A function that is called only when there are no tasks to be processed. If it resolves to `true` then the pool's run finishes. Being called only when+there are no tasks being processed means that as long as `isTaskReadyFunction()` keeps resolving to `true`, `isFinishedFunction()` will never be+called. To abort a run, use the [`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort) method.++---++### `minConcurrency`++**Type**: `Number` <code> = 1</code>++The minimum number of tasks running in parallel.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash. If you're+not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `Number` <code> = 1000</code>++The maximum number of tasks running in parallel.++---++### `desiredConcurrency`++**Type**: `Number`++The desired number of tasks that should be running parallel on the start of the pool, if there is a large enough supply of them. By default, it is+`minConcurrency`.++---++### `desiredConcurrencyRatio`++**Type**: `Number` <code> = 0.95</code>++Minimum level of desired concurrency to reach before more scaling up is allowed.++---++### `scaleUpStepRatio`++**Type**: `Number` <code> = 0.05</code>++Defines the fractional amount of desired concurrency to be added with each scaling up. The minimum scaling step is one.++---++### `scaleDownStepRatio`++**Type**: `Number` <code> = 0.05</code>++Defines the amount of desired concurrency to be subtracted with each scaling down. The minimum scaling step is one.++---++### `maybeRunIntervalSecs`++**Type**: `Number` <code> = 0.5</code>++Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds. This has no effect on starting new tasks immediately+after a task completes.++---++### `loggingIntervalSecs`++**Type**: `Number` <code> = 60</code>++Specifies a period in which the instance logs its state, in seconds. Set to `null` to disable periodic logging.++---++### `autoscaleIntervalSecs`++**Type**: `Number` <code> = 10</code>++Defines in seconds how often the pool should attempt to adjust the desired concurrency based on the latest system status. Setting it lower than 1+might have a severe impact on performance. We suggest using a value from 5 to 20.++---++### `snapshotterOptions`++**Type**: [`SnapshotterOptions`](/docs/typedefs/snapshotter-options)++Options to be passed down to the [`Snapshotter`](/docs/api/snapshotter) constructor. This is useful for fine-tuning the snapshot intervals and+history.++---++### `systemStatusOptions`++**Type**: [`SystemStatusOptions`](/docs/typedefs/system-status-options)++Options to be passed down to the [`SystemStatus`](/docs/api/system-status) constructor. This is useful for fine-tuning the system status reports. If a+custom snapshotter is set in the options, it will be used by the pool.++---diff --git a/docs/typedefs/BasicCrawlerOptions.md b/docs/typedefs/BasicCrawlerOptions.mdindex ccce17c14..95f9354ec 100644--- a/docs/typedefs/BasicCrawlerOptions.md+++ b/docs/typedefs/BasicCrawlerOptions.md@@ -1,111 +1,146 @@----id: basiccrawleroptions+id: basic-crawler-optionstitle: BasicCrawlerOptions----<a name="BasicCrawlerOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>handleRequestFunction</code></td><td><code><a href="../typedefs/handlerequest">HandleRequest</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each URL to crawl.</p>-<p> The function receives the following object as an argument:</p>-<pre><code>{+<a name="basiccrawleroptions"></a>++## Properties++### `handleRequestFunction`++**Type**: [`HandleRequest`](/docs/typedefs/handle-request)++User-provided function that performs the logic of the crawler. It is called for each URL to crawl.++The function receives the following object as an argument:++```+{request: Request,autoscaledPool: AutoscaledPool-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>-<p> The function must return a promise, which is then awaited by the crawler.</p>-<p> If the function throws an exception, the crawler will try to re-crawl the- request later, up to <code>option.maxRequestRetries</code> times.- If all the retries fail, the crawler calls the function- provided to the <code>handleFailedRequestFunction</code> parameter.- To make this work, you should <strong>always</strong>- let your function throw exceptions rather than catch them.- The exceptions are logged to the request using the- <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>-</td></tr><tr>-<td><code>[requestList]</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Static list of URLs to be processed.- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>-</td></tr><tr>-<td><code>[requestQueue]</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>-</td></tr><tr>-<td><code>[handleRequestTimeoutSecs]</code></td><td><code>number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Timeout in which the function passed as <code>handleRequestFunction</code> needs to finish, in seconds.</p>-</td></tr><tr>-<td><code>[handleFailedRequestFunction]</code></td><td><code><a href="../typedefs/handlefailedrequest">HandleFailedRequest</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>-<p> The function receives the following object as an argument:</p>-<pre><code>{+}+```++where the [`Request`](/docs/api/request) instance represents the URL to crawl.++The function must return a promise, which is then awaited by the crawler.++If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your+function throw exceptions rather than catch them. The exceptions are logged to the request using the+[`Request.pushErrorMessage()`](/docs/api/request#pusherrormessage) function.++---++### `requestList`++**Type**: [`RequestList`](/docs/api/request-list)++Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).++---++### `requestQueue`++**Type**: [`RequestQueue`](/docs/api/request-queue)++Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be+provided (or both).++---++### `handleRequestTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.++---++### `handleFailedRequestFunction`++**Type**: [`HandleFailedRequest`](/docs/typedefs/handle-failed-request)++A function to handle requests that failed more than `option.maxRequestRetries` times.++The function receives the following object as an argument:++```+{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance- represents the last error thrown during processing of the request.</p>-<p> See- <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>- for the default implementation of this function.</p>-</td></tr><tr>-<td><code>[maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how many times the request is retried if <a href="#new_BasicCrawler_new"><code>handleRequestFunction()</code></a> fails.</p>-</td></tr><tr>-<td><code>[maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.- Always set this value in order to prevent infinite loops in misconfigured crawlers.- Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>-</td></tr><tr>-<td><code>[autoscaledPoolOptions]</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.- Note that the <code>runTaskFunction</code> and <code>isTaskReadyFunction</code> options- are provided by <code>BasicCrawler</code> and cannot be overridden.- However, you can provide a custom implementation of <code>isFinishedFunction</code>.</p>-</td></tr><tr>-<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.- If you're not sure, just keep the default value and the concurrency will scale up automatically.</p>-</td></tr><tr>-<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>-</td></tr><tr>-<td><code>[useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to true. Basic crawler will initialize the <a href="sessionpool"><code>SessionPool</code></a> with the corresponding <code>sessionPoolOptions</code>.- The session instance will be than available in the <code>handleRequestFunction</code>.</p>-</td></tr><tr>-<td><code>[sessionPoolOptions]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>The configuration options for {SessionPool} to use.</p>-</td></tr></tbody>-</table>+}+```++where the [`Request`](/docs/api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during+processing of the request.++See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11) for the default implementation of this+function.++---++### `maxRequestRetries`++**Type**: `Number` <code> = 3</code>++Indicates how many times the request is retried if+[`BasicCrawlerOptions.handleRequestFunction`](/docs/typedefs/basic-crawler-options#handlerequestfunction) fails.++---++### `maxRequestsPerCrawl`++**Type**: `Number`++Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.++---++### `autoscaledPoolOptions`++**Type**: [`AutoscaledPoolOptions`](/docs/typedefs/autoscaled-pool-options)++Custom options passed to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) constructor. Note that the `runTaskFunction` and+`isTaskReadyFunction` options are provided by `BasicCrawler` and cannot be overridden. However, you can provide a custom implementation of+`isFinishedFunction`.++---++### `minConcurrency`++**Type**: `Number` <code> = 1</code>++Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If+you're not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `Number` <code> = 1000</code>++Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.++---++### `useSessionPool`++**Type**: `Boolean` <code> = false</code>++If set to true. Basic crawler will initialize the [`SessionPool`](/docs/api/session-pool) with the corresponding `sessionPoolOptions`. The session+instance will be than available in the `handleRequestFunction`.++---++### `sessionPoolOptions`++**Type**: [`SessionPoolOptions`](/docs/typedefs/session-pool-options)++The configuration options for {SessionPool} to use.++---diff --git a/docs/typedefs/CheerioCrawlerOptions.md b/docs/typedefs/CheerioCrawlerOptions.mdindex 06683f7bf..c09f4299f 100644--- a/docs/typedefs/CheerioCrawlerOptions.md+++ b/docs/typedefs/CheerioCrawlerOptions.md@@ -1,228 +1,301 @@----id: cheeriocrawleroptions+id: cheerio-crawler-optionstitle: CheerioCrawlerOptions----<a name="CheerioCrawlerOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>handlePageFunction</code></td><td><code><a href="../typedefs/cheeriohandlepage">CheerioHandlePage</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each page- loaded and parsed by the crawler.</p>-<p> The function receives the following object as an argument:</p>-<pre><code class="language-javascript">{- // The Cheerio object's function with the parsed HTML.+<a name="cheeriocrawleroptions"></a>++## Properties++### `handlePageFunction`++**Type**: [`CheerioHandlePage`](/docs/typedefs/cheerio-handle-page)++User-provided function that performs the logic of the crawler. It is called for each page loaded and parsed by the crawler.++The function receives the following object as an argument:++```+{+ // The Cheerio object's function with the parsed HTML.$: Cheerio,-// The request body of the web page, whose type depends on the content type. body: String|Buffer,--// The parsed object from JSON for responses with the "application/json" content types. // For other content types it's null. json:-Object,--// Apify.Request object with details of the requested web page request: Request,--// Parsed Content-Type HTTP header: { type, encoding } contentType: Object,--// An instance of Node's http.IncomingMessage object, response: Object,--// Underlying AutoscaledPool instance used to manage the concurrency of crawler autoscaledPool: AutoscaledPool,--// Session object, useful to work around anti-scraping protections session: Session }</code></pre>--<p> Type of <code>body</code> depends on the <code>Content-Type</code> header of the web page:</p>-<ul>-<li><p>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> MIME content types</p>-</li>-<li><p>Buffer for others MIME content types</p>-<p>Parsed <code>Content-Type</code> header using-<a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>-is stored in <code>contentType</code>.</p>-<p>Cheerio is available only for HTML and XML content types.</p>-<p>With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>-<p>If the function returns, the returned promise is awaited by the crawler.</p>-<p>If the function throws an exception, the crawler will try to re-crawl the-request later, up to <code>option.maxRequestRetries</code> times.-If all the retries fail, the crawler calls the function-provided to the <code>handleFailedRequestFunction</code> parameter.-To make this work, you should <strong>always</strong>-let your function throw exceptions rather than catch them.-The exceptions are logged to the request using the-<a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>-</li>-</ul>-</td></tr><tr>-<td><code>[requestList]</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Static list of URLs to be processed.- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>-</td></tr><tr>-<td><code>[requestQueue]</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>-</td></tr><tr>-<td><code>[requestOptions]</code></td><td><code><a href="../typedefs/requestasbrowseroptions">RequestAsBrowserOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Represents the options passed to the <a href="requestasbrowser"><code>requestAsBrowser</code></a> function that makes the HTTP requests to fetch the web pages.- Provided <code>requestOptions</code> are added to internal defaults that cannot be overridden to ensure- the operation of <code>CheerioCrawler</code> and all its options. Headers will not be merged,- use <a href="requestlist"><code>RequestList</code></a> and/or <a href="requestqueue"><code>RequestQueue</code></a> to initialize your <a href="request"><code>Request</code></a> with the- correct headers or use <code>prepareRequestFunction</code> to modify your <a href="request"><code>Request</code></a> dynamically.- If you need more granular control over your requests, use <a href="basiccrawler"><code>BasicCrawler</code></a>.</p>-<p> The mandatory internal defaults that <strong>CANNOT BE OVERRIDDEN</strong> by <code>requestOptions</code>:</p>-<pre><code> {- url, // Provided by RequestList and/or RequestQueue- method, // Provided by RequestList and/or RequestQueue- headers, // Provided by RequestList and/or RequestQueue- payload, // Provided by RequestList and/or RequestQueue- strictSSL, // Use ignoreSslErrors- proxy, // Use useApifyProxy or proxyUrls- }</code></pre></td></tr><tr>-<td><code>[prepareRequestFunction]</code></td><td><code><a href="../typedefs/preparerequest">PrepareRequest</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function that executes before the HTTP request is made to the target resource.- This function is suitable for setting dynamic properties such as cookies to the <a href="request"><code>Request</code></a>.</p>-<p> The function receives the following object as an argument:</p>-<pre><code>{+ // The request body of the web page, whose type depends on the content type.+ body: String|Buffer,++ // The parsed object from JSON for responses with the "application/json" content types.+ // For other content types it's null.+ json: Object,++ // Apify.Request object with details of the requested web page+ request: Request,++ // Parsed Content-Type HTTP header: { type, encoding }+ contentType: Object,++ // An instance of Node's http.IncomingMessage object,+ response: Object,++ // Underlying AutoscaledPool instance used to manage the concurrency of crawler+ autoscaledPool: AutoscaledPool,++ // Session object, useful to work around anti-scraping protections+ session: Session+}+```++Type of `body` depends on the `Content-Type` header of the web page:++- String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types+- Buffer for others MIME content types++Parsed `Content-Type` header using [content-type package](https://www.npmjs.com/package/content-type) is stored in `contentType`.++Cheerio is available only for HTML and XML content types.++With the [`Request`](/docs/api/request) object representing the URL to crawl.++If the function returns, the returned promise is awaited by the crawler.++If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your+function throw exceptions rather than catch them. The exceptions are logged to the request using the+[`Request.pushErrorMessage()`](/docs/api/request#pusherrormessage) function.++---++### `requestList`++**Type**: [`RequestList`](/docs/api/request-list)++Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).++---++### `requestQueue`++**Type**: [`RequestQueue`](/docs/api/request-queue)++Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be+provided (or both).++---++### `requestOptions`++**Type**: [`RequestAsBrowserOptions`](/docs/typedefs/request-as-browser-options)++Represents the options passed to the `requestAsBrowser` function that makes the HTTP requests to fetch the web pages. Provided `requestOptions` are+added to internal defaults that cannot be overridden to ensure the operation of `CheerioCrawler` and all its options. Headers will not be merged, use+[`RequestList`](/docs/api/request-list) and/or [`RequestQueue`](/docs/api/request-queue) to initialize your [`Request`](/docs/api/request) with the+correct headers or use `prepareRequestFunction` to modify your [`Request`](/docs/api/request) dynamically. If you need more granular control over your+requests, use [`BasicCrawler`](/docs/api/basic-crawler).++The mandatory internal defaults that **CANNOT BE OVERRIDDEN** by `requestOptions`:++```+{+ url, // Provided by RequestList and/or RequestQueue+ method, // Provided by RequestList and/or RequestQueue+ headers, // Provided by RequestList and/or RequestQueue+ payload, // Provided by RequestList and/or RequestQueue+ strictSSL, // Use ignoreSslErrors+ proxy, // Use useApifyProxy or proxyUrls+}+```++---++### `prepareRequestFunction`++**Type**: [`PrepareRequest`](/docs/typedefs/prepare-request)++A function that executes before the HTTP request is made to the target resource. This function is suitable for setting dynamic properties such as+cookies to the [`Request`](/docs/api/request).++The function receives the following object as an argument:++```+{request: Request,session: Session-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request- and the <a href="session"><code>Session</code></a> instance corresponds to used session.</p>-<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance- in place because there are already earlier references to it. Making a copy and returning it from- this function is therefore not supported, because it would create inconsistencies where- different parts of SDK would have access to a different <a href="request"><code>Request</code></a> instance.</p>-</td></tr><tr>-<td><code>[handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Timeout in which the function passed as <code>handlePageFunction</code> needs to finish, given in seconds.</p>-</td></tr><tr>-<td><code>[requestTimeoutSecs]</code></td><td><code>Number</code></td><td><code>30</code></td>-</tr>-<tr>-<td colspan="3"><p>Timeout in which the HTTP request to the resource needs to finish, given in seconds.</p>-</td></tr><tr>-<td><code>[ignoreSslErrors]</code></td><td><code>Boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to true, SSL certificate errors will be ignored.</p>-</td></tr><tr>-<td><code>[useApifyProxy]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to <code>true</code>, <code>CheerioCrawler</code> will be configured to use- <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a></p>-</td></tr><tr>-<td><code>[apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of proxy groups to be used- by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.- Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>-</td></tr><tr>-<td><code>[apifyProxySession]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Apify Proxy session identifier to be used with requests made by <code>CheerioCrawler</code>.- All HTTP requests going through the proxy with the same session identifier- will use the same target proxy server (i.e. the same IP address).- The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>"."</code>, <code>"_"</code> and <code>"~"</code>.- Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>-</td></tr><tr>-<td><code>[proxyUrls]</code></td><td><code>Array<String></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>CheerioCrawler</code> instance.- The provided custom proxies' order will be randomized and the resulting list rotated.- Custom proxies are not compatible with Apify Proxy and an attempt to use both- configuration options will cause an error to be thrown on startup.</p>-</td></tr><tr>-<td><code>[handleFailedRequestFunction]</code></td><td><code><a href="../typedefs/handlefailedrequest">HandleFailedRequest</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>-<p> The function receives the following object as an argument:</p>-<pre><code>{+}+```++where the [`Request`](/docs/api/request) instance corresponds to the initialized request and the [`Session`](/docs/api/session) instance corresponds+to used session.++The function should modify the properties of the passed [`Request`](/docs/api/request) instance in place because there are already earlier references+to it. Making a copy and returning it from this function is therefore not supported, because it would create inconsistencies where different parts of+SDK would have access to a different [`Request`](/docs/api/request) instance.++---++### `handlePageTimeoutSecs`++**Type**: `Number` <code> = 60</code>++Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.++---++### `requestTimeoutSecs`++**Type**: `Number` <code> = 30</code>++Timeout in which the HTTP request to the resource needs to finish, given in seconds.++---++### `ignoreSslErrors`++**Type**: `Boolean` <code> = true</code>++If set to true, SSL certificate errors will be ignored.++---++### `useApifyProxy`++**Type**: `Boolean` <code> = false</code>++If set to `true`, `CheerioCrawler` will be configured to use [Apify Proxy](https://my.apify.com/proxy) for all connections. For more information, see+the [documentation](https://docs.apify.com/proxy)++---++### `apifyProxyGroups`++**Type**: `Array<String>`++An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/proxy). Only applied if the `useApifyProxy` option is `true`.++---++### `apifyProxySession`++**Type**: `String`++Apify Proxy session identifier to be used with requests made by `CheerioCrawler`. All HTTP requests going through the proxy with the same session+identifier will use the same target proxy server (i.e. the same IP address). The identifier can only contain the following characters: `0-9`, `a-z`,+`A-Z`, `"."`, `"_"` and `"~"`. Only applied if the `useApifyProxy` option is `true`.++---++### `proxyUrls`++**Type**: `Array<String>`++An array of custom proxy URLs to be used by the `CheerioCrawler` instance. The provided custom proxies' order will be randomized and the resulting+list rotated. Custom proxies are not compatible with Apify Proxy and an attempt to use both configuration options will cause an error to be thrown on+startup.++---++### `handleFailedRequestFunction`++**Type**: [`HandleFailedRequest`](/docs/typedefs/handle-failed-request)++A function to handle requests that failed more than `option.maxRequestRetries` times.++The function receives the following object as an argument:++```+{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance- represents the last error thrown during processing of the request.</p>-<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>- for the default implementation of this function.</p>-</td></tr><tr>-<td><code>[additionalMimeTypes]</code></td><td><code>Array<String></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"- target="_blank">MIME types</a> you want the crawler to load and process.- By default, only <code>text/html</code> and <code>application/xhtml+xml</code> MIME types are supported.</p>-</td></tr><tr>-<td><code>[maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how many times the request is retried if either <code>requestFunction</code> or <code>handlePageFunction</code> fails.</p>-</td></tr><tr>-<td><code>[maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.- Always set this value in order to prevent infinite loops in misconfigured crawlers.- Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>-</td></tr><tr>-<td><code>[autoscaledPoolOptions]</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.- Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options- are provided by <code>CheerioCrawler</code> and cannot be overridden. Reasonable <a href="snapshotter"><code>Snapshotter</code></a>- and <a href="systemstatus"><code>SystemStatus</code></a> defaults are provided to account for the fact that <code>cheerio</code>- parses HTML synchronously and therefore blocks the event loop.</p>-</td></tr><tr>-<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.- If you're not sure, just keep the default value and the concurrency will scale up automatically.</p>-</td></tr><tr>-<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>-</td></tr><tr>-<td><code>[useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.- It also marks Session as bad after a request timeout.</p>-</td></tr><tr>-<td><code>[sessionPoolOptions]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>-</td></tr><tr>-<td><code>[persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>-<p> It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.- It passes the "Cookie" header to the request with the session cookies.</p>-</td></tr></tbody>-</table>+}+```++where the [`Request`](/docs/api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during+processing of the request.++See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13) for the default implementation of this+function.++---++### `additionalMimeTypes`++**Type**: `Array<String>`++An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"+ target="_blank">MIME types</a> you want the crawler to load and process. By default, only `text/html` and `application/xhtml+xml` MIME types are+supported.++---++### `maxRequestRetries`++**Type**: `Number` <code> = 3</code>++Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.++---++### `maxRequestsPerCrawl`++**Type**: `Number`++Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.++---++### `autoscaledPoolOptions`++**Type**: [`AutoscaledPoolOptions`](/docs/typedefs/autoscaled-pool-options)++Custom options passed to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) constructor. Note that the `runTaskFunction`,+`isTaskReadyFunction` and `isFinishedFunction` options are provided by `CheerioCrawler` and cannot be overridden. Reasonable+[`Snapshotter`](/docs/api/snapshotter) and [`SystemStatus`](/docs/api/system-status) defaults are provided to account for the fact that `cheerio`+parses HTML synchronously and therefore blocks the event loop.++---++### `minConcurrency`++**Type**: `Number` <code> = 1</code>++Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If+you're not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `Number` <code> = 1000</code>++Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.++---++### `useSessionPool`++**Type**: `Boolean` <code> = false</code>++If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes. It also marks+Session as bad after a request timeout.++---++### `sessionPoolOptions`++**Type**: [`SessionPoolOptions`](/docs/typedefs/session-pool-options)++Custom options passed to the underlying [`SessionPool`](/docs/api/session-pool) constructor.++---++### `persistCookiesPerSession`++**Type**: `Boolean`++Automatically saves cookies to Session. Works only if Session Pool is used.++It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request. It passes the+"Cookie" header to the request with the session cookies.++---diff --git a/docs/typedefs/CheerioHandlePage.md b/docs/typedefs/CheerioHandlePage.mdindex e27fca142..2e1ec4050 100644--- a/docs/typedefs/CheerioHandlePage.md+++ b/docs/typedefs/CheerioHandlePage.md@@ -1,21 +1,14 @@----id: cheeriohandlepage+id: cheerio-handle-pagetitle: CheerioHandlePage----<a name="CheerioHandlePage"></a>+<a name="cheeriohandlepage"></a>-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/cheeriohandlepageinputs">CheerioHandlePageInputs</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Returns**: `Promise<void>`++**Params**++- **`inputs`**: [`CheerioHandlePageInputs`](/docs/typedefs/cheerio-handle-page-inputs) - Arguments passed to this callback.++---diff --git a/docs/typedefs/CheerioHandlePageInputs.md b/docs/typedefs/CheerioHandlePageInputs.mdindex 54153b3d1..45c499032 100644--- a/docs/typedefs/CheerioHandlePageInputs.md+++ b/docs/typedefs/CheerioHandlePageInputs.md@@ -1,56 +1,73 @@----id: cheeriohandlepageinputs+id: cheerio-handle-page-inputstitle: CheerioHandlePageInputs----<a name="CheerioHandlePageInputs"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[$]</code></td><td><code>Cheerio</code></td>-</tr>-<tr>-<td colspan="3"><p>The <a href="https://cheerio.js.org/">Cheerio</a> object with parsed HTML.</p>-</td></tr><tr>-<td><code>body</code></td><td><code>String</code> | <code>Buffer</code></td>-</tr>-<tr>-<td colspan="3"><p>The request body of the web page.</p>-</td></tr><tr>-<td><code>[json]</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>The parsed object from JSON string if the response contains the content type application/json.</p>-</td></tr><tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"><p>The original {Request} object.</p>-</td></tr><tr>-<td><code>contentType</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Parsed <code>Content-Type header: { type, encoding }</code>.</p>-</td></tr><tr>-<td><code>response</code></td><td><code>IncomingMessage</code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of Node's http.IncomingMessage object,</p>-</td></tr><tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[session]</code></td><td><code>session</code></td>-</tr>-<tr>-<td colspan="3"></td></tr></tbody>-</table>+<a name="cheeriohandlepageinputs"></a>++## Properties++### `$`++**Type**: `Cheerio`++The [Cheerio](https://cheerio.js.org/) object with parsed HTML.++---++### `body`++**Type**: `String` | `Buffer`++The request body of the web page.++---++### `json`++**Type**: `Object`++The parsed object from JSON string if the response contains the content type application/json.++---++### `request`++**Type**: [`Request`](/docs/api/request)++The original {Request} object.++---++### `contentType`++**Type**: `Object`++Parsed `Content-Type header: { type, encoding }`.++---++### `response`++**Type**: `IncomingMessage`++An instance of Node's http.IncomingMessage object,++---++### `autoscaledPool`++**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)++A reference to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property+is only initialized after calling the [`CheerioCrawler.run()`](/docs/api/cheerio-crawler#run) function. You can use it to change the concurrency+settings on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause) or to abort it by calling+[`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort).++---++### `session`++**Type**: `session`++---diff --git a/docs/typedefs/CreateSession.md b/docs/typedefs/CreateSession.mdindex e3d2f0edf..34b48f4e3 100644--- a/docs/typedefs/CreateSession.md+++ b/docs/typedefs/CreateSession.md@@ -1,23 +1,14 @@----id: createsession+id: create-sessiontitle: CreateSession----<a name="CreateSession"></a>+<a name="createsession"></a>-Factory user-function which creates customized [`Session`](session) instances.+Factory user-function which creates customized [`Session`](/docs/api/session) instances.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>sessionPool</code></td><td><code><a href="sessionpool">SessionPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Pool requesting the new session.</p>-</td></tr></tbody>-</table>+**Params**++- **`sessionPool`**: [`SessionPool`](/docs/api/session-pool) - Pool requesting the new session.++---diff --git a/docs/typedefs/DatasetConsumer.md b/docs/typedefs/DatasetConsumer.mdindex b942c7d69..2bf5946cc 100644--- a/docs/typedefs/DatasetConsumer.md+++ b/docs/typedefs/DatasetConsumer.md@@ -1,30 +1,17 @@----id: datasetconsumer+id: dataset-consumertitle: DatasetConsumer----<a name="DatasetConsumer"></a>+<a name="datasetconsumer"></a>++**Returns**: TUser-function used in the `Dataset.forEach()` API.-**Returns**: T+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>item</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Current <a href="dataset"><code>Dataset</code></a> entry being processed.</p>-</td></tr><tr>-<td><code>index</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Position of current {Dataset} entry.</p>-</td></tr></tbody>-</table>+- **`item`**: `Object` - Current [`Dataset`](/docs/api/dataset) entry being processed.+- **`index`**: `Number` - Position of current {Dataset} entry.++---diff --git a/docs/typedefs/DatasetContent.md b/docs/typedefs/DatasetContent.mdindex 1c0d3bc06..ece0e9189 100644--- a/docs/typedefs/DatasetContent.md+++ b/docs/typedefs/DatasetContent.md@@ -1,43 +1,48 @@----id: datasetcontent+id: dataset-contenttitle: DatasetContent----<a name="DatasetContent"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>items</code></td><td><code>Array<Object></code> | <code>Array<String></code> | <code>Array.<Buffer></code></td>-</tr>-<tr>-<td colspan="3"><p>Dataset entries based on chosen format parameter.</p>-</td></tr><tr>-<td><code>total</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Total count of entries in the dataset.</p>-</td></tr><tr>-<td><code>offset</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Position of the first returned entry in the dataset.</p>-</td></tr><tr>-<td><code>count</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Count of dataset entries returned in this set.</p>-</td></tr><tr>-<td><code>limit</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of dataset entries requested.</p>-</td></tr></tbody>-</table>+<a name="datasetcontent"></a>++## Properties++### `items`++**Type**: `Array<Object>` | `Array<String>` | `Array.<Buffer>`++Dataset entries based on chosen format parameter.++---++### `total`++**Type**: `Number`++Total count of entries in the dataset.++---++### `offset`++**Type**: `Number`++Position of the first returned entry in the dataset.++---++### `count`++**Type**: `Number`++Count of dataset entries returned in this set.++---++### `limit`++**Type**: `Number`++Maximum number of dataset entries requested.++---diff --git a/docs/typedefs/DatasetMapper.md b/docs/typedefs/DatasetMapper.mdindex 9f07303a6..8b4602081 100644--- a/docs/typedefs/DatasetMapper.md+++ b/docs/typedefs/DatasetMapper.md@@ -1,30 +1,17 @@----id: datasetmapper+id: dataset-mappertitle: DatasetMapper----<a name="DatasetMapper"></a>+<a name="datasetmapper"></a>++**Returns**: TUser-function used in the `Dataset.map()` API.-**Returns**: T+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>item</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Currect <a href="dataset"><code>Dataset</code></a> entry being processed.</p>-</td></tr><tr>-<td><code>index</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Position of current {Dataset} entry.</p>-</td></tr></tbody>-</table>+- **`item`**: `Object` - Currect [`Dataset`](/docs/api/dataset) entry being processed.+- **`index`**: `Number` - Position of current {Dataset} entry.++---diff --git a/docs/typedefs/DatasetReducer.md b/docs/typedefs/DatasetReducer.mdindex 867dca68c..3fe56aee2 100644--- a/docs/typedefs/DatasetReducer.md+++ b/docs/typedefs/DatasetReducer.md@@ -1,35 +1,18 @@----id: datasetreducer+id: dataset-reducertitle: DatasetReducer----<a name="DatasetReducer"></a>+<a name="datasetreducer"></a>++**Returns**: TUser-function used in the `Dataset.reduce()` API.-**Returns**: T+**Params**-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>memo</code></td><td><code>T</code></td>-</tr>-<tr>-<td colspan="3"><p>Previous state of the reduction.</p>-</td></tr><tr>-<td><code>item</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Currect <a href="dataset"><code>Dataset</code></a> entry being processed.</p>-</td></tr><tr>-<td><code>index</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Position of current {Dataset} entry.</p>-</td></tr></tbody>-</table>+- **`memo`**: `T` - Previous state of the reduction.+- **`item`**: `Object` - Currect [`Dataset`](/docs/api/dataset) entry being processed.+- **`index`**: `Number` - Position of current {Dataset} entry.++---diff --git a/docs/typedefs/HandleFailedRequest.md b/docs/typedefs/HandleFailedRequest.mdindex ada6af09f..b6a5e2414 100644--- a/docs/typedefs/HandleFailedRequest.md+++ b/docs/typedefs/HandleFailedRequest.md@@ -1,21 +1,14 @@----id: handlefailedrequest+id: handle-failed-requesttitle: HandleFailedRequest----<a name="HandleFailedRequest"></a>+<a name="handlefailedrequest"></a>-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/handlefailedrequestinput">HandleFailedRequestInput</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Returns**: `void` \| `Promise<void>`++**Params**++- **`inputs`**: [`HandleFailedRequestInput`](/docs/typedefs/handle-failed-request-input) - Arguments passed to this callback.++---diff --git a/docs/typedefs/HandleFailedRequestInput.md b/docs/typedefs/HandleFailedRequestInput.mdindex 15c78af73..08eaa9d79 100644--- a/docs/typedefs/HandleFailedRequestInput.md+++ b/docs/typedefs/HandleFailedRequestInput.md@@ -1,28 +1,24 @@----id: handlefailedrequestinput+id: handle-failed-request-inputtitle: HandleFailedRequestInput----<a name="HandleFailedRequestInput"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"><p>The original {Request} object.</p>-</td></tr><tr>-<td><code>error</code></td><td><code>Error</code></td>-</tr>-<tr>-<td colspan="3"><p>The Error thrown by <code>handleRequestFunction</code>.</p>-</td></tr></tbody>-</table>+<a name="handlefailedrequestinput"></a>++## Properties++### `request`++**Type**: [`Request`](/docs/api/request)++The original {Request} object.++---++### `error`++**Type**: `Error`++The Error thrown by `handleRequestFunction`.++---diff --git a/docs/typedefs/HandleRequest.md b/docs/typedefs/HandleRequest.mdindex d78416a64..e681126dd 100644--- a/docs/typedefs/HandleRequest.md+++ b/docs/typedefs/HandleRequest.md@@ -1,21 +1,14 @@----id: handlerequest+id: handle-requesttitle: HandleRequest----<a name="HandleRequest"></a>+<a name="handlerequest"></a>-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/handlerequestinputs">HandleRequestInputs</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Returns**: `Promise<void>`++**Params**++- **`inputs`**: [`HandleRequestInputs`](/docs/typedefs/handle-request-inputs) - Arguments passed to this callback.++---diff --git a/docs/typedefs/HandleRequestInputs.md b/docs/typedefs/HandleRequestInputs.mdindex 98d1210c0..1f1081456 100644--- a/docs/typedefs/HandleRequestInputs.md+++ b/docs/typedefs/HandleRequestInputs.md@@ -1,31 +1,33 @@----id: handlerequestinputs+id: handle-request-inputstitle: HandleRequestInputs----<a name="HandleRequestInputs"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"><p>The original {Request} object.</p>-</td></tr><tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[session]</code></td><td><code><a href="session">Session</a></code></td>-</tr>-<tr>-<td colspan="3"></td></tr></tbody>-</table>+<a name="handlerequestinputs"></a>++## Properties++### `request`++**Type**: [`Request`](/docs/api/request)++The original {Request} object.++---++### `autoscaledPool`++**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)++A reference to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property+is only initialized after calling the [`BasicCrawler.run()`](/docs/api/basic-crawler#run) function. You can use it to change the concurrency settings+on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause) or to abort it by calling+[`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort).++---++### `session`++**Type**: [`Session`](/docs/api/session)++---diff --git a/docs/typedefs/KeyConsumer.md b/docs/typedefs/KeyConsumer.mdindex 6cb8bfff4..4f47bc8ed 100644--- a/docs/typedefs/KeyConsumer.md+++ b/docs/typedefs/KeyConsumer.md@@ -1,38 +1,17 @@----id: keyconsumer+id: key-consumertitle: KeyConsumer----<a name="KeyConsumer"></a>+<a name="keyconsumer"></a>-User-function used in the [`KeyValueStore.forEachKey()`](../api/keyvaluestore#forEachKey) method.+User-function used in the [`KeyValueStore.forEachKey()`](/docs/api/key-value-store#foreachkey) method.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>key</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Current {KeyValue} key being processed.</p>-</td></tr><tr>-<td><code>index</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Position of the current key in {KeyValuestore}.</p>-</td></tr><tr>-<td><code>info</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Information about the current {KeyValueStore} entry.</p>-</td></tr><tr>-<td><code>info.size</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Size of the value associated with the current key in bytes.</p>-</td></tr></tbody>-</table>+**Params**++- **`key`**: `String` - Current {KeyValue} key being processed.+- **`index`**: `Number` - Position of the current key in {KeyValuestore}.+- **`info`**: `Object` - Information about the current {KeyValueStore} entry.+ - **`.size`**: `Number` - Size of the value associated with the current key in bytes.++---diff --git a/docs/typedefs/LaunchPuppeteer.md b/docs/typedefs/LaunchPuppeteer.mdindex b566b7ad6..e5761a6b4 100644--- a/docs/typedefs/LaunchPuppeteer.md+++ b/docs/typedefs/LaunchPuppeteer.md@@ -1,25 +1,16 @@----id: launchpuppeteer+id: launch-puppeteertitle: LaunchPuppeteer----<a name="LaunchPuppeteer"></a>+<a name="launchpuppeteer"></a>**Returns**: `Promise<Browser>` - Promise that resolves to Puppeteer's `Browser` instance. This might be obtained by calling-<a href="https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions">puppeteer.launch()</a> directly, or by delegating to-[`Apify.launchPuppeteer()`](../api/apify#apifylaunchpuppeteeroptions-%E2%87%92-promisebrowser).+[puppeteer.launch()](https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions) directly, or by delegating to+[`Apify.launchPuppeteer()`](/docs/api/apify#launchpuppeteer).-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Params**++- **`inputs`**: [`LaunchPuppeteerOptions`](/docs/typedefs/launch-puppeteer-options) - Arguments passed to this callback.++---diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.mdindex 1a900bce3..2574c4609 100644--- a/docs/typedefs/LaunchPuppeteerOptions.md+++ b/docs/typedefs/LaunchPuppeteerOptions.md@@ -1,90 +1,97 @@----id: launchpuppeteeroptions+id: launch-puppeteer-optionstitle: LaunchPuppeteerOptions----<a name="LaunchPuppeteerOptions"></a>--Apify extends the launch options of Puppeteer. You can use any of the-<a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a> options in the-[`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer) function and in addition, all the options available below.--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[proxyUrl]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>URL to a HTTP proxy server. It must define the port number,- and it may also contain proxy username and password.</p>-<p> Example: <code>http://bob:[email protected]:1234</code>.</p>-</td></tr><tr>-<td><code>[userAgent]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>The <code>User-Agent</code> HTTP header used by the browser.- If not provided, the function sets <code>User-Agent</code> to a reasonable default- to reduce the chance of detection of the crawler.</p>-</td></tr><tr>-<td><code>[useChrome]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> and <code>executablePath</code> is not set,- Puppeteer will launch full Google Chrome browser available on the machine- rather than the bundled Chromium. The path to Chrome executable- is taken from the <code>APIFY_CHROME_EXECUTABLE_PATH</code> environment variable if provided,- or defaults to the typical Google Chrome executable location specific for the operating system.- By default, this option is <code>false</code>.</p>-</td></tr><tr>-<td><code>[useApifyProxy]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to <code>true</code>, Puppeteer will be configured to use- <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a></p>-</td></tr><tr>-<td><code>[apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of proxy groups to be used- by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.- Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>-</td></tr><tr>-<td><code>[apifyProxySession]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Apify Proxy session identifier to be used by all the Chrome browsers.- All HTTP requests going through the proxy with the same session identifier- will use the same target proxy server (i.e. the same IP address).- The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>"."</code>, <code>"_"</code> and <code>"~"</code>.- Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>-</td></tr><tr>-<td><code>[puppeteerModule]</code></td><td><code>string</code> | <code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Either a require path (<code>string</code>) to a package to be used instead of default <code>puppeteer</code>,- or an already required module (<code>Object</code>). This enables usage of various Puppeteer- wrappers such as <code>puppeteer-extra</code>.</p>-<p> Take caution, because it can cause all kinds of unexpected errors and weird behavior.- Apify SDK is not tested with any other library besides <code>puppeteer</code> itself.</p>-</td></tr><tr>-<td><code>[stealth]</code></td><td><code>boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.- It is recommended to use it together with the <code>useChrome</code> set to <code>true</code>.</p>-</td></tr><tr>-<td><code>[stealthOptions]</code></td><td><code><a href="../typedefs/stealthoptions">StealthOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Using this configuration, you can disable some of the hiding tricks.- For these settings to take effect <code>stealth</code> must be set to true</p>-</td></tr></tbody>-</table>+<a name="launchpuppeteeroptions"></a>++Apify extends the launch options of Puppeteer. You can use any of the Puppeteer compatible+[`LaunchOptions`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions) options in the+[`Apify.launchPuppeteer()`](/docs/api/apify#launchpuppeteer) function and in addition, all the options available below.++## Properties++### `proxyUrl`++**Type**: `String`++URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password.++Example: `http://bob:[email protected]:1234`.++---++### `userAgent`++**Type**: `String`++The `User-Agent` HTTP header used by the browser. If not provided, the function sets `User-Agent` to a reasonable default to reduce the chance of+detection of the crawler.++---++### `useChrome`++**Type**: `Boolean` <code> = false</code>++If `true` and `executablePath` is not set, Puppeteer will launch full Google Chrome browser available on the machine rather than the bundled Chromium.+The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical Google+Chrome executable location specific for the operating system. By default, this option is `false`.++---++### `useApifyProxy`++**Type**: `Boolean` <code> = false</code>++If set to `true`, Puppeteer will be configured to use [Apify Proxy](https://my.apify.com/proxy) for all connections. For more information, see the+[documentation](https://docs.apify.com/proxy)++---++### `apifyProxyGroups`++**Type**: `Array<String>`++An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/proxy). Only applied if the `useApifyProxy` option is `true`.++---++### `apifyProxySession`++**Type**: `String`++Apify Proxy session identifier to be used by all the Chrome browsers. All HTTP requests going through the proxy with the same session identifier will+use the same target proxy server (i.e. the same IP address). The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`,+`"_"` and `"~"`. Only applied if the `useApifyProxy` option is `true`.++---++### `puppeteerModule`++**Type**: `string` | `Object`++Either a require path (`string`) to a package to be used instead of default `puppeteer`, or an already required module (`Object`). This enables usage+of various Puppeteer wrappers such as `puppeteer-extra`.++Take caution, because it can cause all kinds of unexpected errors and weird behavior. Apify SDK is not tested with any other library besides+`puppeteer` itself.++---++### `stealth`++**Type**: `boolean`++This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable. It is recommended to use it together+with the `useChrome` set to `true`.++---++### `stealthOptions`++**Type**: [`StealthOptions`](/docs/typedefs/stealth-options)++Using this configuration, you can disable some of the hiding tricks. For these settings to take effect `stealth` must be set to true++---diff --git a/docs/typedefs/MemoryInfo.md b/docs/typedefs/MemoryInfo.mdindex 484cad953..f838c44ed 100644--- a/docs/typedefs/MemoryInfo.md+++ b/docs/typedefs/MemoryInfo.md@@ -1,45 +1,50 @@----id: memoryinfo+id: memory-infotitle: MemoryInfo----<a name="MemoryInfo"></a>+<a name="memoryinfo"></a>Describes memory usage of an Actor.-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>totalBytes</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Total memory available in the system or container</p>-</td></tr><tr>-<td><code>freeBytes</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Amount of free memory in the system or container</p>-</td></tr><tr>-<td><code>usedBytes</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Amount of memory used (= totalBytes - freeBytes)</p>-</td></tr><tr>-<td><code>mainProcessBytes</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Amount of memory used the current Node.js process</p>-</td></tr><tr>-<td><code>childProcessesBytes</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Amount of memory used by child processes of the current Node.js process</p>-</td></tr></tbody>-</table>+## Properties++### `totalBytes`++**Type**: `Number`++Total memory available in the system or container++---++### `freeBytes`++**Type**: `Number`++Amount of free memory in the system or container++---++### `usedBytes`++**Type**: `Number`++Amount of memory used (= totalBytes - freeBytes)++---++### `mainProcessBytes`++**Type**: `Number`++Amount of memory used the current Node.js process++---++### `childProcessesBytes`++**Type**: `Number`++Amount of memory used by child processes of the current Node.js process++---diff --git a/docs/typedefs/PrepareRequest.md b/docs/typedefs/PrepareRequest.mdindex 57e9a173a..a0684eee8 100644--- a/docs/typedefs/PrepareRequest.md+++ b/docs/typedefs/PrepareRequest.md@@ -1,21 +1,14 @@----id: preparerequest+id: prepare-requesttitle: PrepareRequest----<a name="PrepareRequest"></a>+<a name="preparerequest"></a>-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/preparerequestinputs">PrepareRequestInputs</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Returns**: `void` \| `Promise<void>`++**Params**++- **`inputs`**: [`PrepareRequestInputs`](/docs/typedefs/prepare-request-inputs) - Arguments passed to this callback.++---diff --git a/docs/typedefs/PrepareRequestInputs.md b/docs/typedefs/PrepareRequestInputs.mdindex f310bf85c..ad3a43e8f 100644--- a/docs/typedefs/PrepareRequestInputs.md+++ b/docs/typedefs/PrepareRequestInputs.md@@ -1,23 +1,16 @@----id: preparerequestinputs+id: prepare-request-inputstitle: PrepareRequestInputs----<a name="PrepareRequestInputs"></a>+<a name="preparerequestinputs"></a>-**Properties**+## Properties-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Original instance fo the {Request} object. Must be modified in-place.</p>-</td></tr></tbody>-</table>+### `request`++**Type**: [`Request`](/docs/api/request)++Original instance fo the {Request} object. Must be modified in-place.++---diff --git a/docs/typedefs/PuppeteerCrawlerOptions.md b/docs/typedefs/PuppeteerCrawlerOptions.mdindex d0a20441e..628dc8846 100644--- a/docs/typedefs/PuppeteerCrawlerOptions.md+++ b/docs/typedefs/PuppeteerCrawlerOptions.md@@ -1,165 +1,211 @@----id: puppeteercrawleroptions+id: puppeteer-crawler-optionstitle: PuppeteerCrawlerOptions----<a name="PuppeteerCrawlerOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>handlePageFunction</code></td><td><code><a href="../typedefs/puppeteerhandlepage">PuppeteerHandlePage</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Function that is called to process each request.- It is passed an object with the following fields:</p>-<pre><code>{+<a name="puppeteercrawleroptions"></a>++## Properties++### `handlePageFunction`++**Type**: [`PuppeteerHandlePage`](/docs/typedefs/puppeteer-handle-page)++Function that is called to process each request. It is passed an object with the following fields:++```+{request: Request,response: Response,page: Page,puppeteerPool: PuppeteerPool,autoscaledPool: AutoscaledPool,session: Session,-}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.- <code>page</code> is an instance of the <code>Puppeteer</code>- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>- <code>response</code> is an instance of the <code>Puppeteer</code>- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,- which is the main resource response as returned by <code>page.goto(request.url)</code>.- <code>puppeteerPool</code> is an instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>-<p> The function must return a promise, which is then awaited by the crawler.</p>-<p> If the function throws an exception, the crawler will try to re-crawl the- request later, up to <code>option.maxRequestRetries</code> times.- If all the retries fail, the crawler calls the function- provided to the <code>handleFailedRequestFunction</code> parameter.- To make this work, you should <strong>always</strong>- let your function throw exceptions rather than catch them.- The exceptions are logged to the request using the- <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>-</td></tr><tr>-<td><code>[requestList]</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Static list of URLs to be processed.- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>-</td></tr><tr>-<td><code>[requestQueue]</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>-</td></tr><tr>-<td><code>[handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Timeout in which the function passed as <code>handlePageFunction</code> needs to finish, in seconds.</p>-</td></tr><tr>-<td><code>[gotoFunction]</code></td><td><code><a href="../typedefs/puppeteergoto">PuppeteerGoto</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's- <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,- i.e. a <code>Promise</code> resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.</p>-<p> This is useful if you need to extend the page load timeout or select different criteria- to determine that the navigation succeeded.</p>-<p> Note that a single page object is only used to process a single request and it is closed afterwards.</p>-<p> By default, the function invokes <a href="puppeteer#puppeteer.gotoExtended"><code>Apify.utils.puppeteer.gotoExtended()</code></a> with a timeout of 60 seconds.- For details, see source code on- <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.</p>-</td></tr><tr>-<td><code>[gotoTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Timeout in which page navigation needs to finish, in seconds. When <code>gotoFunction()</code> is used and thus the default- function is overridden, this timeout will not be used and needs to be configured in the new <code>gotoFunction()</code>.</p>-</td></tr><tr>-<td><code>[handleFailedRequestFunction]</code></td><td><code><a href="../typedefs/handlefailedrequest">HandleFailedRequest</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>-<p> The function receives the following object as an argument:</p>-<pre><code>{+}+```++`request` is an instance of the [`Request`](/docs/api/request) object with details about the URL to open, HTTP method etc. `page` is an instance of+the `Puppeteer` [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) `response` is an instance of the `Puppeteer`+[`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response), which is the main resource response as returned by+`page.goto(request.url)`. `puppeteerPool` is an instance of the [`PuppeteerPool`](/docs/api/puppeteer-pool) used by this `PuppeteerCrawler`.++The function must return a promise, which is then awaited by the crawler.++If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your+function throw exceptions rather than catch them. The exceptions are logged to the request using the+[`Request.pushErrorMessage()`](/docs/api/request#pusherrormessage) function.++---++### `requestList`++**Type**: [`RequestList`](/docs/api/request-list)++Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).++---++### `requestQueue`++**Type**: [`RequestQueue`](/docs/api/request-queue)++Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be+provided (or both).++---++### `handlePageTimeoutSecs`++**Type**: `Number` <code> = 60</code>++Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.++---++### `gotoFunction`++**Type**: [`PuppeteerGoto`](/docs/typedefs/puppeteer-goto)++Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's+[page.goto()](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options) function, i.e. a `Promise` resolving to the+[Response](https://pptr.dev/#?product=Puppeteer&show=api-class-response) object.++This is useful if you need to extend the page load timeout or select different criteria to determine that the navigation succeeded.++Note that a single page object is only used to process a single request and it is closed afterwards.++By default, the function invokes [`puppeteer.gotoExtended()`](/docs/api/puppeteer#gotoextended) with a timeout of 60 seconds. For details, see source+code on [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292).++---++### `gotoTimeoutSecs`++**Type**: `Number` <code> = 60</code>++Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default function is overridden, this timeout+will not be used and needs to be configured in the new `gotoFunction()`.++---++### `handleFailedRequestFunction`++**Type**: [`HandleFailedRequest`](/docs/typedefs/handle-failed-request)++A function to handle requests that failed more than `option.maxRequestRetries` times.++The function receives the following object as an argument:++```+{request: Request,error: Error,-}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance- represents the last error thrown during processing of the request.</p>-<p> See- <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>- for the default implementation of this function.</p>-</td></tr><tr>-<td><code>[maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how many times the request is retried if either <code>handlePageFunction()</code> or <code>gotoFunction()</code> fails.</p>-</td></tr><tr>-<td><code>[maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.- Always set this value in order to prevent infinite loops in misconfigured crawlers.- Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>-</td></tr><tr>-<td><code>[puppeteerPoolOptions]</code></td><td><code><a href="../typedefs/puppeteerpooloptions">PuppeteerPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom options passed to the underlying <a href="puppeteerpool"><code>PuppeteerPool</code></a> constructor.- You can tweak those to fine-tune browser management.</p>-</td></tr><tr>-<td><code>[launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.- Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option.- See source code on- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>- for default behavior.</p>-</td></tr><tr>-<td><code>[launchPuppeteerOptions]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Options used by <a href="apify#module_Apify.launchPuppeteer"><code>Apify.launchPuppeteer()</code></a> to start new Puppeteer instances.- Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option. See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>-</td></tr><tr>-<td><code>[autoscaledPoolOptions]</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> instance constructor.- Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options- are provided by <code>PuppeteerCrawler</code> and should not be overridden.</p>-</td></tr><tr>-<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.- If you're not sure, just keep the default value and the concurrency will scale up automatically.</p>-</td></tr><tr>-<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>-</td></tr><tr>-<td><code>[useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.- It also marks Session as bad after a request timeout.</p>-</td></tr><tr>-<td><code>[sessionPoolOptions]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>-</td></tr><tr>-<td><code>[persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>-</td></tr></tbody>-</table>+}+```++Where the [`Request`](/docs/api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during+processing of the request.++See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301) for the default implementation of this+function.++---++### `maxRequestRetries`++**Type**: `Number` <code> = 3</code>++Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.++---++### `maxRequestsPerCrawl`++**Type**: `Number`++Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.++---++### `puppeteerPoolOptions`++**Type**: [`PuppeteerPoolOptions`](/docs/typedefs/puppeteer-pool-options)++Custom options passed to the underlying [`PuppeteerPool`](/docs/api/puppeteer-pool) constructor. You can tweak those to fine-tune browser management.++---++### `launchPuppeteerFunction`++**Type**: `function`++Overrides the default function to launch a new Puppeteer instance. Shortcut to the corresponding [`PuppeteerPool`](/docs/api/puppeteer-pool) option.+See source code on [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28) for default behavior.++---++### `launchPuppeteerOptions`++**Type**: [`LaunchPuppeteerOptions`](/docs/typedefs/launch-puppeteer-options)++Options used by [`Apify.launchPuppeteer()`](/docs/api/apify#launchpuppeteer) to start new Puppeteer instances. Shortcut to the corresponding+[`PuppeteerPool`](/docs/api/puppeteer-pool) option.++---++### `autoscaledPoolOptions`++**Type**: [`AutoscaledPoolOptions`](/docs/typedefs/autoscaled-pool-options)++Custom options passed to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) instance constructor. Note that the `runTaskFunction`,+`isTaskReadyFunction` and `isFinishedFunction` options are provided by `PuppeteerCrawler` and should not be overridden.++---++### `minConcurrency`++**Type**: `Number` <code> = 1</code>++Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding+[`AutoscaledPoolOptions.minConcurrency`](/docs/typedefs/autoscaled-pool-options#minconcurrency) option.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If+you're not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `Number` <code> = 1000</code>++Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding+[`AutoscaledPoolOptions.maxConcurrency`](/docs/typedefs/autoscaled-pool-options#maxconcurrency) option.++---++### `useSessionPool`++**Type**: `Boolean` <code> = false</code>++If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes. It also marks+Session as bad after a request timeout.++---++### `sessionPoolOptions`++**Type**: [`SessionPoolOptions`](/docs/typedefs/session-pool-options)++Custom options passed to the underlying [`SessionPool`](/docs/api/session-pool) constructor.++---++### `persistCookiesPerSession`++**Type**: `Boolean` <code> = false</code>++Automatically saves cookies to Session. Works only if Session Pool is used.++---diff --git a/docs/typedefs/PuppeteerGoto.md b/docs/typedefs/PuppeteerGoto.mdindex 9503aeaa3..09834686c 100644--- a/docs/typedefs/PuppeteerGoto.md+++ b/docs/typedefs/PuppeteerGoto.md@@ -1,25 +1,15 @@----id: puppeteergoto+id: puppeteer-gototitle: PuppeteerGoto----<a name="PuppeteerGoto"></a>+<a name="puppeteergoto"></a>-**Returns**: `Promise<PuppeteerResponse>` - An instance of the `Puppeteer`-<a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>, which is the main resource response-as returned by `page.goto(request.url)`.+**Returns**: `Promise<PuppeteerResponse>` - An instance of the Puppeteer [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),+which is the main resource response as returned by `page.goto(request.url)`.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/puppeteergotoinputs">PuppeteerGotoInputs</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Params**++- **`inputs`**: [`PuppeteerGotoInputs`](/docs/typedefs/puppeteer-goto-inputs) - Arguments passed to this callback.++---diff --git a/docs/typedefs/PuppeteerGotoInputs.md b/docs/typedefs/PuppeteerGotoInputs.mdindex f3bde9bec..561e4e81b 100644--- a/docs/typedefs/PuppeteerGotoInputs.md+++ b/docs/typedefs/PuppeteerGotoInputs.md@@ -1,44 +1,48 @@----id: puppeteergotoinputs+id: puppeteer-goto-inputstitle: PuppeteerGotoInputs----<a name="PuppeteerGotoInputs"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>PuppeteerPage</code></td>-</tr>-<tr>-<td colspan="3"><p>is an instance of the <code>Puppeteer</code>- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a></p>-</td></tr><tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.</p>-</td></tr><tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of the <code>AutoscaledPool</code>.</p>-</td></tr><tr>-<td><code>puppeteerPool</code></td><td><code><a href="puppeteerpool">PuppeteerPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>-</td></tr><tr>-<td><code>[session]</code></td><td><code><a href="session">Session</a></code></td>-</tr>-<tr>-<td colspan="3"><p><code>Session</code> object for this request.</p>-</td></tr></tbody>-</table>+<a name="puppeteergotoinputs"></a>++## Properties++### `page`++**Type**: `PuppeteerPage`++is an instance of the Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)++---++### `request`++**Type**: [`Request`](/docs/api/request)++An instance of the [`Request`](/docs/api/request) object with details about the URL to open, HTTP method etc.++---++### `autoscaledPool`++**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)++An instance of the `AutoscaledPool`.++---++### `puppeteerPool`++**Type**: [`PuppeteerPool`](/docs/api/puppeteer-pool)++An instance of the [`PuppeteerPool`](/docs/api/puppeteer-pool) used by this `PuppeteerCrawler`.++---++### `session`++**Type**: [`Session`](/docs/api/session)++`Session` object for this request.++---diff --git a/docs/typedefs/PuppeteerHandlePage.md b/docs/typedefs/PuppeteerHandlePage.mdindex cd85af529..713829d19 100644--- a/docs/typedefs/PuppeteerHandlePage.md+++ b/docs/typedefs/PuppeteerHandlePage.md@@ -1,21 +1,14 @@----id: puppeteerhandlepage+id: puppeteer-handle-pagetitle: PuppeteerHandlePage----<a name="PuppeteerHandlePage"></a>+<a name="puppeteerhandlepage"></a>-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>inputs</code></td><td><code><a href="../typedefs/puppeteerhandlepageinputs">PuppeteerHandlePageInputs</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Arguments passed to this callback.</p>-</td></tr></tbody>-</table>+**Returns**: `Promise<void>`++**Params**++- **`inputs`**: [`PuppeteerHandlePageInputs`](/docs/typedefs/puppeteer-handle-page-inputs) - Arguments passed to this callback.++---diff --git a/docs/typedefs/PuppeteerHandlePageInputs.md b/docs/typedefs/PuppeteerHandlePageInputs.mdindex 6c2e98647..fc9fa2ca8 100644--- a/docs/typedefs/PuppeteerHandlePageInputs.md+++ b/docs/typedefs/PuppeteerHandlePageInputs.md@@ -1,49 +1,58 @@----id: puppeteerhandlepageinputs+id: puppeteer-handle-page-inputstitle: PuppeteerHandlePageInputs----<a name="PuppeteerHandlePageInputs"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>request</code></td><td><code><a href="request">Request</a></code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.</p>-</td></tr><tr>-<td><code>response</code></td><td><code>PuppeteerResponse</code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of the <code>Puppeteer</code>- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,- which is the main resource response as returned by <code>page.goto(request.url)</code>.</p>-</td></tr><tr>-<td><code>page</code></td><td><code>PuppeteerPage</code></td>-</tr>-<tr>-<td colspan="3"><p>is an instance of the <code>Puppeteer</code>- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a></p>-</td></tr><tr>-<td><code>puppeteerPool</code></td><td><code><a href="puppeteerpool">PuppeteerPool</a></code></td>-</tr>-<tr>-<td colspan="3"><p>An instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>-</td></tr><tr>-<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[session]</code></td><td><code><a href="session">Session</a></code></td>-</tr>-<tr>-<td colspan="3"></td></tr></tbody>-</table>+<a name="puppeteerhandlepageinputs"></a>++## Properties++### `request`++**Type**: [`Request`](/docs/api/request)++An instance of the [`Request`](/docs/api/request) object with details about the URL to open, HTTP method etc.++---++### `response`++**Type**: `PuppeteerResponse`++An instance of the Puppeteer [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response), which is the main resource response as+returned by `page.goto(request.url)`.++---++### `page`++**Type**: `PuppeteerPage`++is an instance of the Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)++---++### `puppeteerPool`++**Type**: [`PuppeteerPool`](/docs/api/puppeteer-pool)++An instance of the [`PuppeteerPool`](/docs/api/puppeteer-pool) used by this `PuppeteerCrawler`.++---++### `autoscaledPool`++**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)++A reference to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property+is only initialized after calling the [`PuppeteerCrawler.run()`](/docs/api/puppeteer-crawler#run) function. You can use it to change the concurrency+settings on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause) or to abort it by calling+[`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort).++---++### `session`++**Type**: [`Session`](/docs/api/session)++---diff --git a/docs/typedefs/PuppeteerPoolOptions.md b/docs/typedefs/PuppeteerPoolOptions.mdindex 0f682319f..529c10ed0 100644--- a/docs/typedefs/PuppeteerPoolOptions.md+++ b/docs/typedefs/PuppeteerPoolOptions.md@@ -1,100 +1,113 @@----id: puppeteerpooloptions+id: puppeteer-pool-optionstitle: PuppeteerPoolOptions----<a name="PuppeteerPoolOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[useLiveView]</code></td><td><code>boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Enables the use of a preconfigured <a href="liveviewserver"><code>LiveViewServer</code></a> that serves snapshots- just before a page would be recycled by <code>PuppeteerPool</code>. If there are no clients- connected, it has close to zero impact on performance.</p>-</td></tr><tr>-<td><code>[maxOpenPagesPerInstance]</code></td><td><code>number</code></td><td><code>50</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.</p>-</td></tr><tr>-<td><code>[retireInstanceAfterRequestCount]</code></td><td><code>number</code></td><td><code>100</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of requests that can be processed by a single browser instance.- After the limit is reached, the browser is retired and new requests are- handled by a new browser instance.</p>-</td></tr><tr>-<td><code>[puppeteerOperationTimeoutSecs]</code></td><td><code>number</code></td><td><code>15</code></td>-</tr>-<tr>-<td colspan="3"><p>All browser management operations such as launching a new browser, opening a new page- or closing a page will timeout after the set number of seconds and the connected- browser will be retired.</p>-</td></tr><tr>-<td><code>[instanceKillerIntervalSecs]</code></td><td><code>number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how often are the open Puppeteer instances checked whether they can be closed.</p>-</td></tr><tr>-<td><code>[killInstanceAfterSecs]</code></td><td><code>number</code></td><td><code>300</code></td>-</tr>-<tr>-<td colspan="3"><p>When Puppeteer instance reaches the <code>retireInstanceAfterRequestCount</code> limit then- it is considered retired and no more tabs will be opened. After the last tab is closed the- whole browser is closed too. This parameter defines a time limit between the last tab was opened and- before the browser is closed even if there are pending open tabs.</p>-</td></tr><tr>-<td><code>[launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.- The function must return a promise resolving to- <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser"><code>Browser</code></a> instance.- See the source code on- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>- for the default implementation.</p>-</td></tr><tr>-<td><code>[launchPuppeteerOptions]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Options used by <code>Apify.launchPuppeteer()</code> to start new Puppeteer instances.- See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>-</td></tr><tr>-<td><code>[recycleDiskCache]</code></td><td><code>boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>Enables recycling of disk cache directories by Chrome instances.- When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.- This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.- Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.</p>-<p> Beware that the disk cache directories can consume a lot of disk space.- To limit the space consumed, you can pass the <code>--disk-cache-size=X</code> argument to <code>launchPuppeteerargs</code>,- where <code>X</code> is the approximate maximum number of bytes for disk cache.</p>-<p> Do not use the <code>recycleDiskCache</code> setting together with <code>--disk-cache-dir</code>- argument in <code>launchPuppeteerargs</code>, the behavior is undefined.</p>-</td></tr><tr>-<td><code>[useIncognitoPages]</code></td><td><code>boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>With this option selected, all pages will be opened in a new incognito browser context, which means- that they will not share cookies or cache and their resources will not be throttled by one another.</p>-</td></tr><tr>-<td><code>[proxyUrls]</code></td><td><code>Array<string></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>PuppeteerPool</code> instance.- The provided custom proxies' order will be randomized and the resulting list rotated.- Custom proxies are not compatible with Apify Proxy and an attempt to use both- configuration options will cause an error to be thrown on startup.</p>-</td></tr></tbody>-</table>+<a name="puppeteerpooloptions"></a>++## Properties++### `useLiveView`++**Type**: `boolean`++Enables the use of a preconfigured [`LiveViewServer`](/docs/api/live-view-server) that serves snapshots just before a page would be recycled by+`PuppeteerPool`. If there are no clients connected, it has close to zero impact on performance.++---++### `maxOpenPagesPerInstance`++**Type**: `number` <code> = 50</code>++Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.++---++### `retireInstanceAfterRequestCount`++**Type**: `number` <code> = 100</code>++Maximum number of requests that can be processed by a single browser instance. After the limit is reached, the browser is retired and new requests are+handled by a new browser instance.++---++### `puppeteerOperationTimeoutSecs`++**Type**: `number` <code> = 15</code>++All browser management operations such as launching a new browser, opening a new page or closing a page will timeout after the set number of seconds+and the connected browser will be retired.++---++### `instanceKillerIntervalSecs`++**Type**: `number` <code> = 60</code>++Indicates how often are the open Puppeteer instances checked whether they can be closed.++---++### `killInstanceAfterSecs`++**Type**: `number` <code> = 300</code>++When Puppeteer instance reaches the `retireInstanceAfterRequestCount` limit then it is considered retired and no more tabs will be opened. After the+last tab is closed the whole browser is closed too. This parameter defines a time limit between the last tab was opened and before the browser is+closed even if there are pending open tabs.++---++### `launchPuppeteerFunction`++**Type**: `function`++Overrides the default function to launch a new Puppeteer instance. The function must return a promise resolving to+[`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance. See the source code on+[GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28) for the default implementation.++---++### `launchPuppeteerOptions`++**Type**: [`LaunchPuppeteerOptions`](/docs/typedefs/launch-puppeteer-options)++Options used by [`Apify.launchPuppeteer()`](/docs/api/apify#launchpuppeteer) to start new Puppeteer instances.++---++### `recycleDiskCache`++**Type**: `boolean` <code> = false</code>++Enables recycling of disk cache directories by Chrome instances. When a browser instance is closed, its disk cache directory is not deleted but it's+used by a newly opened browser instance. This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy+usage. Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.++Beware that the disk cache directories can consume a lot of disk space. To limit the space consumed, you can pass the `--disk-cache-size=X` argument+to `launchPuppeteerargs`, where `X` is the approximate maximum number of bytes for disk cache.++Do not use the `recycleDiskCache` setting together with `--disk-cache-dir` argument in `launchPuppeteerargs`, the behavior is undefined.++---++### `useIncognitoPages`++**Type**: `boolean`++With this option selected, all pages will be opened in a new incognito browser context, which means that they will not share cookies or cache and+their resources will not be throttled by one another.++---++### `proxyUrls`++**Type**: `Array<string>`++An array of custom proxy URLs to be used by the `PuppeteerPool` instance. The provided custom proxies' order will be randomized and the resulting list+rotated. Custom proxies are not compatible with Apify Proxy and an attempt to use both configuration options will cause an error to be thrown on+startup.++---diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.mdindex 5e9396750..e85b7f267 100644--- a/docs/typedefs/QueueOperationInfo.md+++ b/docs/typedefs/QueueOperationInfo.md@@ -1,41 +1,43 @@----id: queueoperationinfo+id: queue-operation-infotitle: QueueOperationInfo----<a name="QueueOperationInfo"></a>--A helper class that is used to report results from various [`RequestQueue`](../api/requestqueue) functions as well as-[`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>wasAlreadyPresent</code></td><td><code>Boolean</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates if request was already present in the queue.</p>-</td></tr><tr>-<td><code>wasAlreadyHandled</code></td><td><code>Boolean</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates if request was already marked as handled.</p>-</td></tr><tr>-<td><code>requestId</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>The ID of the added request</p>-</td></tr><tr>-<td><code>request</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>The original <a href="../api/request"><code>Request</code></a> object passed to the <code>RequestQueue</code> function.</p>-</td></tr></tbody>-</table>+<a name="queueoperationinfo"></a>++A helper class that is used to report results from various [`RequestQueue`](/docs/api/request-queue) functions as well as+[`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks).++## Properties++### `wasAlreadyPresent`++**Type**: `Boolean`++Indicates if request was already present in the queue.++---++### `wasAlreadyHandled`++**Type**: `Boolean`++Indicates if request was already marked as handled.++---++### `requestId`++**Type**: `String`++The ID of the added request++---++### `request`++**Type**: `Object`++The original [`Request`](/docs/api/request) object passed to the `RequestQueue` function.++---diff --git a/docs/typedefs/RequestAsBrowserOptions.md b/docs/typedefs/RequestAsBrowserOptions.mdindex 822b6f13e..8b5c2e056 100644--- a/docs/typedefs/RequestAsBrowserOptions.md+++ b/docs/typedefs/RequestAsBrowserOptions.md@@ -1,76 +1,94 @@----id: requestasbrowseroptions+id: request-as-browser-optionstitle: RequestAsBrowserOptions----<a name="RequestAsBrowserOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>url</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>URL of the target endpoint. Supports both HTTP and HTTPS schemes.</p>-</td></tr><tr>-<td><code>[method]</code></td><td><code>String</code></td><td><code>GET</code></td>-</tr>-<tr>-<td colspan="3"><p>HTTP method.</p>-</td></tr><tr>-<td><code>[headers]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Additional HTTP headers to add. It's only recommended to use this option,- with headers that are typically added by websites, such as cookies. Overriding- default browser headers will remove the masking this function provides.</p>-</td></tr><tr>-<td><code>[proxyUrl]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.</p>-</td></tr><tr>-<td><code>[languageCode]</code></td><td><code>String</code></td><td><code>en</code></td>-</tr>-<tr>-<td colspan="3"><p>Two-letter ISO 639 language code.</p>-</td></tr><tr>-<td><code>[countryCode]</code></td><td><code>String</code></td><td><code>US</code></td>-</tr>-<tr>-<td colspan="3"><p>Two-letter ISO 3166 country code.</p>-</td></tr><tr>-<td><code>[useMobileVersion]</code></td><td><code>Boolean</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code>, the function uses User-Agent of a mobile browser.</p>-</td></tr><tr>-<td><code>[abortFunction]</code></td><td><code>function</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.- If function returns true request gets aborted. This function is passed to the- (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>-</td></tr><tr>-<td><code>[ignoreSslErrors]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>If set to true, SSL/TLS certificate errors will be ignored.</p>-</td></tr><tr>-<td><code>[useInsecureHttpParser]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Node.js HTTP parser is stricter than Browser parsers. This prevents fetching of some websites- whose servers do not comply with HTTP specs or employ anti-scraping protections due to- various parse errors, typically <code>invalid header value char</code> error. This option forces- the parser to ignore certain errors which allows those websites to be scraped.- However, it will also open your application to various security vulnerabilities.</p>-</td></tr></tbody>-</table>+<a name="requestasbrowseroptions"></a>++## Properties++### `url`++**Type**: `String`++URL of the target endpoint. Supports both HTTP and HTTPS schemes.++---++### `method`++**Type**: `String` <code> = GET</code>++HTTP method.++---++### `headers`++**Type**: `Object`++Additional HTTP headers to add. It's only recommended to use this option, with headers that are typically added by websites, such as cookies.+Overriding default browser headers will remove the masking this function provides.++---++### `proxyUrl`++**Type**: `String`++An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.++---++### `languageCode`++**Type**: `String` <code> = en</code>++Two-letter ISO 639 language code.++---++### `countryCode`++**Type**: `String` <code> = US</code>++Two-letter ISO 3166 country code.++---++### `useMobileVersion`++**Type**: `Boolean`++If `true`, the function uses User-Agent of a mobile browser.++---++### `abortFunction`++**Type**: `function`++Function accepts `response` object as a single parameter and should return true or false. If function returns true request gets aborted. This function+is passed to the [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.++---++### `ignoreSslErrors`++**Type**: `boolean` <code> = true</code>++If set to true, SSL/TLS certificate errors will be ignored.++---++### `useInsecureHttpParser`++**Type**: `boolean` <code> = true</code>++Node.js' HTTP parser is stricter than parsers used by web browsers, which prevents scraping of websites whose servers do not comply with HTTP specs,+either by accident or due to some anti-scraping protections, causing e.g. the `invalid header value char` error. The `useInsecureHttpParser` option+forces the HTTP parser to ignore certain errors which lets you scrape such websites. However, it will also open your application to some security+vulnerabilities, although the risk should be negligible as these vulnerabilities mainly relate to server applications, not clients. Learn more in this+[blog post](https://snyk.io/blog/node-js-release-fixes-a-critical-http-security-vulnerability/).++---diff --git a/docs/typedefs/RequestListOptions.md b/docs/typedefs/RequestListOptions.mdindex 639848c70..f16ba240c 100644--- a/docs/typedefs/RequestListOptions.md+++ b/docs/typedefs/RequestListOptions.md@@ -1,93 +1,97 @@----id: requestlistoptions+id: request-list-optionstitle: RequestListOptions----<a name="RequestListOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>sources</code></td><td><code>Array<(RequestOptions|Request)></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>. It can be either an array of plain objects that- define the <code>url</code> property, or an array of instances of the <a href="request"><code>Request</code></a> class.- Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,- which will instruct <code>RequestList</code> to download the source URLs from a given remote location.- The URLs will be parsed from the received response.</p>-<pre><code>[+<a name="requestlistoptions"></a>++## Properties++### `sources`++**Type**: [`Array<(RequestOptions|Request)>`](/docs/typedefs/request-options)++An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that define the `url` property, or an array of instances+of the [`Request`](/docs/api/request) class. Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct+`RequestList` to download the source URLs from a given remote location. The URLs will be parsed from the received response.++```+[// A single URL- { method: 'GET', url: 'http://example.com/a/b' },+ { method: 'GET', url: 'http://example.com/a/b' },// Batch import of URLs from a file hosted on the web,// where the URLs should be requested using the HTTP POST request- { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },+ { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },// Batch import from remote file, using a specific regular expression to extract the URLs.- { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },- // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.+ // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.// For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document- { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }--]</code></pre></td></tr><tr>--<td><code>[persistStateKey]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Identifies the key in the default key-value store under which <code>RequestList</code> periodically stores its- state (i.e. which URLs were crawled and which not).- If the actor is restarted, <code>RequestList</code> will read the state- and continue where it left off.</p>-<p> If <code>persistStateKey</code> is not set, <code>RequestList</code> will always start from the beginning,- and all the source URLs will be crawled again.</p>-</td></tr><tr>-<td><code>[persistSourcesKey]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Identifies the key in the default key-value store under which the <code>RequestList</code> persists its- sources (i.e. the lists of URLs) during the <a href="#RequestList+initialize"><code>initialize</code></a> call.- This is necessary if <code>persistStateKey</code> is set and the source URLs might potentially change,- to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.</p>-<p> If <code>persistSourcesKey</code> is not set, <a href="#RequestList+initialize"><code>initialize</code></a> will always fetch the sources- from their origin, check that they are consistent with the restored state (if any)- and throw an error if they are not.</p>-</td></tr><tr>-<td><code>[state]</code></td><td><code><a href="../typedefs/requestliststate">RequestListState</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>The state object that the <code>RequestList</code> will be initialized from.- It is in the form as returned by <code>RequestList.getState()</code>, such as follows:</p>-<pre><code>{+ { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }+]+```++---++### `persistStateKey`++**Type**: `String`++Identifies the key in the default key-value store under which `RequestList` periodically stores its state (i.e. which URLs were crawled and which+not). If the actor is restarted, `RequestList` will read the state and continue where it left off.++If `persistStateKey` is not set, `RequestList` will always start from the beginning, and all the source URLs will be crawled again.++---++### `persistSourcesKey`++**Type**: `String`++Identifies the key in the default key-value store under which the `RequestList` persists its sources (i.e. the lists of URLs) during the+[`RequestList.initialize()`](/docs/api/request-list#initialize) call. This is necessary if `persistStateKey` is set and the source URLs might+potentially change, to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.++If `persistSourcesKey` is not set, [`RequestList.initialize()`](/docs/api/request-list#initialize) will always fetch the sources from their origin,+check that they are consistent with the restored state (if any) and throw an error if they are not.++---++### `state`++**Type**: [`RequestListState`](/docs/typedefs/request-list-state)++The state object that the `RequestList` will be initialized from. It is in the form as returned by `RequestList.getState()`, such as follows:++```+{nextIndex: 5,- nextUniqueKey: 'unique-key-5'+ nextUniqueKey: 'unique-key-5'inProgress: {- 'unique-key-1': true,- 'unique-key-4': true,+ 'unique-key-1': true,+ 'unique-key-4': true,},-}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>- is to use the <code>stateKeyPrefix</code> parameter instead.</p>-</td></tr><tr>-<td><code>[keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>By default, <code>RequestList</code> will deduplicate the provided URLs. Default deduplication is based- on the <code>uniqueKey</code> property of passed source <a href="request"><code>Request</code></a> objects.</p>-<p> If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.- In any case, only one request per <code>uniqueKey</code> is added to the <code>RequestList</code> resulting in removal- of duplicate URLs / unique keys.</p>-<p> Setting <code>keepDuplicateUrls</code> to <code>true</code> will append an additional identifier to the <code>uniqueKey</code>- of each request that does not already include a <code>uniqueKey</code>. Therefore, duplicate- URLs will be kept in the list. It does not protect the user from having duplicates in user set- <code>uniqueKey</code>s however. It is the user's responsibility to ensure uniqueness of their unique keys- if they wish to keep more than just a single copy in the <code>RequestList</code>.</p>-</td></tr></tbody>-</table>+}+```++Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList` is to use the `stateKeyPrefix` parameter instead.++---++### `keepDuplicateUrls`++**Type**: `Boolean` <code> = false</code>++By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based on the `uniqueKey` property of passed source+[`Request`](/docs/api/request) objects.++If the property is not present, it is generated by normalizing the URL. If present, it is kept intact. In any case, only one request per `uniqueKey`+is added to the `RequestList` resulting in removal of duplicate URLs / unique keys.++Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey` of each request that does not already include a+`uniqueKey`. Therefore, duplicate URLs will be kept in the list. It does not protect the user from having duplicates in user set `uniqueKey`s however.+It is the user's responsibility to ensure uniqueness of their unique keys if they wish to keep more than just a single copy in the `RequestList`.++---diff --git a/docs/typedefs/RequestListState.md b/docs/typedefs/RequestListState.mdindex 1590b472e..907c2247c 100644--- a/docs/typedefs/RequestListState.md+++ b/docs/typedefs/RequestListState.md@@ -1,12 +1,12 @@----id: requestliststate+id: request-list-statetitle: RequestListState----<a name="RequestListState"></a>+<a name="requestliststate"></a>Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed. You can obtain the state by calling-[`RequestList.getState()`](<../api/requestlist#getState()>) and receive an object with the following structure:+[`RequestList.getState()`](/docs/api/request-list#getstate) and receive an object with the following structure:```{@@ -19,29 +19,28 @@ Represents state of a {RequestList}. It can be used to resume a {RequestList} wh}```-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>nextIndex</code></td><td><code>Number</code></td>-</tr>-<tr>-<td colspan="3"><p>Position of the next request to be processed.</p>-</td></tr><tr>-<td><code>nextUniqueKey</code></td><td><code>String</code></td>-</tr>-<tr>-<td colspan="3"><p>Key of the next request to be processed.</p>-</td></tr><tr>-<td><code>inProgress</code></td><td><code>Object<String, Boolean></code></td>-</tr>-<tr>-<td colspan="3"><p>An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.</p>-</td></tr></tbody>-</table>+## Properties++### `nextIndex`++**Type**: `Number`++Position of the next request to be processed.++---++### `nextUniqueKey`++**Type**: `String`++Key of the next request to be processed.++---++### `inProgress`++**Type**: `Object<String, Boolean>`++An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.++---diff --git a/docs/typedefs/RequestOptions.md b/docs/typedefs/RequestOptions.mdindex 964dd4767..bb3b02ffb 100644--- a/docs/typedefs/RequestOptions.md+++ b/docs/typedefs/RequestOptions.md@@ -1,78 +1,94 @@----id: requestoptions+id: request-optionstitle: RequestOptions----<a name="RequestOptions"></a>--Specifies required and optional fields for constructing a [`Request`](../api/request).--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>url</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>URL of the web page to crawl. It must be a non-empty string.</p>-</td></tr><tr>-<td><code>[uniqueKey]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A unique key identifying the request.- Two requests with the same <code>uniqueKey</code> are considered as pointing to the same URL.</p>-<p> If <code>uniqueKey</code> is not provided, then it is automatically generated by normalizing the URL.- For example, the URL of <code>HTTP://www.EXAMPLE.com/something/</code> will produce the <code>uniqueKey</code>- of <code>http://www.example.com/something</code>.</p>-<p> The <code>keepUrlFragment</code> option determines whether URL hash fragment is included in the <code>uniqueKey</code> or not.</p>-<p> The <code>useExtendedUniqueKey</code> options determines whether method and payload are included in the <code>uniqueKey</code>,- producing a <code>uniqueKey</code> in the following format: <code>METHOD(payloadHash):normalizedUrl</code>. This is useful- when requests point to the same URL, but with different methods and payloads. For example: form submits.</p>-<p> Pass an arbitrary non-empty text value to the <code>uniqueKey</code> property- to override the default behavior and specify which URLs shall be considered equal.</p>-</td></tr><tr>-<td><code>[method]</code></td><td><code>String</code></td><td><code>'GET'</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>[payload]</code></td><td><code>String</code> | <code>Buffer</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p>-</td></tr><tr>-<td><code>[headers]</code></td><td><code>Object</code></td><td><code>{}</code></td>-</tr>-<tr>-<td colspan="3"><p>HTTP headers in the following format:</p>-<pre><code> {- Accept: 'text/html',- 'Content-Type': 'application/json'- }</code></pre></td></tr><tr>-<td><code>[userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>-</tr>-<tr>-<td colspan="3"><p>Custom user data assigned to the request. Use this to save any request related data to the- request's scope, keeping them accessible on retries, failures etc.</p>-</td></tr><tr>-<td><code>[keepUrlFragment]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>false</code> then the hash part of a URL is removed when computing the <code>uniqueKey</code> property.- For example, this causes the <code>http://www.example.com#foo</code> and <code>http://www.example.com#bar</code> URLs- to have the same <code>uniqueKey</code> of <code>http://www.example.com</code> and thus the URLs are considered equal.- Note that this option only has an effect if <code>uniqueKey</code> is not set.</p>-</td></tr><tr>-<td><code>[useExtendedUniqueKey]</code></td><td><code>Boolean</code></td><td><code>false</code></td>-</tr>-<tr>-<td colspan="3"><p>If <code>true</code> then the <code>uniqueKey</code> is computed not only from the URL, but also from the method and payload- properties. This is useful when making requests to the same URL that are differentiated by method- or payload, such as form submit navigations in browsers.</p>-</td></tr></tbody>-</table>+<a name="requestoptions"></a>++Specifies required and optional fields for constructing a [`Request.setValue()`](/docs/api/request#setvalue).++## Properties++### `url`++**Type**: `String`++URL of the web page to crawl. It must be a non-empty string.++---++### `uniqueKey`++**Type**: `String`++A unique key identifying the request. Two requests with the same `uniqueKey` are considered as pointing to the same URL.++If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL. For example, the URL of `HTTP://www.EXAMPLE.com/something/`+will produce the `uniqueKey` of `http://www.example.com/something`.++The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.++The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`, producing a `uniqueKey` in the following+format: `METHOD(payloadHash):normalizedUrl`. This is useful when requests point to the same URL, but with different methods and payloads. For example:+form submits.++Pass an arbitrary non-empty text value to the `uniqueKey` property to override the default behavior and specify which URLs shall be considered equal.++---++### `method`++**Type**: `String` <code> = 'GET'</code>++---++### `payload`++**Type**: `String` | `Buffer`++HTTP request payload, e.g. for POST requests.++---++### `headers`++**Type**: `Object` <code> = {}</code>++HTTP headers in the following format:++```+{+ Accept: 'text/html',+ 'Content-Type': 'application/json'+}+```++---++### `userData`++**Type**: `Object` <code> = {}</code>++Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries,+failures etc.++---++### `keepUrlFragment`++**Type**: `Boolean` <code> = false</code>++If `false` then the hash part of a URL is removed when computing the `uniqueKey` property. For example, this causes the `http://www.example.com#foo`+and `http://www.example.com#bar` URLs to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal. Note that this+option only has an effect if `uniqueKey` is not set.++---++### `useExtendedUniqueKey`++**Type**: `Boolean` <code> = false</code>++If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload properties. This is useful when making requests+to the same URL that are differentiated by method or payload, such as form submit navigations in browsers.++---diff --git a/docs/typedefs/RequestTransform.md b/docs/typedefs/RequestTransform.mdindex 07c0a4bbd..7e151af04 100644--- a/docs/typedefs/RequestTransform.md+++ b/docs/typedefs/RequestTransform.md@@ -1,26 +1,17 @@----id: requesttransform+id: request-transformtitle: RequestTransform----<a name="RequestTransform"></a>+<a name="requesttransform"></a>++**Returns**: [`RequestOptions`](/docs/typedefs/request-options) - The modified request options to enqueue.Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used-[`Apify.utils.enqueueLinks`](../api/utils#utils.enqueueLinks) to modify requests before enqueuing them.+[`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks) to modify requests before enqueuing them.++**Params**-**Returns**: [`RequestOptions`](../typedefs/requestoptions) - The modified request options to enqueue.+- **`original`**: [`RequestOptions`](/docs/typedefs/request-options) - Request options to be modified.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>original</code></td><td><code><a href="../typedefs/requestoptions">RequestOptions</a></code></td>-</tr>-<tr>-<td colspan="3"><p>Request options to be modified.</p>-</td></tr></tbody>-</table>+---diff --git a/docs/typedefs/SessionOptions.md b/docs/typedefs/SessionOptions.mdindex 15f5543ca..4da2f1bd2 100644--- a/docs/typedefs/SessionOptions.md+++ b/docs/typedefs/SessionOptions.md@@ -1,79 +1,99 @@----id: sessionoptions+id: session-optionstitle: SessionOptions----<a name="SessionOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[id]</code></td><td><code>string</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Id of session used for generating fingerprints. It is used as proxy session name.</p>-</td></tr><tr>-<td><code>[maxAgeSecs]</code></td><td><code>number</code></td><td><code>3000</code></td>-</tr>-<tr>-<td colspan="3"><p>Number of seconds after which the session is considered as expired.</p>-</td></tr><tr>-<td><code>userData</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Object where custom user data can be stored. For example custom headers.</p>-</td></tr><tr>-<td><code>[maxErrorScore]</code></td><td><code>number</code></td><td><code>3</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum number of marking session as blocked usage.- If the <code>errorScore</code> reaches the <code>maxErrorScore</code> session is marked as block and it is thrown away.- It starts at 0. Calling the <code>markBad</code> function increases the <code>errorScore</code> by 1.- Calling the <code>markGood</code> will decrease the <code>errorScore</code> by <code>errorScoreDecrement</code></p>-</td></tr><tr>-<td><code>[errorScoreDecrement]</code></td><td><code>number</code></td><td><code>0.5</code></td>-</tr>-<tr>-<td colspan="3"><p>It is used for healing the session.- For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this- number.</p>-</td></tr><tr>-<td><code>[createdAt]</code></td><td><code>Date</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Date of creation.</p>-</td></tr><tr>-<td><code>[expiresAt]</code></td><td><code>Date</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Date of expiration.</p>-</td></tr><tr>-<td><code>[usageCount]</code></td><td><code>number</code></td><td><code>0</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how many times the session has been used.</p>-</td></tr><tr>-<td><code>[errorCount]</code></td><td><code>number</code></td><td><code>0</code></td>-</tr>-<tr>-<td colspan="3"><p>Indicates how many times the session is marked bad.</p>-</td></tr><tr>-<td><code>[maxUsageCount]</code></td><td><code>number</code></td><td><code>50</code></td>-</tr>-<tr>-<td colspan="3"><p>Session should be used only a limited amount of times.- This number indicates how many times the session is going to be used, before it is thrown away.</p>-</td></tr><tr>-<td><code>sessionPool</code></td><td><code><a href="sessionpool">SessionPool</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>SessionPool instance. Session will emit the <code>sessionRetired</code> event on this instance.</p>-</td></tr></tbody>-</table>+<a name="sessionoptions"></a>++## Properties++### `id`++**Type**: `string`++Id of session used for generating fingerprints. It is used as proxy session name.++---++### `maxAgeSecs`++**Type**: `number` <code> = 3000</code>++Number of seconds after which the session is considered as expired.++---++### `userData`++**Type**: `Object`++Object where custom user data can be stored. For example custom headers.++---++### `maxErrorScore`++**Type**: `number` <code> = 3</code>++Maximum number of marking session as blocked usage. If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.+It starts at 0. Calling the `markBad` function increases the `errorScore` by 1. Calling the `markGood` will decrease the `errorScore` by+`errorScoreDecrement`++---++### `errorScoreDecrement`++**Type**: `number` <code> = 0.5</code>++It is used for healing the session. For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is+decremented by this number.++---++### `createdAt`++**Type**: `Date`++Date of creation.++---++### `expiresAt`++**Type**: `Date`++Date of expiration.++---++### `usageCount`++**Type**: `number` <code> = 0</code>++Indicates how many times the session has been used.++---++### `errorCount`++**Type**: `number` <code> = 0</code>++Indicates how many times the session is marked bad.++---++### `maxUsageCount`++**Type**: `number` <code> = 50</code>++Session should be used only a limited amount of times. This number indicates how many times the session is going to be used, before it is thrown away.++---++### `sessionPool`++**Type**: [`SessionPool`](/docs/api/session-pool)++SessionPool instance. Session will emit the `sessionRetired` event on this instance.++---diff --git a/docs/typedefs/SessionPoolOptions.md b/docs/typedefs/SessionPoolOptions.mdindex 488b19a5e..0bf631e11 100644--- a/docs/typedefs/SessionPoolOptions.md+++ b/docs/typedefs/SessionPoolOptions.md@@ -1,45 +1,48 @@----id: sessionpooloptions+id: session-pool-optionstitle: SessionPoolOptions----<a name="SessionPoolOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[maxPoolSize]</code></td><td><code>Number</code></td><td><code>1000</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum size of the pool.-Indicates how many sessions are rotated.</p>-</td></tr><tr>-<td><code>[sessionOptions]</code></td><td><code><a href="../typedefs/sessionoptions">SessionOptions</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>The configuration options for {Session} instances.</p>-</td></tr><tr>-<td><code>[persistStateKeyValueStoreId]</code></td><td><code>String</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Name or Id of <code>KeyValueStore</code> where is the <code>SessionPool</code> state stored.</p>-</td></tr><tr>-<td><code>[persistStateKey]</code></td><td><code>String</code></td><td><code>"SESSION_POOL_STATE"</code></td>-</tr>-<tr>-<td colspan="3"><p>Session pool persists it's state under this key in Key value store.</p>-</td></tr><tr>-<td><code>[createSessionFunction]</code></td><td><code><a href="../typedefs/createsession">CreateSession</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Custom function that should return <code>Session</code> instance.-Function receives <code>SessionPool</code> instance as a parameter</p>-</td></tr></tbody>-</table>+<a name="sessionpooloptions"></a>++## Properties++### `maxPoolSize`++**Type**: `Number` <code> = 1000</code>++Maximum size of the pool. Indicates how many sessions are rotated.++---++### `sessionOptions`++**Type**: [`SessionOptions`](/docs/typedefs/session-options)++The configuration options for {Session} instances.++---++### `persistStateKeyValueStoreId`++**Type**: `String`++Name or Id of `KeyValueStore` where is the `SessionPool` state stored.++---++### `persistStateKey`++**Type**: `String` <code> = "SESSION_POOL_STATE"</code>++Session pool persists it's state under this key in Key value store.++---++### `createSessionFunction`++**Type**: [`CreateSession`](/docs/typedefs/create-session)++Custom function that should return `Session` instance. Function receives `SessionPool` instance as a parameter++---diff --git a/docs/typedefs/SessionState.md b/docs/typedefs/SessionState.mdindex cb83d2355..f29201834 100644--- a/docs/typedefs/SessionState.md+++ b/docs/typedefs/SessionState.md@@ -1,56 +1,64 @@----id: sessionstate+id: session-statetitle: SessionState----<a name="SessionState"></a>--Persistable [`Session`](session) state.--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>id</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>cookieJar</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr><tr>-<td><code>userData</code></td><td><code>Object</code></td>-</tr>-<tr>-</tr><tr>-<td><code>errorScore</code></td><td><code>number</code></td>-</tr>-<tr>-</tr><tr>-<td><code>maxErrorScore</code></td><td><code>number</code></td>-</tr>-<tr>-</tr><tr>-<td><code>errorScoreDecrement</code></td><td><code>number</code></td>-</tr>-<tr>-</tr><tr>-<td><code>usageCount</code></td><td><code>number</code></td>-</tr>-<tr>-</tr><tr>-<td><code>expiresAt</code></td><td><code>string</code></td>-</tr>-<tr>-</tr><tr>-<td><code>createdAt</code></td><td><code>string</code></td>-</tr>-<tr>-</tr></tbody>-</table>+<a name="sessionstate"></a>++Persistable [`Session`](/docs/api/session) state.++## Properties++### `id`++**Type**: `string`++---++### `cookieJar`++**Type**: `Object`++---++### `userData`++**Type**: `Object`++---++### `errorScore`++**Type**: `number`++---++### `maxErrorScore`++**Type**: `number`++---++### `errorScoreDecrement`++**Type**: `number`++---++### `usageCount`++**Type**: `number`++---++### `expiresAt`++**Type**: `string`++---++### `createdAt`++**Type**: `string`++---diff --git a/docs/typedefs/SnapshotterOptions.md b/docs/typedefs/SnapshotterOptions.mdindex 1be68c18c..c28e72d9e 100644--- a/docs/typedefs/SnapshotterOptions.md+++ b/docs/typedefs/SnapshotterOptions.md@@ -1,75 +1,83 @@----id: snapshotteroptions+id: snapshotter-optionstitle: SnapshotterOptions----<a name="SnapshotterOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[eventLoopSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>0.5</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the interval of measuring the event loop response time.</p>-</td></tr><tr>-<td><code>[clientSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the interval of checking the current state- of the remote API client.</p>-</td></tr><tr>-<td><code>[maxBlockedMillis]</code></td><td><code>Number</code></td><td><code>50</code></td>-</tr>-<tr>-<td colspan="3"><p>Maximum allowed delay of the event loop in milliseconds.- Exceeding this limit overloads the event loop.</p>-</td></tr><tr>-<td><code>[cpuSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the interval of measuring CPU usage.- This is only used when running locally. On the Apify platform,- the statistics are provided externally at a fixed interval.</p>-</td></tr><tr>-<td><code>[maxUsedCpuRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the maximum usage of CPU.- Exceeding this limit overloads the CPU.</p>-</td></tr><tr>-<td><code>[memorySnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the interval of measuring memory consumption.- This is only used when running locally. On the Apify platform,- the statistics are provided externally at a fixed interval.- The measurement itself is resource intensive (25 - 50ms async).- Therefore, setting this interval below 1 second is not recommended.</p>-</td></tr><tr>-<td><code>[maxUsedMemoryRatio]</code></td><td><code>Number</code></td><td><code>0.7</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the maximum ratio of total memory that can be used.- Exceeding this limit overloads the memory.</p>-</td></tr><tr>-<td><code>[maxClientErrors]</code></td><td><code>Number</code></td><td><code>1</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines the maximum number of new rate limit errors within- the given interval.</p>-</td></tr><tr>-<td><code>[snapshotHistorySecs]</code></td><td><code>Number</code></td><td><code>60</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the interval in seconds for which a history of resource snapshots- will be kept. Increasing this to very high numbers will affect performance.</p>-</td></tr></tbody>-</table>+<a name="snapshotteroptions"></a>++## Properties++### `eventLoopSnapshotIntervalSecs`++**Type**: `Number` <code> = 0.5</code>++Defines the interval of measuring the event loop response time.++---++### `clientSnapshotIntervalSecs`++**Type**: `Number` <code> = 1</code>++Defines the interval of checking the current state of the remote API client.++---++### `maxBlockedMillis`++**Type**: `Number` <code> = 50</code>++Maximum allowed delay of the event loop in milliseconds. Exceeding this limit overloads the event loop.++---++### `cpuSnapshotIntervalSecs`++**Type**: `Number` <code> = 1</code>++Defines the interval of measuring CPU usage. This is only used when running locally. On the Apify platform, the statistics are provided externally at+a fixed interval.++---++### `maxUsedCpuRatio`++**Type**: `Number` <code> = 0.95</code>++Defines the maximum usage of CPU. Exceeding this limit overloads the CPU.++---++### `memorySnapshotIntervalSecs`++**Type**: `Number` <code> = 1</code>++Defines the interval of measuring memory consumption. This is only used when running locally. On the Apify platform, the statistics are provided+externally at a fixed interval. The measurement itself is resource intensive (25 - 50ms async). Therefore, setting this interval below 1 second is not+recommended.++---++### `maxUsedMemoryRatio`++**Type**: `Number` <code> = 0.7</code>++Defines the maximum ratio of total memory that can be used. Exceeding this limit overloads the memory.++---++### `maxClientErrors`++**Type**: `Number` <code> = 1</code>++Defines the maximum number of new rate limit errors within the given interval.++---++### `snapshotHistorySecs`++**Type**: `Number` <code> = 60</code>++Sets the interval in seconds for which a history of resource snapshots will be kept. Increasing this to very high numbers will affect performance.++---diff --git a/docs/typedefs/SocialHandles.md b/docs/typedefs/SocialHandles.mdindex d75a05986..15aabb580 100644--- a/docs/typedefs/SocialHandles.md+++ b/docs/typedefs/SocialHandles.md@@ -1,9 +1,9 @@----id: socialhandles+id: social-handlestitle: SocialHandles----<a name="SocialHandles"></a>+<a name="socialhandles"></a>Representation of social handles parsed from a HTML page.@@ -22,46 +22,52 @@ The object has the following structure:}```-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>emails</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>phones</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>phonesUncertain</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>linkedIns</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>twitters</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>instagrams</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>facebooks</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr><tr>-<td><code>youtubes</code></td><td><code>Array<String></code></td>-</tr>-<tr>-</tr></tbody>-</table>+## Properties++### `emails`++**Type**: `Array<String>`++---++### `phones`++**Type**: `Array<String>`++---++### `phonesUncertain`++**Type**: `Array<String>`++---++### `linkedIns`++**Type**: `Array<String>`++---++### `twitters`++**Type**: `Array<String>`++---++### `instagrams`++**Type**: `Array<String>`++---++### `facebooks`++**Type**: `Array<String>`++---++### `youtubes`++**Type**: `Array<String>`++---diff --git a/docs/typedefs/StealthOptions.md b/docs/typedefs/StealthOptions.mdindex 3c36f8faa..39ecf82c7 100644--- a/docs/typedefs/StealthOptions.md+++ b/docs/typedefs/StealthOptions.md@@ -1,71 +1,91 @@----id: stealthoptions+id: stealth-optionstitle: StealthOptions----<a name="StealthOptions"></a>+<a name="stealthoptions"></a>Configuration of stealth tricks for a proper hiding effect all of them should be set to true. These tricks are applied only when the `stealth` optionis set to `true`.-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[addPlugins]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>If plugins should be added to the navigator.</p>-</td></tr><tr>-<td><code>[emulateWindowFrame]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Emulates window Iframe.</p>-</td></tr><tr>-<td><code>[emulateWebGL]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Emulates graphic card.</p>-</td></tr><tr>-<td><code>[emulateConsoleDebug]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Emulates console.debug to return null.</p>-</td></tr><tr>-<td><code>[addLanguage]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Adds languages to the navigator.</p>-</td></tr><tr>-<td><code>[hideWebDriver]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Hides the webdriver by changing the navigator proto.</p>-</td></tr><tr>-<td><code>[hackPermissions]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Fakes interaction with permissions.</p>-</td></tr><tr>-<td><code>[mockChrome]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Adds the chrome runtime properties.</p>-</td></tr><tr>-<td><code>[mockChromeInIframe]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Adds the chrome runtime properties inside the every newly created iframe.</p>-</td></tr><tr>-<td><code>[mockDeviceMemory]</code></td><td><code>boolean</code></td><td><code>true</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets device memory to other value than 0.</p>-</td></tr></tbody>-</table>+## Properties++### `addPlugins`++**Type**: `boolean` <code> = true</code>++If plugins should be added to the navigator.++---++### `emulateWindowFrame`++**Type**: `boolean` <code> = true</code>++Emulates window Iframe.++---++### `emulateWebGL`++**Type**: `boolean` <code> = true</code>++Emulates graphic card.++---++### `emulateConsoleDebug`++**Type**: `boolean` <code> = true</code>++Emulates console.debug to return null.++---++### `addLanguage`++**Type**: `boolean` <code> = true</code>++Adds languages to the navigator.++---++### `hideWebDriver`++**Type**: `boolean` <code> = true</code>++Hides the webdriver by changing the navigator proto.++---++### `hackPermissions`++**Type**: `boolean` <code> = true</code>++Fakes interaction with permissions.++---++### `mockChrome`++**Type**: `boolean` <code> = true</code>++Adds the chrome runtime properties.++---++### `mockChromeInIframe`++**Type**: `boolean` <code> = true</code>++Adds the chrome runtime properties inside the every newly created iframe.++---++### `mockDeviceMemory`++**Type**: `boolean` <code> = true</code>++Sets device memory to other value than 0.++---diff --git a/docs/typedefs/SystemInfo.md b/docs/typedefs/SystemInfo.mdindex dd074d7f8..feadca8c8 100644--- a/docs/typedefs/SystemInfo.md+++ b/docs/typedefs/SystemInfo.md@@ -1,38 +1,38 @@----id: systeminfo+id: system-infotitle: SystemInfo----<a name="SystemInfo"></a>+<a name="systeminfo"></a>Represents the current status of the system.-**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>isSystemIdle</code></td><td><code>Boolean</code></td>-</tr>-<tr>-<td colspan="3"><p>If true, system is being overloaded.</p>-</td></tr><tr>-<td><code>memInfo</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"><p>Memory</p>-</td></tr><tr>-<td><code>eventLoopInfo</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>cpuInfo</code></td><td><code>Object</code></td>-</tr>-<tr>-<td colspan="3"></td></tr></tbody>-</table>+## Properties++### `isSystemIdle`++**Type**: `Boolean`++If true, system is being overloaded.++---++### `memInfo`++**Type**: `Object`++Memory++---++### `eventLoopInfo`++**Type**: `Object`++---++### `cpuInfo`++**Type**: `Object`++---diff --git a/docs/typedefs/SystemStatusOptions.md b/docs/typedefs/SystemStatusOptions.mdindex 697d84375..3cf34d8da 100644--- a/docs/typedefs/SystemStatusOptions.md+++ b/docs/typedefs/SystemStatusOptions.md@@ -1,53 +1,56 @@----id: systemstatusoptions+id: system-status-optionstitle: SystemStatusOptions----<a name="SystemStatusOptions"></a>--**Properties**--<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>[currentHistorySecs]</code></td><td><code>Number</code></td><td><code>5</code></td>-</tr>-<tr>-<td colspan="3"><p>Defines max age of snapshots used in the- <a href="#SystemStatus+getCurrentStatus"><code>getCurrentStatus()</code></a> measurement.</p>-</td></tr><tr>-<td><code>[maxMemoryOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a memory sample.- If the sample exceeds this ratio, the system will be overloaded.</p>-</td></tr><tr>-<td><code>[maxEventLoopOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in an event loop sample.- If the sample exceeds this ratio, the system will be overloaded.</p>-</td></tr><tr>-<td><code>[maxCpuOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.4</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a CPU sample.- If the sample exceeds this ratio, the system will be overloaded.</p>-</td></tr><tr>-<td><code>[maxClientOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>-</tr>-<tr>-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a Client sample.- If the sample exceeds this ratio, the system will be overloaded.</p>-</td></tr><tr>-<td><code>[snapshotter]</code></td><td><code><a href="snapshotter">Snapshotter</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>The <code>Snapshotter</code> instance to be queried for <code>SystemStatus</code>.</p>-</td></tr></tbody>-</table>+<a name="systemstatusoptions"></a>++## Properties++### `currentHistorySecs`++**Type**: `Number` <code> = 5</code>++Defines max age of snapshots used in the [`SystemStatus.getCurrentStatus()`](/docs/api/system-status#getcurrentstatus) measurement.++---++### `maxMemoryOverloadedRatio`++**Type**: `Number` <code> = 0.2</code>++Sets the maximum ratio of overloaded snapshots in a memory sample. If the sample exceeds this ratio, the system will be overloaded.++---++### `maxEventLoopOverloadedRatio`++**Type**: `Number` <code> = 0.2</code>++Sets the maximum ratio of overloaded snapshots in an event loop sample. If the sample exceeds this ratio, the system will be overloaded.++---++### `maxCpuOverloadedRatio`++**Type**: `Number` <code> = 0.4</code>++Sets the maximum ratio of overloaded snapshots in a CPU sample. If the sample exceeds this ratio, the system will be overloaded.++---++### `maxClientOverloadedRatio`++**Type**: `Number` <code> = 0.2</code>++Sets the maximum ratio of overloaded snapshots in a Client sample. If the sample exceeds this ratio, the system will be overloaded.++---++### `snapshotter`++**Type**: [`Snapshotter`](/docs/api/snapshotter)++The `Snapshotter` instance to be queried for `SystemStatus`.++---diff --git a/examples/basic_crawler.js b/examples/basic_crawler.jsindex 30b1891c5..f34283c97 100644--- a/examples/basic_crawler.js+++ b/examples/basic_crawler.js@@ -1,14 +1,14 @@/*** This is the most basic example of the Apify SDK, which demonstrates some of its* elementary tools, such as the- * [`BasicCrawler`](../api/basiccrawler)- * and [`RequestList`](../api/requestlist) classes.+ * [`BasicCrawler`](/docs/api/basic-crawler)+ * and [`RequestList`](/docs/api/request-list) classes.* The script just downloads several web pages with plain HTTP requests (using the- * <a href="https://www.npmjs.com/package/request-promise" target="_blank">request-promise</a> library)+ * [`request-promise`](https://www.npmjs.com/package/request-promise) library)* and stores their raw HTML and URL to the default dataset.* In local configuration, the data will be stored as JSON files in `./apify_storage/datasets/default`.*- * To run this example on the Apify Platform, select the `Node.js 10 on Alpine Linux (apify/actor-node-basic)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 on Alpine Linux (apify/actor-node-basic)` base image* on the source tab of your actor configuration.*/diff --git a/examples/call_actor.js b/examples/call_actor.jsindex 3078ded60..ca8489f9d 100644--- a/examples/call_actor.js+++ b/examples/call_actor.js@@ -1,18 +1,17 @@/*** This example demonstrates how to start an Apify actor using- * [`Apify.call()`](../api/apify#module_Apify.call)- * and how to call Apify API using- * [`Apify.client`](../api/apify#module_Apify.client).- * The script extracts the current Bitcoin prices from <a href="https://www.kraken.com/" target="_blank">Kraken.com</a>- * and sends them to your email using the <a href="https://apify.com/apify/send-mail" target="_blank">apify/send-mail</a> actor.+ * [`Apify.call()`](/docs/api/apify#call) and how to call Apify API using+ * [`Apify.client`](/docs/api/apify#client).+ * The script extracts the current Bitcoin prices from [Kraken.com](https://www.kraken.com/)+ * and sends them to your email using the [`apify/send-mail`](https://apify.com/apify/send-mail) actor.*- * To make the example work, you'll need an <a href="https://my.apify.com/" target="_blank">Apify Account</a>.- * Go to <a href="https://my.apify.com/account#/integrations" target="_blank">Account - Integrations</a> page to obtain your API token- * and set it to the `APIFY_TOKEN` [environment variable](../guides/environmentvariables), or run the script using the CLI.+ * To make the example work, you'll need an [Apify Account](https://my.apify.com/).+ * Go to [Account - Integrations](https://my.apify.com/account#/integrations) page to obtain your API token+ * and set it to the [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable, or run the script using the CLI.* If you deploy this actor to the Apify Cloud then you can set up a scheduler for early* morning.*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/cheerio_crawler.js b/examples/cheerio_crawler.jsindex 53f7361e9..b9d1eed63 100644--- a/examples/cheerio_crawler.js+++ b/examples/cheerio_crawler.js@@ -1,10 +1,10 @@/**- * This example demonstrates how to use [`CheerioCrawler`](../api/cheeriocrawler)+ * This example demonstrates how to use [`CheerioCrawler`](/docs/api/cheerio-crawler)* to crawl a list of URLs from an external file,- * load each URL using a plain HTTP request, parse the HTML using <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a>+ * load each URL using a plain HTTP request, parse the HTML using [cheerio](https://www.npmjs.com/package/cheerio)* and extract some data from it: the page title and all H1 tags.*- * To run this example on the Apify Platform, select the `Node.js 10 on Alpine Linux (apify/actor-node-basic)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 on Alpine Linux (apify/actor-node-basic)` base image* on the source tab of your actor configuration.*/diff --git a/examples/forms.js b/examples/forms.jsindex 1db225025..3c0db4269 100644--- a/examples/forms.js+++ b/examples/forms.js@@ -1,16 +1,16 @@/*** This example demonstrates how to use- * [`PuppeteerCrawler`](../api/puppeteercrawler)+ * [`PuppeteerCrawler`](/docs/api/puppeteer-crawler)* to automatically fill and submit a search form to look up repositories on- * <a href="https://news.ycombinator.com" target="_blank">GitHub</a>+ * [GitHub](https://news.ycombinator.com)* using headless Chrome / Puppeteer.* The actor first fills in the search term, repository owner, start date and* language of the repository, then submits the form and prints out the results.* Finally, the results are saved either on the Apify platform to the default- * <a href="https://sdk.apify.com/docs/api/dataset" target="_blank">dataset</a>- * or on the local machine as JSON files in ./apify_storage/datasets/default.+ * [`Dataset`](/docs/api/dataset)+ * or on the local machine as JSON files in `./apify_storage/datasets/default`.*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/hello_world.js b/examples/hello_world.jsindex c6bacc417..f94254592 100644--- a/examples/hello_world.js+++ b/examples/hello_world.js@@ -1,7 +1,7 @@/*** Run the following example to perform a recursive crawl of a website using Puppeteer.*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/puppeteer_crawler.js b/examples/puppeteer_crawler.jsindex 8b499bb74..f5bafde7e 100644--- a/examples/puppeteer_crawler.js+++ b/examples/puppeteer_crawler.js@@ -1,12 +1,12 @@/**- * This example demonstrates how to use [`PuppeteerCrawler`](../api/puppeteercrawler)- * in combination with [`RequestQueue`](../api/requestqueue) to recursively scrape the- * <a href="https://news.ycombinator.com" target="_blank">Hacker News website</a> using headless Chrome / Puppeteer.+ * This example demonstrates how to use [`PuppeteerCrawler`](/docs/api/puppeteer-crawler)+ * in combination with [`RequestQueue`](/docs/api/request-queue) to recursively scrape the+ * [Hacker News website](https://news.ycombinator.com) using headless Chrome / Puppeteer.* The crawler starts with a single URL, finds links to next pages,* enqueues them and continues until no more desired links are available.* The results are stored to the default dataset. In local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/puppeteer_sitemap.js b/examples/puppeteer_sitemap.jsindex e74acfabb..0119e5bd8 100644--- a/examples/puppeteer_sitemap.js+++ b/examples/puppeteer_sitemap.js@@ -1,10 +1,10 @@/**- * This example demonstrates how to use [`PuppeteerCrawler`](../api/puppeteercrawler) to crawl a list of web pages+ * This example demonstrates how to use [`PuppeteerCrawler`](/docs/api/puppeteer-crawler) to crawl a list of web pages* specified in a sitemap. The crawler extracts page title and URL from each page* and stores them as a record to the default dataset.* In local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/puppeteer_with_proxy.js b/examples/puppeteer_with_proxy.jsindex e4950db71..362ba87ab 100644--- a/examples/puppeteer_with_proxy.js+++ b/examples/puppeteer_with_proxy.js@@ -1,13 +1,12 @@/*** This example demonstrates how to load pages in headless Chrome / Puppeteer- * over <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.- * To make it work, you'll need an Apify Account- * that has access to the proxy.- * The proxy password is available on the <a href="https://my.apify.com/proxy" target="_blank">Proxy</a> page in the app.- * Just set it to the `APIFY_PROXY_PASSWORD` [environment variable](../guides/environmentvariables)- * or run the script using the CLI.+ * over [Apify Proxy](https://docs.apify.com/proxy).+ * To make it work, you'll need an Apify Account that has access to the proxy.+ * The proxy password is available on the [Proxy](https://my.apify.com/proxy) page in the app.+ * Just set it to the [`APIFY_PROXY_PASSWORD`](../guides/environment-variables#APIFY_PROXY_PASSWORD)+ * environment variable or run the script using the CLI.*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/screenshots.js b/examples/screenshots.jsindex 14ed40121..3caab512e 100644--- a/examples/screenshots.js+++ b/examples/screenshots.js@@ -1,9 +1,7 @@/**- * This example demonstrates how to read and write- * data to the default key-value store using- * [`Apify.getValue()`](../api/apify#module_Apify.getValue)- * and- * [`Apify.setValue()`](../api/apify#module_Apify.setValue).+ * This example demonstrates how to read and write data to the default key-value store using+ * [`Apify.getValue()`](/docs/api/apify#getValue) and [`Apify.setValue()`](/docs/api/apify#setValue).+ ** The script crawls a list of URLs using Puppeteer,* captures a screenshot of each page and saves it to the store. The list of URLs is* provided as actor input that is also read from the store.@@ -17,11 +15,11 @@** On the Apify cloud, the input can be either set manually* in the UI app or passed as the POST payload to the- * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>.- * For more details, see <a href="https://docs.apify.com/actor/run#input-and-output" target="_blank">Input and output</a>+ * [Run actor API call](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor).+ * For more details, see [Input and output](https://docs.apify.com/actor/run#input-and-output)* in the Apify Actor documentation.*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/diff --git a/examples/synchronous_run.js b/examples/synchronous_run.jsindex fc1797d75..4a8838d79 100644--- a/examples/synchronous_run.js+++ b/examples/synchronous_run.js@@ -1,19 +1,19 @@/*** This example shows a quick actor that has a run time of just a few seconds.- * It opens a <a href="http://goldengatebridge75.org/news/webcam.html" target="_blank">web page</a>+ * It opens a [web page](http://goldengatebridge75.org/news/webcam.html)* that contains a webcam stream from the Golden Gate Bridge, takes a screenshot of the page and saves it as output.** This actor- * can be invoked synchronously using a single HTTP request to directly obtain its output as a reponse, using the- * <a href="https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a>+ * can be invoked synchronously using a single HTTP request to directly obtain its output as a response, using the+ * [Run actor synchronously](https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input)* Apify API endpoint. The example is also shared as the- * <a href="https://apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a>+ * [`apify/example-golden-gate-webcam`](https://apify.com/apify/example-golden-gate-webcam)* actor in the Apify library, so you can test it directly there simply by sending a POST request to* ```http* https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]* ```*- * To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image+ * To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.*/@@ -21,20 +21,20 @@ const Apify = require('apify');Apify.main(async () => {// Launch web browser.- const browser = await Apify.launchPuppeteer();+ const browser = await Apify.launchPuppeteer({ headless: true });// Load http://goldengatebridge75.org/news/webcam.html and get an IFRAME with the webcam streamconsole.log('Opening web page...');const page = await browser.newPage();await page.goto('http://goldengatebridge75.org/news/webcam.html');- const iframe = (await page.frames()).pop();+ const frames = await page.frames();+ const iframeWithVideo = frames[1];- // Get webcam image element handle.- const imageElementHandle = await iframe.$('.VideoColm img');-- // Give the webcam image some time to load.- console.log('Waiting for page to load...');- await Apify.utils.sleep(3000);+ // Wait for the webcam to load.+ console.log('Waiting for the webcam image.');+ const imageElementHandle = await iframeWithVideo.waitForSelector('.VideoColm img[src^="blob:"]');+ // Wait a little more to give the browser a chance to display the image.+ await page.waitFor(1000);// Get a screenshot of that image.const imageBuffer = await imageElementHandle.screenshot();diff --git a/src/actor.js b/src/actor.jsindex 19dec6cd7..322eccf5e 100644--- a/src/actor.js+++ b/src/actor.js@@ -121,7 +121,7 @@ const addInputOptionsOrThrow = (input, contentType, options) => {* Returns a new {@link ApifyEnv} object which contains information parsed from all the `APIFY_XXX` environment variables.** For the list of the `APIFY_XXX` environment variables, see- * <a href="https://docs.apify.com/actor/run#environment-variables" target="_blank">Actor documentation</a>.+ * [Actor documentation](https://docs.apify.com/actor/run#environment-variables).* If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.** @memberof module:Apify@@ -168,7 +168,7 @@ export const getEnv = () => {* <li>When running on the Apify platform (i.e. <code>APIFY_IS_AT_HOME</code> environment variable is set),* it sets up a connection to listen for platform events.* For example, to get a notification about an imminent migration to another server.- * See <a href="apify#apifyevents"><code>Apify.events</code></a> for details.+ * See [](apify#apifyevents) for details.* </li>* <li>It checks that either <code>APIFY_TOKEN</code> or <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable* is defined. If not, the functions sets <code>APIFY_LOCAL_STORAGE_DIR</code> to <code>./apify_storage</code>@@ -293,10 +293,10 @@ let callMemoryWarningIssued = false;* If the actor run fails, the function throws the {@link ApifyCallError} exception.** If you want to run an actor task rather than an actor, please use the- * [`Apify.callTask()`](../api/apify#module_Apify.callTask) function instead.+ * {@link Apify#callTask} function instead.** For more information about actors, read the- * <a href="https://docs.apify.com/actor" target="_blank">documentation</a>.+ * [documentation](https://docs.apify.com/actor).** **Example usage:***@@ -306,7 +306,7 @@ let callMemoryWarningIssued = false;* ```** Internally, the `call()` function invokes the- * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor</a>+ * [Run actor](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor)* and several other API endpoints to obtain the output.** @param {String} actId@@ -434,9 +434,9 @@ export const call = async (actId, input, options = {}) => {** Note that an actor task is a saved input configuration and options for an actor.* If you want to run an actor directly rather than an actor task, please use the- * [`Apify.call()`](../api/apify#module_Apify.call) function instead.+ * {@link Apify#call} function instead.*- * For more information about actor tasks, read the [`documentation`](https://docs.apify.com/tasks).+ * For more information about actor tasks, read the [documentation](https://docs.apify.com/tasks).** **Example usage:***@@ -446,7 +446,7 @@ export const call = async (actId, input, options = {}) => {* ```** Internally, the `callTask()` function calls the- * <a href="https://apify.com/docs/api/v2#/reference/actor-tasks/run-collection/run-task" target="_blank">Run task</a>+ * [Run task](https://apify.com/docs/api/v2#/reference/actor-tasks/run-collection/run-task)* and several other API endpoints to obtain the output.** @param {String} taskId@@ -612,15 +612,15 @@ export const metamorph = async (targetActorId, input, options = {}) => {* proxy-enabled applications.** For more information, see- * the <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the app- * or the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>.+ * the [Apify Proxy](https://my.apify.com/proxy) page in the app+ * or the [documentation](https://docs.apify.com/proxy).** @param {Object} options* Object with the settings below:* @param {String} [options.password] User's password for the proxy.* By default, it is taken from the `APIFY_PROXY_PASSWORD` environment variable,* which is automatically set by the system when running the actors on the Apify cloud,- * or when using the <a href="https://github.com/apifytech/apify-cli" target="_blank">Apify CLI</a>+ * or when using the [Apify CLI](https://github.com/apifytech/apify-cli)* package and the user previously logged in (called `apify login`).* @param {String[]} [options.groups] Array of Apify Proxy groups to be used.* If not provided, the proxy will select the groups automatically.@@ -716,7 +716,7 @@ export const getApifyProxyUrl = (options = {}) => {/**** Creates an ad-hoc webhook for the current actor run, which lets you receive a notification when the actor run finished or failed.- * For more information about Apify actor webhooks, please see the <a href="https://docs.apify.com/webhooks" target="_blank">documentation</a>.+ * For more information about Apify actor webhooks, please see the [documentation](https://docs.apify.com/webhooks).** Note that webhooks are only supported for actors running on the Apify platform.* In local environment, the function will print a warning and have no effect.@@ -724,22 +724,22 @@ export const getApifyProxyUrl = (options = {}) => {* @param {Object} options* @param {string[]} options.eventTypes* Array of event types, which you can set for actor run, see- * the <a href="https://docs.apify.com/webhooks/events#actor-run" target="_blank">actor run events</a> in the Apify doc.+ * the [actor run events](https://docs.apify.com/webhooks/events#actor-run) in the Apify doc.* @param {string} options.requestUrl* URL which will be requested using HTTP POST request, when actor run will reach the set event type.* @param {string} [options.payloadTemplate]* Payload template is a JSON-like string that describes the structure of the webhook POST request payload.* It uses JSON syntax, extended with a double curly braces syntax for injecting variables `{{variable}}`.* Those variables are resolved at the time of the webhook's dispatch, and a list of available variables with their descriptions- * is available in the <a href="https://docs.apify.com/webhooks" target="_blank">Apify webhook documentation</a>.+ * is available in the [Apify webhook documentation](https://docs.apify.com/webhooks).** When omitted, the default payload template will be used.- * <a href="https://docs.apify.com/webhooks" target="_blank">See the docs for the default payload template</a>.+ * [See the docs for the default payload template](https://docs.apify.com/webhooks).* @param {string} [options.idempotencyKey]* Idempotency key enables you to ensure that a webhook will not be added multiple times in case of* an actor restart or other situation that would cause the `addWebhook()` function to be called again.* We suggest using the actor run ID as the idempotency key. You can get the run ID by calling- * [`Apify.getEnv()`](apify#module_Apify.getEnv) function.+ * {@link Apify#getEnv} function.* @return {Promise<Object>} The return value is the Webhook object.* For more information, see the [Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.*diff --git a/src/autoscaling/autoscaled_pool.js b/src/autoscaling/autoscaled_pool.jsindex ee8b57bbe..cfaac1c8a 100644--- a/src/autoscaling/autoscaled_pool.js+++ b/src/autoscaling/autoscaled_pool.js@@ -34,7 +34,7 @@ const DEFAULT_OPTIONS = {* If it resolves to `true` then the pool's run finishes. Being called only* when there are no tasks being processed means that as long as `isTaskReadyFunction()`* keeps resolving to `true`, `isFinishedFunction()` will never be called.- * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.+ * To abort a run, use the {@link AutoscaledPool#abort} method.** @property {Number} [minConcurrency=1]* The minimum number of tasks running in parallel.@@ -85,16 +85,16 @@ const DEFAULT_OPTIONS = {* Meaningful data gathered from these snapshots is provided to `AutoscaledPool` by the {@link SystemStatus} class.** Before running the pool, you need to implement the following three functions:- * [`runTaskFunction()`](#new_AutoscaledPool_new),- * [`isTaskReadyFunction()`](#new_AutoscaledPool_new) and- * [`isFinishedFunction()`](#new_AutoscaledPool_new).+ * {@link AutoscaledPoolOptions#runTaskFunction},+ * {@link AutoscaledPoolOptions#isTaskReadyFunction} and+ * {@link AutoscaledPoolOptions#isFinishedFunction}.*- * The auto-scaled pool is started by calling the [`run()`](#AutoscaledPool+run) function.- * The pool periodically queries the [`isTaskReadyFunction()`](#new_AutoscaledPool_new) function+ * The auto-scaled pool is started by calling the {@link AutoscaledPool#run} function.+ * The pool periodically queries the {@link AutoscaledPoolOptions#isTaskReadyFunction} function* for more tasks, managing optimal concurrency, until the function resolves to `false`. The pool then queries- * the [`isFinishedFunction()`](#new_AutoscaledPool_new). If it resolves to `true`, the run finishes after all running tasks complete.+ * the {@link AutoscaledPoolOptions#isFinishedFunction}. If it resolves to `true`, the run finishes after all running tasks complete.* If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for tasks.- * If any of the tasks throws then the [`run()`](#AutoscaledPool+run) function rejects the promise with an error.+ * If any of the tasks throws then the {@link AutoscaledPool#run} function rejects the promise with an error.** The pool evaluates whether it should start a new task every time one of the tasks finishes* and also in the interval set by the `options.maybeRunIntervalSecs` parameter.@@ -309,7 +309,7 @@ class AutoscaledPool {/*** Aborts the run of the auto-scaled pool and destroys it. The promise returned from- * the [`run()`](#AutoscaledPool+run) function will immediately resolve, no more new tasks+ * the {@link AutoscaledPool#run} function will immediately resolve, no more new tasks* will be spawned and all running tasks will be left in their current state.** Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion@@ -330,13 +330,13 @@ class AutoscaledPool {/*** Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish- * (unlike abort, which terminates them). Used together with [`resume()`](#AutoscaledPool+resume)+ * (unlike abort, which terminates them). Used together with {@link AutoscaledPool#resume}** The function's promise will resolve once all running tasks have completed and the pool* is effectively idle. If the `timeoutSecs` argument is provided, the promise will reject* with a timeout error after the `timeoutSecs` seconds.*- * The promise returned from the [`run()`](#AutoscaledPool+run) function will not resolve+ * The promise returned from the {@link AutoscaledPool#run} function will not resolve* when `.pause()` is invoked (unlike abort, which resolves it).** @param {number} [timeoutSecs]@@ -368,7 +368,7 @@ class AutoscaledPool {/*** Resumes the operation of the autoscaled-pool by allowing more tasks to be run.- * Used together with [`pause()`](#AutoscaledPool+pause)+ * Used together with {@link AutoscaledPool#pause}** Tasks will automatically start running again in `options.maybeRunIntervalSecs`.*/diff --git a/src/autoscaling/system_status.js b/src/autoscaling/system_status.jsindex a8b8f8c33..12b1bc39e 100644--- a/src/autoscaling/system_status.js+++ b/src/autoscaling/system_status.js@@ -34,7 +34,7 @@ const DEFAULT_OPTIONS = {* @typedef {Object} SystemStatusOptions* @property {Number} [currentHistorySecs=5]* Defines max age of snapshots used in the- * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.+ * {@link SystemStatus#getCurrentStatus} measurement.* @property {Number} [maxMemoryOverloadedRatio=0.2]* Sets the maximum ratio of overloaded snapshots in a memory sample.* If the sample exceeds this ratio, the system will be overloaded.@@ -53,21 +53,21 @@ const DEFAULT_OPTIONS = {/*** Provides a simple interface to reading system status from a {@link Snapshotter} instance.- * It only exposes two functions [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)- * and [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus).+ * It only exposes two functions {@link SystemStatus#getCurrentStatus}+ * and {@link SystemStatus#getHistoricalStatus}.* The system status is calculated using a weighted average of overloaded* messages in the snapshots, with the weights being the time intervals* between the snapshots. Each resource is calculated separately* and the system is overloaded whenever at least one resource is overloaded.* The class is used by the {@link AutoscaledPool} class.*- * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)+ * {@link SystemStatus#getCurrentStatus}* returns a boolean that represents the current status of the system.* The length of the current timeframe in seconds is configurable* by the `currentHistorySecs` option and represents the max age* of snapshots to be considered for the calculation.*- * [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus)+ * {@link SystemStatus#getHistoricalStatus}* returns a boolean that represents the long-term status* of the system. It considers the full snapshot history available* in the {@link Snapshotter} instance.diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.jsindex 3d2dfda28..9bd37bf5a 100644--- a/src/crawlers/basic_crawler.js+++ b/src/crawlers/basic_crawler.js@@ -67,7 +67,7 @@ const DEFAULT_OPTIONS = {* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.* @property {RequestList} [requestList]* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).@@ -90,10 +90,10 @@ const DEFAULT_OPTIONS = {* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>+ * [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)* for the default implementation of this function.* @property {Number} [maxRequestRetries=3]- * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.+ * Indicates how many times the request is retried if {@link BasicCrawlerOptions.handleRequestFunction} fails.* @property {Number} [maxRequestsPerCrawl]* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.* Always set this value in order to prevent infinite loops in misconfigured crawlers.@@ -126,13 +126,13 @@ const DEFAULT_OPTIONS = {* If you want a crawler that already facilitates this functionality,* please consider using {@link PuppeteerCrawler} or {@link CheerioCrawler}.*- * `BasicCrawler` invokes the user-provided [`handleRequestFunction()`](#new_BasicCrawler_new)+ * `BasicCrawler` invokes the user-provided {@link BasicCrawlerOptions.handleRequestFunction}* for each {@link Request} object, which represents a single URL to crawl.* The {@link Request} objects are fed from the {@link RequestList} or the {@link RequestQueue}- * instances provided by the [`requestList`](#new_BasicCrawler_new) or [`requestQueue`](#new_BasicCrawler_new)+ * instances provided by the {@link BasicCrawlerOptions.requestList} or {@link BasicCrawlerOptions.requestQueue}* constructor options, respectively.*- * If both [`requestList`](#new_BasicCrawler_new) and [`requestQueue`](#new_BasicCrawler_new) options are used,+ * If both {@link BasicCrawlerOptions.requestList} and {@link BasicCrawlerOptions.requestQueue} options are used,* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.*@@ -173,17 +173,11 @@ const DEFAULT_OPTIONS = {** await crawler.run();* ```- *- * @property {AutoscaledPool} autoscaledPool- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.- * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.- * You can use it to change the concurrency settings on the fly,- * to pause the crawler by calling {@link AutoscaledPool#pause}- * or to abort it by calling {@link AutoscaledPool#abort}.*/class BasicCrawler {/*** @param {BasicCrawlerOptions} options+ * All `BasicCrawler` parameters are passed via an options object.*/constructor(options) {const {@@ -513,6 +507,11 @@ export default BasicCrawler;* @typedef HandleRequestInputs* @property {Request} request The original {Request} object.* @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.* @property {Session} [session]*/diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex 64691237f..104bb6a24 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -84,7 +84,7 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {* - Buffer for others MIME content types** Parsed `Content-Type` header using- * <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>+ * [content-type package](https://www.npmjs.com/package/content-type)* is stored in `contentType`.** Cheerio is available only for HTML and XML content types.@@ -100,7 +100,7 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.* @property {RequestList} [requestList]* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).@@ -153,11 +153,11 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {* If set to true, SSL certificate errors will be ignored.* @property {Boolean} [useApifyProxy=false]* If set to `true`, `CheerioCrawler` will be configured to use- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>+ * [Apify Proxy](https://my.apify.com/proxy) for all connections.+ * For more information, see the [documentation](https://docs.apify.com/proxy)* @property {String[]} [apifyProxyGroups]* An array of proxy groups to be used- * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.+ * by the [Apify Proxy](https://docs.apify.com/proxy).* Only applied if the `useApifyProxy` option is `true`.* @property {String} [apifyProxySession]* Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.@@ -183,7 +183,7 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {* where the {@link Request} instance corresponds to the failed request, and the `Error` instance* represents the last error thrown during processing of the request.*- * See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>+ * See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13)* for the default implementation of this function.* @property {String[]} [additionalMimeTypes]* An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"@@ -222,7 +222,7 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {/*** Provides a framework for the parallel crawling of web pages using plain HTTP requests and- * <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.+ * [cheerio](https://www.npmjs.com/package/cheerio) HTML parser.* The URLs to crawl are fed either from a static list of URLs* or from a dynamic queue of URLs enabling recursive crawling of websites.*@@ -232,29 +232,29 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {* because it loads the pages using full-featured headless Chrome browser.** `CheerioCrawler` downloads each URL using a plain HTTP request,- * parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>- * and then invokes the user-provided [`handlePageFunction()`](#new_CheerioCrawler_new) to extract page data- * using a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface to the parsed HTML DOM.+ * parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio)+ * and then invokes the user-provided {@link CheerioCrawlerOptions.handlePageFunction} to extract page data+ * using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.** The source URLs are represented using {@link Request} objects that are fed from- * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_CheerioCrawler_new)- * or [`requestQueue`](#new_CheerioCrawler_new) constructor options, respectively.+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link CheerioCrawlerOptions.requestList}+ * or {@link CheerioCrawlerOptions.requestQueue} constructor options, respectively.*- * If both [`requestList`](#new_CheerioCrawler_new) and [`requestQueue`](#new_CheerioCrawler_new) are used,+ * If both {@link CheerioCrawlerOptions.requestList} and {@link CheerioCrawlerOptions.requestQueue} are used,* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.** The crawler finishes when there are no more {@link Request} objects to crawl.*- * `CheerioCrawler` downloads the web pages using the {@link requestAsBrowser} utility function.+ * `CheerioCrawler` downloads the web pages using the {@link utils#requestAsBrowser} utility function.* You can use the `requestOptions` parameter to pass additional options to this function.** By default, `CheerioCrawler` only processes web pages with the `text/html`* and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),* and skips pages with other content types. If you want the crawler to process other content types,- * use the [`additionalMimeTypes`](#new_CheerioCrawler_new) constructor option.+ * use the {@link CheerioCrawlerOptions.additionalMimeTypes} constructor option.* Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.- * For details, see {@link CheerioCrawlerOptions#handlePageFunction}.+ * For details, see {@link CheerioCrawlerOptions.handlePageFunction}.** New requests are only dispatched when there is enough free CPU and memory available,* using the functionality provided by the {@link AutoscaledPool} class.@@ -296,16 +296,11 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {** await crawler.run();* ```- * @property {AutoscaledPool} autoscaledPool- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.- * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.- * You can use it to change the concurrency settings on the fly,- * to pause the crawler by calling {@link AutoscaledPool#pause}- * or to abort it by calling {@link AutoscaledPool#abort}.*/class CheerioCrawler {/*** @param {CheerioCrawlerOptions} options+ * All `CheerioCrawler` parameters are passed via an options object.*/constructor(options = {}) {const {@@ -700,13 +695,24 @@ export default CheerioCrawler;/*** @typedef CheerioHandlePageInputs- * @property {Cheerio} [$] The <a href="https://cheerio.js.org/">Cheerio</a> object with parsed HTML.- * @property {String|Buffer} body The request body of the web page.- * @property {Object} [json] The parsed object from JSON string if the response contains the content type application/json.- * @property {Request} request The original {Request} object.- * @property {{ type: string, encoding: string }} contentType Parsed `Content-Type header: { type, encoding }`.- * @property {IncomingMessage} response An instance of Node's http.IncomingMessage object,+ * @property {Cheerio} [$]+ * The [Cheerio](https://cheerio.js.org/) object with parsed HTML.+ * @property {String|Buffer} body+ * The request body of the web page.+ * @property {Object} [json]+ * The parsed object from JSON string if the response contains the content type application/json.+ * @property {Request} request+ * The original {Request} object.+ * @property {{ type: string, encoding: string }} contentType+ * Parsed `Content-Type header: { type, encoding }`.+ * @property {IncomingMessage} response+ * An instance of Node's http.IncomingMessage object,* @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.* @property {session} [session]*//**diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.jsindex 8068fea8a..6aa95d688 100644--- a/src/crawlers/puppeteer_crawler.js+++ b/src/crawlers/puppeteer_crawler.js@@ -41,9 +41,9 @@ import { SessionPoolOptions } from '../session_pool/session_pool';** `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.* `page` is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)* `response` is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),* which is the main resource response as returned by `page.goto(request.url)`.* `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.*@@ -56,7 +56,7 @@ import { SessionPoolOptions } from '../session_pool/session_pool';* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.* @property {RequestList} [requestList]* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).@@ -67,17 +67,17 @@ import { SessionPoolOptions } from '../session_pool/session_pool';* Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.* @property {PuppeteerGoto} [gotoFunction]* Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,- * i.e. a `Promise` resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.+ * [page.goto()](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options) function,+ * i.e. a `Promise` resolving to the [Response](https://pptr.dev/#?product=Puppeteer&show=api-class-response) object.** This is useful if you need to extend the page load timeout or select different criteria* to determine that the navigation succeeded.** Note that a single page object is only used to process a single request and it is closed afterwards.*- * By default, the function invokes [`Apify.utils.puppeteer.gotoExtended()`](puppeteer#puppeteer.gotoExtended) with a timeout of 60 seconds.+ * By default, the function invokes {@link puppeteer#gotoExtended} with a timeout of 60 seconds.* For details, see source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292).* @property {Number} [gotoTimeoutSecs=60]* Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default* function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.@@ -95,7 +95,7 @@ import { SessionPoolOptions } from '../session_pool/session_pool';* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>+ * [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301)* for the default implementation of this function.* @property {Number} [maxRequestRetries=3]* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.@@ -110,34 +110,37 @@ import { SessionPoolOptions } from '../session_pool/session_pool';* Overrides the default function to launch a new Puppeteer instance.* Shortcut to the corresponding {@link PuppeteerPool} option.* See source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)* for default behavior.* @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]- * Options used by [`Apify.launchPuppeteer()`](apify#module_Apify.launchPuppeteer) to start new Puppeteer instances.- * Shortcut to the corresponding {@link PuppeteerPool} option. See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.+ * Shortcut to the corresponding {@link PuppeteerPool} option.* @property {AutoscaledPoolOptions} [autoscaledPoolOptions]* Custom options passed to the underlying {@link AutoscaledPool} instance constructor.* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options* are provided by `PuppeteerCrawler` and should not be overridden.* @property {Number} [minConcurrency=1]- * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the+ * corresponding {@link AutoscaledPoolOptions.minConcurrency} option.*- * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.- * If you're not sure, just keep the default value and the concurrency will scale up automatically.+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU,+ * your crawler will run extremely slow or crash. If you're not sure, just keep the default value+ * and the concurrency will scale up automatically.* @property {Number} [maxConcurrency=1000]- * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the+ * corresponding {@link AutoscaledPoolOptions.maxConcurrency} option.* @property {Boolean} [useSessionPool=false]- * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.- * It also marks Session as bad after a request timeout.+ * If set to true Crawler will automatically use Session Pool. It will automatically retire+ * sessions on 403, 401 and 429 status codes. It also marks Session as bad after a request timeout.* @property {SessionPoolOptions} [sessionPoolOptions]* Custom options passed to the underlying {@link SessionPool} constructor.- * @property {Boolean} [persistCookiesPerSession]+ * @property {Boolean} [persistCookiesPerSession=false]* Automatically saves cookies to Session. Works only if Session Pool is used.*//*** Provides a simple framework for parallel crawling of web pages- * using headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.+ * using headless Chrome with [Puppeteer](https://github.com/GoogleChrome/puppeteer).* The URLs to crawl are fed either from a static list of URLs* or from a dynamic queue of URLs enabling recursive crawling of websites.*@@ -147,28 +150,27 @@ import { SessionPoolOptions } from '../session_pool/session_pool';* which downloads the pages using raw HTTP requests and is about 10x faster.** The source URLs are represented using {@link Request} objects that are fed from- * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_PuppeteerCrawler_new)- * or [`requestQueue`](#new_PuppeteerCrawler_new) constructor options, respectively.+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link PuppeteerCrawlerOptions.requestList}+ * or {@link PuppeteerCrawlerOptions.requestQueue} constructor options, respectively.*- * If both [`requestList`](#new_PuppeteerCrawler_new) and [`requestQueue`](#new_PuppeteerCrawler_new) are used,+ * If both {@link PuppeteerCrawlerOptions.requestList} and {@link PuppeteerCrawlerOptions.requestQueue} are used,* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.** The crawler finishes when there are no more {@link Request} objects to crawl.** `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl- * and then calls the function provided by user as the [`handlePageFunction()`](#new_PuppeteerCrawler_new) option.+ * and then calls the function provided by user as the {@link PuppeteerCrawlerOptions.handlePageFunction} option.** New pages are only opened when there is enough free CPU and memory available,* using the functionality provided by the {@link AutoscaledPool} class.- * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`+ * All {@link AutoscaledPool} configuration options can be passed to the {@link PuppeteerCrawlerOptions.autoscaledPoolOptions}* parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`- * {@link AutoscaledPool} options are available directly in the `PuppeteerCrawler` constructor.+ * {@link AutoscaledPoolOptions} are available directly in the `PuppeteerCrawler` constructor.*- * Note that the pool of Puppeteer instances is internally managed by- * the {@link PuppeteerPool} class. Many constructor options- * such as `maxOpenPagesPerInstance` or `launchPuppeteerFunction` are passed directly- * to {@link PuppeteerPool} constructor.+ * Note that the pool of Puppeteer instances is internally managed by the {@link PuppeteerPool} class.+ * Many constructor options such as {@link PuppeteerPoolOptions.maxOpenPagesPerInstance} or+ * {@link PuppeteerPoolOptions.launchPuppeteerFunction} are passed directly to the {@link PuppeteerPool} constructor.** **Example usage:***@@ -197,17 +199,11 @@ import { SessionPoolOptions } from '../session_pool/session_pool';** await crawler.run();* ```- * @property {AutoscaledPool} autoscaledPool- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.- * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.- * You can use it to change the concurrency settings on the fly,- * to pause the crawler by calling {@link AutoscaledPool#pause}- * or to abort it by calling {@link AutoscaledPool#abort}.*/class PuppeteerCrawler {/**- * @param {PuppeteerCrawlerOptions} options All `PuppeteerCrawler` parameters are passed- * via an options object with the following keys:+ * @param {PuppeteerCrawlerOptions} options+ * All `PuppeteerCrawler` parameters are passed via an options object.*/constructor(options) {const {@@ -386,14 +382,21 @@ export default PuppeteerCrawler;/*** @typedef PuppeteerHandlePageInputs- * @property {Request} request An instance of the {@link Request} object with details about the URL to open, HTTP method etc.- * @property {PuppeteerResponse} response An instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ * @property {Request} request+ * An instance of the {@link Request} object with details about the URL to open, HTTP method etc.+ * @property {PuppeteerResponse} response An instance of the Puppeteer+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),* which is the main resource response as returned by `page.goto(request.url)`.- * @property {PuppeteerPage} page is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>- * @property {PuppeteerPool} puppeteerPool An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.+ * @property {PuppeteerPage} page is an instance of the Puppeteer+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)+ * @property {PuppeteerPool} puppeteerPool+ * An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.* @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.* @property {Session} [session]*//**@@ -404,8 +407,8 @@ export default PuppeteerCrawler;/*** @typedef PuppeteerGotoInputs- * @property {PuppeteerPage} page is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * @property {PuppeteerPage} page is an instance of the Puppeteer+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)* @property {Request} request An instance of the {@link Request} object with details about the URL to open, HTTP method etc.* @property {AutoscaledPool} autoscaledPool An instance of the `AutoscaledPool`.* @property {PuppeteerPool} puppeteerPool An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.@@ -414,8 +417,8 @@ export default PuppeteerCrawler;/*** @callback PuppeteerGoto* @param {PuppeteerGotoInputs} inputs Arguments passed to this callback.- * @return {Promise<PuppeteerResponse>} An instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ * @return {Promise<PuppeteerResponse>} An instance of the Puppeteer+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),* which is the main resource response as returned by `page.goto(request.url)`.*/@@ -424,7 +427,7 @@ export default PuppeteerCrawler;* @param {LaunchPuppeteerOptions} inputs Arguments passed to this callback.* @return {Promise<Browser>} Promise that resolves to Puppeteer's `Browser` instance.* This might be obtained by calling- * <a href="https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions">puppeteer.launch()</a>+ * [puppeteer.launch()](https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions)* directly, or by delegating to- * [`Apify.launchPuppeteer()`](../api/apify#apifylaunchpuppeteeroptions-%E2%87%92-promisebrowser).+ * {@link Apify#launchPuppeteer}.*/diff --git a/src/dataset.js b/src/dataset.jsindex 0eb062a87..e1c563002 100644--- a/src/dataset.js+++ b/src/dataset.js@@ -104,7 +104,7 @@ export const chunkBySize = (items, limitBytes) => {* Typically it is used to store crawling results.** Do not instantiate this class directly, use the- * [`Apify.openDataset()`](apify#module_Apify.openDataset) function instead.+ * {@link Apify#openDataset} function instead.** `Dataset` stores its data either on local disk or in the Apify cloud,* depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.@@ -119,9 +119,9 @@ export const chunkBySize = (items, limitBytes) => {* Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://docs.apify.com/storage/dataset" target="_blank">Apify Dataset</a>+ * [Apify Dataset](https://docs.apify.com/storage/dataset)* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`- * option to [`Apify.openDataset()`](apify#module_Apify.openDataset) function,+ * option to {@link Apify#openDataset} function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.** **Example usage:**@@ -695,7 +695,7 @@ export const openDataset = (datasetIdOrName, options = {}) => {/*** Stores an object or an array of objects to the default {@link Dataset} of the current actor run.*- * This is just a convenient shortcut for [`dataset.pushData()`](dataset#Dataset+pushData).+ * This is just a convenient shortcut for {@link Dataset#pushData}.* For example, calling the following code:* ```javascript* await Apify.pushData({ myValue: 123 });@@ -707,7 +707,7 @@ export const openDataset = (datasetIdOrName, options = {}) => {* await dataset.pushData({ myValue: 123 });* ```*- * For more information, see [`Apify.openDataset()`](apify#module_Apify.openDataset) and [`dataset.pushData()`](dataset#Dataset+pushData)+ * For more information, see {@link Apify#openDataset} and {@link Dataset#pushData}** **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,* otherwise the actor process might finish before the data are stored!diff --git a/src/enqueue_links/click_elements.js b/src/enqueue_links/click_elements.jsindex fc83d5c37..275ec0ba1 100644--- a/src/enqueue_links/click_elements.js+++ b/src/enqueue_links/click_elements.js@@ -16,7 +16,7 @@ const STARTING_Z_INDEX = 2147400000;* requests, including their methods, headers and payloads are then enqueued to a provided* {@link RequestQueue}. This is useful to crawl JavaScript heavy pages where links are not available* in `href` elements, but rather navigations are triggered in click handlers.- * If you're looking to find URLs in `href` attributes of the page, see [`enqueueLinks()`](utils#utils.enqueueLinks).+ * If you're looking to find URLs in `href` attributes of the page, see {@link utils#enqueueLinks}.** Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects* and override settings of the enqueued {@link Request} objects.@@ -58,11 +58,11 @@ const STARTING_Z_INDEX = 2147400000;* All `enqueueLinksByClickingElements()` parameters are passed* via an options object with the following keys:* @param {Page} options.page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {RequestQueue} options.requestQueue* A request queue to which the URLs will be enqueued.* @param {String} options.selector- * A CSS selector matching elements to be clicked on. Unlike in [`enqueueLinks()`](utils#utils.enqueueLinks), there is no default+ * A CSS selector matching elements to be clicked on. Unlike in {@link utils#enqueueLinks}, there is no default* value. This is to prevent suboptimal use of this function by using it too broadly.* @param {Array<String|RegExp|Object>} [options.pseudoUrls]* An array of {@link PseudoUrl}s matching the URLs to be enqueued,@@ -115,6 +115,7 @@ const STARTING_Z_INDEX = 2147400000;* Promise that resolves to an array of {@link QueueOperationInfo} objects.* @memberOf puppeteer* @name enqueueLinksByClickingElements+ * @function*/export async function enqueueLinksByClickingElements(options = {}) {const {diff --git a/src/enqueue_links/enqueue_links.js b/src/enqueue_links/enqueue_links.jsindex 6b1f24cfe..8dec5e63f 100644--- a/src/enqueue_links/enqueue_links.js+++ b/src/enqueue_links/enqueue_links.js@@ -23,7 +23,7 @@ import { Cheerio } from '../typedefs';* and enqueues the URLs in their `href` attributes to the provided {@link RequestQueue}.* If you're looking to find URLs in JavaScript heavy pages where links are not available* in `href` elements, but rather navigations are triggered in click handlers- * see [`enqueueLinksByClickingElements()`](puppeteer#puppeteer.enqueueLinksByClickingElements).+ * see {@link puppeteer#enqueueLinksByClickingElements}.** Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects* and override settings of the enqueued {@link Request} objects.@@ -52,10 +52,10 @@ import { Cheerio } from '../typedefs';* All `enqueueLinks()` parameters are passed* via an options object with the following keys:* @param {Page} options.page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* Either `page` or `$` option must be provided.* @param {Cheerio} options.$- * <a href="https://github.com/cheeriojs/cheerio" target="_blank"><code>Cheerio</code></a> object with loaded HTML.+ * [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML.* Either `page` or `$` option must be provided.* @param {RequestQueue} options.requestQueue* A request queue to which the URLs will be enqueued.@@ -97,6 +97,7 @@ import { Cheerio } from '../typedefs';* Promise that resolves to an array of {@link QueueOperationInfo} objects.* @memberOf utils* @name enqueueLinks+ * @function*/export async function enqueueLinks(options = {}) {const {diff --git a/src/enqueue_links/shared.js b/src/enqueue_links/shared.jsindex 60f8ab1cb..4234f80ec 100644--- a/src/enqueue_links/shared.js+++ b/src/enqueue_links/shared.js@@ -107,7 +107,7 @@ export async function addRequestsToQueueInBatches(requests, requestQueue, batchS/*** Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used- * [`Apify.utils.enqueueLinks`](../api/utils#utils.enqueueLinks) to modify requests before enqueuing them.+ * {@link utils#enqueueLinks} to modify requests before enqueuing them.* @callback RequestTransform* @param {RequestOptions} original Request options to be modified.* @return {RequestOptions} The modified request options to enqueue.diff --git a/src/errors.js b/src/errors.jsindex 24df0757f..280fca823 100644--- a/src/errors.js+++ b/src/errors.js@@ -7,7 +7,7 @@ export const APIFY_CALL_ERROR_NAME = 'ApifyCallError';/*** The class represents exceptions thrown- * by the [`Apify.call()`](../api/apify#module_Apify.call) function.+ * by the {@link Apify#call} function.** @typedef {Object} ApifyCallError* @property {String} messagediff --git a/src/events.js b/src/events.jsindex d8d750799..295e42b90 100644--- a/src/events.js+++ b/src/events.js@@ -27,9 +27,9 @@ let persistStateInterval = null;/*** Gets an instance of a Node.js'- * <a href="https://nodejs.org/api/events.html#events_class_eventemitter" target="_blank">EventEmitter</a>+ * [EventEmitter](https://nodejs.org/api/events.html#events_class_eventemitter)* class that emits various events from the SDK or the Apify platform.- * The event emitter is initialized by calling the [`Apify.main()`](#module_Apify.main) function.+ * The event emitter is initialized by calling the {@link Apify#main} function.** **Example usage:***@@ -56,7 +56,7 @@ let persistStateInterval = null;* The event is emitted approximately every second* and it indicates whether the actor is using the maximum of available CPU resources.* If that's the case, the actor should not add more workload.- * For example, this event is used by the <a href="autoscaledpool"><code>AutoscaledPool</code></a> class.+ * For example, this event is used by the {@link AutoscaledPool} class.* </td>* </tr>* <tr>@@ -67,7 +67,7 @@ let persistStateInterval = null;* <td colspan="2">* Emitted when the actor running on the Apify platform is going to be migrated to another worker server soon.* You can use it to persist the state of the actor and abort the run, to speed up migration.- * For example, this is used by the <a href="requestlist"><code>RequestList</code></a> class.+ * For example, this is used by the {@link RequestList} class.* </td>* </tr>* <tr>diff --git a/src/index.esm.js b/src/index.esm.jsindex c472763e6..7587a75d4 100644--- a/src/index.esm.js+++ b/src/index.esm.js@@ -40,7 +40,8 @@ const exportedUtils = Object.assign(publicUtils, {/*** The following section describes all functions and properties provided by the `apify` package,* except individual classes and namespaces that have their separate, detailed, documentation pages- * accessible from the left sidebar.+ * accessible from the left sidebar. To learn how Apify SDK works, we suggest following+ * the [Getting Started](/docs/guides/getting-started) tutorial.** @module Apify*/diff --git a/src/key_value_store.js b/src/key_value_store.jsindex c81762402..a5ec102ce 100644--- a/src/key_value_store.js+++ b/src/key_value_store.js@@ -103,20 +103,20 @@ export const maybeStringify = (value, options) => {* for saving screenshots, actor inputs and outputs, web pages, PDFs or to persist the state of crawlers.** Do not instantiate this class directly, use the- * [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function instead.+ * {@link Apify#openKeyValueStore} function instead.** Each actor run is associated with a default key-value store, which is created exclusively* for the run. By convention, the actor input and output are stored into the* default key-value store under the `INPUT` and `OUTPUT` key, respectively.* Typically, input and output are JSON files, although it can be any other format.* To access the default key-value store directly, you can use the- * [`Apify.getValue()`](apify#module_Apify.getValue)- * and [`Apify.setValue()`](apify#module_Apify.setValue) convenience functions.+ * {@link Apify#getValue} and {@link Apify#setValue} convenience functions.*- * To access the input, you can also use the [`Apify.getInput()`](apify#module_Apify.getInput) convenience function.+ * To access the input, you can also use the {@link Apify#getInput} convenience function.** `KeyValueStore` stores its data either on local disk or in the Apify cloud,- * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.+ * depending on whether the [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR)+ * or [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variables are set.** If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in* the local directory in the following files:@@ -127,11 +127,12 @@ export const maybeStringify = (value, options) => {* unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.* The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.*- * If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://docs.apify.com/storage/key-value-store" target="_blank">Apify Key-value store</a>+ * If the [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable is set but+ * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) not,+ * the data is stored in the [Apify Key-value store](https://docs.apify.com/storage/key-value-store)* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`- * option to [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function,- * even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.+ * option to {@link Apify#openKeyValueStore} function, even if the+ * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) variable is set.** **Example usage:***@@ -181,7 +182,7 @@ export class KeyValueStore {* content type are automatically parsed and returned as a JavaScript object.* Similarly, records with `text/plain` content types are returned as a string.* For all other content types, the value is returned as a raw- * <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a> instance.+ * [](https://nodejs.org/api/buffer.html) instance.** If the record does not exist, the function resolves to `null`.*@@ -191,7 +192,7 @@ export class KeyValueStore {* **Example usage:**** ```javascript- * const store = await Apify.openKeyValueStore('my-screenshots');+ * const store = await Apify.openKeyValueStore();* const buffer = await store.getValue('screenshot1.png');* ```*@@ -200,7 +201,7 @@ export class KeyValueStore {* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`* @returns {Promise<Object|String|Buffer>}* Returns a promise that resolves to an object, string- * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, depending+ * or [](https://nodejs.org/api/buffer.html), depending* on the MIME content type of the record.*/getValue(key) {@@ -221,8 +222,8 @@ export class KeyValueStore {* **Example usage:**** ```javascript- * const store = await Apify.openKeyValueStore('my-store');- * await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });+ * const store = await Apify.openKeyValueStore();+ * await store.setValue('OUTPUT', { foo: 'bar' });* ```** Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`@@ -231,11 +232,11 @@ export class KeyValueStore {* `application/json; charset=utf-8` MIME content type.* To store the value with another content type, pass it in the options as follows:* ```javascript- * const store = await Apify.openKeyValueStore('my-store');+ * const store = await Apify.openKeyValueStore('my-text-store');* await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });* ```* If you set custom content type, `value` must be either a string or- * <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, otherwise an error will be thrown.+ * [](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.** If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds* regardless whether the record existed or not.@@ -255,7 +256,7 @@ export class KeyValueStore {* <li>If `null`, the record in the key-value store is deleted.</li>* <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>* <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`- * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>.</li>+ * or [](https://nodejs.org/api/buffer.html).</li>* </ul>* For any other value an error will be thrown.* @param {Object} [options]@@ -611,7 +612,7 @@ export const openKeyValueStore = (storeIdOrName, options = {}) => {/*** Gets a value from the default {@link KeyValueStore} associated with the current actor run.*- * This is just a convenient shortcut for [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+ * This is just a convenient shortcut for {@link KeyValueStore#getValue}.* For example, calling the following code:* ```javascript* const value = await Apify.getValue('my-key');@@ -623,10 +624,10 @@ export const openKeyValueStore = (storeIdOrName, options = {}) => {* const value = await store.getValue('my-key');* ```*- * To store the value to the default-key value store, you can use the [`Apify.setValue()`](#module_Apify.setValue) function.+ * To store the value to the default-key value store, you can use the {@link Apify#setValue} function.*- * For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)- * and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+ * For more information, see {@link Apify#openKeyValueStore}+ * and {@link KeyValueStore#getValue}.** @param {String} key* Unique record key.@@ -646,7 +647,7 @@ export const getValue = async (key) => {/*** Stores or deletes a value in the default {@link KeyValueStore} associated with the current actor run.*- * This is just a convenient shortcut for [`keyValueStore.setValue()`](keyvaluestore#KeyValueStore+setValue).+ * This is just a convenient shortcut for {@link KeyValueStore#setValue}.* For example, calling the following code:* ```javascript* await Apify.setValue('OUTPUT', { foo: "bar" });@@ -658,10 +659,10 @@ export const getValue = async (key) => {* await store.setValue('OUTPUT', { foo: "bar" });* ```*- * To get a value from the default-key value store, you can use the [`Apify.getValue()`](#module_Apify.getValue) function.+ * To get a value from the default-key value store, you can use the {@link Apify#getValue} function.*- * For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)- * and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+ * For more information, see {@link Apify#openKeyValueStore}+ * and {@link KeyValueStore#getValue}.** @param {String} key* Unique record key.@@ -671,7 +672,7 @@ export const getValue = async (key) => {* <li>If `null`, the record in the key-value store is deleted.</li>* <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>* <li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`- * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>.</li>+ * or [](https://nodejs.org/api/buffer.html).</li>* </ul>* For any other value an error will be thrown.* @param {Object} [options]@@ -691,7 +692,7 @@ export const setValue = async (key, value, options) => {/*** Gets the actor input value from the default {@link KeyValueStore} associated with the current actor run.*- * This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](keyvaluestore#KeyValueStore+getValue).+ * This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](keyvaluestore#getvalue).* For example, calling the following code:* ```javascript* const input = await Apify.getInput();@@ -703,19 +704,20 @@ export const setValue = async (key, value, options) => {* await store.getValue('INPUT');* ```*- * For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)- * and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).+ * For more information, see {@link Apify#openKeyValueStore}+ * and {@link KeyValueStore#getValue}.** @returns {Promise<Object>}* Returns a promise that resolves once the record is stored.* @memberof module:Apify* @name getInput+ * @function*/export const getInput = async () => getValue(process.env[ENV_VARS.INPUT_KEY] || KEY_VALUE_STORE_KEYS.INPUT);/**- * User-function used in the [`KeyValueStore.forEachKey()`](../api/keyvaluestore#forEachKey) method.+ * User-function used in the {@link KeyValueStore#forEachKey} method.* @callback KeyConsumer* @param {String} key* Current {KeyValue} key being processed.diff --git a/src/pseudo_url.js b/src/pseudo_url.jsindex c36f69636..bb52b5f7e 100644--- a/src/pseudo_url.js+++ b/src/pseudo_url.js@@ -52,7 +52,7 @@ const parsePurl = (purl) => {/*** Represents a pseudo-URL (PURL) - an URL pattern used by web crawlers* to specify which URLs should the crawler visit.- * This class is used by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.+ * This class is used by the {@link utils#enqueueLinks} function.** A PURL is simply a URL with special directives enclosed in `[]` brackets.* Currently, the only supported directive is `[RegExp]`,@@ -107,7 +107,7 @@ class PseudoUrl {* such as making the matching case sensitive.* @param {RequestOptions} requestTemplate* Options for the new {@link Request} instances created for matching URLs- * by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.+ * by the {@link utils#enqueueLinks} function.*/constructor(purl, requestTemplate = {}) {checkParamOrThrow(purl, 'purl', 'String|RegExp');diff --git a/src/puppeteer.js b/src/puppeteer.jsindex 6f9a9388b..e68f5c9fb 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -105,9 +105,9 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {// https://github.com/Microsoft/TypeScript/issues/20077/*** Apify extends the launch options of Puppeteer.- * You can use any of the- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>- * options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)+ * You can use any of the Puppeteer compatible+ * [`LaunchOptions`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions)+ * options in the {@link Apify#launchPuppeteer}* function and in addition, all the options available below.** @typedef {Object} LaunchPuppeteerOptions@@ -129,11 +129,11 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {* By default, this option is `false`.* @property {Boolean} [useApifyProxy=false]* If set to `true`, Puppeteer will be configured to use- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>+ * [Apify Proxy](https://my.apify.com/proxy) for all connections.+ * For more information, see the [documentation](https://docs.apify.com/proxy)* @property {String[]} [apifyProxyGroups]* An array of proxy groups to be used- * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.+ * by the [Apify Proxy](https://docs.apify.com/proxy).* Only applied if the `useApifyProxy` option is `true`.* @property {String} [apifyProxySession]* Apify Proxy session identifier to be used by all the Chrome browsers.@@ -182,7 +182,7 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {* </li>* <li>* If <code>options.useApifyProxy</code> is <code>true</code> then the function generates a URL of- * <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>+ * [Apify Proxy](https://docs.apify.com/proxy)* based on <code>options.apifyProxyGroups</code> and <code>options.apifyProxySession</code> and passes it as <code>options.proxyUrl</code>.* </li>* <li>@@ -197,11 +197,11 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {* </li>* </ul>*- * To use this function, you need to have the <a href="https://www.npmjs.com/package/puppeteer" target="_blank">puppeteer</a>+ * To use this function, you need to have the [puppeteer](https://www.npmjs.com/package/puppeteer)* NPM package installed in your project.* When running on the Apify cloud, you can achieve that simply* by using the `apify/actor-node-chrome` base Docker image for your actor - see- * <a href="https://docs.apify.com/actor/build#base-images" target="_blank">Apify Actor documentation</a>+ * [Apify Actor documentation](https://docs.apify.com/actor/build#base-images)* for details.** For an example of usage, see the [Synchronous run Example](../examples/synchronousrun)@@ -209,8 +209,8 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {** @param {LaunchPuppeteerOptions} [options]* Optional settings passed to `puppeteer.launch()`. In addition to- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank">Puppeteer's options</a>- * the object may contain our own [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions) that enable additional features.+ * [Puppeteer's options](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions)+ * the object may contain our own {@link LaunchPuppeteerOptions} that enable additional features.* @returns {Promise<Browser>}* Promise that resolves to Puppeteer's `Browser` instance.* @memberof module:Apifydiff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.jsindex ff76a91f2..921123024 100644--- a/src/puppeteer_pool.js+++ b/src/puppeteer_pool.js@@ -100,11 +100,10 @@ class PuppeteerInstance {* The function must return a promise resolving to* [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.* See the source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)* for the default implementation.* @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]- * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.- * See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.* @property {boolean} [recycleDiskCache=false]* Enables recycling of disk cache directories by Chrome instances.* When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.@@ -129,7 +128,7 @@ class PuppeteerInstance {/*** Manages a pool of Chrome browser instances controlled using- * <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.+ * [Puppeteer](https://github.com/GoogleChrome/puppeteer).** `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.* This is useful in order to facilitate rotation of proxies, cookies@@ -168,7 +167,7 @@ class PuppeteerPool {/*** @param {PuppeteerPoolOptions} [options]* All `PuppeteerPool` parameters are passed- * via an options object with the following keys:+ * via an options object.*/constructor(options = {}) {const {@@ -503,7 +502,7 @@ class PuppeteerPool {/*** Opens new tab in one of the browsers in the pool and returns a `Promise`* that resolves to an instance of a Puppeteer- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>.+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page).** @return {Promise<Page>}* @ignore@@ -654,7 +653,7 @@ class PuppeteerPool {/*** Manually retires a Puppeteer- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-browser" target="_blank"><code>Browser</code></a>+ * [`Browser`](https://pptr.dev/#?product=Puppeteer&show=api-class-browser)* instance from the pool. The browser will continue to process open pages so that they may gracefully finish.* This is unlike `browser.close()` which will forcibly terminate the browser and all open pages will be closed.* @param {Browser} browserdiff --git a/src/puppeteer_request_interception.js b/src/puppeteer_request_interception.jsindex ef8e50808..2c17ea194 100644--- a/src/puppeteer_request_interception.js+++ b/src/puppeteer_request_interception.js@@ -107,7 +107,7 @@ const handleRequest = (request, interceptRequestHandlers) => {* ```** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Function} handler Request interception handler.* @return {Promise}* @memberOf puppeteer@@ -142,7 +142,7 @@ export const addInterceptRequestHandler = async (page, handler) => {* Removes request interception handler for given page.** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Function} handler Request interception handler.* @return {Promise}* @memberOf puppeteerdiff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.jsindex a070fa317..72bb85346 100644--- a/src/puppeteer_utils.js+++ b/src/puppeteer_utils.js@@ -27,7 +27,7 @@ const DEFAULT_BLOCK_REQUEST_URL_PATTERNS = ['.css', '.jpg', '.jpeg', '.png', '.s* The function should be called on a newly-created page object before navigating to the target crawled page.** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @return {Promise}* @memberOf puppeteer* @ignore@@ -78,7 +78,7 @@ const injectedFilesCache = new LruCache({ maxLength: MAX_INJECT_FILE_CACHE_SIZE* File contents are cached for up to 10 files to limit file system access.** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {String} filePath File path* @param {Object} [options]* @param {boolean} [options.surviveNavigations]@@ -105,7 +105,7 @@ const injectFile = async (page, filePath, options = {}) => {};/**- * Injects the <a href="https://jquery.com/" target="_blank"><code>jQuery</code></a> library into a Puppeteer page.+ * Injects the [](https://jquery.com/) library into a Puppeteer page.* jQuery is often useful for various web scraping and crawling tasks.* For example, it can help extract text from HTML elements using CSS selectors.*@@ -124,11 +124,11 @@ const injectFile = async (page, filePath, options = {}) => {* ```** Note that `injectJQuery()` does not affect the Puppeteer's- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-pageselector" target="_blank"><code>Page.$()</code></a>+ * [](https://pptr.dev/#?product=Puppeteer&show=api-pageselector)* function in any way.** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @return {Promise}* @memberOf puppeteer*/@@ -138,7 +138,7 @@ const injectJQuery = (page) => {};/**- * Injects the <a href="https://underscorejs.org/" target="_blank"><code>Underscore.js</code></a> library into a Puppeteer page.+ * Injects the [](https://underscorejs.org/) library into a Puppeteer page.** Beware that the injected Underscore object will be set to the `window._` variable and thus it might cause conflicts with* libraries included by the page that use the same variable name.@@ -230,7 +230,7 @@ const enqueueRequestsFromClickableElements = async (page, selector, purls, reque* ```** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} [options]* @param {string[]} [options.urlPatterns]* The patterns of URLs to block from being loaded by the browser.@@ -282,7 +282,7 @@ const blockResources = async (page, resourceTypes = ['stylesheet', 'font', 'imag* *IMPORTANT*: Caching responses stores them to memory, so too loose rules could cause memory leaks for longer running crawlers.* This issue should be resolved or atleast mitigated in future iterations of this feature.* @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} cache* Object in which responses are stored* @param {Array<String|RegExp>} responseUrlRules@@ -349,7 +349,7 @@ const cacheResponses = async (page, cache, responseUrlRules) => {* request: Request,* }* ```- * Where `page` is a Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * Where `page` is a Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page)* and `request` is a {@link Request}.** The function is compiled by using the `scriptString` parameter as the function's body,@@ -396,7 +396,7 @@ const compileScript = (scriptString, context = Object.create(null)) => {* browser cache which degrades performance.** @param {Page} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Request} request* @param {Object} gotoOptions Custom options for `page.goto()`.* @return {Promise<Response>}@@ -441,7 +441,7 @@ export const gotoExtended = async (page, request, gotoOptions = {}) => {* Scrolls to the bottom of a page, or until it times out.* Loads dynamic content when it hits the bottom of a page, and then continues scrolling.* @param {Object} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} [options]* @param {Number} [options.timeoutSecs=0]* How many seconds to scroll for. If 0, will scroll until bottom of page.@@ -510,7 +510,7 @@ export const infiniteScroll = async (page, options = {}) => {/*** Saves a full screenshot and HTML of the current page into a Key-Value store.* @param {Object} page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} [options]* @param {String} [options.key=SNAPSHOT]* Key under which the screenshot and HTML will be saved. `.jpg` will be appended for screenshot and `.html` for HTML.diff --git a/src/request.js b/src/request.jsindex 8f2055c87..6fa9be832 100644--- a/src/request.js+++ b/src/request.js@@ -31,7 +31,7 @@ export function hashPayload(payload) {}/**- * Specifies required and optional fields for constructing a [`Request`](../api/request).+ * Specifies required and optional fields for constructing a {@link Request#setValue}.** @typedef {Object} RequestOptions* @property {String} url URL of the web page to crawl. It must be a non-empty string.diff --git a/src/request_list.js b/src/request_list.jsindex 98cf9600a..98b34bee7 100644--- a/src/request_list.js+++ b/src/request_list.js@@ -649,23 +649,23 @@ export class RequestList {* If `null`, the list will not be persisted and will only be stored in memory. Process restart* will then cause the list to be crawled again from the beginning. We suggest always using a name.* @param {Array<Request|RequestOptions|string>} sources- * An array of sources of URLs for the `RequestList`.- * It can be either an array of plain objects that- * define the `url` property, or an array of instances of the {@link Request} class.+ * An array of sources of URLs for the {@link RequestList}. It can be either an array of plain objects+ * that define at least the `url` property, or an array of instances of the {@link Request} class.** Additionally, the `requestsFromUrl` property may be used instead of `url`,- * which will instruct `RequestList` to download the source URLs from a given remote location.+ * which will instruct {@link RequestList} to download the source URLs from a given remote location.* The URLs will be parsed from the received response. In this case you can limit the URLs- * using `regex` parameter containing regular expression pattern for URLs to be included.+ * using `regex` parameter containing regular expression pattern for URLs to be included.*- * For details, see the [`RequestList`](requestlist#new_RequestList_new)- * constructor options.+ * For details, see the {@link RequestListOptions.sources}* @param {RequestListOptions} [options]- * The [`new RequestList`](requestlist#new_RequestList_new) options. Note that the listName parameter supersedes- * the `persistStateKey` and `persistSourcesKey` options and the sources parameter supersedes the `sources` option.+ * The {@link RequestList} options. Note that the `listName` parameter supersedes+ * the {@link RequestListOptions.persistStateKey} and {@link RequestListOptions.persistSourcesKey}+ * options and the `sources` parameter supersedes the {@link RequestListOptions.sources} option.* @returns {Promise<RequestList>}* @memberof module:Apify* @name openRequestList+ * @function*/export const openRequestList = async (listName, sources, options = {}) => {checkParamOrThrow(listName, 'listName', 'String | Null');@@ -688,7 +688,7 @@ export const openRequestList = async (listName, sources, options = {}) => {/*** Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed.- * You can obtain the state by calling [`RequestList.getState()`](../api/requestlist#getState()) and receive an object with+ * You can obtain the state by calling {@link RequestList#getState} and receive an object with* the following structure:** ```diff --git a/src/request_queue.js b/src/request_queue.jsindex 0a088e933..d50f2b022 100644--- a/src/request_queue.js+++ b/src/request_queue.js@@ -111,14 +111,14 @@ const getRequestId = (uniqueKey) => {/*** A helper class that is used to report results from various- * [`RequestQueue`](../api/requestqueue) functions as well as- * [`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).+ * {@link RequestQueue} functions as well as+ * {@link utils#enqueueLinks}.** @typedef {Object} QueueOperationInfo* @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.* @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.* @property {String} requestId The ID of the added request- * @property {Object} request The original [`Request`](../api/request) object passed to the `RequestQueue` function.+ * @property {Object} request The original {@link Request} object passed to the `RequestQueue` function.*//**@@ -133,7 +133,7 @@ const getRequestId = (uniqueKey) => {* corresponding {@link Request} objects will need to have different `uniqueKey` properties.** Do not instantiate this class directly, use the- * [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function instead.+ * {@link Apify#openRequestQueue} function instead.** `RequestQueue` is used by {@link BasicCrawler}, {@link CheerioCrawler}* and {@link PuppeteerCrawler} as a source of URLs to crawl.@@ -154,9 +154,9 @@ const getRequestId = (uniqueKey) => {* and `{NUMBER}` is an integer indicating the position of the request in the queue.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://docs.apify.com/storage/request-queue" target="_blank">Apify Request Queue</a>+ * [Apify Request Queue](https://docs.apify.com/storage/request-queue)* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`- * option to [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function,+ * option to {@link Apify#openRequestQueue} function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.** **Example usage:**@@ -239,7 +239,7 @@ export class RequestQueue {* {@link QueueOperationInfo} object.** To add multiple requests to the queue by extracting links from a webpage,- * see the [`Apify.utils.enqueueLinks()`](utils#utils.enqueueLinks) helper function.+ * see the {@link utils#enqueueLinks} helper function.** @param {Request|RequestOptions} request {@link Request} object or vanilla object with request data.* Note that the function sets the `uniqueKey` and `id` fields to the passed object.@@ -314,15 +314,15 @@ export class RequestQueue {* Returns a next request in the queue to be processed, or `null` if there are no more pending requests.** Once you successfully finish processing of the request, you need to call- * [`requestQueue.markRequestHandled()`](#RequestQueue+markRequestHandled)+ * {@link RequestQueue#markRequestHandled}* to mark the request as handled in the queue. If there was some error in processing the request,- * call [`requestQueue.reclaimRequest()`](#RequestQueue+reclaimRequest) instead,+ * call {@link RequestQueue#reclaimRequest} instead,* so that the queue will give the request to some other consumer in another call to the `fetchNextRequest` function.** Note that the `null` return value doesn't mean the queue processing finished,* it means there are currently no pending requests.* To check whether all requests in queue were finished,- * use [`requestQueue.isFinished()`](#RequestQueue+isFinished) instead.+ * use {@link RequestQueue#isFinished} instead.** @returns {Promise<Request>}* Returns the request object or `null` if there are no more pending requests.@@ -386,7 +386,7 @@ export class RequestQueue {/*** Marks a request that was previously returned by the- * [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)+ * {@link RequestQueue#fetchNextRequest}* function as handled after successful processing.* Handled requests will never again be returned by the `fetchNextRequest` function.*@@ -425,7 +425,7 @@ export class RequestQueue {/*** Reclaims a failed request back to the queue, so that it can be returned for processed later again- * by another call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).+ * by another call to {@link RequestQueue#fetchNextRequest}.* The request record in the queue is updated using the provided `request` parameter.* For example, this lets you store the number of retries or error messages for the request.*@@ -433,7 +433,7 @@ export class RequestQueue {* @param {Object} [options]* @param {Boolean} [options.forefront=false]* If `true` then the request it placed to the beginning of the queue, so that it's returned- * in the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).+ * in the next call to {@link RequestQueue#fetchNextRequest}.* By default, it's put to the end of the queue.* @return {Promise<QueueOperationInfo>}*/@@ -476,10 +476,10 @@ export class RequestQueue {}/**- * Resolves to `true` if the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)+ * Resolves to `true` if the next call to {@link RequestQueue#fetchNextRequest}* would return `null`, otherwise it resolves to `false`.* Note that even if the queue is empty, there might be some pending requests currently being processed.- * If you need to ensure that there is no activity in the queue, use [`requestQueue.isFinished()`](#RequestQueue+isFinished).+ * If you need to ensure that there is no activity in the queue, use {@link RequestQueue#isFinished}.** @returns {Promise<Boolean>}*/@@ -1080,6 +1080,7 @@ const getOrCreateQueue = async (queueIdOrName) => {* @returns {Promise<RequestQueue>}* @memberof module:Apify* @name openRequestQueue+ * @function*/export const openRequestQueue = (queueIdOrName, options = {}) => {checkParamOrThrow(queueIdOrName, 'queueIdOrName', 'Maybe String');diff --git a/src/session_pool/session_pool.js b/src/session_pool/session_pool.jsindex b17a92395..6f2deaf47 100644--- a/src/session_pool/session_pool.js+++ b/src/session_pool/session_pool.js@@ -29,10 +29,11 @@ import { ACTOR_EVENT_NAMES_EX } from '../constants';* Handles the sessions rotation, creation and persistence.* Creates a pool of {@link Session} instances, that are randomly rotated.* When some session is marked as blocked. It is removed and new one is created instead.- * Learn more in the [`Session management guide`](../guides/sessionmanagement).+ * Learn more in the [`Session management guide`](/docs/guides/session-management).** Session pool is by default persisted in default {@link KeyValueStore}.- * If you want to have one pool for all runs you have to specify `persistStateKeyValueStoreId`.+ * If you want to have one pool for all runs you have to specify+ * {@link SessionPoolOptions.persistStateKeyValueStoreId}.** **Example usage:***@@ -319,10 +320,11 @@ export class SessionPool extends EventEmitter {** For more details and code examples, see the {@link SessionPool} class.*- * @param {SessionPoolOptions} sessionPoolOptions The [`new SessionPool`](sessionpool#new_SessionPool_new) options+ * @param {SessionPoolOptions} sessionPoolOptions* @return {Promise<SessionPool>}* @memberof module:Apify* @name openSessionPool+ * @function*/export const openSessionPool = async (sessionPoolOptions) => {const sessionPool = new SessionPool(sessionPoolOptions);diff --git a/src/typedefs.js b/src/typedefs.jsindex fa53336d5..7f26b3d46 100644--- a/src/typedefs.js+++ b/src/typedefs.js@@ -7,13 +7,13 @@/*** Represents information about an actor run, as returned by the- * [`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.+ * {@link Apify#call} or {@link Apify#callTask} function.* The object is almost equivalent to the JSON response* of the- * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>+ * [Actor run](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor)* Apify API endpoint and extended with certain fields.* For more details, see- * <a href="https://docs.apify.com/actor/run" target="_blank">Runs.</a>+ * [Runs.](https://docs.apify.com/actor/run)** @typedef {Object} ActorRun* @property {String} id@@ -26,7 +26,7 @@* Time when the actor run finished. Contains `null` for running actors.* @property {String} status* Status of the run. For possible values, see- * <a href="https://docs.apify.com/actor/run#lifecycle" target="_blank">Run lifecycle</a>+ * [Run lifecycle](https://docs.apify.com/actor/run#lifecycle)* in Apify actor documentation.* @property {Object} meta* Actor run meta-data. For example:@@ -59,26 +59,26 @@* ```* @property {String} buildId* ID of the actor build used for the run. For details, see- * <a href="https://docs.apify.com/actor/build" target="_blank">Builds</a>+ * [Builds](https://docs.apify.com/actor/build)* in Apify actor documentation.* @property {String} buildNumber* Number of the actor build used for the run. For example, `0.0.10`.* @property {Number} exitCode* Exit code of the actor run process. It's `null` if actor is still running.* @property {String} defaultKeyValueStoreId- * ID of the default key-value store associated with the actor run. See [`KeyValueStore`](../api/keyvaluestore) for details.+ * ID of the default key-value store associated with the actor run. See {@link KeyValueStore} for details.* @property {String} defaultDatasetId- * ID of the default dataset associated with the actor run. See [`Dataset`](../api/dataset) for details.+ * ID of the default dataset associated with the actor run. See {@link Dataset} for details.* @property {String} defaultRequestQueueId- * ID of the default request queue associated with the actor run. See [`RequestQueue`](../api/requestqueue) for details.+ * ID of the default request queue associated with the actor run. See {@link RequestQueue} for details.* @property {String} containerUrl* URL on which the web server running inside actor run's Docker container can be accessed.* For more details, see- * <a href="https://docs.apify.com/actor/run#container-web-server" target="_blank">Container web server</a>+ * [Container web server](https://docs.apify.com/actor/run#container-web-server)* in Apify actor documentation.* @property {Object} output* Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,- * or if you pass `false` to the `fetchOutput` option of [`Apify.call()`](../api/apify#module_Apify.call).+ * or if you pass `false` to the `fetchOutput` option of {@link Apify#call}.** For example:* ```diff --git a/src/utils.js b/src/utils.jsindex d32b67b0d..02a1dac87 100644--- a/src/utils.js+++ b/src/utils.js@@ -84,7 +84,7 @@ export const logSystemInfo = () => {* `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN` environment variables.** The instance is used for all underlying calls to the Apify API in functions such as- * [`Apify.getValue()`](#module_Apify.getValue) or [`Apify.call()`](#module_Apify.call).+ * {@link Apify#getValue} or {@link Apify#call}.* The settings of the client can be globally altered by calling the* <a href="https://docs.apify.com/api/apify-client-js/latest#ApifyClient-setOptions"* target="_blank">`Apify.client.setOptions()`</a> function.@@ -389,6 +389,7 @@ export const isAtHome = () => !!process.env[ENV_VARS.IS_AT_HOME];* @param {Number} millis Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.* @memberof utils* @name sleep+ * @function* @return {Promise<void>}*/export const sleep = (millis) => {@@ -531,6 +532,7 @@ const BLOCK_TAGS_REGEX = /^(p|h1|h2|h3|h4|h5|h6|ol|ul|li|pre|address|blockquote|* [cheerio](https://www.npmjs.com/package/cheerio) function.* @return {String} Plain text* @memberOf utils+ * @function*/const htmlToText = (html) => {if (!html) return '';@@ -540,8 +542,10 @@ const htmlToText = (html) => {// Also, it seems this doesn't work well in CheerioScraper, e.g. htmlToText($)// produces really text with a lot of HTML elements in it. Let's just deprecate this sort of usage,// and make the parameter "htmlOrCheerioElement"-- /** @type {Cheerio} */+ /**+ * @type {Cheerio}+ * @ignore+ */const $ = typeof html === 'function' ? html : cheerio.load(html, { decodeEntities: true });let text = '';@@ -586,8 +590,8 @@ const htmlToText = (html) => {** @param {Request|RequestOptions} request [Apify.Request](https://sdk.apify.com/docs/api/request) object.* @param {IncomingMessage|PuppeteerResponse} [response]- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&version=v1.11.0&show=api-class-response" target="_blank"><code>Response</code></a>- * or NodeJS <a href="https://nodejs.org/api/http.html#http_class_http_serverresponse" target="_blank"><code>http.ServerResponse</code></a>.+ * Puppeteer [`Response`](https://pptr.dev/#?product=Puppeteer&version=v1.11.0&show=api-class-response)+ * or NodeJS [`http.IncomingMessage`](https://nodejs.org/api/http.html#http_class_http_serverresponse).* @param {Object} [additionalFields] Object containing additional fields to be added.* @return {Object}diff --git a/src/utils_request.js b/src/utils_request.jsindex e856f0916..bb0a12ec6 100644--- a/src/utils_request.js+++ b/src/utils_request.js@@ -80,6 +80,7 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {* however, if returned from the cache it will be a [response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.* @memberOf utils* @name requestAsBrowser+ * @function*/export const requestAsBrowser = async (options) => {const {diff --git a/src/utils_social.js b/src/utils_social.jsindex 3cb1b175c..18d168ec8 100644--- a/src/utils_social.js+++ b/src/utils_social.js@@ -525,7 +525,7 @@ try {* The function removes duplicates from the resulting arrays and sorts the items alphabetically.** Note that the `phones` field contains phone numbers extracted from the special phone links- * such as `<a href="tel:+1234556789">call us</a>` (see [`social.phonesFromUrls()`](#social.phonesFromUrls)])+ * such as `[call us](tel:+1234556789)` (see {@link social#phonesFromUrls})* and potentially other sources with high certainty, while `phonesUncertain` contains phone numbers* extracted from the plain text, which might be very inaccurate.*diff --git a/tools/build_readme.js b/tools/build_readme.jsindex a81c8fcad..f366434e1 100644--- a/tools/build_readme.js+++ b/tools/build_readme.js@@ -9,7 +9,7 @@ console.log('Reading documentation files.');const introduction = readFileSync(path.join(docPath, 'readme', 'introduction.md'), 'utf8');const motivation = readFileSync(path.join(docPath, 'guides', 'motivation.md'), 'utf8');const overview = readFileSync(path.join(docPath, 'readme', 'overview.md'), 'utf8');-const gettingStarted = readFileSync(path.join(docPath, 'guides', 'getting_started_quick.md'), 'utf8');+const gettingStarted = readFileSync(path.join(docPath, 'guides', 'quick_start.md'), 'utf8');const support = readFileSync(path.join(docPath, 'readme', 'support.md'), 'utf8');const headerRx = /---.*?title: (.+)\n.*?---/s;@@ -21,7 +21,7 @@ const fixHeaders = doc => doc.replace(hashRx, hashReplace).replace(headerRx, heaconst linkRx = /\[(.*)]\((.*)\)/g;const fixLinks = (match, p1, p2) => {- const url = new URL(p2, 'https://sdk.apify.com/docs/guides/');+ const url = new URL(p2, 'https://sdk.apify.com/docs/');return `[${p1}](${url})`;};diff --git a/types/autoscaling/autoscaled_pool.d.ts b/types/autoscaling/autoscaled_pool.d.tsindex d565ca4ca..87cc43943 100644--- a/types/autoscaling/autoscaled_pool.d.ts+++ b/types/autoscaling/autoscaled_pool.d.ts@@ -17,7 +17,7 @@ export type AutoscaledPoolOptions = {* If it resolves to `true` then the pool's run finishes. Being called only* when there are no tasks being processed means that as long as `isTaskReadyFunction()`* keeps resolving to `true`, `isFinishedFunction()` will never be called.- * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.+ * To abort a run, use the {@link AutoscaledPool#abort} method.*/isFinishedFunction: Function;/**@@ -96,7 +96,7 @@ export type AutoscaledPoolOptions = {* If it resolves to `true` then the pool's run finishes. Being called only* when there are no tasks being processed means that as long as `isTaskReadyFunction()`* keeps resolving to `true`, `isFinishedFunction()` will never be called.- * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.+ * To abort a run, use the {@link AutoscaledPool#abort} method.** @property {Number} [minConcurrency=1]* The minimum number of tasks running in parallel.@@ -146,16 +146,16 @@ export type AutoscaledPoolOptions = {* Meaningful data gathered from these snapshots is provided to `AutoscaledPool` by the {@link SystemStatus} class.** Before running the pool, you need to implement the following three functions:- * [`runTaskFunction()`](#new_AutoscaledPool_new),- * [`isTaskReadyFunction()`](#new_AutoscaledPool_new) and- * [`isFinishedFunction()`](#new_AutoscaledPool_new).+ * {@link AutoscaledPoolOptions#runTaskFunction},+ * {@link AutoscaledPoolOptions#isTaskReadyFunction} and+ * {@link AutoscaledPoolOptions#isFinishedFunction}.*- * The auto-scaled pool is started by calling the [`run()`](#AutoscaledPool+run) function.- * The pool periodically queries the [`isTaskReadyFunction()`](#new_AutoscaledPool_new) function+ * The auto-scaled pool is started by calling the {@link AutoscaledPool#run} function.+ * The pool periodically queries the {@link AutoscaledPoolOptions#isTaskReadyFunction} function* for more tasks, managing optimal concurrency, until the function resolves to `false`. The pool then queries- * the [`isFinishedFunction()`](#new_AutoscaledPool_new). If it resolves to `true`, the run finishes after all running tasks complete.+ * the {@link AutoscaledPoolOptions#isFinishedFunction}. If it resolves to `true`, the run finishes after all running tasks complete.* If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for tasks.- * If any of the tasks throws then the [`run()`](#AutoscaledPool+run) function rejects the promise with an error.+ * If any of the tasks throws then the {@link AutoscaledPool#run} function rejects the promise with an error.** The pool evaluates whether it should start a new task every time one of the tasks finishes* and also in the interval set by the `options.maybeRunIntervalSecs` parameter.@@ -292,7 +292,7 @@ declare class AutoscaledPool {maybeRunInterval: any;/*** Aborts the run of the auto-scaled pool and destroys it. The promise returned from- * the [`run()`](#AutoscaledPool+run) function will immediately resolve, no more new tasks+ * the {@link AutoscaledPool#run} function will immediately resolve, no more new tasks* will be spawned and all running tasks will be left in their current state.** Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion@@ -306,13 +306,13 @@ declare class AutoscaledPool {abort(): Promise<any>;/*** Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish- * (unlike abort, which terminates them). Used together with [`resume()`](#AutoscaledPool+resume)+ * (unlike abort, which terminates them). Used together with {@link AutoscaledPool#resume}** The function's promise will resolve once all running tasks have completed and the pool* is effectively idle. If the `timeoutSecs` argument is provided, the promise will reject* with a timeout error after the `timeoutSecs` seconds.*- * The promise returned from the [`run()`](#AutoscaledPool+run) function will not resolve+ * The promise returned from the {@link AutoscaledPool#run} function will not resolve* when `.pause()` is invoked (unlike abort, which resolves it).** @param {number} [timeoutSecs]@@ -321,7 +321,7 @@ declare class AutoscaledPool {pause(timeoutSecs?: number): Promise<any>;/*** Resumes the operation of the autoscaled-pool by allowing more tasks to be run.- * Used together with [`pause()`](#AutoscaledPool+pause)+ * Used together with {@link AutoscaledPool#pause}** Tasks will automatically start running again in `options.maybeRunIntervalSecs`.*/diff --git a/types/autoscaling/system_status.d.ts b/types/autoscaling/system_status.d.tsindex 1782e0c58..c7c99bc28 100644--- a/types/autoscaling/system_status.d.ts+++ b/types/autoscaling/system_status.d.ts@@ -17,7 +17,7 @@ export type SystemInfo = {export type SystemStatusOptions = {/*** Defines max age of snapshots used in the- * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.+ * {@link SystemStatus#getCurrentStatus} measurement.*/currentHistorySecs?: number;/**@@ -60,7 +60,7 @@ export type SystemStatusOptions = {* @typedef {Object} SystemStatusOptions* @property {Number} [currentHistorySecs=5]* Defines max age of snapshots used in the- * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.+ * {@link SystemStatus#getCurrentStatus} measurement.* @property {Number} [maxMemoryOverloadedRatio=0.2]* Sets the maximum ratio of overloaded snapshots in a memory sample.* If the sample exceeds this ratio, the system will be overloaded.@@ -78,21 +78,21 @@ export type SystemStatusOptions = {*//*** Provides a simple interface to reading system status from a {@link Snapshotter} instance.- * It only exposes two functions [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)- * and [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus).+ * It only exposes two functions {@link SystemStatus#getCurrentStatus}+ * and {@link SystemStatus#getHistoricalStatus}.* The system status is calculated using a weighted average of overloaded* messages in the snapshots, with the weights being the time intervals* between the snapshots. Each resource is calculated separately* and the system is overloaded whenever at least one resource is overloaded.* The class is used by the {@link AutoscaledPool} class.*- * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)+ * {@link SystemStatus#getCurrentStatus}* returns a boolean that represents the current status of the system.* The length of the current timeframe in seconds is configurable* by the `currentHistorySecs` option and represents the max age* of snapshots to be considered for the calculation.*- * [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus)+ * {@link SystemStatus#getHistoricalStatus}* returns a boolean that represents the long-term status* of the system. It considers the full snapshot history available* in the {@link Snapshotter} instance.diff --git a/types/crawlers/basic_crawler.d.ts b/types/crawlers/basic_crawler.d.tsindex 2fd2a42c9..d13f8fe38 100644--- a/types/crawlers/basic_crawler.d.ts+++ b/types/crawlers/basic_crawler.d.ts@@ -21,7 +21,7 @@ export type BasicCrawlerOptions = {* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.*/handleRequestFunction: HandleRequest;/**@@ -52,12 +52,12 @@ export type BasicCrawlerOptions = {* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>+ * [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)* for the default implementation of this function.*/handleFailedRequestFunction?: HandleFailedRequest;/**- * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.+ * Indicates how many times the request is retried if {@link BasicCrawlerOptions.handleRequestFunction} fails.*/maxRequestRetries?: number;/**@@ -100,6 +100,13 @@ export type HandleRequestInputs = {* The original {Request} object.*/request: Request;+ /**+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.+ */autoscaledPool: AutoscaledPool;session?: Session;};@@ -137,7 +144,7 @@ export type HandleFailedRequestInput = {* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.* @property {RequestList} [requestList]* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).@@ -160,10 +167,10 @@ export type HandleFailedRequestInput = {* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>+ * [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)* for the default implementation of this function.* @property {Number} [maxRequestRetries=3]- * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.+ * Indicates how many times the request is retried if {@link BasicCrawlerOptions.handleRequestFunction} fails.* @property {Number} [maxRequestsPerCrawl]* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.* Always set this value in order to prevent infinite loops in misconfigured crawlers.@@ -195,13 +202,13 @@ export type HandleFailedRequestInput = {* If you want a crawler that already facilitates this functionality,* please consider using {@link PuppeteerCrawler} or {@link CheerioCrawler}.*- * `BasicCrawler` invokes the user-provided [`handleRequestFunction()`](#new_BasicCrawler_new)+ * `BasicCrawler` invokes the user-provided {@link BasicCrawlerOptions.handleRequestFunction}* for each {@link Request} object, which represents a single URL to crawl.* The {@link Request} objects are fed from the {@link RequestList} or the {@link RequestQueue}- * instances provided by the [`requestList`](#new_BasicCrawler_new) or [`requestQueue`](#new_BasicCrawler_new)+ * instances provided by the {@link BasicCrawlerOptions.requestList} or {@link BasicCrawlerOptions.requestQueue}* constructor options, respectively.*- * If both [`requestList`](#new_BasicCrawler_new) and [`requestQueue`](#new_BasicCrawler_new) options are used,+ * If both {@link BasicCrawlerOptions.requestList} and {@link BasicCrawlerOptions.requestQueue} options are used,* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.*@@ -242,17 +249,11 @@ export type HandleFailedRequestInput = {** await crawler.run();* ```- *- * @property {AutoscaledPool} autoscaledPool- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.- * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.- * You can use it to change the concurrency settings on the fly,- * to pause the crawler by calling {@link AutoscaledPool#pause}- * or to abort it by calling {@link AutoscaledPool#abort}.*/declare class BasicCrawler {/*** @param {BasicCrawlerOptions} options+ * All `BasicCrawler` parameters are passed via an options object.*/constructor(options: BasicCrawlerOptions);requestList: any;diff --git a/types/crawlers/cheerio_crawler.d.ts b/types/crawlers/cheerio_crawler.d.tsindex e864367bc..d36da7942 100644--- a/types/crawlers/cheerio_crawler.d.ts+++ b/types/crawlers/cheerio_crawler.d.ts@@ -5,7 +5,7 @@ export type CheerioCrawlerOptions = {* loaded and parsed by the crawler.** The function receives the following object as an argument:- * ```javascript+ * ```* {* // The Cheerio object's function with the parsed HTML.* $: Cheerio,@@ -33,12 +33,13 @@ export type CheerioCrawlerOptions = {* session: Session* }* ```+ ** Type of `body` depends on the `Content-Type` header of the web page:* - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types* - Buffer for others MIME content types** Parsed `Content-Type` header using- * <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>+ * [content-type package](https://www.npmjs.com/package/content-type)* is stored in `contentType`.** Cheerio is available only for HTML and XML content types.@@ -54,7 +55,7 @@ export type CheerioCrawlerOptions = {* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.*/handlePageFunction: CheerioHandlePage;/**@@ -122,13 +123,13 @@ export type CheerioCrawlerOptions = {ignoreSslErrors?: boolean;/*** If set to `true`, `CheerioCrawler` will be configured to use- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>+ * [Apify Proxy](https://my.apify.com/proxy) for all connections.+ * For more information, see the [documentation](https://docs.apify.com/proxy)*/useApifyProxy?: boolean;/*** An array of proxy groups to be used- * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.+ * by the [Apify Proxy](https://docs.apify.com/proxy).* Only applied if the `useApifyProxy` option is `true`.*/apifyProxyGroups?: string[];@@ -160,7 +161,7 @@ export type CheerioCrawlerOptions = {* where the {@link Request} instance corresponds to the failed request, and the `Error` instance* represents the last error thrown during processing of the request.*- * See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>+ * See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13)* for the default implementation of this function.*/handleFailedRequestFunction?: HandleFailedRequest;@@ -225,7 +226,7 @@ export type PrepareRequestInputs = {export type PrepareRequest = (inputs: PrepareRequestInputs) => void | Promise<void>;export type CheerioHandlePageInputs = {/**- * The <a href="https://cheerio.js.org/">Cheerio</a> object with parsed HTML.+ * The [Cheerio](https://cheerio.js.org/) object with parsed HTML.*/$?: CheerioStatic;/**@@ -251,6 +252,13 @@ export type CheerioHandlePageInputs = {* An instance of Node's http.IncomingMessage object,*/response: IncomingMessage;+ /**+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.+ */autoscaledPool: AutoscaledPool;session?: any;};@@ -262,7 +270,7 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* loaded and parsed by the crawler.** The function receives the following object as an argument:- * ```javascript+ * ```* {* // The Cheerio object's function with the parsed HTML.* $: Cheerio,@@ -290,12 +298,13 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* session: Session* }* ```+ ** Type of `body` depends on the `Content-Type` header of the web page:* - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types* - Buffer for others MIME content types** Parsed `Content-Type` header using- * <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>+ * [content-type package](https://www.npmjs.com/package/content-type)* is stored in `contentType`.** Cheerio is available only for HTML and XML content types.@@ -311,7 +320,7 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.* @property {RequestList} [requestList]* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).@@ -364,11 +373,11 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* If set to true, SSL certificate errors will be ignored.* @property {Boolean} [useApifyProxy=false]* If set to `true`, `CheerioCrawler` will be configured to use- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>+ * [Apify Proxy](https://my.apify.com/proxy) for all connections.+ * For more information, see the [documentation](https://docs.apify.com/proxy)* @property {String[]} [apifyProxyGroups]* An array of proxy groups to be used- * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.+ * by the [Apify Proxy](https://docs.apify.com/proxy).* Only applied if the `useApifyProxy` option is `true`.* @property {String} [apifyProxySession]* Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.@@ -394,7 +403,7 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* where the {@link Request} instance corresponds to the failed request, and the `Error` instance* represents the last error thrown during processing of the request.*- * See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>+ * See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13)* for the default implementation of this function.* @property {String[]} [additionalMimeTypes]* An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"@@ -432,7 +441,7 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi*//*** Provides a framework for the parallel crawling of web pages using plain HTTP requests and- * <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.+ * [cheerio](https://www.npmjs.com/package/cheerio) HTML parser.* The URLs to crawl are fed either from a static list of URLs* or from a dynamic queue of URLs enabling recursive crawling of websites.*@@ -442,29 +451,29 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi* because it loads the pages using full-featured headless Chrome browser.** `CheerioCrawler` downloads each URL using a plain HTTP request,- * parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>- * and then invokes the user-provided [`handlePageFunction()`](#new_CheerioCrawler_new) to extract page data- * using a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface to the parsed HTML DOM.+ * parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio)+ * and then invokes the user-provided {@link CheerioCrawlerOptions.handlePageFunction} to extract page data+ * using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.** The source URLs are represented using {@link Request} objects that are fed from- * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_CheerioCrawler_new)- * or [`requestQueue`](#new_CheerioCrawler_new) constructor options, respectively.+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link CheerioCrawlerOptions.requestList}+ * or {@link CheerioCrawlerOptions.requestQueue} constructor options, respectively.*- * If both [`requestList`](#new_CheerioCrawler_new) and [`requestQueue`](#new_CheerioCrawler_new) are used,+ * If both {@link CheerioCrawlerOptions.requestList} and {@link CheerioCrawlerOptions.requestQueue} are used,* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.** The crawler finishes when there are no more {@link Request} objects to crawl.*- * `CheerioCrawler` downloads the web pages using the {@link requestAsBrowser} utility function.+ * `CheerioCrawler` downloads the web pages using the {@link utils#requestAsBrowser} utility function.* You can use the `requestOptions` parameter to pass additional options to this function.** By default, `CheerioCrawler` only processes web pages with the `text/html`* and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),* and skips pages with other content types. If you want the crawler to process other content types,- * use the [`additionalMimeTypes`](#new_CheerioCrawler_new) constructor option.+ * use the {@link CheerioCrawlerOptions.additionalMimeTypes} constructor option.* Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.- * For details, see {@link CheerioCrawlerOptions#handlePageFunction}.+ * For details, see {@link CheerioCrawlerOptions.handlePageFunction}.** New requests are only dispatched when there is enough free CPU and memory available,* using the functionality provided by the {@link AutoscaledPool} class.@@ -506,16 +515,11 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi** await crawler.run();* ```- * @property {AutoscaledPool} autoscaledPool- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.- * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.- * You can use it to change the concurrency settings on the fly,- * to pause the crawler by calling {@link AutoscaledPool#pause}- * or to abort it by calling {@link AutoscaledPool#abort}.*/declare class CheerioCrawler {/*** @param {CheerioCrawlerOptions} options+ * All `CheerioCrawler` parameters are passed via an options object.*/constructor(options?: CheerioCrawlerOptions);supportedMimeTypes: Set<string>;diff --git a/types/crawlers/puppeteer_crawler.d.ts b/types/crawlers/puppeteer_crawler.d.tsindex cefaafd7e..b29196959 100644--- a/types/crawlers/puppeteer_crawler.d.ts+++ b/types/crawlers/puppeteer_crawler.d.ts@@ -17,9 +17,9 @@ export type PuppeteerCrawlerOptions = {** `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.* `page` is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)* `response` is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),* which is the main resource response as returned by `page.goto(request.url)`.* `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.*@@ -32,7 +32,7 @@ export type PuppeteerCrawlerOptions = {* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.*/handlePageFunction: PuppeteerHandlePage;/**@@ -51,17 +51,17 @@ export type PuppeteerCrawlerOptions = {handlePageTimeoutSecs?: number;/*** Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,- * i.e. a `Promise` resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.+ * [page.goto()](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options) function,+ * i.e. a `Promise` resolving to the [Response](https://pptr.dev/#?product=Puppeteer&show=api-class-response) object.** This is useful if you need to extend the page load timeout or select different criteria* to determine that the navigation succeeded.** Note that a single page object is only used to process a single request and it is closed afterwards.*- * By default, the function invokes [`Apify.utils.puppeteer.gotoExtended()`](puppeteer#puppeteer.gotoExtended) with a timeout of 60 seconds.+ * By default, the function invokes {@link puppeteer#gotoExtended} with a timeout of 60 seconds.* For details, see source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292).*/gotoFunction?: PuppeteerGoto;/**@@ -83,7 +83,7 @@ export type PuppeteerCrawlerOptions = {* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>+ * [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301)* for the default implementation of this function.*/handleFailedRequestFunction?: HandleFailedRequest;@@ -106,13 +106,13 @@ export type PuppeteerCrawlerOptions = {* Overrides the default function to launch a new Puppeteer instance.* Shortcut to the corresponding {@link PuppeteerPool} option.* See source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)* for default behavior.*/launchPuppeteerFunction?: Function;/**- * Options used by [`Apify.launchPuppeteer()`](apify#module_Apify.launchPuppeteer) to start new Puppeteer instances.- * Shortcut to the corresponding {@link PuppeteerPool} option. See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.+ * Shortcut to the corresponding {@link PuppeteerPool} option.*/launchPuppeteerOptions?: LaunchPuppeteerOptions;/**@@ -122,19 +122,22 @@ export type PuppeteerCrawlerOptions = {*/autoscaledPoolOptions?: AutoscaledPoolOptions;/**- * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the+ * corresponding {@link AutoscaledPoolOptions.minConcurrency} option.*- * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.- * If you're not sure, just keep the default value and the concurrency will scale up automatically.+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU,+ * your crawler will run extremely slow or crash. If you're not sure, just keep the default value+ * and the concurrency will scale up automatically.*/minConcurrency?: number;/**- * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the+ * corresponding {@link AutoscaledPoolOptions.maxConcurrency} option.*/maxConcurrency?: number;/**- * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.- * It also marks Session as bad after a request timeout.+ * If set to true Crawler will automatically use Session Pool. It will automatically retire+ * sessions on 403, 401 and 429 status codes. It also marks Session as bad after a request timeout.*/useSessionPool?: boolean;/**@@ -152,28 +155,35 @@ export type PuppeteerHandlePageInputs = {*/request: Request;/**- * An instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ * An instance of the Puppeteer+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),* which is the main resource response as returned by `page.goto(request.url)`.*/response: PuppeteerResponse;/**- * is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * is an instance of the Puppeteer+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)*/page: PuppeteerPage;/*** An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.*/puppeteerPool: PuppeteerPool;+ /**+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.+ */autoscaledPool: AutoscaledPool;session?: Session;};export type PuppeteerHandlePage = (inputs: PuppeteerHandlePageInputs) => Promise<void>;export type PuppeteerGotoInputs = {/**- * is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * is an instance of the Puppeteer+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)*/page: PuppeteerPage;/**@@ -214,9 +224,9 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse** `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.* `page` is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)* `response` is an instance of the `Puppeteer`- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),* which is the main resource response as returned by `page.goto(request.url)`.* `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.*@@ -229,7 +239,7 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse* To make this work, you should **always*** let your function throw exceptions rather than catch them.* The exceptions are logged to the request using the- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.+ * {@link Request#pushErrorMessage} function.* @property {RequestList} [requestList]* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).@@ -240,17 +250,17 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse* Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.* @property {PuppeteerGoto} [gotoFunction]* Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,- * i.e. a `Promise` resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.+ * [page.goto()](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options) function,+ * i.e. a `Promise` resolving to the [Response](https://pptr.dev/#?product=Puppeteer&show=api-class-response) object.** This is useful if you need to extend the page load timeout or select different criteria* to determine that the navigation succeeded.** Note that a single page object is only used to process a single request and it is closed afterwards.*- * By default, the function invokes [`Apify.utils.puppeteer.gotoExtended()`](puppeteer#puppeteer.gotoExtended) with a timeout of 60 seconds.+ * By default, the function invokes {@link puppeteer#gotoExtended} with a timeout of 60 seconds.* For details, see source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292).* @property {Number} [gotoTimeoutSecs=60]* Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default* function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.@@ -268,7 +278,7 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse* represents the last error thrown during processing of the request.** See- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>+ * [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301)* for the default implementation of this function.* @property {Number} [maxRequestRetries=3]* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.@@ -283,33 +293,36 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse* Overrides the default function to launch a new Puppeteer instance.* Shortcut to the corresponding {@link PuppeteerPool} option.* See source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)* for default behavior.* @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]- * Options used by [`Apify.launchPuppeteer()`](apify#module_Apify.launchPuppeteer) to start new Puppeteer instances.- * Shortcut to the corresponding {@link PuppeteerPool} option. See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.+ * Shortcut to the corresponding {@link PuppeteerPool} option.* @property {AutoscaledPoolOptions} [autoscaledPoolOptions]* Custom options passed to the underlying {@link AutoscaledPool} instance constructor.* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options* are provided by `PuppeteerCrawler` and should not be overridden.* @property {Number} [minConcurrency=1]- * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the+ * corresponding {@link AutoscaledPoolOptions.minConcurrency} option.*- * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.- * If you're not sure, just keep the default value and the concurrency will scale up automatically.+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU,+ * your crawler will run extremely slow or crash. If you're not sure, just keep the default value+ * and the concurrency will scale up automatically.* @property {Number} [maxConcurrency=1000]- * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the+ * corresponding {@link AutoscaledPoolOptions.maxConcurrency} option.* @property {Boolean} [useSessionPool=false]- * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.- * It also marks Session as bad after a request timeout.+ * If set to true Crawler will automatically use Session Pool. It will automatically retire+ * sessions on 403, 401 and 429 status codes. It also marks Session as bad after a request timeout.* @property {SessionPoolOptions} [sessionPoolOptions]* Custom options passed to the underlying {@link SessionPool} constructor.- * @property {Boolean} [persistCookiesPerSession]+ * @property {Boolean} [persistCookiesPerSession=false]* Automatically saves cookies to Session. Works only if Session Pool is used.*//*** Provides a simple framework for parallel crawling of web pages- * using headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.+ * using headless Chrome with [Puppeteer](https://github.com/GoogleChrome/puppeteer).* The URLs to crawl are fed either from a static list of URLs* or from a dynamic queue of URLs enabling recursive crawling of websites.*@@ -319,28 +332,27 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse* which downloads the pages using raw HTTP requests and is about 10x faster.** The source URLs are represented using {@link Request} objects that are fed from- * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_PuppeteerCrawler_new)- * or [`requestQueue`](#new_PuppeteerCrawler_new) constructor options, respectively.+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link PuppeteerCrawlerOptions.requestList}+ * or {@link PuppeteerCrawlerOptions.requestQueue} constructor options, respectively.*- * If both [`requestList`](#new_PuppeteerCrawler_new) and [`requestQueue`](#new_PuppeteerCrawler_new) are used,+ * If both {@link PuppeteerCrawlerOptions.requestList} and {@link PuppeteerCrawlerOptions.requestQueue} are used,* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.** The crawler finishes when there are no more {@link Request} objects to crawl.** `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl- * and then calls the function provided by user as the [`handlePageFunction()`](#new_PuppeteerCrawler_new) option.+ * and then calls the function provided by user as the {@link PuppeteerCrawlerOptions.handlePageFunction} option.** New pages are only opened when there is enough free CPU and memory available,* using the functionality provided by the {@link AutoscaledPool} class.- * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`+ * All {@link AutoscaledPool} configuration options can be passed to the {@link PuppeteerCrawlerOptions.autoscaledPoolOptions}* parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`- * {@link AutoscaledPool} options are available directly in the `PuppeteerCrawler` constructor.+ * {@link AutoscaledPoolOptions} are available directly in the `PuppeteerCrawler` constructor.*- * Note that the pool of Puppeteer instances is internally managed by- * the {@link PuppeteerPool} class. Many constructor options- * such as `maxOpenPagesPerInstance` or `launchPuppeteerFunction` are passed directly- * to {@link PuppeteerPool} constructor.+ * Note that the pool of Puppeteer instances is internally managed by the {@link PuppeteerPool} class.+ * Many constructor options such as {@link PuppeteerPoolOptions.maxOpenPagesPerInstance} or+ * {@link PuppeteerPoolOptions.launchPuppeteerFunction} are passed directly to the {@link PuppeteerPool} constructor.** **Example usage:***@@ -369,17 +381,11 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse** await crawler.run();* ```- * @property {AutoscaledPool} autoscaledPool- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.- * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.- * You can use it to change the concurrency settings on the fly,- * to pause the crawler by calling {@link AutoscaledPool#pause}- * or to abort it by calling {@link AutoscaledPool#abort}.*/declare class PuppeteerCrawler {/**- * @param {PuppeteerCrawlerOptions} options All `PuppeteerCrawler` parameters are passed- * via an options object with the following keys:+ * @param {PuppeteerCrawlerOptions} options+ * All `PuppeteerCrawler` parameters are passed via an options object.*/constructor(options: PuppeteerCrawlerOptions);handlePageFunction: PuppeteerHandlePage;diff --git a/types/dataset.d.ts b/types/dataset.d.tsindex 3942834ac..b69bc321f 100644--- a/types/dataset.d.ts+++ b/types/dataset.d.ts@@ -12,7 +12,7 @@ export function chunkBySize(items: any[], limitBytes: number): any[];* Typically it is used to store crawling results.** Do not instantiate this class directly, use the- * [`Apify.openDataset()`](apify#module_Apify.openDataset) function instead.+ * {@link Apify#openDataset} function instead.** `Dataset` stores its data either on local disk or in the Apify cloud,* depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.@@ -27,9 +27,9 @@ export function chunkBySize(items: any[], limitBytes: number): any[];* Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://docs.apify.com/storage/dataset" target="_blank">Apify Dataset</a>+ * [Apify Dataset](https://docs.apify.com/storage/dataset)* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`- * option to [`Apify.openDataset()`](apify#module_Apify.openDataset) function,+ * option to {@link Apify#openDataset} function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.** **Example usage:**diff --git a/types/enqueue_links/click_elements.d.ts b/types/enqueue_links/click_elements.d.tsindex 260fe2d63..f4addf78d 100644--- a/types/enqueue_links/click_elements.d.ts+++ b/types/enqueue_links/click_elements.d.ts@@ -5,7 +5,7 @@* requests, including their methods, headers and payloads are then enqueued to a provided* {@link RequestQueue}. This is useful to crawl JavaScript heavy pages where links are not available* in `href` elements, but rather navigations are triggered in click handlers.- * If you're looking to find URLs in `href` attributes of the page, see [`enqueueLinks()`](utils#utils.enqueueLinks).+ * If you're looking to find URLs in `href` attributes of the page, see {@link utils#enqueueLinks}.** Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects* and override settings of the enqueued {@link Request} objects.@@ -47,11 +47,11 @@* All `enqueueLinksByClickingElements()` parameters are passed* via an options object with the following keys:* @param {Page} options.page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {RequestQueue} options.requestQueue* A request queue to which the URLs will be enqueued.* @param {String} options.selector- * A CSS selector matching elements to be clicked on. Unlike in [`enqueueLinks()`](utils#utils.enqueueLinks), there is no default+ * A CSS selector matching elements to be clicked on. Unlike in {@link utils#enqueueLinks}, there is no default* value. This is to prevent suboptimal use of this function by using it too broadly.* @param {Array<String|RegExp|Object>} [options.pseudoUrls]* An array of {@link PseudoUrl}s matching the URLs to be enqueued,@@ -104,6 +104,7 @@* Promise that resolves to an array of {@link QueueOperationInfo} objects.* @memberOf puppeteer* @name enqueueLinksByClickingElements+ * @function*/export function enqueueLinksByClickingElements(options?: {page: any;diff --git a/types/enqueue_links/enqueue_links.d.ts b/types/enqueue_links/enqueue_links.d.tsindex ec0e57876..9edfa7b50 100644--- a/types/enqueue_links/enqueue_links.d.ts+++ b/types/enqueue_links/enqueue_links.d.ts@@ -4,7 +4,7 @@* and enqueues the URLs in their `href` attributes to the provided {@link RequestQueue}.* If you're looking to find URLs in JavaScript heavy pages where links are not available* in `href` elements, but rather navigations are triggered in click handlers- * see [`enqueueLinksByClickingElements()`](puppeteer#puppeteer.enqueueLinksByClickingElements).+ * see {@link puppeteer#enqueueLinksByClickingElements}.** Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects* and override settings of the enqueued {@link Request} objects.@@ -33,10 +33,10 @@* All `enqueueLinks()` parameters are passed* via an options object with the following keys:* @param {Page} options.page- * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* Either `page` or `$` option must be provided.* @param {Cheerio} options.$- * <a href="https://github.com/cheeriojs/cheerio" target="_blank"><code>Cheerio</code></a> object with loaded HTML.+ * [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML.* Either `page` or `$` option must be provided.* @param {RequestQueue} options.requestQueue* A request queue to which the URLs will be enqueued.@@ -78,6 +78,7 @@* Promise that resolves to an array of {@link QueueOperationInfo} objects.* @memberOf utils* @name enqueueLinks+ * @function*/export function enqueueLinks(options?: {page: Page;diff --git a/types/enqueue_links/shared.d.ts b/types/enqueue_links/shared.d.tsindex 947501231..ad7dfad6f 100644--- a/types/enqueue_links/shared.d.ts+++ b/types/enqueue_links/shared.d.ts@@ -28,7 +28,7 @@ export function createRequestOptions(sources: any[] | string[], userData?: any):export function addRequestsToQueueInBatches(requests: Request[], requestQueue: any, batchSize?: number): Promise<any[]>;/*** Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used- * [`Apify.utils.enqueueLinks`](../api/utils#utils.enqueueLinks) to modify requests before enqueuing them.+ * {@link utils#enqueueLinks} to modify requests before enqueuing them.*/export type RequestTransform = (original: any) => any;import PseudoUrl from "../pseudo_url";diff --git a/types/errors.d.ts b/types/errors.d.tsindex 09814a25c..4610947e8 100644--- a/types/errors.d.ts+++ b/types/errors.d.ts@@ -1,7 +1,7 @@export const APIFY_CALL_ERROR_NAME: "ApifyCallError";/*** The class represents exceptions thrown- * by the [`Apify.call()`](../api/apify#module_Apify.call) function.+ * by the {@link Apify#call} function.** @typedef {Object} ApifyCallError* @property {String} messagediff --git a/types/key_value_store.d.ts b/types/key_value_store.d.tsindex 92ac8c45b..b0c6832c7 100644--- a/types/key_value_store.d.ts+++ b/types/key_value_store.d.ts@@ -7,20 +7,20 @@ export function maybeStringify(value: any, options: any): any;* for saving screenshots, actor inputs and outputs, web pages, PDFs or to persist the state of crawlers.** Do not instantiate this class directly, use the- * [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function instead.+ * {@link Apify#openKeyValueStore} function instead.** Each actor run is associated with a default key-value store, which is created exclusively* for the run. By convention, the actor input and output are stored into the* default key-value store under the `INPUT` and `OUTPUT` key, respectively.* Typically, input and output are JSON files, although it can be any other format.* To access the default key-value store directly, you can use the- * [`Apify.getValue()`](apify#module_Apify.getValue)- * and [`Apify.setValue()`](apify#module_Apify.setValue) convenience functions.+ * {@link Apify#getValue} and {@link Apify#setValue} convenience functions.*- * To access the input, you can also use the [`Apify.getInput()`](apify#module_Apify.getInput) convenience function.+ * To access the input, you can also use the {@link Apify#getInput} convenience function.** `KeyValueStore` stores its data either on local disk or in the Apify cloud,- * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.+ * depending on whether the [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR)+ * or [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variables are set.** If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in* the local directory in the following files:@@ -31,11 +31,12 @@ export function maybeStringify(value: any, options: any): any;* unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.* The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.*- * If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://docs.apify.com/storage/key-value-store" target="_blank">Apify Key-value store</a>+ * If the [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable is set but+ * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) not,+ * the data is stored in the [Apify Key-value store](https://docs.apify.com/storage/key-value-store)* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`- * option to [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function,- * even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.+ * option to {@link Apify#openKeyValueStore} function, even if the+ * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) variable is set.** **Example usage:***@@ -80,7 +81,7 @@ export class KeyValueStore {* content type are automatically parsed and returned as a JavaScript object.* Similarly, records with `text/plain` content types are returned as a string.* For all other content types, the value is returned as a raw- * <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a> instance.+ * [](https://nodejs.org/api/buffer.html) instance.** If the record does not exist, the function resolves to `null`.*@@ -90,7 +91,7 @@ export class KeyValueStore {* **Example usage:**** ```javascript- * const store = await Apify.openKeyValueStore('my-screenshots');+ * const store = await Apify.openKeyValueStore();* const buffer = await store.getValue('screenshot1.png');* ```*@@ -99,7 +100,7 @@ export class KeyValueStore {* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`* @returns {Promise<Object|String|Buffer>}* Returns a promise that resolves to an object, string- * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, depending+ * or [](https://nodejs.org/api/buffer.html), depending* on the MIME content type of the record.*/getValue(key: string): Promise<any>;@@ -110,8 +111,8 @@ export class KeyValueStore {* **Example usage:**** ```javascript- * const store = await Apify.openKeyValueStore('my-store');- * await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });+ * const store = await Apify.openKeyValueStore();+ * await store.setValue('OUTPUT', { foo: 'bar' });* ```** Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`@@ -120,11 +121,11 @@ export class KeyValueStore {* `application/json; charset=utf-8` MIME content type.* To store the value with another content type, pass it in the options as follows:* ```javascript- * const store = await Apify.openKeyValueStore('my-store');+ * const store = await Apify.openKeyValueStore('my-text-store');* await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });* ```* If you set custom content type, `value` must be either a string or- * <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, otherwise an error will be thrown.+ * [](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.** If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds* regardless whether the record existed or not.@@ -144,7 +145,7 @@ export class KeyValueStore {* <li>If `null`, the record in the key-value store is deleted.</li>* <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>* <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`- * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>.</li>+ * or [](https://nodejs.org/api/buffer.html).</li>* </ul>* For any other value an error will be thrown.* @param {Object} [options]@@ -272,6 +273,6 @@ export function setValue(key: string, value: any, options?: {}): Promise<any>;export function getInput(): Promise<any>;/**- * User-function used in the [`KeyValueStore.forEachKey()`](../api/keyvaluestore#forEachKey) method.+ * User-function used in the {@link KeyValueStore#forEachKey} method.*/export type KeyConsumer = (key: string, index: number, info: any, size: number) => any;diff --git a/types/pseudo_url.d.ts b/types/pseudo_url.d.tsindex a90fbd104..ef29f477c 100644--- a/types/pseudo_url.d.ts+++ b/types/pseudo_url.d.ts@@ -2,7 +2,7 @@ export default PseudoUrl;/*** Represents a pseudo-URL (PURL) - an URL pattern used by web crawlers* to specify which URLs should the crawler visit.- * This class is used by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.+ * This class is used by the {@link utils#enqueueLinks} function.** A PURL is simply a URL with special directives enclosed in `[]` brackets.* Currently, the only supported directive is `[RegExp]`,@@ -57,7 +57,7 @@ declare class PseudoUrl {* such as making the matching case sensitive.* @param {RequestOptions} requestTemplate* Options for the new {@link Request} instances created for matching URLs- * by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.+ * by the {@link utils#enqueueLinks} function.*/constructor(purl: string | RegExp, requestTemplate?: any);regex: RegExp;diff --git a/types/puppeteer.d.ts b/types/puppeteer.d.tsindex 060a2d4cb..f7ff7f528 100644--- a/types/puppeteer.d.ts+++ b/types/puppeteer.d.ts@@ -1,9 +1,9 @@export function launchPuppeteer(options?: LaunchPuppeteerOptions): Promise<any>;/*** Apify extends the launch options of Puppeteer.- * You can use any of the- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>- * options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)+ * You can use any of the Puppeteer compatible+ * [`LaunchOptions`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions)+ * options in the {@link Apify#launchPuppeteer}* function and in addition, all the options available below.*/export type LaunchPuppeteerOptions = {@@ -31,13 +31,13 @@ export type LaunchPuppeteerOptions = {useChrome?: boolean;/*** If set to `true`, Puppeteer will be configured to use- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>+ * [Apify Proxy](https://my.apify.com/proxy) for all connections.+ * For more information, see the [documentation](https://docs.apify.com/proxy)*/useApifyProxy?: boolean;/*** An array of proxy groups to be used- * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.+ * by the [Apify Proxy](https://docs.apify.com/proxy).* Only applied if the `useApifyProxy` option is `true`.*/apifyProxyGroups?: string[];diff --git a/types/puppeteer_pool.d.ts b/types/puppeteer_pool.d.tsindex 6029b2971..a59106b15 100644--- a/types/puppeteer_pool.d.ts+++ b/types/puppeteer_pool.d.ts@@ -39,13 +39,12 @@ export type PuppeteerPoolOptions = {* The function must return a promise resolving to* [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.* See the source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)* for the default implementation.*/launchPuppeteerFunction?: Function;/**- * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.- * See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.*/launchPuppeteerOptions?: LaunchPuppeteerOptions;/**@@ -103,11 +102,10 @@ export type PuppeteerPoolOptions = {* The function must return a promise resolving to* [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.* See the source code on- * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>+ * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)* for the default implementation.* @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]- * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.- * See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.* @property {boolean} [recycleDiskCache=false]* Enables recycling of disk cache directories by Chrome instances.* When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.@@ -131,7 +129,7 @@ export type PuppeteerPoolOptions = {*//*** Manages a pool of Chrome browser instances controlled using- * <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.+ * [Puppeteer](https://github.com/GoogleChrome/puppeteer).** `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.* This is useful in order to facilitate rotation of proxies, cookies@@ -170,7 +168,7 @@ declare class PuppeteerPool {/*** @param {PuppeteerPoolOptions} [options]* All `PuppeteerPool` parameters are passed- * via an options object with the following keys:+ * via an options object.*/constructor(options?: PuppeteerPoolOptions);sessionPool: any;@@ -254,7 +252,7 @@ declare class PuppeteerPool {/*** Opens new tab in one of the browsers in the pool and returns a `Promise`* that resolves to an instance of a Puppeteer- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>.+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page).** @return {Promise<Page>}* @ignore@@ -289,7 +287,7 @@ declare class PuppeteerPool {_findInstanceByBrowser(browser: Browser): Promise<any>;/*** Manually retires a Puppeteer- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-browser" target="_blank"><code>Browser</code></a>+ * [`Browser`](https://pptr.dev/#?product=Puppeteer&show=api-class-browser)* instance from the pool. The browser will continue to process open pages so that they may gracefully finish.* This is unlike `browser.close()` which will forcibly terminate the browser and all open pages will be closed.* @param {Browser} browserdiff --git a/types/request.d.ts b/types/request.d.tsindex 3ea5ba439..39956d8da 100644--- a/types/request.d.ts+++ b/types/request.d.ts@@ -8,7 +8,7 @@ export function computeUniqueKey({ url, method, payload, keepUrlFragment, useExtexport function hashPayload(payload: any): string;export default Request;/**- * Specifies required and optional fields for constructing a [`Request`](../api/request).+ * Specifies required and optional fields for constructing a {@link Request#setValue}.*/export type RequestOptions = {/**@@ -68,7 +68,7 @@ export type RequestOptions = {useExtendedUniqueKey?: boolean;};/**- * Specifies required and optional fields for constructing a [`Request`](../api/request).+ * Specifies required and optional fields for constructing a {@link Request#setValue}.** @typedef {Object} RequestOptions* @property {String} url URL of the web page to crawl. It must be a non-empty string.diff --git a/types/request_list.d.ts b/types/request_list.d.tsindex 1b9d56803..9c8a5792e 100644--- a/types/request_list.d.ts+++ b/types/request_list.d.ts@@ -387,7 +387,7 @@ export type RequestListOptions = {};/*** Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed.- * You can obtain the state by calling [`RequestList.getState()`](../api/requestlist#getState()) and receive an object with+ * You can obtain the state by calling {@link RequestList#getState} and receive an object with* the following structure:** ```diff --git a/types/request_queue.d.ts b/types/request_queue.d.tsindex 1a35b7967..109383bb8 100644--- a/types/request_queue.d.ts+++ b/types/request_queue.d.ts@@ -6,14 +6,14 @@ export const MAX_QUERIES_FOR_CONSISTENCY: 6;export const STORAGE_CONSISTENCY_DELAY_MILLIS: 3000;/*** A helper class that is used to report results from various- * [`RequestQueue`](../api/requestqueue) functions as well as- * [`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).+ * {@link RequestQueue} functions as well as+ * {@link utils#enqueueLinks}.** @typedef {Object} QueueOperationInfo* @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.* @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.* @property {String} requestId The ID of the added request- * @property {Object} request The original [`Request`](../api/request) object passed to the `RequestQueue` function.+ * @property {Object} request The original {@link Request} object passed to the `RequestQueue` function.*//*** Represents a queue of URLs to crawl, which is used for deep crawling of websites@@ -27,7 +27,7 @@ export const STORAGE_CONSISTENCY_DELAY_MILLIS: 3000;* corresponding {@link Request} objects will need to have different `uniqueKey` properties.** Do not instantiate this class directly, use the- * [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function instead.+ * {@link Apify#openRequestQueue} function instead.** `RequestQueue` is used by {@link BasicCrawler}, {@link CheerioCrawler}* and {@link PuppeteerCrawler} as a source of URLs to crawl.@@ -48,9 +48,9 @@ export const STORAGE_CONSISTENCY_DELAY_MILLIS: 3000;* and `{NUMBER}` is an integer indicating the position of the request in the queue.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://docs.apify.com/storage/request-queue" target="_blank">Apify Request Queue</a>+ * [Apify Request Queue](https://docs.apify.com/storage/request-queue)* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`- * option to [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function,+ * option to {@link Apify#openRequestQueue} function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.** **Example usage:**@@ -104,7 +104,7 @@ export class RequestQueue {* {@link QueueOperationInfo} object.** To add multiple requests to the queue by extracting links from a webpage,- * see the [`Apify.utils.enqueueLinks()`](utils#utils.enqueueLinks) helper function.+ * see the {@link utils#enqueueLinks} helper function.** @param {Request|RequestOptions} request {@link Request} object or vanilla object with request data.* Note that the function sets the `uniqueKey` and `id` fields to the passed object.@@ -126,15 +126,15 @@ export class RequestQueue {* Returns a next request in the queue to be processed, or `null` if there are no more pending requests.** Once you successfully finish processing of the request, you need to call- * [`requestQueue.markRequestHandled()`](#RequestQueue+markRequestHandled)+ * {@link RequestQueue#markRequestHandled}* to mark the request as handled in the queue. If there was some error in processing the request,- * call [`requestQueue.reclaimRequest()`](#RequestQueue+reclaimRequest) instead,+ * call {@link RequestQueue#reclaimRequest} instead,* so that the queue will give the request to some other consumer in another call to the `fetchNextRequest` function.** Note that the `null` return value doesn't mean the queue processing finished,* it means there are currently no pending requests.* To check whether all requests in queue were finished,- * use [`requestQueue.isFinished()`](#RequestQueue+isFinished) instead.+ * use {@link RequestQueue#isFinished} instead.** @returns {Promise<Request>}* Returns the request object or `null` if there are no more pending requests.@@ -142,7 +142,7 @@ export class RequestQueue {fetchNextRequest(): Promise<Request>;/*** Marks a request that was previously returned by the- * [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)+ * {@link RequestQueue#fetchNextRequest}* function as handled after successful processing.* Handled requests will never again be returned by the `fetchNextRequest` function.*@@ -152,7 +152,7 @@ export class RequestQueue {markRequestHandled(request: Request): Promise<QueueOperationInfo>;/*** Reclaims a failed request back to the queue, so that it can be returned for processed later again- * by another call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).+ * by another call to {@link RequestQueue#fetchNextRequest}.* The request record in the queue is updated using the provided `request` parameter.* For example, this lets you store the number of retries or error messages for the request.*@@ -160,7 +160,7 @@ export class RequestQueue {* @param {Object} [options]* @param {Boolean} [options.forefront=false]* If `true` then the request it placed to the beginning of the queue, so that it's returned- * in the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).+ * in the next call to {@link RequestQueue#fetchNextRequest}.* By default, it's put to the end of the queue.* @return {Promise<QueueOperationInfo>}*/@@ -168,10 +168,10 @@ export class RequestQueue {forefront?: boolean;}): Promise<QueueOperationInfo>;/**- * Resolves to `true` if the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)+ * Resolves to `true` if the next call to {@link RequestQueue#fetchNextRequest}* would return `null`, otherwise it resolves to `false`.* Note that even if the queue is empty, there might be some pending requests currently being processed.- * If you need to ensure that there is no activity in the queue, use [`requestQueue.isFinished()`](#RequestQueue+isFinished).+ * If you need to ensure that there is no activity in the queue, use {@link RequestQueue#isFinished}.** @returns {Promise<Boolean>}*/@@ -330,8 +330,8 @@ export function openRequestQueue(queueIdOrName?: string, options?: {}): Promise<RequestQueue>;/*** A helper class that is used to report results from various- * [`RequestQueue`](../api/requestqueue) functions as well as- * [`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).+ * {@link RequestQueue} functions as well as+ * {@link utils#enqueueLinks}.*/export type QueueOperationInfo = {/**@@ -347,7 +347,7 @@ export type QueueOperationInfo = {*/requestId: string;/**- * The original [`Request`](../api/request) object passed to the `RequestQueue` function.+ * The original {@link Request} object passed to the `RequestQueue` function.*/request: any;};diff --git a/types/session_pool/session_pool.d.ts b/types/session_pool/session_pool.d.tsindex 61ba68f54..74b837d2d 100644--- a/types/session_pool/session_pool.d.ts+++ b/types/session_pool/session_pool.d.ts@@ -17,10 +17,11 @@* Handles the sessions rotation, creation and persistence.* Creates a pool of {@link Session} instances, that are randomly rotated.* When some session is marked as blocked. It is removed and new one is created instead.- * Learn more in the [`Session management guide`](../guides/sessionmanagement).+ * Learn more in the [`Session management guide`](/docs/guides/session-management).** Session pool is by default persisted in default {@link KeyValueStore}.- * If you want to have one pool for all runs you have to specify `persistStateKeyValueStoreId`.+ * If you want to have one pool for all runs you have to specify+ * {@link SessionPoolOptions.persistStateKeyValueStoreId}.** **Example usage:***diff --git a/types/typedefs.d.ts b/types/typedefs.d.tsindex cb26ac6a7..b4f4b8884 100644--- a/types/typedefs.d.ts+++ b/types/typedefs.d.ts@@ -5,13 +5,13 @@export type Cheerio = CheerioStatic;/*** Represents information about an actor run, as returned by the- * [`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.+ * {@link Apify#call} or {@link Apify#callTask} function.* The object is almost equivalent to the JSON response* of the- * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>+ * [Actor run](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor)* Apify API endpoint and extended with certain fields.* For more details, see- * <a href="https://docs.apify.com/actor/run" target="_blank">Runs.</a>+ * [Runs.](https://docs.apify.com/actor/run)*/export type ActorRun = {/**@@ -32,7 +32,7 @@ export type ActorRun = {finishedAt: Date;/*** Status of the run. For possible values, see- * <a href="https://docs.apify.com/actor/run#lifecycle" target="_blank">Run lifecycle</a>+ * [Run lifecycle](https://docs.apify.com/actor/run#lifecycle)* in Apify actor documentation.*/status: string;@@ -73,7 +73,7 @@ export type ActorRun = {options: any;/*** ID of the actor build used for the run. For details, see- * <a href="https://docs.apify.com/actor/build" target="_blank">Builds</a>+ * [Builds](https://docs.apify.com/actor/build)* in Apify actor documentation.*/buildId: string;@@ -86,27 +86,27 @@ export type ActorRun = {*/exitCode: number;/**- * ID of the default key-value store associated with the actor run. See [`KeyValueStore`](../api/keyvaluestore) for details.+ * ID of the default key-value store associated with the actor run. See {@link KeyValueStore} for details.*/defaultKeyValueStoreId: string;/**- * ID of the default dataset associated with the actor run. See [`Dataset`](../api/dataset) for details.+ * ID of the default dataset associated with the actor run. See {@link Dataset} for details.*/defaultDatasetId: string;/**- * ID of the default request queue associated with the actor run. See [`RequestQueue`](../api/requestqueue) for details.+ * ID of the default request queue associated with the actor run. See {@link RequestQueue} for details.*/defaultRequestQueueId: string;/*** URL on which the web server running inside actor run's Docker container can be accessed.* For more details, see- * <a href="https://docs.apify.com/actor/run#container-web-server" target="_blank">Container web server</a>+ * [Container web server](https://docs.apify.com/actor/run#container-web-server)* in Apify actor documentation.*/containerUrl: string;/*** Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,- * or if you pass `false` to the `fetchOutput` option of [`Apify.call()`](../api/apify#module_Apify.call).+ * or if you pass `false` to the `fetchOutput` option of {@link Apify#call}.** For example:* ```diff --git a/types/utils.d.ts b/types/utils.d.tsindex c2eb47fd8..5a304a667 100644--- a/types/utils.d.ts+++ b/types/utils.d.ts@@ -8,7 +8,7 @@ export function logSystemInfo(): void;* `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN` environment variables.** The instance is used for all underlying calls to the Apify API in functions such as- * [`Apify.getValue()`](#module_Apify.getValue) or [`Apify.call()`](#module_Apify.call).+ * {@link Apify#getValue} or {@link Apify#call}.* The settings of the client can be globally altered by calling the* <a href="https://docs.apify.com/api/apify-client-js/latest#ApifyClient-setOptions"* target="_blank">`Apify.client.setOptions()`</a> function.diff --git a/types/utils_request.d.ts b/types/utils_request.d.tsindex f23231525..f2bc3e54a 100644--- a/types/utils_request.d.ts+++ b/types/utils_request.d.ts@@ -2,7 +2,7 @@ export const FIREFOX_MOBILE_USER_AGENT: "Mozilla/5.0 (Android; Mobile; rv:14.0)export const FIREFOX_DESKTOP_USER_AGENT: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0";export function requestAsBrowser(options: RequestAsBrowserOptions): Promise<any>;/**- * (@apify/http-request)[https://www.npmjs.com/package/@apify/http-request] NPM package.+ * [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.*/export type RequestAsBrowserOptions = {/**@@ -38,7 +38,7 @@ export type RequestAsBrowserOptions = {/*** Function accepts `response` object as a single parameter and should return true or false.* If function returns true request gets aborted. This function is passed to the- * (+ * [*/abortFunction?: Function;};diff --git a/website/i18n/en.json b/website/i18n/en.jsonindex 4114c250f..6374433bd 100644--- a/website/i18n/en.json+++ b/website/i18n/en.json@@ -8,52 +8,52 @@"api/apify": {"title": "Apify"},- "api/autoscaledpool": {+ "api/autoscaled-pool": {"title": "AutoscaledPool"},- "api/basiccrawler": {+ "api/basic-crawler": {"title": "BasicCrawler"},- "api/cheeriocrawler": {+ "api/cheerio-crawler": {"title": "CheerioCrawler"},"api/dataset": {"title": "Dataset"},- "api/keyvaluestore": {+ "api/key-value-store": {"title": "KeyValueStore"},- "api/liveviewserver": {+ "api/live-view-server": {"title": "LiveViewServer"},"api/log": {"title": "utils.log"},- "api/pseudourl": {+ "api/pseudo-url": {"title": "PseudoUrl"},"api/puppeteer": {"title": "utils.puppeteer"},- "api/puppeteercrawler": {+ "api/puppeteer-crawler": {"title": "PuppeteerCrawler"},- "api/puppeteerpool": {+ "api/puppeteer-pool": {"title": "PuppeteerPool"},"api/request": {"title": "Request"},- "api/requestlist": {+ "api/request-list": {"title": "RequestList"},- "api/requestqueue": {+ "api/request-queue": {"title": "RequestQueue"},"api/session": {"title": "Session"},- "api/sessionpool": {+ "api/session-pool": {"title": "SessionPool"},"api/snapshotter": {@@ -62,67 +62,67 @@"api/social": {"title": "utils.social"},- "api/systemstatus": {+ "api/system-status": {"title": "SystemStatus"},"api/utils": {"title": "utils"},- "examples/basiccrawler": {+ "examples/basic-crawler": {"title": "Basic Crawler"},- "examples/callactor": {+ "examples/call-actor": {"title": "Call Actor"},- "examples/cheeriocrawler": {+ "examples/cheerio-crawler": {"title": "Cheerio Crawler"},"examples/forms": {"title": "Forms"},- "examples/helloworld": {+ "examples/hello-world": {"title": "Hello World"},- "examples/puppeteercrawler": {+ "examples/puppeteer-crawler": {"title": "Puppeteer Crawler"},- "examples/puppeteersitemap": {+ "examples/puppeteer-sitemap": {"title": "Puppeteer Sitemap"},- "examples/puppeteerwithproxy": {+ "examples/puppeteer-with-proxy": {"title": "Puppeteer With Proxy"},"examples/screenshots": {"title": "Screenshots"},- "examples/synchronousrun": {+ "examples/synchronous-run": {"title": "Synchronous Run"},- "guides/datastorage": {+ "guides/data-storage": {"title": "Data Storage"},- "guides/environmentvariables": {+ "guides/environment-variables": {"title": "Environment Variables"},- "guides/quickstart": {- "title": "Quick Start"- },- "guides/gettingstarted": {+ "guides/getting-started": {"title": "Getting Started"},"guides/motivation": {"title": "Motivation"},- "guides/puppeteerliveview": {+ "guides/puppeteer-live-view": {"title": "Puppeteer Live View"},- "guides/sessionmanagement": {+ "guides/quick-start": {+ "title": "Quick Start"+ },+ "guides/session-management": {"title": "Session Management"},- "guides/typescriptactor": {- "title": "Typescript Actors"+ "guides/type-script-actor": {+ "title": "TypeScript Actors"},- "guides/whatisanactor": {+ "guides/what-is-an-actor": {"title": "What is an Actor"},"readme/introduction": {@@ -134,155 +134,147 @@"readme/support": {"title": "readme/support"},- "typedefs/actorrun": {+ "typedefs/actor-run": {"title": "ActorRun"},- "typedefs/apifycallerror": {+ "typedefs/apify-call-error": {"title": "ApifyCallError"},- "typedefs/apifyenv": {+ "typedefs/apify-env": {"title": "ApifyEnv"},- "typedefs/autoscaledpooloptions": {+ "typedefs/autoscaled-pool-options": {"title": "AutoscaledPoolOptions"},- "typedefs/basiccrawleroptions": {+ "typedefs/basic-crawler-options": {"title": "BasicCrawlerOptions"},- "typedefs/cheeriocrawleroptions": {+ "typedefs/cheerio-crawler-options": {"title": "CheerioCrawlerOptions"},- "typedefs/cheeriohandlepage": {+ "typedefs/cheerio-handle-page": {"title": "CheerioHandlePage"},- "typedefs/cheeriohandlepageinputs": {+ "typedefs/cheerio-handle-page-inputs": {"title": "CheerioHandlePageInputs"},- "typedefs/createsession": {+ "typedefs/create-session": {"title": "CreateSession"},- "typedefs/datasetconsumer": {+ "typedefs/dataset-consumer": {"title": "DatasetConsumer"},- "typedefs/datasetcontent": {+ "typedefs/dataset-content": {"title": "DatasetContent"},- "typedefs/datasetmapper": {+ "typedefs/dataset-mapper": {"title": "DatasetMapper"},- "typedefs/datasetreducer": {+ "typedefs/dataset-reducer": {"title": "DatasetReducer"},- "typedefs/handlefailedrequest": {+ "typedefs/handle-failed-request": {"title": "HandleFailedRequest"},- "typedefs/handlefailedrequestinput": {+ "typedefs/handle-failed-request-input": {"title": "HandleFailedRequestInput"},- "typedefs/handlerequest": {+ "typedefs/handle-request": {"title": "HandleRequest"},- "typedefs/handlerequestinputs": {+ "typedefs/handle-request-inputs": {"title": "HandleRequestInputs"},- "typedefs/keyconsumer": {+ "typedefs/key-consumer": {"title": "KeyConsumer"},- "typedefs/launchpuppeteer": {+ "typedefs/launch-puppeteer": {"title": "LaunchPuppeteer"},- "typedefs/launchpuppeteeroptions": {+ "typedefs/launch-puppeteer-options": {"title": "LaunchPuppeteerOptions"},- "typedefs/memoryinfo": {+ "typedefs/memory-info": {"title": "MemoryInfo"},- "typedefs/preparerequest": {+ "typedefs/prepare-request": {"title": "PrepareRequest"},- "typedefs/preparerequestinputs": {+ "typedefs/prepare-request-inputs": {"title": "PrepareRequestInputs"},- "typedefs/puppeteercrawleroptions": {+ "typedefs/puppeteer-crawler-options": {"title": "PuppeteerCrawlerOptions"},- "typedefs/puppeteergoto": {+ "typedefs/puppeteer-goto": {"title": "PuppeteerGoto"},- "typedefs/puppeteergotoinputs": {+ "typedefs/puppeteer-goto-inputs": {"title": "PuppeteerGotoInputs"},- "typedefs/puppeteerhandlepage": {+ "typedefs/puppeteer-handle-page": {"title": "PuppeteerHandlePage"},- "typedefs/puppeteerhandlepageinputs": {+ "typedefs/puppeteer-handle-page-inputs": {"title": "PuppeteerHandlePageInputs"},- "typedefs/puppeteerpooloptions": {+ "typedefs/puppeteer-pool-options": {"title": "PuppeteerPoolOptions"},- "typedefs/queueoperationinfo": {+ "typedefs/queue-operation-info": {"title": "QueueOperationInfo"},- "typedefs/requestasbrowseroptions": {+ "typedefs/request-as-browser-options": {"title": "RequestAsBrowserOptions"},- "typedefs/requestlistoptions": {+ "typedefs/request-list-options": {"title": "RequestListOptions"},- "typedefs/requestliststate": {+ "typedefs/request-list-state": {"title": "RequestListState"},- "typedefs/requestoptions": {+ "typedefs/request-options": {"title": "RequestOptions"},- "typedefs/requesttransform": {+ "typedefs/request-transform": {"title": "RequestTransform"},- "typedefs/sessionoptions": {+ "typedefs/session-options": {"title": "SessionOptions"},- "typedefs/sessionpooloptions": {+ "typedefs/session-pool-options": {"title": "SessionPoolOptions"},- "typedefs/sessionstate": {+ "typedefs/session-state": {"title": "SessionState"},- "typedefs/snapshotteroptions": {+ "typedefs/snapshotter-options": {"title": "SnapshotterOptions"},- "typedefs/socialhandles": {+ "typedefs/social-handles": {"title": "SocialHandles"},- "typedefs/stealthoptions": {+ "typedefs/stealth-options": {"title": "StealthOptions"},- "typedefs/systeminfo": {+ "typedefs/system-info": {"title": "SystemInfo"},- "typedefs/systemstatusoptions": {+ "typedefs/system-status-options": {"title": "SystemStatusOptions"}},"links": {"Guide": "Guide","Examples": "Examples",- "Reference": "Reference",+ "API Reference": "API Reference","GitHub": "GitHub"},"categories": {"Guide": "Guide",- "Reference": "Reference",- "Examples": "Examples",- "Autoscaling Tools": "Autoscaling Tools",+ "API Reference": "API Reference","Type Definitions": "Type Definitions",- "Constructor Options": "Constructor Options",- "Crawler User-Functions": "Crawler User-Functions",- "Puppeteer User-Functions": "Puppeteer User-Functions",- "Dataset User-Functions": "Dataset User-Functions",- "KeyValueStore User-Functions": "KeyValueStore User-Functions",- "SessionPool User-Functions": "SessionPool User-Functions",- "Utility User-Functions": "Utility User-Functions"+ "Examples": "Examples"}},"pages-strings": {diff --git a/website/package.json b/website/package.jsonindex d2e9f8fd6..dc7e51582 100644--- a/website/package.json+++ b/website/package.json@@ -11,8 +11,13 @@"prettify": "prettier --write --config ./tools/prettier.config.js ../docs/guides/*.md"},"devDependencies": {+ "camelcase": "^5.3.1",+ "decamelize": "^4.0.0","docusaurus": "^1.13.0","jsdoc-to-markdown": "^5.0.1","prettier": "^1.18.2"+ },+ "dependencies": {+ "remarkable-extlink": "^1.1.0"}}diff --git a/website/pages/en/index.js b/website/pages/en/index.jsindex aa7c2f0fe..19fe9c67e 100755--- a/website/pages/en/index.js+++ b/website/pages/en/index.js@@ -82,9 +82,9 @@ class HomeSplash extends React.Component {<div className="inner"><ProjectTitle/><PromoSection>- <Button href={docUrl('api/apify', language)}>Read the Docs</Button>- <Button href={docUrl('examples/basiccrawler', language)}>See Examples</Button><Button href="#try">Try It Out</Button>+ <Button href={docUrl('guides/getting-started', language)}>Learn the Basics</Button>+ <Button href={docUrl('examples/puppeteer-crawler', language)}>See Examples</Button></PromoSection></div><a@@ -198,7 +198,7 @@ const TryOut = () => (<Block id="try" background="light" gridBlockAlign="left">{[{- content: 'Install **Apify SDK** into a Node.js project. You must have Node.js 8 or higher installed.\n' ++ content: 'Install **Apify SDK** into a Node.js project. You must have Node.js 10 or higher installed.\n' +'```\n' +'npm install apify\n' +'```\n' +diff --git a/website/sidebars.json b/website/sidebars.jsonindex 5152e7b27..c7c1a00c2 100644--- a/website/sidebars.json+++ b/website/sidebars.json@@ -2,116 +2,165 @@"docs": {"Guide": ["guides/motivation",- "guides/gettingstarted",- "guides/quickstart",- "guides/whatisanactor",- "guides/typescriptactor",- "guides/environmentvariables",- "guides/datastorage",- "guides/puppeteerliveview",- "guides/sessionmanagement"+ "guides/quick-start",+ "guides/getting-started",+ "guides/what-is-an-actor",+ "guides/data-storage",+ "guides/environment-variables",+ "guides/session-management",+ "guides/type-script-actor",+ "guides/puppeteer-live-view"],- "Reference": [+ "API Reference": ["api/apify",- "api/autoscaledpool",- "api/basiccrawler",- "api/cheeriocrawler",- "api/dataset",- "api/keyvaluestore",- "api/liveviewserver",- "api/pseudourl",- "api/puppeteercrawler",- "api/puppeteerpool",- "api/request",- "api/requestlist",- "api/requestqueue",- "api/sessionpool",- "api/session",- "api/utils",- "api/log",- "api/puppeteer",- "api/social"+ {+ "type": "subcategory",+ "label": "Crawlers",+ "ids": [+ "api/basic-crawler",+ "api/cheerio-crawler",+ "api/puppeteer-crawler"+ ]+ },+ {+ "type": "subcategory",+ "label": "Result Stores",+ "ids": [+ "api/dataset",+ "api/key-value-store"+ ]+ },+ {+ "type": "subcategory",+ "label": "Scaling",+ "ids": [+ "api/autoscaled-pool",+ "api/puppeteer-pool",+ "api/session",+ "api/session-pool",+ "api/snapshotter",+ "api/system-status"+ ]+ },+ {+ "type": "subcategory",+ "label": "Sources",+ "ids": [+ "api/request",+ "api/request-list",+ "api/request-queue",+ "api/pseudo-url"+ ]+ },+ {+ "type": "subcategory",+ "label": "Utilities",+ "ids": [+ "api/utils",+ "api/log",+ "api/puppeteer",+ "api/social"+ ]+ }+ ],+ "Type Definitions": [+ {+ "type": "subcategory",+ "label": "Constructor Options",+ "ids": [+ "typedefs/autoscaled-pool-options",+ "typedefs/basic-crawler-options",+ "typedefs/cheerio-crawler-options",+ "typedefs/puppeteer-crawler-options",+ "typedefs/puppeteer-pool-options",+ "typedefs/launch-puppeteer-options",+ "typedefs/request-options",+ "typedefs/request-list-options",+ "typedefs/request-as-browser-options",+ "typedefs/session-options",+ "typedefs/session-pool-options",+ "typedefs/snapshotter-options",+ "typedefs/stealth-options",+ "typedefs/system-status-options"+ ]+ },+ {+ "type": "subcategory",+ "label": "Functions - Crawlers",+ "ids": [+ "typedefs/cheerio-handle-page",+ "typedefs/cheerio-handle-page-inputs",+ "typedefs/handle-failed-request",+ "typedefs/handle-failed-request-input",+ "typedefs/handle-request",+ "typedefs/handle-request-inputs",+ "typedefs/prepare-request",+ "typedefs/prepare-request-inputs",+ "typedefs/puppeteer-handle-page",+ "typedefs/puppeteer-handle-page-inputs",+ "typedefs/launch-puppeteer",+ "typedefs/puppeteer-goto",+ "typedefs/puppeteer-goto-inputs"+ ]+ },+ {+ "type": "subcategory",+ "label": "Functions - Dataset",+ "ids": [+ "typedefs/dataset-consumer",+ "typedefs/dataset-mapper",+ "typedefs/dataset-reducer"+ ]+ },+ {+ "type": "subcategory",+ "label": "Functions - KeyValueStore",+ "ids": [+ "typedefs/key-consumer"+ ]+ },+ {+ "type": "subcategory",+ "label": "Functions - Sessions",+ "ids": [+ "typedefs/create-session"+ ]+ },+ {+ "type": "subcategory",+ "label": "Functions - Utilities",+ "ids": [+ "typedefs/request-transform"+ ]+ },+ {+ "type": "subcategory",+ "label": "Return Types",+ "ids": [+ "typedefs/actor-run",+ "typedefs/apify-call-error",+ "typedefs/apify-env",+ "typedefs/dataset-content",+ "typedefs/memory-info",+ "typedefs/queue-operation-info",+ "typedefs/request-list-state",+ "typedefs/session-state",+ "typedefs/social-handles",+ "typedefs/system-info"+ ]+ }]},"examples": {"Examples": [- "examples/basiccrawler",- "examples/cheeriocrawler",- "examples/puppeteercrawler",- "examples/puppeteerwithproxy",- "examples/puppeteersitemap",+ "examples/basic-crawler",+ "examples/cheerio-crawler",+ "examples/puppeteer-crawler",+ "examples/puppeteer-with-proxy",+ "examples/puppeteer-sitemap","examples/screenshots",- "examples/callactor",- "examples/synchronousrun"- ]- },- "autoscaling": {- "Autoscaling Tools": [- "api/snapshotter",- "api/systemstatus"- ]- },- "typedefs": {- "Type Definitions": [- "typedefs/actorrun",- "typedefs/apifycallerror",- "typedefs/apifyenv",- "typedefs/datasetcontent",- "typedefs/memoryinfo",- "typedefs/queueoperationinfo",- "typedefs/requestliststate",- "typedefs/sessionstate",- "typedefs/socialhandles",- "typedefs/systeminfo"- ],- "Constructor Options": [- "typedefs/autoscaledpooloptions",- "typedefs/basiccrawleroptions",- "typedefs/cheeriocrawleroptions",- "typedefs/puppeteercrawleroptions",- "typedefs/puppeteerpooloptions",- "typedefs/launchpuppeteeroptions",- "typedefs/requestoptions",- "typedefs/requestlistoptions",- "typedefs/requestasbrowseroptions",- "typedefs/sessionoptions",- "typedefs/sessionpooloptions",- "typedefs/snapshotteroptions",- "typedefs/stealhtoptions",- "typedefs/systemstatusoptions"- ]- },- "User-Functions": {- "Crawler User-Functions": [- "typedefs/cheeriohandlepage",- "typedefs/cheeriohandlepageinputs",- "typedefs/handlefailedrequest",- "typedefs/handlefailedrequestinput",- "typedefs/handlerequest",- "typedefs/handlerequestinputs",- "typedefs/preparerequest",- "typedefs/preparerequestinputs"- ],- "Puppeteer User-Functions": [- "typedefs/puppeteerhandlepage",- "typedefs/puppeteerhandlepageinputs",- "typedefs/launchpuppeteer",- "typedefs/puppeteergoto",- "typedefs/puppeteergotoinputs"- ],- "Dataset User-Functions": [- "typedefs/datasetconsumer",- "typedefs/datasetmapper",- "typedefs/datasetreducer"- ],- "KeyValueStore User-Functions": [- "typedefs/keyconsumer"- ],- "SessionPool User-Functions": [- "typedefs/createsession"- ],- "Utility User-Functions": [- "typedefs/requesttransform"+ "examples/call-actor",+ "examples/synchronous-run"]}}diff --git a/website/siteConfig.js b/website/siteConfig.jsindex 1e1d4dae9..39f1708a0 100644--- a/website/siteConfig.js+++ b/website/siteConfig.js@@ -19,6 +19,7 @@// pinned: true,// },// ];+const extlink = require('remarkable-extlink');const repoUrl = 'https://github.com/apifytech/apify-js';@@ -46,8 +47,8 @@ const siteConfig = {headerLinks: [{ search: true },{ doc: 'guides/motivation', label: 'Guide' },- { doc: 'examples/basiccrawler', label: 'Examples' },- { doc: 'api/apify', label: 'Reference' },+ { doc: 'examples/basic-crawler', label: 'Examples' },+ { doc: 'api/apify', label: 'API Reference' },{ href: repoUrl, label: 'GitHub' },// { page: 'help', label: 'Help' },// { blog: true, label: 'Blog' },@@ -99,6 +100,18 @@ const siteConfig = {// Using Prism for syntax highlightingusePrism: true,+ docsSideNavCollapsible: true,+ markdownOptions: {+ html: true,+ },+ markdownPlugins: [+ (md) => {+ extlink(md, {+ host: 'sdk.apify.com', // The hrefs that you DON'T want to be external+ });+ },+ ],+// Add custom scripts here that would be placed in <script> tags.scripts: ['https://buttons.github.io/buttons.js',diff --git a/website/tools/build_docs.js b/website/tools/build_docs.jsindex 51aa7f327..b39e47a85 100644--- a/website/tools/build_docs.js+++ b/website/tools/build_docs.js@@ -1,12 +1,14 @@-const jsdoc2md = require('jsdoc-to-markdown'); // eslint-disable-line-const readline = require('readline');+const decamelize = require('decamelize');const fs = require('fs');+const jsdoc2md = require('jsdoc-to-markdown'); // eslint-disable-lineconst path = require('path');-const { promisify } = require('util');const prettier = require('prettier'); // eslint-disable-line+const readline = require('readline');+const { promisify } = require('util');const prettierConfig = require('./prettier.config');const writeFile = promisify(fs.writeFile);+const BASE_URL = '/docs';/* eslint-disable no-shadow */@@ -14,9 +16,13 @@ const classNames = [];const namespaces = [];const typedefs = [];+const toId = (name) => {+ return decamelize(name, '-').replace(/\s/g, '-');+};+const getHeader = (title) => {const prefix = /puppeteer|social|log/.test(title) ? 'utils.' : '';- const id = title.replace(/\s/g, '').toLowerCase();+ const id = toId(title);return `---\nid: ${id}\ntitle: ${prefix}${title}\n---\n`;};@@ -24,13 +30,20 @@ const getRenderOptions = (template, data) => ({template,data,'name-format': true,- 'param-list-format': 'table',+ separators: true,+ 'param-list-format': 'list',+ 'property-list-format': 'list','heading-depth': 1,helper: [path.join(__dirname, 'helpers.js')],partial: [- path.join(__dirname, 'partials', 'params-table.hbs'),- path.join(__dirname, 'partials', 'properties-table.hbs'),+ path.join(__dirname, 'partials', 'params-list.hbs'),+ path.join(__dirname, 'partials', 'properties-list.hbs'),path.join(__dirname, 'partials', 'link.hbs'),+ path.join(__dirname, 'partials', 'body.hbs'),+ path.join(__dirname, 'partials', 'docs.hbs'),+ path.join(__dirname, 'partials', 'returns.hbs'),+ path.join(__dirname, 'partials', 'sig-name.hbs'),+ path.join(__dirname, 'partials', 'header.hbs'),],});@@ -51,7 +64,26 @@ const readFileFromLine = async (path, lineNumber = 1) => {});};-const generateFinalMarkdown = (title, text) => {+const getLinkToEntity = (entityName, entityMap) => {+ const entity = entityMap.get(entityName);+ const folder = entity.kind === 'typedef' ? 'typedefs' : 'api';+ const id = toId(entityName);+ return `${BASE_URL}/${folder}/${id}`;+};++/**+ * Creates a regular expression that will match any entity from the provided map.+ * @param {Map} entityMap+ * @returns {RegExp}+ */+const createEntityRegex = (entityMap) => {+ const entities = Array.from(entityMap.keys())+ .sort((a, b) => b.length - a.length)+ .join('|');+ return new RegExp(entities);+};++const generateFinalMarkdown = (title, text, entityMap) => {const header = getHeader(title);// Remove Class titles so we don't have double page titles with Docusaurus.const rx = new RegExp(`# \`?${title}\`?.*?\n`);@@ -64,12 +96,40 @@ const generateFinalMarkdown = (title, text) => {return `${p1}<${p2.replace(dotsRx, replacer)}>`;};text = text.replace(dotsRx, replacer);- // Fix class links- const linksRx = new RegExp(`([("])#(module_)?(${classNames.join('|')})([)"])`, 'gi');- text = text.replace(linksRx, (match, p1, p2, p3, p4) => p1 + p3.toLowerCase() + p4);- // Fix typedef links- const typeLinkRx = new RegExp(`([("])#(module_)?(${typedefs.join('|')})([)"])`, 'gi');- text = text.replace(typeLinkRx, (match, p1, p2, p3, p4) => `${p1}../typedefs/${p3.toLowerCase()}${p4}`);+ // Remove 'module_' prefix from Apify+ text = text.replace(/module_Apify/g, 'Apify');+ // Generate correct links+ const linkRx = /(<code>)?LINK!(.*?)!LINK(<\/code>)?/g;+ const entityRx = createEntityRegex(entityMap);+ text = text.replace(linkRx, (linkToken, openTag, target, closeTag) => {+ const isHtml = !!(openTag && closeTag);+ const entityMatch = linkToken.match(entityRx);+ if (!entityMatch) {+ return isHtml ? `${openTag}${target}${closeTag}` : `\`${target}\``;+ }+ const [entity] = entityMatch;+ let link = getLinkToEntity(entity, entityMap);+ // Link to target can include method delimited by hash+ // such as RequestList.initialize => requestList.initialize()+ let caption = target;+ const [parentName, propertyName] = target.split(/[#.]/);+ if (propertyName) {+ link = `${link}#${propertyName}`;+ caption = `${parentName}.${propertyName}`;+ // Add parens to methods.+ if (target.includes('#')) caption += '()';+ // We want typedefs uppercased and class instances lowercased.+ const { kind } = entityMap.get(entity);+ caption = kind === 'class' ? `${caption[0].toLowerCase() + caption.substring(1)}` : caption;+ }+ // Normalize link to work with Docusaurus heading anchors which are lowercase+ link = link.toLowerCase();+ return isHtml+ ? `${openTag}<a href="${link}">${caption}</a>${closeTag}`+ : `[\`${caption}\`](${link})`;+ });+ // Remove annoying "new exports."+ text = text.replace(/new exports\./g, 'new ');// Format Markdown with Prettierreturn prettier.format(header + text, prettierConfig);};@@ -108,6 +168,13 @@ const main = async () => {if (identifier.kind === 'typedef' && !identifier.ignore) typedefs.push(identifier.name);});+ // build a map of all available entities for link generation.+ // Apify needs to be added manually since its actually a module+ const entityMap = new Map();+ entityMap.set('Apify', { name: 'Apify', kind: 'namespace' });+ classNames.forEach(name => entityMap.set(name, { name, kind: 'className' }));+ namespaces.forEach(name => entityMap.set(name, { name, kind: 'namespace' }));+ typedefs.forEach(name => entityMap.set(name, { name, kind: 'typedef' }));// create a doc file for Apifyconst mainModule = 'Apify';@@ -129,7 +196,7 @@ const main = async () => {const finalData = moduleData.sort(comparator).concat(otherData);const output = jsdoc2md.renderSync(getRenderOptions(template, finalData));- const markdown = generateFinalMarkdown(mainModule, output);+ const markdown = generateFinalMarkdown(mainModule, output, entityMap);fs.writeFileSync(path.resolve(sourceFilesOutputDir, `${mainModule}.md`), markdown);// create a doc file file for each class@@ -137,7 +204,7 @@ const main = async () => {const template = `{{#class name="${className}"}}{{>docs}}{{/class}}`;console.log(`Rendering ${className}, template: ${template}`); // eslint-disable-line no-consoleconst output = jsdoc2md.renderSync(getRenderOptions(template, templateData));- const markdown = generateFinalMarkdown(className, output);+ const markdown = generateFinalMarkdown(className, output, entityMap);fs.writeFileSync(path.resolve(sourceFilesOutputDir, `${className}.md`), markdown);});@@ -146,7 +213,7 @@ const main = async () => {const template = `{{#namespace name="${name}"}}{{>docs}}{{/namespace}}`;console.log(`Rendering ${name}, template: ${template}`); // eslint-disable-line no-consoleconst output = jsdoc2md.renderSync(getRenderOptions(template, templateData));- const markdown = generateFinalMarkdown(name, output);+ const markdown = generateFinalMarkdown(name, output, entityMap);fs.writeFileSync(path.resolve(sourceFilesOutputDir, `${name}.md`), markdown);});@@ -155,7 +222,7 @@ const main = async () => {const template = `{{#identifier name="${name}"}}{{>docs}}{{/identifier}}`;console.log(`Rendering ${name}, template: ${template}`); // eslint-disable-line no-consoleconst output = jsdoc2md.renderSync(getRenderOptions(template, templateData));- const markdown = generateFinalMarkdown(name, output);+ const markdown = generateFinalMarkdown(name, output, entityMap);fs.writeFileSync(path.resolve(typeFilesOutputDir, `${name}.md`), markdown);});};diff --git a/website/tools/helpers.js b/website/tools/helpers.jsindex e7d3643e1..600cb4655 100644--- a/website/tools/helpers.js+++ b/website/tools/helpers.js@@ -2,15 +2,15 @@ const ddata = require('dmd/helpers/ddata'); // eslint-disable-line/* eslint-disable no-underscore-dangle */-exports.inlineLinks = (text, options) => {+exports.inlineLinks = (text) => {if (text) {const links = ddata.parseLink(text);links.forEach((link) => {- const linked = ddata._link(link.url, options);- if (link.caption === link.url) link.caption = linked.name;- if (linked.url) link.url = linked.url;- const url = link.url.includes('+') ? link.url : link.url.toLowerCase();- text = text.replace(link.original, `[\`${link.caption}\`](${url})`);+ // const linked = ddata._link(link.url, options);+ // if (link.caption === link.url) link.caption = linked.name;+ // if (linked.url) link.url = linked.url;+ // const url = link.url.includes('+') ? link.url : link.url.toLowerCase();+ text = text.replace(link.original, `LINK!${link.caption}!LINK`);});}return text;@@ -21,6 +21,8 @@ exports.escape = function escape(input) {return input.replace(/([*|])/g, '$1');};+exports.lowerCase = text => text.toLowerCase();+exports.class = (options) => {options.hash.kind = 'class';const result = ddata._identifier(options);diff --git a/website/tools/partials/body.hbs b/website/tools/partials/body.hbsnew file mode 100644index 000000000..6cf9d1616--- /dev/null+++ b/website/tools/partials/body.hbs@@ -0,0 +1,29 @@+{{>deprecated~}}+{{>returns~}}+{{>description~}}+{{>scope~}}+{{>summary~}}+{{>augments~}}+{{>implements~}}+{{>mixes~}}+{{>default~}}+{{>chainable~}}+{{>overrides~}}+{{>category~}}+{{>throws~}}+{{>fires~}}+{{>this~}}+{{>access~}}+{{>readOnly~}}+{{>requires~}}+{{>customTags~}}+{{>see~}}+{{>since~}}+{{>version~}}+{{>authors~}}+{{>license~}}+{{>copyright~}}+{{>todo~}}+{{>params~}}+{{>properties~}}+{{>examples~}}diff --git a/website/tools/partials/docs.hbs b/website/tools/partials/docs.hbsnew file mode 100644index 000000000..7bf675e19--- /dev/null+++ b/website/tools/partials/docs.hbs@@ -0,0 +1,4 @@+{{>header~}}+{{>body}}+{{>separator~}}+{{>members~}}diff --git a/website/tools/partials/header.hbs b/website/tools/partials/header.hbsnew file mode 100644index 000000000..71c6a8c00--- /dev/null+++ b/website/tools/partials/header.hbs@@ -0,0 +1,3 @@+<a name="{{{lowerCase name}}}"></a>++{{>heading-indent}}{{>sig-name}}diff --git a/website/tools/partials/link.hbs b/website/tools/partials/link.hbsindex 612ad1f7f..605f37517 100644--- a/website/tools/partials/link.hbs+++ b/website/tools/partials/link.hbs@@ -1,25 +1,7 @@{{! usage: link to="namepath" html=true/false caption="optional caption"~}}{{~#if html~}}- <code>-- {{~#link to~}}- {{#if url~}}- <a href="{{{url}}}">{{#if ../../caption}}{{../../../caption}}{{else}}{{name}}{{/if}}</a>- {{~else~}}- {{#if ../../caption}}{{../../../caption}}{{else}}{{name}}{{/if~}}- {{/if~}}- {{/link~}}-- </code>+ <code>{{~#link to~}}LINK!{{name}}!LINK{{/link~}}</code>{{~else~}}-- {{#link to~}}- {{#if url~}}- [`{{#if ../../caption}}{{escape ../../../caption}}{{else}}{{escape name}}{{/if}}`]({{{url}}})- {{~else~}}- `{{#if ../../caption}}{{escape ../../../caption}}{{else}}{{escape name}}{{/if~}}`- {{~/if~}}- {{/link~}}-+ {{#link to~}}LINK!{{name}}!LINK{{/link~}}{{/if~}}diff --git a/website/tools/partials/params-list.hbs b/website/tools/partials/params-list.hbsnew file mode 100644index 000000000..8b2492eab--- /dev/null+++ b/website/tools/partials/params-list.hbs@@ -0,0 +1,9 @@+{{#if params}}+{{#params}}**Params**++{{#each this~}}+{{indent}}- **`{{name}}`**:{{#if type}} {{>linked-type-list types=type.names delimiter=" | " }}{{/if}}{{#unless (equal defaultvalue undefined)}} {{>defaultvalue equals=true ~}}{{/unless}}{{#if description}} - {{{inlineLinks description}}}{{/if}}+{{/each}}++{{/params~}}+{{/if}}diff --git a/website/tools/partials/properties-list.hbs b/website/tools/partials/properties-list.hbsnew file mode 100644index 000000000..f8207ed0a--- /dev/null+++ b/website/tools/partials/properties-list.hbs@@ -0,0 +1,11 @@+{{#if properties}}+## Properties+{{#each properties~}}+### `{{{name}}}`+**Type**: {{#if type}} {{>linked-type-list types=type.names delimiter=" | " }}{{/if}}{{#unless (equal defaultvalue undefined)}} {{>defaultvalue equals=true ~}}{{/unless}}++{{#if description}}{{{inlineLinks description}}}{{/if}}+{{#unless @last}}{{>separator~}}{{/unless~}}+{{/each}}++{{/if~}}diff --git a/website/tools/partials/returns.hbs b/website/tools/partials/returns.hbsnew file mode 100644index 000000000..8fb55516c--- /dev/null+++ b/website/tools/partials/returns.hbs@@ -0,0 +1,12 @@+{{#if returns}}+**Returns**: {{#each returns~}}+ {{#if type~}}+ {{#if type.names}}{{>linked-type-list types=type.names delimiter=" \| " ~}}{{/if}}+ {{~#if description}} - {{{inlineLinks description}}}{{/if~}}+ {{else~}}+ {{{inlineLinks description}~}}+ {{/if~}}+{{~/each}}+++{{/if}}diff --git a/website/tools/partials/sig-name.hbs b/website/tools/partials/sig-name.hbsnew file mode 100644index 000000000..97eef0a8f--- /dev/null+++ b/website/tools/partials/sig-name.hbs@@ -0,0 +1,12 @@+{{#if virtual}}*{{/if}}{{#with (parentObject)}}{{#if virtual}}*{{/if~}}{{/with~}}+{{#if name}}{{#sig~}}+{{{@depOpen}~}}+{{{@codeOpen}~}}+{{#if @prefix}}{{@prefix}} {{/if~}}+{{@parent~}}+{{@accessSymbol}}{{#if (isEvent)}}"{{{name}}}"{{else}}{{{escape name}}}{{/if~}}+{{#if @methodSign}}{{#if (isEvent)}} {{@methodSign}}{{else}}{{@methodSign}}{{/if}}{{/if~}}+{{{@codeClose}~}}+{{{@depClose}~}}+{{~/sig}}{{/if~}}+{{#if virtual}}*{{/if}}{{#with (parentObject)}}{{#if virtual}}*{{/if~}}{{/with~}}