Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Typescript declarations automatically generated from JSDoc comments (#558)

Matej Gagyi1b78219e7b20
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index 28b5e0bd8..2b48c4c48 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -12,17 +12,17 @@ separate, detailed, documentation pages accessible from the left sidebar.
- [`.addWebhook(options)`](#module_Apify.addWebhook) ⇒ `Promise<Object>`
- [`.call(actId, [input], [options])`](#module_Apify.call) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)
- [`.callTask(taskId, [input], [options])`](#module_Apify.callTask) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)
- - [`.client`](#module_Apify.client)
+ - [`.client`](#module_Apify.client) : `*`
- [`.events`](#module_Apify.events)
- [`.getApifyProxyUrl(options)`](#module_Apify.getApifyProxyUrl) ⇒ `String`
- - [`.getEnv()`](#module_Apify.getEnv) ⇒ `Object`
+ - [`.getEnv()`](#module_Apify.getEnv) ⇒ [`ApifyEnv`](../typedefs/apifyenv)
- [`.getInput`](#module_Apify.getInput) ⇒ `Promise<Object>`
- - [`.getMemoryInfo()`](#module_Apify.getMemoryInfo) ⇒ `Promise<Object>`
+ - [`.getMemoryInfo()`](#module_Apify.getMemoryInfo) ⇒ [`Promise<MemoryInfo>`](../typedefs/memoryinfo)
- [`.getValue(key)`](#module_Apify.getValue) ⇒ `Promise<Object>`
- [`.isAtHome()`](#module_Apify.isAtHome) ⇒ `Boolean`
- [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ `Promise<Browser>`
- [`.main(userFunc)`](#module_Apify.main)
- - [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ `Promise`
+ - [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ `Promise<void>`
- [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [`Promise<Dataset>`](dataset)
- [`.openKeyValueStore([storeIdOrName], [options])`](#module_Apify.openKeyValueStore) ⇒ [`Promise<KeyValueStore>`](keyvaluestore)
- [`.openRequestList`](#module_Apify.openRequestList) ⇒ [`Promise<RequestList>`](requestlist)
@@ -308,7 +308,7 @@ obtain the output.
</table>
<a name="module_Apify.client"></a>
-## `Apify.client`
+## `Apify.client` : `*`
Gets the default instance of the `ApifyClient` class provided <a href="https://docs.apify.com/api/apify-client-js/latest"
target="_blank">apify-client</a> by the NPM package. The instance is created automatically by the Apify SDK and it is configured using the
@@ -450,39 +450,9 @@ full list of available country codes
</table>
<a name="module_Apify.getEnv"></a>
-## `Apify.getEnv()` ⇒ `Object`
+## `Apify.getEnv()` ⇒ [`ApifyEnv`](../typedefs/apifyenv)
-Returns a new object which contains information parsed from all the `APIFY_XXX` environment variables. It has properties such as the following:
-
-```javascript
-{
- // ID of the actor (APIFY_ACTOR_ID)
- actorId: String,
- // ID of the actor run (APIFY_ACTOR_RUN_ID)
- actorRunId: String,
- // ID of the actor task (APIFY_ACTOR_TASK_ID)
- actorTaskId: String,
- // ID of the user who started the actor - note that it might be
- // different than the owner of the actor (APIFY_USER_ID)
- userId: String,
- // Authentication token representing privileges given to the actor run,
- // it can be passed to various Apify APIs (APIFY_TOKEN).
- token: String,
- // Date when the actor was started (APIFY_STARTED_AT)
- startedAt: Date,
- // Date when the actor will time out (APIFY_TIMEOUT_AT)
- timeoutAt: Date,
- // ID of the key-value store where input and output data of this
- // actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
- defaultKeyValueStoreId: String,
- // ID of the dataset where input and output data of this
- // actor is stored (APIFY_DEFAULT_DATASET_ID)
- defaultDatasetId: String,
- // Amount of memory allocated for the actor,
- // in megabytes (APIFY_MEMORY_MBYTES)
- memoryMbytes: Number,
-}
-```
+Returns a new [`ApifyEnv`](../typedefs/apifyenv) object which contains information parsed from all the `APIFY_XXX` environment variables.
For the list of the `APIFY_XXX` environment variables, see <a href="https://docs.apify.com/actor/run#environment-variables" target="_blank">Actor
documentation</a>. If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.
@@ -513,24 +483,9 @@ For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyVal
**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.
<a name="module_Apify.getMemoryInfo"></a>
-## `Apify.getMemoryInfo()` ⇒ `Promise<Object>`
+## `Apify.getMemoryInfo()` ⇒ [`Promise<MemoryInfo>`](../typedefs/memoryinfo)
-Returns memory statistics of the process and the system, which is an object with the following properties:
-
-```javascript
-{
- // Total memory available in the system or container
- totalBytes: Number,
- // Amount of free memory in the system or container
- freeBytes: Number,
- // Amount of memory used (= totalBytes - freeBytes)
- usedBytes: Number,
- // Amount of memory used the current Node.js process
- mainProcessBytes: Number,
- // Amount of memory used by child processes of the current Node.js process
- childProcessesBytes: Number,
-}
-```
+Returns memory statistics of the process and the system, see [`MemoryInfo`](../typedefs/memoryinfo).
If the process runs inside of Docker, the `getMemoryInfo` gets container memory limits, otherwise it gets system memory limits.
@@ -731,7 +686,7 @@ the promise will be awaited. The user function is called with no arguments.</p>
</table>
<a name="module_Apify.metamorph"></a>
-## `Apify.metamorph(targetActorId, [input], [options])` ⇒ `Promise`
+## `Apify.metamorph(targetActorId, [input], [options])` ⇒ `Promise<void>`
Transforms this actor run to an actor run of a given actor. The system stops the current container and starts the new container instead. All the
default storages are preserved and the new input is stored under the `INPUT-METAMORPH-1` key in the same default key-value store.
@@ -888,7 +843,7 @@ const requestList = await Apify.openRequestList('my-name', sources);
<p> If <code>null</code>, the list will not be persisted and will only be stored in memory. Process restart
will then cause the list to be crawled again from the beginning. We suggest always using a name.</p>
</td></tr><tr>
-<td><code>sources</code></td><td><code>Array<Object&gt;</code> | <code>Array.&lt;string></code></td>
+<td><code>sources</code></td><td><code>Array<(Request|RequestOptions|string)></code></td>
</tr>
<tr>
<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>.
@@ -901,7 +856,7 @@ using <code>regex</code> parameter containing regular expression pattern for URL
<p> For details, see the <a href="requestlist#new_RequestList_new"><code>RequestList</code></a>
constructor options.</p>
</td></tr><tr>
-<td><code>[options]</code></td><td><code>Object</code></td>
+<td><code>[options]</code></td><td><code><a href="../typedefs/requestlistoptions">RequestListOptions</a></code></td>
</tr>
<tr>
<td colspan="3"><p>The <a href="requestlist#new_RequestList_new"><code>new RequestList</code></a> options. Note that the listName parameter supersedes
@@ -961,7 +916,7 @@ For more details and code examples, see the [`SessionPool`](sessionpool) class.
</thead>
<tbody>
<tr>
-<td><code>sessionPoolOptions</code></td><td><code>Object</code></td>
+<td><code>sessionPoolOptions</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td>
</tr>
<tr>
<td colspan="3"><p>The <a href="sessionpool#new_SessionPool_new"><code>new SessionPool</code></a> options</p>
diff --git a/docs/api/AutoscaledPool.md b/docs/api/AutoscaledPool.md
index 8b9b635fe..0fda68935 100644
--- a/docs/api/AutoscaledPool.md
+++ b/docs/api/AutoscaledPool.md
@@ -67,107 +67,15 @@ await pool.run();
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>All <code>AutoscaledPool</code> parameters are passed
- via an options object with the following keys.</p>
-</td></tr><tr>
-<td><code>options.runTaskFunction</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function that performs an asynchronous resource-intensive task.
- The function must either be labeled <code>async</code> or return a promise.</p>
-</td></tr><tr>
-<td><code>options.isTaskReadyFunction</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function that indicates whether <code>runTaskFunction</code> should be called.
- This function is called every time there is free capacity for a new task and it should
- indicate whether it should start a new task or not by resolving to either <code>true</code> or `false.
- Besides its obvious use, it is also useful for task throttling to save resources.</p>
-</td></tr><tr>
-<td><code>options.isFinishedFunction</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function that is called only when there are no tasks to be processed.
- If it resolves to <code>true</code> then the pool&#39;s run finishes. Being called only
- when there are no tasks being processed means that as long as <code>isTaskReadyFunction()</code>
- keeps resolving to <code>true</code>, <code>isFinishedFunction()</code> will never be called.
- To abort a run, use the <a href="#AutoscaledPool+abort"><code>abort()</code></a> method.</p>
-</td></tr><tr>
-<td><code>[options.minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>The minimum number of tasks running in parallel.</p>
-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
- If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
-</td></tr><tr>
-<td><code>[options.maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>The maximum number of tasks running in parallel.</p>
-</td></tr><tr>
-<td><code>[options.desiredConcurrency]</code></td><td><code>Number</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>The desired number of tasks that should be running parallel on the start of the pool,
- if there is a large enough supply of them.
- By default, it is <code>options.minConcurrency</code>.</p>
-</td></tr><tr>
-<td><code>[options.desiredConcurrencyRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Minimum level of desired concurrency to reach before more scaling up is allowed.</p>
-</td></tr><tr>
-<td><code>[options.scaleUpStepRatio]</code></td><td><code>Number</code></td><td><code>0.05</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the fractional amount of desired concurrency to be added with each scaling up.
- The minimum scaling step is one.</p>
-</td></tr><tr>
-<td><code>[options.scaleDownStepRatio]</code></td><td><code>Number</code></td><td><code>0.05</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the amount of desired concurrency to be subtracted with each scaling down.
- The minimum scaling step is one.</p>
-</td></tr><tr>
-<td><code>[options.maybeRunIntervalSecs]</code></td><td><code>Number</code></td><td><code>0.5</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how often the pool should call the <code>runTaskFunction()</code> to start a new task, in seconds.
- This has no effect on starting new tasks immediately after a task completes.</p>
-</td></tr><tr>
-<td><code>[options.loggingIntervalSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Specifies a period in which the instance logs its state, in seconds.
- Set to <code>null</code> to disable periodic logging.</p>
-</td></tr><tr>
-<td><code>[options.autoscaleIntervalSecs]</code></td><td><code>Number</code></td><td><code>10</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines in seconds how often the pool should attempt to adjust the desired concurrency
- based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
- We suggest using a value from 5 to 20.</p>
-</td></tr><tr>
-<td><code>[options.snapshotterOptions]</code></td><td><code>Number</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Options to be passed down to the <a href="snapshotter"><code>Snapshotter</code></a> constructor. This is useful for fine-tuning
- the snapshot intervals and history.</p>
-</td></tr><tr>
-<td><code>[options.systemStatusOptions]</code></td><td><code>Number</code></td><td></td>
+<td><code>options</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>Options to be passed down to the <a href="systemstatus"><code>SystemStatus</code></a> constructor. This is useful for fine-tuning
- the system status reports. If a custom snapshotter is set in the options, it will be used
- by the pool.</p>
+<td colspan="3"><p>All <code>AutoscaledPool</code> configuration options.</p>
</td></tr></tbody>
</table>
<a name="AutoscaledPool+minConcurrency"></a>
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index fc848a313..1ad927bcc 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -55,123 +55,28 @@ await crawler.run();
```
- [BasicCrawler](basiccrawler)
- - [`new BasicCrawler(options, sessionPoolOptions)`](#new_BasicCrawler_new)
- - [`.run()`](#BasicCrawler+run) ⇒ `Promise`
+ - [`new BasicCrawler(options)`](#new_BasicCrawler_new)
+ - [`.run()`](#BasicCrawler+run) ⇒ `Promise<void>`
<a name="new_BasicCrawler_new"></a>
-## `new BasicCrawler(options, sessionPoolOptions)`
+## `new BasicCrawler(options)`
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td><code>Object</code></td><td></td>
+<td><code>options</code></td><td><code><a href="../typedefs/basiccrawleroptions">BasicCrawlerOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>All <code>BasicCrawler</code> parameters are passed
- via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>options.handleRequestFunction</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each URL to crawl.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- request: Request,
- autoscaledPool: AutoscaledPool
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
-<p> The function must return a promise, which is then awaited by the crawler.</p>
-<p> If the function throws an exception, the crawler will try to re-crawl the
- request later, up to <code>option.maxRequestRetries</code> times.
- If all the retries fail, the crawler calls the function
- provided to the <code>options.handleFailedRequestFunction</code> parameter.
- To make this work, you should <strong>always</strong>
- let your function throw exceptions rather than catch them.
- The exceptions are logged to the request using the
- <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
-</td></tr><tr>
-<td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Static list of URLs to be processed.
- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
-</td></tr><tr>
-<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
-</td></tr><tr>
-<td><code>[options.handleRequestTimeoutSecs]</code></td><td><code>number</code></td><td><code>60</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Timeout in which the function passed as <code>options.handleRequestFunction</code> needs to finish, in seconds.</p>
-</td></tr><tr>
-<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- request: Request,
- error: Error,
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
- represents the last error thrown during processing of the request.</p>
-<p> See
- <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
- for the default implementation of this function.</p>
-</td></tr><tr>
-<td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how many times the request is retried if <a href="#new_BasicCrawler_new"><code>handleRequestFunction()</code></a> fails.</p>
-</td></tr><tr>
-<td><code>[options.maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
- Always set this value in order to prevent infinite loops in misconfigured crawlers.
- Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
-</td></tr><tr>
-<td><code>[options.autoscaledPoolOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.
- Note that the <code>runTaskFunction</code> and <code>isTaskReadyFunction</code> options
- are provided by <code>BasicCrawler</code> and cannot be overridden.
- However, you can provide a custom implementation of <code>isFinishedFunction</code>.</p>
-</td></tr><tr>
-<td><code>[options.minConcurrency]</code></td><td><code>Object</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
- If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
-</td></tr><tr>
-<td><code>[options.maxConcurrency]</code></td><td><code>Object</code></td><td><code>1000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
-</td></tr><tr>
-<td><code>[options.useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If set to true. Basic crawler will initialize the <a href="sessionpool"><code>SessionPool</code></a> with the corresponding <code>sessionPoolOptions</code>.
- The session instance will be than available in the <code>handleRequestFunction</code>.</p>
-</td></tr><tr>
-<td><code>sessionPoolOptions</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>The <a href="sessionpool#new_SessionPool_new"><code>new SessionPool</code></a> options</p>
-</td></tr></tbody>
+</tr></tbody>
</table>
<a name="BasicCrawler+run"></a>
-## `basicCrawler.run()` ⇒ `Promise`
+## `basicCrawler.run()` ⇒ `Promise<void>`
Runs the crawler. Returns a promise that gets resolved once all the requests are processed.
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 68d0d4ddc..072e484f0 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -82,219 +82,15 @@ await crawler.run();
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td><code>Object</code></td><td></td>
+<td><code>options</code></td><td><code><a href="../typedefs/cheeriocrawleroptions">CheerioCrawlerOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>All <code>CheerioCrawler</code> parameters are passed
- via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>options.handlePageFunction</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each page
- loaded and parsed by the crawler.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- $: Cheerio, // the Cheerio object with parsed HTML
- body: String|Buffer // the request body of the web page
- // the parsed object from JSON string
- // if the response contains the content type application/json
- json: Object,
- request: Request,
- contentType: Object, // Parsed Content-Type header: { type, encoding }
- response: Object // An instance of Node&#39;s http.IncomingMessage object,
- autoscaledPool: AutoscaledPool,
- session: Session
-}</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p>
-<ul>
-<li><p>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> mime types</p>
-</li>
-<li><p>Buffer for others mime types</p>
-<p>Parsed <code>Content-Type</code> header using
-<a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>
-is stored in <code>contentType</code>.</p>
-<p>Cheerio is available only for HTML and XML content types.</p>
-<p>With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
-<p>If the function returns a promise, it is awaited by the crawler.</p>
-<p>If the function throws an exception, the crawler will try to re-crawl the
-request later, up to <code>option.maxRequestRetries</code> times.
-If all the retries fail, the crawler calls the function
-provided to the <code>options.handleFailedRequestFunction</code> parameter.
-To make this work, you should <strong>always</strong>
-let your function throw exceptions rather than catch them.
-The exceptions are logged to the request using the
-<a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
-</li>
-</ul>
-</td></tr><tr>
-<td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Static list of URLs to be processed.
- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
-</td></tr><tr>
-<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
-</td></tr><tr>
-<td><code>[options.requestOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Represents the options passed to
- <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.
- Provided <code>requestOptions</code> are added to internal defaults that cannot be overridden to ensure
- the operation of <code>CheerioCrawler</code> and all its options. Headers will not be merged,
- use <a href="requestlist"><code>RequestList</code></a> and/or <a href="requestqueue"><code>RequestQueue</code></a> to initialize your <a href="request"><code>Request</code></a> with the
- correct headers or use <code>options.prepareRequestFunction</code> to modify your <a href="request"><code>Request</code></a> dynamically.
- If you need more granular control over your requests, use <a href="basiccrawler"><code>BasicCrawler</code></a>.</p>
-<p> The mandatory internal defaults that <strong>CANNOT BE OVERRIDDEN</strong> by <code>requestOptions</code>:</p>
-<pre><code> {
- url, // Provided by RequestList and/or RequestQueue
- method, // Provided by RequestList and/or RequestQueue
- headers, // Provided by RequestList and/or RequestQueue
- payload, // Provided by RequestList and/or RequestQueue
- strictSSL, // Use options.ignoreSslErrors
- proxy, // Use options.useApifyProxy or options.proxyUrls
- }</code></pre></td></tr><tr>
-<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function that executes before the HTTP request is made to the target resource.
- This function is suitable for setting dynamic properties such as cookies to the <a href="request"><code>Request</code></a>.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- request: Request,
- session: Session
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request and the <a href="session"><code>Session</code></a> instance corresponds to used session.</p>
-<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance
- in place because there are already earlier references to it. Making a copy and returning it from
- this function is therefore not supported, because it would create inconsistencies where
- different parts of SDK would have access to a different <a href="request"><code>Request</code></a> instance.</p>
-</td></tr><tr>
-<td><code>[options.handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Timeout in which the function passed as <code>options.handlePageFunction</code> needs to finish, given in seconds.</p>
-</td></tr><tr>
-<td><code>[options.requestTimeoutSecs]</code></td><td><code>Number</code></td><td><code>30</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Timeout in which the HTTP request to the resource needs to finish, given in seconds.</p>
-</td></tr><tr>
-<td><code>[options.ignoreSslErrors]</code></td><td><code>Boolean</code></td><td><code>true</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If set to true, SSL certificate errors will be ignored.</p>
-</td></tr><tr>
-<td><code>[options.useApifyProxy]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If set to <code>true</code>, <code>CheerioCrawler</code> will be configured to use
- <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
- For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a></p>
-</td></tr><tr>
-<td><code>[options.apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>An array of proxy groups to be used
- by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
- Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>
-</td></tr><tr>
-<td><code>[options.apifyProxySession]</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Apify Proxy session identifier to be used with requests made by <code>CheerioCrawler</code>.
- All HTTP requests going through the proxy with the same session identifier
- will use the same target proxy server (i.e. the same IP address).
- The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>&quot;.&quot;</code>, <code>&quot;_&quot;</code> and <code>&quot;~&quot;</code>.
- Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>
-</td></tr><tr>
-<td><code>[options.proxyUrls]</code></td><td><code>Array<String></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>CheerioCrawler</code> instance.
- The provided custom proxies&#39; order will be randomized and the resulting list rotated.
- Custom proxies are not compatible with Apify Proxy and an attempt to use both
- configuration options will cause an error to be thrown on startup.</p>
-</td></tr><tr>
-<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- request: Request,
- error: Error,
-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
- represents the last error thrown during processing of the request.</p>
-<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
- for the default implementation of this function.</p>
-</td></tr><tr>
-<td><code>[options.additionalMimeTypes]</code></td><td><code>Array<String></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
- target="_blank">mime types</a> you want to process.
- By default <code>text/html</code>, <code>application/xhtml+xml</code> mime types are supported.</p>
-</td></tr><tr>
-<td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how many times the request is retried if either <code>requestFunction</code> or <code>handlePageFunction</code> fails.</p>
-</td></tr><tr>
-<td><code>[options.maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
- Always set this value in order to prevent infinite loops in misconfigured crawlers.
- Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
-</td></tr><tr>
-<td><code>[options.autoscaledPoolOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.
- Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options
- are provided by <code>CheerioCrawler</code> and cannot be overridden. Reasonable <a href="snapshotter"><code>Snapshotter</code></a>
- and <a href="systemstatus"><code>SystemStatus</code></a> defaults are provided to account for the fact that <code>cheerio</code>
- parses HTML synchronously and therefore blocks the event loop.</p>
-</td></tr><tr>
-<td><code>[options.minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
- If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
-</td></tr><tr>
-<td><code>[options.maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
-</td></tr><tr>
-<td><code>[options.useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
- It also marks Session as bad after a request timeout.</p>
-</td></tr><tr>
-<td><code>[options.sessionPoolOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>
-</td></tr><tr>
-<td><code>[options.persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>
-<p> It parses cookie from response &quot;set-cookie&quot; header saves or updates cookies for session and once the session is used for next request.
- It passes the &quot;Cookie&quot; header to the request with the session cookies.</p>
-</td></tr></tbody>
+</tr></tbody>
</table>
<a name="CheerioCrawler+run"></a>
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.md
index a9c9f47d9..6a1195d90 100644
--- a/docs/api/Dataset.md
+++ b/docs/api/Dataset.md
@@ -46,17 +46,39 @@ await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);
```
- [Dataset](dataset)
- - [`.pushData(data)`](#Dataset+pushData) ⇒ `Promise`
- - [`.getData([options])`](#Dataset+getData) ⇒ `Promise<Object>`
+ - [`new exports.Dataset(datasetId, datasetName)`](#new_Dataset_new)
+ - [`.pushData(data)`](#Dataset+pushData) ⇒ `Promise<void>`
+ - [`.getData([options])`](#Dataset+getData) ⇒ [`Promise<DatasetContent>`](../typedefs/datasetcontent)
- [`.getInfo()`](#Dataset+getInfo) ⇒ `Promise<Object>`
- - [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ `Promise`
- - [`.map(iteratee, options)`](#Dataset+map) ⇒ `Promise<Array>`
- - [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ `Promise<*>`
- - [`.drop()`](#Dataset+drop) ⇒ `Promise`
+ - [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ `Promise<void>`
+ - [`.map(iteratee, options)`](#Dataset+map) ⇒ `Promise<Array<T>>`
+ - [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ `Promise<T>`
+ - [`.drop()`](#Dataset+drop) ⇒ `Promise<void>`
+<a name="new_Dataset_new"></a>
+
+## `new exports.Dataset(datasetId, datasetName)`
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>datasetId</code></td><td><code>string</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>datasetName</code></td><td><code>string</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
<a name="Dataset+pushData"></a>
-## `dataset.pushData(data)` ⇒ `Promise`
+## `dataset.pushData(data)` ⇒ `Promise<void>`
Stores an object or an array of objects to the dataset. The function returns a promise that resolves when the operation finishes. It has no result,
but throws on invalid args or other errors.
@@ -89,19 +111,9 @@ The objects must be serializable to JSON and the JSON representation of each obj
</table>
<a name="Dataset+getData"></a>
-## `dataset.getData([options])` ⇒ `Promise<Object>`
+## `dataset.getData([options])` ⇒ [`Promise<DatasetContent>`](../typedefs/datasetcontent)
-Returns items in the dataset based on the provided parameters. The returned object has the following structure:
-
-```javascript
-{
- items, // Array|String|Buffer based on chosen format parameter.
- total, // Number
- offset, // Number
- count, // Number
- limit, // Number
-}
-```
+Returns {DatasetContent} object holding the items in the dataset based on the provided parameters.
**NOTE**: If using dataset with local disk storage, the `format` option must be `json` and the following options are not supported: `unwind`,
`disableBodyParser`, `attachment`, `bom` and `simplified`. If you try to use them, you will receive an error.
@@ -250,7 +262,7 @@ The function returns the same object as the Apify API Client's
<a name="Dataset+forEach"></a>
-## `dataset.forEach(iteratee, [options], [index])` ⇒ `Promise`
+## `dataset.forEach(iteratee, [options], [index])` ⇒ `Promise<void>`
Iterates over dataset items, yielding each in turn to an `iteratee` function. Each invocation of `iteratee` is called with two arguments:
`(item, index)`.
@@ -275,7 +287,7 @@ await dataset.forEach(async (item, index) => {
</thead>
<tbody>
<tr>
-<td><code>iteratee</code></td><td><code>function</code></td><td></td>
+<td><code>iteratee</code></td><td><code><a href="../typedefs/datasetconsumer">DatasetConsumer</a></code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>A function that is called for every item in the dataset.</p>
@@ -309,7 +321,7 @@ await dataset.forEach(async (item, index) => {
</table>
<a name="Dataset+map"></a>
-## `dataset.map(iteratee, options)` ⇒ `Promise<Array>`
+## `dataset.map(iteratee, options)` ⇒ `Promise<Array<T>>`
Produces a new array of values by mapping each value in list through a transformation function `iteratee()`. Each invocation of `iteratee()` is called
with two arguments: `(element, index)`.
@@ -324,7 +336,7 @@ If `iteratee` returns a `Promise` then it's awaited before a next call.
</thead>
<tbody>
<tr>
-<td><code>iteratee</code></td><td><code>function</code></td><td></td>
+<td><code>iteratee</code></td><td><code><a href="../typedefs/datasetmapper">DatasetMapper</a></code></td><td></td>
</tr>
<tr>
<td colspan="3"></td></tr><tr>
@@ -352,7 +364,7 @@ If `iteratee` returns a `Promise` then it's awaited before a next call.
</table>
<a name="Dataset+reduce"></a>
-## `dataset.reduce(iteratee, memo, options)` ⇒ `Promise<*>`
+## `dataset.reduce(iteratee, memo, options)` ⇒ `Promise<T>`
Reduces a list of values down to a single value.
@@ -372,11 +384,11 @@ If `iteratee()` returns a `Promise` then it's awaited before a next call.
</thead>
<tbody>
<tr>
-<td><code>iteratee</code></td><td><code>function</code></td><td></td>
+<td><code>iteratee</code></td><td><code><a href="../typedefs/datasetreducer">DatasetReducer</a></code></td><td></td>
</tr>
<tr>
<td colspan="3"></td></tr><tr>
-<td><code>memo</code></td><td><code>*</code></td><td></td>
+<td><code>memo</code></td><td><code>T</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Initial state of the reduction.</p>
@@ -405,6 +417,6 @@ If `iteratee()` returns a `Promise` then it's awaited before a next call.
</table>
<a name="Dataset+drop"></a>
-## `dataset.drop()` ⇒ `Promise`
+## `dataset.drop()` ⇒ `Promise<void>`
Removes the dataset either from the Apify cloud storage or from the local directory, depending on the mode of operation.
diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.md
index f0c6c7805..a528f447d 100644
--- a/docs/api/KeyValueStore.md
+++ b/docs/api/KeyValueStore.md
@@ -62,12 +62,34 @@ await store.drop();
```
- [KeyValueStore](keyvaluestore)
+ - [`new exports.KeyValueStore(storeId, storeName)`](#new_KeyValueStore_new)
- [`.getValue(key)`](#KeyValueStore+getValue) ⇒ `Promise<(Object|String|Buffer)>`
- - [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ `Promise`
- - [`.drop()`](#KeyValueStore+drop) ⇒ `Promise`
+ - [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ `Promise<void>`
+ - [`.drop()`](#KeyValueStore+drop) ⇒ `Promise<void>`
- [`.getPublicUrl(key)`](#KeyValueStore+getPublicUrl) ⇒ `string`
- - [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ `Promise`
+ - [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ `Promise<void>`
+<a name="new_KeyValueStore_new"></a>
+
+## `new exports.KeyValueStore(storeId, storeName)`
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>storeId</code></td><td><code>string</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>storeName</code></td><td><code>string</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
<a name="KeyValueStore+getValue"></a>
## `keyValueStore.getValue(key)` ⇒ `Promise<(Object|String|Buffer)>`
@@ -110,7 +132,7 @@ const buffer = await store.getValue('screenshot1.png');
</table>
<a name="KeyValueStore+setValue"></a>
-## `keyValueStore.setValue(key, value, [options])` ⇒ `Promise`
+## `keyValueStore.setValue(key, value, [options])` ⇒ `Promise<void>`
Saves or deletes a record in the key-value store. The function returns a promise that resolves once the record has been saved or deleted.
@@ -179,7 +201,7 @@ stored!
</table>
<a name="KeyValueStore+drop"></a>
-## `keyValueStore.drop()` ⇒ `Promise`
+## `keyValueStore.drop()` ⇒ `Promise<void>`
Removes the key-value store either from the Apify cloud storage or from the local directory, depending on the mode of operation.
@@ -204,7 +226,7 @@ Returns a URL for the given key that may be used to publicly access the value in
</table>
<a name="KeyValueStore+forEachKey"></a>
-## `keyValueStore.forEachKey(iteratee, [options])` ⇒ `Promise`
+## `keyValueStore.forEachKey(iteratee, [options])` ⇒ `Promise<void>`
Iterates over key value store keys, yielding each in turn to an `iteratee` function. Each invocation of `iteratee` is called with three arguments:
`(key, index, info)`, where `key` is the record key, `index` is a zero-based index of the key in the current iteration (regardless of
@@ -230,7 +252,7 @@ await keyValueStore.forEachKey(async (key, index, info) => {
</thead>
<tbody>
<tr>
-<td><code>iteratee</code></td><td><code>function</code></td>
+<td><code>iteratee</code></td><td><code><a href="../typedefs/keyconsumer">KeyConsumer</a></code></td>
</tr>
<tr>
<td colspan="3"><p>A function that is called for every key in the key value store.</p>
diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.md
index 391f65101..d065d6660 100644
--- a/docs/api/PseudoUrl.md
+++ b/docs/api/PseudoUrl.md
@@ -76,7 +76,7 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M
Using a <code>RegExp</code> instance enables more granular control,
such as making the matching case sensitive.</p>
</td></tr><tr>
-<td><code>requestTemplate</code></td><td><code>Object</code></td>
+<td><code>requestTemplate</code></td><td><code><a href="../typedefs/requestoptions">RequestOptions</a></code></td>
</tr>
<tr>
<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index dbc9535dc..184df6cb5 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -63,7 +63,7 @@ await crawler.run();
- [PuppeteerCrawler](puppeteercrawler)
- [`new PuppeteerCrawler(options)`](#new_PuppeteerCrawler_new)
- - [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise`
+ - [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise<void>`
<a name="new_PuppeteerCrawler_new"></a>
@@ -72,167 +72,20 @@ await crawler.run();
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td><code>Object</code></td><td></td>
+<td><code>options</code></td><td><code><a href="../typedefs/puppeteercrawleroptions">PuppeteerCrawlerOptions</a></code></td>
</tr>
<tr>
<td colspan="3"><p>All <code>PuppeteerCrawler</code> parameters are passed
via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>options.handlePageFunction</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Function that is called to process each request.
- It is passed an object with the following fields:</p>
-<pre><code>{
- request: Request,
- response: Response,
- page: Page,
- puppeteerPool: PuppeteerPool,
- autoscaledPool: AutoscaledPool,
- session: Session,
-}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
- <code>page</code> is an instance of the <code>Puppeteer</code>
- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>.
- <code>response</code> is an instance of the <code>Puppeteer</code>
- <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
- which is the main resource response as returned by <code>page.goto(request.url)</code>.
- <code>puppeteerPool</code> is an instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>
-<p> The function must return a promise, which is then awaited by the crawler.</p>
-<p> If the function throws an exception, the crawler will try to re-crawl the
- request later, up to <code>option.maxRequestRetries</code> times.
- If all the retries fail, the crawler calls the function
- provided to the <code>options.handleFailedRequestFunction</code> parameter.
- To make this work, you should <strong>always</strong>
- let your function throw exceptions rather than catch them.
- The exceptions are logged to the request using the
- <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
-</td></tr><tr>
-<td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Static list of URLs to be processed.
- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
-</td></tr><tr>
-<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
-</td></tr><tr>
-<td><code>[options.handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Timeout in which the function passed as <code>options.handlePageFunction</code> needs to finish, in seconds.</p>
-</td></tr><tr>
-<td><code>[options.gotoFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer&#39;s
- <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,
- i.e. a <code>Promise</code> resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.</p>
-<p> This is useful if you need to extend the page load timeout or select different criteria
- to determine that the navigation succeeded.</p>
-<p> Note that a single page object is only used to process a single request and it is closed afterwards.</p>
-<p> By default, the function invokes <a href="puppeteer#puppeteer.gotoExtended"><code>Apify.utils.puppeteer.gotoExtended()</code></a> with a timeout of 60 seconds.
- For details, see source code on
- <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.</p>
-</td></tr><tr>
-<td><code>[options.gotoTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Timeout in which page navigation needs to finish, in seconds. When <code>options.gotoFunction()</code> is used and thus the default
- function is overridden, this timeout will not be used and needs to be configured in the new <code>gotoFunction()</code>.</p>
-</td></tr><tr>
-<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- request: Request,
- error: Error,
-}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
- represents the last error thrown during processing of the request.</p>
-<p> See
- <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
- for the default implementation of this function.</p>
-</td></tr><tr>
-<td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how many times the request is retried if either <code>handlePageFunction()</code> or <code>gotoFunction()</code> fails.</p>
-</td></tr><tr>
-<td><code>[options.maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
- Always set this value in order to prevent infinite loops in misconfigured crawlers.
- Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
-</td></tr><tr>
-<td><code>[options.puppeteerPoolOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom options passed to the underlying <a href="puppeteerpool"><code>PuppeteerPool</code></a> constructor.
- You can tweak those to fine-tune browser management.</p>
-</td></tr><tr>
-<td><code>[options.launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.
- Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option.
- See source code on
- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
- for default behavior.</p>
-</td></tr><tr>
-<td><code>[options.launchPuppeteerOptions]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Options used by <a href="apify#module_Apify.launchPuppeteer"><code>Apify.launchPuppeteer()</code></a> to start new Puppeteer instances.
- Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option. See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
-</td></tr><tr>
-<td><code>[options.autoscaledPoolOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> instance constructor.
- Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options
- are provided by <code>PuppeteerCrawler</code> and should not be overridden.</p>
-</td></tr><tr>
-<td><code>[options.minConcurrency]</code></td><td><code>Object</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
-<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
- If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
-</td></tr><tr>
-<td><code>[options.maxConcurrency]</code></td><td><code>Object</code></td><td><code>1000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
-</td></tr><tr>
-<td><code>[options.useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
- It also marks Session as bad after a request timeout.</p>
-</td></tr><tr>
-<td><code>[options.sessionPoolOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>
-</td></tr><tr>
-<td><code>[options.persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>
</td></tr></tbody>
</table>
<a name="PuppeteerCrawler+run"></a>
-## `puppeteerCrawler.run()` ⇒ `Promise`
+## `puppeteerCrawler.run()` ⇒ `Promise<void>`
Runs the crawler. Returns promise that gets resolved once all the requests got processed.
diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.md
index faebf0e93..86cfe2825 100644
--- a/docs/api/PuppeteerPool.md
+++ b/docs/api/PuppeteerPool.md
@@ -41,10 +41,10 @@ await puppeteerPool.destroy();
- [PuppeteerPool](puppeteerpool)
- [`new PuppeteerPool([options])`](#new_PuppeteerPool_new)
- [`.newPage()`](#PuppeteerPool+newPage) ⇒ `Promise<Page>`
- - [`.destroy()`](#PuppeteerPool+destroy) ⇒ `Promise`
- - [`.retire(browser)`](#PuppeteerPool+retire) ⇒ `Promise`
- - [`.recyclePage(page)`](#PuppeteerPool+recyclePage) ⇒ `Promise`
- - [`.serveLiveViewSnapshot(page)`](#PuppeteerPool+serveLiveViewSnapshot) ⇒ `Promise`
+ - [`.destroy()`](#PuppeteerPool+destroy) ⇒ `Promise<void>`
+ - [`.retire(browser)`](#PuppeteerPool+retire) ⇒ `Promise<void>`
+ - [`.recyclePage(page)`](#PuppeteerPool+recyclePage) ⇒ `Promise<void>`
+ - [`.serveLiveViewSnapshot(page)`](#PuppeteerPool+serveLiveViewSnapshot) ⇒ `Promise<void>`
<a name="new_PuppeteerPool_new"></a>
@@ -53,98 +53,16 @@ await puppeteerPool.destroy();
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>[options]</code></td><td><code>Object</code></td><td></td>
+<td><code>[options]</code></td><td><code><a href="../typedefs/puppeteerpooloptions">PuppeteerPoolOptions</a></code></td>
</tr>
<tr>
<td colspan="3"><p>All <code>PuppeteerPool</code> parameters are passed
via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>[options.useLiveView]</code></td><td><code>boolean</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Enables the use of a preconfigured <a href="liveviewserver"><code>LiveViewServer</code></a> that serves snapshots
- just before a page would be recycled by <code>PuppeteerPool</code>. If there are no clients
- connected, it has close to zero impact on performance.</p>
-</td></tr><tr>
-<td><code>[options.maxOpenPagesPerInstance]</code></td><td><code>number</code></td><td><code>50</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.</p>
-</td></tr><tr>
-<td><code>[options.retireInstanceAfterRequestCount]</code></td><td><code>number</code></td><td><code>100</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of requests that can be processed by a single browser instance.
- After the limit is reached, the browser is retired and new requests are
- handled by a new browser instance.</p>
-</td></tr><tr>
-<td><code>[options.puppeteerOperationTimeoutSecs]</code></td><td><code>number</code></td><td><code>15</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>All browser management operations such as launching a new browser, opening a new page
- or closing a page will timeout after the set number of seconds and the connected
- browser will be retired.</p>
-</td></tr><tr>
-<td><code>[options.instanceKillerIntervalSecs]</code></td><td><code>number</code></td><td><code>60</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how often are the open Puppeteer instances checked whether they can be closed.</p>
-</td></tr><tr>
-<td><code>[options.killInstanceAfterSecs]</code></td><td><code>number</code></td><td><code>300</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>When Puppeteer instance reaches the <code>options.retireInstanceAfterRequestCount</code> limit then
- it is considered retired and no more tabs will be opened. After the last tab is closed the
- whole browser is closed too. This parameter defines a time limit between the last tab was opened and
- before the browser is closed even if there are pending open tabs.</p>
-</td></tr><tr>
-<td><code>[options.launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.
- The function must return a promise resolving to
- <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser"><code>Browser</code></a> instance.
- See the source code on
- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
- for the default implementation.</p>
-</td></tr><tr>
-<td><code>[options.launchPuppeteerOptions]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Options used by <code>Apify.launchPuppeteer()</code> to start new Puppeteer instances.
- See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
-</td></tr><tr>
-<td><code>[options.recycleDiskCache]</code></td><td><code>boolean</code></td><td><code>false</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Enables recycling of disk cache directories by Chrome instances.
- When a browser instance is closed, its disk cache directory is not deleted but it&#39;s used by a newly opened browser instance.
- This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
- Note that the new browser starts with empty cookies, local storage etc. so this setting doesn&#39;t affect anonymity of your crawler.</p>
-<p> Beware that the disk cache directories can consume a lot of disk space.
- To limit the space consumed, you can pass the <code>--disk-cache-size=X</code> argument to <code>options.launchPuppeteerOptions.args</code>,
- where <code>X</code> is the approximate maximum number of bytes for disk cache.</p>
-<p> Do not use the <code>options.recycleDiskCache</code> setting together with <code>--disk-cache-dir</code>
- argument in <code>options.launchPuppeteerOptions.args</code>, the behavior is undefined.</p>
-</td></tr><tr>
-<td><code>[options.useIncognitoPages]</code></td><td><code>boolean</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>With this option selected, all pages will be opened in a new incognito browser context, which means
- that they will not share cookies or cache and their resources will not be throttled by one another.</p>
-</td></tr><tr>
-<td><code>[options.proxyUrls]</code></td><td><code>Array<string></code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>PuppeteerPool</code> instance.
- The provided custom proxies&#39; order will be randomized and the resulting list rotated.
- Custom proxies are not compatible with Apify Proxy and an attempt to use both
- configuration options will cause an error to be thrown on startup.</p>
</td></tr></tbody>
</table>
<a name="PuppeteerPool+newPage"></a>
@@ -158,13 +76,13 @@ To spawn a new browser tab for each page, set the `reusePages` constructor optio
<a name="PuppeteerPool+destroy"></a>
-## `puppeteerPool.destroy()` ⇒ `Promise`
+## `puppeteerPool.destroy()` ⇒ `Promise<void>`
Closes all open browsers.
<a name="PuppeteerPool+retire"></a>
-## `puppeteerPool.retire(browser)` ⇒ `Promise`
+## `puppeteerPool.retire(browser)` ⇒ `Promise<void>`
Manually retires a Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-browser" target="_blank"><code>Browser</code></a> instance
from the pool. The browser will continue to process open pages so that they may gracefully finish. This is unlike `browser.close()` which will
@@ -185,7 +103,7 @@ forcibly terminate the browser and all open pages will be closed.
</table>
<a name="PuppeteerPool+recyclePage"></a>
-## `puppeteerPool.recyclePage(page)` ⇒ `Promise`
+## `puppeteerPool.recyclePage(page)` ⇒ `Promise<void>`
Closes the page, unless the `reuseTabs` option is set to true. Then it would only flag the page for a future reuse, without actually closing it.
@@ -207,7 +125,7 @@ closing the page or flagging it for reuse.
</table>
<a name="PuppeteerPool+serveLiveViewSnapshot"></a>
-## `puppeteerPool.serveLiveViewSnapshot(page)` ⇒ `Promise`
+## `puppeteerPool.serveLiveViewSnapshot(page)` ⇒ `Promise<void>`
Tells the connected LiveViewServer to serve a snapshot when available.
diff --git a/docs/api/Request.md b/docs/api/Request.md
index 808308a20..6dd5765a8 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -119,74 +119,15 @@ const foo = request.userData.foo;
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td><code>object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>All <code>Request</code> parameters are passed
- via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>options.url</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>URL of the web page to crawl. It must be a non-empty string.</p>
-</td></tr><tr>
-<td><code>[options.uniqueKey]</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A unique key identifying the request.
-Two requests with the same <code>uniqueKey</code> are considered as pointing to the same URL.</p>
-<p>If <code>uniqueKey</code> is not provided, then it is automatically generated by normalizing the URL.
-For example, the URL of <code>HTTP://www.EXAMPLE.com/something/</code> will produce the <code>uniqueKey</code>
-of <code>http://www.example.com/something</code>.</p>
-<p>The <code>keepUrlFragment</code> option determines whether URL hash fragment is included in the <code>uniqueKey</code> or not.</p>
-<p>The <code>useExtendedUniqueKey</code> options determines whether method and payload are included in the <code>uniqueKey</code>,
-producing a <code>uniqueKey</code> in the following format: <code>METHOD(payloadHash):normalizedUrl</code>. This is useful
-when requests point to the same URL, but with different methods and payloads. For example: form submits.</p>
-<p>Pass an arbitrary non-empty text value to the <code>uniqueKey</code> property
-to override the default behavior and specify which URLs shall be considered equal.</p>
-</td></tr><tr>
-<td><code>[options.method]</code></td><td><code>String</code></td><td><code>&#x27;GET&#x27;</code></td>
-</tr>
-<tr>
-<td colspan="3"></td></tr><tr>
-<td><code>[options.payload]</code></td><td><code>String</code> | <code>Buffer</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p>
-</td></tr><tr>
-<td><code>[options.headers]</code></td><td><code>Object</code></td><td><code>{}</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>HTTP headers in the following format:</p>
-<pre><code> {
- Accept: &#39;text/html&#39;,
- &#39;Content-Type&#39;: &#39;application/json&#39;
- }</code></pre></td></tr><tr>
-<td><code>[options.userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom user data assigned to the request. Use this to save any request related data to the
- request&#39;s scope, keeping them accessible on retries, failures etc.</p>
-</td></tr><tr>
-<td><code>[options.keepUrlFragment]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If <code>false</code> then the hash part of a URL is removed when computing the <code>uniqueKey</code> property.
- For example, this causes the <code>http://www.example.com#foo</code> and <code>http://www.example.com#bar</code> URLs
- to have the same <code>uniqueKey</code> of <code>http://www.example.com</code> and thus the URLs are considered equal.
- Note that this option only has an effect if <code>uniqueKey</code> is not set.</p>
-</td></tr><tr>
-<td><code>[options.useExtendedUniqueKey]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+<td><code>options</code></td><td><code><a href="../typedefs/requestoptions">RequestOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>If <code>true</code> then the <code>uniqueKey</code> is computed not only from the URL, but also from the method and payload
- properties. This is useful when making requests to the same URL that are differentiated by method
- or payload, such as form submit navigations in browsers.</p>
+<td colspan="3"><p><code>Request</code> parameters including the URL, HTTP method and headers, and others.</p>
</td></tr></tbody>
</table>
<a name="Request+pushErrorMessage"></a>
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index 830cc1032..2c811590a 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -65,14 +65,14 @@ await requestList.reclaimRequest(request2);
- [RequestList](requestlist)
- [`new exports.RequestList(options)`](#new_RequestList_new)
- - [`.initialize()`](#RequestList+initialize) ⇒ `Promise`
- - [`.persistState()`](#RequestList+persistState) ⇒ `Promise`
- - [`.getState()`](#RequestList+getState) ⇒ `Object`
+ - [`.initialize()`](#RequestList+initialize) ⇒ `Promise<void>`
+ - [`.persistState()`](#RequestList+persistState) ⇒ `Promise<void>`
+ - [`.getState()`](#RequestList+getState) ⇒ [`RequestListState`](../typedefs/requestliststate)
- [`.isEmpty()`](#RequestList+isEmpty) ⇒ `Promise<Boolean>`
- [`.isFinished()`](#RequestList+isFinished) ⇒ `Promise<Boolean>`
- [`.fetchNextRequest()`](#RequestList+fetchNextRequest) ⇒ [`Promise<Request>`](request)
- - [`.markRequestHandled(request)`](#RequestList+markRequestHandled) ⇒ `Promise`
- - [`.reclaimRequest(request)`](#RequestList+reclaimRequest) ⇒ `Promise`
+ - [`.markRequestHandled(request)`](#RequestList+markRequestHandled) ⇒ `Promise<void>`
+ - [`.reclaimRequest(request)`](#RequestList+reclaimRequest) ⇒ `Promise<void>`
- [`.length()`](#RequestList+length) ⇒ `Number`
- [`.handledCount()`](#RequestList+handledCount) ⇒ `Number`
@@ -83,94 +83,27 @@ await requestList.reclaimRequest(request2);
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>All <code>RequestList</code> parameters are passed
- via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>options.sources</code></td><td><code>Array</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>. It can be either an array of plain objects that
- define the <code>url</code> property, or an array of instances of the <a href="request"><code>Request</code></a> class.
- Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,
- which will instruct <code>RequestList</code> to download the source URLs from a given remote location.
- The URLs will be parsed from the received response.</p>
-<pre><code>[
- // One URL
- { method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
- // Batch import of URLs from a file hosted on the web
- { method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
- // Batch import combined with regex.
- { method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39;, regex: /https:\/\/example.com\/.+/ },
-]</code></pre></td></tr><tr>
-<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Identifies the key in the default key-value store under which <code>RequestList</code> periodically stores its
- state (i.e. which URLs were crawled and which not).
- If the actor is restarted, <code>RequestList</code> will read the state
- and continue where it left off.</p>
-<p> If <code>persistStateKey</code> is not set, <code>RequestList</code> will always start from the beginning,
- and all the source URLs will be crawled again.</p>
-</td></tr><tr>
-<td><code>[options.persistSourcesKey]</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Identifies the key in the default key-value store under which the <code>RequestList</code> persists its
- sources (i.e. the lists of URLs) during the <a href="#RequestList+initialize"><code>initialize</code></a> call.
- This is necessary if <code>persistStateKey</code> is set and the source URLs might potentially change,
- to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.</p>
-<p> If <code>persistSourcesKey</code> is not set, <a href="#RequestList+initialize"><code>initialize</code></a> will always fetch the sources
- from their origin, check that they are consistent with the restored state (if any)
- and throw an error if they are not.</p>
-</td></tr><tr>
-<td><code>[options.state]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>The state object that the <code>RequestList</code> will be initialized from.
- It is in the form as returned by <code>RequestList.getState()</code>, such as follows:</p>
-<pre><code>{
- nextIndex: 5,
- nextUniqueKey: &#39;unique-key-5&#39;
- inProgress: {
- &#39;unique-key-1&#39;: true,
- &#39;unique-key-4&#39;: true,
- },
-}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
- is to use the <code>stateKeyPrefix</code> parameter instead.</p>
-</td></tr><tr>
-<td><code>[options.keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+<td><code>options</code></td><td><code><a href="../typedefs/requestlistoptions">RequestListOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>By default, <code>RequestList</code> will deduplicate the provided URLs. Default deduplication is based
- on the <code>uniqueKey</code> property of passed source <a href="request"><code>Request</code></a> objects.</p>
-<p> If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
- In any case, only one request per <code>uniqueKey</code> is added to the <code>RequestList</code> resulting in removal
- of duplicate URLs / unique keys.</p>
-<p> Setting <code>keepDuplicateUrls</code> to <code>true</code> will append an additional identifier to the <code>uniqueKey</code>
- of each request that does not already include a <code>uniqueKey</code>. Therefore, duplicate
- URLs will be kept in the list. It does not protect the user from having duplicates in user set
- <code>uniqueKey</code>s however. It is the user&#39;s responsibility to ensure uniqueness of their unique keys
- if they wish to keep more than just a single copy in the <code>RequestList</code>.</p>
+<td colspan="3"><p>All <code>RequestList</code> configuration options</p>
</td></tr></tbody>
</table>
<a name="RequestList+initialize"></a>
-## `requestList.initialize()` ⇒ `Promise`
+## `requestList.initialize()` ⇒ `Promise<void>`
Loads all remote sources of URLs and potentially starts periodic state persistence. This function must be called before you can start using the
instance in a meaningful way.
<a name="RequestList+persistState"></a>
-## `requestList.persistState()` ⇒ `Promise`
+## `requestList.persistState()` ⇒ `Promise<void>`
Persists the current state of the `RequestList` into the default [`KeyValueStore`](keyvaluestore). The state is persisted automatically in regular
intervals, but calling this method manually is useful in cases where you want to have the most current state available after you pause or stop
@@ -178,7 +111,7 @@ fetching its requests. For example after you pause or abort a crawl. Or just bef
<a name="RequestList+getState"></a>
-## `requestList.getState()` ⇒ `Object`
+## `requestList.getState()` ⇒ [`RequestListState`](../typedefs/requestliststate)
Returns an object representing the internal state of the `RequestList` instance. Note that the object's fields can change in future releases.
@@ -206,7 +139,7 @@ The function's `Promise` resolves to `null` if there are no more requests to pro
<a name="RequestList+markRequestHandled"></a>
-## `requestList.markRequestHandled(request)` ⇒ `Promise`
+## `requestList.markRequestHandled(request)` ⇒ `Promise<void>`
Marks request as handled after successful processing.
@@ -225,7 +158,7 @@ Marks request as handled after successful processing.
</table>
<a name="RequestList+reclaimRequest"></a>
-## `requestList.reclaimRequest(request)` ⇒ `Promise`
+## `requestList.reclaimRequest(request)` ⇒ `Promise<void>`
Reclaims request to the list if its processing failed. The request will become available in the next `this.fetchNextRequest()`.
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index aa56fd51a..bbd3f6f78 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -95,7 +95,7 @@ function.
</thead>
<tbody>
<tr>
-<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td><td></td>
+<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code><a href="../typedefs/requestoptions">RequestOptions</a></code></td><td></td>
</tr>
<tr>
<td colspan="3"><p><a href="request"><code>Request</code></a> object or vanilla object with request data.
diff --git a/docs/api/Session.md b/docs/api/Session.md
index 8fe4a4dc9..d530c7d92 100644
--- a/docs/api/Session.md
+++ b/docs/api/Session.md
@@ -10,96 +10,46 @@ a specific user, with its own cookies, IP (via proxy) and potentially a unique b
user data for example some authorization tokens and specific headers in general.
- [Session](session)
- - [`new exports.Session()`](#new_Session_new)
+ - [`new exports.Session(options)`](#new_Session_new)
+ - [`.cookieJar`](#Session+cookieJar) : `CookieJar`
- [`.isBlocked()`](#Session+isBlocked) ⇒ `boolean`
- [`.isExpired()`](#Session+isExpired) ⇒ `boolean`
- [`.isMaxUsageCountReached()`](#Session+isMaxUsageCountReached) ⇒ `boolean`
- [`.isUsable()`](#Session+isUsable) ⇒ `boolean`
- [`.markGood()`](#Session+markGood)
- - [`.getState()`](#Session+getState) ⇒ `Object`
+ - [`.getState()`](#Session+getState) ⇒ [`SessionState`](../typedefs/sessionstate)
- [`.retire()`](#Session+retire)
- [`.markBad()`](#Session+markBad)
- [`.retireOnBlockedStatusCodes(statusCode, blockedStatusCodes)`](#Session+retireOnBlockedStatusCodes) ⇒ `boolean`
- [`.setCookiesFromResponse(response)`](#Session+setCookiesFromResponse)
+ - [`.putPuppeteerCookies(puppeteerCookies, url)`](#Session+putPuppeteerCookies)
- [`.setPuppeteerCookies(cookies, url)`](#Session+setPuppeteerCookies)
- - [`.getPuppeteerCookies(url)`](#Session+getPuppeteerCookies) ⇒ `Array<Object>`
+ - [`.getPuppeteerCookies(url)`](#Session+getPuppeteerCookies) ⇒ `Array<PuppeteerCookie>`
- [`.getCookieString(url)`](#Session+getCookieString) ⇒ `String`
<a name="new_Session_new"></a>
-## `new exports.Session()`
+## `new exports.Session(options)`
Session configuration.
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>[options.id]</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Id of session used for generating fingerprints. It is used as proxy session name.</p>
-</td></tr><tr>
-<td><code>[options.maxAgeSecs]</code></td><td><code>Number</code></td><td><code>3000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Number of seconds after which the session is considered as expired.</p>
-</td></tr><tr>
-<td><code>options.userData</code></td><td><code>Object</code></td><td></td>
+<td><code>options</code></td><td><code><a href="../typedefs/sessionoptions">SessionOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>Object where custom user data can be stored. For example custom headers.</p>
-</td></tr><tr>
-<td><code>[options.maxErrorScore]</code></td><td><code>number</code></td><td><code>3</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of marking session as blocked usage.
-If the <code>errorScore</code> reaches the <code>maxErrorScore</code> session is marked as block and it is thrown away.
-It starts at 0. Calling the <code>markBad</code> function increases the <code>errorScore</code> by 1.
-Calling the <code>markGood</code> will decrease the <code>errorScore</code> by <code>errorScoreDecrement</code></p>
-</td></tr><tr>
-<td><code>[options.errorScoreDecrement]</code></td><td><code>number</code></td><td><code>0.5</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>It is used for healing the session.
-For example: if your session is marked bad two times, but it is successful on the third attempt it&#39;s errorScore is decremented by this number.</p>
-</td></tr><tr>
-<td><code>options.createdAt</code></td><td><code>Date</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Date of creation.</p>
-</td></tr><tr>
-<td><code>options.expiredAt</code></td><td><code>Date</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Date of expiration.</p>
-</td></tr><tr>
-<td><code>[options.usageCount]</code></td><td><code>Number</code></td><td><code>0</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how many times the session has been used.</p>
-</td></tr><tr>
-<td><code>[options.errorCount]</code></td><td><code>Number</code></td><td><code>0</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how many times the session is marked bad.</p>
-</td></tr><tr>
-<td><code>[options.maxUsageCount]</code></td><td><code>Number</code></td><td><code>50</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Session should be used only a limited amount of times.
-This number indicates how many times the session is going to be used, before it is thrown away.</p>
-</td></tr><tr>
-<td><code>options.sessionPool</code></td><td><code>EventEmitter</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>SessionPool instance. Session will emit the <code>sessionRetired</code> event on this instance.</p>
-</td></tr></tbody>
+</tr></tbody>
</table>
+<a name="Session+cookieJar"></a>
+
+## `session.cookieJar` : `CookieJar`
+
<a name="Session+isBlocked"></a>
## `session.isBlocked()` ⇒ `boolean`
@@ -135,11 +85,11 @@ This method should be called after a successful session usage. It increases `usa
<a name="Session+getState"></a>
-## `session.getState()` ⇒ `Object`
+## `session.getState()` ⇒ [`SessionState`](../typedefs/sessionstate)
Gets session state for persistence in KeyValueStore.
-**Returns**: `Object` - represents session internal state.
+**Returns**: [`SessionState`](../typedefs/sessionstate) - represents session internal state.
<a name="Session+retire"></a>
## `session.retire()`
@@ -190,16 +140,41 @@ Sets cookies from response to the cookieJar. Parses cookies from `set-cookie` he
<table>
<thead>
<tr>
-<th>Param</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>response</code></td>
+<td><code>response</code></td><td><code>Object</code></td>
</tr>
<tr>
</tr></tbody>
</table>
+<a name="Session+putPuppeteerCookies"></a>
+
+## `session.putPuppeteerCookies(puppeteerCookies, url)`
+
+Persists puppeteer cookies to session for reuse.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>puppeteerCookies</code></td><td><code>PuppeteerCookie</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>cookie from puppeteer <code>page.cookies</code> method.</p>
+</td></tr><tr>
+<td><code>url</code></td><td><code>String</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Loaded url from page function.</p>
+</td></tr></tbody>
+</table>
<a name="Session+setPuppeteerCookies"></a>
## `session.setPuppeteerCookies(cookies, url)`
@@ -215,7 +190,7 @@ cookie compatible.
</thead>
<tbody>
<tr>
-<td><code>cookies</code></td><td><code>Array<Object></code></td>
+<td><code>cookies</code></td><td><code>Array<PuppeteerCookie></code></td>
</tr>
<tr>
</tr><tr>
@@ -226,7 +201,7 @@ cookie compatible.
</table>
<a name="Session+getPuppeteerCookies"></a>
-## `session.getPuppeteerCookies(url)` ⇒ `Array<Object>`
+## `session.getPuppeteerCookies(url)` ⇒ `Array<PuppeteerCookie>`
Gets cookies in puppeteer ready to be used with `page.setCookie`.
@@ -255,12 +230,12 @@ Wrapper around `tough-cookie` Cookie jar `getCookieString` method.
<table>
<thead>
<tr>
-<th>Param</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>url</code></td>
+<td><code>url</code></td><td><code>String</code></td>
</tr>
<tr>
</tr></tbody>
diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.md
index d60ce8d47..08ef31400 100644
--- a/docs/api/SessionPool.md
+++ b/docs/api/SessionPool.md
@@ -66,41 +66,15 @@ Session pool configuration.
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>[options]</code></td><td></td><td></td>
+<td><code>[options]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"></td></tr><tr>
-<td><code>[options.maxPoolSize]</code></td><td><code>Number</code></td><td><code>1000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum size of the pool.
-Indicates how many sessions are rotated.</p>
-</td></tr><tr>
-<td><code>[options.sessionOptions]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>The <a href="session#new_SessionPool_new"><code>new Session</code></a> options</p>
-</td></tr><tr>
-<td><code>[options.persistStateKeyValueStoreId]</code></td><td><code>String</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Name or Id of <code>KeyValueStore</code> where is the <code>SessionPool</code> state stored.</p>
-</td></tr><tr>
-<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td><code>&quot;SESSION_POOL_STATE&quot;</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Session pool persists it&#39;s state under this key in Key value store.</p>
-</td></tr><tr>
-<td><code>[options.createSessionFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Custom function that should return <code>Session</code> instance.
-Function receives <code>SessionPool</code> instance as a parameter</p>
+<td colspan="3"><p>All <code>SessionPool</code> configuration options.</p>
</td></tr></tbody>
</table>
<a name="SessionPool+usableSessionsCount"></a>
diff --git a/docs/api/Snapshotter.md b/docs/api/Snapshotter.md
index f029686b7..ed69fc0bd 100644
--- a/docs/api/Snapshotter.md
+++ b/docs/api/Snapshotter.md
@@ -39,73 +39,15 @@ set interval.
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>[options]</code></td><td><code>Object</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>All <code>Snapshotter</code> parameters are passed
- via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>[options.eventLoopSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>0.5</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the interval of measuring the event loop response time.</p>
-</td></tr><tr>
-<td><code>[options.clientSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the interval of checking the current state
- of the remote API client.</p>
-</td></tr><tr>
-<td><code>[options.maxBlockedMillis]</code></td><td><code>Number</code></td><td><code>50</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum allowed delay of the event loop in milliseconds.
- Exceeding this limit overloads the event loop.</p>
-</td></tr><tr>
-<td><code>[options.cpuSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the interval of measuring CPU usage.
- This is only used when running locally. On the Apify platform,
- the statistics are provided externally at a fixed interval.</p>
-</td></tr><tr>
-<td><code>[options.maxUsedCpuRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the maximum usage of CPU.
- Exceeding this limit overloads the CPU.</p>
-</td></tr><tr>
-<td><code>[options.memorySnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the interval of measuring memory consumption.
- This is only used when running locally. On the Apify platform,
- the statistics are provided externally at a fixed interval.
- The measurement itself is resource intensive (25 - 50ms async).
- Therefore, setting this interval below 1 second is not recommended.</p>
-</td></tr><tr>
-<td><code>[options.maxUsedMemoryRatio]</code></td><td><code>Number</code></td><td><code>0.7</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the maximum ratio of total memory that can be used.
- Exceeding this limit overloads the memory.</p>
-</td></tr><tr>
-<td><code>[options.maxClientErrors]</code></td><td><code>Number</code></td><td><code>1</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines the maximum number of new rate limit errors within
- the given interval.</p>
-</td></tr><tr>
-<td><code>[options.snapshotHistorySecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+<td><code>[options]</code></td><td><code><a href="../typedefs/snapshotteroptions">SnapshotterOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>Sets the interval in seconds for which a history of resource snapshots
- will be kept. Increasing this to very high numbers will affect performance.</p>
+<td colspan="3"><p>All <code>Snapshotter</code> configuration options.</p>
</td></tr></tbody>
</table>
<a name="Snapshotter+start"></a>
diff --git a/docs/api/SystemStatus.md b/docs/api/SystemStatus.md
index d6e611dc8..599de99d0 100644
--- a/docs/api/SystemStatus.md
+++ b/docs/api/SystemStatus.md
@@ -19,7 +19,7 @@ full snapshot history available in the [`Snapshotter`](snapshotter) instance.
- [SystemStatus](systemstatus)
- [`new SystemStatus([options])`](#new_SystemStatus_new)
- - [`.getCurrentStatus()`](#SystemStatus+getCurrentStatus) ⇒ `Object`
+ - [`.getCurrentStatus()`](#SystemStatus+getCurrentStatus) ⇒ [`SystemInfo`](../typedefs/systeminfo)
- [`.getHistoricalStatus()`](#SystemStatus+getHistoricalStatus) ⇒ `Object`
<a name="new_SystemStatus_new"></a>
@@ -29,53 +29,22 @@ full snapshot history available in the [`Snapshotter`](snapshotter) instance.
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>[options]</code></td><td><code>Object</code></td><td></td>
+<td><code>[options]</code></td><td><code><a href="../typedefs/systemstatusoptions">SystemStatusOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>All <code>SystemStatus</code> parameters are passed
- via an options object with the following keys:</p>
-</td></tr><tr>
-<td><code>[options.currentHistorySecs]</code></td><td><code>Number</code></td><td><code>5</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Defines max age of snapshots used in the
- <a href="#SystemStatus+getCurrentStatus"><code>getCurrentStatus()</code></a> measurement.</p>
-</td></tr><tr>
-<td><code>[options.maxMemoryOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a memory sample.
- If the sample exceeds this ratio, the system will be overloaded.</p>
-</td></tr><tr>
-<td><code>[options.maxEventLoopOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in an event loop sample.
- If the sample exceeds this ratio, the system will be overloaded.</p>
-</td></tr><tr>
-<td><code>[options.maxCpuOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.4</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a CPU sample.
- If the sample exceeds this ratio, the system will be overloaded.</p>
-</td></tr><tr>
-<td><code>[options.maxClientOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a Client sample.
- If the sample exceeds this ratio, the system will be overloaded.</p>
+<td colspan="3"><p>All <code>SystemStatus</code> configuration options.</p>
</td></tr></tbody>
</table>
<a name="SystemStatus+getCurrentStatus"></a>
-## `systemStatus.getCurrentStatus()` ⇒ `Object`
+## `systemStatus.getCurrentStatus()` ⇒ [`SystemInfo`](../typedefs/systeminfo)
-Returns an object with the following structure:
+Returns an [`SystemInfo`](../typedefs/systeminfo) object with the following structure:
```javascript
{
@@ -93,7 +62,7 @@ otherwise.
## `systemStatus.getHistoricalStatus()` ⇒ `Object`
-Returns an object with the following structure:
+Returns an [`SystemInfo`](../typedefs/systeminfo) object with the following structure:
```javascript
{
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 874e1c51a..8209c096a 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -124,11 +124,10 @@ objects.
<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function
enqueues all links found on the page.</p>
</td></tr><tr>
-<td><code>[options.transformRequestFunction]</code></td><td><code>function</code></td><td></td>
+<td><code>[options.transformRequestFunction]</code></td><td><code><a href="../typedefs/requesttransform">RequestTransform</a></code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p><strong>Signature:</strong> (<a href="request"><code>Request</code></a>): <a href="request"><code>Request</code></a></p>
-<p> Just before a new <a href="request"><code>Request</code></a> is constructed and enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used
+<td colspan="3"><p>Just before a new <a href="request"><code>Request</code></a> is constructed and enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used
to remove it or modify its contents such as <code>userData</code>, <code>payload</code> or, most importantly <code>uniqueKey</code>. This is useful
when you need to enqueue multiple <code>Requests</code> to the queue that share the same URL, but differ in methods or payloads,
or to dynamically update or create <code>userData</code>.</p>
diff --git a/docs/api/social.md b/docs/api/social.md
index d75953642..a8e138981 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -32,7 +32,7 @@ const emails = Apify.utils.social.emailsFromText('[email protected] bob@example.
- [`.emailsFromUrls(urls)`](#social.emailsFromUrls) ⇒ `Array<String>`
- [`.phonesFromText(text)`](#social.phonesFromText) ⇒ `Array<String>`
- [`.phonesFromUrls(urls)`](#social.phonesFromUrls) ⇒ `Array<String>`
- - [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ `*`
+ - [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ [`SocialHandles`](../typedefs/socialhandles)
<a name="social.LINKEDIN_REGEX"></a>
@@ -407,26 +407,11 @@ Finds phone number links in an array of URLs and extracts the phone numbers from
</table>
<a name="social.parseHandlesFromHtml"></a>
-## `social.parseHandlesFromHtml(html, data)` ⇒ `*`
+## `social.parseHandlesFromHtml(html, data)` ⇒ [`SocialHandles`](../typedefs/socialhandles)
The function attempts to extract emails, phone numbers and social profile URLs from a HTML document, specifically LinkedIn, Twitter, Instagram and
Facebook profile URLs. The function removes duplicates from the resulting arrays and sorts the items alphabetically.
-The result of the function is an object with the following structure:
-
-```
-{
- emails: String[],
- phones: String[],
- phonesUncertain: String[],
- linkedIns: String[],
- twitters: String[],
- instagrams: String[],
- facebooks: String[],
- youtubes: String[],
-}
-```
-
Note that the `phones` field contains phone numbers extracted from the special phone links such as `<a href="tel:+1234556789">call us</a>` (see
[`social.phonesFromUrls()`](#social.phonesFromUrls)]) and potentially other sources with high certainty, while `phonesUncertain` contains phone
numbers extracted from the plain text, which might be very inaccurate.
@@ -446,7 +431,7 @@ console.log('Social handles:');
console.dir(result);
```
-**Returns**: `*` - An object with the social handles.
+**Returns**: [`SocialHandles`](../typedefs/socialhandles) - An object with the social handles.
<table>
<thead>
diff --git a/docs/api/utils.md b/docs/api/utils.md
index cf8107c11..ce06e63f7 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -21,14 +21,15 @@ await Apify.utils.sleep(1500);
- [`utils`](#utils) : `object`
- [`.enqueueLinks`](#utils.enqueueLinks) ⇒ `Promise<Array<QueueOperationInfo>>`
- [`.requestAsBrowser`](#utils.requestAsBrowser) ⇒ `Promise<(http.IncomingMessage|stream.Readable)>`
- - [`.sleep`](#utils.sleep) ⇒ `Promise`
+ - [`.sleep`](#utils.sleep) ⇒ `Promise<void>`
- [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)
- [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)
- - [`.isDocker()`](#utils.isDocker) ⇒ `Promise`
+ - [`.isDocker(forceReset)`](#utils.isDocker) ⇒ `Promise<boolean>`
- [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`
- [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`
- [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`
- [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`
+ - [`~$`](#utils.htmlToText..$) : `Cheerio`
<a name="utils.enqueueLinks"></a>
@@ -115,11 +116,10 @@ objects.
<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function
enqueues all links found on the page.</p>
</td></tr><tr>
-<td><code>[options.transformRequestFunction]</code></td><td><code>function</code></td><td></td>
+<td><code>[options.transformRequestFunction]</code></td><td><code><a href="../typedefs/requesttransform">RequestTransform</a></code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p><strong>Signature:</strong> (<a href="request"><code>Request</code></a>): <a href="request"><code>Request</code></a></p>
-<p> Just before a new <a href="request"><code>Request</code></a> is constructed and enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used
+<td colspan="3"><p>Just before a new <a href="request"><code>Request</code></a> is constructed and enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used
to remove it or modify its contents such as <code>userData</code>, <code>payload</code> or, most importantly <code>uniqueKey</code>. This is useful
when you need to enqueue multiple <code>Requests</code> to the queue that share the same URL, but differ in methods or payloads,
or to dynamically update or create <code>userData</code>.</p>
@@ -151,57 +151,27 @@ Currently, the function sends requests the same way as Firefox web browser does.
Internally, the function uses httpRequest function from the [@apify/httpRequest](https://github.com/apifytech/http-request) NPM package to perform the
request. All `options` not recognized by this function are passed to it, so see it for more details.
+**Returns**: `Promise<(http.IncomingMessage|stream.Readable)>` - This will typically be a
+[Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage), however, if returned from the cache it will be a
+[response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.
+
<table>
<thead>
<tr>
-<th>Param</th><th>Default</th>
+<th>Param</th><th>Type</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options.url</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>URL of the target endpoint. Supports both HTTP and HTTPS schemes.</p>
-</td></tr><tr>
-<td><code>[options.method]</code></td><td><code>GET</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>HTTP method.</p>
-</td></tr><tr>
-<td><code>[options.headers]</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Additional HTTP headers to add. It&#39;s only recommended to use this option,
- with headers that are typically added by websites, such as cookies. Overriding
- default browser headers will remove the masking this function provides.</p>
-</td></tr><tr>
-<td><code>[options.languageCode]</code></td><td><code>en</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Two-letter ISO 639 language code.</p>
-</td></tr><tr>
-<td><code>[options.countryCode]</code></td><td><code>US</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Two-letter ISO 3166 country code.</p>
-</td></tr><tr>
-<td><code>[options.isMobile]</code></td><td></td>
+<td><code>options</code></td><td><code><a href="../typedefs/requestasbrowseroptions">RequestAsBrowserOptions</a></code></td>
</tr>
<tr>
-<td colspan="3"><p>If <code>true</code>, the function uses User-Agent of a mobile browser.</p>
-</td></tr><tr>
-<td><code>[options.abortFunction]</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.
- If function returns true request gets aborted. This function is passed to the
- (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>
+<td colspan="3"><p>All <code>requestAsBrowser</code> configuration options.</p>
</td></tr></tbody>
</table>
<a name="utils.sleep"></a>
-## `utils.sleep` ⇒ `Promise`
+## `utils.sleep` ⇒ `Promise<void>`
Returns a `Promise` that resolves after a specific period of time. This is useful to implement waiting in your code, e.g. to prevent overloading of
target website or to avoid bot detection.
@@ -247,10 +217,23 @@ however, that this may prevent parsing URLs from comma delimited lists, or the U
<a name="utils.isDocker"></a>
-## `utils.isDocker()` ⇒ `Promise`
+## `utils.isDocker(forceReset)` ⇒ `Promise<boolean>`
Returns a `Promise` that resolves to true if the code is running in a Docker container.
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>forceReset</code></td><td><code>boolean</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
<a name="utils.downloadListOfUrls"></a>
## `utils.downloadListOfUrls(options)` ⇒ `Promise<Array<String>>`
@@ -358,10 +341,13 @@ const text = htmlToText(cheerio.load(html, { decodeEntities: true }));
</thead>
<tbody>
<tr>
-<td><code>html</code></td><td><code>String</code> | <code>function</code></td>
+<td><code>html</code></td><td><code>String</code> | <code>Cheerio</code></td>
</tr>
<tr>
<td colspan="3"><p>HTML text or parsed HTML represented using a
<a href="https://www.npmjs.com/package/cheerio">cheerio</a> function.</p>
</td></tr></tbody>
</table>
+<a name="utils.htmlToText..$"></a>
+
+### `htmlToText~$` : `Cheerio`
diff --git a/docs/guides/typescript_actor.md b/docs/guides/typescript_actor.md
new file mode 100644
index 000000000..b7d94bd6a
--- /dev/null
+++ b/docs/guides/typescript_actor.md
@@ -0,0 +1,136 @@
+---
+id: typescriptactor
+title: Typescript Actors
+---
+
+Apify SDK supports Typescript by covering public APIs with type declarations. This
+allows writing code with auto-completion for Typedcript and Javascript code alike.
+Besides that, actors written in Typescript can take advantage of compile-time
+type-checking and avoid many mistakes.
+
+Setting up Typescript project
+=============================
+
+To use Typescript in your actors, you'll need the following prerequisities.
+
+1. Typescript compiler `tsc` installed somewhere:
+
+ ```shell script
+ npm install --dev typescript
+ ```
+
+ Type can be a development dependency in your project, as shown above. There's no
+ need to pollute your production environment or your system's package repository
+ with Typescript.
+
+2. A build script invoking `tsc` and a correctly specified `main` entry point defined
+ in your `package.json`:
+
+ ```json
+ {
+ "scripts": {
+ "build": "tsc"
+ },
+ "main": "build/main.js"
+ }
+ ```
+
+3. Type declarations for NodeJS, and optionally Cheerio, or Puppeteer (or both), so
+ you can take advantage of type-checking in all the features you'll use:
+
+ ```shell script
+ npm install --dev @types/node
+ npm install --dev @types/cheerio
+ npm install --dev @types/puppeteer
+ ```
+
+ If you're using other Javascript packages in your actor, you'll want to include their
+ type declarations too (if they are available).
+
+4. Typescript configuration file allowing `tsc` to understand your project layout and
+ the features used in your project and a targeted language level:
+
+ ```json
+ {
+ "compilerOptions": {
+ "target": "es2018",
+ "module": "commonjs",
+ "moduleResolution": "node",
+ "lib": [
+ "es2018",
+ "es2018.asynciterable",
+ "dom",
+ "dom.iterable"
+ ],
+ "types": [
+ "node",
+ "cheerio"
+ ],
+ "esModuleInterop": true,
+ "outDir": "build/"
+ },
+ "include": [
+ "src/"
+ ]
+ }
+ ```
+
+ NOTE: You'll need to mention `cheerio` in `tsconfig.json` explicitly, but you don't
+ need to do so for `puppeteer`. The reason is that the `cheerio` module exports only
+ one function. Puppeteer's type declarations explicitly export all the types required.
+
+Auto-completion
+==============
+
+IDE auto-completion should work in most places. That's true even if you are writting
+actors in pure Javascript. For time constraints, we left out the amendment of an
+internal API for the time being, and these need to be added as SDK developers write
+new and enhance old code.
+
+SDK Documentation
+=================
+
+SDK documentation has grown a lot. There is a new API Reference section and one
+sub-section:
+
+- **Compiler options** in the **Type definitions** sidebar - Holds documentation for
+ all constructible types in SDK.
+- **User-Functions** sidebar - Holds documentation for user-provided functions like
+ `CheerioHandlePage`, `PuppeteerHandlePage`, `DatasetConsumer`, `DatasetMapper`, etc.
+ and their associated value-types `CheerioHandlePageInput`, etc.
+
+Expanding and enhancing documentation in those new places and adding more
+details may be a potential priority.
+
+Problems
+========
+
+Typescript sometimes generates invalid or incorrect declarations from JSDoc comments
+until Typescript developers fix these problems. This needs to be handled in the future,
+though all critical errors have been avoided or fixed already. Examples:
+
+- Typescript does not handle `@extends` and `@typedef {BaseType}` as expected
+- In `types/session_pool/session_pool.d.ts`, it adds methods from the base-class
+ `EventEmitter` and changes their return types, so they are not compatible with the
+ `EventEmitter` (i.e., `addListener(): SessionPool`, instead of `addListener(): this`).
+
+How to `launchPuppeteer()`
+==========================
+
+We are not able to `@extend` puppeteer's `LaunchOptions` as mentioned above and
+JSDoc considers intersection types a syntactic error.
+For this reason we left out the original launch options from our declarations and
+let users to handle types in his code.
+
+You can just cast a as PuppeteerOptions after the object, as such:
+
+```typescript
+// manually import the types from puppeteer
+import { PuppeteerOptions } from 'puppeteer'
+import { LaunchPuppeteerOptions } from 'apify'
+
+Apify.launchPuppeteer({ } as PuppeteerOptions & LaunchPuppeteerOptions)
+// should show all available options intersected
+```
+
+or slap an `as any` for limited time
diff --git a/docs/typedefs/ApifyEnv.md b/docs/typedefs/ApifyEnv.md
new file mode 100644
index 000000000..52867afe0
--- /dev/null
+++ b/docs/typedefs/ApifyEnv.md
@@ -0,0 +1,75 @@
+---
+id: apifyenv
+title: ApifyEnv
+---
+
+<a name="ApifyEnv"></a>
+
+Parsed representation of the `APIFY_XXX` environmental variables.
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>actorId</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>ID of the actor (APIFY_ACTOR_ID)</p>
+</td></tr><tr>
+<td><code>actorRunId</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>ID of the actor run (APIFY_ACTOR_RUN_ID)</p>
+</td></tr><tr>
+<td><code>actorTaskId</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>ID of the actor task (APIFY_ACTOR_TASK_ID)</p>
+</td></tr><tr>
+<td><code>userId</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>ID of the user who started the actor - note that it might be
+ different than the owner ofthe actor (APIFY_USER_ID)</p>
+</td></tr><tr>
+<td><code>token</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Authentication token representing privileges given to the actor run,
+ it can be passed to various Apify APIs (APIFY_TOKEN)</p>
+</td></tr><tr>
+<td><code>startedAt</code></td><td><code>Date</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Date when the actor was started (APIFY_STARTED_AT)</p>
+</td></tr><tr>
+<td><code>timeoutAt</code></td><td><code>Date</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Date when the actor will time out (APIFY_TIMEOUT_AT)</p>
+</td></tr><tr>
+<td><code>defaultKeyValueStoreId</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>ID of the key-value store where input and output data of this
+ actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)</p>
+</td></tr><tr>
+<td><code>defaultDatasetId</code></td><td><code>String</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>ID of the dataset where input and output data of this
+ actor is stored (APIFY_DEFAULT_DATASET_ID)</p>
+</td></tr><tr>
+<td><code>memoryMbytes</code></td><td><code>Number</code> | <code>null</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Amount of memory allocated for the actor,
+ in megabytes (APIFY_MEMORY_MBYTES)</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/AutoscaledPoolOptions.md b/docs/typedefs/AutoscaledPoolOptions.md
new file mode 100644
index 000000000..597246e43
--- /dev/null
+++ b/docs/typedefs/AutoscaledPoolOptions.md
@@ -0,0 +1,109 @@
+---
+id: autoscaledpooloptions
+title: AutoscaledPoolOptions
+---
+
+<a name="AutoscaledPoolOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>runTaskFunction</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function that performs an asynchronous resource-intensive task.
+ The function must either be labeled <code>async</code> or return a promise.</p>
+</td></tr><tr>
+<td><code>isTaskReadyFunction</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function that indicates whether <code>runTaskFunction</code> should be called.
+ This function is called every time there is free capacity for a new task and it should
+ indicate whether it should start a new task or not by resolving to either <code>true</code> or <code>false</code>.
+ Besides its obvious use, it is also useful for task throttling to save resources.</p>
+</td></tr><tr>
+<td><code>isFinishedFunction</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function that is called only when there are no tasks to be processed.
+ If it resolves to <code>true</code> then the pool&#39;s run finishes. Being called only
+ when there are no tasks being processed means that as long as <code>isTaskReadyFunction()</code>
+ keeps resolving to <code>true</code>, <code>isFinishedFunction()</code> will never be called.
+ To abort a run, use the <a href="#AutoscaledPool+abort"><code>abort()</code></a> method.</p>
+</td></tr><tr>
+<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The minimum number of tasks running in parallel.</p>
+<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
+ If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
+</td></tr><tr>
+<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The maximum number of tasks running in parallel.</p>
+</td></tr><tr>
+<td><code>[desiredConcurrency]</code></td><td><code>Number</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>The desired number of tasks that should be running parallel on the start of the pool,
+ if there is a large enough supply of them.
+ By default, it is <code>minConcurrency</code>.</p>
+</td></tr><tr>
+<td><code>[desiredConcurrencyRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Minimum level of desired concurrency to reach before more scaling up is allowed.</p>
+</td></tr><tr>
+<td><code>[scaleUpStepRatio]</code></td><td><code>Number</code></td><td><code>0.05</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the fractional amount of desired concurrency to be added with each scaling up.
+ The minimum scaling step is one.</p>
+</td></tr><tr>
+<td><code>[scaleDownStepRatio]</code></td><td><code>Number</code></td><td><code>0.05</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the amount of desired concurrency to be subtracted with each scaling down.
+ The minimum scaling step is one.</p>
+</td></tr><tr>
+<td><code>[maybeRunIntervalSecs]</code></td><td><code>Number</code></td><td><code>0.5</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how often the pool should call the <code>runTaskFunction()</code> to start a new task, in seconds.
+ This has no effect on starting new tasks immediately after a task completes.</p>
+</td></tr><tr>
+<td><code>[loggingIntervalSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Specifies a period in which the instance logs its state, in seconds.
+ Set to <code>null</code> to disable periodic logging.</p>
+</td></tr><tr>
+<td><code>[autoscaleIntervalSecs]</code></td><td><code>Number</code></td><td><code>10</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines in seconds how often the pool should attempt to adjust the desired concurrency
+ based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
+ We suggest using a value from 5 to 20.</p>
+</td></tr><tr>
+<td><code>[snapshotterOptions]</code></td><td><code><a href="../typedefs/snapshotteroptions">SnapshotterOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Options to be passed down to the <a href="snapshotter"><code>Snapshotter</code></a> constructor. This is useful for fine-tuning
+ the snapshot intervals and history.</p>
+</td></tr><tr>
+<td><code>[systemStatusOptions]</code></td><td><code><a href="../typedefs/systemstatusoptions">SystemStatusOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Options to be passed down to the <a href="systemstatus"><code>SystemStatus</code></a> constructor. This is useful for fine-tuning
+ the system status reports. If a custom snapshotter is set in the options, it will be used
+ by the pool.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/BasicCrawlerOptions.md b/docs/typedefs/BasicCrawlerOptions.md
new file mode 100644
index 000000000..ccce17c14
--- /dev/null
+++ b/docs/typedefs/BasicCrawlerOptions.md
@@ -0,0 +1,111 @@
+---
+id: basiccrawleroptions
+title: BasicCrawlerOptions
+---
+
+<a name="BasicCrawlerOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>handleRequestFunction</code></td><td><code><a href="../typedefs/handlerequest">HandleRequest</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each URL to crawl.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ request: Request,
+ autoscaledPool: AutoscaledPool
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>
+<p> The function must return a promise, which is then awaited by the crawler.</p>
+<p> If the function throws an exception, the crawler will try to re-crawl the
+ request later, up to <code>option.maxRequestRetries</code> times.
+ If all the retries fail, the crawler calls the function
+ provided to the <code>handleFailedRequestFunction</code> parameter.
+ To make this work, you should <strong>always</strong>
+ let your function throw exceptions rather than catch them.
+ The exceptions are logged to the request using the
+ <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
+</td></tr><tr>
+<td><code>[requestList]</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Static list of URLs to be processed.
+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
+</td></tr><tr>
+<td><code>[requestQueue]</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
+</td></tr><tr>
+<td><code>[handleRequestTimeoutSecs]</code></td><td><code>number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Timeout in which the function passed as <code>handleRequestFunction</code> needs to finish, in seconds.</p>
+</td></tr><tr>
+<td><code>[handleFailedRequestFunction]</code></td><td><code><a href="../typedefs/handlefailedrequest">HandleFailedRequest</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ request: Request,
+ error: Error,
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+ represents the last error thrown during processing of the request.</p>
+<p> See
+ <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
+ for the default implementation of this function.</p>
+</td></tr><tr>
+<td><code>[maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how many times the request is retried if <a href="#new_BasicCrawler_new"><code>handleRequestFunction()</code></a> fails.</p>
+</td></tr><tr>
+<td><code>[maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
+</td></tr><tr>
+<td><code>[autoscaledPoolOptions]</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.
+ Note that the <code>runTaskFunction</code> and <code>isTaskReadyFunction</code> options
+ are provided by <code>BasicCrawler</code> and cannot be overridden.
+ However, you can provide a custom implementation of <code>isFinishedFunction</code>.</p>
+</td></tr><tr>
+<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
+</td></tr><tr>
+<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+</td></tr><tr>
+<td><code>[useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If set to true. Basic crawler will initialize the <a href="sessionpool"><code>SessionPool</code></a> with the corresponding <code>sessionPoolOptions</code>.
+ The session instance will be than available in the <code>handleRequestFunction</code>.</p>
+</td></tr><tr>
+<td><code>[sessionPoolOptions]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>The configuration options for {SessionPool} to use.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/CheerioCrawlerOptions.md b/docs/typedefs/CheerioCrawlerOptions.md
new file mode 100644
index 000000000..74d0d00f6
--- /dev/null
+++ b/docs/typedefs/CheerioCrawlerOptions.md
@@ -0,0 +1,221 @@
+---
+id: cheeriocrawleroptions
+title: CheerioCrawlerOptions
+---
+
+<a name="CheerioCrawlerOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>handlePageFunction</code></td><td><code><a href="../typedefs/cheeriohandlepage">CheerioHandlePage</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each page
+ loaded and parsed by the crawler.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ $: Cheerio, // the Cheerio object with parsed HTML
+ body: String|Buffer // the request body of the web page
+ // the parsed object from JSON string
+ // if the response contains the content type application/json
+ json: Object,
+ request: Request,
+ contentType: Object, // Parsed Content-Type header: { type, encoding }
+ response: Object // An instance of Node&#39;s http.IncomingMessage object,
+ autoscaledPool: AutoscaledPool,
+ session: Session
+}</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p>
+<ul>
+<li><p>String for <code>text/html</code>, <code>application/xhtml+xml</code>, <code>application/xml</code> mime types</p>
+</li>
+<li><p>Buffer for others mime types</p>
+<p>Parsed <code>Content-Type</code> header using
+<a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>
+is stored in <code>contentType</code>.</p>
+<p>Cheerio is available only for HTML and XML content types.</p>
+<p>With the <a href="request"><code>Request</code></a> object representing the URL to crawl.</p>
+<p>If the function returns, the returned promise is awaited by the crawler.</p>
+<p>If the function throws an exception, the crawler will try to re-crawl the
+request later, up to <code>option.maxRequestRetries</code> times.
+If all the retries fail, the crawler calls the function
+provided to the <code>handleFailedRequestFunction</code> parameter.
+To make this work, you should <strong>always</strong>
+let your function throw exceptions rather than catch them.
+The exceptions are logged to the request using the
+<a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
+</li>
+</ul>
+</td></tr><tr>
+<td><code>[requestList]</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Static list of URLs to be processed.
+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
+</td></tr><tr>
+<td><code>[requestQueue]</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
+</td></tr><tr>
+<td><code>[requestOptions]</code></td><td><code><a href="../typedefs/requestasbrowseroptions">RequestAsBrowserOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Represents the options passed to
+ <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.
+ Provided <code>requestOptions</code> are added to internal defaults that cannot be overridden to ensure
+ the operation of <code>CheerioCrawler</code> and all its options. Headers will not be merged,
+ use <a href="requestlist"><code>RequestList</code></a> and/or <a href="requestqueue"><code>RequestQueue</code></a> to initialize your <a href="request"><code>Request</code></a> with the
+ correct headers or use <code>prepareRequestFunction</code> to modify your <a href="request"><code>Request</code></a> dynamically.
+ If you need more granular control over your requests, use <a href="basiccrawler"><code>BasicCrawler</code></a>.</p>
+<p> The mandatory internal defaults that <strong>CANNOT BE OVERRIDDEN</strong> by <code>requestOptions</code>:</p>
+<pre><code> {
+ url, // Provided by RequestList and/or RequestQueue
+ method, // Provided by RequestList and/or RequestQueue
+ headers, // Provided by RequestList and/or RequestQueue
+ payload, // Provided by RequestList and/or RequestQueue
+ strictSSL, // Use ignoreSslErrors
+ proxy, // Use useApifyProxy or proxyUrls
+ }</code></pre></td></tr><tr>
+<td><code>[prepareRequestFunction]</code></td><td><code><a href="../typedefs/preparerequest">PrepareRequest</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function that executes before the HTTP request is made to the target resource.
+ This function is suitable for setting dynamic properties such as cookies to the <a href="request"><code>Request</code></a>.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ request: Request,
+ session: Session
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request
+ and the <a href="session"><code>Session</code></a> instance corresponds to used session.</p>
+<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance
+ in place because there are already earlier references to it. Making a copy and returning it from
+ this function is therefore not supported, because it would create inconsistencies where
+ different parts of SDK would have access to a different <a href="request"><code>Request</code></a> instance.</p>
+</td></tr><tr>
+<td><code>[handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Timeout in which the function passed as <code>handlePageFunction</code> needs to finish, given in seconds.</p>
+</td></tr><tr>
+<td><code>[requestTimeoutSecs]</code></td><td><code>Number</code></td><td><code>30</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Timeout in which the HTTP request to the resource needs to finish, given in seconds.</p>
+</td></tr><tr>
+<td><code>[ignoreSslErrors]</code></td><td><code>Boolean</code></td><td><code>true</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If set to true, SSL certificate errors will be ignored.</p>
+</td></tr><tr>
+<td><code>[useApifyProxy]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If set to <code>true</code>, <code>CheerioCrawler</code> will be configured to use
+ <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
+ For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a></p>
+</td></tr><tr>
+<td><code>[apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>An array of proxy groups to be used
+ by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
+ Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>
+</td></tr><tr>
+<td><code>[apifyProxySession]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Apify Proxy session identifier to be used with requests made by <code>CheerioCrawler</code>.
+ All HTTP requests going through the proxy with the same session identifier
+ will use the same target proxy server (i.e. the same IP address).
+ The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>&quot;.&quot;</code>, <code>&quot;_&quot;</code> and <code>&quot;~&quot;</code>.
+ Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>
+</td></tr><tr>
+<td><code>[proxyUrls]</code></td><td><code>Array<String></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>CheerioCrawler</code> instance.
+ The provided custom proxies&#39; order will be randomized and the resulting list rotated.
+ Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ configuration options will cause an error to be thrown on startup.</p>
+</td></tr><tr>
+<td><code>[handleFailedRequestFunction]</code></td><td><code><a href="../typedefs/handlefailedrequest">HandleFailedRequest</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ request: Request,
+ error: Error,
+}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+ represents the last error thrown during processing of the request.</p>
+<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
+ for the default implementation of this function.</p>
+</td></tr><tr>
+<td><code>[additionalMimeTypes]</code></td><td><code>Array<String></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
+ target="_blank">mime types</a> you want to process.
+ By default <code>text/html</code>, <code>application/xhtml+xml</code> mime types are supported.</p>
+</td></tr><tr>
+<td><code>[maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how many times the request is retried if either <code>requestFunction</code> or <code>handlePageFunction</code> fails.</p>
+</td></tr><tr>
+<td><code>[maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
+</td></tr><tr>
+<td><code>[autoscaledPoolOptions]</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.
+ Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options
+ are provided by <code>CheerioCrawler</code> and cannot be overridden. Reasonable <a href="snapshotter"><code>Snapshotter</code></a>
+ and <a href="systemstatus"><code>SystemStatus</code></a> defaults are provided to account for the fact that <code>cheerio</code>
+ parses HTML synchronously and therefore blocks the event loop.</p>
+</td></tr><tr>
+<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
+</td></tr><tr>
+<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+</td></tr><tr>
+<td><code>[useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ It also marks Session as bad after a request timeout.</p>
+</td></tr><tr>
+<td><code>[sessionPoolOptions]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>
+</td></tr><tr>
+<td><code>[persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>
+<p> It parses cookie from response &quot;set-cookie&quot; header saves or updates cookies for session and once the session is used for next request.
+ It passes the &quot;Cookie&quot; header to the request with the session cookies.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/CheerioHandlePage.md b/docs/typedefs/CheerioHandlePage.md
new file mode 100644
index 000000000..e27fca142
--- /dev/null
+++ b/docs/typedefs/CheerioHandlePage.md
@@ -0,0 +1,21 @@
+---
+id: cheeriohandlepage
+title: CheerioHandlePage
+---
+
+<a name="CheerioHandlePage"></a>
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/cheeriohandlepageinputs">CheerioHandlePageInputs</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/CheerioHandlePageInputs.md b/docs/typedefs/CheerioHandlePageInputs.md
new file mode 100644
index 000000000..54153b3d1
--- /dev/null
+++ b/docs/typedefs/CheerioHandlePageInputs.md
@@ -0,0 +1,56 @@
+---
+id: cheeriohandlepageinputs
+title: CheerioHandlePageInputs
+---
+
+<a name="CheerioHandlePageInputs"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>[$]</code></td><td><code>Cheerio</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The <a href="https://cheerio.js.org/">Cheerio</a> object with parsed HTML.</p>
+</td></tr><tr>
+<td><code>body</code></td><td><code>String</code> | <code>Buffer</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The request body of the web page.</p>
+</td></tr><tr>
+<td><code>[json]</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The parsed object from JSON string if the response contains the content type application/json.</p>
+</td></tr><tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The original {Request} object.</p>
+</td></tr><tr>
+<td><code>contentType</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Parsed <code>Content-Type header: { type, encoding }</code>.</p>
+</td></tr><tr>
+<td><code>response</code></td><td><code>IncomingMessage</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of Node&#39;s http.IncomingMessage object,</p>
+</td></tr><tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>[session]</code></td><td><code>session</code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr></tbody>
+</table>
diff --git a/docs/typedefs/CreateSession.md b/docs/typedefs/CreateSession.md
new file mode 100644
index 000000000..e3d2f0edf
--- /dev/null
+++ b/docs/typedefs/CreateSession.md
@@ -0,0 +1,23 @@
+---
+id: createsession
+title: CreateSession
+---
+
+<a name="CreateSession"></a>
+
+Factory user-function which creates customized [`Session`](session) instances.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>sessionPool</code></td><td><code><a href="sessionpool">SessionPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Pool requesting the new session.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/DatasetConsumer.md b/docs/typedefs/DatasetConsumer.md
new file mode 100644
index 000000000..b942c7d69
--- /dev/null
+++ b/docs/typedefs/DatasetConsumer.md
@@ -0,0 +1,30 @@
+---
+id: datasetconsumer
+title: DatasetConsumer
+---
+
+<a name="DatasetConsumer"></a>
+
+User-function used in the `Dataset.forEach()` API.
+
+**Returns**: T
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>item</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Current <a href="dataset"><code>Dataset</code></a> entry being processed.</p>
+</td></tr><tr>
+<td><code>index</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Position of current {Dataset} entry.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/DatasetContent.md b/docs/typedefs/DatasetContent.md
new file mode 100644
index 000000000..1c0d3bc06
--- /dev/null
+++ b/docs/typedefs/DatasetContent.md
@@ -0,0 +1,43 @@
+---
+id: datasetcontent
+title: DatasetContent
+---
+
+<a name="DatasetContent"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>items</code></td><td><code>Array<Object&gt;</code> | <code>Array<String></code> | <code>Array.&lt;Buffer></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Dataset entries based on chosen format parameter.</p>
+</td></tr><tr>
+<td><code>total</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Total count of entries in the dataset.</p>
+</td></tr><tr>
+<td><code>offset</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Position of the first returned entry in the dataset.</p>
+</td></tr><tr>
+<td><code>count</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Count of dataset entries returned in this set.</p>
+</td></tr><tr>
+<td><code>limit</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of dataset entries requested.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/DatasetMapper.md b/docs/typedefs/DatasetMapper.md
new file mode 100644
index 000000000..9f07303a6
--- /dev/null
+++ b/docs/typedefs/DatasetMapper.md
@@ -0,0 +1,30 @@
+---
+id: datasetmapper
+title: DatasetMapper
+---
+
+<a name="DatasetMapper"></a>
+
+User-function used in the `Dataset.map()` API.
+
+**Returns**: T
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>item</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Currect <a href="dataset"><code>Dataset</code></a> entry being processed.</p>
+</td></tr><tr>
+<td><code>index</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Position of current {Dataset} entry.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/DatasetReducer.md b/docs/typedefs/DatasetReducer.md
new file mode 100644
index 000000000..867dca68c
--- /dev/null
+++ b/docs/typedefs/DatasetReducer.md
@@ -0,0 +1,35 @@
+---
+id: datasetreducer
+title: DatasetReducer
+---
+
+<a name="DatasetReducer"></a>
+
+User-function used in the `Dataset.reduce()` API.
+
+**Returns**: T
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>memo</code></td><td><code>T</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Previous state of the reduction.</p>
+</td></tr><tr>
+<td><code>item</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Currect <a href="dataset"><code>Dataset</code></a> entry being processed.</p>
+</td></tr><tr>
+<td><code>index</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Position of current {Dataset} entry.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/HandleFailedRequest.md b/docs/typedefs/HandleFailedRequest.md
new file mode 100644
index 000000000..ada6af09f
--- /dev/null
+++ b/docs/typedefs/HandleFailedRequest.md
@@ -0,0 +1,21 @@
+---
+id: handlefailedrequest
+title: HandleFailedRequest
+---
+
+<a name="HandleFailedRequest"></a>
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/handlefailedrequestinput">HandleFailedRequestInput</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/HandleFailedRequestInput.md b/docs/typedefs/HandleFailedRequestInput.md
new file mode 100644
index 000000000..15c78af73
--- /dev/null
+++ b/docs/typedefs/HandleFailedRequestInput.md
@@ -0,0 +1,28 @@
+---
+id: handlefailedrequestinput
+title: HandleFailedRequestInput
+---
+
+<a name="HandleFailedRequestInput"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The original {Request} object.</p>
+</td></tr><tr>
+<td><code>error</code></td><td><code>Error</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The Error thrown by <code>handleRequestFunction</code>.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/HandleRequest.md b/docs/typedefs/HandleRequest.md
new file mode 100644
index 000000000..d78416a64
--- /dev/null
+++ b/docs/typedefs/HandleRequest.md
@@ -0,0 +1,21 @@
+---
+id: handlerequest
+title: HandleRequest
+---
+
+<a name="HandleRequest"></a>
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/handlerequestinputs">HandleRequestInputs</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/HandleRequestInputs.md b/docs/typedefs/HandleRequestInputs.md
new file mode 100644
index 000000000..98d1210c0
--- /dev/null
+++ b/docs/typedefs/HandleRequestInputs.md
@@ -0,0 +1,31 @@
+---
+id: handlerequestinputs
+title: HandleRequestInputs
+---
+
+<a name="HandleRequestInputs"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The original {Request} object.</p>
+</td></tr><tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>[session]</code></td><td><code><a href="session">Session</a></code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr></tbody>
+</table>
diff --git a/docs/typedefs/KeyConsumer.md b/docs/typedefs/KeyConsumer.md
new file mode 100644
index 000000000..6cb8bfff4
--- /dev/null
+++ b/docs/typedefs/KeyConsumer.md
@@ -0,0 +1,38 @@
+---
+id: keyconsumer
+title: KeyConsumer
+---
+
+<a name="KeyConsumer"></a>
+
+User-function used in the [`KeyValueStore.forEachKey()`](../api/keyvaluestore#forEachKey) method.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>key</code></td><td><code>String</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Current {KeyValue} key being processed.</p>
+</td></tr><tr>
+<td><code>index</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Position of the current key in {KeyValuestore}.</p>
+</td></tr><tr>
+<td><code>info</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Information about the current {KeyValueStore} entry.</p>
+</td></tr><tr>
+<td><code>info.size</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Size of the value associated with the current key in bytes.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/LaunchPuppeteer.md b/docs/typedefs/LaunchPuppeteer.md
new file mode 100644
index 000000000..b566b7ad6
--- /dev/null
+++ b/docs/typedefs/LaunchPuppeteer.md
@@ -0,0 +1,25 @@
+---
+id: launchpuppeteer
+title: LaunchPuppeteer
+---
+
+<a name="LaunchPuppeteer"></a>
+
+**Returns**: `Promise<Browser>` - Promise that resolves to Puppeteer's `Browser` instance. This might be obtained by calling
+<a href="https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions">puppeteer.launch()</a> directly, or by delegating to
+[`Apify.launchPuppeteer()`](../api/apify#apifylaunchpuppeteeroptions-%E2%87%92-promisebrowser).
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.md
index cccfd910d..1a900bce3 100644
--- a/docs/typedefs/LaunchPuppeteerOptions.md
+++ b/docs/typedefs/LaunchPuppeteerOptions.md
@@ -19,13 +19,6 @@ Apify extends the launch options of Puppeteer. You can use any of the
</thead>
<tbody>
<tr>
-<td><code>...</code></td><td></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>You can use any of the
- <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
- options.</p>
-</td></tr><tr>
<td><code>[proxyUrl]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
diff --git a/docs/typedefs/MemoryInfo.md b/docs/typedefs/MemoryInfo.md
new file mode 100644
index 000000000..484cad953
--- /dev/null
+++ b/docs/typedefs/MemoryInfo.md
@@ -0,0 +1,45 @@
+---
+id: memoryinfo
+title: MemoryInfo
+---
+
+<a name="MemoryInfo"></a>
+
+Describes memory usage of an Actor.
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>totalBytes</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Total memory available in the system or container</p>
+</td></tr><tr>
+<td><code>freeBytes</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Amount of free memory in the system or container</p>
+</td></tr><tr>
+<td><code>usedBytes</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Amount of memory used (= totalBytes - freeBytes)</p>
+</td></tr><tr>
+<td><code>mainProcessBytes</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Amount of memory used the current Node.js process</p>
+</td></tr><tr>
+<td><code>childProcessesBytes</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Amount of memory used by child processes of the current Node.js process</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PrepareRequest.md b/docs/typedefs/PrepareRequest.md
new file mode 100644
index 000000000..57e9a173a
--- /dev/null
+++ b/docs/typedefs/PrepareRequest.md
@@ -0,0 +1,21 @@
+---
+id: preparerequest
+title: PrepareRequest
+---
+
+<a name="PrepareRequest"></a>
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/preparerequestinputs">PrepareRequestInputs</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PrepareRequestInputs.md b/docs/typedefs/PrepareRequestInputs.md
new file mode 100644
index 000000000..f310bf85c
--- /dev/null
+++ b/docs/typedefs/PrepareRequestInputs.md
@@ -0,0 +1,23 @@
+---
+id: preparerequestinputs
+title: PrepareRequestInputs
+---
+
+<a name="PrepareRequestInputs"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Original instance fo the {Request} object. Must be modified in-place.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PuppeteerCrawlerOptions.md b/docs/typedefs/PuppeteerCrawlerOptions.md
new file mode 100644
index 000000000..d0a20441e
--- /dev/null
+++ b/docs/typedefs/PuppeteerCrawlerOptions.md
@@ -0,0 +1,165 @@
+---
+id: puppeteercrawleroptions
+title: PuppeteerCrawlerOptions
+---
+
+<a name="PuppeteerCrawlerOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>handlePageFunction</code></td><td><code><a href="../typedefs/puppeteerhandlepage">PuppeteerHandlePage</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Function that is called to process each request.
+ It is passed an object with the following fields:</p>
+<pre><code>{
+ request: Request,
+ response: Response,
+ page: Page,
+ puppeteerPool: PuppeteerPool,
+ autoscaledPool: AutoscaledPool,
+ session: Session,
+}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.
+ <code>page</code> is an instance of the <code>Puppeteer</code>
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ <code>response</code> is an instance of the <code>Puppeteer</code>
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ which is the main resource response as returned by <code>page.goto(request.url)</code>.
+ <code>puppeteerPool</code> is an instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>
+<p> The function must return a promise, which is then awaited by the crawler.</p>
+<p> If the function throws an exception, the crawler will try to re-crawl the
+ request later, up to <code>option.maxRequestRetries</code> times.
+ If all the retries fail, the crawler calls the function
+ provided to the <code>handleFailedRequestFunction</code> parameter.
+ To make this work, you should <strong>always</strong>
+ let your function throw exceptions rather than catch them.
+ The exceptions are logged to the request using the
+ <a href="request#Request+pushErrorMessage"><code>request.pushErrorMessage</code></a> function.</p>
+</td></tr><tr>
+<td><code>[requestList]</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Static list of URLs to be processed.
+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
+</td></tr><tr>
+<td><code>[requestQueue]</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
+</td></tr><tr>
+<td><code>[handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Timeout in which the function passed as <code>handlePageFunction</code> needs to finish, in seconds.</p>
+</td></tr><tr>
+<td><code>[gotoFunction]</code></td><td><code><a href="../typedefs/puppeteergoto">PuppeteerGoto</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer&#39;s
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,
+ i.e. a <code>Promise</code> resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.</p>
+<p> This is useful if you need to extend the page load timeout or select different criteria
+ to determine that the navigation succeeded.</p>
+<p> Note that a single page object is only used to process a single request and it is closed afterwards.</p>
+<p> By default, the function invokes <a href="puppeteer#puppeteer.gotoExtended"><code>Apify.utils.puppeteer.gotoExtended()</code></a> with a timeout of 60 seconds.
+ For details, see source code on
+ <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.</p>
+</td></tr><tr>
+<td><code>[gotoTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Timeout in which page navigation needs to finish, in seconds. When <code>gotoFunction()</code> is used and thus the default
+ function is overridden, this timeout will not be used and needs to be configured in the new <code>gotoFunction()</code>.</p>
+</td></tr><tr>
+<td><code>[handleFailedRequestFunction]</code></td><td><code><a href="../typedefs/handlefailedrequest">HandleFailedRequest</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ request: Request,
+ error: Error,
+}</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance
+ represents the last error thrown during processing of the request.</p>
+<p> See
+ <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
+ for the default implementation of this function.</p>
+</td></tr><tr>
+<td><code>[maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how many times the request is retried if either <code>handlePageFunction()</code> or <code>gotoFunction()</code> fails.</p>
+</td></tr><tr>
+<td><code>[maxRequestsPerCrawl]</code></td><td><code>Number</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
+</td></tr><tr>
+<td><code>[puppeteerPoolOptions]</code></td><td><code><a href="../typedefs/puppeteerpooloptions">PuppeteerPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="puppeteerpool"><code>PuppeteerPool</code></a> constructor.
+ You can tweak those to fine-tune browser management.</p>
+</td></tr><tr>
+<td><code>[launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.
+ Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option.
+ See source code on
+ <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
+ for default behavior.</p>
+</td></tr><tr>
+<td><code>[launchPuppeteerOptions]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Options used by <a href="apify#module_Apify.launchPuppeteer"><code>Apify.launchPuppeteer()</code></a> to start new Puppeteer instances.
+ Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option. See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
+</td></tr><tr>
+<td><code>[autoscaledPoolOptions]</code></td><td><code><a href="../typedefs/autoscaledpooloptions">AutoscaledPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> instance constructor.
+ Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options
+ are provided by <code>PuppeteerCrawler</code> and should not be overridden.</p>
+</td></tr><tr>
+<td><code>[minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
+</td></tr><tr>
+<td><code>[maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+</td></tr><tr>
+<td><code>[useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ It also marks Session as bad after a request timeout.</p>
+</td></tr><tr>
+<td><code>[sessionPoolOptions]</code></td><td><code><a href="../typedefs/sessionpooloptions">SessionPoolOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>
+</td></tr><tr>
+<td><code>[persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PuppeteerGoto.md b/docs/typedefs/PuppeteerGoto.md
new file mode 100644
index 000000000..9503aeaa3
--- /dev/null
+++ b/docs/typedefs/PuppeteerGoto.md
@@ -0,0 +1,25 @@
+---
+id: puppeteergoto
+title: PuppeteerGoto
+---
+
+<a name="PuppeteerGoto"></a>
+
+**Returns**: `Promise<PuppeteerResponse>` - An instance of the `Puppeteer`
+<a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>, which is the main resource response
+as returned by `page.goto(request.url)`.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/puppeteergotoinputs">PuppeteerGotoInputs</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PuppeteerGotoInputs.md b/docs/typedefs/PuppeteerGotoInputs.md
new file mode 100644
index 000000000..f3bde9bec
--- /dev/null
+++ b/docs/typedefs/PuppeteerGotoInputs.md
@@ -0,0 +1,44 @@
+---
+id: puppeteergotoinputs
+title: PuppeteerGotoInputs
+---
+
+<a name="PuppeteerGotoInputs"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>page</code></td><td><code>PuppeteerPage</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>is an instance of the <code>Puppeteer</code>
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a></p>
+</td></tr><tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.</p>
+</td></tr><tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of the <code>AutoscaledPool</code>.</p>
+</td></tr><tr>
+<td><code>puppeteerPool</code></td><td><code><a href="puppeteerpool">PuppeteerPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>
+</td></tr><tr>
+<td><code>[session]</code></td><td><code><a href="session">Session</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p><code>Session</code> object for this request.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PuppeteerHandlePage.md b/docs/typedefs/PuppeteerHandlePage.md
new file mode 100644
index 000000000..cd85af529
--- /dev/null
+++ b/docs/typedefs/PuppeteerHandlePage.md
@@ -0,0 +1,21 @@
+---
+id: puppeteerhandlepage
+title: PuppeteerHandlePage
+---
+
+<a name="PuppeteerHandlePage"></a>
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>inputs</code></td><td><code><a href="../typedefs/puppeteerhandlepageinputs">PuppeteerHandlePageInputs</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Arguments passed to this callback.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PuppeteerHandlePageInputs.md b/docs/typedefs/PuppeteerHandlePageInputs.md
new file mode 100644
index 000000000..6c2e98647
--- /dev/null
+++ b/docs/typedefs/PuppeteerHandlePageInputs.md
@@ -0,0 +1,49 @@
+---
+id: puppeteerhandlepageinputs
+title: PuppeteerHandlePageInputs
+---
+
+<a name="PuppeteerHandlePageInputs"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.</p>
+</td></tr><tr>
+<td><code>response</code></td><td><code>PuppeteerResponse</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of the <code>Puppeteer</code>
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ which is the main resource response as returned by <code>page.goto(request.url)</code>.</p>
+</td></tr><tr>
+<td><code>page</code></td><td><code>PuppeteerPage</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>is an instance of the <code>Puppeteer</code>
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a></p>
+</td></tr><tr>
+<td><code>puppeteerPool</code></td><td><code><a href="puppeteerpool">PuppeteerPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An instance of the <a href="puppeteerpool"><code>PuppeteerPool</code></a> used by this <code>PuppeteerCrawler</code>.</p>
+</td></tr><tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>[session]</code></td><td><code><a href="session">Session</a></code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr></tbody>
+</table>
diff --git a/docs/typedefs/PuppeteerPoolOptions.md b/docs/typedefs/PuppeteerPoolOptions.md
new file mode 100644
index 000000000..0f682319f
--- /dev/null
+++ b/docs/typedefs/PuppeteerPoolOptions.md
@@ -0,0 +1,100 @@
+---
+id: puppeteerpooloptions
+title: PuppeteerPoolOptions
+---
+
+<a name="PuppeteerPoolOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>[useLiveView]</code></td><td><code>boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Enables the use of a preconfigured <a href="liveviewserver"><code>LiveViewServer</code></a> that serves snapshots
+ just before a page would be recycled by <code>PuppeteerPool</code>. If there are no clients
+ connected, it has close to zero impact on performance.</p>
+</td></tr><tr>
+<td><code>[maxOpenPagesPerInstance]</code></td><td><code>number</code></td><td><code>50</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.</p>
+</td></tr><tr>
+<td><code>[retireInstanceAfterRequestCount]</code></td><td><code>number</code></td><td><code>100</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of requests that can be processed by a single browser instance.
+ After the limit is reached, the browser is retired and new requests are
+ handled by a new browser instance.</p>
+</td></tr><tr>
+<td><code>[puppeteerOperationTimeoutSecs]</code></td><td><code>number</code></td><td><code>15</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>All browser management operations such as launching a new browser, opening a new page
+ or closing a page will timeout after the set number of seconds and the connected
+ browser will be retired.</p>
+</td></tr><tr>
+<td><code>[instanceKillerIntervalSecs]</code></td><td><code>number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how often are the open Puppeteer instances checked whether they can be closed.</p>
+</td></tr><tr>
+<td><code>[killInstanceAfterSecs]</code></td><td><code>number</code></td><td><code>300</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>When Puppeteer instance reaches the <code>retireInstanceAfterRequestCount</code> limit then
+ it is considered retired and no more tabs will be opened. After the last tab is closed the
+ whole browser is closed too. This parameter defines a time limit between the last tab was opened and
+ before the browser is closed even if there are pending open tabs.</p>
+</td></tr><tr>
+<td><code>[launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.
+ The function must return a promise resolving to
+ <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser"><code>Browser</code></a> instance.
+ See the source code on
+ <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
+ for the default implementation.</p>
+</td></tr><tr>
+<td><code>[launchPuppeteerOptions]</code></td><td><code><a href="../typedefs/launchpuppeteeroptions">LaunchPuppeteerOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Options used by <code>Apify.launchPuppeteer()</code> to start new Puppeteer instances.
+ See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
+</td></tr><tr>
+<td><code>[recycleDiskCache]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Enables recycling of disk cache directories by Chrome instances.
+ When a browser instance is closed, its disk cache directory is not deleted but it&#39;s used by a newly opened browser instance.
+ This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
+ Note that the new browser starts with empty cookies, local storage etc. so this setting doesn&#39;t affect anonymity of your crawler.</p>
+<p> Beware that the disk cache directories can consume a lot of disk space.
+ To limit the space consumed, you can pass the <code>--disk-cache-size=X</code> argument to <code>launchPuppeteerargs</code>,
+ where <code>X</code> is the approximate maximum number of bytes for disk cache.</p>
+<p> Do not use the <code>recycleDiskCache</code> setting together with <code>--disk-cache-dir</code>
+ argument in <code>launchPuppeteerargs</code>, the behavior is undefined.</p>
+</td></tr><tr>
+<td><code>[useIncognitoPages]</code></td><td><code>boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>With this option selected, all pages will be opened in a new incognito browser context, which means
+ that they will not share cookies or cache and their resources will not be throttled by one another.</p>
+</td></tr><tr>
+<td><code>[proxyUrls]</code></td><td><code>Array<string></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>PuppeteerPool</code> instance.
+ The provided custom proxies&#39; order will be randomized and the resulting list rotated.
+ Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ configuration options will cause an error to be thrown on startup.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/RequestAsBrowserOptions.md b/docs/typedefs/RequestAsBrowserOptions.md
new file mode 100644
index 000000000..eb249229a
--- /dev/null
+++ b/docs/typedefs/RequestAsBrowserOptions.md
@@ -0,0 +1,62 @@
+---
+id: requestasbrowseroptions
+title: RequestAsBrowserOptions
+---
+
+<a name="RequestAsBrowserOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>url</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>URL of the target endpoint. Supports both HTTP and HTTPS schemes.</p>
+</td></tr><tr>
+<td><code>[method]</code></td><td><code>String</code></td><td><code>GET</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>HTTP method.</p>
+</td></tr><tr>
+<td><code>[headers]</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Additional HTTP headers to add. It&#39;s only recommended to use this option,
+ with headers that are typically added by websites, such as cookies. Overriding
+ default browser headers will remove the masking this function provides.</p>
+</td></tr><tr>
+<td><code>[proxyUrl]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.</p>
+</td></tr><tr>
+<td><code>[languageCode]</code></td><td><code>String</code></td><td><code>en</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Two-letter ISO 639 language code.</p>
+</td></tr><tr>
+<td><code>[countryCode]</code></td><td><code>String</code></td><td><code>US</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Two-letter ISO 3166 country code.</p>
+</td></tr><tr>
+<td><code>[isMobile]</code></td><td><code>Boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>true</code>, the function uses User-Agent of a mobile browser.</p>
+</td></tr><tr>
+<td><code>[abortFunction]</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.
+ If function returns true request gets aborted. This function is passed to the
+ (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/RequestListOptions.md b/docs/typedefs/RequestListOptions.md
new file mode 100644
index 000000000..5d273ede8
--- /dev/null
+++ b/docs/typedefs/RequestListOptions.md
@@ -0,0 +1,84 @@
+---
+id: requestlistoptions
+title: RequestListOptions
+---
+
+<a name="RequestListOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>sources</code></td><td><code>Array<(RequestOptions|Request)></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>. It can be either an array of plain objects that
+ define the <code>url</code> property, or an array of instances of the <a href="request"><code>Request</code></a> class.
+ Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,
+ which will instruct <code>RequestList</code> to download the source URLs from a given remote location.
+ The URLs will be parsed from the received response.</p>
+<pre><code>[
+ // One URL
+ { method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
+ // Batch import of URLs from a file hosted on the web
+ { method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
+ // Batch import combined with regex.
+ { method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39;, regex: /https:\/\/example.com\/.+/ },
+]</code></pre></td></tr><tr>
+<td><code>[persistStateKey]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Identifies the key in the default key-value store under which <code>RequestList</code> periodically stores its
+ state (i.e. which URLs were crawled and which not).
+ If the actor is restarted, <code>RequestList</code> will read the state
+ and continue where it left off.</p>
+<p> If <code>persistStateKey</code> is not set, <code>RequestList</code> will always start from the beginning,
+ and all the source URLs will be crawled again.</p>
+</td></tr><tr>
+<td><code>[persistSourcesKey]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Identifies the key in the default key-value store under which the <code>RequestList</code> persists its
+ sources (i.e. the lists of URLs) during the <a href="#RequestList+initialize"><code>initialize</code></a> call.
+ This is necessary if <code>persistStateKey</code> is set and the source URLs might potentially change,
+ to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.</p>
+<p> If <code>persistSourcesKey</code> is not set, <a href="#RequestList+initialize"><code>initialize</code></a> will always fetch the sources
+ from their origin, check that they are consistent with the restored state (if any)
+ and throw an error if they are not.</p>
+</td></tr><tr>
+<td><code>[state]</code></td><td><code><a href="../typedefs/requestliststate">RequestListState</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>The state object that the <code>RequestList</code> will be initialized from.
+ It is in the form as returned by <code>RequestList.getState()</code>, such as follows:</p>
+<pre><code>{
+ nextIndex: 5,
+ nextUniqueKey: &#39;unique-key-5&#39;
+ inProgress: {
+ &#39;unique-key-1&#39;: true,
+ &#39;unique-key-4&#39;: true,
+ },
+}</code></pre><p> Note that the preferred (and simpler) way to persist the state of crawling of the <code>RequestList</code>
+ is to use the <code>stateKeyPrefix</code> parameter instead.</p>
+</td></tr><tr>
+<td><code>[keepDuplicateUrls]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>By default, <code>RequestList</code> will deduplicate the provided URLs. Default deduplication is based
+ on the <code>uniqueKey</code> property of passed source <a href="request"><code>Request</code></a> objects.</p>
+<p> If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
+ In any case, only one request per <code>uniqueKey</code> is added to the <code>RequestList</code> resulting in removal
+ of duplicate URLs / unique keys.</p>
+<p> Setting <code>keepDuplicateUrls</code> to <code>true</code> will append an additional identifier to the <code>uniqueKey</code>
+ of each request that does not already include a <code>uniqueKey</code>. Therefore, duplicate
+ URLs will be kept in the list. It does not protect the user from having duplicates in user set
+ <code>uniqueKey</code>s however. It is the user&#39;s responsibility to ensure uniqueness of their unique keys
+ if they wish to keep more than just a single copy in the <code>RequestList</code>.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/RequestListState.md b/docs/typedefs/RequestListState.md
new file mode 100644
index 000000000..1590b472e
--- /dev/null
+++ b/docs/typedefs/RequestListState.md
@@ -0,0 +1,47 @@
+---
+id: requestliststate
+title: RequestListState
+---
+
+<a name="RequestListState"></a>
+
+Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed. You can obtain the state by calling
+[`RequestList.getState()`](<../api/requestlist#getState()>) and receive an object with the following structure:
+
+```
+{
+ nextIndex: 5,
+ nextUniqueKey: 'unique-key-5'
+ inProgress: {
+ 'unique-key-1': true,
+ 'unique-key-4': true
+ },
+}
+```
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>nextIndex</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Position of the next request to be processed.</p>
+</td></tr><tr>
+<td><code>nextUniqueKey</code></td><td><code>String</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Key of the next request to be processed.</p>
+</td></tr><tr>
+<td><code>inProgress</code></td><td><code>Object<String, Boolean></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/RequestOptions.md b/docs/typedefs/RequestOptions.md
new file mode 100644
index 000000000..964dd4767
--- /dev/null
+++ b/docs/typedefs/RequestOptions.md
@@ -0,0 +1,78 @@
+---
+id: requestoptions
+title: RequestOptions
+---
+
+<a name="RequestOptions"></a>
+
+Specifies required and optional fields for constructing a [`Request`](../api/request).
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>url</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>URL of the web page to crawl. It must be a non-empty string.</p>
+</td></tr><tr>
+<td><code>[uniqueKey]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A unique key identifying the request.
+ Two requests with the same <code>uniqueKey</code> are considered as pointing to the same URL.</p>
+<p> If <code>uniqueKey</code> is not provided, then it is automatically generated by normalizing the URL.
+ For example, the URL of <code>HTTP://www.EXAMPLE.com/something/</code> will produce the <code>uniqueKey</code>
+ of <code>http://www.example.com/something</code>.</p>
+<p> The <code>keepUrlFragment</code> option determines whether URL hash fragment is included in the <code>uniqueKey</code> or not.</p>
+<p> The <code>useExtendedUniqueKey</code> options determines whether method and payload are included in the <code>uniqueKey</code>,
+ producing a <code>uniqueKey</code> in the following format: <code>METHOD(payloadHash):normalizedUrl</code>. This is useful
+ when requests point to the same URL, but with different methods and payloads. For example: form submits.</p>
+<p> Pass an arbitrary non-empty text value to the <code>uniqueKey</code> property
+ to override the default behavior and specify which URLs shall be considered equal.</p>
+</td></tr><tr>
+<td><code>[method]</code></td><td><code>String</code></td><td><code>&#x27;GET&#x27;</code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>[payload]</code></td><td><code>String</code> | <code>Buffer</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p>
+</td></tr><tr>
+<td><code>[headers]</code></td><td><code>Object</code></td><td><code>{}</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>HTTP headers in the following format:</p>
+<pre><code> {
+ Accept: &#39;text/html&#39;,
+ &#39;Content-Type&#39;: &#39;application/json&#39;
+ }</code></pre></td></tr><tr>
+<td><code>[userData]</code></td><td><code>Object</code></td><td><code>{}</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom user data assigned to the request. Use this to save any request related data to the
+ request&#39;s scope, keeping them accessible on retries, failures etc.</p>
+</td></tr><tr>
+<td><code>[keepUrlFragment]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>false</code> then the hash part of a URL is removed when computing the <code>uniqueKey</code> property.
+ For example, this causes the <code>http://www.example.com#foo</code> and <code>http://www.example.com#bar</code> URLs
+ to have the same <code>uniqueKey</code> of <code>http://www.example.com</code> and thus the URLs are considered equal.
+ Note that this option only has an effect if <code>uniqueKey</code> is not set.</p>
+</td></tr><tr>
+<td><code>[useExtendedUniqueKey]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If <code>true</code> then the <code>uniqueKey</code> is computed not only from the URL, but also from the method and payload
+ properties. This is useful when making requests to the same URL that are differentiated by method
+ or payload, such as form submit navigations in browsers.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/RequestTransform.md b/docs/typedefs/RequestTransform.md
new file mode 100644
index 000000000..07c0a4bbd
--- /dev/null
+++ b/docs/typedefs/RequestTransform.md
@@ -0,0 +1,26 @@
+---
+id: requesttransform
+title: RequestTransform
+---
+
+<a name="RequestTransform"></a>
+
+Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used
+[`Apify.utils.enqueueLinks`](../api/utils#utils.enqueueLinks) to modify requests before enqueuing them.
+
+**Returns**: [`RequestOptions`](../typedefs/requestoptions) - The modified request options to enqueue.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>original</code></td><td><code><a href="../typedefs/requestoptions">RequestOptions</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Request options to be modified.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/SessionOptions.md b/docs/typedefs/SessionOptions.md
new file mode 100644
index 000000000..1accce233
--- /dev/null
+++ b/docs/typedefs/SessionOptions.md
@@ -0,0 +1,79 @@
+---
+id: sessionoptions
+title: SessionOptions
+---
+
+<a name="SessionOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>[id]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Id of session used for generating fingerprints. It is used as proxy session name.</p>
+</td></tr><tr>
+<td><code>[maxAgeSecs]</code></td><td><code>Number</code></td><td><code>3000</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Number of seconds after which the session is considered as expired.</p>
+</td></tr><tr>
+<td><code>userData</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Object where custom user data can be stored. For example custom headers.</p>
+</td></tr><tr>
+<td><code>[maxErrorScore]</code></td><td><code>number</code></td><td><code>3</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum number of marking session as blocked usage.
+ If the <code>errorScore</code> reaches the <code>maxErrorScore</code> session is marked as block and it is thrown away.
+ It starts at 0. Calling the <code>markBad</code> function increases the <code>errorScore</code> by 1.
+ Calling the <code>markGood</code> will decrease the <code>errorScore</code> by <code>errorScoreDecrement</code></p>
+</td></tr><tr>
+<td><code>[errorScoreDecrement]</code></td><td><code>number</code></td><td><code>0.5</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>It is used for healing the session.
+ For example: if your session is marked bad two times, but it is successful on the third attempt it&#39;s errorScore is decremented by this
+ number.</p>
+</td></tr><tr>
+<td><code>[createdAt]</code></td><td><code>Date</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Date of creation.</p>
+</td></tr><tr>
+<td><code>[expiresAt]</code></td><td><code>Date</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Date of expiration.</p>
+</td></tr><tr>
+<td><code>[usageCount]</code></td><td><code>Number</code></td><td><code>0</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how many times the session has been used.</p>
+</td></tr><tr>
+<td><code>[errorCount]</code></td><td><code>Number</code></td><td><code>0</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Indicates how many times the session is marked bad.</p>
+</td></tr><tr>
+<td><code>[maxUsageCount]</code></td><td><code>Number</code></td><td><code>50</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Session should be used only a limited amount of times.
+ This number indicates how many times the session is going to be used, before it is thrown away.</p>
+</td></tr><tr>
+<td><code>sessionPool</code></td><td><code><a href="sessionpool">SessionPool</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>SessionPool instance. Session will emit the <code>sessionRetired</code> event on this instance.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/SessionPoolOptions.md b/docs/typedefs/SessionPoolOptions.md
new file mode 100644
index 000000000..488b19a5e
--- /dev/null
+++ b/docs/typedefs/SessionPoolOptions.md
@@ -0,0 +1,45 @@
+---
+id: sessionpooloptions
+title: SessionPoolOptions
+---
+
+<a name="SessionPoolOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>[maxPoolSize]</code></td><td><code>Number</code></td><td><code>1000</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum size of the pool.
+Indicates how many sessions are rotated.</p>
+</td></tr><tr>
+<td><code>[sessionOptions]</code></td><td><code><a href="../typedefs/sessionoptions">SessionOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>The configuration options for {Session} instances.</p>
+</td></tr><tr>
+<td><code>[persistStateKeyValueStoreId]</code></td><td><code>String</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Name or Id of <code>KeyValueStore</code> where is the <code>SessionPool</code> state stored.</p>
+</td></tr><tr>
+<td><code>[persistStateKey]</code></td><td><code>String</code></td><td><code>&quot;SESSION_POOL_STATE&quot;</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Session pool persists it&#39;s state under this key in Key value store.</p>
+</td></tr><tr>
+<td><code>[createSessionFunction]</code></td><td><code><a href="../typedefs/createsession">CreateSession</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom function that should return <code>Session</code> instance.
+Function receives <code>SessionPool</code> instance as a parameter</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/SessionState.md b/docs/typedefs/SessionState.md
new file mode 100644
index 000000000..dc7f4860f
--- /dev/null
+++ b/docs/typedefs/SessionState.md
@@ -0,0 +1,60 @@
+---
+id: sessionstate
+title: SessionState
+---
+
+<a name="SessionState"></a>
+
+Persistable {Session} state.
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>id</code></td><td><code>String</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>cookies</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>cookieJar</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>userData</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>maxErrorScore</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>errorScoreDecrement</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>expiresAt</code></td><td><code>String</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>createdAt</code></td><td><code>String</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>usageCount</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>errorScore</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
diff --git a/docs/typedefs/SnapshotterOptions.md b/docs/typedefs/SnapshotterOptions.md
new file mode 100644
index 000000000..1be68c18c
--- /dev/null
+++ b/docs/typedefs/SnapshotterOptions.md
@@ -0,0 +1,75 @@
+---
+id: snapshotteroptions
+title: SnapshotterOptions
+---
+
+<a name="SnapshotterOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>[eventLoopSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>0.5</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the interval of measuring the event loop response time.</p>
+</td></tr><tr>
+<td><code>[clientSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the interval of checking the current state
+ of the remote API client.</p>
+</td></tr><tr>
+<td><code>[maxBlockedMillis]</code></td><td><code>Number</code></td><td><code>50</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Maximum allowed delay of the event loop in milliseconds.
+ Exceeding this limit overloads the event loop.</p>
+</td></tr><tr>
+<td><code>[cpuSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the interval of measuring CPU usage.
+ This is only used when running locally. On the Apify platform,
+ the statistics are provided externally at a fixed interval.</p>
+</td></tr><tr>
+<td><code>[maxUsedCpuRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the maximum usage of CPU.
+ Exceeding this limit overloads the CPU.</p>
+</td></tr><tr>
+<td><code>[memorySnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the interval of measuring memory consumption.
+ This is only used when running locally. On the Apify platform,
+ the statistics are provided externally at a fixed interval.
+ The measurement itself is resource intensive (25 - 50ms async).
+ Therefore, setting this interval below 1 second is not recommended.</p>
+</td></tr><tr>
+<td><code>[maxUsedMemoryRatio]</code></td><td><code>Number</code></td><td><code>0.7</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the maximum ratio of total memory that can be used.
+ Exceeding this limit overloads the memory.</p>
+</td></tr><tr>
+<td><code>[maxClientErrors]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the maximum number of new rate limit errors within
+ the given interval.</p>
+</td></tr><tr>
+<td><code>[snapshotHistorySecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the interval in seconds for which a history of resource snapshots
+ will be kept. Increasing this to very high numbers will affect performance.</p>
+</td></tr></tbody>
+</table>
diff --git a/docs/typedefs/SocialHandles.md b/docs/typedefs/SocialHandles.md
new file mode 100644
index 000000000..d75a05986
--- /dev/null
+++ b/docs/typedefs/SocialHandles.md
@@ -0,0 +1,67 @@
+---
+id: socialhandles
+title: SocialHandles
+---
+
+<a name="SocialHandles"></a>
+
+Representation of social handles parsed from a HTML page.
+
+The object has the following structure:
+
+```
+{
+ emails: String[],
+ phones: String[],
+ phonesUncertain: String[],
+ linkedIns: String[],
+ twitters: String[],
+ instagrams: String[],
+ facebooks: String[],
+ youtubes: String[],
+}
+```
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>emails</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>phones</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>phonesUncertain</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>linkedIns</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>twitters</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>instagrams</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>facebooks</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>youtubes</code></td><td><code>Array<String></code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
diff --git a/docs/typedefs/SystemInfo.md b/docs/typedefs/SystemInfo.md
new file mode 100644
index 000000000..dd074d7f8
--- /dev/null
+++ b/docs/typedefs/SystemInfo.md
@@ -0,0 +1,38 @@
+---
+id: systeminfo
+title: SystemInfo
+---
+
+<a name="SystemInfo"></a>
+
+Represents the current status of the system.
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>isSystemIdle</code></td><td><code>Boolean</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If true, system is being overloaded.</p>
+</td></tr><tr>
+<td><code>memInfo</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Memory</p>
+</td></tr><tr>
+<td><code>eventLoopInfo</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>cpuInfo</code></td><td><code>Object</code></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr></tbody>
+</table>
diff --git a/docs/typedefs/SystemStatusOptions.md b/docs/typedefs/SystemStatusOptions.md
new file mode 100644
index 000000000..697d84375
--- /dev/null
+++ b/docs/typedefs/SystemStatusOptions.md
@@ -0,0 +1,53 @@
+---
+id: systemstatusoptions
+title: SystemStatusOptions
+---
+
+<a name="SystemStatusOptions"></a>
+
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>[currentHistorySecs]</code></td><td><code>Number</code></td><td><code>5</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines max age of snapshots used in the
+ <a href="#SystemStatus+getCurrentStatus"><code>getCurrentStatus()</code></a> measurement.</p>
+</td></tr><tr>
+<td><code>[maxMemoryOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a memory sample.
+ If the sample exceeds this ratio, the system will be overloaded.</p>
+</td></tr><tr>
+<td><code>[maxEventLoopOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in an event loop sample.
+ If the sample exceeds this ratio, the system will be overloaded.</p>
+</td></tr><tr>
+<td><code>[maxCpuOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.4</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a CPU sample.
+ If the sample exceeds this ratio, the system will be overloaded.</p>
+</td></tr><tr>
+<td><code>[maxClientOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a Client sample.
+ If the sample exceeds this ratio, the system will be overloaded.</p>
+</td></tr><tr>
+<td><code>[snapshotter]</code></td><td><code><a href="snapshotter">Snapshotter</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>The <code>Snapshotter</code> instance to be queried for <code>SystemStatus</code>.</p>
+</td></tr></tbody>
+</table>
diff --git a/jest.config.js b/jest.config.js
new file mode 100644
index 000000000..bcf0df8f5
--- /dev/null
+++ b/jest.config.js
@@ -0,0 +1,11 @@
+const path = require('path');
+
+module.exports = {
+ verbose: true,
+ rootDir: path.join(__dirname, './'),
+ testMatch: [path.join(__dirname, '**/?(*.)+(spec|test).[tj]s?(x)')],
+ transform: {
+ '^.+\\.ts$': 'ts-jest',
+ '^.+\\.js$': 'babel-jest',
+ },
+};
diff --git a/package.json b/package.json
index dc318cdb7..d7dbea93a 100644
--- a/package.json
+++ b/package.json
@@ -6,6 +6,7 @@
"node": ">=10.17.0"
},
"main": "build/index.js",
+ "types": "types/index.esm.d.ts",
"keywords": [
"apify",
"headless",
@@ -34,13 +35,15 @@
},
"homepage": "https://sdk.apify.com/",
"files": [
- "build"
+ "build",
+ "types"
],
"scripts": {
"build": "rm -rf ./build && babel src --out-dir build --copy-files",
- "build-docs": "npm run clean && npm run build && node ./website/tools/build_docs.js",
+ "build-docs": "npm run clean && npm run build && node ./website/tools/build_docs.js && npm run build-types",
"build-readme": "node ./tools/build_readme.js",
- "test": "npm run build && jest --testTimeout=20000 --maxWorkers=3",
+ "build-types": "./node_modules/.bin/tsc -p tsconfig.json && node ./tools/typescript_fixes.js",
+ "test": "npm run build && npm run build-types && jest --testTimeout=20000 --maxWorkers=3",
"prepare": "npm run build",
"prepublishOnly": "(test $TRAVIS || (echo \"Publishing is reserved to the Travis CI!\"; exit 1))",
"clean": "rm -rf build",
@@ -50,6 +53,10 @@
"dependencies": {
"@apify/http-request": "^1.1.5",
"@apify/ps-tree": "^1.1.3",
+ "@types/cheerio": "^0.22.15",
+ "@types/node": "^13.1.8",
+ "@types/puppeteer": "^2.0.0",
+ "@types/tough-cookie": "^2.3.6",
"apify-client": "^0.5.25",
"apify-shared": "^0.1.66",
"cheerio": "^1.0.0-rc.3",
@@ -67,6 +74,7 @@
"semver": "^6.3.0",
"socket.io": "^2.2.0",
"tough-cookie": "^3.0.1",
+ "ts-jest": "^24.3.0",
"underscore": "^1.9.1",
"ws": "^6.2.1"
},
@@ -78,7 +86,6 @@
"@babel/register": "^7.0.0",
"@types/fs-extra": "^7.0.0",
"@types/jest": "^24.0.21",
- "@types/puppeteer": "^1.19.1",
"@types/sinon": "^7.0.13",
"@types/socket.io": "^2.1.2",
"basic-auth-parser": "^0.0.2",
@@ -96,6 +103,7 @@
"sinon": "^7.4.1",
"sinon-stub-promise": "^4.0.0",
"socket.io-client": "^2.2.0",
+ "typescript": "^3.7.4",
"ua-parser-js": "^0.7.20",
"why-is-node-running": "^2.0.3"
},
diff --git a/src/actor.js b/src/actor.js
index 621ffc37d..a72c6c037 100644
--- a/src/actor.js
+++ b/src/actor.js
@@ -10,6 +10,11 @@ import { apifyClient, addCharsetToContentType, sleep, snakeCaseToCamelCase, isAt
import { maybeStringify } from './key_value_store';
import { ApifyCallError } from './errors';
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { ActorRun } from './typedefs';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
+
const METAMORPH_AFTER_SLEEP_MILLIS = 300000;
/**
@@ -92,47 +97,37 @@ const addInputOptionsOrThrow = (input, contentType, options) => {
};
/**
- * Returns a new object which contains information parsed from all the `APIFY_XXX` environment variables.
- * It has properties such as the following:
+ * Parsed representation of the `APIFY_XXX` environmental variables.
+ *
+ * @typedef {Object} ApifyEnv
+ * @property {String|null} actorId ID of the actor (APIFY_ACTOR_ID)
+ * @property {String|null} actorRunId ID of the actor run (APIFY_ACTOR_RUN_ID)
+ * @property {String|null} actorTaskId ID of the actor task (APIFY_ACTOR_TASK_ID)
+ * @property {String|null} userId ID of the user who started the actor - note that it might be
+ * different than the owner ofthe actor (APIFY_USER_ID)
+ * @property {String|null} token Authentication token representing privileges given to the actor run,
+ * it can be passed to various Apify APIs (APIFY_TOKEN)
+ * @property {Date|null} startedAt Date when the actor was started (APIFY_STARTED_AT)
+ * @property {Date|null} timeoutAt Date when the actor will time out (APIFY_TIMEOUT_AT)
+ * @property {String|null} defaultKeyValueStoreId ID of the key-value store where input and output data of this
+ * actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
+ * @property {String|null} defaultDatasetId ID of the dataset where input and output data of this
+ * actor is stored (APIFY_DEFAULT_DATASET_ID)
+ * @property {Number|null} memoryMbytes Amount of memory allocated for the actor,
+ * in megabytes (APIFY_MEMORY_MBYTES)
+ */
+
+/**
+ * Returns a new {@link ApifyEnv} object which contains information parsed from all the `APIFY_XXX` environment variables.
*
- * ```javascript
- * {
- * // ID of the actor (APIFY_ACTOR_ID)
- * actorId: String,
- * // ID of the actor run (APIFY_ACTOR_RUN_ID)
- * actorRunId: String,
- * // ID of the actor task (APIFY_ACTOR_TASK_ID)
- * actorTaskId: String,
- * // ID of the user who started the actor - note that it might be
- * // different than the owner of the actor (APIFY_USER_ID)
- * userId: String,
- * // Authentication token representing privileges given to the actor run,
- * // it can be passed to various Apify APIs (APIFY_TOKEN).
- * token: String,
- * // Date when the actor was started (APIFY_STARTED_AT)
- * startedAt: Date,
- * // Date when the actor will time out (APIFY_TIMEOUT_AT)
- * timeoutAt: Date,
- * // ID of the key-value store where input and output data of this
- * // actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
- * defaultKeyValueStoreId: String,
- * // ID of the dataset where input and output data of this
- * // actor is stored (APIFY_DEFAULT_DATASET_ID)
- * defaultDatasetId: String,
- * // Amount of memory allocated for the actor,
- * // in megabytes (APIFY_MEMORY_MBYTES)
- * memoryMbytes: Number,
- * }
- * ```
* For the list of the `APIFY_XXX` environment variables, see
* <a href="https://docs.apify.com/actor/run#environment-variables" target="_blank">Actor documentation</a>.
* If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.
*
- * @returns {Object}
- *
* @memberof module:Apify
* @function
* @name getEnv
+ * @returns {ApifyEnv}
*/
export const getEnv = () => {
// NOTE: Don't throw if env vars are invalid to simplify local development and debugging of actors
@@ -567,7 +562,7 @@ export const callTask = async (taskId, input, options = {}) => {
* @param {String} [options.build]
* Tag or number of the target actor build to metamorph into (e.g. `beta` or `1.2.345`).
* If not provided, the run uses build tag or number from the default actor run configuration (typically `latest`).
- * @returns {Promise}
+ * @returns {Promise<void>}
*
* @memberof module:Apify
* @function
@@ -608,93 +603,6 @@ export const metamorph = async (targetActorId, input, options = {}) => {
await sleep(optionsCopy.customAfterSleepMillis || METAMORPH_AFTER_SLEEP_MILLIS);
};
-/**
- * Represents information about an actor run, as returned by the
- * [`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.
- * The object is almost equivalent to the JSON response
- * of the
- * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>
- * Apify API endpoint and extended with certain fields.
- * For more details, see
- * <a href="https://docs.apify.com/actor/run" target="_blank">Runs.</a>
- *
- * @typedef {Object} ActorRun
- * @property {String} id
- * Actor run ID
- * @property {String} actId
- * Actor ID
- * @property {Date} startedAt
- * Time when the actor run started
- * @property {Date} finishedAt
- * Time when the actor run finished. Contains `null` for running actors.
- * @property {String} status
- * Status of the run. For possible values, see
- * <a href="https://docs.apify.com/actor/run#lifecycle" target="_blank">Run lifecycle</a>
- * in Apify actor documentation.
- * @property {Object} meta
- * Actor run meta-data. For example:
- * ```
- * {
- * "origin": "API",
- * "clientIp": "1.2.3.4",
- * "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"
- * }
- * ```
- * @property {Object} stats
- * An object containing various actor run statistics. For example:
- * ```
- * {
- * "inputBodyLen": 22,
- * "restartCount": 0,
- * "workersUsed": 1,
- * }
- * ```
- * Beware that object fields might change in future releases.
- * @property {Object} options
- * Actor run options. For example:
- * ```
- * {
- * "build": "latest",
- * "waitSecs": 0,
- * "memoryMbytes": 256,
- * "diskMbytes": 512
- * }
- * ```
- * @property {String} buildId
- * ID of the actor build used for the run. For details, see
- * <a href="https://docs.apify.com/actor/build" target="_blank">Builds</a>
- * in Apify actor documentation.
- * @property {String} buildNumber
- * Number of the actor build used for the run. For example, `0.0.10`.
- * @property {Number} exitCode
- * Exit code of the actor run process. It's `null` if actor is still running.
- * @property {String} defaultKeyValueStoreId
- * ID of the default key-value store associated with the actor run. See [`KeyValueStore`](../api/keyvaluestore) for details.
- * @property {String} defaultDatasetId
- * ID of the default dataset associated with the actor run. See [`Dataset`](../api/dataset) for details.
- * @property {String} defaultRequestQueueId
- * ID of the default request queue associated with the actor run. See [`RequestQueue`](../api/requestqueue) for details.
- * @property {String} containerUrl
- * URL on which the web server running inside actor run's Docker container can be accessed.
- * For more details, see
- * <a href="https://docs.apify.com/actor/run#container-web-server" target="_blank">Container web server</a>
- * in Apify actor documentation.
- * @property {Object} output
- * Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,
- * or if you pass `false` to the `fetchOutput` option of [`Apify.call()`](../api/apify#module_Apify.call).
- *
- * For example:
- * ```
- * {
- * "contentType": "application/json; charset=utf-8",
- * "body": {
- * "message": "Hello world!"
- * }
- * }
- * ```
- */
-
-
/**
* Constructs an Apify Proxy URL using the specified settings.
* The proxy URL can be used from Apify actors, web browsers or any other HTTP
diff --git a/src/autoscaling/autoscaled_pool.js b/src/autoscaling/autoscaled_pool.js
index 2ad7b97c3..ee8b57bbe 100644
--- a/src/autoscaling/autoscaled_pool.js
+++ b/src/autoscaling/autoscaled_pool.js
@@ -2,8 +2,8 @@ import _ from 'underscore';
import { betterSetInterval, betterClearInterval } from 'apify-shared/utilities';
import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import Snapshotter from './snapshotter';
-import SystemStatus from './system_status';
+import Snapshotter, { SnapshotterOptions } from './snapshotter'; // eslint-disable-line import/named,no-unused-vars
+import SystemStatus, { SystemStatusOptions } from './system_status'; // eslint-disable-line import/named,no-unused-vars
const DEFAULT_OPTIONS = {
maxConcurrency: 1000,
@@ -16,6 +16,64 @@ const DEFAULT_OPTIONS = {
autoscaleIntervalSecs: 10,
};
+
+/**
+ * @typedef {Object} AutoscaledPoolOptions
+ * @property {Function} runTaskFunction
+ * A function that performs an asynchronous resource-intensive task.
+ * The function must either be labeled `async` or return a promise.
+ *
+ * @property {Function} isTaskReadyFunction
+ * A function that indicates whether `runTaskFunction` should be called.
+ * This function is called every time there is free capacity for a new task and it should
+ * indicate whether it should start a new task or not by resolving to either `true` or `false`.
+ * Besides its obvious use, it is also useful for task throttling to save resources.
+ *
+ * @property {Function} isFinishedFunction
+ * A function that is called only when there are no tasks to be processed.
+ * If it resolves to `true` then the pool's run finishes. Being called only
+ * when there are no tasks being processed means that as long as `isTaskReadyFunction()`
+ * keeps resolving to `true`, `isFinishedFunction()` will never be called.
+ * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.
+ *
+ * @property {Number} [minConcurrency=1]
+ * The minimum number of tasks running in parallel.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {Number} [maxConcurrency=1000]
+ * The maximum number of tasks running in parallel.
+ * @property {Number} [desiredConcurrency]
+ * The desired number of tasks that should be running parallel on the start of the pool,
+ * if there is a large enough supply of them.
+ * By default, it is `minConcurrency`.
+ * @property {Number} [desiredConcurrencyRatio=0.95]
+ * Minimum level of desired concurrency to reach before more scaling up is allowed.
+ * @property {Number} [scaleUpStepRatio=0.05]
+ * Defines the fractional amount of desired concurrency to be added with each scaling up.
+ * The minimum scaling step is one.
+ * @property {Number} [scaleDownStepRatio=0.05]
+ * Defines the amount of desired concurrency to be subtracted with each scaling down.
+ * The minimum scaling step is one.
+ * @property {Number} [maybeRunIntervalSecs=0.5]
+ * Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds.
+ * This has no effect on starting new tasks immediately after a task completes.
+ * @property {Number} [loggingIntervalSecs=60]
+ * Specifies a period in which the instance logs its state, in seconds.
+ * Set to `null` to disable periodic logging.
+ * @property {Number} [autoscaleIntervalSecs=10]
+ * Defines in seconds how often the pool should attempt to adjust the desired concurrency
+ * based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
+ * We suggest using a value from 5 to 20.
+ * @property {SnapshotterOptions} [snapshotterOptions]
+ * Options to be passed down to the {@link Snapshotter} constructor. This is useful for fine-tuning
+ * the snapshot intervals and history.
+ * @property {SystemStatusOptions} [systemStatusOptions]
+ * Options to be passed down to the {@link SystemStatus} constructor. This is useful for fine-tuning
+ * the system status reports. If a custom snapshotter is set in the options, it will be used
+ * by the pool.
+ */
+
/**
* Manages a pool of asynchronous resource-intensive tasks that are executed in parallel.
* The pool only starts new tasks if there is enough free CPU and memory available
@@ -62,64 +120,11 @@ const DEFAULT_OPTIONS = {
*
* await pool.run();
* ```
- *
- * @param {Object} options All `AutoscaledPool` parameters are passed
- * via an options object with the following keys.
- * @param {Function} options.runTaskFunction
- * A function that performs an asynchronous resource-intensive task.
- * The function must either be labeled `async` or return a promise.
- *
- * @param {Function} options.isTaskReadyFunction
- * A function that indicates whether `runTaskFunction` should be called.
- * This function is called every time there is free capacity for a new task and it should
- * indicate whether it should start a new task or not by resolving to either `true` or `false.
- * Besides its obvious use, it is also useful for task throttling to save resources.
- *
- * @param {Function} options.isFinishedFunction
- * A function that is called only when there are no tasks to be processed.
- * If it resolves to `true` then the pool's run finishes. Being called only
- * when there are no tasks being processed means that as long as `isTaskReadyFunction()`
- * keeps resolving to `true`, `isFinishedFunction()` will never be called.
- * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.
- *
- * @param {Number} [options.minConcurrency=1]
- * The minimum number of tasks running in parallel.
- *
- * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
- * If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @param {Number} [options.maxConcurrency=1000]
- * The maximum number of tasks running in parallel.
- * @param {Number} [options.desiredConcurrency]
- * The desired number of tasks that should be running parallel on the start of the pool,
- * if there is a large enough supply of them.
- * By default, it is `options.minConcurrency`.
- * @param {Number} [options.desiredConcurrencyRatio=0.95]
- * Minimum level of desired concurrency to reach before more scaling up is allowed.
- * @param {Number} [options.scaleUpStepRatio=0.05]
- * Defines the fractional amount of desired concurrency to be added with each scaling up.
- * The minimum scaling step is one.
- * @param {Number} [options.scaleDownStepRatio=0.05]
- * Defines the amount of desired concurrency to be subtracted with each scaling down.
- * The minimum scaling step is one.
- * @param {Number} [options.maybeRunIntervalSecs=0.5]
- * Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds.
- * This has no effect on starting new tasks immediately after a task completes.
- * @param {Number} [options.loggingIntervalSecs=60]
- * Specifies a period in which the instance logs its state, in seconds.
- * Set to `null` to disable periodic logging.
- * @param {Number} [options.autoscaleIntervalSecs=10]
- * Defines in seconds how often the pool should attempt to adjust the desired concurrency
- * based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
- * We suggest using a value from 5 to 20.
- * @param {Number} [options.snapshotterOptions]
- * Options to be passed down to the {@link Snapshotter} constructor. This is useful for fine-tuning
- * the snapshot intervals and history.
- * @param {Number} [options.systemStatusOptions]
- * Options to be passed down to the {@link SystemStatus} constructor. This is useful for fine-tuning
- * the system status reports. If a custom snapshotter is set in the options, it will be used
- * by the pool.
*/
class AutoscaledPool {
+ /**
+ * @param {AutoscaledPoolOptions} options All `AutoscaledPool` configuration options.
+ */
constructor(options = {}) {
const {
maxConcurrency,
diff --git a/src/autoscaling/snapshotter.js b/src/autoscaling/snapshotter.js
index 5315cfd44..db9bc6933 100644
--- a/src/autoscaling/snapshotter.js
+++ b/src/autoscaling/snapshotter.js
@@ -22,6 +22,40 @@ const RESERVE_MEMORY_RATIO = 0.5;
const CLIENT_RATE_LIMIT_ERROR_RETRY_COUNT = 2;
const CRITICAL_OVERLOAD_RATE_LIMIT_MILLIS = 10000;
+/**
+ * @typedef SnapshotterOptions
+ * @property {Number} [eventLoopSnapshotIntervalSecs=0.5]
+ * Defines the interval of measuring the event loop response time.
+ * @property {Number} [clientSnapshotIntervalSecs=1]
+ * Defines the interval of checking the current state
+ * of the remote API client.
+ * @property {Number} [maxBlockedMillis=50]
+ * Maximum allowed delay of the event loop in milliseconds.
+ * Exceeding this limit overloads the event loop.
+ * @property {Number} [cpuSnapshotIntervalSecs=1]
+ * Defines the interval of measuring CPU usage.
+ * This is only used when running locally. On the Apify platform,
+ * the statistics are provided externally at a fixed interval.
+ * @property {Number} [maxUsedCpuRatio=0.95]
+ * Defines the maximum usage of CPU.
+ * Exceeding this limit overloads the CPU.
+ * @property {Number} [memorySnapshotIntervalSecs=1]
+ * Defines the interval of measuring memory consumption.
+ * This is only used when running locally. On the Apify platform,
+ * the statistics are provided externally at a fixed interval.
+ * The measurement itself is resource intensive (25 - 50ms async).
+ * Therefore, setting this interval below 1 second is not recommended.
+ * @property {Number} [maxUsedMemoryRatio=0.7]
+ * Defines the maximum ratio of total memory that can be used.
+ * Exceeding this limit overloads the memory.
+ * @property {Number} [maxClientErrors=1]
+ * Defines the maximum number of new rate limit errors within
+ * the given interval.
+ * @property {Number} [snapshotHistorySecs=60]
+ * Sets the interval in seconds for which a history of resource snapshots
+ * will be kept. Increasing this to very high numbers will affect performance.
+ */
+
/**
* Creates snapshots of system resources at given intervals and marks the resource
* as either overloaded or not during the last interval. Keeps a history of the snapshots.
@@ -45,41 +79,11 @@ const CRITICAL_OVERLOAD_RATE_LIMIT_MILLIS = 10000;
*
* Client becomes overloaded when rate limit errors (429 - Too Many Requests),
* typically received from the request queue, exceed the set limit within the set interval.
- *
- * @param {Object} [options] All `Snapshotter` parameters are passed
- * via an options object with the following keys:
- * @param {Number} [options.eventLoopSnapshotIntervalSecs=0.5]
- * Defines the interval of measuring the event loop response time.
- * @param {Number} [options.clientSnapshotIntervalSecs=1]
- * Defines the interval of checking the current state
- * of the remote API client.
- * @param {Number} [options.maxBlockedMillis=50]
- * Maximum allowed delay of the event loop in milliseconds.
- * Exceeding this limit overloads the event loop.
- * @param {Number} [options.cpuSnapshotIntervalSecs=1]
- * Defines the interval of measuring CPU usage.
- * This is only used when running locally. On the Apify platform,
- * the statistics are provided externally at a fixed interval.
- * @param {Number} [options.maxUsedCpuRatio=0.95]
- * Defines the maximum usage of CPU.
- * Exceeding this limit overloads the CPU.
- * @param {Number} [options.memorySnapshotIntervalSecs=1]
- * Defines the interval of measuring memory consumption.
- * This is only used when running locally. On the Apify platform,
- * the statistics are provided externally at a fixed interval.
- * The measurement itself is resource intensive (25 - 50ms async).
- * Therefore, setting this interval below 1 second is not recommended.
- * @param {Number} [options.maxUsedMemoryRatio=0.7]
- * Defines the maximum ratio of total memory that can be used.
- * Exceeding this limit overloads the memory.
- * @param {Number} [options.maxClientErrors=1]
- * Defines the maximum number of new rate limit errors within
- * the given interval.
- * @param {Number} [options.snapshotHistorySecs=60]
- * Sets the interval in seconds for which a history of resource snapshots
- * will be kept. Increasing this to very high numbers will affect performance.
*/
class Snapshotter {
+ /**
+ * @param {SnapshotterOptions} [options] All `Snapshotter` configuration options.
+ */
constructor(options = {}) {
const {
eventLoopSnapshotIntervalSecs,
@@ -400,7 +404,6 @@ class Snapshotter {
* overloading.
*
* @param intervalCallback
- * @return {number}
* @private
*/
_snapshotClient(intervalCallback) {
diff --git a/src/autoscaling/system_status.js b/src/autoscaling/system_status.js
index ce594ab9d..a8b8f8c33 100644
--- a/src/autoscaling/system_status.js
+++ b/src/autoscaling/system_status.js
@@ -1,8 +1,14 @@
-import _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import Snapshotter from './snapshotter';
+import _ from 'underscore';
+import Snapshotter from './snapshotter'; // eslint-disable-line import/no-duplicates
import { weightedAvg } from '../utils';
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { SnapshotterOptions } from './snapshotter';
+/* eslint-anable no-unused-vars,import/named,import/no-duplicates,import/order */
+
+
const DEFAULT_OPTIONS = {
currentHistorySecs: 5,
maxMemoryOverloadedRatio: 0.2,
@@ -11,6 +17,40 @@ const DEFAULT_OPTIONS = {
maxClientOverloadedRatio: 0.3,
};
+// TODO yin: Add `@property clientInfo` as in `SystemStatus._isSystemIdle()`
+/**
+ * Represents the current status of the system.
+ *
+ * @typedef {Object} SystemInfo
+ * @property {Boolean} isSystemIdle
+ * If true, system is being overloaded.
+ * @property {Object} memInfo
+ * Memory
+ * @property {Object} eventLoopInfo
+ * @property {Object} cpuInfo
+ */
+
+/**
+ * @typedef {Object} SystemStatusOptions
+ * @property {Number} [currentHistorySecs=5]
+ * Defines max age of snapshots used in the
+ * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.
+ * @property {Number} [maxMemoryOverloadedRatio=0.2]
+ * Sets the maximum ratio of overloaded snapshots in a memory sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Number} [maxEventLoopOverloadedRatio=0.2]
+ * Sets the maximum ratio of overloaded snapshots in an event loop sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Number} [maxCpuOverloadedRatio=0.4]
+ * Sets the maximum ratio of overloaded snapshots in a CPU sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Number} [maxClientOverloadedRatio=0.2]
+ * Sets the maximum ratio of overloaded snapshots in a Client sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Snapshotter} [snapshotter]
+ * The `Snapshotter` instance to be queried for `SystemStatus`.
+ */
+
/**
* Provides a simple interface to reading system status from a {@link Snapshotter} instance.
* It only exposes two functions [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)
@@ -31,26 +71,11 @@ const DEFAULT_OPTIONS = {
* returns a boolean that represents the long-term status
* of the system. It considers the full snapshot history available
* in the {@link Snapshotter} instance.
- *
- * @param {Object} [options] All `SystemStatus` parameters are passed
- * via an options object with the following keys:
- * @param {Number} [options.currentHistorySecs=5]
- * Defines max age of snapshots used in the
- * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.
- * @param {Number} [options.maxMemoryOverloadedRatio=0.2]
- * Sets the maximum ratio of overloaded snapshots in a memory sample.
- * If the sample exceeds this ratio, the system will be overloaded.
- * @param {Number} [options.maxEventLoopOverloadedRatio=0.2]
- * Sets the maximum ratio of overloaded snapshots in an event loop sample.
- * If the sample exceeds this ratio, the system will be overloaded.
- * @param {Number} [options.maxCpuOverloadedRatio=0.4]
- * Sets the maximum ratio of overloaded snapshots in a CPU sample.
- * If the sample exceeds this ratio, the system will be overloaded.
- * @param {Number} [options.maxClientOverloadedRatio=0.2]
- * Sets the maximum ratio of overloaded snapshots in a Client sample.
- * If the sample exceeds this ratio, the system will be overloaded.
*/
class SystemStatus {
+ /**
+ * @param {SystemStatusOptions} [options] All `SystemStatus` configuration options.
+ */
constructor(options = {}) {
const {
currentHistorySecs,
@@ -73,12 +98,11 @@ class SystemStatus {
this.maxEventLoopOverloadedRatio = maxEventLoopOverloadedRatio;
this.maxCpuOverloadedRatio = maxCpuOverloadedRatio;
this.maxClientOverloadedRatio = maxClientOverloadedRatio;
-
this.snapshotter = snapshotter || new Snapshotter();
}
/**
- * Returns an object with the following structure:
+ * Returns an {@link SystemInfo} object with the following structure:
*
* ```javascript
* {
@@ -92,14 +116,14 @@ class SystemStatus {
* Where the `isSystemIdle` property is set to `false` if the system
* has been overloaded in the last `options.currentHistorySecs` seconds,
* and `true` otherwise.
- * @return {Object}
+ * @return {SystemInfo}
*/
getCurrentStatus() {
return this._isSystemIdle(this.currentHistorySecs);
}
/**
- * Returns an object with the following structure:
+ * Returns an {@link SystemInfo} object with the following structure:
*
* ```javascript
* {
@@ -123,7 +147,7 @@ class SystemStatus {
* Returns a system status object.
*
* @param {Number} [sampleDurationMillis]
- * @return {Boolean}
+ * @return {SystemInfo}
* @ignore
*/
_isSystemIdle(sampleDurationMillis) {
diff --git a/src/constants.js b/src/constants.js
index cd580b00c..83fcdb628 100644
--- a/src/constants.js
+++ b/src/constants.js
@@ -19,6 +19,9 @@ export const EXIT_CODES = {
/**
* These events are just internal for Apify package, so we don't need them in apify-shared package.
+ *
+ * @type {{CPU_INFO: String, SYSTEM_INFO: String, MIGRATING: String, PERSIST_STATE: String}}
+ *
*/
export const ACTOR_EVENT_NAMES_EX = Object.assign({}, ACTOR_EVENT_NAMES, { PERSIST_STATE: 'persistState' });
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index d8f433759..12d9085fb 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -1,16 +1,23 @@
import { checkParamOrThrow } from 'apify-client/build/utils';
-import _ from 'underscore';
-import log from 'apify-shared/log';
import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';
+import log from 'apify-shared/log';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
-import AutoscaledPool from '../autoscaling/autoscaled_pool';
+import _ from 'underscore';
+import AutoscaledPool from '../autoscaling/autoscaled_pool'; // eslint-disable-line import/no-duplicates
import { RequestList } from '../request_list';
import { RequestQueue, RequestQueueLocal } from '../request_queue';
import events from '../events';
-import { addTimeoutToPromise } from '../utils';
+import { openSessionPool } from '../session_pool/session_pool'; // eslint-disable-line import/no-duplicates
import Statistics from './statistics';
-import { openSessionPool } from '../session_pool/session_pool';
-import Request from '../request'; // eslint-disable-line no-unused-vars
+import { addTimeoutToPromise } from '../utils';
+
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool';
+import Request from '../request';
+import { Session } from '../session_pool/session';
+import { SessionPoolOptions } from '../session_pool/session_pool';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
/**
* Since there's no set number of seconds before the container is terminated after
@@ -37,6 +44,78 @@ const DEFAULT_OPTIONS = {
useSessionPool: false,
};
+/**
+ * @typedef {Object} BasicCrawlerOptions
+ * @property {HandleRequest} handleRequestFunction
+ * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * autoscaledPool: AutoscaledPool
+ * }
+ * ```
+ * where the {@link Request} instance represents the URL to crawl.
+ *
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {number} [handleRequestTimeoutSecs=60]
+ * Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
+ * for the default implementation of this function.
+ * @property {Number} [maxRequestRetries=3]
+ * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.
+ * @property {Number} [maxRequestsPerCrawl]
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
+ * @property {Number} [minConcurrency=1]
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {Number} [maxConcurrency=1000]
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @property {Boolean} [useSessionPool=false]
+ * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
+ * The session instance will be than available in the `handleRequestFunction`.
+ * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {SessionPool} to use.
+ */
+
/**
* Provides a simple framework for parallel crawling of web pages.
* The URLs to crawl are fed either from a static list of URLs
@@ -94,79 +173,11 @@ const DEFAULT_OPTIONS = {
*
* await crawler.run();
* ```
- *
- * @param {Object} options All `BasicCrawler` parameters are passed
- * via an options object with the following keys:
- * @param {Function} options.handleRequestFunction
- * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
- *
- * The function receives the following object as an argument:
- * ```
- * {
- * request: Request,
- * autoscaledPool: AutoscaledPool
- * }
- * ```
- * where the {@link Request} instance represents the URL to crawl.
- *
- * The function must return a promise, which is then awaited by the crawler.
- *
- * If the function throws an exception, the crawler will try to re-crawl the
- * request later, up to `option.maxRequestRetries` times.
- * If all the retries fail, the crawler calls the function
- * provided to the `options.handleFailedRequestFunction` parameter.
- * To make this work, you should **always**
- * let your function throw exceptions rather than catch them.
- * The exceptions are logged to the request using the
- * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
- * @param {RequestList} options.requestList
- * Static list of URLs to be processed.
- * Either `requestList` or `requestQueue` option must be provided (or both).
- * @param {RequestQueue} options.requestQueue
- * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * Either `requestList` or `requestQueue` option must be provided (or both).
- * @param {number} [options.handleRequestTimeoutSecs=60]
- * Timeout in which the function passed as `options.handleRequestFunction` needs to finish, in seconds.
- * @param {Function} [options.handleFailedRequestFunction]
- * A function to handle requests that failed more than `option.maxRequestRetries` times.
- *
- * The function receives the following object as an argument:
- * ```
- * {
- * request: Request,
- * error: Error,
- * }
- * ```
- * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
- * represents the last error thrown during processing of the request.
- *
- * See
- * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
- * for the default implementation of this function.
- * @param {Number} [options.maxRequestRetries=3]
- * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.
- * @param {Number} [options.maxRequestsPerCrawl]
- * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
- * Always set this value in order to prevent infinite loops in misconfigured crawlers.
- * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
- * @param {Object} [options.autoscaledPoolOptions]
- * Custom options passed to the underlying {@link AutoscaledPool} constructor.
- * Note that the `runTaskFunction` and `isTaskReadyFunction` options
- * are provided by `BasicCrawler` and cannot be overridden.
- * However, you can provide a custom implementation of `isFinishedFunction`.
- * @param {Object} [options.minConcurrency=1]
- * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- *
- * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
- * If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @param {Object} [options.maxConcurrency=1000]
- * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @param {Boolean} [options.useSessionPool=false]
- * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
- * The session instance will be than available in the `handleRequestFunction`.
- * @param sessionPoolOptions {Object} The [`new SessionPool`](sessionpool#new_SessionPool_new) options
*/
class BasicCrawler {
+ /**
+ * @param {BasicCrawlerOptions} options
+ */
constructor(options) {
const {
requestList,
@@ -266,7 +277,7 @@ class BasicCrawler {
/**
* Runs the crawler. Returns a promise that gets resolved once all the requests are processed.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async run() {
if (this.isRunningPromise) return this.isRunningPromise;
@@ -472,7 +483,7 @@ class BasicCrawler {
* list will first be dumped into the queue and then left
* empty.
*
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
async _loadHandledRequestCount() {
@@ -485,3 +496,26 @@ class BasicCrawler {
}
export default BasicCrawler;
+
+/**
+ * @callback HandleRequest
+ * @param {HandleRequestInputs} inputs Arguments passed to this callback.
+ * @returns {Promise<void>}
+ */
+/**
+ * @typedef HandleRequestInputs
+ * @property {Request} request The original {Request} object.
+ * @property {AutoscaledPool} autoscaledPool
+ * @property {Session} [session]
+ */
+
+/**
+ * @callback HandleFailedRequest
+ * @param {HandleFailedRequestInput} inputs Arguments passed to this callback.
+ * @returns {void|Promise<void>}
+ */
+/**
+ * @typedef HandleFailedRequestInput
+ * @property {Request} request The original {Request} object.
+ * @property {Error} error The Error thrown by `handleRequestFunction`.
+ */
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 8dc6dc9d5..bdb879f45 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -1,19 +1,33 @@
/* eslint-disable class-methods-use-this */
-import _ from 'underscore';
+import log from 'apify-shared/log';
+import { checkParamOrThrow } from 'apify-client/build/utils';
+import { readStreamToString, concatStreamToBuffer } from 'apify-shared/streams_utilities';
import cheerio from 'cheerio';
+import contentTypeParser from 'content-type';
import htmlparser from 'htmlparser2';
import * as iconv from 'iconv-lite';
+import _ from 'underscore';
import util from 'util';
-import log from 'apify-shared/log';
-import { checkParamOrThrow } from 'apify-client/build/utils';
-import contentTypeParser from 'content-type';
-import { readStreamToString, concatStreamToBuffer } from 'apify-shared/streams_utilities';
-import BasicCrawler from './basic_crawler';
-import { addTimeoutToPromise, parseContentTypeFromResponse } from '../utils';
import { getApifyProxyUrl } from '../actor';
import { BASIC_CRAWLER_TIMEOUT_MULTIPLIER } from '../constants';
-import * as utilsRequest from '../utils_request';
import { TimeoutError } from '../errors';
+import { addTimeoutToPromise, parseContentTypeFromResponse } from '../utils';
+import * as utilsRequest from '../utils_request'; // eslint-disable-line import/no-duplicates
+import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
+
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { IncomingMessage } from 'http';
+import AutoscaledPool, { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool';
+import { HandleFailedRequest } from './basic_crawler';
+import Request, { RequestOptions } from '../request';
+import { RequestList } from '../request_list';
+import { RequestQueue } from '../request_queue';
+import { Session } from '../session_pool/session';
+import { SessionPoolOptions } from '../session_pool/session_pool';
+import { Cheerio } from '../typedefs';
+import { RequestAsBrowserOptions } from '../utils_request';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
/**
* Default mime types, which CheerioScraper supports.
@@ -46,79 +60,8 @@ const DEFAULT_OPTIONS = {
};
/**
- * Provides a framework for the parallel crawling of web pages using plain HTTP requests and
- * <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.
- * The URLs to crawl are fed either from a static list of URLs
- * or from a dynamic queue of URLs enabling recursive crawling of websites.
- *
- * Since `CheerioCrawler` uses raw HTTP requests to download web pages,
- * it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript
- * to display the content, you might need to use {@link PuppeteerCrawler} instead,
- * because it loads the pages using full-featured headless Chrome browser.
- *
- * `CheerioCrawler` downloads each URL using a plain HTTP request,
- * parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>
- * and then invokes the user-provided [`handlePageFunction()`](#new_CheerioCrawler_new) to extract page data
- * using a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface to the parsed HTML DOM.
- *
- * The source URLs are represented using {@link Request} objects that are fed from
- * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_CheerioCrawler_new)
- * or [`requestQueue`](#new_CheerioCrawler_new) constructor options, respectively.
- *
- * If both [`requestList`](#new_CheerioCrawler_new) and [`requestQueue`](#new_CheerioCrawler_new) are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
- *
- * The crawler finishes when there are no more {@link Request} objects to crawl.
- *
- * By default, `CheerioCrawler` downloads HTML using the
- * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> NPM package.
- * You can use the `requestOptions` parameter to pass additional options to `request`.
- *
- * New requests are only dispatched when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
- * parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
- * {@link AutoscaledPool} options are available directly in the `CheerioCrawler` constructor.
- *
- * **Example usage:**
- *
- * ```javascript
- * // Prepare a list of URLs to crawl
- * const requestList = new Apify.RequestList({
- * sources: [
- * { url: 'http://www.example.com/page-1' },
- * { url: 'http://www.example.com/page-2' },
- * ],
- * });
- * await requestList.initialize();
- *
- * // Crawl the URLs
- * const crawler = new Apify.CheerioCrawler({
- * requestList,
- * handlePageFunction: async ({ request, response, body, contentType, $ }) => {
- * const data = [];
- *
- * // Do some data extraction from the page with Cheerio.
- * $('.some-collection').each((index, el) => {
- * data.push({ title: $(el).find('.some-title').text() });
- * });
- *
- * // Save the data to dataset.
- * await Apify.pushData({
- * url: request.url,
- * html: body,
- * data,
- * })
- * },
- * });
- *
- * await crawler.run();
- * ```
- *
- * @param {Object} options All `CheerioCrawler` parameters are passed
- * via an options object with the following keys:
- * @param {Function} options.handlePageFunction
+ * @typedef {Object} CheerioCrawlerOptions
+ * @property {CheerioHandlePage} handlePageFunction
* User-provided function that performs the logic of the crawler. It is called for each page
* loaded and parsed by the crawler.
*
@@ -149,29 +92,29 @@ const DEFAULT_OPTIONS = {
*
* With the {@link Request} object representing the URL to crawl.
*
- * If the function returns a promise, it is awaited by the crawler.
+ * If the function returns, the returned promise is awaited by the crawler.
*
* If the function throws an exception, the crawler will try to re-crawl the
* request later, up to `option.maxRequestRetries` times.
* If all the retries fail, the crawler calls the function
- * provided to the `options.handleFailedRequestFunction` parameter.
+ * provided to the `handleFailedRequestFunction` parameter.
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
* [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
- * @param {RequestList} options.requestList
+ * @property {RequestList} [requestList]
* Static list of URLs to be processed.
* Either `requestList` or `requestQueue` option must be provided (or both).
- * @param {RequestQueue} options.requestQueue
+ * @property {RequestQueue} [requestQueue]
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
* Either `requestList` or `requestQueue` option must be provided (or both).
- * @param {Object} [options.requestOptions]
+ * @property {RequestAsBrowserOptions} [requestOptions]
* Represents the options passed to
* <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.
* Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure
* the operation of `CheerioCrawler` and all its options. Headers will not be merged,
* use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the
- * correct headers or use `options.prepareRequestFunction` to modify your {@link Request} dynamically.
+ * correct headers or use `prepareRequestFunction` to modify your {@link Request} dynamically.
* If you need more granular control over your requests, use {@link BasicCrawler}.
*
* The mandatory internal defaults that **CANNOT BE OVERRIDDEN** by `requestOptions`:
@@ -181,11 +124,11 @@ const DEFAULT_OPTIONS = {
* method, // Provided by RequestList and/or RequestQueue
* headers, // Provided by RequestList and/or RequestQueue
* payload, // Provided by RequestList and/or RequestQueue
- * strictSSL, // Use options.ignoreSslErrors
- * proxy, // Use options.useApifyProxy or options.proxyUrls
+ * strictSSL, // Use ignoreSslErrors
+ * proxy, // Use useApifyProxy or proxyUrls
* }
* ```
- * @param {Function} [options.prepareRequestFunction]
+ * @property {PrepareRequest} [prepareRequestFunction]
* A function that executes before the HTTP request is made to the target resource.
* This function is suitable for setting dynamic properties such as cookies to the {@link Request}.
*
@@ -196,39 +139,40 @@ const DEFAULT_OPTIONS = {
* session: Session
* }
* ```
- * where the {@link Request} instance corresponds to the initialized request and the {@link Session} instance corresponds to used session.
+ * where the {@link Request} instance corresponds to the initialized request
+ * and the {@link Session} instance corresponds to used session.
*
* The function should modify the properties of the passed {@link Request} instance
* in place because there are already earlier references to it. Making a copy and returning it from
* this function is therefore not supported, because it would create inconsistencies where
* different parts of SDK would have access to a different {@link Request} instance.
*
- * @param {Number} [options.handlePageTimeoutSecs=60]
- * Timeout in which the function passed as `options.handlePageFunction` needs to finish, given in seconds.
- * @param {Number} [options.requestTimeoutSecs=30]
+ * @property {Number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
+ * @property {Number} [requestTimeoutSecs=30]
* Timeout in which the HTTP request to the resource needs to finish, given in seconds.
- * @param {Boolean} [options.ignoreSslErrors=true]
+ * @property {Boolean} [ignoreSslErrors=true]
* If set to true, SSL certificate errors will be ignored.
- * @param {Boolean} [options.useApifyProxy=false]
+ * @property {Boolean} [useApifyProxy=false]
* If set to `true`, `CheerioCrawler` will be configured to use
* <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
* For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>
- * @param {String[]} [options.apifyProxyGroups]
+ * @property {String[]} [apifyProxyGroups]
* An array of proxy groups to be used
* by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
* Only applied if the `useApifyProxy` option is `true`.
- * @param {String} [options.apifyProxySession]
+ * @property {String} [apifyProxySession]
* Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.
* All HTTP requests going through the proxy with the same session identifier
* will use the same target proxy server (i.e. the same IP address).
* The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
* Only applied if the `useApifyProxy` option is `true`.
- * @param {String[]} [options.proxyUrls]
+ * @property {String[]} [proxyUrls]
* An array of custom proxy URLs to be used by the `CheerioCrawler` instance.
* The provided custom proxies' order will be randomized and the resulting list rotated.
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
* configuration options will cause an error to be thrown on startup.
- * @param {Function} [options.handleFailedRequestFunction]
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
* A function to handle requests that failed more than `option.maxRequestRetries` times.
*
* The function receives the following object as an argument:
@@ -243,41 +187,116 @@ const DEFAULT_OPTIONS = {
*
* See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
* for the default implementation of this function.
- * @param {String[]} [options.additionalMimeTypes]
+ * @property {String[]} [additionalMimeTypes]
* An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
* target="_blank">mime types</a> you want to process.
* By default `text/html`, `application/xhtml+xml` mime types are supported.
- * @param {Number} [options.maxRequestRetries=3]
+ * @property {Number} [maxRequestRetries=3]
* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
- * @param {Number} [options.maxRequestsPerCrawl]
+ * @property {Number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
- * @param {Object} [options.autoscaledPoolOptions]
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
* Custom options passed to the underlying {@link AutoscaledPool} constructor.
* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
* are provided by `CheerioCrawler` and cannot be overridden. Reasonable {@link Snapshotter}
* and {@link SystemStatus} defaults are provided to account for the fact that `cheerio`
* parses HTML synchronously and therefore blocks the event loop.
- * @param {Number} [options.minConcurrency=1]
+ * @property {Number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @param {Number} [options.maxConcurrency=1000]
+ * @property {Number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @param {Boolean} [options.useSessionPool=false]
+ * @property {Boolean} [useSessionPool=false]
* If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
* It also marks Session as bad after a request timeout.
- * @param {Object} [options.sessionPoolOptions]
+ * @property {SessionPoolOptions} [sessionPoolOptions]
* Custom options passed to the underlying {@link SessionPool} constructor.
- * @param {Boolean} [options.persistCookiesPerSession]
+ * @property {Boolean} [persistCookiesPerSession]
* Automatically saves cookies to Session. Works only if Session Pool is used.
*
* It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.
* It passes the "Cookie" header to the request with the session cookies.
*/
+
+/**
+ * Provides a framework for the parallel crawling of web pages using plain HTTP requests and
+ * <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `CheerioCrawler` uses raw HTTP requests to download web pages,
+ * it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript
+ * to display the content, you might need to use {@link PuppeteerCrawler} instead,
+ * because it loads the pages using full-featured headless Chrome browser.
+ *
+ * `CheerioCrawler` downloads each URL using a plain HTTP request,
+ * parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>
+ * and then invokes the user-provided [`handlePageFunction()`](#new_CheerioCrawler_new) to extract page data
+ * using a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface to the parsed HTML DOM.
+ *
+ * The source URLs are represented using {@link Request} objects that are fed from
+ * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_CheerioCrawler_new)
+ * or [`requestQueue`](#new_CheerioCrawler_new) constructor options, respectively.
+ *
+ * If both [`requestList`](#new_CheerioCrawler_new) and [`requestQueue`](#new_CheerioCrawler_new) are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes when there are no more {@link Request} objects to crawl.
+ *
+ * By default, `CheerioCrawler` downloads HTML using the
+ * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> NPM package.
+ * You can use the `requestOptions` parameter to pass additional options to `request`.
+ *
+ * New requests are only dispatched when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPool} options are available directly in the `CheerioCrawler` constructor.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * // Prepare a list of URLs to crawl
+ * const requestList = new Apify.RequestList({
+ * sources: [
+ * { url: 'http://www.example.com/page-1' },
+ * { url: 'http://www.example.com/page-2' },
+ * ],
+ * });
+ * await requestList.initialize();
+ *
+ * // Crawl the URLs
+ * const crawler = new Apify.CheerioCrawler({
+ * requestList,
+ * handlePageFunction: async ({ request, response, body, contentType, $ }) => {
+ * const data = [];
+ *
+ * // Do some data extraction from the page with Cheerio.
+ * $('.some-collection').each((index, el) => {
+ * data.push({ title: $(el).find('.some-title').text() });
+ * });
+ *
+ * // Save the data to dataset.
+ * await Apify.pushData({
+ * url: request.url,
+ * html: body,
+ * data,
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ */
class CheerioCrawler {
+ /**
+ * @param {CheerioCrawlerOptions} options
+ */
constructor(options = {}) {
const {
requestOptions,
@@ -388,6 +407,10 @@ class CheerioCrawler {
/**
* Wrapper around handlePageFunction that opens and closes pages etc.
*
+ * @param {Object} options
+ * @param {Request} options.request
+ * @param {AutoscaledPool} options.autoscaledPool
+ * @param {Session} options.session
* @ignore
*/
async _handleRequestFunction({ request, autoscaledPool, session }) {
@@ -445,6 +468,10 @@ class CheerioCrawler {
* Function to make the HTTP request. It performs optimizations
* on the request such as only downloading the request body if the
* received content type matches text/html, application/xml, application/xhtml+xml.
+ *
+ * @param {Object} options
+ * @param {Request} options.request
+ * @param {Session} options.session
* @ignore
*/
async _requestFunction({ request, session }) {
@@ -541,6 +568,7 @@ class CheerioCrawler {
* Enables the use of a proxy by returning a proxy URL
* based on configured options or null if no proxy is used.
* @param {Session?} session
+ * @returns {string|null}
* @ignore
*/
_getProxyUrl(session = {}) {
@@ -592,7 +620,7 @@ class CheerioCrawler {
/**
* Checks and extends supported mime types
- * @param additionalMimeTypes
+ * @param {Array<String|Object>} additionalMimeTypes
* @ignore
*/
_extendSupportedMimeTypes(additionalMimeTypes) {
@@ -631,5 +659,31 @@ class CheerioCrawler {
}
}
-
export default CheerioCrawler;
+
+/**
+ * @typedef PrepareRequestInputs
+ * @property {Request} request Original instance fo the {Request} object. Must be modified in-place.
+ */
+/**
+ * @callback PrepareRequest
+ * @param {PrepareRequestInputs} inputs Arguments passed to this callback.
+ * @returns {void|Promise<void>}
+ */
+
+/**
+ * @typedef CheerioHandlePageInputs
+ * @property {Cheerio} [$] The <a href="https://cheerio.js.org/">Cheerio</a> object with parsed HTML.
+ * @property {String|Buffer} body The request body of the web page.
+ * @property {Object} [json] The parsed object from JSON string if the response contains the content type application/json.
+ * @property {Request} request The original {Request} object.
+ * @property {{ type: string, encoding: string }} contentType Parsed `Content-Type header: { type, encoding }`.
+ * @property {IncomingMessage} response An instance of Node's http.IncomingMessage object,
+ * @property {AutoscaledPool} autoscaledPool
+ * @property {session} [session]
+ */
+/**
+ * @callback CheerioHandlePage
+ * @param {CheerioHandlePageInputs} inputs Arguments passed to this callback.
+ * @returns {Promise<void>}
+ */
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 16ef95f6d..97a80c8d8 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -1,78 +1,30 @@
import { checkParamOrThrow } from 'apify-client/build/utils';
import log from 'apify-shared/log';
import _ from 'underscore';
-import BasicCrawler from './basic_crawler';
-import PuppeteerPool, { BROWSER_SESSION_KEY_NAME } from '../puppeteer_pool';
-import { addTimeoutToPromise } from '../utils';
+import PuppeteerPool, { BROWSER_SESSION_KEY_NAME } from '../puppeteer_pool'; // eslint-disable-line import/no-duplicates
import { BASIC_CRAWLER_TIMEOUT_MULTIPLIER } from '../constants';
import { gotoExtended } from '../puppeteer_utils';
-import { openSessionPool } from '../session_pool/session_pool';
+import { openSessionPool } from '../session_pool/session_pool'; // eslint-disable-line import/no-duplicates
+import { addTimeoutToPromise } from '../utils';
+import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
+
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { Browser, Page as PuppeteerPage, Response as PuppeteerResponse } from 'puppeteer';
+import { HandleFailedRequest } from './basic_crawler';
+import { PuppeteerPoolOptions } from '../puppeteer_pool';
+import Request from '../request'; // eslint-disable-line no-unused-vars
+import { RequestList } from '../request_list'; // eslint-disable-line no-unused-vars
+import { RequestQueue } from '../request_queue'; // eslint-disable-line no-unused-vars
+import AutoscaledPool, { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool'; // eslint-disable-line no-unused-vars,import/named
+import { LaunchPuppeteerOptions } from '../puppeteer'; // eslint-disable-line no-unused-vars,import/named
+import { Session } from '../session_pool/session'; // eslint-disable-line no-unused-vars
+import { SessionPoolOptions } from '../session_pool/session_pool';
+// eslint-enable-line import/no-duplicates
/**
- * Provides a simple framework for parallel crawling of web pages
- * using headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
- * The URLs to crawl are fed either from a static list of URLs
- * or from a dynamic queue of URLs enabling recursive crawling of websites.
- *
- * Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data,
- * it is useful for crawling of websites that require to execute JavaScript.
- * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
- * which downloads the pages using raw HTTP requests and is about 10x faster.
- *
- * The source URLs are represented using {@link Request} objects that are fed from
- * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_PuppeteerCrawler_new)
- * or [`requestQueue`](#new_PuppeteerCrawler_new) constructor options, respectively.
- *
- * If both [`requestList`](#new_PuppeteerCrawler_new) and [`requestQueue`](#new_PuppeteerCrawler_new) are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
- *
- * The crawler finishes when there are no more {@link Request} objects to crawl.
- *
- * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
- * and then calls the function provided by user as the [`handlePageFunction()`](#new_PuppeteerCrawler_new) option.
- *
- * New pages are only opened when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
- * parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
- * {@link AutoscaledPool} options are available directly in the `PuppeteerCrawler` constructor.
- *
- * Note that the pool of Puppeteer instances is internally managed by
- * the {@link PuppeteerPool} class. Many constructor options
- * such as `maxOpenPagesPerInstance` or `launchPuppeteerFunction` are passed directly
- * to {@link PuppeteerPool} constructor.
- *
- * **Example usage:**
- *
- * ```javascript
- * const crawler = new Apify.PuppeteerCrawler({
- * requestList,
- * handlePageFunction: async ({ page, request }) => {
- * // This function is called to extract data from a single web page
- * // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
- * // 'request' is an instance of Request class with information about the page to load
- * await Apify.pushData({
- * title: await page.title(),
- * url: request.url,
- * succeeded: true,
- * })
- * },
- * handleFailedRequestFunction: async ({ request }) => {
- * // This function is called when the crawling of a request failed too many times
- * await Apify.pushData({
- * url: request.url,
- * succeeded: false,
- * errors: request.errorMessages,
- * })
- * },
- * });
- *
- * await crawler.run();
- * ```
- * @param {Object} options All `PuppeteerCrawler` parameters are passed
- * via an options object with the following keys:
- * @param {Function} options.handlePageFunction
+ * @typedef {Object} PuppeteerCrawlerOptions
+ * @property {PuppeteerHandlePage} handlePageFunction
* Function that is called to process each request.
* It is passed an object with the following fields:
*
@@ -89,7 +41,7 @@ import { openSessionPool } from '../session_pool/session_pool';
*
* `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
* `page` is an instance of the `Puppeteer`
- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>.
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
* `response` is an instance of the `Puppeteer`
* <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
* which is the main resource response as returned by `page.goto(request.url)`.
@@ -100,20 +52,20 @@ import { openSessionPool } from '../session_pool/session_pool';
* If the function throws an exception, the crawler will try to re-crawl the
* request later, up to `option.maxRequestRetries` times.
* If all the retries fail, the crawler calls the function
- * provided to the `options.handleFailedRequestFunction` parameter.
+ * provided to the `handleFailedRequestFunction` parameter.
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
* [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
- * @param {RequestList} options.requestList
+ * @property {RequestList} [requestList]
* Static list of URLs to be processed.
* Either `requestList` or `requestQueue` option must be provided (or both).
- * @param {RequestQueue} options.requestQueue
+ * @property {RequestQueue} [requestQueue]
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
* Either `requestList` or `requestQueue` option must be provided (or both).
- * @param {Number} [options.handlePageTimeoutSecs=60]
- * Timeout in which the function passed as `options.handlePageFunction` needs to finish, in seconds.
- * @param {Function} [options.gotoFunction]
+ * @property {Number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+ * @property {PuppeteerGoto} [gotoFunction]
* Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's
* <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,
* i.e. a `Promise` resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.
@@ -126,10 +78,10 @@ import { openSessionPool } from '../session_pool/session_pool';
* By default, the function invokes [`Apify.utils.puppeteer.gotoExtended()`](puppeteer#puppeteer.gotoExtended) with a timeout of 60 seconds.
* For details, see source code on
* <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.
- * @param {Number} [options.gotoTimeoutSecs=60]
- * Timeout in which page navigation needs to finish, in seconds. When `options.gotoFunction()` is used and thus the default
+ * @property {Number} [gotoTimeoutSecs=60]
+ * Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
* function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
- * @param {Function} [options.handleFailedRequestFunction]
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
* A function to handle requests that failed more than `option.maxRequestRetries` times.
*
* The function receives the following object as an argument:
@@ -145,44 +97,112 @@ import { openSessionPool } from '../session_pool/session_pool';
* See
* <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
* for the default implementation of this function.
- * @param {Number} [options.maxRequestRetries=3]
+ * @property {Number} [maxRequestRetries=3]
* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
- * @param {Number} [options.maxRequestsPerCrawl]
+ * @property {Number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
- * @param {Object} [options.puppeteerPoolOptions]
+ * @property {PuppeteerPoolOptions} [puppeteerPoolOptions]
* Custom options passed to the underlying {@link PuppeteerPool} constructor.
* You can tweak those to fine-tune browser management.
- * @param {Function} [options.launchPuppeteerFunction]
+ * @property {Function} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* Shortcut to the corresponding {@link PuppeteerPool} option.
* See source code on
* <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
* for default behavior.
- * @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]
+ * @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]
* Options used by [`Apify.launchPuppeteer()`](apify#module_Apify.launchPuppeteer) to start new Puppeteer instances.
* Shortcut to the corresponding {@link PuppeteerPool} option. See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
- * @param {Object} [options.autoscaledPoolOptions]
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
* Custom options passed to the underlying {@link AutoscaledPool} instance constructor.
* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
* are provided by `PuppeteerCrawler` and should not be overridden.
- * @param {Object} [options.minConcurrency=1]
+ * @property {Number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @param {Object} [options.maxConcurrency=1000]
+ * @property {Number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @param {Boolean} [options.useSessionPool=false]
+ * @property {Boolean} [useSessionPool=false]
* If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
* It also marks Session as bad after a request timeout.
- * @param {Object} [options.sessionPoolOptions]
+ * @property {SessionPoolOptions} [sessionPoolOptions]
* Custom options passed to the underlying {@link SessionPool} constructor.
- * @param {Boolean} [options.persistCookiesPerSession]
+ * @property {Boolean} [persistCookiesPerSession]
* Automatically saves cookies to Session. Works only if Session Pool is used.
*/
+
+/**
+ * Provides a simple framework for parallel crawling of web pages
+ * using headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data,
+ * it is useful for crawling of websites that require to execute JavaScript.
+ * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * which downloads the pages using raw HTTP requests and is about 10x faster.
+ *
+ * The source URLs are represented using {@link Request} objects that are fed from
+ * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_PuppeteerCrawler_new)
+ * or [`requestQueue`](#new_PuppeteerCrawler_new) constructor options, respectively.
+ *
+ * If both [`requestList`](#new_PuppeteerCrawler_new) and [`requestQueue`](#new_PuppeteerCrawler_new) are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes when there are no more {@link Request} objects to crawl.
+ *
+ * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
+ * and then calls the function provided by user as the [`handlePageFunction()`](#new_PuppeteerCrawler_new) option.
+ *
+ * New pages are only opened when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPool} options are available directly in the `PuppeteerCrawler` constructor.
+ *
+ * Note that the pool of Puppeteer instances is internally managed by
+ * the {@link PuppeteerPool} class. Many constructor options
+ * such as `maxOpenPagesPerInstance` or `launchPuppeteerFunction` are passed directly
+ * to {@link PuppeteerPool} constructor.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const crawler = new Apify.PuppeteerCrawler({
+ * requestList,
+ * handlePageFunction: async ({ page, request }) => {
+ * // This function is called to extract data from a single web page
+ * // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
+ * // 'request' is an instance of Request class with information about the page to load
+ * await Apify.pushData({
+ * title: await page.title(),
+ * url: request.url,
+ * succeeded: true,
+ * })
+ * },
+ * handleFailedRequestFunction: async ({ request }) => {
+ * // This function is called when the crawling of a request failed too many times
+ * await Apify.pushData({
+ * url: request.url,
+ * succeeded: false,
+ * errors: request.errorMessages,
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ */
class PuppeteerCrawler {
+ /**
+ * @param {PuppeteerCrawlerOptions} options All `PuppeteerCrawler` parameters are passed
+ * via an options object with the following keys:
+ */
constructor(options) {
const {
handlePageFunction,
@@ -264,7 +284,7 @@ class PuppeteerCrawler {
/**
* Runs the crawler. Returns promise that gets resolved once all the requests got processed.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async run() {
if (this.isRunningPromise) return this.isRunningPromise;
@@ -288,6 +308,9 @@ class PuppeteerCrawler {
/**
* Wrapper around handlePageFunction that opens and closes pages etc.
*
+ * @param {Object} options
+ * @param {Request} options.request
+ * @param {AutoscaledPool} options.autoscaledPool
* @ignore
*/
async _handleRequestFunction({ request, autoscaledPool }) {
@@ -328,9 +351,10 @@ class PuppeteerCrawler {
}
/**
- * @param {Page} page
- * @param {Request} request
- * @return {Promise<Response>}
+ * @param {Object} options
+ * @param {PuppeteerPage} options.page
+ * @param {Request} options.request
+ * @return {Promise<PuppeteerResponse>}
* @ignore
*/
async _defaultGotoFunction({ page, request }) {
@@ -338,8 +362,9 @@ class PuppeteerCrawler {
}
/**
- * @param {Request} request
- * @return {Promise}
+ * @param {Object} options
+ * @param {Request} options.request
+ * @return {Promise<void>}
* @ignore
*/
async _defaultHandleFailedRequestFunction({ request }) { // eslint-disable-line class-methods-use-this
@@ -348,5 +373,49 @@ class PuppeteerCrawler {
}
}
-
export default PuppeteerCrawler;
+
+/**
+ * @typedef PuppeteerHandlePageInputs
+ * @property {Request} request An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * @property {PuppeteerResponse} response An instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ * @property {PuppeteerPage} page is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ * @property {PuppeteerPool} puppeteerPool An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
+ * @property {AutoscaledPool} autoscaledPool
+ * @property {Session} [session]
+ */
+/**
+ * @callback PuppeteerHandlePage
+ * @param {PuppeteerHandlePageInputs} inputs Arguments passed to this callback.
+ * @return {Promise<void>}
+ */
+
+/**
+ * @typedef PuppeteerGotoInputs
+ * @property {PuppeteerPage} page is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ * @property {Request} request An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * @property {AutoscaledPool} autoscaledPool An instance of the `AutoscaledPool`.
+ * @property {PuppeteerPool} puppeteerPool An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
+ * @property {Session} [session] `Session` object for this request.
+ */
+/**
+ * @callback PuppeteerGoto
+ * @param {PuppeteerGotoInputs} inputs Arguments passed to this callback.
+ * @return {Promise<PuppeteerResponse>} An instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ */
+
+/**
+ * @callback LaunchPuppeteer
+ * @param {LaunchPuppeteerOptions} inputs Arguments passed to this callback.
+ * @return {Promise<Browser>} Promise that resolves to Puppeteer's `Browser` instance.
+ * This might be obtained by calling
+ * <a href="https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions">puppeteer.launch()</a>
+ * directly, or by delegating to
+ * [`Apify.launchPuppeteer()`](../api/apify#apifylaunchpuppeteeroptions-%E2%87%92-promisebrowser).
+ */
diff --git a/src/dataset.js b/src/dataset.js
index 015c80ff8..0eb062a87 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -145,6 +145,10 @@ export const chunkBySize = (items, limitBytes) => {
* @hideconstructor
*/
export class Dataset {
+ /**
+ * @param {string} datasetId
+ * @param {string} datasetName
+ */
constructor(datasetId, datasetName) {
checkParamOrThrow(datasetId, 'datasetId', 'String');
checkParamOrThrow(datasetName, 'datasetName', 'Maybe String');
@@ -177,7 +181,7 @@ export class Dataset {
*
* @param {Object|Array} data Object or array of objects containing data to be stored in the default dataset.
* The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async pushData(data) {
checkParamOrThrow(data, 'data', 'Array | Object');
@@ -201,18 +205,7 @@ export class Dataset {
}
/**
- * Returns items in the dataset based on the provided parameters. The returned object
- * has the following structure:
- *
- * ```javascript
- * {
- * items, // Array|String|Buffer based on chosen format parameter.
- * total, // Number
- * offset, // Number
- * count, // Number
- * limit, // Number
- * }
- * ```
+ * Returns {DatasetContent} object holding the items in the dataset based on the provided parameters.
*
* **NOTE**: If using dataset with local disk storage, the `format` option must be `json` and
* the following options are not supported:
@@ -269,7 +262,7 @@ export class Dataset {
* If `true` then, the all the items with errorInfo property will be skipped from the output.
* This feature is here to emulate functionality of Apify API version 1 used for
* the legacy Apify Crawler product and it's not recommended to use it in new integrations.
- * @return {Promise<Object>}
+ * @return {Promise<DatasetContent>}
*/
async getData(options = {}) {
// TODO (JC): Do we really need this function? It only works with API but not locally,
@@ -290,6 +283,7 @@ export class Dataset {
}
}
+ // TODO yin: After ApifyClient declarations, re-export this typedef for {DatasetInfo}.
/**
* Returns an object containing general information about the dataset.
*
@@ -334,14 +328,14 @@ export class Dataset {
* });
* ```
*
- * @param {Function} iteratee A function that is called for every item in the dataset.
+ * @param {DatasetConsumer} iteratee A function that is called for every item in the dataset.
* @param {Object} [options] All `forEach()` parameters are passed
* via an options object with the following keys:
* @param {Boolean} [options.desc=false] If `true` then the objects are sorted by `createdAt` in descending order.
* @param {Array} [options.fields] If provided then returned objects will only contain specified keys.
* @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
* @param {Number} [index=0] Specifies the initial index number passed to the `iteratee` function.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async forEach(iteratee, options = {}, index = 0) {
if (!options.offset) options.offset = 0;
@@ -369,13 +363,14 @@ export class Dataset {
*
* If `iteratee` returns a `Promise` then it's awaited before a next call.
*
- * @param {Function} iteratee
+ * @template T
+ * @param {DatasetMapper} iteratee
* @param {Object} options All `map()` parameters are passed
* via an options object with the following keys:
* @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
* @param {Array} [options.fields] If provided then returned objects will only contain specified keys
* @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @return {Promise<Array>}
+ * @return {Promise<T[]>}
*/
map(iteratee, options) {
const result = [];
@@ -403,14 +398,15 @@ export class Dataset {
*
* If `iteratee()` returns a `Promise` then it's awaited before a next call.
*
- * @param {Function} iteratee
- * @param {*} memo Initial state of the reduction.
+ * @template T
+ * @param {DatasetReducer} iteratee
+ * @param {T} memo Initial state of the reduction.
* @param {Object} options All `reduce()` parameters are passed
* via an options object with the following keys:
* @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
* @param {Array} [options.fields] If provided then returned objects will only contain specified keys
* @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @return {Promise<*>}
+ * @return {Promise<T>}
*/
reduce(iteratee, memo, options) {
let currentMemo = memo;
@@ -437,7 +433,7 @@ export class Dataset {
* Removes the dataset either from the Apify cloud storage or from the local directory,
* depending on the mode of operation.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async drop() {
await datasets.deleteDataset({ datasetId: this.datasetId });
@@ -725,3 +721,40 @@ export const openDataset = (datasetIdOrName, options = {}) => {
* @function
*/
export const pushData = item => openDataset().then(dataset => dataset.pushData(item));
+
+/**
+ * @typedef DatasetContent
+ * @property {Object[]|String[]|Buffer[]} items Dataset entries based on chosen format parameter.
+ * @property {Number} total Total count of entries in the dataset.
+ * @property {Number} offset Position of the first returned entry in the dataset.
+ * @property {Number} count Count of dataset entries returned in this set.
+ * @property {Number} limit Maximum number of dataset entries requested.
+ */
+
+// TODO yin: Typescript candoes not understand `@callback` with generic `@template T`. Change after this is fixed
+/**
+ * User-function used in the `Dataset.forEach()` API.
+ * @callback DatasetConsumer
+ * @param {Object} item Current {@link Dataset} entry being processed.
+ * @param {Number} index Position of current {Dataset} entry.
+ * @returns T
+ */
+
+// TODO yin: Typescript candoes not understand `@callback` with generic `@template T`. Change after this is fixed
+/**
+ * User-function used in the `Dataset.map()` API.
+ * @callback DatasetMapper
+ * @param {Object} item Currect {@link Dataset} entry being processed.
+ * @param {Number} index Position of current {Dataset} entry.
+ * @returns T
+ */
+
+// TODO yin: Typescript candoes not understand `@callback` with generic `@template T`. Change after this is fixed
+/**
+ * User-function used in the `Dataset.reduce()` API.
+ * @callback DatasetReducer
+ * @param {T} memo Previous state of the reduction.
+ * @param {Object} item Currect {@link Dataset} entry being processed.
+ * @param {Number} index Position of current {Dataset} entry.
+ * @returns T
+ */
diff --git a/src/enqueue_links/click_elements.js b/src/enqueue_links/click_elements.js
index f1dffbb47..fc83d5c37 100644
--- a/src/enqueue_links/click_elements.js
+++ b/src/enqueue_links/click_elements.js
@@ -2,9 +2,10 @@ import { URL } from 'url';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
import log from 'apify-shared/log';
-import { RequestQueue, RequestQueueLocal } from '../request_queue';
+import { RequestQueue, RequestQueueLocal, QueueOperationInfo } from '../request_queue'; // eslint-disable-line import/named,no-unused-vars
import { addInterceptRequestHandler, removeInterceptRequestHandler } from '../puppeteer_request_interception';
-import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatches, createRequestOptions } from './shared';
+// eslint-disable-next-line import/named,no-unused-vars
+import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatches, createRequestOptions, RequestTransform } from './shared';
const STARTING_Z_INDEX = 2147400000;
@@ -73,9 +74,7 @@ const STARTING_Z_INDEX = 2147400000;
*
* If `pseudoUrls` is an empty array, `null` or `undefined`, then the function
* enqueues all links found on the page.
- * @param {Function} [options.transformRequestFunction]
- * **Signature:** ({@link Request}): {@link Request}
- *
+ * @param {RequestTransform} [options.transformRequestFunction]
* Just before a new {@link Request} is constructed and enqueued to the {@link RequestQueue}, this function can be used
* to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful
* when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads,
diff --git a/src/enqueue_links/enqueue_links.js b/src/enqueue_links/enqueue_links.js
index fec56dc3d..6b1f24cfe 100644
--- a/src/enqueue_links/enqueue_links.js
+++ b/src/enqueue_links/enqueue_links.js
@@ -2,8 +2,20 @@ import { URL } from 'url';
import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
+/* eslint-disable import/no-duplicates */
import { RequestQueue, RequestQueueLocal } from '../request_queue';
import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatches, createRequestOptions } from './shared';
+/* eslint-enable import/no-duplicates */
+
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { Page } from 'puppeteer';
+import { RequestOptions } from '../request';
+import { QueueOperationInfo } from '../request_queue';
+import { RequestTransform } from './shared';
+import { Cheerio } from '../typedefs';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
+
/**
* The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default)
@@ -62,9 +74,7 @@ import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatche
*
* If `pseudoUrls` is an empty array, `null` or `undefined`, then the function
* enqueues all links found on the page.
- * @param {Function} [options.transformRequestFunction]
- * **Signature:** ({@link Request}): {@link Request}
- *
+ * @param {RequestTransform} [options.transformRequestFunction]
* Just before a new {@link Request} is constructed and enqueued to the {@link RequestQueue}, this function can be used
* to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful
* when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads,
@@ -148,7 +158,7 @@ export async function extractUrlsFromPage(page, selector) {
/**
* Extracts URLs from a given Cheerio object.
*
- * @param {Function} $
+ * @param {Cheerio} $
* @param {string} selector
* @param {string} baseUrl
* @return {string[]}
diff --git a/src/enqueue_links/shared.js b/src/enqueue_links/shared.js
index d17d794f8..60f8ab1cb 100644
--- a/src/enqueue_links/shared.js
+++ b/src/enqueue_links/shared.js
@@ -4,6 +4,7 @@ import { checkParamOrThrow } from 'apify-client/build/utils';
import PseudoUrl from '../pseudo_url';
import Request from '../request';
+
const MAX_ENQUEUE_LINKS_CACHE_SIZE = 1000;
/**
@@ -103,3 +104,11 @@ export async function addRequestsToQueueInBatches(requests, requestQueue, batchS
}
return Promise.all(queueOperationInfos);
}
+
+/**
+ * Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used
+ * [`Apify.utils.enqueueLinks`](../api/utils#utils.enqueueLinks) to modify requests before enqueuing them.
+ * @callback RequestTransform
+ * @param {RequestOptions} original Request options to be modified.
+ * @return {RequestOptions} The modified request options to enqueue.
+ */
diff --git a/src/errors.js b/src/errors.js
index bb8ed7d66..24df0757f 100644
--- a/src/errors.js
+++ b/src/errors.js
@@ -1,3 +1,8 @@
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { ActorRun } from './typedefs';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
+
export const APIFY_CALL_ERROR_NAME = 'ApifyCallError';
/**
diff --git a/src/events.js b/src/events.js
index dd483dcd5..d8d750799 100644
--- a/src/events.js
+++ b/src/events.js
@@ -1,4 +1,4 @@
-import EventEmitter from 'events';
+import { EventEmitter } from 'events';
import WebSocket from 'ws';
import log from 'apify-shared/log';
import { ENV_VARS, ACTOR_EVENT_NAMES } from 'apify-shared/consts';
diff --git a/src/index.esm.js b/src/index.esm.js
new file mode 100644
index 000000000..c472763e6
--- /dev/null
+++ b/src/index.esm.js
@@ -0,0 +1,94 @@
+import { EventEmitter } from 'events';
+import log from 'apify-shared/log';
+import { main, getEnv, call, callTask, getApifyProxyUrl, metamorph, addWebhook } from './actor';
+import AutoscaledPool from './autoscaling/autoscaled_pool';
+import BasicCrawler from './crawlers/basic_crawler';
+import CheerioCrawler from './crawlers/cheerio_crawler';
+import { pushData, openDataset } from './dataset';
+import events, { initializeEvents, stopEvents } from './events';
+import { getValue, setValue, getInput, openKeyValueStore } from './key_value_store';
+import { launchPuppeteer } from './puppeteer';
+import PuppeteerCrawler from './crawlers/puppeteer_crawler';
+import PuppeteerPool from './puppeteer_pool';
+import Request from './request';
+import { RequestList, openRequestList } from './request_list';
+import { openRequestQueue } from './request_queue';
+import { apifyClient, getMemoryInfo, isAtHome, publicUtils } from './utils';
+import { puppeteerUtils } from './puppeteer_utils';
+import { socialUtils } from './utils_social';
+import { enqueueLinks } from './enqueue_links/enqueue_links';
+import PseudoUrl from './pseudo_url';
+import LiveViewServer from './live_view/live_view_server';
+import { requestAsBrowser } from './utils_request';
+import { openSessionPool } from './session_pool/session_pool';
+import { Session } from './session_pool/session';
+
+// Increase the global limit for event emitter memory leak warnings.
+EventEmitter.defaultMaxListeners = 50;
+
+// Log as plain text not JSON
+log.logJson = false;
+
+const exportedUtils = Object.assign(publicUtils, {
+ puppeteer: puppeteerUtils,
+ social: socialUtils,
+ log,
+ enqueueLinks,
+ requestAsBrowser,
+});
+
+/**
+ * The following section describes all functions and properties provided by the `apify` package,
+ * except individual classes and namespaces that have their separate, detailed, documentation pages
+ * accessible from the left sidebar.
+ *
+ * @module Apify
+ */
+export {
+ main,
+ getEnv,
+ call,
+ callTask,
+ metamorph,
+ getMemoryInfo,
+ getApifyProxyUrl,
+ isAtHome,
+ apifyClient as client,
+ addWebhook,
+
+ AutoscaledPool,
+
+ BasicCrawler,
+
+ CheerioCrawler,
+
+ pushData,
+ openDataset,
+
+ events,
+ initializeEvents,
+ stopEvents,
+
+ getValue,
+ setValue,
+ getInput,
+ openKeyValueStore,
+
+ launchPuppeteer,
+ PuppeteerPool,
+ PuppeteerCrawler,
+
+ PseudoUrl,
+
+ Request,
+ RequestList,
+ openRequestList,
+ openRequestQueue,
+
+ openSessionPool,
+
+ LiveViewServer,
+ Session,
+
+ exportedUtils as utils,
+};
diff --git a/src/index.js b/src/index.js
index ce55cd791..d88667e23 100644
--- a/src/index.js
+++ b/src/index.js
@@ -1,228 +1,4 @@
-import EventEmitter from 'events';
-import log from 'apify-shared/log';
-import { main, getEnv, call, callTask, getApifyProxyUrl, metamorph, addWebhook } from './actor';
-import AutoscaledPool from './autoscaling/autoscaled_pool';
-import BasicCrawler from './crawlers/basic_crawler';
-import CheerioCrawler from './crawlers/cheerio_crawler';
-import { pushData, openDataset } from './dataset';
-import events, { initializeEvents, stopEvents } from './events';
-import { getValue, setValue, getInput, openKeyValueStore } from './key_value_store';
-import { launchPuppeteer } from './puppeteer';
-import PuppeteerCrawler from './crawlers/puppeteer_crawler';
-import PuppeteerPool from './puppeteer_pool';
-import Request from './request';
-import { RequestList, openRequestList } from './request_list';
-import { openRequestQueue } from './request_queue';
-import { apifyClient, getMemoryInfo, isAtHome, publicUtils } from './utils';
-import { puppeteerUtils } from './puppeteer_utils';
-import { socialUtils } from './utils_social';
-import { enqueueLinks } from './enqueue_links/enqueue_links';
-import PseudoUrl from './pseudo_url';
-import LiveViewServer from './live_view/live_view_server';
-import { requestAsBrowser } from './utils_request';
-import { openSessionPool } from './session_pool/session_pool';
-import { Session } from './session_pool/session';
+import * as Apify from './index.esm';
-// Increase the global limit for event emitter memory leak warnings.
-EventEmitter.defaultMaxListeners = 50;
-
-// Log as plain text not JSON
-log.logJson = false;
-
-/**
- * The following section describes all functions and properties provided by the `apify` package,
- * except individual classes and namespaces that have their separate, detailed, documentation pages
- * accessible from the left sidebar.
- *
- * @module Apify
- */
-module.exports = {
- main,
- getEnv,
- call,
- callTask,
- metamorph,
- getMemoryInfo,
- getApifyProxyUrl,
- isAtHome,
- client: apifyClient,
- addWebhook,
-
- AutoscaledPool,
-
- BasicCrawler,
-
- CheerioCrawler,
-
- pushData,
- openDataset,
-
- events,
- initializeEvents,
- stopEvents,
-
- getValue,
- setValue,
- getInput,
- openKeyValueStore,
-
- launchPuppeteer,
- PuppeteerPool,
- PuppeteerCrawler,
-
- PseudoUrl,
-
- Request,
- RequestList,
- openRequestList,
- openRequestQueue,
-
- openSessionPool,
-
- LiveViewServer,
- Session,
-
- utils: Object.assign(publicUtils, {
- puppeteer: puppeteerUtils,
- social: socialUtils,
- log,
- enqueueLinks,
- requestAsBrowser,
- }),
-};
-
-// Add docs for log separately, as it's imported from apify-shared.
-// Adding them directly to the log object in utils breaks JSDoc.
-
-/**
- * The log instance enables level aware logging of messages and we advise
- * to use it instead of `console.log()` and its aliases in most development
- * scenarios.
- *
- * A very useful use case for `log` is using `log.debug` liberally throughout
- * the codebase to get useful logging messages only when appropriate log level is set
- * and keeping the console tidy in production environments.
- *
- * The available logging levels are, in this order: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `OFF`
- * and can be referenced from the `log.LEVELS` constant, such as `log.LEVELS.ERROR`.
- *
- * To log messages to the system console, use the `log.level(message)` invocation,
- * such as `log.debug('this is a debug message')`.
- *
- * To prevent writing of messages above a certain log level to the console, simply
- * set the appropriate level. The default log level is `INFO`, which means that
- * `DEBUG` messages will not be printed, unless enabled.
- *
- * **Example:**
- * ```
- * const Apify = require('apify');
- * const { log } = Apify.utils;
- *
- * log.info('Information message', { someData: 123 }); // prints message
- * log.debug('Debug message', { debugData: 'hello' }); // doesn't print anything
- *
- * log.setLevel(log.LEVELS.DEBUG);
- * log.debug('Debug message'); // prints message
- *
- * log.setLevel(log.LEVELS.ERROR);
- * log.debug('Debug message'); // doesn't print anything
- * log.info('Info message'); // doesn't print anything
- *
- * log.error('Error message', { errorDetails: 'This is bad!' }); // prints message
- * try {
- * throw new Error('Not good!');
- * } catch (e) {
- * log.exception(e, 'Exception occurred', { errorDetails: 'This is really bad!' }); // prints message
- * }
- * ```
- *
- * Another very useful way of setting the log level is by setting the `APIFY_LOG_LEVEL`
- * environment variable, such as `APIFY_LOG_LEVEL=DEBUG`. This way, no code changes
- * are necessary to turn on your debug messages and start debugging right away.
- * @namespace log
- */
-
-/**
- * Map of available log levels that's useful for easy setting of appropriate log levels.
- * Each log level is represented internally by a number. Eg. `log.LEVELS.DEBUG === 5`.
- * @name LEVELS
- * @type Object
- * @memberOf log
- */
-
-/**
- * Sets the log level to the given value, preventing messages from less important log levels
- * from being printed to the console. Use in conjunction with the `log.LEVELS` constants such as
- *
- * ```
- * log.setLevel(log.LEVELS.DEBUG);
- * ```
- *
- * Default log level is INFO.
- * @name setLevel
- * @param {number} level
- * @method
- * @memberOf log
- */
-
-/**
- * Returns the currently selected logging level. This is useful for checking whether a message
- * will actually be printed to the console before one actually performs a resource intensive operation
- * to construct the message, such as querying a DB for some metadata that need to be added. If the log
- * level is not high enough at the moment, it doesn't make sense to execute the query.
- * @name getLevel
- * @method
- * @memberOf log
- */
-
-/**
- * Logs a `DEBUG` message. By default, it will not be written to the console. To see `DEBUG`
- * messages in the console, set the log level to `DEBUG` either using the `log.setLevel(log.LEVELS.DEBUG)`
- * method or using the environment variable `APIFY_LOG_LEVEL=DEBUG`. Data are stringified and appended
- * to the message.
- * @name debug
- * @param {string} message
- * @param {Object} [data]
- * @method
- * @memberOf log
- */
-
-/**
- * Logs an `INFO` message. `INFO` is the default log level so info messages will be always logged,
- * unless the log level is changed. Data are stringified and appended to the message.
- * @name info
- * @param {string} message
- * @param {Object} [data]
- * @method
- * @memberOf log
- */
-
-/**
- * Logs a `WARNING` level message. Data are stringified and appended to the message.
- * @name warning
- * @param {string} message
- * @param {Object} [data]
- * @method
- * @memberOf log
- */
-
-/**
- * Logs an `ERROR` message. Use this method to log error messages that are not directly connected
- * to an exception. For logging exceptions, use the `log.exception` method.
- * @name error
- * @param {string} message
- * @param {Object} [data]
- * @method
- * @memberOf log
- */
-
-/**
- * Logs an `ERROR` level message with a nicely formatted exception. Note that the exception is the first parameter
- * here and an additional message is only optional.
- * @name exception
- * @param {Error} exception
- * @param {string} [message]
- * @param {Object} [data]
- * @method
- * @memberOf log
- */
+export default Apify;
+export * from './index.esm';
diff --git a/src/key_value_store.js b/src/key_value_store.js
index 7ab8de91d..c81762402 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -160,6 +160,10 @@ export const maybeStringify = (value, options) => {
* @hideconstructor
*/
export class KeyValueStore {
+ /**
+ * @param {string} storeId
+ * @param {string} storeName
+ */
constructor(storeId, storeName) {
checkParamOrThrow(storeId, 'storeId', 'String');
checkParamOrThrow(storeName, 'storeName', 'Maybe String');
@@ -257,7 +261,7 @@ export class KeyValueStore {
* @param {Object} [options]
* @param {String} [options.contentType]
* Specifies a custom MIME content type of the record.
- * @returns {Promise}
+ * @returns {Promise<void>}
*
*/
setValue(key, value, options = {}) {
@@ -284,7 +288,7 @@ export class KeyValueStore {
* Removes the key-value store either from the Apify cloud storage or from the local directory,
* depending on the mode of operation.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async drop() {
await keyValueStores.deleteStore({ storeId: this.storeId });
@@ -328,11 +332,11 @@ export class KeyValueStore {
* });
* ```
*
- * @param {Function} iteratee A function that is called for every key in the key value store.
+ * @param {KeyConsumer} iteratee A function that is called for every key in the key value store.
* @param {Object} [options] All `forEachKey()` parameters are passed
* via an options object with the following keys:
* @param {string} [options.exclusiveStartKey] All keys up to this one (including) are skipped from the result.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async forEachKey(iteratee, options = {}, index = 0) {
const { exclusiveStartKey } = options;
@@ -708,3 +712,17 @@ export const setValue = async (key, value, options) => {
* @name getInput
*/
export const getInput = async () => getValue(process.env[ENV_VARS.INPUT_KEY] || KEY_VALUE_STORE_KEYS.INPUT);
+
+
+/**
+ * User-function used in the [`KeyValueStore.forEachKey()`](../api/keyvaluestore#forEachKey) method.
+ * @callback KeyConsumer
+ * @param {String} key
+ * Current {KeyValue} key being processed.
+ * @param {Number} index
+ * Position of the current key in {KeyValuestore}.
+ * @param {Object} info
+ * Information about the current {KeyValueStore} entry.
+ * @param {Number} info.size
+ * Size of the value associated with the current key in bytes.
+ */
diff --git a/src/live_view/live_view_server.js b/src/live_view/live_view_server.js
index 2a7d2fd0b..d8e2e206c 100644
--- a/src/live_view/live_view_server.js
+++ b/src/live_view/live_view_server.js
@@ -8,6 +8,7 @@ import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { promisifyServerListen } from 'apify-shared/utilities';
import { ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';
+import { Page } from 'puppeteer'; // eslint-disable-line no-unused-vars
import { addTimeoutToPromise } from '../utils';
import Snapshot from './snapshot';
diff --git a/src/pseudo_url.js b/src/pseudo_url.js
index b945978bc..c36f69636 100644
--- a/src/pseudo_url.js
+++ b/src/pseudo_url.js
@@ -98,16 +98,17 @@ const parsePurl = (purl) => {
*
* if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');
* ```
- *
- * @param {String|RegExp} purl
- * A pseudo-URL string or a regular expression object.
- * Using a `RegExp` instance enables more granular control,
- * such as making the matching case sensitive.
- * @param {Object} requestTemplate
- * Options for the new {@link Request} instances created for matching URLs
- * by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.
*/
class PseudoUrl {
+ /**
+ * @param {String|RegExp} purl
+ * A pseudo-URL string or a regular expression object.
+ * Using a `RegExp` instance enables more granular control,
+ * such as making the matching case sensitive.
+ * @param {RequestOptions} requestTemplate
+ * Options for the new {@link Request} instances created for matching URLs
+ * by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.
+ */
constructor(purl, requestTemplate = {}) {
checkParamOrThrow(purl, 'purl', 'String|RegExp');
checkParamOrThrow(requestTemplate, 'requestTemplate', 'Object');
diff --git a/src/puppeteer.js b/src/puppeteer.js
index a8abef146..6f9a9388b 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -3,10 +3,11 @@ import { checkParamOrThrow } from 'apify-client/build/utils';
import { anonymizeProxy, closeAnonymizedProxy, redactUrl, parseUrl } from 'proxy-chain';
import log from 'apify-shared/log';
import { ENV_VARS } from 'apify-shared/consts';
+import { LaunchOptions } from 'puppeteer'; // eslint-disable-line no-unused-vars
import { DEFAULT_USER_AGENT } from './constants';
import { getTypicalChromeExecutablePath, isAtHome } from './utils';
import { getApifyProxyUrl } from './actor';
-import applyStealthToBrowser from './stealth/stealth';
+import applyStealthToBrowser, { StealthOptions } from './stealth/stealth'; // eslint-disable-line no-unused-vars,import/named
const LAUNCH_PUPPETEER_LOG_OMIT_OPTS = [
'proxyUrl', 'userAgent', 'useApifyProxy', 'apifyProxyGroups',
@@ -23,63 +24,6 @@ const LAUNCH_PUPPETEER_APIFY_OPTIONS = [
'useChrome', 'stealth',
];
-/**
- * Apify extends the launch options of Puppeteer.
- * You can use any of the
- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
- * options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
- * function and in addition, all the options available below.
- *
- * @typedef {Object} LaunchPuppeteerOptions
- * @property ...
- * You can use any of the
- * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
- * options.
- * @property {String} [proxyUrl]
- * URL to a HTTP proxy server. It must define the port number,
- * and it may also contain proxy username and password.
- *
- * Example: `http://bob:[email protected]:1234`.
- * @property {String} [userAgent]
- * The `User-Agent` HTTP header used by the browser.
- * If not provided, the function sets `User-Agent` to a reasonable default
- * to reduce the chance of detection of the crawler.
- * @property {Boolean} [useChrome=false]
- * If `true` and `executablePath` is not set,
- * Puppeteer will launch full Google Chrome browser available on the machine
- * rather than the bundled Chromium. The path to Chrome executable
- * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
- * or defaults to the typical Google Chrome executable location specific for the operating system.
- * By default, this option is `false`.
- * @property {Boolean} [useApifyProxy=false]
- * If set to `true`, Puppeteer will be configured to use
- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
- * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>
- * @property {String[]} [apifyProxyGroups]
- * An array of proxy groups to be used
- * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
- * Only applied if the `useApifyProxy` option is `true`.
- * @property {String} [apifyProxySession]
- * Apify Proxy session identifier to be used by all the Chrome browsers.
- * All HTTP requests going through the proxy with the same session identifier
- * will use the same target proxy server (i.e. the same IP address).
- * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
- * Only applied if the `useApifyProxy` option is `true`.
- * @property {string|Object} [puppeteerModule]
- * Either a require path (`string`) to a package to be used instead of default `puppeteer`,
- * or an already required module (`Object`). This enables usage of various Puppeteer
- * wrappers such as `puppeteer-extra`.
- *
- * Take caution, because it can cause all kinds of unexpected errors and weird behavior.
- * Apify SDK is not tested with any other library besides `puppeteer` itself.
- * @property {boolean} [stealth]
- * This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.
- * It is recommended to use it together with the `useChrome` set to `true`.
- * @property {StealthOptions} [stealthOptions]
- * Using this configuration, you can disable some of the hiding tricks.
- * For these settings to take effect `stealth` must be set to true
- */
-
/**
* Launches Puppeteer with proxy used via `proxy-chain` package.
*
@@ -157,6 +101,61 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
}
};
+// TODO yin: `@property ...` didn't work. Extend Puppeteer's `LaunchOptions` didn't work. There is a GitHub issue for that:
+// https://github.com/Microsoft/TypeScript/issues/20077
+/**
+ * Apify extends the launch options of Puppeteer.
+ * You can use any of the
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
+ * options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
+ * function and in addition, all the options available below.
+ *
+ * @typedef {Object} LaunchPuppeteerOptions
+ * @property {String} [proxyUrl]
+ * URL to a HTTP proxy server. It must define the port number,
+ * and it may also contain proxy username and password.
+ *
+ * Example: `http://bob:[email protected]:1234`.
+ * @property {String} [userAgent]
+ * The `User-Agent` HTTP header used by the browser.
+ * If not provided, the function sets `User-Agent` to a reasonable default
+ * to reduce the chance of detection of the crawler.
+ * @property {Boolean} [useChrome=false]
+ * If `true` and `executablePath` is not set,
+ * Puppeteer will launch full Google Chrome browser available on the machine
+ * rather than the bundled Chromium. The path to Chrome executable
+ * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
+ * or defaults to the typical Google Chrome executable location specific for the operating system.
+ * By default, this option is `false`.
+ * @property {Boolean} [useApifyProxy=false]
+ * If set to `true`, Puppeteer will be configured to use
+ * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
+ * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>
+ * @property {String[]} [apifyProxyGroups]
+ * An array of proxy groups to be used
+ * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
+ * Only applied if the `useApifyProxy` option is `true`.
+ * @property {String} [apifyProxySession]
+ * Apify Proxy session identifier to be used by all the Chrome browsers.
+ * All HTTP requests going through the proxy with the same session identifier
+ * will use the same target proxy server (i.e. the same IP address).
+ * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
+ * Only applied if the `useApifyProxy` option is `true`.
+ * @property {string|Object} [puppeteerModule]
+ * Either a require path (`string`) to a package to be used instead of default `puppeteer`,
+ * or an already required module (`Object`). This enables usage of various Puppeteer
+ * wrappers such as `puppeteer-extra`.
+ *
+ * Take caution, because it can cause all kinds of unexpected errors and weird behavior.
+ * Apify SDK is not tested with any other library besides `puppeteer` itself.
+ * @property {boolean} [stealth]
+ * This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.
+ * It is recommended to use it together with the `useChrome` set to `true`.
+ * @property {StealthOptions} [stealthOptions]
+ * Using this configuration, you can disable some of the hiding tricks.
+ * For these settings to take effect `stealth` must be set to true
+ */
+
/**
* Launches headless Chrome using Puppeteer pre-configured to work within the Apify platform.
* The function has the same argument and the return value as `puppeteer.launch()`.
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index b54ae5be4..ff76a91f2 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -7,7 +7,9 @@ import log from 'apify-shared/log';
import LinkedList from 'apify-shared/linked_list';
import rimraf from 'rimraf';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import { launchPuppeteer } from './puppeteer';
+import { Page, Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
+
+import { launchPuppeteer, LaunchPuppeteerOptions } from './puppeteer'; // eslint-disable-line no-unused-vars,import/named
import { addTimeoutToPromise } from './utils';
import LiveViewServer from './live_view/live_view_server';
import EVENTS from './session_pool/events';
@@ -71,97 +73,103 @@ class PuppeteerInstance {
}
/**
- * Manages a pool of Chrome browser instances controlled using
- * <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
- *
- * `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.
- * This is useful in order to facilitate rotation of proxies, cookies
- * or other settings in order to prevent detection of your web scraping bot,
- * access web pages from various countries etc.
- *
- * Additionally, the reuse of browser instances instances speeds up crawling,
- * and the retirement of instances helps mitigate effects of memory leaks in Chrome.
- *
- * `PuppeteerPool` is internally used by the {@link PuppeteerCrawler} class.
- *
- * **Example usage:**
- *
- * ```javascript
- * const puppeteerPool = new PuppeteerPool({
- * launchPuppeteerFunction: () => {
- * // Use a new proxy with a new IP address for each new Chrome instance
- * return Apify.launchPuppeteer({
- * useApifyProxy: true,
- * apifyProxySession: Math.random(),
- * });
- * },
- * });
- *
- * const page1 = await puppeteerPool.newPage();
- * const page2 = await puppeteerPool.newPage();
- * const page3 = await puppeteerPool.newPage();
- *
- * // ... do something with the pages ...
- *
- * // Close all browsers.
- * await puppeteerPool.destroy();
- * ```
- * @param {Object} [options]
- * All `PuppeteerPool` parameters are passed
- * via an options object with the following keys:
- * @param {boolean} [options.useLiveView]
+ * @typedef {Object} PuppeteerPoolOptions
+ * @property {boolean} [useLiveView]
* Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
* just before a page would be recycled by `PuppeteerPool`. If there are no clients
* connected, it has close to zero impact on performance.
- * @param {number} [options.maxOpenPagesPerInstance=50]
+ * @property {number} [maxOpenPagesPerInstance=50]
* Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.
- * @param {number} [options.retireInstanceAfterRequestCount=100]
+ * @property {number} [retireInstanceAfterRequestCount=100]
* Maximum number of requests that can be processed by a single browser instance.
* After the limit is reached, the browser is retired and new requests are
* handled by a new browser instance.
- * @param {number} [options.puppeteerOperationTimeoutSecs=15]
+ * @property {number} [puppeteerOperationTimeoutSecs=15]
* All browser management operations such as launching a new browser, opening a new page
* or closing a page will timeout after the set number of seconds and the connected
* browser will be retired.
- * @param {number} [options.instanceKillerIntervalSecs=60]
+ * @property {number} [instanceKillerIntervalSecs=60]
* Indicates how often are the open Puppeteer instances checked whether they can be closed.
- * @param {number} [options.killInstanceAfterSecs=300]
- * When Puppeteer instance reaches the `options.retireInstanceAfterRequestCount` limit then
+ * @property {number} [killInstanceAfterSecs=300]
+ * When Puppeteer instance reaches the `retireInstanceAfterRequestCount` limit then
* it is considered retired and no more tabs will be opened. After the last tab is closed the
* whole browser is closed too. This parameter defines a time limit between the last tab was opened and
* before the browser is closed even if there are pending open tabs.
- * @param {Function} [options.launchPuppeteerFunction]
+ * @property {Function} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* The function must return a promise resolving to
* [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.
* See the source code on
* <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
* for the default implementation.
- * @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]
+ * @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]
* Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.
* See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
- * @param {boolean} [options.recycleDiskCache=false]
+ * @property {boolean} [recycleDiskCache=false]
* Enables recycling of disk cache directories by Chrome instances.
* When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.
* This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
* Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.
*
* Beware that the disk cache directories can consume a lot of disk space.
- * To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `options.launchPuppeteerOptions.args`,
+ * To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `launchPuppeteerargs`,
* where `X` is the approximate maximum number of bytes for disk cache.
*
- * Do not use the `options.recycleDiskCache` setting together with `--disk-cache-dir`
- * argument in `options.launchPuppeteerOptions.args`, the behavior is undefined.
- * @param {boolean} [options.useIncognitoPages]
+ * Do not use the `recycleDiskCache` setting together with `--disk-cache-dir`
+ * argument in `launchPuppeteerargs`, the behavior is undefined.
+ * @property {boolean} [useIncognitoPages]
* With this option selected, all pages will be opened in a new incognito browser context, which means
* that they will not share cookies or cache and their resources will not be throttled by one another.
- * @param {string[]} [options.proxyUrls]
+ * @property {string[]} [proxyUrls]
* An array of custom proxy URLs to be used by the `PuppeteerPool` instance.
* The provided custom proxies' order will be randomized and the resulting list rotated.
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
* configuration options will cause an error to be thrown on startup.
*/
+
+/**
+ * Manages a pool of Chrome browser instances controlled using
+ * <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
+ *
+ * `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.
+ * This is useful in order to facilitate rotation of proxies, cookies
+ * or other settings in order to prevent detection of your web scraping bot,
+ * access web pages from various countries etc.
+ *
+ * Additionally, the reuse of browser instances instances speeds up crawling,
+ * and the retirement of instances helps mitigate effects of memory leaks in Chrome.
+ *
+ * `PuppeteerPool` is internally used by the {@link PuppeteerCrawler} class.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const puppeteerPool = new PuppeteerPool({
+ * launchPuppeteerFunction: () => {
+ * // Use a new proxy with a new IP address for each new Chrome instance
+ * return Apify.launchPuppeteer({
+ * useApifyProxy: true,
+ * apifyProxySession: Math.random(),
+ * });
+ * },
+ * });
+ *
+ * const page1 = await puppeteerPool.newPage();
+ * const page2 = await puppeteerPool.newPage();
+ * const page3 = await puppeteerPool.newPage();
+ *
+ * // ... do something with the pages ...
+ *
+ * // Close all browsers.
+ * await puppeteerPool.destroy();
+ * ```
+ */
class PuppeteerPool {
+ /**
+ * @param {PuppeteerPoolOptions} [options]
+ * All `PuppeteerPool` parameters are passed
+ * via an options object with the following keys:
+ */
constructor(options = {}) {
const {
maxOpenPagesPerInstance,
@@ -208,6 +216,7 @@ class PuppeteerPool {
checkParamOrThrow(sessionPool, 'options.sessionPool', 'Maybe Object');
// Config.
+
this.sessionPool = sessionPool;
this.reusePages = reusePages;
this.maxOpenPagesPerInstance = maxOpenPagesPerInstance;
@@ -311,7 +320,7 @@ class PuppeteerPool {
* Takes care of async processes in PuppeteerInstance construction with a Browser.
* @param {Promise<Browser>} browserPromise
* @param {PuppeteerInstance} instance
- * @returns {Promise}
+ * @returns {Promise<void>}
* @ignore
*/
async _initBrowser(browserPromise, instance) {
@@ -580,7 +589,7 @@ class PuppeteerPool {
/**
* Closes all open browsers.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async destroy() {
clearInterval(this.instanceKillerInterval);
@@ -649,7 +658,7 @@ class PuppeteerPool {
* instance from the pool. The browser will continue to process open pages so that they may gracefully finish.
* This is unlike `browser.close()` which will forcibly terminate the browser and all open pages will be closed.
* @param {Browser} browser
- * @return {Promise}
+ * @return {Promise<void>}
*/
async retire(browser) {
const instance = await this._findInstanceByBrowser(browser);
@@ -666,7 +675,7 @@ class PuppeteerPool {
* or flagging it for reuse.
*
* @param {Page} page
- * @return {Promise}
+ * @return {Promise<void>}
*/
async recyclePage(page) {
const snapshotPromise = this.liveViewSnapshotsInProgress.get(page);
@@ -692,7 +701,7 @@ class PuppeteerPool {
* Tells the connected LiveViewServer to serve a snapshot when available.
*
* @param page
- * @return {Promise}
+ * @return {Promise<void>}
*/
async serveLiveViewSnapshot(page) {
const isLiveViewConnected = this.liveViewServer && this.liveViewServer.hasClients();
diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.js
index ed382497d..a070fa317 100644
--- a/src/puppeteer_utils.js
+++ b/src/puppeteer_utils.js
@@ -6,6 +6,8 @@ import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
import LruCache from 'apify-shared/lru_cache';
+import { Page, Response } from 'puppeteer'; // eslint-disable-line no-unused-vars
+
import { RequestQueue, RequestQueueLocal } from './request_queue';
import Request from './request';
import { enqueueLinks } from './enqueue_links/enqueue_links';
diff --git a/src/request.js b/src/request.js
index 6a025c996..8f2055c87 100644
--- a/src/request.js
+++ b/src/request.js
@@ -30,6 +30,51 @@ export function hashPayload(payload) {
.substr(0, 8);
}
+/**
+ * Specifies required and optional fields for constructing a [`Request`](../api/request).
+ *
+ * @typedef {Object} RequestOptions
+ * @property {String} url URL of the web page to crawl. It must be a non-empty string.
+ * @property {String} [uniqueKey] A unique key identifying the request.
+ * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
+ *
+ * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
+ * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
+ * of `http://www.example.com/something`.
+ *
+ * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
+ *
+ * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
+ * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
+ * when requests point to the same URL, but with different methods and payloads. For example: form submits.
+ *
+ * Pass an arbitrary non-empty text value to the `uniqueKey` property
+ * to override the default behavior and specify which URLs shall be considered equal.
+ * @property {String} [method='GET']
+ * @property {String|Buffer} [payload]
+ * HTTP request payload, e.g. for POST requests.
+ * @property {Object} [headers={}]
+ * HTTP headers in the following format:
+ * ```
+ * {
+ * Accept: 'text/html',
+ * 'Content-Type': 'application/json'
+ * }
+ * ```
+ * @property {Object} [userData={}]
+ * Custom user data assigned to the request. Use this to save any request related data to the
+ * request's scope, keeping them accessible on retries, failures etc.
+ * @property {Boolean} [keepUrlFragment=false]
+ * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
+ * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
+ * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
+ * Note that this option only has an effect if `uniqueKey` is not set.
+ * @property {Boolean} [useExtendedUniqueKey=false]
+ * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
+ * properties. This is useful when making requests to the same URL that are differentiated by method
+ * or payload, such as form submit navigations in browsers.
+ */
+
/**
* Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
* The `Request` object also stores information about errors that occurred during processing of the request.
@@ -56,47 +101,7 @@ export function hashPayload(payload) {
*
* const foo = request.userData.foo;
* ```
- * @param {object} options All `Request` parameters are passed
- * via an options object with the following keys:
- * @param {String} options.url URL of the web page to crawl. It must be a non-empty string.
- * @param {String} [options.uniqueKey] A unique key identifying the request.
- * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
- *
- * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
- * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
- * of `http://www.example.com/something`.
- *
- * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
- *
- * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
- * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
- * when requests point to the same URL, but with different methods and payloads. For example: form submits.
- *
- * Pass an arbitrary non-empty text value to the `uniqueKey` property
- * to override the default behavior and specify which URLs shall be considered equal.
- * @param {String} [options.method='GET']
- * @param {String|Buffer} [options.payload]
- * HTTP request payload, e.g. for POST requests.
- * @param {Object} [options.headers={}]
- * HTTP headers in the following format:
- * ```
- * {
- * Accept: 'text/html',
- * 'Content-Type': 'application/json'
- * }
- * ```
- * @param {Object} [options.userData={}]
- * Custom user data assigned to the request. Use this to save any request related data to the
- * request's scope, keeping them accessible on retries, failures etc.
- * @param {Boolean} [options.keepUrlFragment=false]
- * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
- * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
- * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
- * Note that this option only has an effect if `uniqueKey` is not set.
- * @param {Boolean} [options.useExtendedUniqueKey=false]
- * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
- * properties. This is useful when making requests to the same URL that are differentiated by method
- * or payload, such as form submit navigations in browsers.
+ * @param {RequestOptions} options `Request` parameters including the URL, HTTP method and headers, and others.
*
* @property {String} id
* Request ID
diff --git a/src/request_list.js b/src/request_list.js
index ab3fdc208..fb4a4e98a 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -10,6 +10,73 @@ import { getValue, setValue } from './key_value_store';
export const STATE_PERSISTENCE_KEY = 'REQUEST_LIST_STATE';
export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
+/**
+ * @typedef {Object} RequestListOptions
+ * @property {Array<RequestOptions|Request>} sources
+ * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
+ * define the `url` property, or an array of instances of the {@link Request} class.
+ * Additionally, the `requestsFromUrl` property may be used instead of `url`,
+ * which will instruct `RequestList` to download the source URLs from a given remote location.
+ * The URLs will be parsed from the received response.
+ *
+ * ```
+ * [
+ * // One URL
+ * { method: 'GET', url: 'http://example.com/a/b' },
+ * // Batch import of URLs from a file hosted on the web
+ * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
+ * // Batch import combined with regex.
+ * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ * ]
+ * ```
+ * @property {String} [persistStateKey]
+ * Identifies the key in the default key-value store under which `RequestList` periodically stores its
+ * state (i.e. which URLs were crawled and which not).
+ * If the actor is restarted, `RequestList` will read the state
+ * and continue where it left off.
+ *
+ * If `persistStateKey` is not set, `RequestList` will always start from the beginning,
+ * and all the source URLs will be crawled again.
+ * @property {String} [persistSourcesKey]
+ * Identifies the key in the default key-value store under which the `RequestList` persists its
+ * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.
+ * This is necessary if `persistStateKey` is set and the source URLs might potentially change,
+ * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.
+ *
+ * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources
+ * from their origin, check that they are consistent with the restored state (if any)
+ * and throw an error if they are not.
+ * @property {RequestListState} [state]
+ * The state object that the `RequestList` will be initialized from.
+ * It is in the form as returned by `RequestList.getState()`, such as follows:
+ *
+ * ```
+ * {
+ * nextIndex: 5,
+ * nextUniqueKey: 'unique-key-5'
+ * inProgress: {
+ * 'unique-key-1': true,
+ * 'unique-key-4': true,
+ * },
+ * }
+ * ```
+ *
+ * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
+ * is to use the `stateKeyPrefix` parameter instead.
+ * @property {Boolean} [keepDuplicateUrls=false]
+ * By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
+ * on the `uniqueKey` property of passed source {@link Request} objects.
+ *
+ * If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
+ * In any case, only one request per `uniqueKey` is added to the `RequestList` resulting in removal
+ * of duplicate URLs / unique keys.
+ *
+ * Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey`
+ * of each request that does not already include a `uniqueKey`. Therefore, duplicate
+ * URLs will be kept in the list. It does not protect the user from having duplicates in user set
+ * `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys
+ * if they wish to keep more than just a single copy in the `RequestList`.
+ */
/**
* Represents a static list of URLs to crawl.
* The URLs can be provided either in code or parsed from a text file hosted on the web.
@@ -74,75 +141,11 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
* // If processing fails then reclaim it back to the list
* await requestList.reclaimRequest(request2);
* ```
- *
- * @param {Object} options All `RequestList` parameters are passed
- * via an options object with the following keys:
- * @param {Array} options.sources
- * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
- * define the `url` property, or an array of instances of the {@link Request} class.
- * Additionally, the `requestsFromUrl` property may be used instead of `url`,
- * which will instruct `RequestList` to download the source URLs from a given remote location.
- * The URLs will be parsed from the received response.
- *
- * ```
- * [
- * // One URL
- * { method: 'GET', url: 'http://example.com/a/b' },
- * // Batch import of URLs from a file hosted on the web
- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
- * // Batch import combined with regex.
- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
- * ]
- * ```
- * @param {String} [options.persistStateKey]
- * Identifies the key in the default key-value store under which `RequestList` periodically stores its
- * state (i.e. which URLs were crawled and which not).
- * If the actor is restarted, `RequestList` will read the state
- * and continue where it left off.
- *
- * If `persistStateKey` is not set, `RequestList` will always start from the beginning,
- * and all the source URLs will be crawled again.
- * @param {String} [options.persistSourcesKey]
- * Identifies the key in the default key-value store under which the `RequestList` persists its
- * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.
- * This is necessary if `persistStateKey` is set and the source URLs might potentially change,
- * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.
- *
- * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources
- * from their origin, check that they are consistent with the restored state (if any)
- * and throw an error if they are not.
- * @param {Object} [options.state]
- * The state object that the `RequestList` will be initialized from.
- * It is in the form as returned by `RequestList.getState()`, such as follows:
- *
- * ```
- * {
- * nextIndex: 5,
- * nextUniqueKey: 'unique-key-5'
- * inProgress: {
- * 'unique-key-1': true,
- * 'unique-key-4': true,
- * },
- * }
- * ```
- *
- * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
- * is to use the `stateKeyPrefix` parameter instead.
- * @param {Boolean} [options.keepDuplicateUrls=false]
- * By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
- * on the `uniqueKey` property of passed source {@link Request} objects.
- *
- * If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
- * In any case, only one request per `uniqueKey` is added to the `RequestList` resulting in removal
- * of duplicate URLs / unique keys.
- *
- * Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey`
- * of each request that does not already include a `uniqueKey`. Therefore, duplicate
- * URLs will be kept in the list. It does not protect the user from having duplicates in user set
- * `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys
- * if they wish to keep more than just a single copy in the `RequestList`.
*/
export class RequestList {
+ /**
+ * @param {RequestListOptions} options All `RequestList` configuration options
+ */
constructor(options = {}) {
checkParamOrThrow(options, 'options', 'Object');
@@ -195,7 +198,7 @@ export class RequestList {
* Loads all remote sources of URLs and potentially starts periodic state persistence.
* This function must be called before you can start using the instance in a meaningful way.
*
- * @returns {Promise}
+ * @returns {Promise<void>}
*/
async initialize() {
if (this.isLoading) {
@@ -240,7 +243,7 @@ export class RequestList {
* or stop fetching its requests. For example after you pause or abort a crawl. Or just before
* a server migration.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async persistState() {
if (!this.persistStateKey) {
@@ -260,7 +263,7 @@ export class RequestList {
* are automatically persisted at RequestList initialization (if the persistSourcesKey is set),
* but there's no reason to persist it again afterwards, because RequestList is immutable.
*
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
async _persistSources() {
@@ -271,7 +274,7 @@ export class RequestList {
/**
* Restores RequestList state from a state object.
*
- * @param {Object} state
+ * @param {RequestListState} state
* @ignore
*/
_restoreState(state) {
@@ -354,7 +357,7 @@ export class RequestList {
* Returns an object representing the internal state of the `RequestList` instance.
* Note that the object's fields can change in future releases.
*
- * @returns {Object}
+ * @returns {RequestListState}
*/
getState() {
this._ensureIsInitialized();
@@ -430,7 +433,7 @@ export class RequestList {
*
* @param {Request} request
*
- * @returns {Promise}
+ * @returns {Promise<void>}
*/
async markRequestHandled(request) {
const { uniqueKey } = request;
@@ -449,7 +452,7 @@ export class RequestList {
*
* @param {Request} request
*
- * @returns {Promise}
+ * @returns {Promise<void>}
*/
async reclaimRequest(request) {
const { uniqueKey } = request;
@@ -488,7 +491,7 @@ export class RequestList {
/**
* Fetches URLs from requestsFromUrl and returns them in format of list of requests
* @param source
- * @return {Promise<Object[]|Array>}
+ * @return {Promise<RequestOptions[]>}
* @ignore
*/
async _fetchRequestsFromUrl(source) {
@@ -638,7 +641,7 @@ export class RequestList {
*
* If `null`, the list will not be persisted and will only be stored in memory. Process restart
* will then cause the list to be crawled again from the beginning. We suggest always using a name.
- * @param {Object[]|string[]} sources
+ * @param {Array<Request|RequestOptions|string>} sources
* An array of sources of URLs for the `RequestList`.
* It can be either an array of plain objects that
* define the `url` property, or an array of instances of the {@link Request} class.
@@ -650,7 +653,7 @@ export class RequestList {
*
* For details, see the [`RequestList`](requestlist#new_RequestList_new)
* constructor options.
- * @param {Object} [options]
+ * @param {RequestListOptions} [options]
* The [`new RequestList`](requestlist#new_RequestList_new) options. Note that the listName parameter supersedes
* the `persistStateKey` and `persistSourcesKey` options and the sources parameter supersedes the `sources` option.
* @returns {Promise<RequestList>}
@@ -675,3 +678,28 @@ export const openRequestList = async (listName, sources, options = {}) => {
await rl.initialize();
return rl;
};
+
+/**
+ * Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed.
+ * You can obtain the state by calling [`RequestList.getState()`](../api/requestlist#getState()) and receive an object with
+ * the following structure:
+ *
+ * ```
+ * {
+ * nextIndex: 5,
+ * nextUniqueKey: 'unique-key-5'
+ * inProgress: {
+ * 'unique-key-1': true,
+ * 'unique-key-4': true
+ * },
+ * }
+ * ```
+ *
+ * @typedef {Object} RequestListState
+ * @property {Number} nextIndex
+ * Position of the next request to be processed.
+ * @property {String} nextUniqueKey
+ * Key of the next request to be processed.
+ * @property {Object<String,Boolean>} inProgress
+ * An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.
+ */
diff --git a/src/request_queue.js b/src/request_queue.js
index fa6c27f21..0a088e933 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -7,7 +7,7 @@ import ListDictionary from 'apify-shared/list_dictionary';
import { ENV_VARS, LOCAL_STORAGE_SUBDIRS, REQUEST_QUEUE_HEAD_MAX_LIMIT } from 'apify-shared/consts';
import { checkParamPrototypeOrThrow, cryptoRandomObjectId } from 'apify-shared/utilities';
import log from 'apify-shared/log';
-import Request from './request';
+import Request, { RequestOptions } from './request'; // eslint-disable-line import/named,no-unused-vars
import { ensureDirExists, apifyClient, openRemoteStorage, openLocalStorage, ensureTokenOrLocalStorageEnvExists, sleep } from './utils';
export const LOCAL_STORAGE_SUBDIR = LOCAL_STORAGE_SUBDIRS.requestQueues;
@@ -241,7 +241,7 @@ export class RequestQueue {
* To add multiple requests to the queue by extracting links from a webpage,
* see the [`Apify.utils.enqueueLinks()`](utils#utils.enqueueLinks) helper function.
*
- * @param {Request|Object} request {@link Request} object or vanilla object with request data.
+ * @param {Request|RequestOptions} request {@link Request} object or vanilla object with request data.
* Note that the function sets the `uniqueKey` and `id` fields to the passed object.
* @param {Object} [options]
* @param {Boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.
diff --git a/src/session_pool/session.js b/src/session_pool/session.js
index 142f461d5..b79baa5dc 100644
--- a/src/session_pool/session.js
+++ b/src/session_pool/session.js
@@ -1,14 +1,52 @@
import { cryptoRandomObjectId } from 'apify-shared/utilities';
import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import tough from 'tough-cookie';
-import EVENTS from './events';
+import { Cookie, CookieJar } from 'tough-cookie';
+import EVENTS from './events'; // eslint-disable-line import/named,no-unused-vars
import { STATUS_CODES_BLOCKED } from '../constants';
import { getCookiesFromResponse } from './session_utils';
-const { Cookie } = tough;
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order,import/no-cycle */
+import { SessionPool } from './session_pool';
+import { Cookie as PuppeteerCookie } from 'puppeteer';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order,import/no-cycle */
+/**
+ * Persistable {Session} state.
+ * @typedef {Object} SessionState
+ * @property {String} id
+ * @property {Object} cookies
+ * @property {Object} cookieJar
+ * @property {Object} userData
+ * @property {Number} maxErrorScore
+ * @property {Number} errorScoreDecrement
+ * @property {String} expiresAt
+ * @property {String} createdAt
+ * @property {Number} usageCount
+ * @property {Number} errorScore
+ */
+/**
+ * @typedef {Object} SessionOptions
+ * @property {String} [id] - Id of session used for generating fingerprints. It is used as proxy session name.
+ * @property {Number} [maxAgeSecs=3000] - Number of seconds after which the session is considered as expired.
+ * @property {Object} userData - Object where custom user data can be stored. For example custom headers.
+ * @property {number} [maxErrorScore=3] - Maximum number of marking session as blocked usage.
+ * If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
+ * It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
+ * Calling the `markGood` will decrease the `errorScore` by `errorScoreDecrement`
+ * @property {number} [errorScoreDecrement=0.5] - It is used for healing the session.
+ * For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this
+ * number.
+ * @property {Date} [createdAt] - Date of creation.
+ * @property {Date} [expiresAt] - Date of expiration.
+ * @property {Number} [usageCount=0] - Indicates how many times the session has been used.
+ * @property {Number} [errorCount=0] - Indicates how many times the session is marked bad.
+ * @property {Number} [maxUsageCount=50] - Session should be used only a limited amount of times.
+ * This number indicates how many times the session is going to be used, before it is thrown away.
+ * @property {SessionPool} sessionPool - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
+ */
/**
* Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions.
* You can imagine each session as a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint.
@@ -17,28 +55,13 @@ const { Cookie } = tough;
export class Session {
/**
* Session configuration.
- * @param [options.id] {String} - Id of session used for generating fingerprints. It is used as proxy session name.
- * @param [options.maxAgeSecs=3000] {Number} - Number of seconds after which the session is considered as expired.
- * @param options.userData {Object} - Object where custom user data can be stored. For example custom headers.
- * @param [options.maxErrorScore=3] {number} - Maximum number of marking session as blocked usage.
- * If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
- * It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
- * Calling the `markGood` will decrease the `errorScore` by `errorScoreDecrement`
- * @param [options.errorScoreDecrement=0.5] {number} - It is used for healing the session.
- * For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this number.
- * @param options.createdAt {Date} - Date of creation.
- * @param options.expiredAt {Date} - Date of expiration.
- * @param [options.usageCount=0] {Number} - Indicates how many times the session has been used.
- * @param [options.errorCount=0] {Number} - Indicates how many times the session is marked bad.
- * @param [options.maxUsageCount=50] {Number} - Session should be used only a limited amount of times.
- * This number indicates how many times the session is going to be used, before it is thrown away.
- * @param options.sessionPool {EventEmitter} - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
+ * @param {SessionOptions} options
*/
constructor(options = {}) {
const {
id = `session_${cryptoRandomObjectId(10)}`,
cookies = [], // @TODO: Delete, deprecate or leave it as custom cookie persistance?
- cookieJar = new tough.CookieJar(),
+ cookieJar = new CookieJar(),
maxAgeSecs = 3000,
userData = {},
maxErrorScore = 3,
@@ -72,7 +95,8 @@ export class Session {
// Configurable
this.id = id;
this.cookies = cookies;
- this.cookieJar = cookieJar.setCookie ? cookieJar : tough.CookieJar.fromJSON(JSON.stringify(cookieJar));
+ /** @type CookieJar */
+ this.cookieJar = cookieJar.setCookie ? cookieJar : CookieJar.fromJSON(JSON.stringify(cookieJar));
this.maxAgeSecs = maxAgeSecs;
this.userData = userData;
this.maxErrorScore = maxErrorScore;
@@ -138,7 +162,7 @@ export class Session {
/**
* Gets session state for persistence in KeyValueStore.
- * @return {Object} represents session internal state.
+ * @return {SessionState} represents session internal state.
*/
getState() {
return {
@@ -197,7 +221,7 @@ export class Session {
/**
* Sets cookies from response to the cookieJar.
* Parses cookies from `set-cookie` header and sets them to `Session.cookieJar`.
- * @param response
+ * @param {{ headers }} response
*/
setCookiesFromResponse(response) {
try {
@@ -210,10 +234,21 @@ export class Session {
}
}
+ /**
+ * Persists puppeteer cookies to session for reuse.
+ * @param {PuppeteerCookie} puppeteerCookies - cookie from puppeteer `page.cookies` method.
+ * @param {String} url - Loaded url from page function.
+ */
+ putPuppeteerCookies(puppeteerCookies, url) {
+ const cookies = puppeteerCookies.map(puppeteerCookie => this._transformPuppeteerCookie(puppeteerCookie));
+
+ this.setCookies(cookies, url);
+ }
+
/**
* Set cookies to session cookieJar.
* Cookies array should be [puppeteer](https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-pagecookiesurls) cookie compatible.
- * @param cookies {Array<Object>}
+ * @param cookies {PuppeteerCookie[]}
* @param url {String}
*/
setPuppeteerCookies(cookies, url) {
@@ -228,7 +263,7 @@ export class Session {
/**
* Gets cookies in puppeteer ready to be used with `page.setCookie`.
* @param url {String} - website url. Only cookies stored for this url will be returned
- * @return {Array<Object>}
+ * @return {PuppeteerCookie[]}
*/
getPuppeteerCookies(url) {
const cookies = this.cookieJar.getCookiesSync(url);
@@ -238,7 +273,7 @@ export class Session {
/**
* Wrapper around `tough-cookie` Cookie jar `getCookieString` method.
- * @param url
+ * @param {String} url
* @return {String} - represents `Cookie` header.
*/
getCookieString(url) {
@@ -248,7 +283,7 @@ export class Session {
/**
* Transforms puppeteer cookie to tough-cookie.
- * @param puppeteerCookie {Object} - Cookie from puppeteer `page.cookies method.
+ * @param puppeteerCookie {PuppeteerCookie} - Cookie from puppeteer `page.cookies method.
* @return {Cookie}
* @private
*/
@@ -266,8 +301,8 @@ export class Session {
/**
* Transforms tough-cookie cookie to puppeteer Cookie .
- * @param toughCookie - Cookie from CookieJar.
- * @return {Object} - puppeteer cookie
+ * @param {Cookie} toughCookie - Cookie from CookieJar.
+ * @return {PuppeteerCookie} - puppeteer cookie
* @private
*/
_toughCookieToPuppeteer(toughCookie) {
@@ -284,8 +319,8 @@ export class Session {
/**
* Sets cookies.
- * @param cookies
- * @param url
+ * @param {Cookie} cookies
+ * @param {String} url
* @private
*/
_setCookies(cookies, url) {
diff --git a/src/session_pool/session_pool.js b/src/session_pool/session_pool.js
index 639862ead..3e4215b71 100644
--- a/src/session_pool/session_pool.js
+++ b/src/session_pool/session_pool.js
@@ -1,13 +1,30 @@
-import EventEmitter from 'events';
+import { EventEmitter } from 'events';
import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { openKeyValueStore } from '../key_value_store';
-import { Session } from './session';
+import { Session, SessionOptions } from './session'; // eslint-disable-line no-unused-vars,import/named,import/no-cycle
import events from '../events';
import { ACTOR_EVENT_NAMES_EX } from '../constants';
+/**
+ * Factory user-function which creates customized {@link Session} instances.
+ * @callback CreateSession
+ * @param {SessionPool} sessionPool Pool requesting the new session.
+ */
+
+/**
+ * @typedef {Object} SessionPoolOptions
+ * @property {Number} [maxPoolSize=1000] - Maximum size of the pool.
+ * Indicates how many sessions are rotated.
+ * @property {SessionOptions} [sessionOptions] The configuration options for {Session} instances.
+ * @property {String} [persistStateKeyValueStoreId] - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
+ * @property {String} [persistStateKey="SESSION_POOL_STATE"] - Session pool persists it's state under this key in Key value store.
+ * @property {CreateSession} [createSessionFunction] - Custom function that should return `Session` instance.
+ * Function receives `SessionPool` instance as a parameter
+ */
+// TODO yin: `tsc` generates a class declaration containing EventEmitter methods with wrong return type (`:SessionPool instead of `:this`).
/**
* Handles the sessions rotation, creation and persistence.
* Creates a pool of {@link Session} instances, that are randomly rotated.
@@ -57,14 +74,7 @@ import { ACTOR_EVENT_NAMES_EX } from '../constants';
export class SessionPool extends EventEmitter {
/**
* Session pool configuration.
- * @param [options]
- * @param [options.maxPoolSize=1000] {Number} - Maximum size of the pool.
- * Indicates how many sessions are rotated.
- * @param [options.sessionOptions] {Object} The [`new Session`](session#new_SessionPool_new) options
- * @param [options.persistStateKeyValueStoreId] {String} - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
- * @param [options.persistStateKey="SESSION_POOL_STATE"] {String} - Session pool persists it's state under this key in Key value store.
- * @param [options.createSessionFunction] {function} - Custom function that should return `Session` instance.
- * Function receives `SessionPool` instance as a parameter
+ * @param {SessionPoolOptions} [options] All `SessionPool` configuration options.
*/
constructor(options = {}) {
const {
@@ -308,7 +318,7 @@ export class SessionPool extends EventEmitter {
*
* For more details and code examples, see the {@link SessionPool} class.
*
- * @param sessionPoolOptions {Object} The [`new SessionPool`](sessionpool#new_SessionPool_new) options
+ * @param {SessionPoolOptions} sessionPoolOptions The [`new SessionPool`](sessionpool#new_SessionPool_new) options
* @return {Promise<SessionPool>}
* @memberof module:Apify
* @name openSessionPool
diff --git a/src/stealth/stealth.js b/src/stealth/stealth.js
index bfcce9228..d474bf462 100644
--- a/src/stealth/stealth.js
+++ b/src/stealth/stealth.js
@@ -1,4 +1,5 @@
import _ from 'underscore';
+import { Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
import hidingTricks from './hiding_tricks';
/**
diff --git a/src/typedefs.js b/src/typedefs.js
new file mode 100644
index 000000000..fa53336d5
--- /dev/null
+++ b/src/typedefs.js
@@ -0,0 +1,94 @@
+/**
+ * `CherioStatic`, the return type of `cheerio.load()`, is not exported from `@types/cheerio`, so it can not be imported and aliased, it is a global
+ * type. We reference the type and redefine it with a bettter name here.
+ * @typedef {CheerioStatic} Cheerio
+ * @ignore
+ */
+
+/**
+ * Represents information about an actor run, as returned by the
+ * [`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.
+ * The object is almost equivalent to the JSON response
+ * of the
+ * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>
+ * Apify API endpoint and extended with certain fields.
+ * For more details, see
+ * <a href="https://docs.apify.com/actor/run" target="_blank">Runs.</a>
+ *
+ * @typedef {Object} ActorRun
+ * @property {String} id
+ * Actor run ID
+ * @property {String} actId
+ * Actor ID
+ * @property {Date} startedAt
+ * Time when the actor run started
+ * @property {Date} finishedAt
+ * Time when the actor run finished. Contains `null` for running actors.
+ * @property {String} status
+ * Status of the run. For possible values, see
+ * <a href="https://docs.apify.com/actor/run#lifecycle" target="_blank">Run lifecycle</a>
+ * in Apify actor documentation.
+ * @property {Object} meta
+ * Actor run meta-data. For example:
+ * ```
+ * {
+ * "origin": "API",
+ * "clientIp": "1.2.3.4",
+ * "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"
+ * }
+ * ```
+ * @property {Object} stats
+ * An object containing various actor run statistics. For example:
+ * ```
+ * {
+ * "inputBodyLen": 22,
+ * "restartCount": 0,
+ * "workersUsed": 1,
+ * }
+ * ```
+ * Beware that object fields might change in future releases.
+ * @property {Object} options
+ * Actor run options. For example:
+ * ```
+ * {
+ * "build": "latest",
+ * "waitSecs": 0,
+ * "memoryMbytes": 256,
+ * "diskMbytes": 512
+ * }
+ * ```
+ * @property {String} buildId
+ * ID of the actor build used for the run. For details, see
+ * <a href="https://docs.apify.com/actor/build" target="_blank">Builds</a>
+ * in Apify actor documentation.
+ * @property {String} buildNumber
+ * Number of the actor build used for the run. For example, `0.0.10`.
+ * @property {Number} exitCode
+ * Exit code of the actor run process. It's `null` if actor is still running.
+ * @property {String} defaultKeyValueStoreId
+ * ID of the default key-value store associated with the actor run. See [`KeyValueStore`](../api/keyvaluestore) for details.
+ * @property {String} defaultDatasetId
+ * ID of the default dataset associated with the actor run. See [`Dataset`](../api/dataset) for details.
+ * @property {String} defaultRequestQueueId
+ * ID of the default request queue associated with the actor run. See [`RequestQueue`](../api/requestqueue) for details.
+ * @property {String} containerUrl
+ * URL on which the web server running inside actor run's Docker container can be accessed.
+ * For more details, see
+ * <a href="https://docs.apify.com/actor/run#container-web-server" target="_blank">Container web server</a>
+ * in Apify actor documentation.
+ * @property {Object} output
+ * Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,
+ * or if you pass `false` to the `fetchOutput` option of [`Apify.call()`](../api/apify#module_Apify.call).
+ *
+ * For example:
+ * ```
+ * {
+ * "contentType": "application/json; charset=utf-8",
+ * "body": {
+ * "message": "Hello world!"
+ * }
+ * }
+ * ```
+ */
+
+export {};
diff --git a/src/utils.js b/src/utils.js
index 195c628e6..f1d977737 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -1,24 +1,31 @@
-import contentTypeParser from 'content-type';
-import os from 'os';
-import _ from 'underscore';
-import fs from 'fs';
-import util from 'util';
-import fsExtra from 'fs-extra';
-import ApifyClient from 'apify-client';
import psTree from '@apify/ps-tree';
-import requestPromise from 'request-promise-native';
-import cheerio from 'cheerio';
-import log from 'apify-shared/log';
-import semver from 'semver';
-import { getRandomInt } from 'apify-shared/utilities';
-import { ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';
+import ApifyClient from 'apify-client';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { version as apifyClientVersion } from 'apify-client/package.json';
+import { ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';
+import log from 'apify-shared/log';
+import { getRandomInt } from 'apify-shared/utilities';
+import cheerio from 'cheerio';
+import contentTypeParser from 'content-type';
+import fs from 'fs';
+import fsExtra from 'fs-extra';
import mime from 'mime-types';
+import os from 'os';
import path from 'path';
+import requestPromise from 'request-promise-native';
+import semver from 'semver';
+import _ from 'underscore';
import { URL } from 'url';
-import { version as apifyVersion } from '../package.json';
+import util from 'util';
import { USER_AGENT_LIST } from './constants';
+import { version as apifyVersion } from '../package.json';
+
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
+import { IncomingMessage } from 'http';
+import { Response as PuppeteerResponse } from 'puppeteer';
+import { Cheerio } from './typedefs';
+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
/**
* Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters
@@ -83,6 +90,8 @@ export const logSystemInfo = () => {
* target="_blank">`Apify.client.setOptions()`</a> function.
* Beware that altering these settings might have unintended effects on the entire Apify SDK package.
*
+ * @type {*}
+ *
* @memberof module:Apify
* @name client
*/
@@ -91,7 +100,7 @@ export const apifyClient = newClient();
/**
* Returns a result of `Promise.resolve()`.
*
- * @returns {*}
+ * @returns {Promise<void>}
*
* @ignore
*/
@@ -139,7 +148,8 @@ const createIsDockerPromise = () => {
/**
* Returns a `Promise` that resolves to true if the code is running in a Docker container.
*
- * @return {Promise}
+ * @param {boolean} forceReset
+ * @return {Promise<boolean>}
*
* @memberof utils
* @name isDocker
@@ -155,7 +165,7 @@ export const isDocker = (forceReset) => {
/**
* Sums an array of numbers.
*
- * @param {Array} arr An array of numbers.
+ * @param {Number[]} arr An array of numbers.
* @return {Number} Sum of the numbers.
*
* @ignore
@@ -165,7 +175,7 @@ export const sum = arr => arr.reduce((total, c) => total + c, 0);
/**
* Computes an average of an array of numbers.
*
- * @param {Array} arr An array of numbers.
+ * @param {Number[]} arr An array of numbers.
* @return {Number} Average value.
*
* @ignore
@@ -175,8 +185,8 @@ export const avg = arr => sum(arr) / arr.length;
/**
* Computes a weighted average of an array of numbers, complemented by an array of weights.
*
- * @param {Array} arrValues
- * @param {Array} arrWeights
+ * @param {Number[]} arrValues
+ * @param {Number[]} arrWeights
* @return {Number}
*
* @ignore
@@ -193,22 +203,18 @@ export const weightedAvg = (arrValues, arrWeights) => {
};
/**
- * Returns memory statistics of the process and the system, which is an object with the following properties:
+ * Describes memory usage of an Actor.
*
- * ```javascript
- * {
- * // Total memory available in the system or container
- * totalBytes: Number,
- * // Amount of free memory in the system or container
- * freeBytes: Number,
- * // Amount of memory used (= totalBytes - freeBytes)
- * usedBytes: Number,
- * // Amount of memory used the current Node.js process
- * mainProcessBytes: Number,
- * // Amount of memory used by child processes of the current Node.js process
- * childProcessesBytes: Number,
- * }
- * ```
+ * @typedef {Object} MemoryInfo
+ * @property {Number} totalBytes Total memory available in the system or container
+ * @property {Number} freeBytes Amount of free memory in the system or container
+ * @property {Number} usedBytes Amount of memory used (= totalBytes - freeBytes)
+ * @property {Number} mainProcessBytes Amount of memory used the current Node.js process
+ * @property {Number} childProcessesBytes Amount of memory used by child processes of the current Node.js process
+ */
+
+/**
+ * Returns memory statistics of the process and the system, see {@link MemoryInfo}.
*
* If the process runs inside of Docker, the `getMemoryInfo` gets container memory limits,
* otherwise it gets system memory limits.
@@ -216,7 +222,7 @@ export const weightedAvg = (arrValues, arrWeights) => {
* Beware that the function is quite inefficient because it spawns a new process.
* Therefore you shouldn't call it too often, like more than once per second.
*
- * @returns {Promise<Object>}
+ * @returns {Promise<MemoryInfo>}
*
* @memberof module:Apify
* @name getMemoryInfo
@@ -383,7 +389,7 @@ export const isAtHome = () => !!process.env[ENV_VARS.IS_AT_HOME];
* @param {Number} millis Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
* @memberof utils
* @name sleep
- * @return {Promise}
+ * @return {Promise<void>}
*/
export const sleep = (millis) => {
return new Promise(res => setTimeout(res, millis));
@@ -521,7 +527,7 @@ const BLOCK_TAGS_REGEX = /^(p|h1|h2|h3|h4|h5|h6|ol|ul|li|pre|address|blockquote|
* const html = '<html><body>Some text</body></html>';
* const text = htmlToText(cheerio.load(html, { decodeEntities: true }));
* ```
- * @param {String|Function} html HTML text or parsed HTML represented using a
+ * @param {String|Cheerio} html HTML text or parsed HTML represented using a
* [cheerio](https://www.npmjs.com/package/cheerio) function.
* @return {String} Plain text
* @memberOf utils
@@ -529,6 +535,7 @@ const BLOCK_TAGS_REGEX = /^(p|h1|h2|h3|h4|h5|h6|ol|ul|li|pre|address|blockquote|
const htmlToText = (html) => {
if (!html) return '';
+ /** @type {Cheerio} */
const $ = typeof html === 'function' ? html : cheerio.load(html, { decodeEntities: true });
let text = '';
@@ -571,8 +578,8 @@ const htmlToText = (html) => {
/**
* Creates a standardized debug info from request and response. This info is usually added to dataset under the hidden `#debug` field.
*
- * @param {Object} request [Apify.Request](https://sdk.apify.com/docs/api/request) object.
- * @param {Object} [response]
+ * @param {Request|RequestOptions} request [Apify.Request](https://sdk.apify.com/docs/api/request) object.
+ * @param {IncomingMessage|PuppeteerResponse} [response]
* Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&version=v1.11.0&show=api-class-response" target="_blank"><code>Response</code></a>
* or NodeJS <a href="https://nodejs.org/api/http.html#http_class_http_serverresponse" target="_blank"><code>http.ServerResponse</code></a>.
* @param {Object} [additionalFields] Object containing additional fields to be added.
diff --git a/src/utils_log.d.ts b/src/utils_log.d.ts
new file mode 100644
index 000000000..bbf38366e
--- /dev/null
+++ b/src/utils_log.d.ts
@@ -0,0 +1,37 @@
+declare module log {
+ const LEVELS: {
+ // Turns off logging completely
+ OFF: number,
+ // For unexpected errors in Apify system
+ ERROR: number,
+ // For situations where error is caused by user (e.g. Meteor.Error), i.e. when the error is not
+ // caused by Apify system, avoid the word "ERROR" to simplify searching in log
+ SOFT_FAIL: number,
+ WARNING: number,
+ INFO: number,
+ DEBUG: number,
+ // for performance stats
+ PERF: number
+ };
+
+ let logJson: boolean;
+
+ // Indicates whether DEBUG messages will be printed or not
+ let isDebugMode: boolean;
+
+ // Sets log level
+ function setLevel(level: number): void;
+
+ function getLevel(): number;
+
+ // Helper functions for common usage
+ function warning(message: string, data?: any): void;
+
+ function info(message: string, data?: any): void;
+
+ function debug(message: string, data?: any): void;
+
+ function error(message: string, data?: any): void;
+
+ function exception(exception: Error, message?: string, data?: any): void;
+}
diff --git a/src/utils_log.js b/src/utils_log.js
new file mode 100644
index 000000000..2414ece14
--- /dev/null
+++ b/src/utils_log.js
@@ -0,0 +1,139 @@
+import log from 'apify-shared/log';
+
+// Add docs for log separately, as it's imported from apify-shared.
+// Adding them directly to the log object in utils in `index.js` breaks JSDoc.
+
+/**
+ * The log instance enables level aware logging of messages and we advise
+ * to use it instead of `console.log()` and its aliases in most development
+ * scenarios.
+ *
+ * A very useful use case for `log` is using `log.debug` liberally throughout
+ * the codebase to get useful logging messages only when appropriate log level is set
+ * and keeping the console tidy in production environments.
+ *
+ * The available logging levels are, in this order: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `OFF`
+ * and can be referenced from the `log.LEVELS` constant, such as `log.LEVELS.ERROR`.
+ *
+ * To log messages to the system console, use the `log.level(message)` invocation,
+ * such as `log.debug('this is a debug message')`.
+ *
+ * To prevent writing of messages above a certain log level to the console, simply
+ * set the appropriate level. The default log level is `INFO`, which means that
+ * `DEBUG` messages will not be printed, unless enabled.
+ *
+ * **Example:**
+ * ```
+ * const Apify = require('apify');
+ * const { log } = Apify.utils;
+ *
+ * log.info('Information message', { someData: 123 }); // prints message
+ * log.debug('Debug message', { debugData: 'hello' }); // doesn't print anything
+ *
+ * log.setLevel(log.LEVELS.DEBUG);
+ * log.debug('Debug message'); // prints message
+ *
+ * log.setLevel(log.LEVELS.ERROR);
+ * log.debug('Debug message'); // doesn't print anything
+ * log.info('Info message'); // doesn't print anything
+ *
+ * log.error('Error message', { errorDetails: 'This is bad!' }); // prints message
+ * try {
+ * throw new Error('Not good!');
+ * } catch (e) {
+ * log.exception(e, 'Exception occurred', { errorDetails: 'This is really bad!' }); // prints message
+ * }
+ * ```
+ *
+ * Another very useful way of setting the log level is by setting the `APIFY_LOG_LEVEL`
+ * environment variable, such as `APIFY_LOG_LEVEL=DEBUG`. This way, no code changes
+ * are necessary to turn on your debug messages and start debugging right away.
+ * @namespace log
+ */
+
+/**
+ * Map of available log levels that's useful for easy setting of appropriate log levels.
+ * Each log level is represented internally by a number. Eg. `log.LEVELS.DEBUG === 5`.
+ * @name LEVELS
+ * @type Object
+ * @memberOf log
+ */
+
+/**
+ * Sets the log level to the given value, preventing messages from less important log levels
+ * from being printed to the console. Use in conjunction with the `log.LEVELS` constants such as
+ *
+ * ```
+ * log.setLevel(log.LEVELS.DEBUG);
+ * ```
+ *
+ * Default log level is INFO.
+ * @name setLevel
+ * @param {number} level
+ * @method
+ * @memberOf log
+ */
+
+/**
+ * Returns the currently selected logging level. This is useful for checking whether a message
+ * will actually be printed to the console before one actually performs a resource intensive operation
+ * to construct the message, such as querying a DB for some metadata that need to be added. If the log
+ * level is not high enough at the moment, it doesn't make sense to execute the query.
+ * @name getLevel
+ * @method
+ * @memberOf log
+ */
+
+/**
+ * Logs a `DEBUG` message. By default, it will not be written to the console. To see `DEBUG`
+ * messages in the console, set the log level to `DEBUG` either using the `log.setLevel(log.LEVELS.DEBUG)`
+ * method or using the environment variable `APIFY_LOG_LEVEL=DEBUG`. Data are stringified and appended
+ * to the message.
+ * @name debug
+ * @param {string} message
+ * @param {Object} [data]
+ * @method
+ * @memberOf log
+ */
+
+/**
+ * Logs an `INFO` message. `INFO` is the default log level so info messages will be always logged,
+ * unless the log level is changed. Data are stringified and appended to the message.
+ * @name info
+ * @param {string} message
+ * @param {Object} [data]
+ * @method
+ * @memberOf log
+ */
+
+/**
+ * Logs a `WARNING` level message. Data are stringified and appended to the message.
+ * @name warning
+ * @param {string} message
+ * @param {Object} [data]
+ * @method
+ * @memberOf log
+ */
+
+/**
+ * Logs an `ERROR` message. Use this method to log error messages that are not directly connected
+ * to an exception. For logging exceptions, use the `log.exception` method.
+ * @name error
+ * @param {string} message
+ * @param {Object} [data]
+ * @method
+ * @memberOf log
+ */
+
+/**
+ * Logs an `ERROR` level message with a nicely formatted exception. Note that the exception is the first parameter
+ * here and an additional message is only optional.
+ * @name exception
+ * @param {Error} exception
+ * @param {string} [message]
+ * @param {Object} [data]
+ * @method
+ * @memberOf log
+ */
+
+export default log;
diff --git a/src/utils_request.js b/src/utils_request.js
index d0fa0f80c..6664023c8 100644
--- a/src/utils_request.js
+++ b/src/utils_request.js
@@ -26,6 +26,31 @@ export const REQUEST_AS_BROWSER_DEFAULT_OPTIONS = {
proxyUrl: null,
timeoutSecs: 30,
};
+
+/**
+ * @typedef {Object} RequestAsBrowserOptions
+ * @property {String} url
+ * URL of the target endpoint. Supports both HTTP and HTTPS schemes.
+ * @property {String} [method=GET]
+ * HTTP method.
+ * @property {Object} [headers]
+ * Additional HTTP headers to add. It's only recommended to use this option,
+ * with headers that are typically added by websites, such as cookies. Overriding
+ * default browser headers will remove the masking this function provides.
+ * @property {String} [proxyUrl]
+ * An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.
+ * @property {String} [languageCode=en]
+ * Two-letter ISO 639 language code.
+ * @property {String} [countryCode=US]
+ * Two-letter ISO 3166 country code.
+ * @property {Boolean} [isMobile]
+ * If `true`, the function uses User-Agent of a mobile browser.
+ * @property {Function} [abortFunction]
+ * Function accepts `response` object as a single parameter and should return true or false.
+ * If function returns true request gets aborted. This function is passed to the
+ * (@apify/http-request)[https://www.npmjs.com/package/@apify/http-request] NPM package.
+ */
+
/**
* Sends a HTTP request that looks like a request sent by a web browser,
* fully emulating browser's HTTP headers.
@@ -45,26 +70,11 @@ export const REQUEST_AS_BROWSER_DEFAULT_OPTIONS = {
* All `options` not recognized by this function are passed to it,
* so see it for more details.
*
- * @param options.url
- * URL of the target endpoint. Supports both HTTP and HTTPS schemes.
- * @param [options.method=GET]
- * HTTP method.
- * @param [options.headers]
- * Additional HTTP headers to add. It's only recommended to use this option,
- * with headers that are typically added by websites, such as cookies. Overriding
- * default browser headers will remove the masking this function provides.
- * @param [options.languageCode=en]
- * Two-letter ISO 639 language code.
- * @param [options.countryCode=US]
- * Two-letter ISO 3166 country code.
- * @param [options.isMobile]
- * If `true`, the function uses User-Agent of a mobile browser.
- * @param [options.abortFunction]
- * Function accepts `response` object as a single parameter and should return true or false.
- * If function returns true request gets aborted. This function is passed to the
- * (@apify/http-request)[https://www.npmjs.com/package/@apify/http-request] NPM package.
+ * @param {RequestAsBrowserOptions} options All `requestAsBrowser` configuration options.
*
- * @return {Promise<http.IncomingMessage|stream.Readable>}
+ * @return {Promise<http.IncomingMessage|stream.Readable>} This will typically be a
+ * [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage),
+ * however, if returned from the cache it will be a [response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.
* @memberOf utils
* @name requestAsBrowser
*/
diff --git a/src/utils_social.js b/src/utils_social.js
index c57e9643a..3cb1b175c 100644
--- a/src/utils_social.js
+++ b/src/utils_social.js
@@ -203,15 +203,25 @@ const FACEBOOK_REGEX_STRING = `(?<!\\w)(?:http(?:s)?:\\/\\/)?(?:www.)?(?:faceboo
// eslint-disable-next-line max-len, quotes
const YOUTUBE_REGEX_STRING = '(?:https?:\\/\\/)?(?:youtu\\.be\\/|(?:www\\.|m\\.)?youtube\\.com\\/(?:watch|v|embed)(?:\\.php)?(?:\\?.*v=|\\/))([a-zA-Z0-9\\-_]+)';
+/** @type RegExp */
let LINKEDIN_REGEX;
+/** @type RegExp */
let LINKEDIN_REGEX_GLOBAL;
+/** @type RegExp */
let INSTAGRAM_REGEX;
+/** @type RegExp */
let INSTAGRAM_REGEX_GLOBAL;
+/** @type RegExp */
let TWITTER_REGEX;
+/** @type RegExp */
let TWITTER_REGEX_GLOBAL;
+/** @type RegExp */
let FACEBOOK_REGEX;
+/** @type RegExp */
let FACEBOOK_REGEX_GLOBAL;
+/** @type RegExp */
let YOUTUBE_REGEX;
+/** @type RegExp */
let YOUTUBE_REGEX_GLOBAL;
try {
@@ -481,13 +491,11 @@ try {
}
}
-
/**
- * The function attempts to extract emails, phone numbers and social profile URLs from a HTML document,
- * specifically LinkedIn, Twitter, Instagram and Facebook profile URLs.
- * The function removes duplicates from the resulting arrays and sorts the items alphabetically.
+ * Representation of social handles parsed from a HTML page.
+ *
+ * The object has the following structure:
*
- * The result of the function is an object with the following structure:
* ```
* {
* emails: String[],
@@ -500,6 +508,21 @@ try {
* youtubes: String[],
* }
* ```
+ * @typedef SocialHandles
+ * @property {String[]} emails
+ * @property {String[]} phones
+ * @property {String[]} phonesUncertain
+ * @property {String[]} linkedIns
+ * @property {String[]} twitters
+ * @property {String[]} instagrams
+ * @property {String[]} facebooks
+ * @property {String[]} youtubes
+ */
+
+/**
+ * The function attempts to extract emails, phone numbers and social profile URLs from a HTML document,
+ * specifically LinkedIn, Twitter, Instagram and Facebook profile URLs.
+ * The function removes duplicates from the resulting arrays and sorts the items alphabetically.
*
* Note that the `phones` field contains phone numbers extracted from the special phone links
* such as `<a href="tel:+1234556789">call us</a>` (see [`social.phonesFromUrls()`](#social.phonesFromUrls)])
@@ -524,7 +547,7 @@ try {
* @param {Object} data Optional object which will receive the `text` and `$` properties
* that contain text content of the HTML and `cheerio` object, respectively. This is an optimization
* so that the caller doesn't need to parse the HTML document again, if needed.
- * @return {*} An object with the social handles.
+ * @return {SocialHandles} An object with the social handles.
*
* @memberOf social
*/
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index ab0529be4..9368683c5 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -1,5 +1,5 @@
import { Readable } from 'stream';
-import EventEmitter from 'events';
+import { EventEmitter } from 'events';
import rqst from 'request';
import fs from 'fs';
import path from 'path';
diff --git a/test/index.esm.test.js b/test/index.esm.test.js
new file mode 100644
index 000000000..7e9286f33
--- /dev/null
+++ b/test/index.esm.test.js
@@ -0,0 +1,21 @@
+// eslint-disable-next-line import/no-duplicates
+import ApifyDefault from '../build/index';
+// eslint-disable-next-line import/no-duplicates
+import * as ApifyWithWildcard from '../build/index';
+// eslint-disable-next-line global-require
+const apify = require('../build');
+
+describe('Apify module', () => {
+ test('import Apify from \'apify\' - should fail', () => {
+ expect(ApifyDefault).not.toBeUndefined();
+ expect(ApifyDefault.default).toBeUndefined();
+ });
+ test('import * as Apify from \'apify\'', () => {
+ expect(ApifyWithWildcard).not.toBeUndefined();
+ expect(ApifyWithWildcard.default).not.toBeUndefined();
+ });
+ test('const apify = require(\'apify\')', () => {
+ expect(apify).not.toBeUndefined();
+ expect(apify.default).not.toBeUndefined();
+ });
+});
diff --git a/test/typescript/crawlers/cheerio_crawler.test.ts b/test/typescript/crawlers/cheerio_crawler.test.ts
new file mode 100644
index 000000000..fcb78e10c
--- /dev/null
+++ b/test/typescript/crawlers/cheerio_crawler.test.ts
@@ -0,0 +1,71 @@
+/// <reference types="jest"/>
+import Apify, {CheerioCrawler, CheerioHandlePage, CheerioHandlePageInputs, RequestList} from "../../..";
+import cheerio from "cheerio";
+
+describe('CheerioHandlePage', () => {
+ let testInputs: CheerioHandlePageInputs;
+
+ beforeEach(() => {
+ const body = '<a href="#">';
+ testInputs = {
+ $: cheerio.load(body),
+ body: body,
+ request: new Apify.Request({url: '#'}),
+ contentType: {type: 'text/html', encoding: 'utf-8'},
+ response: null,
+ autoscaledPool: null
+ };
+ });
+
+ test('Can pass around and call `handler(var: CheerioHandlePageInputs)`', async () => {
+ // This form can be easily reused returning arbitrary type by passing `inputs` from another handler function and
+ // processing the return value there. If returning anything, this function can't be used directly in a crawler.
+ // Auto-completion works on all input variables in parameter `inputs`.
+ const x = async (inputs: CheerioHandlePageInputs) => {
+ expect(inputs.$('a').attr('href')).toEqual('#');
+ };
+
+ const crawlerX = new CheerioCrawler({
+ handlePageFunction: x,
+ requestList: new RequestList({ sources: [] }),
+ });
+
+ await x(testInputs);
+ });
+
+ test('Can pass around and call `handler({ var }: { var: Type})`', async () => {
+ // This form can also be easily reused as above.
+ // Auto-completion works on defined input variables in parameter list.
+ const y = async ({$}: { $?: CheerioStatic }) => {
+ expect($('a').attr('href')).toEqual('#');
+ };
+
+ const crawler = new CheerioCrawler({
+ handlePageFunction: y,
+ requestList: new RequestList({ sources: [] }),
+
+ });
+
+ await y(testInputs);
+ });
+
+
+ test('Can pass around and call `handler: CheerioHandlePage`', async () => {
+ // In this form, the return type of the function is strictly enforced to be `Promise<void>`. This form can be reused
+ // if it produces the desired side-effects (or nothing) without returning a value. On the other hand, we can describe
+ // the parameter list in any of the above styles without attaching a typing to input variables.
+ // Auto-completion works in both cases.
+ // Type-checking guards from potential errors in logic, be raising an error if return type does not match what is
+ // expected by the crawler.
+ const z: CheerioHandlePage = async ({$ = null}) => {
+ expect($('a').attr('href')).toEqual('#');
+ };
+
+ const crawler = new CheerioCrawler({
+ handlePageFunction: z,
+ requestList: new RequestList({ sources: [] }),
+ });
+
+ await z(testInputs);
+ });
+});
diff --git a/test/typescript/utils_log.test.ts b/test/typescript/utils_log.test.ts
new file mode 100644
index 000000000..a27551ea7
--- /dev/null
+++ b/test/typescript/utils_log.test.ts
@@ -0,0 +1,22 @@
+/// <reference types="jest"/>
+import Apify from "../..";
+
+describe('Typescript with Apify.utils.log', () => {
+ const log = Apify.utils.log;
+
+ test('log.debug()', () => {
+ log.debug('Debug level log');
+ });
+
+ test('log.info()', () => {
+ log.info('Info Level log');
+ });
+
+ test('log.warning()', () => {
+ log.warning('Warning Level log');
+ });
+
+ test('log.error()', () => {
+ log.error('Error Level log');
+ });
+});
diff --git a/tools/typescript_fixes.js b/tools/typescript_fixes.js
new file mode 100644
index 000000000..1f6ceab5a
--- /dev/null
+++ b/tools/typescript_fixes.js
@@ -0,0 +1,215 @@
+const { readdirSync, readFileSync, writeFileSync, realpathSync, statSync } = require('fs');
+const path = require('path');
+
+console.debugFixLog = false ? console.debug : () => {
+};
+console.debugReadTypes = false ? console.debug : () => {
+};
+console.debugRemoveReferences = false ? console.debug : () => {
+};
+
+console.log('Currect directory:', realpathSync('./'));
+const typesPath = './types';
+const paths = {
+ 'index.esm': path.join(typesPath, 'index.esm.d.ts'),
+};
+
+console.log('Processing', paths['index.esm']);
+
+/**
+ * Removes `namespace log` and adds appropriate import and reference to index.esm.d.ts.
+ *
+ * @param {string[]} input
+ * @return {string[]}
+ */
+const fixLog = (input) => {
+ const output = [];
+
+ const modes = {
+ base: 0, // Most reading and writing is in `base` mode: keep each line as is.
+ 'log-ns-replace': 1, // Erasing log namespace declaration and adding proper import
+ };
+
+ let index = 0;
+ let mode = modes.base;
+ for (const line of input) {
+ try {
+ if (mode === modes.base) {
+ if (index === 0) {
+ output.push('/// <reference path="../src/utils_log.d.ts" />');
+ console.debugFixLog('line #1', mode);
+ }
+ if (line.match(/^declare namespace log {/)) {
+ mode = modes['log-ns-replace'];
+ console.debugFixLog('switch to', mode, line);
+ } else {
+ output.push(line);
+ console.debugFixLog('out', line);
+ }
+ } else if (mode === modes['log-ns-replace']) {
+ if (line.match(/^\}/)) {
+ mode = modes.base;
+ console.debugFixLog('switch to', mode, line);
+ } else {
+ // nothing
+ console.debugFixLog('remove ', line);
+ }
+ }
+ } finally {
+ index++;
+ }
+ }
+ return output;
+};
+
+/**
+ * @callback FilenameFilter
+ * @param {String} path Relative path to file considered.
+ * @returns {Boolen} `true` is the path should be processed, `false` otherwise.
+ */
+
+/**
+ * @callback FileHandler
+ * @param {String} content Relative path to file.
+ * @returns {String} Content after transformation.
+ */
+
+/**
+ * Traverses directory recursively, applies `filter()` on it's relative path path and if `true` is returned,
+ * applies `handleFile()` on the file collecting the return values into a dictionary object.
+ * @param {String} dir Starting directory.
+ * @param {FilenameFilter} filter Pathname filter function.
+ * @param {FileHandler} handleFile File processing function.
+ * @returns {object<string,object|array>} Hierarchical collection of `handleFile`'s return values.
+ */
+const traverseDirs = async (dir, filter, handleFile) => {
+ console.log('Reading directory', dir);
+ const types = {};
+
+ const dirContent = readdirSync(dir);
+ dirContent.map(async (entry) => {
+ const entryPath = path.join(dir, entry);
+ const entryStat = statSync(entryPath);
+
+ if (entryStat.isFile()) {
+ if (filter(entry)) {
+ types[entry.replace(/\.d\.ts$/, '')] = handleFile(entryPath);
+ }
+ } else if (entryStat.isDirectory()) {
+ types[entry] = await traverseDirs(entryPath, filter, handleFile);
+ }
+ });
+ return types;
+};
+
+/**
+ * Blindly extracts exported typenames from a `*.d.ts` file.
+ * @param {string} filepath
+ * @return {string[]}
+ */
+const readTypes = (filepath) => {
+ const input = readFileSync(filepath)
+ .toString();
+ const inputByLine = input.split('\n');
+ const types = [];
+ for (const line of inputByLine) {
+ const matches = line.match(/^export type (\S+)/);
+ console.debugReadTypes(!!matches, line);
+ if (matches) {
+ types.push(matches[1]);
+ }
+ }
+ return types;
+};
+
+/**
+ * Flattens hierarchy of type exports into a mapping from relative paths to exported types.
+ * @param {object<string,object|array>} types
+ * @param {string} prefix
+ * @param {object<string,string[]>} output
+ * @return {object<string,string[]>} Dictionary mapping module declaration paths to exported types.
+ */
+const typeHierarchyToExports = (types, prefix = null, output = {}) => {
+ for (const key of Object.keys(types)) {
+ const entry = types[key];
+ if (entry instanceof Array && entry.length > 0) {
+ output[`${prefix}${key}`] = entry;
+ } else if (entry instanceof Object) {
+ typeHierarchyToExports(entry, `${prefix !== null ? prefix : ''}${key}/`, output);
+ }
+ }
+ return output;
+};
+
+const fixTypes = (input, types = {}) => {
+ const output = [...input];
+ const outputCache = {};
+
+ for (const line of output) {
+ outputCache[line] = true;
+ }
+
+ const exports = typeHierarchyToExports(types, './');
+ for (const file of Object.keys(exports)) {
+ const exportedTypes = exports[file];
+ const line = `export { ${exportedTypes.join(', ')} } from '${file}'`;
+ if (outputCache[line] === undefined) {
+ output.push(line);
+ outputCache[line] = true;
+ }
+ }
+ return output;
+};
+
+const processIndexEsm = async () => {
+ const types = await traverseDirs(
+ typesPath,
+ (filename) => {
+ return filename.endsWith('.d.ts');
+ },
+ readTypes,
+ );
+ const input = readFileSync(paths['index.esm'])
+ .toString()
+ .split('\n');
+ console.log('Fixing log module', paths['index.esm']);
+ const fixedLog = fixLog(input);
+ console.log('Fixing type exports', paths['index.esm']);
+ const fixedTypes = fixTypes(fixedLog, types);
+ console.log('Writing', paths['index.esm']);
+ writeFileSync(paths['index.esm'], fixedTypes.join('\n'));
+};
+
+const removeAllNodeReferences = async () => {
+ await traverseDirs(
+ typesPath,
+ (filename) => {
+ return filename.endsWith('.d.ts');
+ },
+ (filepath) => {
+ const input = readFileSync(filepath)
+ .toString()
+ .split('\n');
+ const output = [];
+ let changed = false;
+ for (const line of input) {
+ if (line.match(/^\/\/\/\s*<reference\s*types="node"\s*\/>/)) {
+ console.debugRemoveReferences('removing node types reference from file', filepath);
+ changed = true;
+ } else {
+ output.push(line);
+ }
+ }
+
+ if (changed === true) {
+ console.log('Writing', filepath);
+ writeFileSync(filepath, output.join('\n'));
+ }
+ },
+ );
+};
+
+(async () => {
+ await processIndexEsm();
+ await removeAllNodeReferences();
+})().then();
diff --git a/tsconfig.json b/tsconfig.json
index 5570755b9..3283c410d 100644
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -1,25 +1,32 @@
{
"compilerOptions": {
- "target": "es2015",
+ "target": "es2018",
"module": "commonjs",
"moduleResolution": "node",
"lib": [
- "es2017",
+ "es2018",
"dom"
],
"types": [
- "node"
+ "node",
+ "cheerio"
],
"allowJs": true,
"checkJs": false,
- "noEmit": true,
- "declaration": false,
+ "noEmit": false,
+ "declaration": true,
+ "emitDeclarationOnly": true,
"strict": false,
"noImplicitThis": true,
"alwaysStrict": true,
- "esModuleInterop": true
+ "esModuleInterop": true,
+ "outDir": "types/"
},
"include": [
+ "src/index.esm.js",
+ "src/utils_log.d.ts"
+ ],
+ "exclude": [
"src/index.js"
]
}
diff --git a/types-apify/apify-shared/log.d.ts b/types-apify/apify-shared/log.d.ts
new file mode 100644
index 000000000..193e0cd52
--- /dev/null
+++ b/types-apify/apify-shared/log.d.ts
@@ -0,0 +1,36 @@
+declare module 'apify-log/log' {
+ const LEVELS: {
+ // Turns off logging completely
+ OFF: number,
+ // For unexpected errors in Apify system
+ ERROR: number,
+ // For situations where error is caused by user (e.g. Meteor.Error), i.e. when the error is not
+ // caused by Apify system, avoid the word "ERROR" to simplify searching in log
+ SOFT_FAIL: number,
+ WARNING: number,
+ INFO: number,
+ DEBUG: number,
+ // for performance stats
+ PERF: number
+ };
+
+ // Indicates whether DEBUG messages will be printed or not
+ let isDebugMode: boolean;
+
+ // Sets log level
+ function setLevel(level: number): void;
+
+ function getLevel(): number;
+
+ // Helper functions for common usage
+ function warning(message: string, data?: any): void;
+
+ function info(message: string, data?: any): void;
+
+ function debug(message: string, data?: any): void;
+
+ function error(message: string, data?: any): void;
+
+ function exception(exception: Error, message?: string, data?: any): void;
+}
+export {}
diff --git a/types/actor.d.ts b/types/actor.d.ts
new file mode 100644
index 000000000..b1e58ac9a
--- /dev/null
+++ b/types/actor.d.ts
@@ -0,0 +1,89 @@
+export function getEnv(): ApifyEnv;
+export function main(userFunc: Function): void;
+export function call(actId: string, input?: any, options?: {
+ contentType?: string;
+ token?: string;
+ memoryMbytes?: number;
+ timeoutSecs?: number;
+ build?: string;
+ waitSecs?: string;
+ fetchOutput?: boolean;
+ disableBodyParser?: boolean;
+ webhooks?: any[];
+}): Promise<ActorRun>;
+export function callTask(taskId: string, input?: any, options?: {
+ contentType?: string;
+ token?: string;
+ memoryMbytes?: number;
+ timeoutSecs?: number;
+ build?: string;
+ waitSecs?: string;
+ webhooks?: any[];
+}): Promise<ActorRun>;
+export function metamorph(targetActorId: string, input?: any, options?: {
+ contentType?: string;
+ build?: string;
+}): Promise<void>;
+export function getApifyProxyUrl(options?: {
+ password?: string;
+ groups?: string[];
+ session?: string;
+ country?: string;
+}): string;
+export function addWebhook({ eventTypes, requestUrl, payloadTemplate, idempotencyKey }: {
+ eventTypes: string[];
+ requestUrl: string;
+ payloadTemplate?: string;
+ idempotencyKey?: string;
+}): Promise<any>;
+/**
+ * Parsed representation of the `APIFY_XXX` environmental variables.
+ */
+export type ApifyEnv = {
+ /**
+ * ID of the actor (APIFY_ACTOR_ID)
+ */
+ actorId: string;
+ /**
+ * ID of the actor run (APIFY_ACTOR_RUN_ID)
+ */
+ actorRunId: string;
+ /**
+ * ID of the actor task (APIFY_ACTOR_TASK_ID)
+ */
+ actorTaskId: string;
+ /**
+ * ID of the user who started the actor - note that it might be
+ * different than the owner ofthe actor (APIFY_USER_ID)
+ */
+ userId: string;
+ /**
+ * Authentication token representing privileges given to the actor run,
+ * it can be passed to various Apify APIs (APIFY_TOKEN)
+ */
+ token: string;
+ /**
+ * Date when the actor was started (APIFY_STARTED_AT)
+ */
+ startedAt: Date;
+ /**
+ * Date when the actor will time out (APIFY_TIMEOUT_AT)
+ */
+ timeoutAt: Date;
+ /**
+ * ID of the key-value store where input and output data of this
+ * actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
+ */
+ defaultKeyValueStoreId: string;
+ /**
+ * ID of the dataset where input and output data of this
+ * actor is stored (APIFY_DEFAULT_DATASET_ID)
+ */
+ defaultDatasetId: string;
+ /**
+ * Amount of memory allocated for the actor,
+ * in megabytes (APIFY_MEMORY_MBYTES)
+ */
+ memoryMbytes: number;
+};
+import { ActorRun } from "./typedefs";
diff --git a/types/autoscaling/autoscaled_pool.d.ts b/types/autoscaling/autoscaled_pool.d.ts
new file mode 100644
index 000000000..d565ca4ca
--- /dev/null
+++ b/types/autoscaling/autoscaled_pool.d.ts
@@ -0,0 +1,365 @@
+export default AutoscaledPool;
+export type AutoscaledPoolOptions = {
+ /**
+ * A function that performs an asynchronous resource-intensive task.
+ * The function must either be labeled `async` or return a promise.
+ */
+ runTaskFunction: Function;
+ /**
+ * A function that indicates whether `runTaskFunction` should be called.
+ * This function is called every time there is free capacity for a new task and it should
+ * indicate whether it should start a new task or not by resolving to either `true` or `false`.
+ * Besides its obvious use, it is also useful for task throttling to save resources.
+ */
+ isTaskReadyFunction: Function;
+ /**
+ * A function that is called only when there are no tasks to be processed.
+ * If it resolves to `true` then the pool's run finishes. Being called only
+ * when there are no tasks being processed means that as long as `isTaskReadyFunction()`
+ * keeps resolving to `true`, `isFinishedFunction()` will never be called.
+ * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.
+ */
+ isFinishedFunction: Function;
+ /**
+ * The minimum number of tasks running in parallel.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ */
+ minConcurrency?: number;
+ /**
+ * The maximum number of tasks running in parallel.
+ */
+ maxConcurrency?: number;
+ /**
+ * The desired number of tasks that should be running parallel on the start of the pool,
+ * if there is a large enough supply of them.
+ * By default, it is `minConcurrency`.
+ */
+ desiredConcurrency?: number;
+ /**
+ * Minimum level of desired concurrency to reach before more scaling up is allowed.
+ */
+ desiredConcurrencyRatio?: number;
+ /**
+ * Defines the fractional amount of desired concurrency to be added with each scaling up.
+ * The minimum scaling step is one.
+ */
+ scaleUpStepRatio?: number;
+ /**
+ * Defines the amount of desired concurrency to be subtracted with each scaling down.
+ * The minimum scaling step is one.
+ */
+ scaleDownStepRatio?: number;
+ /**
+ * Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds.
+ * This has no effect on starting new tasks immediately after a task completes.
+ */
+ maybeRunIntervalSecs?: number;
+ /**
+ * Specifies a period in which the instance logs its state, in seconds.
+ * Set to `null` to disable periodic logging.
+ */
+ loggingIntervalSecs?: number;
+ /**
+ * Defines in seconds how often the pool should attempt to adjust the desired concurrency
+ * based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
+ * We suggest using a value from 5 to 20.
+ */
+ autoscaleIntervalSecs?: number;
+ /**
+ * Options to be passed down to the {@link Snapshotter} constructor. This is useful for fine-tuning
+ * the snapshot intervals and history.
+ */
+ snapshotterOptions?: SnapshotterOptions;
+ /**
+ * Options to be passed down to the {@link SystemStatus} constructor. This is useful for fine-tuning
+ * the system status reports. If a custom snapshotter is set in the options, it will be used
+ * by the pool.
+ */
+ systemStatusOptions?: SystemStatusOptions;
+};
+/**
+ * @typedef {Object} AutoscaledPoolOptions
+ * @property {Function} runTaskFunction
+ * A function that performs an asynchronous resource-intensive task.
+ * The function must either be labeled `async` or return a promise.
+ *
+ * @property {Function} isTaskReadyFunction
+ * A function that indicates whether `runTaskFunction` should be called.
+ * This function is called every time there is free capacity for a new task and it should
+ * indicate whether it should start a new task or not by resolving to either `true` or `false`.
+ * Besides its obvious use, it is also useful for task throttling to save resources.
+ *
+ * @property {Function} isFinishedFunction
+ * A function that is called only when there are no tasks to be processed.
+ * If it resolves to `true` then the pool's run finishes. Being called only
+ * when there are no tasks being processed means that as long as `isTaskReadyFunction()`
+ * keeps resolving to `true`, `isFinishedFunction()` will never be called.
+ * To abort a run, use the [`abort()`](#AutoscaledPool+abort) method.
+ *
+ * @property {Number} [minConcurrency=1]
+ * The minimum number of tasks running in parallel.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {Number} [maxConcurrency=1000]
+ * The maximum number of tasks running in parallel.
+ * @property {Number} [desiredConcurrency]
+ * The desired number of tasks that should be running parallel on the start of the pool,
+ * if there is a large enough supply of them.
+ * By default, it is `minConcurrency`.
+ * @property {Number} [desiredConcurrencyRatio=0.95]
+ * Minimum level of desired concurrency to reach before more scaling up is allowed.
+ * @property {Number} [scaleUpStepRatio=0.05]
+ * Defines the fractional amount of desired concurrency to be added with each scaling up.
+ * The minimum scaling step is one.
+ * @property {Number} [scaleDownStepRatio=0.05]
+ * Defines the amount of desired concurrency to be subtracted with each scaling down.
+ * The minimum scaling step is one.
+ * @property {Number} [maybeRunIntervalSecs=0.5]
+ * Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds.
+ * This has no effect on starting new tasks immediately after a task completes.
+ * @property {Number} [loggingIntervalSecs=60]
+ * Specifies a period in which the instance logs its state, in seconds.
+ * Set to `null` to disable periodic logging.
+ * @property {Number} [autoscaleIntervalSecs=10]
+ * Defines in seconds how often the pool should attempt to adjust the desired concurrency
+ * based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
+ * We suggest using a value from 5 to 20.
+ * @property {SnapshotterOptions} [snapshotterOptions]
+ * Options to be passed down to the {@link Snapshotter} constructor. This is useful for fine-tuning
+ * the snapshot intervals and history.
+ * @property {SystemStatusOptions} [systemStatusOptions]
+ * Options to be passed down to the {@link SystemStatus} constructor. This is useful for fine-tuning
+ * the system status reports. If a custom snapshotter is set in the options, it will be used
+ * by the pool.
+ */
+/**
+ * Manages a pool of asynchronous resource-intensive tasks that are executed in parallel.
+ * The pool only starts new tasks if there is enough free CPU and memory available
+ * and the Javascript event loop is not blocked.
+ *
+ * The information about the CPU and memory usage is obtained by the {@link Snapshotter} class,
+ * which makes regular snapshots of system resources that may be either local
+ * or from the Apify cloud infrastructure in case the process is running on the Apify platform.
+ * Meaningful data gathered from these snapshots is provided to `AutoscaledPool` by the {@link SystemStatus} class.
+ *
+ * Before running the pool, you need to implement the following three functions:
+ * [`runTaskFunction()`](#new_AutoscaledPool_new),
+ * [`isTaskReadyFunction()`](#new_AutoscaledPool_new) and
+ * [`isFinishedFunction()`](#new_AutoscaledPool_new).
+ *
+ * The auto-scaled pool is started by calling the [`run()`](#AutoscaledPool+run) function.
+ * The pool periodically queries the [`isTaskReadyFunction()`](#new_AutoscaledPool_new) function
+ * for more tasks, managing optimal concurrency, until the function resolves to `false`. The pool then queries
+ * the [`isFinishedFunction()`](#new_AutoscaledPool_new). If it resolves to `true`, the run finishes after all running tasks complete.
+ * If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for tasks.
+ * If any of the tasks throws then the [`run()`](#AutoscaledPool+run) function rejects the promise with an error.
+ *
+ * The pool evaluates whether it should start a new task every time one of the tasks finishes
+ * and also in the interval set by the `options.maybeRunIntervalSecs` parameter.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const pool = new Apify.AutoscaledPool({
+ * maxConcurrency: 50,
+ * runTaskFunction: async () => {
+ * // Run some resource-intensive asynchronous operation here.
+ * },
+ * isTaskReadyFunction: async () => {
+ * // Tell the pool whether more tasks are ready to be processed.
+ * // Return true or false
+ * },
+ * isFinishedFunction: async () => {
+ * // Tell the pool whether it should finish
+ * // or wait for more tasks to become available.
+ * // Return true or false
+ * }
+ * });
+ *
+ * await pool.run();
+ * ```
+ */
+declare class AutoscaledPool {
+ /**
+ * @param {AutoscaledPoolOptions} options All `AutoscaledPool` configuration options.
+ */
+ constructor(options?: AutoscaledPoolOptions);
+ desiredConcurrencyRatio: any;
+ scaleUpStepRatio: any;
+ scaleDownStepRatio: any;
+ maybeRunIntervalMillis: number;
+ loggingIntervalMillis: number;
+ autoscaleIntervalMillis: number;
+ runTaskFunction: any;
+ isFinishedFunction: any;
+ isTaskReadyFunction: any;
+ _minConcurrency: any;
+ _maxConcurrency: any;
+ _desiredConcurrency: any;
+ _currentConcurrency: number;
+ isStopped: boolean;
+ lastLoggingTime: number;
+ resolve: (value?: any) => void;
+ reject: (reason?: any) => void;
+ /**
+ * Gets called every autoScaleIntervalSecs and evaluates the current system status.
+ * If the system IS NOT overloaded and the settings allow it, it scales up.
+ * If the system IS overloaded and the settings allow it, it scales down.
+ *
+ * @ignore
+ */
+ _autoscale(intervalCallback: any): any;
+ /**
+ * Starts a new task
+ * if the number of running tasks (current concurrency) is lower than desired concurrency
+ * and the system is not currently overloaded
+ * and this.isTaskReadyFunction() returns true.
+ *
+ * It doesn't allow multiple concurrent runs of this method.
+ *
+ * @ignore
+ */
+ _maybeRunTask(intervalCallback: any): Promise<void>;
+ snapshotter: any;
+ systemStatus: SystemStatus;
+ /**
+ * @ignore
+ */
+ setMaxConcurrency(maxConcurrency: any): void;
+ /**
+ * @ignore
+ */
+ setMinConcurrency(minConcurrency: any): void;
+ /**
+ * Sets the minimum number of tasks running in parallel.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ *
+ * @param {number} value
+ */
+ set minConcurrency(arg: number);
+ /**
+ * Gets the minimum number of tasks running in parallel.
+ *
+ * @return {number}
+ */
+ get minConcurrency(): number;
+ /**
+ * Sets the maximum number of tasks running in parallel.
+ *
+ * @param {number} value
+ */
+ set maxConcurrency(arg: number);
+ /**
+ * Gets the maximum number of tasks running in parallel.
+ *
+ * @return {number}
+ */
+ get maxConcurrency(): number;
+ /**
+ * Sets the desired concurrency for the pool, i.e. the number of tasks that should be running
+ * in parallel if there's large enough supply of tasks.
+ *
+ * @param {number} value
+ */
+ set desiredConcurrency(arg: number);
+ /**
+ * Gets the desired concurrency for the pool,
+ * which is an estimated number of parallel tasks that the system can currently support.
+ *
+ * @return {number}
+ */
+ get desiredConcurrency(): number;
+ /**
+ * Gets the the number of parallel tasks currently running in the pool.
+ *
+ * @return {number}
+ */
+ get currentConcurrency(): number;
+ /**
+ * Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once
+ * all the tasks are finished or one of them fails.
+ *
+ * @return {Promise}
+ */
+ run(): Promise<any>;
+ poolPromise: Promise<any>;
+ autoscaleInterval: any;
+ maybeRunInterval: any;
+ /**
+ * Aborts the run of the auto-scaled pool and destroys it. The promise returned from
+ * the [`run()`](#AutoscaledPool+run) function will immediately resolve, no more new tasks
+ * will be spawned and all running tasks will be left in their current state.
+ *
+ * Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion
+ * of all the running tasks, therefore, no abortion is attempted and some of the tasks
+ * may finish, while others may not. Essentially, auto-scaled pool doesn't care about
+ * their state after the invocation of `.abort()`, but that does not mean that some
+ * parts of their asynchronous chains of commands will not execute.
+ *
+ * @return {Promise}
+ */
+ abort(): Promise<any>;
+ /**
+ * Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish
+ * (unlike abort, which terminates them). Used together with [`resume()`](#AutoscaledPool+resume)
+ *
+ * The function's promise will resolve once all running tasks have completed and the pool
+ * is effectively idle. If the `timeoutSecs` argument is provided, the promise will reject
+ * with a timeout error after the `timeoutSecs` seconds.
+ *
+ * The promise returned from the [`run()`](#AutoscaledPool+run) function will not resolve
+ * when `.pause()` is invoked (unlike abort, which resolves it).
+ *
+ * @param {number} [timeoutSecs]
+ * @return {Promise}
+ */
+ pause(timeoutSecs?: number): Promise<any>;
+ /**
+ * Resumes the operation of the autoscaled-pool by allowing more tasks to be run.
+ * Used together with [`pause()`](#AutoscaledPool+pause)
+ *
+ * Tasks will automatically start running again in `options.maybeRunIntervalSecs`.
+ */
+ resume(): void;
+ queryingIsTaskReady: boolean;
+ /**
+ * Scales the pool up by increasing
+ * the desired concurrency by the scaleUpStepRatio.
+ *
+ * @param {Object} systemStatus for logging
+ * @ignore
+ */
+ _scaleUp(systemStatus: any): void;
+ /**
+ * Scales the pool down by decreasing
+ * the desired concurrency by the scaleDownStepRatio.
+ *
+ * @param {Object} systemStatus for logging
+ * @ignore
+ */
+ _scaleDown(systemStatus: any): void;
+ /**
+ * If there are no running tasks and this.isFinishedFunction() returns true then closes
+ * the pool and resolves the pool's promise returned by the run() method.
+ *
+ * It doesn't allow multiple concurrent runs of this method.
+ *
+ * @ignore
+ */
+ _maybeFinish(): Promise<void>;
+ queryingIsFinished: boolean;
+ /**
+ * Cleans up resources.
+ *
+ * @ignore
+ */
+ _destroy(): Promise<void>;
+}
+import { SnapshotterOptions } from "./snapshotter";
+import { SystemStatusOptions } from "./system_status";
+import SystemStatus from "./system_status";
diff --git a/types/autoscaling/snapshotter.d.ts b/types/autoscaling/snapshotter.d.ts
new file mode 100644
index 000000000..80b08859b
--- /dev/null
+++ b/types/autoscaling/snapshotter.d.ts
@@ -0,0 +1,249 @@
+export default Snapshotter;
+export type SnapshotterOptions = {
+ /**
+ * Defines the interval of measuring the event loop response time.
+ */
+ eventLoopSnapshotIntervalSecs?: number;
+ /**
+ * Defines the interval of checking the current state
+ * of the remote API client.
+ */
+ clientSnapshotIntervalSecs?: number;
+ /**
+ * Maximum allowed delay of the event loop in milliseconds.
+ * Exceeding this limit overloads the event loop.
+ */
+ maxBlockedMillis?: number;
+ /**
+ * Defines the interval of measuring CPU usage.
+ * This is only used when running locally. On the Apify platform,
+ * the statistics are provided externally at a fixed interval.
+ */
+ cpuSnapshotIntervalSecs?: number;
+ /**
+ * Defines the maximum usage of CPU.
+ * Exceeding this limit overloads the CPU.
+ */
+ maxUsedCpuRatio?: number;
+ /**
+ * Defines the interval of measuring memory consumption.
+ * This is only used when running locally. On the Apify platform,
+ * the statistics are provided externally at a fixed interval.
+ * The measurement itself is resource intensive (25 - 50ms async).
+ * Therefore, setting this interval below 1 second is not recommended.
+ */
+ memorySnapshotIntervalSecs?: number;
+ /**
+ * Defines the maximum ratio of total memory that can be used.
+ * Exceeding this limit overloads the memory.
+ */
+ maxUsedMemoryRatio?: number;
+ /**
+ * Defines the maximum number of new rate limit errors within
+ * the given interval.
+ */
+ maxClientErrors?: number;
+ /**
+ * Sets the interval in seconds for which a history of resource snapshots
+ * will be kept. Increasing this to very high numbers will affect performance.
+ */
+ snapshotHistorySecs?: number;
+};
+/**
+ * @typedef SnapshotterOptions
+ * @property {Number} [eventLoopSnapshotIntervalSecs=0.5]
+ * Defines the interval of measuring the event loop response time.
+ * @property {Number} [clientSnapshotIntervalSecs=1]
+ * Defines the interval of checking the current state
+ * of the remote API client.
+ * @property {Number} [maxBlockedMillis=50]
+ * Maximum allowed delay of the event loop in milliseconds.
+ * Exceeding this limit overloads the event loop.
+ * @property {Number} [cpuSnapshotIntervalSecs=1]
+ * Defines the interval of measuring CPU usage.
+ * This is only used when running locally. On the Apify platform,
+ * the statistics are provided externally at a fixed interval.
+ * @property {Number} [maxUsedCpuRatio=0.95]
+ * Defines the maximum usage of CPU.
+ * Exceeding this limit overloads the CPU.
+ * @property {Number} [memorySnapshotIntervalSecs=1]
+ * Defines the interval of measuring memory consumption.
+ * This is only used when running locally. On the Apify platform,
+ * the statistics are provided externally at a fixed interval.
+ * The measurement itself is resource intensive (25 - 50ms async).
+ * Therefore, setting this interval below 1 second is not recommended.
+ * @property {Number} [maxUsedMemoryRatio=0.7]
+ * Defines the maximum ratio of total memory that can be used.
+ * Exceeding this limit overloads the memory.
+ * @property {Number} [maxClientErrors=1]
+ * Defines the maximum number of new rate limit errors within
+ * the given interval.
+ * @property {Number} [snapshotHistorySecs=60]
+ * Sets the interval in seconds for which a history of resource snapshots
+ * will be kept. Increasing this to very high numbers will affect performance.
+ */
+/**
+ * Creates snapshots of system resources at given intervals and marks the resource
+ * as either overloaded or not during the last interval. Keeps a history of the snapshots.
+ * It tracks the following resources: Memory, EventLoop, API and CPU.
+ * The class is used by the {@link AutoscaledPool} class.
+ *
+ * When running on the Apify platform, the CPU and memory statistics are provided by the platform,
+ * as collected from the running Docker container. When running locally, `Snapshotter`
+ * makes its own statistics by querying the OS.
+ *
+ * CPU becomes overloaded locally when its current use exceeds the `maxUsedCpuRatio` option or
+ * when Apify platform marks it as overloaded.
+ *
+ * Memory becomes overloaded if its current use exceeds the `maxUsedMemoryRatio` option.
+ * It's computed using the total memory available to the container when running on
+ * the Apify platform and a quarter of total system memory when running locally.
+ * Max total memory when running locally may be overridden by using the `APIFY_MEMORY_MBYTES`
+ * environment variable.
+ *
+ * Event loop becomes overloaded if it slows down by more than the `maxBlockedMillis` option.
+ *
+ * Client becomes overloaded when rate limit errors (429 - Too Many Requests),
+ * typically received from the request queue, exceed the set limit within the set interval.
+ */
+declare class Snapshotter {
+ /**
+ * @param {SnapshotterOptions} [options] All `Snapshotter` configuration options.
+ */
+ constructor(options?: SnapshotterOptions);
+ eventLoopSnapshotIntervalMillis: number;
+ memorySnapshotIntervalMillis: number;
+ clientSnapshotIntervalMillis: number;
+ cpuSnapshotIntervalMillis: number;
+ snapshotHistoryMillis: number;
+ maxBlockedMillis: any;
+ maxUsedMemoryRatio: any;
+ maxUsedCpuRatio: any;
+ maxClientErrors: any;
+ maxMemoryBytes: number;
+ cpuSnapshots: any[];
+ eventLoopSnapshots: any[];
+ memorySnapshots: any[];
+ clientSnapshots: any[];
+ eventLoopInterval: any;
+ memoryInterval: any;
+ clientInterval: any;
+ cpuInterval: any;
+ lastLoggedCriticalMemoryOverloadAt: Date;
+ /**
+ * Creates a snapshot of current CPU usage
+ * using the Apify platform `systemInfo` event.
+ * @param {Object} systemInfo
+ * @ignore
+ */
+ _snapshotCpuOnPlatform(systemInfo: any): void;
+ /**
+ * Creates a snapshot of current memory usage
+ * using the Apify platform `systemInfo` event.
+ * @param {Object} systemInfo
+ * @ignore
+ */
+ _snapshotMemoryOnPlatform(systemInfo: any): void;
+ /**
+ * Starts capturing snapshots at configured intervals.
+ * @return {Promise}
+ */
+ start(): Promise<any>;
+ /**
+ * Stops all resource capturing.
+ * @return {Promise}
+ */
+ stop(): Promise<any>;
+ /**
+ * Returns a sample of latest memory snapshots, with the size of the sample defined
+ * by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
+ * @param {Number} [sampleDurationMillis]
+ * @return {Array}
+ */
+ getMemorySample(sampleDurationMillis?: number): any[];
+ /**
+ * Returns a sample of latest event loop snapshots, with the size of the sample defined
+ * by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
+ * @param {Number} [sampleDurationMillis]
+ * @return {Array}
+ */
+ getEventLoopSample(sampleDurationMillis?: number): any[];
+ /**
+ * Returns a sample of latest CPU snapshots, with the size of the sample defined
+ * by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
+ * @param {Number} [sampleDurationMillis]
+ * @return {Array}
+ */
+ getCpuSample(sampleDurationMillis?: number): any[];
+ /**
+ * Returns a sample of latest Client snapshots, with the size of the sample defined
+ * by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
+ * @param {Number} sampleDurationMillis
+ * @return {Array}
+ */
+ getClientSample(sampleDurationMillis: number): any[];
+ /**
+ * Finds the latest snapshots by sampleDurationMillis in the provided array.
+ * @param {Array} snapshots
+ * @param {Number} [sampleDurationMillis]
+ * @return {Array}
+ * @ignore
+ */
+ _getSample(snapshots: any[], sampleDurationMillis?: number): any[];
+ /**
+ * Creates a snapshot of current memory usage
+ * using the Apify platform `systemInfo` event.
+ * @param {Function} intervalCallback
+ * @return {Promise}
+ * @ignore
+ */
+ _snapshotMemoryOnLocal(intervalCallback: Function): Promise<any>;
+ /**
+ * Checks for critical memory overload and logs it to the console.
+ * @ignore
+ * @param {Object} systemInfo
+ */
+ _memoryOverloadWarning({ memCurrentBytes }: any): void;
+ /**
+ * Creates a snapshot of current event loop delay.
+ * @param {Function} intervalCallback
+ * @ignore
+ */
+ _snapshotEventLoop(intervalCallback: Function): void;
+ /**
+ * Creates a snapshot of current CPU usage
+ * using OS provided metrics.
+ * @param {Function} intervalCallback
+ * @ignore
+ */
+ _snapshotCpuOnLocal(intervalCallback: Function): void;
+ _getCurrentCpuTicks(): {
+ idle: number;
+ total: number;
+ };
+ /**
+ * Creates a snapshot of current API state by checking for
+ * rate limit errors. Only errors produced by a 2nd retry
+ * of the API call are considered for snapshotting since
+ * earlier errors may just be caused by a random spike in
+ * number of requests and do not necessarily signify API
+ * overloading.
+ *
+ * @param intervalCallback
+ * @private
+ */
+ _snapshotClient(intervalCallback: any): void;
+ /**
+ * Removes snapshots that are older than the snapshotHistorySecs option
+ * from the array (destructively - in place).
+ * @param {Array} snapshots
+ * @param {Date} now
+ * @ignore
+ */
+ _pruneSnapshots(snapshots: any[], now: Date): void;
+ /**
+ * Calculate max memory for platform or local usage.
+ * @ignore
+ */
+ _ensureCorrectMaxMemory(): Promise<void>;
+}
diff --git a/types/autoscaling/system_status.d.ts b/types/autoscaling/system_status.d.ts
new file mode 100644
index 000000000..1782e0c58
--- /dev/null
+++ b/types/autoscaling/system_status.d.ts
@@ -0,0 +1,205 @@
+export default SystemStatus;
+/**
+ * Represents the current status of the system.
+ */
+export type SystemInfo = {
+ /**
+ * If true, system is being overloaded.
+ */
+ isSystemIdle: boolean;
+ /**
+ * Memory
+ */
+ memInfo: any;
+ eventLoopInfo: any;
+ cpuInfo: any;
+};
+export type SystemStatusOptions = {
+ /**
+ * Defines max age of snapshots used in the
+ * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.
+ */
+ currentHistorySecs?: number;
+ /**
+ * Sets the maximum ratio of overloaded snapshots in a memory sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ */
+ maxMemoryOverloadedRatio?: number;
+ /**
+ * Sets the maximum ratio of overloaded snapshots in an event loop sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ */
+ maxEventLoopOverloadedRatio?: number;
+ /**
+ * Sets the maximum ratio of overloaded snapshots in a CPU sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ */
+ maxCpuOverloadedRatio?: number;
+ /**
+ * Sets the maximum ratio of overloaded snapshots in a Client sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ */
+ maxClientOverloadedRatio?: number;
+ /**
+ * The `Snapshotter` instance to be queried for `SystemStatus`.
+ */
+ snapshotter?: Snapshotter;
+};
+/**
+ * Represents the current status of the system.
+ *
+ * @typedef {Object} SystemInfo
+ * @property {Boolean} isSystemIdle
+ * If true, system is being overloaded.
+ * @property {Object} memInfo
+ * Memory
+ * @property {Object} eventLoopInfo
+ * @property {Object} cpuInfo
+ */
+/**
+ * @typedef {Object} SystemStatusOptions
+ * @property {Number} [currentHistorySecs=5]
+ * Defines max age of snapshots used in the
+ * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus) measurement.
+ * @property {Number} [maxMemoryOverloadedRatio=0.2]
+ * Sets the maximum ratio of overloaded snapshots in a memory sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Number} [maxEventLoopOverloadedRatio=0.2]
+ * Sets the maximum ratio of overloaded snapshots in an event loop sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Number} [maxCpuOverloadedRatio=0.4]
+ * Sets the maximum ratio of overloaded snapshots in a CPU sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Number} [maxClientOverloadedRatio=0.2]
+ * Sets the maximum ratio of overloaded snapshots in a Client sample.
+ * If the sample exceeds this ratio, the system will be overloaded.
+ * @property {Snapshotter} [snapshotter]
+ * The `Snapshotter` instance to be queried for `SystemStatus`.
+ */
+/**
+ * Provides a simple interface to reading system status from a {@link Snapshotter} instance.
+ * It only exposes two functions [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)
+ * and [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus).
+ * The system status is calculated using a weighted average of overloaded
+ * messages in the snapshots, with the weights being the time intervals
+ * between the snapshots. Each resource is calculated separately
+ * and the system is overloaded whenever at least one resource is overloaded.
+ * The class is used by the {@link AutoscaledPool} class.
+ *
+ * [`getCurrentStatus()`](#SystemStatus+getCurrentStatus)
+ * returns a boolean that represents the current status of the system.
+ * The length of the current timeframe in seconds is configurable
+ * by the `currentHistorySecs` option and represents the max age
+ * of snapshots to be considered for the calculation.
+ *
+ * [`getHistoricalStatus()`](#SystemStatus+getHistoricalStatus)
+ * returns a boolean that represents the long-term status
+ * of the system. It considers the full snapshot history available
+ * in the {@link Snapshotter} instance.
+ */
+declare class SystemStatus {
+ /**
+ * @param {SystemStatusOptions} [options] All `SystemStatus` configuration options.
+ */
+ constructor(options?: SystemStatusOptions);
+ currentHistorySecs: number;
+ maxMemoryOverloadedRatio: any;
+ maxEventLoopOverloadedRatio: any;
+ maxCpuOverloadedRatio: any;
+ maxClientOverloadedRatio: any;
+ snapshotter: any;
+ /**
+ * Returns an {@link SystemInfo} object with the following structure:
+ *
+ * ```javascript
+ * {
+ * isSystemIdle: Boolean,
+ * memInfo: Object,
+ * eventLoopInfo: Object,
+ * cpuInfo: Object
+ * }
+ * ```
+ *
+ * Where the `isSystemIdle` property is set to `false` if the system
+ * has been overloaded in the last `options.currentHistorySecs` seconds,
+ * and `true` otherwise.
+ * @return {SystemInfo}
+ */
+ getCurrentStatus(): SystemInfo;
+ /**
+ * Returns an {@link SystemInfo} object with the following structure:
+ *
+ * ```javascript
+ * {
+ * isSystemIdle: Boolean,
+ * memInfo: Object,
+ * eventLoopInfo: Object,
+ * cpuInfo: Object
+ * }
+ * ```
+ *
+ * Where the `isSystemIdle` property is set to `false` if the system
+ * has been overloaded in the full history of the {@link Snapshotter}
+ * (which is configurable in the {@link Snapshotter}) and `true` otherwise.
+ * @return {Object}
+ */
+ getHistoricalStatus(): any;
+ /**
+ * Returns a system status object.
+ *
+ * @param {Number} [sampleDurationMillis]
+ * @return {SystemInfo}
+ * @ignore
+ */
+ _isSystemIdle(sampleDurationMillis?: number): SystemInfo;
+ /**
+ * Returns an object with an isOverloaded property set to true
+ * if the memory has been overloaded in the last sampleDurationMillis.
+ *
+ * @param {Number} sampleDurationMillis
+ * @return {Object}
+ * @ignore
+ */
+ _isMemoryOverloaded(sampleDurationMillis: number): any;
+ /**
+ * Returns an object with an isOverloaded property set to true
+ * if the event loop has been overloaded in the last sampleDurationMillis.
+ *
+ * @param {Number} sampleDurationMillis
+ * @return {Object}
+ * @ignore
+ */
+ _isEventLoopOverloaded(sampleDurationMillis: number): any;
+ /**
+ * Returns an object with an isOverloaded property set to true
+ * if the CPU has been overloaded in the last sampleDurationMillis.
+ *
+ * @param {Number} sampleDurationMillis
+ * @return {Object}
+ * @ignore
+ */
+ _isCpuOverloaded(sampleDurationMillis: number): any;
+ /**
+ * Returns an object with an isOverloaded property set to true
+ * if the client has been overloaded in the last sampleDurationMillis.
+ * @param sampleDurationMillis
+ * @return {{isOverloaded, maxOverloadedRatio, actualRatio}}
+ * @private
+ */
+ _isClientOverloaded(sampleDurationMillis: any): {
+ isOverloaded: any;
+ maxOverloadedRatio: any;
+ actualRatio: any;
+ };
+ /**
+ * Returns an object with sample information and an isOverloaded property
+ * set to true if at least the ratio of snapshots in the sample are overloaded.
+ *
+ * @param {Array} sample
+ * @param {Number} ratio
+ * @return {Object}
+ * @ignore
+ */
+ _isSampleOverloaded(sample: any[], ratio: number): any;
+}
+import Snapshotter from "./snapshotter";
diff --git a/types/constants.d.ts b/types/constants.d.ts
new file mode 100644
index 000000000..23c4e1255
--- /dev/null
+++ b/types/constants.d.ts
@@ -0,0 +1,42 @@
+/**
+ * The default user agent used by `Apify.launchPuppeteer`.
+ * Last updated on 2018-12-30.
+ */
+export const DEFAULT_USER_AGENT: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36";
+export namespace EXIT_CODES {
+ export const SUCCESS: number;
+ export const ERROR_USER_FUNCTION_THREW: number;
+ export const ERROR_UNKNOWN: number;
+}
+/**
+ * These events are just internal for Apify package, so we don't need them in apify-shared package.
+ *
+ * @type {{CPU_INFO: String, SYSTEM_INFO: String, MIGRATING: String, PERSIST_STATE: String}}
+ *
+ */
+export const ACTOR_EVENT_NAMES_EX: {
+ CPU_INFO: String;
+ SYSTEM_INFO: String;
+ MIGRATING: String;
+ PERSIST_STATE: String;
+};
+/**
+ * Most common user agents from https://techblog.willshouse.com/2012/01/03/most-common-user-agents/
+ *
+ * Last updated on 2019-02-18.
+ */
+export const USER_AGENT_LIST: string[];
+/**
+ * Base URL of Apify's API endpoints.
+ * @type {string}
+ */
+export const APIFY_API_BASE_URL: string;
+/**
+ * Multiplier used in CheerioCrawler and PuppeteerCrawler to set a reasonable
+ * handleRequestTimeoutSecs in BasicCrawler that would not impare functionality.
+ *
+ * @type {number}
+ */
+export const BASIC_CRAWLER_TIMEOUT_MULTIPLIER: number;
+export const COUNTRY_CODE_REGEX: RegExp;
+export const STATUS_CODES_BLOCKED: number[];
diff --git a/types/crawlers/basic_crawler.d.ts b/types/crawlers/basic_crawler.d.ts
new file mode 100644
index 000000000..d27191baa
--- /dev/null
+++ b/types/crawlers/basic_crawler.d.ts
@@ -0,0 +1,327 @@
+export default BasicCrawler;
+export type BasicCrawlerOptions = {
+ /**
+ * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * autoscaledPool: AutoscaledPool
+ * }
+ * ```
+ * where the {@link Request} instance represents the URL to crawl.
+ *
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ */
+ handleRequestFunction: HandleRequest;
+ /**
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ */
+ requestList?: RequestList;
+ /**
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ */
+ requestQueue?: RequestQueue;
+ /**
+ * Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.
+ */
+ handleRequestTimeoutSecs?: number;
+ /**
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
+ * for the default implementation of this function.
+ */
+ handleFailedRequestFunction?: HandleFailedRequest;
+ /**
+ * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.
+ */
+ maxRequestRetries?: number;
+ /**
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ */
+ maxRequestsPerCrawl?: number;
+ /**
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
+ */
+ autoscaledPoolOptions?: AutoscaledPoolOptions;
+ /**
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ */
+ minConcurrency?: number;
+ /**
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ */
+ maxConcurrency?: number;
+ /**
+ * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
+ * The session instance will be than available in the `handleRequestFunction`.
+ */
+ useSessionPool?: boolean;
+ /**
+ * The configuration options for {SessionPool} to use.
+ */
+ sessionPoolOptions?: SessionPoolOptions;
+};
+export type HandleRequest = (inputs: HandleRequestInputs) => Promise<void>;
+export type HandleRequestInputs = {
+ /**
+ * The original {Request} object.
+ */
+ request: Request;
+ autoscaledPool: AutoscaledPool;
+ session?: Session;
+};
+export type HandleFailedRequest = (inputs: HandleFailedRequestInput) => void | Promise<void>;
+export type HandleFailedRequestInput = {
+ /**
+ * The original {Request} object.
+ */
+ request: Request;
+ /**
+ * The Error thrown by `handleRequestFunction`.
+ */
+ error: Error;
+};
+/**
+ * @typedef {Object} BasicCrawlerOptions
+ * @property {HandleRequest} handleRequestFunction
+ * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * autoscaledPool: AutoscaledPool
+ * }
+ * ```
+ * where the {@link Request} instance represents the URL to crawl.
+ *
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {number} [handleRequestTimeoutSecs=60]
+ * Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>
+ * for the default implementation of this function.
+ * @property {Number} [maxRequestRetries=3]
+ * Indicates how many times the request is retried if [`handleRequestFunction()`](#new_BasicCrawler_new) fails.
+ * @property {Number} [maxRequestsPerCrawl]
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
+ * @property {Number} [minConcurrency=1]
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {Number} [maxConcurrency=1000]
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @property {Boolean} [useSessionPool=false]
+ * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
+ * The session instance will be than available in the `handleRequestFunction`.
+ * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {SessionPool} to use.
+ */
+/**
+ * Provides a simple framework for parallel crawling of web pages.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * `BasicCrawler` is a low-level tool that requires the user to implement the page
+ * download and data extraction functionality themselves.
+ * If you want a crawler that already facilitates this functionality,
+ * please consider using {@link PuppeteerCrawler} or {@link CheerioCrawler}.
+ *
+ * `BasicCrawler` invokes the user-provided [`handleRequestFunction()`](#new_BasicCrawler_new)
+ * for each {@link Request} object, which represents a single URL to crawl.
+ * The {@link Request} objects are fed from the {@link RequestList} or the {@link RequestQueue}
+ * instances provided by the [`requestList`](#new_BasicCrawler_new) or [`requestQueue`](#new_BasicCrawler_new)
+ * constructor options, respectively.
+ *
+ * If both [`requestList`](#new_BasicCrawler_new) and [`requestQueue`](#new_BasicCrawler_new) options are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes if there are no more {@link Request} objects to crawl.
+ *
+ * New requests are only dispatched when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * parameter of the `BasicCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPool} options are available directly in the `BasicCrawler` constructor.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const rp = require('request-promise-native');
+ *
+ * // Prepare a list of URLs to crawl
+ * const requestList = new Apify.RequestList({
+ * sources: [
+ * { url: 'http://www.example.com/page-1' },
+ * { url: 'http://www.example.com/page-2' },
+ * ],
+ * });
+ * await requestList.initialize();
+ *
+ * // Crawl the URLs
+ * const crawler = new Apify.BasicCrawler({
+ * requestList,
+ * handleRequestFunction: async ({ request }) => {
+ * // 'request' contains an instance of the Request class
+ * // Here we simply fetch the HTML of the page and store it to a dataset
+ * await Apify.pushData({
+ * url: request.url,
+ * html: await rp(request.url),
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ */
+declare class BasicCrawler {
+ /**
+ * @param {BasicCrawlerOptions} options
+ */
+ constructor(options: BasicCrawlerOptions);
+ requestList: any;
+ requestQueue: any;
+ handleRequestFunction: any;
+ handleRequestTimeoutMillis: number;
+ handleFailedRequestFunction: any;
+ maxRequestRetries: any;
+ handledRequestsCount: number;
+ stats: Statistics;
+ sessionPoolOptions: any;
+ useSessionPool: any;
+ autoscaledPoolOptions: any;
+ isRunningPromise: Promise<any>;
+ /**
+ * Runs the crawler. Returns a promise that gets resolved once all the requests are processed.
+ *
+ * @return {Promise<void>}
+ */
+ run(): Promise<void>;
+ autoscaledPool: AutoscaledPool;
+ sessionPool: import("../session_pool/session_pool").SessionPool;
+ _pauseOnMigration(): Promise<void>;
+ /**
+ * Fetches request from either RequestList or RequestQueue. If request comes from a RequestList
+ * and RequestQueue is present then enqueues it to the queue first.
+ *
+ * @ignore
+ */
+ _fetchNextRequest(): Promise<any>;
+ /**
+ * Wrapper around handleRequestFunction that fetches requests from RequestList/RequestQueue
+ * then retries them in a case of an error, etc.
+ *
+ * @ignore
+ */
+ _runTaskFunction(): Promise<void>;
+ /**
+ * Returns true if either RequestList or RequestQueue have a request ready for processing.
+ *
+ * @ignore
+ */
+ _isTaskReadyFunction(): Promise<boolean>;
+ /**
+ * Returns true if both RequestList and RequestQueue have all requests finished.
+ *
+ * @ignore
+ */
+ _defaultIsFinishedFunction(): Promise<any>;
+ /**
+ * Handles errors thrown by user provided handleRequestFunction()
+ * @param {Error} error
+ * @param {Request} request
+ * @param {RequestList|RequestQueue} source
+ * @return {Promise<Boolean>} willBeRetried
+ * @ignore
+ */
+ _requestFunctionErrorHandler(error: Error, request: Request, source: RequestList | RequestQueue): Promise<boolean>;
+ /**
+ * Updates handledRequestsCount from possibly stored counts,
+ * usually after worker migration. Since one of the stores
+ * needs to have priority when both are present,
+ * it is the request queue, because generally, the request
+ * list will first be dumped into the queue and then left
+ * empty.
+ *
+ * @return {Promise<void>}
+ * @ignore
+ */
+ _loadHandledRequestCount(): Promise<void>;
+}
+import { RequestList } from "../request_list";
+import { RequestQueue } from "../request_queue";
+import { AutoscaledPoolOptions } from "../autoscaling/autoscaled_pool";
+import { SessionPoolOptions } from "../session_pool/session_pool";
+import Request from "../request";
+import AutoscaledPool from "../autoscaling/autoscaled_pool";
+import { Session } from "../session_pool/session";
+import Statistics from "./statistics";
diff --git a/types/crawlers/cheerio_crawler.d.ts b/types/crawlers/cheerio_crawler.d.ts
new file mode 100644
index 000000000..d4f25691d
--- /dev/null
+++ b/types/crawlers/cheerio_crawler.d.ts
@@ -0,0 +1,601 @@
+export default CheerioCrawler;
+export type CheerioCrawlerOptions = {
+ /**
+ * User-provided function that performs the logic of the crawler. It is called for each page
+ * loaded and parsed by the crawler.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * $: Cheerio, // the Cheerio object with parsed HTML
+ * body: String|Buffer // the request body of the web page
+ * // the parsed object from JSON string
+ * // if the response contains the content type application/json
+ * json: Object,
+ * request: Request,
+ * contentType: Object, // Parsed Content-Type header: { type, encoding }
+ * response: Object // An instance of Node's http.IncomingMessage object,
+ * autoscaledPool: AutoscaledPool,
+ * session: Session
+ * }
+ * ```
+ * Type of `body` depends on web page `Content-Type` header.
+ * - String for `text/html`, `application/xhtml+xml`, `application/xml` mime types
+ * - Buffer for others mime types
+ *
+ * Parsed `Content-Type` header using
+ * <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>
+ * is stored in `contentType`.
+ *
+ * Cheerio is available only for HTML and XML content types.
+ *
+ * With the {@link Request} object representing the URL to crawl.
+ *
+ * If the function returns, the returned promise is awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ */
+ handlePageFunction: CheerioHandlePage;
+ /**
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ */
+ requestList?: RequestList;
+ /**
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ */
+ requestQueue?: RequestQueue;
+ /**
+ * Represents the options passed to
+ * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.
+ * Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure
+ * the operation of `CheerioCrawler` and all its options. Headers will not be merged,
+ * use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the
+ * correct headers or use `prepareRequestFunction` to modify your {@link Request} dynamically.
+ * If you need more granular control over your requests, use {@link BasicCrawler}.
+ *
+ * The mandatory internal defaults that **CANNOT BE OVERRIDDEN** by `requestOptions`:
+ * ```
+ * {
+ * url, // Provided by RequestList and/or RequestQueue
+ * method, // Provided by RequestList and/or RequestQueue
+ * headers, // Provided by RequestList and/or RequestQueue
+ * payload, // Provided by RequestList and/or RequestQueue
+ * strictSSL, // Use ignoreSslErrors
+ * proxy, // Use useApifyProxy or proxyUrls
+ * }
+ * ```
+ */
+ requestOptions?: utilsRequest.RequestAsBrowserOptions;
+ /**
+ * A function that executes before the HTTP request is made to the target resource.
+ * This function is suitable for setting dynamic properties such as cookies to the {@link Request}.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * session: Session
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the initialized request
+ * and the {@link Session} instance corresponds to used session.
+ *
+ * The function should modify the properties of the passed {@link Request} instance
+ * in place because there are already earlier references to it. Making a copy and returning it from
+ * this function is therefore not supported, because it would create inconsistencies where
+ * different parts of SDK would have access to a different {@link Request} instance.
+ */
+ prepareRequestFunction?: PrepareRequest;
+ /**
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
+ */
+ handlePageTimeoutSecs?: number;
+ /**
+ * Timeout in which the HTTP request to the resource needs to finish, given in seconds.
+ */
+ requestTimeoutSecs?: number;
+ /**
+ * If set to true, SSL certificate errors will be ignored.
+ */
+ ignoreSslErrors?: boolean;
+ /**
+ * If set to `true`, `CheerioCrawler` will be configured to use
+ * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
+ * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>
+ */
+ useApifyProxy?: boolean;
+ /**
+ * An array of proxy groups to be used
+ * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
+ * Only applied if the `useApifyProxy` option is `true`.
+ */
+ apifyProxyGroups?: string[];
+ /**
+ * Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.
+ * All HTTP requests going through the proxy with the same session identifier
+ * will use the same target proxy server (i.e. the same IP address).
+ * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
+ * Only applied if the `useApifyProxy` option is `true`.
+ */
+ apifyProxySession?: string;
+ /**
+ * An array of custom proxy URLs to be used by the `CheerioCrawler` instance.
+ * The provided custom proxies' order will be randomized and the resulting list rotated.
+ * Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ * configuration options will cause an error to be thrown on startup.
+ */
+ proxyUrls?: string[];
+ /**
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
+ * for the default implementation of this function.
+ */
+ handleFailedRequestFunction?: HandleFailedRequest;
+ /**
+ * An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
+ * target="_blank">mime types</a> you want to process.
+ * By default `text/html`, `application/xhtml+xml` mime types are supported.
+ */
+ additionalMimeTypes?: string[];
+ /**
+ * Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
+ */
+ maxRequestRetries?: number;
+ /**
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ */
+ maxRequestsPerCrawl?: number;
+ /**
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * are provided by `CheerioCrawler` and cannot be overridden. Reasonable {@link Snapshotter}
+ * and {@link SystemStatus} defaults are provided to account for the fact that `cheerio`
+ * parses HTML synchronously and therefore blocks the event loop.
+ */
+ autoscaledPoolOptions?: AutoscaledPoolOptions;
+ /**
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ */
+ minConcurrency?: number;
+ /**
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ */
+ maxConcurrency?: number;
+ /**
+ * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ * It also marks Session as bad after a request timeout.
+ */
+ useSessionPool?: boolean;
+ /**
+ * Custom options passed to the underlying {@link SessionPool} constructor.
+ */
+ sessionPoolOptions?: SessionPoolOptions;
+ /**
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ *
+ * It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.
+ * It passes the "Cookie" header to the request with the session cookies.
+ */
+ persistCookiesPerSession?: boolean;
+};
+export type PrepareRequestInputs = {
+ /**
+ * Original instance fo the {Request} object. Must be modified in-place.
+ */
+ request: Request;
+};
+export type PrepareRequest = (inputs: PrepareRequestInputs) => void | Promise<void>;
+export type CheerioHandlePageInputs = {
+ /**
+ * The <a href="https://cheerio.js.org/">Cheerio</a> object with parsed HTML.
+ */
+ $?: CheerioStatic;
+ /**
+ * The request body of the web page.
+ */
+ body: string | Buffer;
+ /**
+ * The parsed object from JSON string if the response contains the content type application/json.
+ */
+ json?: any;
+ /**
+ * The original {Request} object.
+ */
+ request: Request;
+ /**
+ * Parsed `Content-Type header: { type, encoding }`.
+ */
+ contentType: {
+ type: string;
+ encoding: string;
+ };
+ /**
+ * An instance of Node's http.IncomingMessage object,
+ */
+ response: IncomingMessage;
+ autoscaledPool: AutoscaledPool;
+ session?: any;
+};
+export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<void>;
+/**
+ * @typedef {Object} CheerioCrawlerOptions
+ * @property {CheerioHandlePage} handlePageFunction
+ * User-provided function that performs the logic of the crawler. It is called for each page
+ * loaded and parsed by the crawler.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * $: Cheerio, // the Cheerio object with parsed HTML
+ * body: String|Buffer // the request body of the web page
+ * // the parsed object from JSON string
+ * // if the response contains the content type application/json
+ * json: Object,
+ * request: Request,
+ * contentType: Object, // Parsed Content-Type header: { type, encoding }
+ * response: Object // An instance of Node's http.IncomingMessage object,
+ * autoscaledPool: AutoscaledPool,
+ * session: Session
+ * }
+ * ```
+ * Type of `body` depends on web page `Content-Type` header.
+ * - String for `text/html`, `application/xhtml+xml`, `application/xml` mime types
+ * - Buffer for others mime types
+ *
+ * Parsed `Content-Type` header using
+ * <a href="https://www.npmjs.com/package/content-type" target="_blank">content-type package</a>
+ * is stored in `contentType`.
+ *
+ * Cheerio is available only for HTML and XML content types.
+ *
+ * With the {@link Request} object representing the URL to crawl.
+ *
+ * If the function returns, the returned promise is awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestAsBrowserOptions} [requestOptions]
+ * Represents the options passed to
+ * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.
+ * Provided `requestOptions` are added to internal defaults that cannot be overridden to ensure
+ * the operation of `CheerioCrawler` and all its options. Headers will not be merged,
+ * use {@link RequestList} and/or {@link RequestQueue} to initialize your {@link Request} with the
+ * correct headers or use `prepareRequestFunction` to modify your {@link Request} dynamically.
+ * If you need more granular control over your requests, use {@link BasicCrawler}.
+ *
+ * The mandatory internal defaults that **CANNOT BE OVERRIDDEN** by `requestOptions`:
+ * ```
+ * {
+ * url, // Provided by RequestList and/or RequestQueue
+ * method, // Provided by RequestList and/or RequestQueue
+ * headers, // Provided by RequestList and/or RequestQueue
+ * payload, // Provided by RequestList and/or RequestQueue
+ * strictSSL, // Use ignoreSslErrors
+ * proxy, // Use useApifyProxy or proxyUrls
+ * }
+ * ```
+ * @property {PrepareRequest} [prepareRequestFunction]
+ * A function that executes before the HTTP request is made to the target resource.
+ * This function is suitable for setting dynamic properties such as cookies to the {@link Request}.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * session: Session
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the initialized request
+ * and the {@link Session} instance corresponds to used session.
+ *
+ * The function should modify the properties of the passed {@link Request} instance
+ * in place because there are already earlier references to it. Making a copy and returning it from
+ * this function is therefore not supported, because it would create inconsistencies where
+ * different parts of SDK would have access to a different {@link Request} instance.
+ *
+ * @property {Number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
+ * @property {Number} [requestTimeoutSecs=30]
+ * Timeout in which the HTTP request to the resource needs to finish, given in seconds.
+ * @property {Boolean} [ignoreSslErrors=true]
+ * If set to true, SSL certificate errors will be ignored.
+ * @property {Boolean} [useApifyProxy=false]
+ * If set to `true`, `CheerioCrawler` will be configured to use
+ * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
+ * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>
+ * @property {String[]} [apifyProxyGroups]
+ * An array of proxy groups to be used
+ * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
+ * Only applied if the `useApifyProxy` option is `true`.
+ * @property {String} [apifyProxySession]
+ * Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.
+ * All HTTP requests going through the proxy with the same session identifier
+ * will use the same target proxy server (i.e. the same IP address).
+ * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
+ * Only applied if the `useApifyProxy` option is `true`.
+ * @property {String[]} [proxyUrls]
+ * An array of custom proxy URLs to be used by the `CheerioCrawler` instance.
+ * The provided custom proxies' order will be randomized and the resulting list rotated.
+ * Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ * configuration options will cause an error to be thrown on startup.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13">source code</a>
+ * for the default implementation of this function.
+ * @property {String[]} [additionalMimeTypes]
+ * An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
+ * target="_blank">mime types</a> you want to process.
+ * By default `text/html`, `application/xhtml+xml` mime types are supported.
+ * @property {Number} [maxRequestRetries=3]
+ * Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
+ * @property {Number} [maxRequestsPerCrawl]
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * are provided by `CheerioCrawler` and cannot be overridden. Reasonable {@link Snapshotter}
+ * and {@link SystemStatus} defaults are provided to account for the fact that `cheerio`
+ * parses HTML synchronously and therefore blocks the event loop.
+ * @property {Number} [minConcurrency=1]
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {Number} [maxConcurrency=1000]
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @property {Boolean} [useSessionPool=false]
+ * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ * It also marks Session as bad after a request timeout.
+ * @property {SessionPoolOptions} [sessionPoolOptions]
+ * Custom options passed to the underlying {@link SessionPool} constructor.
+ * @property {Boolean} [persistCookiesPerSession]
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ *
+ * It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.
+ * It passes the "Cookie" header to the request with the session cookies.
+ */
+/**
+ * Provides a framework for the parallel crawling of web pages using plain HTTP requests and
+ * <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `CheerioCrawler` uses raw HTTP requests to download web pages,
+ * it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript
+ * to display the content, you might need to use {@link PuppeteerCrawler} instead,
+ * because it loads the pages using full-featured headless Chrome browser.
+ *
+ * `CheerioCrawler` downloads each URL using a plain HTTP request,
+ * parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>
+ * and then invokes the user-provided [`handlePageFunction()`](#new_CheerioCrawler_new) to extract page data
+ * using a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface to the parsed HTML DOM.
+ *
+ * The source URLs are represented using {@link Request} objects that are fed from
+ * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_CheerioCrawler_new)
+ * or [`requestQueue`](#new_CheerioCrawler_new) constructor options, respectively.
+ *
+ * If both [`requestList`](#new_CheerioCrawler_new) and [`requestQueue`](#new_CheerioCrawler_new) are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes when there are no more {@link Request} objects to crawl.
+ *
+ * By default, `CheerioCrawler` downloads HTML using the
+ * <a href="https://www.npmjs.com/package/request" target="_blank">request</a> NPM package.
+ * You can use the `requestOptions` parameter to pass additional options to `request`.
+ *
+ * New requests are only dispatched when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPool} options are available directly in the `CheerioCrawler` constructor.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * // Prepare a list of URLs to crawl
+ * const requestList = new Apify.RequestList({
+ * sources: [
+ * { url: 'http://www.example.com/page-1' },
+ * { url: 'http://www.example.com/page-2' },
+ * ],
+ * });
+ * await requestList.initialize();
+ *
+ * // Crawl the URLs
+ * const crawler = new Apify.CheerioCrawler({
+ * requestList,
+ * handlePageFunction: async ({ request, response, body, contentType, $ }) => {
+ * const data = [];
+ *
+ * // Do some data extraction from the page with Cheerio.
+ * $('.some-collection').each((index, el) => {
+ * data.push({ title: $(el).find('.some-title').text() });
+ * });
+ *
+ * // Save the data to dataset.
+ * await Apify.pushData({
+ * url: request.url,
+ * html: body,
+ * data,
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ */
+declare class CheerioCrawler {
+ /**
+ * @param {CheerioCrawlerOptions} options
+ */
+ constructor(options?: CheerioCrawlerOptions);
+ supportedMimeTypes: Set<string>;
+ requestOptions: any;
+ handlePageFunction: any;
+ handlePageTimeoutMillis: number;
+ requestTimeoutMillis: number;
+ ignoreSslErrors: any;
+ useApifyProxy: any;
+ apifyProxyGroups: any;
+ apifyProxySession: any;
+ proxyUrls: any;
+ lastUsedProxyUrlIndex: number;
+ prepareRequestFunction: any;
+ persistCookiesPerSession: any;
+ useSessionPool: any;
+ basicCrawler: BasicCrawler;
+ isRunningPromise: Promise<void>;
+ /**
+ * Runs the crawler. Returns promise that gets resolved once all the requests got processed.
+ *
+ * @return {Promise}
+ */
+ run(): Promise<any>;
+ /**
+ * Wrapper around handlePageFunction that opens and closes pages etc.
+ *
+ * @param {Object} options
+ * @param {Request} options.request
+ * @param {AutoscaledPool} options.autoscaledPool
+ * @param {Session} options.session
+ * @ignore
+ */
+ _handleRequestFunction({ request, autoscaledPool, session }: {
+ request: Request;
+ autoscaledPool: AutoscaledPool;
+ session: Session;
+ }): Promise<any>;
+ /**
+ * Function to make the HTTP request. It performs optimizations
+ * on the request such as only downloading the request body if the
+ * received content type matches text/html, application/xml, application/xhtml+xml.
+ *
+ * @param {Object} options
+ * @param {Request} options.request
+ * @param {Session} options.session
+ * @ignore
+ */
+ _requestFunction({ request, session }: {
+ request: Request;
+ session: Session;
+ }): Promise<{
+ dom: any;
+ isXml: boolean;
+ response: any;
+ contentType: {
+ type: string;
+ encoding: any;
+ };
+ body?: undefined;
+ } | {
+ body: any;
+ response: any;
+ contentType: {
+ type: string;
+ encoding: any;
+ };
+ dom?: undefined;
+ isXml?: undefined;
+ }>;
+ /**
+ * Combines the provided `requestOptions` with mandatory (non-overridable) values.
+ * @param {Request} request
+ * @param {Session?} session
+ * @ignore
+ */
+ _getRequestOptions(request: Request, session: Session): any;
+ /**
+ * Enables the use of a proxy by returning a proxy URL
+ * based on configured options or null if no proxy is used.
+ * @param {Session?} session
+ * @returns {string|null}
+ * @ignore
+ */
+ _getProxyUrl(session?: Session): string;
+ _encodeResponse(request: any, response: any, encoding: any): {
+ response: any;
+ encoding: any;
+ };
+ _parseHtmlToDom(response: any): Promise<any>;
+ /**
+ * Checks and extends supported mime types
+ * @param {Array<String|Object>} additionalMimeTypes
+ * @ignore
+ */
+ _extendSupportedMimeTypes(additionalMimeTypes: any[]): void;
+ /**
+ * Handles blocked request
+ * @param session {Session}
+ * @param statusCode {Number}
+ * @private
+ */
+ _throwOnBlockedRequest(session: Session, statusCode: number): void;
+ /**
+ * Handles timeout request
+ * @param session {Session}
+ * @private
+ */
+ _handleRequestTimeout(session: Session): void;
+}
+import { RequestList } from "../request_list";
+import { RequestQueue } from "../request_queue";
+import * as utilsRequest from "../utils_request";
+import { HandleFailedRequest } from "./basic_crawler";
+import { AutoscaledPoolOptions } from "../autoscaling/autoscaled_pool";
+import { SessionPoolOptions } from "../session_pool/session_pool";
+import Request from "../request";
+import { IncomingMessage } from "http";
+import AutoscaledPool from "../autoscaling/autoscaled_pool";
+import BasicCrawler from "./basic_crawler";
+import { Session } from "../session_pool/session";
diff --git a/types/crawlers/puppeteer_crawler.d.ts b/types/crawlers/puppeteer_crawler.d.ts
new file mode 100644
index 000000000..7881a3250
--- /dev/null
+++ b/types/crawlers/puppeteer_crawler.d.ts
@@ -0,0 +1,506 @@
+export default PuppeteerCrawler;
+export type PuppeteerCrawlerOptions = {
+ /**
+ * Function that is called to process each request.
+ * It is passed an object with the following fields:
+ *
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * puppeteerPool: PuppeteerPool,
+ * autoscaledPool: AutoscaledPool,
+ * session: Session,
+ * }
+ * ```
+ *
+ * `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * `page` is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ * `response` is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ * `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
+ *
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ */
+ handlePageFunction: PuppeteerHandlePage;
+ /**
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ */
+ requestList?: RequestList;
+ /**
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ */
+ requestQueue?: RequestQueue;
+ /**
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+ */
+ handlePageTimeoutSecs?: number;
+ /**
+ * Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,
+ * i.e. a `Promise` resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.
+ *
+ * This is useful if you need to extend the page load timeout or select different criteria
+ * to determine that the navigation succeeded.
+ *
+ * Note that a single page object is only used to process a single request and it is closed afterwards.
+ *
+ * By default, the function invokes [`Apify.utils.puppeteer.gotoExtended()`](puppeteer#puppeteer.gotoExtended) with a timeout of 60 seconds.
+ * For details, see source code on
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.
+ */
+ gotoFunction?: PuppeteerGoto;
+ /**
+ * Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
+ * function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
+ */
+ gotoTimeoutSecs?: number;
+ /**
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * Where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
+ * for the default implementation of this function.
+ */
+ handleFailedRequestFunction?: HandleFailedRequest;
+ /**
+ * Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
+ */
+ maxRequestRetries?: number;
+ /**
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ */
+ maxRequestsPerCrawl?: number;
+ /**
+ * Custom options passed to the underlying {@link PuppeteerPool} constructor.
+ * You can tweak those to fine-tune browser management.
+ */
+ puppeteerPoolOptions?: PuppeteerPoolOptions;
+ /**
+ * Overrides the default function to launch a new Puppeteer instance.
+ * Shortcut to the corresponding {@link PuppeteerPool} option.
+ * See source code on
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
+ * for default behavior.
+ */
+ launchPuppeteerFunction?: Function;
+ /**
+ * Options used by [`Apify.launchPuppeteer()`](apify#module_Apify.launchPuppeteer) to start new Puppeteer instances.
+ * Shortcut to the corresponding {@link PuppeteerPool} option. See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
+ */
+ launchPuppeteerOptions?: LaunchPuppeteerOptions;
+ /**
+ * Custom options passed to the underlying {@link AutoscaledPool} instance constructor.
+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * are provided by `PuppeteerCrawler` and should not be overridden.
+ */
+ autoscaledPoolOptions?: AutoscaledPoolOptions;
+ /**
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ */
+ minConcurrency?: number;
+ /**
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ */
+ maxConcurrency?: number;
+ /**
+ * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ * It also marks Session as bad after a request timeout.
+ */
+ useSessionPool?: boolean;
+ /**
+ * Custom options passed to the underlying {@link SessionPool} constructor.
+ */
+ sessionPoolOptions?: SessionPoolOptions;
+ /**
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ */
+ persistCookiesPerSession?: boolean;
+};
+export type PuppeteerHandlePageInputs = {
+ /**
+ * An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ */
+ request: Request;
+ /**
+ * An instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ */
+ response: PuppeteerResponse;
+ /**
+ * is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ */
+ page: PuppeteerPage;
+ /**
+ * An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
+ */
+ puppeteerPool: PuppeteerPool;
+ autoscaledPool: AutoscaledPool;
+ session?: Session;
+};
+export type PuppeteerHandlePage = (inputs: PuppeteerHandlePageInputs) => Promise<void>;
+export type PuppeteerGotoInputs = {
+ /**
+ * is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ */
+ page: PuppeteerPage;
+ /**
+ * An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ */
+ request: Request;
+ /**
+ * An instance of the `AutoscaledPool`.
+ */
+ autoscaledPool: AutoscaledPool;
+ /**
+ * An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
+ */
+ puppeteerPool: PuppeteerPool;
+ /**
+ * `Session` object for this request.
+ */
+ session?: Session;
+};
+export type PuppeteerGoto = (inputs: PuppeteerGotoInputs) => Promise<PuppeteerResponse>;
+export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browser>;
+/**
+ * @typedef {Object} PuppeteerCrawlerOptions
+ * @property {PuppeteerHandlePage} handlePageFunction
+ * Function that is called to process each request.
+ * It is passed an object with the following fields:
+ *
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * puppeteerPool: PuppeteerPool,
+ * autoscaledPool: AutoscaledPool,
+ * session: Session,
+ * }
+ * ```
+ *
+ * `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * `page` is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>
+ * `response` is an instance of the `Puppeteer`
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ * `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
+ *
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {Number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+ * @property {PuppeteerGoto} [gotoFunction]
+ * Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">page.goto()</a> function,
+ * i.e. a `Promise` resolving to the <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank">Response</a> object.
+ *
+ * This is useful if you need to extend the page load timeout or select different criteria
+ * to determine that the navigation succeeded.
+ *
+ * Note that a single page object is only used to process a single request and it is closed afterwards.
+ *
+ * By default, the function invokes [`Apify.utils.puppeteer.gotoExtended()`](puppeteer#puppeteer.gotoExtended) with a timeout of 60 seconds.
+ * For details, see source code on
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292" target="_blank">GitHub</a>.
+ * @property {Number} [gotoTimeoutSecs=60]
+ * Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
+ * function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * }
+ * ```
+ * Where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301" target="_blank">source code</a>
+ * for the default implementation of this function.
+ * @property {Number} [maxRequestRetries=3]
+ * Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
+ * @property {Number} [maxRequestsPerCrawl]
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @property {PuppeteerPoolOptions} [puppeteerPoolOptions]
+ * Custom options passed to the underlying {@link PuppeteerPool} constructor.
+ * You can tweak those to fine-tune browser management.
+ * @property {Function} [launchPuppeteerFunction]
+ * Overrides the default function to launch a new Puppeteer instance.
+ * Shortcut to the corresponding {@link PuppeteerPool} option.
+ * See source code on
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
+ * for default behavior.
+ * @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]
+ * Options used by [`Apify.launchPuppeteer()`](apify#module_Apify.launchPuppeteer) to start new Puppeteer instances.
+ * Shortcut to the corresponding {@link PuppeteerPool} option. See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
+ * Custom options passed to the underlying {@link AutoscaledPool} instance constructor.
+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * are provided by `PuppeteerCrawler` and should not be overridden.
+ * @property {Number} [minConcurrency=1]
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {Number} [maxConcurrency=1000]
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @property {Boolean} [useSessionPool=false]
+ * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ * It also marks Session as bad after a request timeout.
+ * @property {SessionPoolOptions} [sessionPoolOptions]
+ * Custom options passed to the underlying {@link SessionPool} constructor.
+ * @property {Boolean} [persistCookiesPerSession]
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ */
+/**
+ * Provides a simple framework for parallel crawling of web pages
+ * using headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data,
+ * it is useful for crawling of websites that require to execute JavaScript.
+ * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * which downloads the pages using raw HTTP requests and is about 10x faster.
+ *
+ * The source URLs are represented using {@link Request} objects that are fed from
+ * {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_PuppeteerCrawler_new)
+ * or [`requestQueue`](#new_PuppeteerCrawler_new) constructor options, respectively.
+ *
+ * If both [`requestList`](#new_PuppeteerCrawler_new) and [`requestQueue`](#new_PuppeteerCrawler_new) are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes when there are no more {@link Request} objects to crawl.
+ *
+ * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
+ * and then calls the function provided by user as the [`handlePageFunction()`](#new_PuppeteerCrawler_new) option.
+ *
+ * New pages are only opened when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPool} options are available directly in the `PuppeteerCrawler` constructor.
+ *
+ * Note that the pool of Puppeteer instances is internally managed by
+ * the {@link PuppeteerPool} class. Many constructor options
+ * such as `maxOpenPagesPerInstance` or `launchPuppeteerFunction` are passed directly
+ * to {@link PuppeteerPool} constructor.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const crawler = new Apify.PuppeteerCrawler({
+ * requestList,
+ * handlePageFunction: async ({ page, request }) => {
+ * // This function is called to extract data from a single web page
+ * // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
+ * // 'request' is an instance of Request class with information about the page to load
+ * await Apify.pushData({
+ * title: await page.title(),
+ * url: request.url,
+ * succeeded: true,
+ * })
+ * },
+ * handleFailedRequestFunction: async ({ request }) => {
+ * // This function is called when the crawling of a request failed too many times
+ * await Apify.pushData({
+ * url: request.url,
+ * succeeded: false,
+ * errors: request.errorMessages,
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ */
+declare class PuppeteerCrawler {
+ /**
+ * @param {PuppeteerCrawlerOptions} options All `PuppeteerCrawler` parameters are passed
+ * via an options object with the following keys:
+ */
+ constructor(options: PuppeteerCrawlerOptions);
+ handlePageFunction: PuppeteerHandlePage;
+ gotoFunction: PuppeteerGoto;
+ handlePageTimeoutMillis: number;
+ gotoTimeoutMillis: number;
+ puppeteerPoolOptions: {
+ launchPuppeteerFunction: Function;
+ launchPuppeteerOptions: LaunchPuppeteerOptions;
+ /**
+ * Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
+ * just before a page would be recycled by `PuppeteerPool`. If there are no clients
+ * connected, it has close to zero impact on performance.
+ */
+ useLiveView?: boolean;
+ /**
+ * Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.
+ */
+ maxOpenPagesPerInstance?: number;
+ /**
+ * Maximum number of requests that can be processed by a single browser instance.
+ * After the limit is reached, the browser is retired and new requests are
+ * handled by a new browser instance.
+ */
+ retireInstanceAfterRequestCount?: number;
+ /**
+ * All browser management operations such as launching a new browser, opening a new page
+ * or closing a page will timeout after the set number of seconds and the connected
+ * browser will be retired.
+ */
+ puppeteerOperationTimeoutSecs?: number;
+ /**
+ * Indicates how often are the open Puppeteer instances checked whether they can be closed.
+ */
+ instanceKillerIntervalSecs?: number;
+ /**
+ * When Puppeteer instance reaches the `retireInstanceAfterRequestCount` limit then
+ * it is considered retired and no more tabs will be opened. After the last tab is closed the
+ * whole browser is closed too. This parameter defines a time limit between the last tab was opened and
+ * before the browser is closed even if there are pending open tabs.
+ */
+ killInstanceAfterSecs?: number;
+ /**
+ * Enables recycling of disk cache directories by Chrome instances.
+ * When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.
+ * This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
+ * Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.
+ *
+ * Beware that the disk cache directories can consume a lot of disk space.
+ * To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `launchPuppeteerargs`,
+ * where `X` is the approximate maximum number of bytes for disk cache.
+ *
+ * Do not use the `recycleDiskCache` setting together with `--disk-cache-dir`
+ * argument in `launchPuppeteerargs`, the behavior is undefined.
+ */
+ recycleDiskCache?: boolean;
+ /**
+ * With this option selected, all pages will be opened in a new incognito browser context, which means
+ * that they will not share cookies or cache and their resources will not be throttled by one another.
+ */
+ useIncognitoPages?: boolean;
+ /**
+ * An array of custom proxy URLs to be used by the `PuppeteerPool` instance.
+ * The provided custom proxies' order will be randomized and the resulting list rotated.
+ * Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ * configuration options will cause an error to be thrown on startup.
+ */
+ proxyUrls?: string[];
+ };
+ puppeteerPool: PuppeteerPool;
+ useSessionPool: boolean;
+ sessionPoolOptions: SessionPoolOptions;
+ persistCookiesPerSession: boolean;
+ basicCrawler: BasicCrawler;
+ /**
+ * Runs the crawler. Returns promise that gets resolved once all the requests got processed.
+ *
+ * @return {Promise<void>}
+ */
+ run(): Promise<void>;
+ sessionPool: import("../session_pool/session_pool").SessionPool;
+ isRunningPromise: Promise<void>;
+ /**
+ * Wrapper around handlePageFunction that opens and closes pages etc.
+ *
+ * @param {Object} options
+ * @param {Request} options.request
+ * @param {AutoscaledPool} options.autoscaledPool
+ * @ignore
+ */
+ _handleRequestFunction({ request, autoscaledPool }: {
+ request: Request;
+ autoscaledPool: AutoscaledPool;
+ }): Promise<void>;
+ /**
+ * @param {Object} options
+ * @param {PuppeteerPage} options.page
+ * @param {Request} options.request
+ * @return {Promise<PuppeteerResponse>}
+ * @ignore
+ */
+ _defaultGotoFunction({ page, request }: {
+ page: PuppeteerPage;
+ request: Request;
+ }): Promise<PuppeteerResponse>;
+ /**
+ * @param {Object} options
+ * @param {Request} options.request
+ * @return {Promise<void>}
+ * @ignore
+ */
+ _defaultHandleFailedRequestFunction({ request }: {
+ request: Request;
+ }): Promise<void>;
+}
+import { RequestList } from "../request_list";
+import { RequestQueue } from "../request_queue";
+import { HandleFailedRequest } from "./basic_crawler";
+import { PuppeteerPoolOptions } from "../puppeteer_pool";
+import { LaunchPuppeteerOptions } from "../puppeteer";
+import { AutoscaledPoolOptions } from "../autoscaling/autoscaled_pool";
+import { SessionPoolOptions } from "../session_pool/session_pool";
+import Request from "../request";
+import { Response as PuppeteerResponse } from "puppeteer";
+import { Page as PuppeteerPage } from "puppeteer";
+import PuppeteerPool from "../puppeteer_pool";
+import AutoscaledPool from "../autoscaling/autoscaled_pool";
+import { Session } from "../session_pool/session";
+import { Browser } from "puppeteer";
+import BasicCrawler from "./basic_crawler";
diff --git a/types/crawlers/statistics.d.ts b/types/crawlers/statistics.d.ts
new file mode 100644
index 000000000..2fdf691a2
--- /dev/null
+++ b/types/crawlers/statistics.d.ts
@@ -0,0 +1,39 @@
+/**
+ * The statistics class provides an interface to collecting and logging run
+ * statistics of arbitrary jobs. Currently it provides the following information:
+ *
+ * - Average run time of successful jobs
+ * - Number of successful jobs per minute
+ * - Total number of successful jobs
+ * - Total number of failed jobs
+ * - A histogram of retry counts = Number of jobs that finished after N retries.
+ *
+ * @ignore
+ */
+export default class Statistics {
+ constructor(options?: {});
+ logIntervalMillis: number;
+ logMessage: any;
+ jobRetryHistogram: any[];
+ finishedJobs: number;
+ failedJobs: number;
+ jobsInProgress: Map<any, any>;
+ minJobDurationMillis: number;
+ maxJobDurationMillis: number;
+ totalJobDurationMillis: number;
+ startedAt: Date;
+ logInterval: NodeJS.Timeout;
+ startJob(id: any): void;
+ finishJob(id: any): void;
+ failJob(id: any): void;
+ getCurrent(): {
+ avgDurationMillis: number;
+ perMinute: number;
+ finished: number;
+ failed: number;
+ retryHistogram: any[];
+ };
+ startLogging(): void;
+ stopLogging(): void;
+ _saveRetryCountForJob(job: any): void;
+}
diff --git a/types/dataset.d.ts b/types/dataset.d.ts
new file mode 100644
index 000000000..3942834ac
--- /dev/null
+++ b/types/dataset.d.ts
@@ -0,0 +1,366 @@
+export const DATASET_ITERATORS_DEFAULT_LIMIT: 10000;
+export const LOCAL_STORAGE_SUBDIR: any;
+export const LOCAL_FILENAME_DIGITS: 9;
+export const LOCAL_GET_ITEMS_DEFAULT_LIMIT: 250000;
+export function checkAndSerialize(item: any, limitBytes: number, index?: number): string;
+export function chunkBySize(items: any[], limitBytes: number): any[];
+/**
+ * The `Dataset` class represents a store for structured data where each object stored has the same attributes,
+ * such as online store products or real estate offers. You can imagine it as a table,
+ * where each object is a row and its attributes are columns.
+ * Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records.
+ * Typically it is used to store crawling results.
+ *
+ * Do not instantiate this class directly, use the
+ * [`Apify.openDataset()`](apify#module_Apify.openDataset) function instead.
+ *
+ * `Dataset` stores its data either on local disk or in the Apify cloud,
+ * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.
+ *
+ * If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in
+ * the local directory in the following files:
+ * ```
+ * {APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json
+ * ```
+ * Note that `{DATASET_ID}` is the name or ID of the dataset. The default dataset has ID: `default`,
+ * unless you override it by setting the `APIFY_DEFAULT_DATASET_ID` environment variable.
+ * Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.
+ *
+ * If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the
+ * <a href="https://docs.apify.com/storage/dataset" target="_blank">Apify Dataset</a>
+ * cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`
+ * option to [`Apify.openDataset()`](apify#module_Apify.openDataset) function,
+ * even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * // Write a single row to the default dataset
+ * await Apify.pushData({ col1: 123, col2: 'val2' });
+ *
+ * // Open a named dataset
+ * const dataset = await Apify.openDataset('some-name');
+ *
+ * // Write a single row
+ * await dataset.pushData({ foo: 'bar' });
+ *
+ * // Write multiple rows
+ * await dataset.pushData([
+ * { foo: 'bar2', col2: 'val2' },
+ * { col3: 123 },
+ * ]);
+ * ```
+ * @hideconstructor
+ */
+export class Dataset {
+ /**
+ * @param {string} datasetId
+ * @param {string} datasetName
+ */
+ constructor(datasetId: string, datasetName: string);
+ datasetId: string;
+ datasetName: string;
+ /**
+ * Stores an object or an array of objects to the dataset.
+ * The function returns a promise that resolves when the operation finishes.
+ * It has no result, but throws on invalid args or other errors.
+ *
+ * **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,
+ * otherwise the actor process might finish before the data is stored!
+ *
+ * The size of the data is limited by the receiving API and therefore `pushData()` will only
+ * allow objects whose JSON representation is smaller than 9MB. When an array is passed,
+ * none of the included objects
+ * may be larger than 9MB, but the array itself may be of any size.
+ *
+ * The function internally
+ * chunks the array into separate items and pushes them sequentially.
+ * The chunking process is stable (keeps order of data), but it does not provide a transaction
+ * safety mechanism. Therefore, in the event of an uploading error (after several automatic retries),
+ * the function's Promise will reject and the dataset will be left in a state where some of
+ * the items have already been saved to the dataset while other items from the source array were not.
+ * To overcome this limitation, the developer may, for example, read the last item saved in the dataset
+ * and re-attempt the save of the data from this item onwards to prevent duplicates.
+ *
+ * @param {Object|Array} data Object or array of objects containing data to be stored in the default dataset.
+ * The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
+ * @return {Promise<void>}
+ */
+ pushData(data: any): Promise<void>;
+ /**
+ * Returns {DatasetContent} object holding the items in the dataset based on the provided parameters.
+ *
+ * **NOTE**: If using dataset with local disk storage, the `format` option must be `json` and
+ * the following options are not supported:
+ * `unwind`, `disableBodyParser`, `attachment`, `bom` and `simplified`.
+ * If you try to use them, you will receive an error.
+ *
+ * @param {Object} [options] All `getData()` parameters are passed
+ * via an options object with the following keys:
+ * @param {String} [options.format='json']
+ * Format of the `items` property, possible values are: `json`, `csv`, `xlsx`, `html`, `xml` and `rss`.
+ * @param {Number} [options.offset=0]
+ * Number of array elements that should be skipped at the start.
+ * @param {Number} [options.limit=250000]
+ * Maximum number of array elements to return.
+ * @param {Boolean} [options.desc=false]
+ * If `true` then the objects are sorted by `createdAt` in descending order.
+ * Otherwise they are sorted in ascending order.
+ * @param {Array} [options.fields]
+ * An array of field names that will be included in the result. If omitted, all fields are included in the results.
+ * @param {String} [options.unwind]
+ * Specifies a name of the field in the result objects that will be used to unwind the resulting objects.
+ * By default, the results are returned as they are.
+ * @param {Boolean} [options.disableBodyParser=false]
+ * If `true` then response from API will not be parsed.
+ * @param {Boolean} [options.attachment=false]
+ * If `true` then the response will define the `Content-Disposition: attachment` HTTP header, forcing a web
+ * browser to download the file rather than to display it. By default, this header is not present.
+ * @param {String} [options.delimiter=',']
+ * A delimiter character for CSV files, only used if `format` is `csv`.
+ * @param {Boolean} [options.bom]
+ * All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte
+ * Order Mark (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default
+ * behavior, set `bom` option to `true` to include the BOM, or set `bom` to `false` to skip it.
+ * @param {String} [options.xmlRoot='results']
+ * Overrides the default root element name of the XML output. By default, the root element is `results`.
+ * @param {String} [options.xmlRow='page']
+ * Overrides the default element name that wraps each page or page function result object in XML output.
+ * By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
+ * @param {Boolean} [options.skipHeaderRow=false]
+ * If set to `true` then header row in CSV format is skipped.
+ * @param {Boolean} [options.clean=false]
+ * If `true` then the function returns only non-empty items and skips hidden fields (i.e. fields starting with `#` character).
+ * Note that the `clean` parameter is a shortcut for `skipHidden: true` and `skipEmpty: true` options.
+ * @param {Boolean} [options.skipHidden=false]
+ * If `true` then the function doesn't return hidden fields (fields starting with "#" character).
+ * @param {Boolean} [options.skipEmpty=false]
+ * If `true` then the function doesn't return empty items.
+ * Note that in this case the returned number of items might be lower than limit parameter and pagination must be done using the `limit` value.
+ * @param {Boolean} [options.simplified]
+ * If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and `unwind: 'pageFunctionResult'` options.
+ * This feature is used to emulate simplified results provided by Apify API version 1 used for
+ * the legacy Apify Crawler and it's not recommended to use it in new integrations.
+ * @param {Boolean} [options.skipFailedPages]
+ * If `true` then, the all the items with errorInfo property will be skipped from the output.
+ * This feature is here to emulate functionality of Apify API version 1 used for
+ * the legacy Apify Crawler product and it's not recommended to use it in new integrations.
+ * @return {Promise<DatasetContent>}
+ */
+ getData(options?: {
+ format?: string;
+ offset?: number;
+ limit?: number;
+ desc?: boolean;
+ fields?: any[];
+ unwind?: string;
+ disableBodyParser?: boolean;
+ attachment?: boolean;
+ delimiter?: string;
+ bom?: boolean;
+ xmlRoot?: string;
+ xmlRow?: string;
+ skipHeaderRow?: boolean;
+ clean?: boolean;
+ skipHidden?: boolean;
+ skipEmpty?: boolean;
+ simplified?: boolean;
+ skipFailedPages?: boolean;
+ }): Promise<DatasetContent>;
+ /**
+ * Returns an object containing general information about the dataset.
+ *
+ * The function returns the same object as the Apify API Client's
+ * [getDataset](https://docs.apify.com/api/apify-client-js/latest#ApifyClient-datasets-getDataset)
+ * function, which in turn calls the
+ * [Get dataset](https://apify.com/docs/api/v2#/reference/datasets/dataset/get-dataset)
+ * API endpoint.
+ *
+ * **Example:**
+ * ```
+ * {
+ * id: "WkzbQMuFYuamGv3YF",
+ * name: "my-dataset",
+ * userId: "wRsJZtadYvn4mBZmm",
+ * createdAt: new Date("2015-12-12T07:34:14.202Z"),
+ * modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
+ * accessedAt: new Date("2015-12-14T08:36:13.202Z"),
+ * itemCount: 14,
+ * cleanItemCount: 10
+ * }
+ * ```
+ *
+ * @returns {Promise<Object>}
+ */
+ getInfo(): Promise<any>;
+ /**
+ * Iterates over dataset items, yielding each in turn to an `iteratee` function.
+ * Each invocation of `iteratee` is called with two arguments: `(item, index)`.
+ *
+ * If the `iteratee` function returns a Promise then it is awaited before the next call.
+ * If it throws an error, the iteration is aborted and the `forEach` function throws the error.
+ *
+ * **Example usage**
+ * ```javascript
+ * const dataset = await Apify.openDataset('my-results');
+ * await dataset.forEach(async (item, index) => {
+ * console.log(`Item at ${index}: ${JSON.stringify(item)}`);
+ * });
+ * ```
+ *
+ * @param {DatasetConsumer} iteratee A function that is called for every item in the dataset.
+ * @param {Object} [options] All `forEach()` parameters are passed
+ * via an options object with the following keys:
+ * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by `createdAt` in descending order.
+ * @param {Array} [options.fields] If provided then returned objects will only contain specified keys.
+ * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @param {Number} [index=0] Specifies the initial index number passed to the `iteratee` function.
+ * @return {Promise<void>}
+ */
+ forEach(iteratee: DatasetConsumer, options?: {
+ desc?: boolean;
+ fields?: any[];
+ unwind?: string;
+ }, index?: number): Promise<void>;
+ /**
+ * Produces a new array of values by mapping each value in list through a transformation function `iteratee()`.
+ * Each invocation of `iteratee()` is called with two arguments: `(element, index)`.
+ *
+ * If `iteratee` returns a `Promise` then it's awaited before a next call.
+ *
+ * @template T
+ * @param {DatasetMapper} iteratee
+ * @param {Object} options All `map()` parameters are passed
+ * via an options object with the following keys:
+ * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
+ * @param {Array} [options.fields] If provided then returned objects will only contain specified keys
+ * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @return {Promise<T[]>}
+ */
+ map<T>(iteratee: DatasetMapper, options: {
+ desc?: boolean;
+ fields?: any[];
+ unwind?: string;
+ }): Promise<T[]>;
+ /**
+ * Reduces a list of values down to a single value.
+ *
+ * Memo is the initial state of the reduction, and each successive step of it should be returned by `iteratee()`.
+ * The `iteratee()` is passed three arguments: the `memo`, then the `value` and `index` of the iteration.
+ *
+ * If no `memo` is passed to the initial invocation of reduce, the `iteratee()` is not invoked on the first element of the list.
+ * The first element is instead passed as the memo in the invocation of the `iteratee()` on the next element in the list.
+ *
+ * If `iteratee()` returns a `Promise` then it's awaited before a next call.
+ *
+ * @template T
+ * @param {DatasetReducer} iteratee
+ * @param {T} memo Initial state of the reduction.
+ * @param {Object} options All `reduce()` parameters are passed
+ * via an options object with the following keys:
+ * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
+ * @param {Array} [options.fields] If provided then returned objects will only contain specified keys
+ * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @return {Promise<T>}
+ */
+ reduce<T_1>(iteratee: DatasetReducer, memo: T_1, options: {
+ desc?: boolean;
+ fields?: any[];
+ unwind?: string;
+ }): Promise<T_1>;
+ /**
+ * Removes the dataset either from the Apify cloud storage or from the local directory,
+ * depending on the mode of operation.
+ *
+ * @return {Promise<void>}
+ */
+ drop(): Promise<void>;
+ /** @ignore */
+ delete(): Promise<void>;
+}
+/**
+ * This is a local emulation of a dataset.
+ *
+ * @ignore
+ */
+export class DatasetLocal {
+ constructor(datasetId: any, localStorageDir: any);
+ localStoragePath: string;
+ counter: number;
+ datasetId: any;
+ createdAt: any;
+ modifiedAt: any;
+ accessedAt: any;
+ initializationPromise: any;
+ _initialize(): any;
+ pushData(data: any): any;
+ getData(opts?: {}): Promise<{
+ items: any[];
+ total: number;
+ offset: any;
+ count: number;
+ limit: any;
+ }>;
+ getInfo(): Promise<{
+ id: any;
+ name: any;
+ userId: string;
+ createdAt: any;
+ modifiedAt: any;
+ accessedAt: any;
+ itemCount: number;
+ cleanItemCount: number;
+ }>;
+ forEach(iteratee: any): Promise<void>;
+ map(iteratee: any): Promise<any[]>;
+ reduce(iteratee: any, memo: any): Promise<any>;
+ drop(): Promise<void>;
+ delete(): Promise<void>;
+ /**
+ * Returns an array of item indexes for given offset and limit.
+ */
+ _getItemIndexes(offset?: number, limit?: number): any;
+ /**
+ * Reads and parses file for given index.
+ */
+ _readAndParseFile(index: any): Promise<any>;
+ _updateMetadata(isModified: any): void;
+}
+export function openDataset(datasetIdOrName?: string, options?: {
+ forceCloud?: boolean;
+}): Promise<Dataset>;
+export function pushData(item: any): Promise<any>;
+export type DatasetContent = {
+ /**
+ * Dataset entries based on chosen format parameter.
+ */
+ items: any[] | string[] | Buffer[];
+ /**
+ * Total count of entries in the dataset.
+ */
+ total: number;
+ /**
+ * Position of the first returned entry in the dataset.
+ */
+ offset: number;
+ /**
+ * Count of dataset entries returned in this set.
+ */
+ count: number;
+ /**
+ * Maximum number of dataset entries requested.
+ */
+ limit: number;
+};
+/**
+ * User-function used in the `Dataset.forEach()` API.
+ */
+export type DatasetConsumer = (item: any, index: number) => any;
+/**
+ * User-function used in the `Dataset.map()` API.
+ */
+export type DatasetMapper = (item: any, index: number) => any;
+/**
+ * User-function used in the `Dataset.reduce()` API.
+ */
+export type DatasetReducer = (memo: any, item: any, index: number) => any;
diff --git a/types/enqueue_links/click_elements.d.ts b/types/enqueue_links/click_elements.d.ts
new file mode 100644
index 000000000..260fe2d63
--- /dev/null
+++ b/types/enqueue_links/click_elements.d.ts
@@ -0,0 +1,155 @@
+/**
+ * The function finds elements matching a specific CSS selector in a Puppeteer page,
+ * clicks all those elements using a mouse move and a left mouse button click and intercepts
+ * all the navigation requests that are subsequently produced by the page. The intercepted
+ * requests, including their methods, headers and payloads are then enqueued to a provided
+ * {@link RequestQueue}. This is useful to crawl JavaScript heavy pages where links are not available
+ * in `href` elements, but rather navigations are triggered in click handlers.
+ * If you're looking to find URLs in `href` attributes of the page, see [`enqueueLinks()`](utils#utils.enqueueLinks).
+ *
+ * Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects
+ * and override settings of the enqueued {@link Request} objects.
+ *
+ * **IMPORTANT**: To be able to do this, this function uses various mutations on the page,
+ * such as changing the Z-index of elements being clicked and their visibility. Therefore,
+ * it is recommended to only use this function as the last operation in the page.
+ *
+ * **USING HEADFUL BROWSER**: When using a headful browser, this function will only be able to click elements
+ * in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved.
+ *
+ * **PERFORMANCE**: Clicking elements with a mouse and intercepting requests is not a low level operation
+ * that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting
+ * the scope of the clicking as much as possible by using a specific selector that targets only the elements
+ * that you assume or know will produce a navigation. You can certainly click everything by using
+ * the `*` selector, but be prepared to wait minutes to get results on a large and complex page.
+ *
+ * **Example usage**
+ *
+ * ```javascript
+ * const Apify = require('apify');
+ *
+ * const browser = await Apify.launchPuppeteer();
+ * const page = await browser.goto('https://www.example.com');
+ * const requestQueue = await Apify.openRequestQueue();
+ *
+ * await Apify.utils.enqueueLinksByClickingElements({
+ * page,
+ * requestQueue,
+ * selector: 'a.product-detail',
+ * pseudoUrls: [
+ * 'https://www.example.com/handbags/[.*]'
+ * 'https://www.example.com/purses/[.*]'
+ * ],
+ * });
+ * ```
+ *
+ * @param {Object} options
+ * All `enqueueLinksByClickingElements()` parameters are passed
+ * via an options object with the following keys:
+ * @param {Page} options.page
+ * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.
+ * @param {RequestQueue} options.requestQueue
+ * A request queue to which the URLs will be enqueued.
+ * @param {String} options.selector
+ * A CSS selector matching elements to be clicked on. Unlike in [`enqueueLinks()`](utils#utils.enqueueLinks), there is no default
+ * value. This is to prevent suboptimal use of this function by using it too broadly.
+ * @param {Array<String|RegExp|Object>} [options.pseudoUrls]
+ * An array of {@link PseudoUrl}s matching the URLs to be enqueued,
+ * or an array of strings or RegExps or plain Objects from which the {@link PseudoUrl}s can be constructed.
+ *
+ * The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp.
+ * All remaining keys will be used as the `requestTemplate` argument of the {@link PseudoUrl} constructor,
+ * which lets you specify special properties for the enqueued {@link Request} objects.
+ *
+ * If `pseudoUrls` is an empty array, `null` or `undefined`, then the function
+ * enqueues all links found on the page.
+ * @param {RequestTransform} [options.transformRequestFunction]
+ * Just before a new {@link Request} is constructed and enqueued to the {@link RequestQueue}, this function can be used
+ * to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful
+ * when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads,
+ * or to dynamically update or create `userData`.
+ *
+ * For example: by adding `useExtendedUniqueKey: true` to the `request` object, `uniqueKey` will be computed from
+ * a combination of `url`, `method` and `payload` which enables crawling of websites that navigate using form submits
+ * (POST requests).
+ *
+ * **Example:**
+ * ```javascript
+ * {
+ * transformRequestFunction: (request) => {
+ * request.userData.foo = 'bar';
+ * request.useExtendedUniqueKey = true;
+ * return request;
+ * }
+ * }
+ * ```
+ * @param {number} [options.waitForPageIdleSecs=1]
+ * Clicking in the page triggers various asynchronous operations that lead to new URLs being shown
+ * by the browser. It could be a simple JavaScript redirect or opening of a new tab in the browser.
+ * These events often happen only some time after the actual click. Requests typically take milliseconds
+ * while new tabs open in hundreds of milliseconds.
+ *
+ * To be able to capture all those events, the `enqueueLinksByClickingElements()` function repeatedly waits
+ * for the `waitForPageIdleSecs`. By repeatedly we mean that whenever a relevant event is triggered, the timer
+ * is restarted. As long as new events keep coming, the function will not return, unless
+ * the below `maxWaitForPageIdleSecs` timeout is reached.
+ *
+ * You may want to reduce this for example when you're sure that your clicks do not open new tabs,
+ * or increase when you're not getting all the expected URLs.
+ * @param {number} [options.maxWaitForPageIdleSecs=5]
+ * This is the maximum period for which the function will keep tracking events, even if more events keep coming.
+ * Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself.
+ * See `waitForPageIdleSecs` above for an explanation.
+ * @return {Promise<QueueOperationInfo[]>}
+ * Promise that resolves to an array of {@link QueueOperationInfo} objects.
+ * @memberOf puppeteer
+ * @name enqueueLinksByClickingElements
+ */
+export function enqueueLinksByClickingElements(options?: {
+ page: any;
+ requestQueue: RequestQueue;
+ selector: string;
+ pseudoUrls?: any[];
+ transformRequestFunction?: RequestTransform;
+ waitForPageIdleSecs?: number;
+ maxWaitForPageIdleSecs?: number;
+}): Promise<QueueOperationInfo[]>;
+/**
+ * Clicks all elements of given page matching given selector.
+ * Catches and intercepts all initiated navigation requests and opened pages.
+ * Returns a list of all target URLs.
+ *
+ * @param {Object} options
+ * @param {Page} options.page
+ * @param {string} options.selector
+ * @return {Promise<Object[]>}
+ * @ignore
+ */
+export function clickElementsAndInterceptNavigationRequests(options: {
+ page: any;
+ selector: string;
+}): Promise<any[]>;
+/**
+ * We're only interested in pages created by the page we're currently clicking in.
+ * There will generally be a lot of other targets being created in the browser.
+ * @param {Page} page
+ * @param {Target} target
+ * @return {boolean}
+ */
+export function isTargetRelevant(page: any, target: any): boolean;
+/**
+ * Click all elements matching the given selector. To be able to do this using
+ * Puppeteer's `.click()` we need to make sure the elements are reachable by mouse,
+ * so we first move them to the top of the page's stacking context and then click.
+ * We do all in series to prevent elements from hiding one another. Therefore,
+ * for large element sets, this will take considerable amount of time.
+ *
+ * @param {Page} page
+ * @param {string} selector
+ * @return {Promise}
+ * @ignore
+ */
+export function clickElements(page: any, selector: string): Promise<any>;
+import { RequestQueue } from "../request_queue";
+import { RequestTransform } from "./shared";
+import { QueueOperationInfo } from "../request_queue";
diff --git a/types/enqueue_links/enqueue_links.d.ts b/types/enqueue_links/enqueue_links.d.ts
new file mode 100644
index 000000000..ec0e57876
--- /dev/null
+++ b/types/enqueue_links/enqueue_links.d.ts
@@ -0,0 +1,113 @@
+/**
+ * The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default)
+ * either in a Puppeteer page, or in a Cheerio object (parsed HTML),
+ * and enqueues the URLs in their `href` attributes to the provided {@link RequestQueue}.
+ * If you're looking to find URLs in JavaScript heavy pages where links are not available
+ * in `href` elements, but rather navigations are triggered in click handlers
+ * see [`enqueueLinksByClickingElements()`](puppeteer#puppeteer.enqueueLinksByClickingElements).
+ *
+ * Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects
+ * and override settings of the enqueued {@link Request} objects.
+ *
+ * **Example usage**
+ *
+ * ```javascript
+ * const Apify = require('apify');
+ *
+ * const browser = await Apify.launchPuppeteer();
+ * const page = await browser.goto('https://www.example.com');
+ * const requestQueue = await Apify.openRequestQueue();
+ *
+ * await Apify.utils.enqueueLinks({
+ * page,
+ * requestQueue,
+ * selector: 'a.product-detail',
+ * pseudoUrls: [
+ * 'https://www.example.com/handbags/[.*]',
+ * 'https://www.example.com/purses/[.*]'
+ * ],
+ * });
+ * ```
+ *
+ * @param {Object} options
+ * All `enqueueLinks()` parameters are passed
+ * via an options object with the following keys:
+ * @param {Page} options.page
+ * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.
+ * Either `page` or `$` option must be provided.
+ * @param {Cheerio} options.$
+ * <a href="https://github.com/cheeriojs/cheerio" target="_blank"><code>Cheerio</code></a> object with loaded HTML.
+ * Either `page` or `$` option must be provided.
+ * @param {RequestQueue} options.requestQueue
+ * A request queue to which the URLs will be enqueued.
+ * @param {String} [options.selector='a']
+ * A CSS selector matching links to be enqueued.
+ * @param {string} [options.baseUrl]
+ * A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer,
+ * since the relative URL resolution is done inside the browser automatically.
+ * @param {Object[]|String[]} [options.pseudoUrls]
+ * An array of {@link PseudoUrl}s matching the URLs to be enqueued,
+ * or an array of strings or RegExps or plain Objects from which the {@link PseudoUrl}s can be constructed.
+ *
+ * The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp.
+ * All remaining keys will be used as the `requestTemplate` argument of the {@link PseudoUrl} constructor,
+ * which lets you specify special properties for the enqueued {@link Request} objects.
+ *
+ * If `pseudoUrls` is an empty array, `null` or `undefined`, then the function
+ * enqueues all links found on the page.
+ * @param {RequestTransform} [options.transformRequestFunction]
+ * Just before a new {@link Request} is constructed and enqueued to the {@link RequestQueue}, this function can be used
+ * to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful
+ * when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads,
+ * or to dynamically update or create `userData`.
+ *
+ * For example: by adding `keepUrlFragment: true` to the `request` object, URL fragments will not be removed
+ * when `uniqueKey` is computed.
+ *
+ * **Example:**
+ * ```javascript
+ * {
+ * transformRequestFunction: (request) => {
+ * request.userData.foo = 'bar';
+ * request.keepUrlFragment = true;
+ * return request;
+ * }
+ * }
+ * ```
+ * @return {Promise<QueueOperationInfo[]>}
+ * Promise that resolves to an array of {@link QueueOperationInfo} objects.
+ * @memberOf utils
+ * @name enqueueLinks
+ */
+export function enqueueLinks(options?: {
+ page: Page;
+ $: CheerioStatic;
+ requestQueue: RequestQueue;
+ selector?: string;
+ baseUrl?: string;
+ pseudoUrls?: any[] | string[];
+ transformRequestFunction?: RequestTransform;
+}): Promise<QueueOperationInfo[]>;
+/**
+ * Extracts URLs from a given Puppeteer Page.
+ *
+ * @param {Page} page
+ * @param {string} selector
+ * @return {string[]}
+ * @ignore
+ */
+export function extractUrlsFromPage(page: Page, selector: string): string[];
+/**
+ * Extracts URLs from a given Cheerio object.
+ *
+ * @param {Cheerio} $
+ * @param {string} selector
+ * @param {string} baseUrl
+ * @return {string[]}
+ * @ignore
+ */
+export function extractUrlsFromCheerio($: CheerioStatic, selector: string, baseUrl: string): string[];
+import { Page } from "puppeteer";
+import { RequestQueue } from "../request_queue";
+import { RequestTransform } from "./shared";
+import { QueueOperationInfo } from "../request_queue";
diff --git a/types/enqueue_links/shared.d.ts b/types/enqueue_links/shared.d.ts
new file mode 100644
index 000000000..947501231
--- /dev/null
+++ b/types/enqueue_links/shared.d.ts
@@ -0,0 +1,35 @@
+/**
+ * Helper factory used in the `enqueueLinks()` and enqueueLinksByClickingElements() function.
+ * @param {string[]|Object[]} pseudoUrls
+ * @return {PseudoUrl[]}
+ * @ignore
+ */
+export function constructPseudoUrlInstances(pseudoUrls: any[] | string[]): PseudoUrl[];
+/**
+ * @param {string[]|Object[]} requestOptions
+ * @param {PseudoUrl[]} pseudoUrls
+ * @return {Request[]}
+ * @ignore
+ */
+export function createRequests(requestOptions: any[] | string[], pseudoUrls: PseudoUrl[]): Request[];
+/**
+ * @param {string[]|Object[]} sources
+ * @param {Object} [userData]
+ * @ignore
+ */
+export function createRequestOptions(sources: any[] | string[], userData?: any): any;
+/**
+ * @param {Request[]} requests
+ * @param {RequestQueue} requestQueue
+ * @param {number} batchSize
+ * @return {Promise<QueueOperationInfo[]>}
+ * @ignore
+ */
+export function addRequestsToQueueInBatches(requests: Request[], requestQueue: any, batchSize?: number): Promise<any[]>;
+/**
+ * Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used
+ * [`Apify.utils.enqueueLinks`](../api/utils#utils.enqueueLinks) to modify requests before enqueuing them.
+ */
+export type RequestTransform = (original: any) => any;
+import PseudoUrl from "../pseudo_url";
+import Request from "../request";
diff --git a/types/errors.d.ts b/types/errors.d.ts
new file mode 100644
index 000000000..09814a25c
--- /dev/null
+++ b/types/errors.d.ts
@@ -0,0 +1,25 @@
+export const APIFY_CALL_ERROR_NAME: "ApifyCallError";
+/**
+ * The class represents exceptions thrown
+ * by the [`Apify.call()`](../api/apify#module_Apify.call) function.
+ *
+ * @typedef {Object} ApifyCallError
+ * @property {String} message
+ * Error message
+ * @property {String} name
+ * Contains `"ApifyCallError"`
+ * @property {ActorRun} run
+ * Object representing the failed actor run.
+ */
+export class ApifyCallError extends Error {
+ constructor(run: any, message?: string);
+ run: any;
+}
+/**
+ * TimeoutError class.
+ * This error should be thrown after request timeout from `requestAsBrowser`.
+ * @ignore
+ */
+export class TimeoutError extends Error {
+ constructor(message?: string);
+}
diff --git a/types/events.d.ts b/types/events.d.ts
new file mode 100644
index 000000000..1b60b4408
--- /dev/null
+++ b/types/events.d.ts
@@ -0,0 +1,9 @@
+export default events;
+export function initializeEvents(): void;
+export function stopEvents(): void;
+/**
+ * Event emitter providing events from underlying Actor infrastructure and Apify package.
+ * @ignore
+ */
+declare const events: EventEmitter;
+import { EventEmitter } from "node/events";
diff --git a/types/index.esm.d.ts b/types/index.esm.d.ts
new file mode 100644
index 000000000..f2ccf7fb8
--- /dev/null
+++ b/types/index.esm.d.ts
@@ -0,0 +1,136 @@
+/// <reference path="../src/utils_log.d.ts" />
+import { main } from "./actor";
+import { getEnv } from "./actor";
+import { call } from "./actor";
+import { callTask } from "./actor";
+import { metamorph } from "./actor";
+import { getMemoryInfo } from "./utils";
+import { getApifyProxyUrl } from "./actor";
+import { isAtHome } from "./utils";
+import { apifyClient } from "./utils";
+import { addWebhook } from "./actor";
+import AutoscaledPool from "./autoscaling/autoscaled_pool";
+import BasicCrawler from "./crawlers/basic_crawler";
+import CheerioCrawler from "./crawlers/cheerio_crawler";
+import { pushData } from "./dataset";
+import { openDataset } from "./dataset";
+import events from "./events";
+import { initializeEvents } from "./events";
+import { stopEvents } from "./events";
+import { getValue } from "./key_value_store";
+import { setValue } from "./key_value_store";
+import { getInput } from "./key_value_store";
+import { openKeyValueStore } from "./key_value_store";
+import { launchPuppeteer } from "./puppeteer";
+import PuppeteerPool from "./puppeteer_pool";
+import PuppeteerCrawler from "./crawlers/puppeteer_crawler";
+import PseudoUrl from "./pseudo_url";
+import Request from "./request";
+import { RequestList } from "./request_list";
+import { openRequestList } from "./request_list";
+import { openRequestQueue } from "./request_queue";
+import { openSessionPool } from "./session_pool/session_pool";
+import LiveViewServer from "./live_view/live_view_server";
+import { Session } from "./session_pool/session";
+declare const exportedUtils: {
+ isDocker: (forceReset: boolean) => Promise<boolean>;
+ sleep: (millis: number) => Promise<void>;
+ downloadListOfUrls: ({ url, encoding, urlRegExp }: {
+ url: string;
+ encoding?: string;
+ urlRegExp?: RegExp;
+ }) => Promise<string[]>;
+ extractUrls: ({ string, urlRegExp }: {
+ string: string;
+ urlRegExp?: RegExp;
+ }) => string[];
+ getRandomUserAgent: () => string;
+ htmlToText: (html: string | CheerioStatic) => string;
+ URL_NO_COMMAS_REGEX: RegExp;
+ URL_WITH_COMMAS_REGEX: RegExp;
+ createRequestDebugInfo: (request: any, response?: import("http").IncomingMessage | import("puppeteer").Response, additionalFields?: any) => any;
+ parseContentTypeFromResponse: (response: any) => {
+ type: string;
+ charset: string;
+ };
+} & {
+ puppeteer: {
+ hideWebDriver: (page: import("puppeteer").Page) => Promise<any>;
+ injectFile: (page: import("puppeteer").Page, filePath: string, options?: {
+ surviveNavigations?: boolean;
+ }) => Promise<any>;
+ injectJQuery: (page: import("puppeteer").Page) => Promise<any>;
+ injectUnderscore: (page: import("puppeteer").Page) => Promise<any>;
+ enqueueRequestsFromClickableElements: (page: any, selector: any, purls: any, requestQueue: any, requestOpts?: {}) => Promise<any[]>;
+ enqueueLinks: (...args: any[]) => Promise<import("./request_queue").QueueOperationInfo[]>;
+ enqueueLinksByClickingElements: typeof import("./enqueue_links/click_elements").enqueueLinksByClickingElements;
+ blockRequests: (page: import("puppeteer").Page, options?: {
+ urlPatterns?: string[];
+ extraUrlPatterns?: string[];
+ }) => Promise<any>;
+ blockResources: (page: any, resourceTypes?: string[]) => Promise<void>;
+ cacheResponses: (page: import("puppeteer").Page, cache: any, responseUrlRules: (string | RegExp)[]) => Promise<any>;
+ compileScript: (scriptString: string, context?: any) => Function;
+ gotoExtended: (page: import("puppeteer").Page, request: Request, gotoOptions?: any) => Promise<import("puppeteer").Response>;
+ addInterceptRequestHandler: (page: any, handler: Function) => Promise<any>;
+ removeInterceptRequestHandler: (page: any, handler: Function) => Promise<any>;
+ infiniteScroll: (page: any, options?: {
+ timeoutSecs?: number;
+ waitForSecs?: number;
+ }) => Promise<any>;
+ saveSnapshot: (page: any, options?: {
+ key?: string;
+ screenshotQuality?: number;
+ saveScreenshot?: boolean;
+ saveHtml?: boolean;
+ keyValueStoreName?: string;
+ }) => Promise<any>;
+ };
+ social: {
+ emailsFromText: (text: string) => string[];
+ emailsFromUrls: (urls: string[]) => string[];
+ phonesFromText: (text: string) => string[];
+ phonesFromUrls: (urls: string[]) => string[];
+ parseHandlesFromHtml: (html: string, data?: any) => import("./utils_social").SocialHandles;
+ EMAIL_REGEX: RegExp;
+ EMAIL_REGEX_GLOBAL: RegExp;
+ LINKEDIN_REGEX: RegExp;
+ LINKEDIN_REGEX_GLOBAL: RegExp;
+ INSTAGRAM_REGEX: RegExp;
+ INSTAGRAM_REGEX_GLOBAL: RegExp;
+ TWITTER_REGEX: RegExp;
+ TWITTER_REGEX_GLOBAL: RegExp;
+ FACEBOOK_REGEX: RegExp;
+ FACEBOOK_REGEX_GLOBAL: RegExp;
+ YOUTUBE_REGEX: RegExp;
+ YOUTUBE_REGEX_GLOBAL: RegExp;
+ };
+ log: typeof log;
+ enqueueLinks: typeof enqueueLinks;
+ requestAsBrowser: (options: import("./utils_request").RequestAsBrowserOptions) => Promise<any>;
+};
+import { enqueueLinks } from "./enqueue_links/enqueue_links";
+export { main, getEnv, call, callTask, metamorph, getMemoryInfo, getApifyProxyUrl, isAtHome, apifyClient as client, addWebhook, AutoscaledPool, BasicCrawler, CheerioCrawler, pushData, openDataset, events, initializeEvents, stopEvents, getValue, setValue, getInput, openKeyValueStore, launchPuppeteer, PuppeteerPool, PuppeteerCrawler, PseudoUrl, Request, RequestList, openRequestList, openRequestQueue, openSessionPool, LiveViewServer, Session, exportedUtils as utils };
+
+export { ApifyEnv } from './actor'
+export { DatasetContent, DatasetConsumer, DatasetMapper, DatasetReducer } from './dataset'
+export { KeyConsumer } from './key_value_store'
+export { LaunchPuppeteerOptions } from './puppeteer'
+export { PuppeteerPoolOptions } from './puppeteer_pool'
+export { RequestOptions } from './request'
+export { RequestListOptions, RequestListState } from './request_list'
+export { QueueOperationInfo } from './request_queue'
+export { Cheerio, ActorRun } from './typedefs'
+export { MemoryInfo } from './utils'
+export { RequestAsBrowserOptions } from './utils_request'
+export { SocialHandles } from './utils_social'
+export { AutoscaledPoolOptions } from './autoscaling/autoscaled_pool'
+export { SnapshotterOptions } from './autoscaling/snapshotter'
+export { SystemInfo, SystemStatusOptions } from './autoscaling/system_status'
+export { BasicCrawlerOptions, HandleRequest, HandleRequestInputs, HandleFailedRequest, HandleFailedRequestInput } from './crawlers/basic_crawler'
+export { CheerioCrawlerOptions, PrepareRequestInputs, PrepareRequest, CheerioHandlePageInputs, CheerioHandlePage } from './crawlers/cheerio_crawler'
+export { PuppeteerCrawlerOptions, PuppeteerHandlePageInputs, PuppeteerHandlePage, PuppeteerGotoInputs, PuppeteerGoto, LaunchPuppeteer } from './crawlers/puppeteer_crawler'
+export { RequestTransform } from './enqueue_links/shared'
+export { SessionState, SessionOptions } from './session_pool/session'
+export { CreateSession, SessionPoolOptions } from './session_pool/session_pool'
+export { StealthOptions } from './stealth/stealth'
\ No newline at end of file
diff --git a/types/key_value_store.d.ts b/types/key_value_store.d.ts
new file mode 100644
index 000000000..92ac8c45b
--- /dev/null
+++ b/types/key_value_store.d.ts
@@ -0,0 +1,277 @@
+export const LOCAL_STORAGE_SUBDIR: any;
+export function maybeStringify(value: any, options: any): any;
+/**
+ * The `KeyValueStore` class represents a key-value store, a simple data storage that is used
+ * for saving and reading data records or files. Each data record is
+ * represented by a unique key and associated with a MIME content type. Key-value stores are ideal
+ * for saving screenshots, actor inputs and outputs, web pages, PDFs or to persist the state of crawlers.
+ *
+ * Do not instantiate this class directly, use the
+ * [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function instead.
+ *
+ * Each actor run is associated with a default key-value store, which is created exclusively
+ * for the run. By convention, the actor input and output are stored into the
+ * default key-value store under the `INPUT` and `OUTPUT` key, respectively.
+ * Typically, input and output are JSON files, although it can be any other format.
+ * To access the default key-value store directly, you can use the
+ * [`Apify.getValue()`](apify#module_Apify.getValue)
+ * and [`Apify.setValue()`](apify#module_Apify.setValue) convenience functions.
+ *
+ * To access the input, you can also use the [`Apify.getInput()`](apify#module_Apify.getInput) convenience function.
+ *
+ * `KeyValueStore` stores its data either on local disk or in the Apify cloud,
+ * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.
+ *
+ * If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in
+ * the local directory in the following files:
+ * ```
+ * {APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}
+ * ```
+ * Note that `{STORE_ID}` is the name or ID of the key-value store. The default key value store has ID: `default`,
+ * unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.
+ * The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.
+ *
+ * If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the
+ * <a href="https://docs.apify.com/storage/key-value-store" target="_blank">Apify Key-value store</a>
+ * cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`
+ * option to [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function,
+ * even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * // Get actor input from the default key-value store
+ * const input = await Apify.getInput();
+ * const otherValue = Apify.getValue('my-key');
+ *
+ * // Write actor output to the default key-value store.
+ * await Apify.setValue('OUTPUT', { myResult: 123 });
+ *
+ * // Open a named key-value store
+ * const store = await Apify.openKeyValueStore('some-name');
+ *
+ * // Write a record. JavaScript object is automatically converted to JSON,
+ * // strings and binary buffers are stored as they are
+ * await store.setValue('some-key', { foo: 'bar' });
+ *
+ * // Read a record. Note that JSON is automatically parsed to a JavaScript object,
+ * // text data returned as a string and other data is returned as binary buffer
+ * const value = await store.getValue('some-key');
+ *
+ * // Drop (delete) the store
+ * await store.drop();
+ * ```
+ * @hideconstructor
+ */
+export class KeyValueStore {
+ /**
+ * @param {string} storeId
+ * @param {string} storeName
+ */
+ constructor(storeId: string, storeName: string);
+ storeId: string;
+ storeName: string;
+ /**
+ * Gets a value from the key-value store.
+ *
+ * The function returns a `Promise` that resolves to the record value,
+ * whose JavaScript type depends on the MIME content type of the record.
+ * Records with the `application/json`
+ * content type are automatically parsed and returned as a JavaScript object.
+ * Similarly, records with `text/plain` content types are returned as a string.
+ * For all other content types, the value is returned as a raw
+ * <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a> instance.
+ *
+ * If the record does not exist, the function resolves to `null`.
+ *
+ * To save or delete a value in the key-value store, use the
+ * {@link KeyValueStore#setValue} function.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const store = await Apify.openKeyValueStore('my-screenshots');
+ * const buffer = await store.getValue('screenshot1.png');
+ * ```
+ *
+ * @param {String} key
+ * Unique key of the record. It can be at most 256 characters long and only consist
+ * of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
+ * @returns {Promise<Object|String|Buffer>}
+ * Returns a promise that resolves to an object, string
+ * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, depending
+ * on the MIME content type of the record.
+ */
+ getValue(key: string): Promise<any>;
+ /**
+ * Saves or deletes a record in the key-value store.
+ * The function returns a promise that resolves once the record has been saved or deleted.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const store = await Apify.openKeyValueStore('my-store');
+ * await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });
+ * ```
+ *
+ * Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`
+ *
+ * By default, `value` is converted to JSON and stored with the
+ * `application/json; charset=utf-8` MIME content type.
+ * To store the value with another content type, pass it in the options as follows:
+ * ```javascript
+ * const store = await Apify.openKeyValueStore('my-store');
+ * await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });
+ * ```
+ * If you set custom content type, `value` must be either a string or
+ * <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, otherwise an error will be thrown.
+ *
+ * If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds
+ * regardless whether the record existed or not.
+ *
+ * To retrieve a value from the key-value store, use the
+ * {@link KeyValueStore#getValue} function.
+ *
+ * **IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`,
+ * otherwise the actor process might finish before the value is stored!
+ *
+ * @param {String} key
+ * Unique key of the record. It can be at most 256 characters long and only consist
+ * of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
+ * @param {Object|String|Buffer} value
+ * Record data, which can be one of the following values:
+ * <ul>
+ * <li>If `null`, the record in the key-value store is deleted.</li>
+ * <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
+ * <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`
+ * or <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>.</li>
+ * </ul>
+ * For any other value an error will be thrown.
+ * @param {Object} [options]
+ * @param {String} [options.contentType]
+ * Specifies a custom MIME content type of the record.
+ * @returns {Promise<void>}
+ *
+ */
+ setValue(key: string, value: any, options?: {
+ contentType?: string;
+ }): Promise<void>;
+ /**
+ * Removes the key-value store either from the Apify cloud storage or from the local directory,
+ * depending on the mode of operation.
+ *
+ * @return {Promise<void>}
+ */
+ drop(): Promise<void>;
+ /** @ignore */
+ delete(): Promise<void>;
+ /**
+ * Returns a URL for the given key that may be used to publicly
+ * access the value in the remote key value store.
+ *
+ * @param {string} key
+ * @return {string}
+ */
+ getPublicUrl(key: string): string;
+ /**
+ * Iterates over key value store keys, yielding each in turn to an `iteratee` function.
+ * Each invocation of `iteratee` is called with three arguments: `(key, index, info)`, where `key`
+ * is the record key, `index` is a zero-based index of the key in the current iteration
+ * (regardless of `options.exclusiveStartKey`) and `info` is an object that contains a single property `size`
+ * indicating size of the record in bytes.
+ *
+ * If the `iteratee` function returns a Promise then it is awaited before the next call.
+ * If it throws an error, the iteration is aborted and the `forEachKey` function throws the error.
+ *
+ * **Example usage**
+ * ```javascript
+ * const keyValueStore = await Apify.openKeyValueStore();
+ * await keyValueStore.forEachKey(async (key, index, info) => {
+ * console.log(`Key at ${index}: ${key} has size ${info.size}`);
+ * });
+ * ```
+ *
+ * @param {KeyConsumer} iteratee A function that is called for every key in the key value store.
+ * @param {Object} [options] All `forEachKey()` parameters are passed
+ * via an options object with the following keys:
+ * @param {string} [options.exclusiveStartKey] All keys up to this one (including) are skipped from the result.
+ * @return {Promise<void>}
+ */
+ forEachKey(iteratee: KeyConsumer, options?: {
+ exclusiveStartKey?: string;
+ }, index?: number): Promise<void>;
+}
+export function getFileNameRegexp(key: string): RegExp;
+/**
+ * This is a local representation of a key-value store.
+ *
+ * @ignore
+ */
+export class KeyValueStoreLocal {
+ constructor(storeId: any, localStorageDir: any);
+ localStoragePath: string;
+ storeId: any;
+ initializationPromise: any;
+ getValue(key: any): Promise<any>;
+ setValue(key: any, value: any, options?: {}): Promise<void>;
+ delete(): Promise<void>;
+ drop(): Promise<void>;
+ forEachKey(iteratee: any, options?: {}, index?: number): Promise<void>;
+ /**
+ * Helper function to handle files. Accepts a promisified 'fs' function as a second parameter
+ * which will be executed against the file saved under the key. Since the file's extension and thus
+ * full path is not known, it first performs a check against common extensions. If no file is found,
+ * it will read a full list of files in the directory and attempt to find the file again.
+ *
+ * Returns an object when a file is found and handler executes successfully, null otherwise.
+ *
+ * @param {String} key
+ * @param {Function} handler
+ * @returns {Promise} null or object in the following format:
+ * {
+ * returnValue: return value of the handler function,
+ * fileName: name of the file including found extension
+ * }
+ * @ignore
+ */
+ _handleFile(key: string, handler: Function): Promise<any>;
+ /**
+ * Performs a lookup for a file in the local emulation directory's file list.
+ * @param {String} key
+ * @param {Function} handler
+ * @returns {Promise}
+ * @ignore
+ */
+ _fullDirectoryLookup(key: string, handler: Function): Promise<any>;
+ /**
+ * Helper function to resolve file paths.
+ * @param {String} fileName
+ * @returns {String}
+ * @ignore
+ */
+ _getPath(fileName: string): string;
+ /**
+ * Returns a file:// URL for the given fileName that may be used to
+ * access the value on the local drive.
+ *
+ * Unlike in the remote store where key is sufficient, a full fileName
+ * must be provided here including the extension for the URL to be valid.
+ *
+ * @param {string} fileName
+ * @return {string}
+ * @ignore
+ */
+ getPublicUrl(fileName: string): string;
+}
+export function openKeyValueStore(storeIdOrName?: string, options?: {
+ forceCloud?: boolean;
+}): Promise<KeyValueStore>;
+export function getValue(key: string): Promise<any>;
+export function setValue(key: string, value: any, options?: {
+ contentType?: string;
+}): Promise<any>;
+export function getInput(): Promise<any>;
+/**
+ * User-function used in the [`KeyValueStore.forEachKey()`](../api/keyvaluestore#forEachKey) method.
+ */
+export type KeyConsumer = (key: string, index: number, info: any, size: number) => any;
diff --git a/types/live_view/live_view_server.d.ts b/types/live_view/live_view_server.d.ts
new file mode 100644
index 000000000..173855c64
--- /dev/null
+++ b/types/live_view/live_view_server.d.ts
@@ -0,0 +1,143 @@
+export default LiveViewServer;
+/**
+ * `LiveViewServer` enables serving of browser snapshots via web sockets. It includes its own client
+ * that provides a simple frontend to viewing the captured snapshots. A snapshot consists of three
+ * pieces of information, the currently opened URL, the content of the page (HTML) and its screenshot.
+ *
+ * ```json
+ * {
+ * "pageUrl": "https://www.example.com",
+ * "htmlContent": "<html><body> ....",
+ * "screenshotIndex": 3,
+ * "createdAt": "2019-04-18T11:50:40.060Z"
+ * }
+ * ```
+ *
+ * `LiveViewServer` is useful when you want to be able to inspect the current browser status on demand.
+ * When no client is connected, the webserver consumes very low resources so it should have a close
+ * to zero impact on performance. Only once a client connects the server will start serving snapshots.
+ * Once no longer needed, it can be disabled again in the client to remove any performance impact.
+ *
+ * NOTE: Screenshot taking in browser typically takes around 300ms. So having the `LiveViewServer`
+ * always serve snapshots will have a significant impact on performance.
+ *
+ * When using {@link PuppeteerPool}, the `LiveViewServer` can be
+ * easily used just by providing the `useLiveView = true` option to the {@link PuppeteerPool}.
+ * It can also be initiated via {@link PuppeteerCrawler} `puppeteerPoolOptions`.
+ *
+ * It will take snapshots of the first page of the latest browser. Taking snapshots of only a
+ * single page improves performance and stability dramatically in high concurrency situations.
+ *
+ * When running locally, it is often best to use a headful browser for debugging, since it provides
+ * a better view into the browser, including DevTools, but `LiveViewServer` works too.
+ */
+declare class LiveViewServer {
+ /**
+ * @param {Object} [options]
+ * All `LiveViewServer` parameters are passed
+ * via an options object with the following keys:
+ * @param {string} [options.screenshotDirectoryPath]
+ * By default, the screenshots are saved to
+ * the `live_view` directory in the Apify local storage directory.
+ * Provide a different absolute path to change the settings.
+ * @param {number} [options.maxScreenshotFiles=10]
+ * Limits the number of screenshots stored
+ * by the server. This is to prevent using up too much disk space.
+ * @param {number} [options.snapshotTimeoutSecs=3]
+ * If a snapshot is not made within the timeout,
+ * its creation will be aborted. This is to prevent
+ * pages from being hung up by a stalled screenshot.
+ * @param {number} [options.maxSnapshotFrequencySecs=2]
+ * Use this parameter to further decrease the resource consumption
+ * of `LiveViewServer` by limiting the frequency at which it'll
+ * serve snapshots.
+ */
+ constructor(options?: {
+ screenshotDirectoryPath?: string;
+ maxScreenshotFiles?: number;
+ snapshotTimeoutSecs?: number;
+ maxSnapshotFrequencySecs?: number;
+ });
+ screenshotDirectoryPath: string;
+ maxScreenshotFiles: number;
+ snapshotTimeoutMillis: number;
+ maxSnapshotFrequencyMillis: number;
+ /**
+ * @type {?Snapshot}
+ * @private
+ */
+ lastSnapshot: Snapshot | null;
+ lastScreenshotIndex: number;
+ clientCount: number;
+ _isRunning: boolean;
+ httpServer: http.Server;
+ socketio: socketio.Server;
+ servingSnapshot: boolean;
+ /**
+ * Starts the HTTP server with web socket connections enabled.
+ * Snapshots will not be created until a client has connected.
+ * @return {Promise}
+ */
+ start(): Promise<any>;
+ /**
+ * Prevents the server from receiving more connections. Existing connections
+ * will not be terminated, but the server will not prevent a process exit.
+ * @return {Promise}
+ */
+ stop(): Promise<any>;
+ /**
+ * Serves a snapshot to all connected clients.
+ * Screenshots are not served directly, only their index number
+ * which is used by client to retrieve the screenshot.
+ *
+ * Will time out and throw in `options.snapshotTimeoutSecs`.
+ *
+ * @param {Page} page
+ * @return {Promise}
+ */
+ serve(page: Page): Promise<any>;
+ /**
+ * @return {boolean}
+ */
+ isRunning(): boolean;
+ /**
+ * @return {boolean}
+ */
+ hasClients(): boolean;
+ /**
+ * Returns an absolute path to the screenshot with the given index.
+ * @param {number} screenshotIndex
+ * @return {string}
+ * @private
+ */
+ _getScreenshotPath(screenshotIndex: number): string;
+ /**
+ * @param {Page} page
+ * @return {Promise<Snapshot>}
+ * @private
+ */
+ _makeSnapshot(page: Page): Promise<Snapshot>;
+ /**
+ * @param {Snapshot} snapshot
+ * @private
+ */
+ _pushSnapshot(snapshot: Snapshot): void;
+ /**
+ * Initiates an async delete and does not wait for it to complete.
+ * @param {number} screenshotIndex
+ * @private
+ */
+ _deleteScreenshot(screenshotIndex: number): void;
+ _setupHttpServer(): void;
+ port: number;
+ liveViewUrl: any;
+ /**
+ * @param {socketio.Socket} socket
+ * @private
+ */
+ _socketConnectionHandler(socket: socketio.Socket): void;
+}
+import Snapshot from "./snapshot";
+import http from "http";
+import socketio from "socket.io";
+import { Page } from "puppeteer";
diff --git a/types/live_view/snapshot.d.ts b/types/live_view/snapshot.d.ts
new file mode 100644
index 000000000..d578bfe0d
--- /dev/null
+++ b/types/live_view/snapshot.d.ts
@@ -0,0 +1,29 @@
+/**
+ * Represents a live view snapshot to be served by {@link LiveViewServer}.
+ * @property {string} pageUrl
+ * @property {string} htmlContent
+ * @property {number} screenshotIndex
+ * @property {Date} createdAt
+ * @ignore
+ */
+export default class Snapshot {
+ /**
+ * @param {Object} props
+ * @param {string} props.pageUrl
+ * @param {string} props.htmlContent
+ * @param {number} props.screenshotIndex
+ */
+ constructor(props: {
+ pageUrl: string;
+ htmlContent: string;
+ screenshotIndex: number;
+ });
+ pageUrl: string;
+ htmlContent: string;
+ screenshotIndex: number;
+ createdAt: Date;
+ /**
+ * @return {number}
+ */
+ age(): number;
+}
diff --git a/types/pseudo_url.d.ts b/types/pseudo_url.d.ts
new file mode 100644
index 000000000..a90fbd104
--- /dev/null
+++ b/types/pseudo_url.d.ts
@@ -0,0 +1,84 @@
+export default PseudoUrl;
+/**
+ * Represents a pseudo-URL (PURL) - an URL pattern used by web crawlers
+ * to specify which URLs should the crawler visit.
+ * This class is used by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.
+ *
+ * A PURL is simply a URL with special directives enclosed in `[]` brackets.
+ * Currently, the only supported directive is `[RegExp]`,
+ * which defines a JavaScript-style regular expression to match against the URL.
+ *
+ * The `PseudoUrl` class can be constructed either using a pseudo-URL string
+ * or a regular expression (an instance of the `RegExp` object).
+ * With a pseudo-URL string, the matching is always case-insensitive.
+ * If you need case-sensitive matching, use an appropriate `RegExp` object.
+ *
+ * For example, a PURL `http://www.example.com/pages/[(\w|-)*]` will match all of the following URLs:
+ *
+ * <ul>
+ * <li><code>http://www.example.com/pages/</code></li>
+ * <li><code>http://www.example.com/pages/my-awesome-page</code></li>
+ * <li><code>http://www.example.com/pages/something</code></li>
+ * </ul>
+ *
+ * Be careful to correctly escape special characters in the pseudo-URL string.
+ * If either `[` or `]` is part of the normal query string, it must be encoded as `[\x5B]` or `[\x5D]`,
+ * respectively. For example, the following PURL:
+ * ```http
+ * http://www.example.com/search?do[\x5B]load[\x5D]=1
+ * ```
+ * will match the URL:
+ * ```http
+ * http://www.example.com/search?do[load]=1
+ * ```
+ *
+ * If the regular expression in the pseudo-URL contains a backslash character (\),
+ * you need to escape it with another back backslash, as shown in the example below.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * // Using a pseudo-URL string
+ * const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\\w|-)+]', {
+ * userData: { foo: 'bar' },
+ * });
+ *
+ * // Using a regular expression
+ * const purl2 = new Apify.PseudoUrl(/http:\/\/www\.example\.com\/pages\/(\w|-)+/);
+ *
+ * if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');
+ * ```
+ */
+declare class PseudoUrl {
+ /**
+ * @param {String|RegExp} purl
+ * A pseudo-URL string or a regular expression object.
+ * Using a `RegExp` instance enables more granular control,
+ * such as making the matching case sensitive.
+ * @param {RequestOptions} requestTemplate
+ * Options for the new {@link Request} instances created for matching URLs
+ * by the [`utils.enqueueLinks()`](utils#utils.enqueueLinks) function.
+ */
+ constructor(purl: string | RegExp, requestTemplate?: any);
+ regex: RegExp;
+ requestTemplate: any;
+ /**
+ * Determines whether a URL matches this pseudo-URL pattern.
+ *
+ * @param {String} url URL to be matched.
+ * @return {Boolean} Returns `true` if given URL matches pseudo-URL.
+ */
+ matches(url: string): boolean;
+ /**
+ * Creates a Request object from a provided `requestTemplate` and a given URL
+ * or an object that specifies ${@link Request} properties. In case of a collision
+ * the properties will override the template, except for `userData`, which will
+ * be merged together, with the `userData` property having preference over the template.
+ * This enables dynamic overriding of the template.
+ *
+ * @param {string|Object} urlOrProps
+ * @return {Request}
+ */
+ createRequest(urlOrProps: any): Request;
+}
+import Request from "./request";
diff --git a/types/puppeteer.d.ts b/types/puppeteer.d.ts
new file mode 100644
index 000000000..060a2d4cb
--- /dev/null
+++ b/types/puppeteer.d.ts
@@ -0,0 +1,72 @@
+export function launchPuppeteer(options?: LaunchPuppeteerOptions): Promise<any>;
+/**
+ * Apify extends the launch options of Puppeteer.
+ * You can use any of the
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
+ * options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
+ * function and in addition, all the options available below.
+ */
+export type LaunchPuppeteerOptions = {
+ /**
+ * URL to a HTTP proxy server. It must define the port number,
+ * and it may also contain proxy username and password.
+ *
+ * Example: `http://bob:[email protected]:1234`.
+ */
+ proxyUrl?: string;
+ /**
+ * The `User-Agent` HTTP header used by the browser.
+ * If not provided, the function sets `User-Agent` to a reasonable default
+ * to reduce the chance of detection of the crawler.
+ */
+ userAgent?: string;
+ /**
+ * If `true` and `executablePath` is not set,
+ * Puppeteer will launch full Google Chrome browser available on the machine
+ * rather than the bundled Chromium. The path to Chrome executable
+ * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
+ * or defaults to the typical Google Chrome executable location specific for the operating system.
+ * By default, this option is `false`.
+ */
+ useChrome?: boolean;
+ /**
+ * If set to `true`, Puppeteer will be configured to use
+ * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.
+ * For more information, see the <a href="https://docs.apify.com/proxy" target="_blank">documentation</a>
+ */
+ useApifyProxy?: boolean;
+ /**
+ * An array of proxy groups to be used
+ * by the <a href="https://docs.apify.com/proxy" target="_blank">Apify Proxy</a>.
+ * Only applied if the `useApifyProxy` option is `true`.
+ */
+ apifyProxyGroups?: string[];
+ /**
+ * Apify Proxy session identifier to be used by all the Chrome browsers.
+ * All HTTP requests going through the proxy with the same session identifier
+ * will use the same target proxy server (i.e. the same IP address).
+ * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
+ * Only applied if the `useApifyProxy` option is `true`.
+ */
+ apifyProxySession?: string;
+ /**
+ * Either a require path (`string`) to a package to be used instead of default `puppeteer`,
+ * or an already required module (`Object`). This enables usage of various Puppeteer
+ * wrappers such as `puppeteer-extra`.
+ *
+ * Take caution, because it can cause all kinds of unexpected errors and weird behavior.
+ * Apify SDK is not tested with any other library besides `puppeteer` itself.
+ */
+ puppeteerModule?: any;
+ /**
+ * This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.
+ * It is recommended to use it together with the `useChrome` set to `true`.
+ */
+ stealth?: boolean;
+ /**
+ * Using this configuration, you can disable some of the hiding tricks.
+ * For these settings to take effect `stealth` must be set to true
+ */
+ stealthOptions?: StealthOptions;
+};
+import { StealthOptions } from "./stealth/stealth";
diff --git a/types/puppeteer_pool.d.ts b/types/puppeteer_pool.d.ts
new file mode 100644
index 000000000..6029b2971
--- /dev/null
+++ b/types/puppeteer_pool.d.ts
@@ -0,0 +1,340 @@
+export const BROWSER_SESSION_KEY_NAME: "APIFY_SESSION";
+export default PuppeteerPool;
+export type PuppeteerPoolOptions = {
+ /**
+ * Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
+ * just before a page would be recycled by `PuppeteerPool`. If there are no clients
+ * connected, it has close to zero impact on performance.
+ */
+ useLiveView?: boolean;
+ /**
+ * Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.
+ */
+ maxOpenPagesPerInstance?: number;
+ /**
+ * Maximum number of requests that can be processed by a single browser instance.
+ * After the limit is reached, the browser is retired and new requests are
+ * handled by a new browser instance.
+ */
+ retireInstanceAfterRequestCount?: number;
+ /**
+ * All browser management operations such as launching a new browser, opening a new page
+ * or closing a page will timeout after the set number of seconds and the connected
+ * browser will be retired.
+ */
+ puppeteerOperationTimeoutSecs?: number;
+ /**
+ * Indicates how often are the open Puppeteer instances checked whether they can be closed.
+ */
+ instanceKillerIntervalSecs?: number;
+ /**
+ * When Puppeteer instance reaches the `retireInstanceAfterRequestCount` limit then
+ * it is considered retired and no more tabs will be opened. After the last tab is closed the
+ * whole browser is closed too. This parameter defines a time limit between the last tab was opened and
+ * before the browser is closed even if there are pending open tabs.
+ */
+ killInstanceAfterSecs?: number;
+ /**
+ * Overrides the default function to launch a new Puppeteer instance.
+ * The function must return a promise resolving to
+ * [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.
+ * See the source code on
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
+ * for the default implementation.
+ */
+ launchPuppeteerFunction?: Function;
+ /**
+ * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.
+ * See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
+ */
+ launchPuppeteerOptions?: LaunchPuppeteerOptions;
+ /**
+ * Enables recycling of disk cache directories by Chrome instances.
+ * When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.
+ * This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
+ * Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.
+ *
+ * Beware that the disk cache directories can consume a lot of disk space.
+ * To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `launchPuppeteerargs`,
+ * where `X` is the approximate maximum number of bytes for disk cache.
+ *
+ * Do not use the `recycleDiskCache` setting together with `--disk-cache-dir`
+ * argument in `launchPuppeteerargs`, the behavior is undefined.
+ */
+ recycleDiskCache?: boolean;
+ /**
+ * With this option selected, all pages will be opened in a new incognito browser context, which means
+ * that they will not share cookies or cache and their resources will not be throttled by one another.
+ */
+ useIncognitoPages?: boolean;
+ /**
+ * An array of custom proxy URLs to be used by the `PuppeteerPool` instance.
+ * The provided custom proxies' order will be randomized and the resulting list rotated.
+ * Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ * configuration options will cause an error to be thrown on startup.
+ */
+ proxyUrls?: string[];
+};
+/**
+ * @typedef {Object} PuppeteerPoolOptions
+ * @property {boolean} [useLiveView]
+ * Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
+ * just before a page would be recycled by `PuppeteerPool`. If there are no clients
+ * connected, it has close to zero impact on performance.
+ * @property {number} [maxOpenPagesPerInstance=50]
+ * Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.
+ * @property {number} [retireInstanceAfterRequestCount=100]
+ * Maximum number of requests that can be processed by a single browser instance.
+ * After the limit is reached, the browser is retired and new requests are
+ * handled by a new browser instance.
+ * @property {number} [puppeteerOperationTimeoutSecs=15]
+ * All browser management operations such as launching a new browser, opening a new page
+ * or closing a page will timeout after the set number of seconds and the connected
+ * browser will be retired.
+ * @property {number} [instanceKillerIntervalSecs=60]
+ * Indicates how often are the open Puppeteer instances checked whether they can be closed.
+ * @property {number} [killInstanceAfterSecs=300]
+ * When Puppeteer instance reaches the `retireInstanceAfterRequestCount` limit then
+ * it is considered retired and no more tabs will be opened. After the last tab is closed the
+ * whole browser is closed too. This parameter defines a time limit between the last tab was opened and
+ * before the browser is closed even if there are pending open tabs.
+ * @property {Function} [launchPuppeteerFunction]
+ * Overrides the default function to launch a new Puppeteer instance.
+ * The function must return a promise resolving to
+ * [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.
+ * See the source code on
+ * <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
+ * for the default implementation.
+ * @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]
+ * Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.
+ * See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
+ * @property {boolean} [recycleDiskCache=false]
+ * Enables recycling of disk cache directories by Chrome instances.
+ * When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.
+ * This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
+ * Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.
+ *
+ * Beware that the disk cache directories can consume a lot of disk space.
+ * To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `launchPuppeteerargs`,
+ * where `X` is the approximate maximum number of bytes for disk cache.
+ *
+ * Do not use the `recycleDiskCache` setting together with `--disk-cache-dir`
+ * argument in `launchPuppeteerargs`, the behavior is undefined.
+ * @property {boolean} [useIncognitoPages]
+ * With this option selected, all pages will be opened in a new incognito browser context, which means
+ * that they will not share cookies or cache and their resources will not be throttled by one another.
+ * @property {string[]} [proxyUrls]
+ * An array of custom proxy URLs to be used by the `PuppeteerPool` instance.
+ * The provided custom proxies' order will be randomized and the resulting list rotated.
+ * Custom proxies are not compatible with Apify Proxy and an attempt to use both
+ * configuration options will cause an error to be thrown on startup.
+ */
+/**
+ * Manages a pool of Chrome browser instances controlled using
+ * <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
+ *
+ * `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.
+ * This is useful in order to facilitate rotation of proxies, cookies
+ * or other settings in order to prevent detection of your web scraping bot,
+ * access web pages from various countries etc.
+ *
+ * Additionally, the reuse of browser instances instances speeds up crawling,
+ * and the retirement of instances helps mitigate effects of memory leaks in Chrome.
+ *
+ * `PuppeteerPool` is internally used by the {@link PuppeteerCrawler} class.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const puppeteerPool = new PuppeteerPool({
+ * launchPuppeteerFunction: () => {
+ * // Use a new proxy with a new IP address for each new Chrome instance
+ * return Apify.launchPuppeteer({
+ * useApifyProxy: true,
+ * apifyProxySession: Math.random(),
+ * });
+ * },
+ * });
+ *
+ * const page1 = await puppeteerPool.newPage();
+ * const page2 = await puppeteerPool.newPage();
+ * const page3 = await puppeteerPool.newPage();
+ *
+ * // ... do something with the pages ...
+ *
+ * // Close all browsers.
+ * await puppeteerPool.destroy();
+ * ```
+ */
+declare class PuppeteerPool {
+ /**
+ * @param {PuppeteerPoolOptions} [options]
+ * All `PuppeteerPool` parameters are passed
+ * via an options object with the following keys:
+ */
+ constructor(options?: PuppeteerPoolOptions);
+ sessionPool: any;
+ reusePages: boolean;
+ maxOpenPagesPerInstance: any;
+ retireInstanceAfterRequestCount: any;
+ puppeteerOperationTimeoutMillis: number;
+ killInstanceAfterMillis: any;
+ recycledDiskCacheDirs: any;
+ useIncognitoPages: any;
+ proxyUrls: any;
+ liveViewServer: LiveViewServer;
+ launchPuppeteerFunction: () => Promise<any>;
+ browserCounter: number;
+ activeInstances: {};
+ retiredInstances: {};
+ lastUsedProxyUrlIndex: number;
+ instanceKillerInterval: NodeJS.Timeout;
+ idlePages: any[];
+ closedPages: WeakSet<object>;
+ pagesToInstancesMap: WeakMap<object, any>;
+ liveViewSnapshotsInProgress: WeakMap<object, any>;
+ sigintListener: () => void;
+ _retireBrowserWithSession(session: any): Promise<void>;
+ /**
+ * Launches new browser instance.
+ *
+ * @ignore
+ */
+ _launchInstance(): PuppeteerInstance;
+ /**
+ * Takes care of async processes in PuppeteerInstance construction with a Browser.
+ * @param {Promise<Browser>} browserPromise
+ * @param {PuppeteerInstance} instance
+ * @returns {Promise<void>}
+ * @ignore
+ */
+ _initBrowser(browserPromise: Promise<Browser>, instance: PuppeteerInstance): Promise<void>;
+ /**
+ * Retires some of the instances for example due to many uses.
+ *
+ * @ignore
+ */
+ _retireInstance(instance: any): any;
+ /**
+ * Kills all the retired instances that:
+ * - have all tabs closed
+ * - or are inactive for more then killInstanceAfterMillis.
+ *
+ * @ignore
+ */
+ _killRetiredInstances(): void;
+ /**
+ * Kills given browser instance.
+ *
+ * @ignore
+ */
+ _killInstance(instance: any): Promise<void>;
+ /**
+ * Kills all running PuppeteerInstances.
+ * @ignore
+ */
+ _killAllInstances(): void;
+ /**
+ * Updates the instance metadata when a new page is opened.
+ *
+ * @param {PuppeteerInstance} instance
+ * @ignore
+ */
+ _incrementPageCount(instance: PuppeteerInstance): void;
+ /**
+ * Produces a new page instance either by reusing an idle page that currently isn't processing
+ * any request or by spawning a new page (new browser tab) in one of the available
+ * browsers when no idle pages are available.
+ *
+ * To spawn a new browser tab for each page, set the `reusePages` constructor option to false.
+ *
+ * @return {Promise<Page>}
+ */
+ newPage(): Promise<Page>;
+ /**
+ * Opens new tab in one of the browsers in the pool and returns a `Promise`
+ * that resolves to an instance of a Puppeteer
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>.
+ *
+ * @return {Promise<Page>}
+ * @ignore
+ */
+ _openNewTab(): Promise<Page>;
+ /**
+ * Adds the necessary boilerplate to allow page reuse and also
+ * captures page.close() errors to prevent meaningless log clutter.
+ * @param {Page} page
+ * @ignore
+ */
+ _decoratePage(page: Page): Page;
+ /**
+ * Tells Chromium to focus oldest tab. This is to work around Chromium
+ * throttling CPU and network in inactive tabs.
+ *
+ * @param {Browser} browser
+ * @ignore
+ */
+ _focusOldestTab(browser: Browser): Promise<void>;
+ /**
+ * Closes all open browsers.
+ * @return {Promise<void>}
+ */
+ destroy(): Promise<void>;
+ /**
+ * Finds a PuppeteerInstance given a Puppeteer Browser running in the instance.
+ * @param {Browser} browser
+ * @return {Promise}
+ * @ignore
+ */
+ _findInstanceByBrowser(browser: Browser): Promise<any>;
+ /**
+ * Manually retires a Puppeteer
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-browser" target="_blank"><code>Browser</code></a>
+ * instance from the pool. The browser will continue to process open pages so that they may gracefully finish.
+ * This is unlike `browser.close()` which will forcibly terminate the browser and all open pages will be closed.
+ * @param {Browser} browser
+ * @return {Promise<void>}
+ */
+ retire(browser: Browser): Promise<void>;
+ /**
+ * Closes the page, unless the `reuseTabs` option is set to true.
+ * Then it would only flag the page for a future reuse, without actually closing it.
+ *
+ * NOTE: LiveView snapshotting is tied to this function. When `useLiveView` option
+ * is set to true, a snapshot of the page will be taken just before closing the page
+ * or flagging it for reuse.
+ *
+ * @param {Page} page
+ * @return {Promise<void>}
+ */
+ recyclePage(page: Page): Promise<void>;
+ /**
+ * Tells the connected LiveViewServer to serve a snapshot when available.
+ *
+ * @param page
+ * @return {Promise<void>}
+ */
+ serveLiveViewSnapshot(page: any): Promise<void>;
+ _findInstancesBySession(session: any): any[];
+ _killInstanceWithNoPages(instance: any): void;
+}
+import { LaunchPuppeteerOptions } from "./puppeteer";
+import LiveViewServer from "./live_view/live_view_server";
+/**
+ * Internal representation of Puppeteer instance.
+ *
+ * @ignore
+ */
+declare class PuppeteerInstance {
+ constructor(id: any, browserPromise: any);
+ id: any;
+ activePages: number;
+ totalPages: number;
+ browserPromise: any;
+ lastPageOpenedAt: number;
+ killed: boolean;
+ childProcess: any;
+ recycleDiskCacheDir: any;
+}
+import { Browser } from "puppeteer";
+import { Page } from "puppeteer";
diff --git a/types/puppeteer_request_interception.d.ts b/types/puppeteer_request_interception.d.ts
new file mode 100644
index 000000000..bc97d3ca6
--- /dev/null
+++ b/types/puppeteer_request_interception.d.ts
@@ -0,0 +1,2 @@
+export function addInterceptRequestHandler(page: any, handler: Function): Promise<any>;
+export function removeInterceptRequestHandler(page: any, handler: Function): Promise<any>;
diff --git a/types/puppeteer_utils.d.ts b/types/puppeteer_utils.d.ts
new file mode 100644
index 000000000..700e68738
--- /dev/null
+++ b/types/puppeteer_utils.d.ts
@@ -0,0 +1,51 @@
+export function gotoExtended(page: Page, request: Request, gotoOptions?: any): Promise<Response>;
+export function infiniteScroll(page: any, options?: {
+ timeoutSecs?: number;
+ waitForSecs?: number;
+}): Promise<any>;
+export namespace puppeteerUtils {
+ export { hideWebDriver };
+ export { injectFile };
+ export { injectJQuery };
+ export { injectUnderscore };
+ export { enqueueRequestsFromClickableElements };
+ export function enqueueLinks(...args: any[]): Promise<import("./request_queue").QueueOperationInfo[]>;
+ export { enqueueLinksByClickingElements };
+ export { blockRequests };
+ export { blockResources };
+ export { cacheResponses };
+ export { compileScript };
+ export { gotoExtended };
+ export { addInterceptRequestHandler };
+ export { removeInterceptRequestHandler };
+ export { infiniteScroll };
+ export { saveSnapshot };
+}
+import { Page } from "puppeteer";
+import Request from "./request";
+import { Response } from "puppeteer";
+declare function hideWebDriver(page: Page): Promise<any>;
+declare function injectFile(page: Page, filePath: string, options?: {
+ surviveNavigations?: boolean;
+}): Promise<any>;
+declare function injectJQuery(page: Page): Promise<any>;
+declare function injectUnderscore(page: Page): Promise<any>;
+declare function enqueueRequestsFromClickableElements(page: any, selector: any, purls: any, requestQueue: any, requestOpts?: {}): Promise<any[]>;
+import { enqueueLinksByClickingElements } from "./enqueue_links/click_elements";
+declare function blockRequests(page: Page, options?: {
+ urlPatterns?: string[];
+ extraUrlPatterns?: string[];
+}): Promise<any>;
+declare function blockResources(page: any, resourceTypes?: string[]): Promise<void>;
+declare function cacheResponses(page: Page, cache: any, responseUrlRules: (string | RegExp)[]): Promise<any>;
+declare function compileScript(scriptString: string, context?: any): Function;
+import { addInterceptRequestHandler } from "./puppeteer_request_interception";
+import { removeInterceptRequestHandler } from "./puppeteer_request_interception";
+declare function saveSnapshot(page: any, options?: {
+ key?: string;
+ screenshotQuality?: number;
+ saveScreenshot?: boolean;
+ saveHtml?: boolean;
+ keyValueStoreName?: string;
+}): Promise<any>;
+export {};
diff --git a/types/request.d.ts b/types/request.d.ts
new file mode 100644
index 000000000..3ea5ba439
--- /dev/null
+++ b/types/request.d.ts
@@ -0,0 +1,218 @@
+export function computeUniqueKey({ url, method, payload, keepUrlFragment, useExtendedUniqueKey }: {
+ url: any;
+ method: any;
+ payload: any;
+ keepUrlFragment: any;
+ useExtendedUniqueKey: any;
+}): any;
+export function hashPayload(payload: any): string;
+export default Request;
+/**
+ * Specifies required and optional fields for constructing a [`Request`](../api/request).
+ */
+export type RequestOptions = {
+ /**
+ * URL of the web page to crawl. It must be a non-empty string.
+ */
+ url: string;
+ /**
+ * A unique key identifying the request.
+ * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
+ *
+ * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
+ * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
+ * of `http://www.example.com/something`.
+ *
+ * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
+ *
+ * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
+ * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
+ * when requests point to the same URL, but with different methods and payloads. For example: form submits.
+ *
+ * Pass an arbitrary non-empty text value to the `uniqueKey` property
+ * to override the default behavior and specify which URLs shall be considered equal.
+ */
+ uniqueKey?: string;
+ method?: string;
+ /**
+ * HTTP request payload, e.g. for POST requests.
+ */
+ payload?: string | Buffer;
+ /**
+ * HTTP headers in the following format:
+ * ```
+ * {
+ * Accept: 'text/html',
+ * 'Content-Type': 'application/json'
+ * }
+ * ```
+ */
+ headers?: any;
+ /**
+ * Custom user data assigned to the request. Use this to save any request related data to the
+ * request's scope, keeping them accessible on retries, failures etc.
+ */
+ userData?: any;
+ /**
+ * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
+ * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
+ * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
+ * Note that this option only has an effect if `uniqueKey` is not set.
+ */
+ keepUrlFragment?: boolean;
+ /**
+ * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
+ * properties. This is useful when making requests to the same URL that are differentiated by method
+ * or payload, such as form submit navigations in browsers.
+ */
+ useExtendedUniqueKey?: boolean;
+};
+/**
+ * Specifies required and optional fields for constructing a [`Request`](../api/request).
+ *
+ * @typedef {Object} RequestOptions
+ * @property {String} url URL of the web page to crawl. It must be a non-empty string.
+ * @property {String} [uniqueKey] A unique key identifying the request.
+ * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
+ *
+ * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
+ * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
+ * of `http://www.example.com/something`.
+ *
+ * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
+ *
+ * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
+ * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
+ * when requests point to the same URL, but with different methods and payloads. For example: form submits.
+ *
+ * Pass an arbitrary non-empty text value to the `uniqueKey` property
+ * to override the default behavior and specify which URLs shall be considered equal.
+ * @property {String} [method='GET']
+ * @property {String|Buffer} [payload]
+ * HTTP request payload, e.g. for POST requests.
+ * @property {Object} [headers={}]
+ * HTTP headers in the following format:
+ * ```
+ * {
+ * Accept: 'text/html',
+ * 'Content-Type': 'application/json'
+ * }
+ * ```
+ * @property {Object} [userData={}]
+ * Custom user data assigned to the request. Use this to save any request related data to the
+ * request's scope, keeping them accessible on retries, failures etc.
+ * @property {Boolean} [keepUrlFragment=false]
+ * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
+ * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
+ * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
+ * Note that this option only has an effect if `uniqueKey` is not set.
+ * @property {Boolean} [useExtendedUniqueKey=false]
+ * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
+ * properties. This is useful when making requests to the same URL that are differentiated by method
+ * or payload, such as form submit navigations in browsers.
+ */
+/**
+ * Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
+ * The `Request` object also stores information about errors that occurred during processing of the request.
+ *
+ * Each `Request` instance has the `uniqueKey` property, which can be either specified
+ * manually in the constructor or generated automatically from the URL. Two requests with the same `uniqueKey`
+ * are considered as pointing to the same web resource. This behavior applies to all Apify SDK classes,
+ * such as {@link RequestList}, {@link RequestQueue} or {@link PuppeteerCrawler}.
+ *
+ * Example use:
+ *
+ * ```javascript
+ * const request = new Apify.Request({
+ * url: 'http://www.example.com',
+ * headers: { Accept: 'application/json' },
+ * });
+ *
+ * ...
+ *
+ * request.userData.foo = 'bar';
+ * request.pushErrorMessage(new Error('Request failed!'));
+ *
+ * ...
+ *
+ * const foo = request.userData.foo;
+ * ```
+ * @param {RequestOptions} options `Request` parameters including the URL, HTTP method and headers, and others.
+ *
+ * @property {String} id
+ * Request ID
+ * @property {String} url
+ * URL of the web page to crawl.
+ * @property {String} loadedUrl
+ * An actually loaded URL after redirects, if present. HTTP redirects are guaranteed
+ * to be included.
+ *
+ * When using {@link PuppeteerCrawler}, meta tag and JavaScript redirects may,
+ * or may not be included, depending on their nature. This generally means that redirects,
+ * which happen immediately will most likely be included, but delayed redirects will not.
+ * @property {String} uniqueKey
+ * A unique key identifying the request.
+ * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
+ * @property {String} method
+ * HTTP method, e.g. `GET` or `POST`.
+ * @property {String|Buffer} payload
+ * HTTP request payload, e.g. for POST requests.
+ * @property {Boolean} noRetry
+ * The `true` value indicates that the request will not be automatically retried on error.
+ * @property {Number} retryCount
+ * Indicates the number of times the crawling of the request has been retried on error.
+ * @property {String[]} errorMessages
+ * An array of error messages from request processing.
+ * @property {Object} headers
+ * Object with HTTP headers. Key is header name, value is the value.
+ * @property {Object} userData
+ * Custom user data assigned to the request.
+ * @property {Date} handledAt
+ * Indicates the time when the request has been processed.
+ * Is `null` if the request has not been crawled yet.
+ */
+declare class Request {
+ constructor(options?: {});
+ id: any;
+ url: any;
+ loadedUrl: any;
+ uniqueKey: any;
+ method: any;
+ payload: any;
+ noRetry: any;
+ retryCount: any;
+ errorMessages: any;
+ headers: any;
+ userData: any;
+ handledAt: any;
+ /**
+ * Stores information about an error that occurred during processing of this request.
+ *
+ * You should always use Error instances when throwing errors in JavaScript.
+ *
+ * Nevertheless, to improve the debugging experience when using third party libraries
+ * that may not always throw an Error instance, the function performs a type
+ * inspection of the passed argument and attempts to extract as much information
+ * as possible, since just throwing a bad type error makes any debugging rather difficult.
+ *
+ * @param {Error|String} errorOrMessage Error object or error message to be stored in the request.
+ * @param {Object} [options]
+ * @param {Boolean} [options.omitStack=false] Only push the error message without stack trace when true.
+ */
+ pushErrorMessage(errorOrMessage: string | Error, options?: {
+ omitStack?: boolean;
+ }): void;
+ /**
+ * Flags the request with no retry which prevents {@link BasicCrawler}
+ * (as well as {@PuppeteerCrawler} and {@CheerioCrawler}, since they use {@BasicCrawler} internally)
+ * from retrying the request after an error occurs.
+ *
+ * Optionally accepts a message that will be used to construct
+ * and throw an Error.
+ *
+ * @param {String} [message]
+ * @deprecated 2019/06/26
+ * @ignore
+ */
+ doNotRetry(message?: string): void;
+}
diff --git a/types/request_list.d.ts b/types/request_list.d.ts
new file mode 100644
index 000000000..ae971dfa5
--- /dev/null
+++ b/types/request_list.d.ts
@@ -0,0 +1,404 @@
+export const STATE_PERSISTENCE_KEY: "REQUEST_LIST_STATE";
+export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";
+/**
+ * @typedef {Object} RequestListOptions
+ * @property {Array<RequestOptions|Request>} sources
+ * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
+ * define the `url` property, or an array of instances of the {@link Request} class.
+ * Additionally, the `requestsFromUrl` property may be used instead of `url`,
+ * which will instruct `RequestList` to download the source URLs from a given remote location.
+ * The URLs will be parsed from the received response.
+ *
+ * ```
+ * [
+ * // One URL
+ * { method: 'GET', url: 'http://example.com/a/b' },
+ * // Batch import of URLs from a file hosted on the web
+ * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
+ * // Batch import combined with regex.
+ * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ * ]
+ * ```
+ * @property {String} [persistStateKey]
+ * Identifies the key in the default key-value store under which `RequestList` periodically stores its
+ * state (i.e. which URLs were crawled and which not).
+ * If the actor is restarted, `RequestList` will read the state
+ * and continue where it left off.
+ *
+ * If `persistStateKey` is not set, `RequestList` will always start from the beginning,
+ * and all the source URLs will be crawled again.
+ * @property {String} [persistSourcesKey]
+ * Identifies the key in the default key-value store under which the `RequestList` persists its
+ * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.
+ * This is necessary if `persistStateKey` is set and the source URLs might potentially change,
+ * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.
+ *
+ * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources
+ * from their origin, check that they are consistent with the restored state (if any)
+ * and throw an error if they are not.
+ * @property {RequestListState} [state]
+ * The state object that the `RequestList` will be initialized from.
+ * It is in the form as returned by `RequestList.getState()`, such as follows:
+ *
+ * ```
+ * {
+ * nextIndex: 5,
+ * nextUniqueKey: 'unique-key-5'
+ * inProgress: {
+ * 'unique-key-1': true,
+ * 'unique-key-4': true,
+ * },
+ * }
+ * ```
+ *
+ * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
+ * is to use the `stateKeyPrefix` parameter instead.
+ * @property {Boolean} [keepDuplicateUrls=false]
+ * By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
+ * on the `uniqueKey` property of passed source {@link Request} objects.
+ *
+ * If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
+ * In any case, only one request per `uniqueKey` is added to the `RequestList` resulting in removal
+ * of duplicate URLs / unique keys.
+ *
+ * Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey`
+ * of each request that does not already include a `uniqueKey`. Therefore, duplicate
+ * URLs will be kept in the list. It does not protect the user from having duplicates in user set
+ * `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys
+ * if they wish to keep more than just a single copy in the `RequestList`.
+ */
+/**
+ * Represents a static list of URLs to crawl.
+ * The URLs can be provided either in code or parsed from a text file hosted on the web.
+ *
+ * Each URL is represented using an instance of the {@link Request} class.
+ * The list can only contain unique URLs. More precisely, it can only contain `Request` instances
+ * with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden.
+ * To add a single URL to the list multiple times, corresponding {@link Request} objects will need to have different
+ * `uniqueKey` properties. You can use the `keepDuplicateUrls` option to do this for you when initializing the
+ * `RequestList` from sources.
+ *
+ * Once you create an instance of `RequestList`, you need to call the {@link RequestList#initialize} function
+ * before the instance can be used. After that, no more URLs can be added to the list.
+ *
+ * `RequestList` is used by {@link BasicCrawler}, {@link CheerioCrawler}
+ * and {@link PuppeteerCrawler} as a source of URLs to crawl.
+ * Unlike {@link RequestQueue}, `RequestList` is static but it can contain even millions of URLs.
+ *
+ * `RequestList` has an internal state where it stores information about which requests were already handled,
+ * which are in progress and which were reclaimed. The state may be automatically persisted to the default
+ * {@link KeyValueStore} by setting the `persistStateKey` option so that if the Node.js process is restarted,
+ * the crawling can continue where it left off. The automated persisting is launched upon receiving the `persistState`
+ * event that is periodically emitted by {@link events|Apify.events}.
+ *
+ * The internal state is closely tied to the provided sources (URLs). If the sources change on actor restart, the state will become corrupted and
+ * `RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web.
+ * In such case, use the `persistSourcesKey` option in conjunction with `persistStateKey`,
+ * to make the `RequestList` store the initial sources to the default key-value store and load them after restart,
+ * which will prevent any issues that a live list of URLs might cause.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const requestList = new Apify.RequestList({
+ * sources: [
+ * // Separate requests
+ * { url: 'http://www.example.com/page-1', method: 'GET', headers: {} },
+ * { url: 'http://www.example.com/page-2', userData: { foo: 'bar' }},
+ *
+ * // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`
+ * // Note that all URLs must start with http:// or https://
+ * { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },
+ * ],
+ *
+ * // Ensure both the sources and crawling state of the request list is persisted,
+ * // so that on actor restart, the crawling will continue where it left off
+ * persistStateKey: 'my-state',
+ * persistSourcesKey: 'my-sources'
+ * });
+ *
+ * // This call loads and parses the URLs from the remote file.
+ * await requestList.initialize();
+ *
+ * // Get requests from list
+ * const request1 = await requestList.fetchNextRequest();
+ * const request2 = await requestList.fetchNextRequest();
+ * const request3 = await requestList.fetchNextRequest();
+ *
+ * // Mark some of them as handled
+ * await requestList.markRequestHandled(request1);
+ *
+ * // If processing fails then reclaim it back to the list
+ * await requestList.reclaimRequest(request2);
+ * ```
+ */
+export class RequestList {
+ /**
+ * @param {RequestListOptions} options All `RequestList` configuration options
+ */
+ constructor(options?: RequestListOptions);
+ requests: any[];
+ nextIndex: number;
+ uniqueKeyToIndex: {};
+ inProgress: {};
+ reclaimed: {};
+ persistStateKey: string;
+ persistSourcesKey: string;
+ initialState: RequestListState;
+ keepDuplicateUrls: boolean;
+ isStatePersisted: boolean;
+ areSourcesPersisted: boolean;
+ isLoading: boolean;
+ isInitialized: boolean;
+ sources: any;
+ /**
+ * Loads all remote sources of URLs and potentially starts periodic state persistence.
+ * This function must be called before you can start using the instance in a meaningful way.
+ *
+ * @returns {Promise<void>}
+ */
+ initialize(): Promise<void>;
+ /**
+ * Persists the current state of the `RequestList` into the default {@link KeyValueStore}.
+ * The state is persisted automatically in regular intervals, but calling this method manually
+ * is useful in cases where you want to have the most current state available after you pause
+ * or stop fetching its requests. For example after you pause or abort a crawl. Or just before
+ * a server migration.
+ *
+ * @return {Promise<void>}
+ */
+ persistState(): Promise<void>;
+ /**
+ * Unlike persistState(), this is used only internally, since the sources
+ * are automatically persisted at RequestList initialization (if the persistSourcesKey is set),
+ * but there's no reason to persist it again afterwards, because RequestList is immutable.
+ *
+ * @return {Promise<void>}
+ * @ignore
+ */
+ _persistSources(): Promise<void>;
+ /**
+ * Restores RequestList state from a state object.
+ *
+ * @param {RequestListState} state
+ * @ignore
+ */
+ _restoreState(state: RequestListState): void;
+ /**
+ * Attempts to load state and sources using the `RequestList` configuration
+ * and returns a tuple of [state, sources] where each may be null if not loaded.
+ *
+ * @return {Promise<Array>}
+ * @ignore
+ */
+ _loadStateAndSources(): Promise<any[]>;
+ /**
+ * Returns an object representing the internal state of the `RequestList` instance.
+ * Note that the object's fields can change in future releases.
+ *
+ * @returns {RequestListState}
+ */
+ getState(): RequestListState;
+ /**
+ * Resolves to `true` if the next call to {@link RequestList#fetchNextRequest} function
+ * would return `null`, otherwise it resolves to `false`.
+ * Note that even if the list is empty, there might be some pending requests currently being processed.
+ *
+ * @returns {Promise<Boolean>}
+ */
+ isEmpty(): Promise<boolean>;
+ /**
+ * Returns `true` if all requests were already handled and there are no more left.
+ *
+ * @returns {Promise<Boolean>}
+ */
+ isFinished(): Promise<boolean>;
+ /**
+ * Gets the next {@link Request} to process. First, the function gets a request previously reclaimed
+ * using the {@link RequestList#reclaimRequest} function, if there is any.
+ * Otherwise it gets the next request from sources.
+ *
+ * The function's `Promise` resolves to `null` if there are no more
+ * requests to process.
+ *
+ * @returns {Promise<Request>}
+ */
+ fetchNextRequest(): Promise<Request>;
+ /**
+ * Marks request as handled after successful processing.
+ *
+ * @param {Request} request
+ *
+ * @returns {Promise<void>}
+ */
+ markRequestHandled(request: Request): Promise<void>;
+ /**
+ * Reclaims request to the list if its processing failed.
+ * The request will become available in the next `this.fetchNextRequest()`.
+ *
+ * @param {Request} request
+ *
+ * @returns {Promise<void>}
+ */
+ reclaimRequest(request: Request): Promise<void>;
+ /**
+ * Adds all fetched requests from a URL from a remote resource.
+ *
+ * @ignore
+ */
+ _addFetchedRequests(source: any, fetchedRequests: any): Promise<void>;
+ /**
+ * Fetches URLs from requestsFromUrl and returns them in format of list of requests
+ * @param source
+ * @return {Promise<RequestOptions[]>}
+ * @ignore
+ */
+ _fetchRequestsFromUrl(source: any): Promise<any[]>;
+ /**
+ * Adds given request.
+ * If the `opts` parameter is a plain object and not an instance of a `Request`, then the function
+ * creates a `Request` instance.
+ *
+ * @ignore
+ */
+ _addRequest(opts: any): void;
+ /**
+ * Helper function that validates unique key.
+ * Throws an error if uniqueKey is not a non-empty string.
+ *
+ * @ignore
+ */
+ _ensureUniqueKeyValid(uniqueKey: any): void;
+ /**
+ * Checks that request is not reclaimed and throws an error if so.
+ *
+ * @ignore
+ */
+ _ensureInProgressAndNotReclaimed(uniqueKey: any): void;
+ /**
+ * Throws an error if request list wasn't initialized.
+ *
+ * @ignore
+ */
+ _ensureIsInitialized(): void;
+ /**
+ * Returns the total number of unique requests present in the `RequestList`.
+ *
+ * @returns {Number}
+ */
+ length(): number;
+ /**
+ * Returns number of handled requests.
+ *
+ * @returns {Number}
+ */
+ handledCount(): number;
+}
+export function openRequestList(listName: string, sources: any[], options?: RequestListOptions): Promise<RequestList>;
+export type RequestListOptions = {
+ /**
+ * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
+ * define the `url` property, or an array of instances of the {@link Request} class.
+ * Additionally, the `requestsFromUrl` property may be used instead of `url`,
+ * which will instruct `RequestList` to download the source URLs from a given remote location.
+ * The URLs will be parsed from the received response.
+ *
+ * ```
+ * [
+ * // One URL
+ * { method: 'GET', url: 'http://example.com/a/b' },
+ * // Batch import of URLs from a file hosted on the web
+ * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
+ * // Batch import combined with regex.
+ * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ * ]
+ * ```
+ */
+ sources: any[];
+ /**
+ * Identifies the key in the default key-value store under which `RequestList` periodically stores its
+ * state (i.e. which URLs were crawled and which not).
+ * If the actor is restarted, `RequestList` will read the state
+ * and continue where it left off.
+ *
+ * If `persistStateKey` is not set, `RequestList` will always start from the beginning,
+ * and all the source URLs will be crawled again.
+ */
+ persistStateKey?: string;
+ /**
+ * Identifies the key in the default key-value store under which the `RequestList` persists its
+ * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.
+ * This is necessary if `persistStateKey` is set and the source URLs might potentially change,
+ * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.
+ *
+ * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources
+ * from their origin, check that they are consistent with the restored state (if any)
+ * and throw an error if they are not.
+ */
+ persistSourcesKey?: string;
+ /**
+ * The state object that the `RequestList` will be initialized from.
+ * It is in the form as returned by `RequestList.getState()`, such as follows:
+ *
+ * ```
+ * {
+ * nextIndex: 5,
+ * nextUniqueKey: 'unique-key-5'
+ * inProgress: {
+ * 'unique-key-1': true,
+ * 'unique-key-4': true,
+ * },
+ * }
+ * ```
+ *
+ * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
+ * is to use the `stateKeyPrefix` parameter instead.
+ */
+ state?: RequestListState;
+ /**
+ * By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
+ * on the `uniqueKey` property of passed source {@link Request} objects.
+ *
+ * If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
+ * In any case, only one request per `uniqueKey` is added to the `RequestList` resulting in removal
+ * of duplicate URLs / unique keys.
+ *
+ * Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey`
+ * of each request that does not already include a `uniqueKey`. Therefore, duplicate
+ * URLs will be kept in the list. It does not protect the user from having duplicates in user set
+ * `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys
+ * if they wish to keep more than just a single copy in the `RequestList`.
+ */
+ keepDuplicateUrls?: boolean;
+};
+/**
+ * Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed.
+ * You can obtain the state by calling [`RequestList.getState()`](../api/requestlist#getState()) and receive an object with
+ * the following structure:
+ *
+ * ```
+ * {
+ * nextIndex: 5,
+ * nextUniqueKey: 'unique-key-5'
+ * inProgress: {
+ * 'unique-key-1': true,
+ * 'unique-key-4': true
+ * },
+ * }
+ * ```
+ */
+export type RequestListState = {
+ /**
+ * Position of the next request to be processed.
+ */
+ nextIndex: number;
+ /**
+ * Key of the next request to be processed.
+ */
+ nextUniqueKey: string;
+ /**
+ * An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.
+ */
+ inProgress: any;
+};
+import Request from "./request";
diff --git a/types/request_queue.d.ts b/types/request_queue.d.ts
new file mode 100644
index 000000000..1a35b7967
--- /dev/null
+++ b/types/request_queue.d.ts
@@ -0,0 +1,355 @@
+export const LOCAL_STORAGE_SUBDIR: any;
+export const QUERY_HEAD_MIN_LENGTH: 100;
+export const QUERY_HEAD_BUFFER: 3;
+export const API_PROCESSED_REQUESTS_DELAY_MILLIS: number;
+export const MAX_QUERIES_FOR_CONSISTENCY: 6;
+export const STORAGE_CONSISTENCY_DELAY_MILLIS: 3000;
+/**
+ * A helper class that is used to report results from various
+ * [`RequestQueue`](../api/requestqueue) functions as well as
+ * [`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).
+ *
+ * @typedef {Object} QueueOperationInfo
+ * @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.
+ * @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.
+ * @property {String} requestId The ID of the added request
+ * @property {Object} request The original [`Request`](../api/request) object passed to the `RequestQueue` function.
+ */
+/**
+ * Represents a queue of URLs to crawl, which is used for deep crawling of websites
+ * where you start with several URLs and then recursively
+ * follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
+ *
+ * Each URL is represented using an instance of the {@link Request} class.
+ * The queue can only contain unique URLs. More precisely, it can only contain {@link Request} instances
+ * with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden.
+ * To add a single URL multiple times to the queue,
+ * corresponding {@link Request} objects will need to have different `uniqueKey` properties.
+ *
+ * Do not instantiate this class directly, use the
+ * [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function instead.
+ *
+ * `RequestQueue` is used by {@link BasicCrawler}, {@link CheerioCrawler}
+ * and {@link PuppeteerCrawler} as a source of URLs to crawl.
+ * Unlike {@link RequestList}, `RequestQueue` supports dynamic adding and removing of requests.
+ * On the other hand, the queue is not optimized for operations that add or remove a large number of URLs in a batch.
+ *
+ * `RequestQueue` stores its data either on local disk or in the Apify Cloud,
+ * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable is set.
+ *
+ * If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the queue data is stored in
+ * that local directory as follows:
+ * ```
+ * {APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/{STATE}/{NUMBER}.json
+ * ```
+ * Note that `{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID: `default`,
+ * unless you override it by setting the `APIFY_DEFAULT_REQUEST_QUEUE_ID` environment variable.
+ * Each request in the queue is stored as a separate JSON file, where `{STATE}` is either `handled` or `pending`,
+ * and `{NUMBER}` is an integer indicating the position of the request in the queue.
+ *
+ * If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the
+ * <a href="https://docs.apify.com/storage/request-queue" target="_blank">Apify Request Queue</a>
+ * cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`
+ * option to [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function,
+ * even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * // Open the default request queue associated with the actor run
+ * const queue = await Apify.openRequestQueue();
+ *
+ * // Open a named request queue
+ * const queueWithName = await Apify.openRequestQueue('some-name');
+ *
+ * // Enqueue few requests
+ * await queue.addRequest({ url: 'http://example.com/aaa' });
+ * await queue.addRequest({ url: 'http://example.com/bbb' });
+ * await queue.addRequest({ url: 'http://example.com/foo/bar' }, { forefront: true });
+ *
+ * // Get requests from queue
+ * const request1 = await queue.fetchNextRequest();
+ * const request2 = await queue.fetchNextRequest();
+ * const request3 = await queue.fetchNextRequest();
+ *
+ * // Mark a request as handled
+ * await queue.markRequestHandled(request1);
+ *
+ * // If processing of a request fails then reclaim it back to the queue, so that it's crawled again
+ * await queue.reclaimRequest(request2);
+ * ```
+ * @hideconstructor
+ */
+export class RequestQueue {
+ constructor(queueId: any, queueName: any, clientKey?: any);
+ clientKey: any;
+ queueId: any;
+ queueName: any;
+ queueHeadDict: any;
+ queryQueueHeadPromise: any;
+ inProgress: Set<any>;
+ recentlyHandled: any;
+ assumedTotalCount: number;
+ assumedHandledCount: number;
+ requestsCache: any;
+ /**
+ * @ignore
+ */
+ inProgressCount(): number;
+ /**
+ * Adds a request to the queue.
+ *
+ * If a request with the same `uniqueKey` property is already present in the queue,
+ * it will not be updated. You can find out whether this happened from the resulting
+ * {@link QueueOperationInfo} object.
+ *
+ * To add multiple requests to the queue by extracting links from a webpage,
+ * see the [`Apify.utils.enqueueLinks()`](utils#utils.enqueueLinks) helper function.
+ *
+ * @param {Request|RequestOptions} request {@link Request} object or vanilla object with request data.
+ * Note that the function sets the `uniqueKey` and `id` fields to the passed object.
+ * @param {Object} [options]
+ * @param {Boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.
+ * @return {Promise<QueueOperationInfo>}
+ */
+ addRequest(request: RequestOptions | Request, options?: {
+ forefront?: boolean;
+ }): Promise<QueueOperationInfo>;
+ /**
+ * Gets the request from the queue specified by ID.
+ *
+ * @param {String} requestId ID of the request.
+ * @return {Promise<Request>} Returns the request object, or `null` if it was not found.
+ */
+ getRequest(requestId: string): Promise<Request>;
+ /**
+ * Returns a next request in the queue to be processed, or `null` if there are no more pending requests.
+ *
+ * Once you successfully finish processing of the request, you need to call
+ * [`requestQueue.markRequestHandled()`](#RequestQueue+markRequestHandled)
+ * to mark the request as handled in the queue. If there was some error in processing the request,
+ * call [`requestQueue.reclaimRequest()`](#RequestQueue+reclaimRequest) instead,
+ * so that the queue will give the request to some other consumer in another call to the `fetchNextRequest` function.
+ *
+ * Note that the `null` return value doesn't mean the queue processing finished,
+ * it means there are currently no pending requests.
+ * To check whether all requests in queue were finished,
+ * use [`requestQueue.isFinished()`](#RequestQueue+isFinished) instead.
+ *
+ * @returns {Promise<Request>}
+ * Returns the request object or `null` if there are no more pending requests.
+ */
+ fetchNextRequest(): Promise<Request>;
+ /**
+ * Marks a request that was previously returned by the
+ * [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)
+ * function as handled after successful processing.
+ * Handled requests will never again be returned by the `fetchNextRequest` function.
+ *
+ * @param {Request} request
+ * @return {Promise<QueueOperationInfo>}
+ */
+ markRequestHandled(request: Request): Promise<QueueOperationInfo>;
+ /**
+ * Reclaims a failed request back to the queue, so that it can be returned for processed later again
+ * by another call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).
+ * The request record in the queue is updated using the provided `request` parameter.
+ * For example, this lets you store the number of retries or error messages for the request.
+ *
+ * @param {Request} request
+ * @param {Object} [options]
+ * @param {Boolean} [options.forefront=false]
+ * If `true` then the request it placed to the beginning of the queue, so that it's returned
+ * in the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).
+ * By default, it's put to the end of the queue.
+ * @return {Promise<QueueOperationInfo>}
+ */
+ reclaimRequest(request: Request, options?: {
+ forefront?: boolean;
+ }): Promise<QueueOperationInfo>;
+ /**
+ * Resolves to `true` if the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)
+ * would return `null`, otherwise it resolves to `false`.
+ * Note that even if the queue is empty, there might be some pending requests currently being processed.
+ * If you need to ensure that there is no activity in the queue, use [`requestQueue.isFinished()`](#RequestQueue+isFinished).
+ *
+ * @returns {Promise<Boolean>}
+ */
+ isEmpty(): Promise<boolean>;
+ /**
+ * Resolves to `true` if all requests were already handled and there are no more left.
+ * Due to the nature of distributed storage used by the queue,
+ * the function might occasionally return a false negative,
+ * but it will never return a false positive.
+ *
+ * @returns {Promise<Boolean>}
+ */
+ isFinished(): Promise<boolean>;
+ /**
+ * Caches information about request to beware of unneeded addRequest() calls.
+ *
+ * @ignore
+ */
+ _cacheRequest(cacheKey: any, queueOperationInfo: any): void;
+ /**
+ * We always request more items than is in progress to ensure that something falls into head.
+ *
+ * @param {Boolean} [ensureConsistency=false] If true then query for queue head is retried until queueModifiedAt
+ * is older than queryStartedAt by at least API_PROCESSED_REQUESTS_DELAY_MILLIS to ensure that queue
+ * head is consistent.
+ * @param {Number} [limit] How many queue head items will be fetched.
+ * @param {Number} [iteration] Used when this function is called recursively to limit the recursion.
+ * @return {Boolean} Indicates if queue head is consistent (true) or inconsistent (false).
+ * @ignore
+ */
+ _ensureHeadIsNonEmpty(ensureConsistency?: boolean, limit?: number, iteration?: number): boolean;
+ /**
+ * Adds a request straight to the queueHeadDict, to improve performance.
+ * @private
+ */
+ _maybeAddRequestToQueueHead(requestId: any, forefront: any): void;
+ /**
+ * Removes the queue either from the Apify Cloud storage or from the local directory,
+ * depending on the mode of operation.
+ *
+ * @return {Promise}
+ */
+ drop(): Promise<any>;
+ /** @ignore */
+ delete(): Promise<void>;
+ /**
+ * Returns the number of handled requests.
+ *
+ * This function is just a convenient shortcut for:
+ *
+ * ```javascript
+ * const { handledRequestCount } = await queue.getInfo();
+ * ```
+ *
+ * @return {Promise<number>}
+ */
+ handledCount(): Promise<number>;
+ /**
+ * Returns an object containing general information about the request queue.
+ *
+ * The function returns the same object as the Apify API Client's
+ * [getQueue](https://docs.apify.com/api/apify-client-js/latest#ApifyClient-requestQueues)
+ * function, which in turn calls the
+ * [Get request queue](https://apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue)
+ * API endpoint.
+ *
+ * **Example:**
+ * ```
+ * {
+ * id: "WkzbQMuFYuamGv3YF",
+ * name: "my-queue",
+ * userId: "wRsJZtadYvn4mBZmm",
+ * createdAt: new Date("2015-12-12T07:34:14.202Z"),
+ * modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
+ * accessedAt: new Date("2015-12-14T08:36:13.202Z"),
+ * totalRequestCount: 25,
+ * handledRequestCount: 5,
+ * pendingRequestCount: 20,
+ * }
+ * ```
+ *
+ * @returns {Promise<Object>}
+ */
+ getInfo(): Promise<any>;
+}
+/**
+ * Local directory-based implementation of the `RequestQueue` class.
+ * TODO: We should implement this class using the RequestQueueRemote, just replace
+ * the underlying API calls with their emulation on filesystem. That will bring
+ * all the goodies such as caching and will enable better and more consistent testing
+ *
+ * @ignore
+ */
+export class RequestQueueLocal {
+ constructor(queueId: any, localStorageDir: any);
+ queueId: any;
+ localStoragePath: string;
+ localHandledEmulationPath: string;
+ localPendingEmulationPath: string;
+ queueOrderNoCounter: number;
+ pendingCount: number;
+ _handledCount: number;
+ inProgressCount: number;
+ requestIdToQueueOrderNo: {};
+ queueOrderNoInProgress: {};
+ requestsBeingWrittenToFile: Map<any, any>;
+ createdAt: Date;
+ modifiedAt: Date;
+ accessedAt: Date;
+ initializationPromise: Promise<void>;
+ _initialize(): Promise<void>;
+ _saveRequestIdToQueueOrderNo(filepath: any): Promise<void>;
+ _getFilePath(queueOrderNo: any, isHandled?: boolean): string;
+ _getQueueOrderNo(forefront?: boolean): number;
+ _getRequestByQueueOrderNo(queueOrderNo: any): Promise<Request>;
+ addRequest(request: any, opts?: {}): Promise<{
+ requestId: any;
+ wasAlreadyHandled: any;
+ wasAlreadyPresent: boolean;
+ request: any;
+ }>;
+ getRequest(requestId: any): Promise<any>;
+ fetchNextRequest(): Promise<Request>;
+ markRequestHandled(request: any): Promise<{
+ requestId: any;
+ wasAlreadyHandled: boolean;
+ wasAlreadyPresent: boolean;
+ request: any;
+ }>;
+ reclaimRequest(request: any, opts?: {}): Promise<{
+ requestId: any;
+ wasAlreadyHandled: boolean;
+ wasAlreadyPresent: boolean;
+ request: any;
+ }>;
+ isEmpty(): Promise<boolean>;
+ isFinished(): Promise<boolean>;
+ drop(): Promise<void>;
+ delete(): Promise<void>;
+ handledCount(): Promise<number>;
+ getInfo(): Promise<{
+ id: any;
+ name: any;
+ userId: string;
+ createdAt: Date;
+ modifiedAt: Date;
+ accessedAt: Date;
+ totalRequestCount: number;
+ handledRequestCount: number;
+ pendingRequestCount: number;
+ }>;
+ _updateMetadata(isModified: any): void;
+}
+export function openRequestQueue(queueIdOrName?: string, options?: {
+ forceCloud?: boolean;
+}): Promise<RequestQueue>;
+/**
+ * A helper class that is used to report results from various
+ * [`RequestQueue`](../api/requestqueue) functions as well as
+ * [`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).
+ */
+export type QueueOperationInfo = {
+ /**
+ * Indicates if request was already present in the queue.
+ */
+ wasAlreadyPresent: boolean;
+ /**
+ * Indicates if request was already marked as handled.
+ */
+ wasAlreadyHandled: boolean;
+ /**
+ * The ID of the added request
+ */
+ requestId: string;
+ /**
+ * The original [`Request`](../api/request) object passed to the `RequestQueue` function.
+ */
+ request: any;
+};
+import { RequestOptions } from "./request";
+import Request from "./request";
diff --git a/types/session_pool/errors.d.ts b/types/session_pool/errors.d.ts
new file mode 100644
index 000000000..fd7363e81
--- /dev/null
+++ b/types/session_pool/errors.d.ts
@@ -0,0 +1,4 @@
+export class CookieParseError extends Error {
+ constructor(cookieHeaderString: any);
+ cookieHeaderString: any;
+}
diff --git a/types/session_pool/events.d.ts b/types/session_pool/events.d.ts
new file mode 100644
index 000000000..5cf171881
--- /dev/null
+++ b/types/session_pool/events.d.ts
@@ -0,0 +1,4 @@
+declare namespace _default {
+ export const SESSION_RETIRED: string;
+}
+export default _default;
diff --git a/types/session_pool/session.d.ts b/types/session_pool/session.d.ts
new file mode 100644
index 000000000..e27b47b6e
--- /dev/null
+++ b/types/session_pool/session.d.ts
@@ -0,0 +1,240 @@
+/**
+ * Persistable {Session} state.
+ * @typedef {Object} SessionState
+ * @property {String} id
+ * @property {Object} cookies
+ * @property {Object} cookieJar
+ * @property {Object} userData
+ * @property {Number} maxErrorScore
+ * @property {Number} errorScoreDecrement
+ * @property {String} expiresAt
+ * @property {String} createdAt
+ * @property {Number} usageCount
+ * @property {Number} errorScore
+ */
+/**
+ * @typedef {Object} SessionOptions
+ * @property {String} [id] - Id of session used for generating fingerprints. It is used as proxy session name.
+ * @property {Number} [maxAgeSecs=3000] - Number of seconds after which the session is considered as expired.
+ * @property {Object} userData - Object where custom user data can be stored. For example custom headers.
+ * @property {number} [maxErrorScore=3] - Maximum number of marking session as blocked usage.
+ * If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
+ * It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
+ * Calling the `markGood` will decrease the `errorScore` by `errorScoreDecrement`
+ * @property {number} [errorScoreDecrement=0.5] - It is used for healing the session.
+ * For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this
+ * number.
+ * @property {Date} [createdAt] - Date of creation.
+ * @property {Date} [expiresAt] - Date of expiration.
+ * @property {Number} [usageCount=0] - Indicates how many times the session has been used.
+ * @property {Number} [errorCount=0] - Indicates how many times the session is marked bad.
+ * @property {Number} [maxUsageCount=50] - Session should be used only a limited amount of times.
+ * This number indicates how many times the session is going to be used, before it is thrown away.
+ * @property {SessionPool} sessionPool - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
+ */
+/**
+ * Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions.
+ * You can imagine each session as a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint.
+ * Session internal state can be enriched with custom user data for example some authorization tokens and specific headers in general.
+ */
+export class Session {
+ /**
+ * Session configuration.
+ * @param {SessionOptions} options
+ */
+ constructor(options?: SessionOptions);
+ id: string;
+ cookies: any;
+ /** @type CookieJar */
+ cookieJar: CookieJar;
+ maxAgeSecs: number;
+ userData: any;
+ maxErrorScore: number;
+ errorScoreDecrement: number;
+ expiresAt: Date;
+ createdAt: Date;
+ usageCount: number;
+ errorScore: any;
+ maxUsageCount: number;
+ sessionPool: SessionPool;
+ /**
+ * indicates whether the session is blocked.
+ * Session is blocked once it reaches the `maxErrorScore`.
+ * @return {boolean}
+ */
+ isBlocked(): boolean;
+ /**
+ * Indicates whether the session is expired.
+ * Session expiration is determined by the `maxAgeSecs`.
+ * Once the session is older than `createdAt + maxAgeSecs` the session is considered expired.
+ * @return {boolean}
+ */
+ isExpired(): boolean;
+ /**
+ * Indicates whether the session is used maximum number of times.
+ * Session maximum usage count can be changed by `maxUsageCount` parameter.
+ * @return {boolean}
+ */
+ isMaxUsageCountReached(): boolean;
+ /**
+ * Indicates whether the session can be used for next requests.
+ * Session is usable when it is not expired, not blocked and the maximum usage count has not be reached.
+ * @return {boolean}
+ */
+ isUsable(): boolean;
+ /**
+ * This method should be called after a successful session usage.
+ * It increases `usageCount` and potentially lowers the `errorScore` by the `errorScoreDecrement`.
+ */
+ markGood(): void;
+ /**
+ * Gets session state for persistence in KeyValueStore.
+ * @return {SessionState} represents session internal state.
+ */
+ getState(): SessionState;
+ /**
+ * Marks session as blocked and emits event on the `SessionPool`
+ * This method should be used if the session usage was unsuccessful
+ * and you are sure that it is because of the session configuration and not any external matters.
+ * For example when server returns 403 status code.
+ * If the session does not work due to some external factors as server error such as 5XX you probably want to use `markBad` method.
+ */
+ retire(): void;
+ /**
+ * Increases usage and error count.
+ * Should be used when the session has been used unsuccessfully. For example because of timeouts.
+ */
+ markBad(): void;
+ /**
+ * Retires session based on status code.
+ * @param statusCode {Number} - HTTP status code
+ * @param blockedStatusCodes {Array<Number>} - Custom HTTP status codes that means blocking on particular website.
+ * @return {boolean} whether the session was retired.
+ */
+ retireOnBlockedStatusCodes(statusCode: number, blockedStatusCodes?: number[]): boolean;
+ /**
+ * Sets cookies from response to the cookieJar.
+ * Parses cookies from `set-cookie` header and sets them to `Session.cookieJar`.
+ * @param {{ headers }} response
+ */
+ setCookiesFromResponse(response: {
+ headers: any;
+ }): void;
+ /**
+ * Persists puppeteer cookies to session for reuse.
+ * @param {PuppeteerCookie} puppeteerCookies - cookie from puppeteer `page.cookies` method.
+ * @param {String} url - Loaded url from page function.
+ */
+ putPuppeteerCookies(puppeteerCookies: PuppeteerCookie, url: string): void;
+ /**
+ * Set cookies to session cookieJar.
+ * Cookies array should be [puppeteer](https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-pagecookiesurls) cookie compatible.
+ * @param cookies {PuppeteerCookie[]}
+ * @param url {String}
+ */
+ setPuppeteerCookies(cookies: PuppeteerCookie[], url: string): void;
+ /**
+ * Gets cookies in puppeteer ready to be used with `page.setCookie`.
+ * @param url {String} - website url. Only cookies stored for this url will be returned
+ * @return {PuppeteerCookie[]}
+ */
+ getPuppeteerCookies(url: string): PuppeteerCookie[];
+ /**
+ * Wrapper around `tough-cookie` Cookie jar `getCookieString` method.
+ * @param {String} url
+ * @return {String} - represents `Cookie` header.
+ */
+ getCookieString(url: string): string;
+ /**
+ * Transforms puppeteer cookie to tough-cookie.
+ * @param puppeteerCookie {PuppeteerCookie} - Cookie from puppeteer `page.cookies method.
+ * @return {Cookie}
+ * @private
+ */
+ _puppeteerCookieToTough(puppeteerCookie: PuppeteerCookie): Cookie;
+ /**
+ * Transforms tough-cookie cookie to puppeteer Cookie .
+ * @param {Cookie} toughCookie - Cookie from CookieJar.
+ * @return {PuppeteerCookie} - puppeteer cookie
+ * @private
+ */
+ _toughCookieToPuppeteer(toughCookie: Cookie): PuppeteerCookie;
+ /**
+ * Sets cookies.
+ * @param {Cookie} cookies
+ * @param {String} url
+ * @private
+ */
+ _setCookies(cookies: Cookie, url: string): void;
+}
+/**
+ * Persistable {Session} state.
+ */
+export type SessionState = {
+ id: string;
+ cookies: any;
+ cookieJar: any;
+ userData: any;
+ maxErrorScore: number;
+ errorScoreDecrement: number;
+ expiresAt: string;
+ createdAt: string;
+ usageCount: number;
+ errorScore: number;
+};
+export type SessionOptions = {
+ /**
+ * - Id of session used for generating fingerprints. It is used as proxy session name.
+ */
+ id?: string;
+ /**
+ * - Number of seconds after which the session is considered as expired.
+ */
+ maxAgeSecs?: number;
+ /**
+ * - Object where custom user data can be stored. For example custom headers.
+ */
+ userData: any;
+ /**
+ * - Maximum number of marking session as blocked usage.
+ * If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
+ * It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
+ * Calling the `markGood` will decrease the `errorScore` by `errorScoreDecrement`
+ */
+ maxErrorScore?: number;
+ /**
+ * - It is used for healing the session.
+ * For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this
+ * number.
+ */
+ errorScoreDecrement?: number;
+ /**
+ * - Date of creation.
+ */
+ createdAt?: Date;
+ /**
+ * - Date of expiration.
+ */
+ expiresAt?: Date;
+ /**
+ * - Indicates how many times the session has been used.
+ */
+ usageCount?: number;
+ /**
+ * - Indicates how many times the session is marked bad.
+ */
+ errorCount?: number;
+ /**
+ * - Session should be used only a limited amount of times.
+ * This number indicates how many times the session is going to be used, before it is thrown away.
+ */
+ maxUsageCount?: number;
+ /**
+ * - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
+ */
+ sessionPool: SessionPool;
+};
+import { CookieJar } from "tough-cookie";
+import { SessionPool } from "./session_pool";
+import { Cookie as PuppeteerCookie } from "puppeteer";
+import { Cookie } from "tough-cookie";
diff --git a/types/session_pool/session_pool.d.ts b/types/session_pool/session_pool.d.ts
new file mode 100644
index 000000000..18edb7695
--- /dev/null
+++ b/types/session_pool/session_pool.d.ts
@@ -0,0 +1,212 @@
+/**
+ * Factory user-function which creates customized {@link Session} instances.
+ * @callback CreateSession
+ * @param {SessionPool} sessionPool Pool requesting the new session.
+ */
+/**
+ * @typedef {Object} SessionPoolOptions
+ * @property {Number} [maxPoolSize=1000] - Maximum size of the pool.
+ * Indicates how many sessions are rotated.
+ * @property {SessionOptions} [sessionOptions] The configuration options for {Session} instances.
+ * @property {String} [persistStateKeyValueStoreId] - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
+ * @property {String} [persistStateKey="SESSION_POOL_STATE"] - Session pool persists it's state under this key in Key value store.
+ * @property {CreateSession} [createSessionFunction] - Custom function that should return `Session` instance.
+ * Function receives `SessionPool` instance as a parameter
+ */
+/**
+ * Handles the sessions rotation, creation and persistence.
+ * Creates a pool of {@link Session} instances, that are randomly rotated.
+ * When some session is marked as blocked. It is removed and new one is created instead.
+ *
+ * Session pool is by default persisted in default {@link KeyValueStore}.
+ * If you want to have one pool for all runs you have to specify `persistStateKeyValueStoreId`.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const sessionPool = new SessionPool({
+ * maxPoolSize: 25,
+ * sessionOptions:{
+ * maxAgeSecs: 10,
+ * maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
+ * },
+ * persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',
+ * persistStateKey: 'my-session-pool',
+ * });
+ *
+ * // Now you have to initialize the `SessionPool`.
+ * // If you already have a persisted state in the selected `KeyValueState`.
+ * // The Session pool is recreated, otherwise it creates a new one.
+ * // It also attaches listener to `Apify.events` so it is persisted periodically and not after every change.
+ * await sessionPool.initialize();
+ *
+ * // Get random session from the pool
+ * const session1 = await sessionPool.getSession();
+ * const session2 = await sessionPool.getSession();
+ * const session3 = await sessionPool.getSession();
+ *
+ * // Now you can mark the session either failed of successful
+ *
+ * // Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
+ * session1.markBad()
+ *
+ * // Marks as successful.
+ * session2.markGood()
+ *
+ * // Retires session -> session is removed from the pool
+ * session3.retire()
+ *
+ * ```
+ * @hideconstructor
+ */
+export class SessionPool extends EventEmitter {
+ /**
+ * Session pool configuration.
+ * @param {SessionPoolOptions} [options] All `SessionPool` configuration options.
+ */
+ constructor(options?: SessionPoolOptions);
+ maxPoolSize: number;
+ createSessionFunction: any;
+ sessionOptions: {};
+ persistStateKeyValueStoreId: any;
+ persistStateKey: string;
+ keyValueStore: import("../key_value_store").KeyValueStore;
+ sessions: any[];
+ /**
+ * Gets count of usable sessions in the pool.
+ * @return {number}
+ */
+ get usableSessionsCount(): number;
+ /**
+ * Gets count of retired sessions in the pool.
+ * @return {number}
+ */
+ get retiredSessionsCount(): number;
+ /**
+ * Starts periodic state persistence and potentially loads SessionPool state from {@link KeyValueStore}.
+ * This function must be called before you can start using the instance in a meaningful way.
+ *
+ * @return {Promise<void>}
+ */
+ initialize(): Promise<void>;
+ _listener: any;
+ /**
+ * Gets session.
+ * If there is space for new session, it creates and return new session.
+ * If the session pool is full, it picks a session from the pool,
+ * If the picked session is usable it is returned, otherwise it creates and returns a new one.
+ *
+ * @return {Promise<Session>}
+ */
+ getSession(): Promise<Session>;
+ /**
+ * Returns an object representing the internal state of the `SessionPool` instance.
+ * Note that the object's fields can change in future releases.
+ *
+ * @returns {Object}
+ */
+ getState(): any;
+ /**
+ * Persists the current state of the `SessionPool` into the default {@link KeyValueStore}.
+ * The state is persisted automatically in regular intervals.
+ *
+ * @return {Promise}
+ */
+ persistState(): Promise<any>;
+ /**
+ * Removes listener from `persistState` event.
+ * This function should be called after you are done with using the `SessionPool` instance.
+ */
+ teardown(): void;
+ /**
+ * Removes `Session` instance from `SessionPool`.
+ * @param session {Session} - Session to be removed
+ * @private
+ */
+ _removeSession(session: Session): void;
+ /**
+ * Adds `Session` instance to `SessionPool`.
+ * @param newSession {Session} - `Session` instance to be added.
+ * @private
+ */
+ _addSession(newSession: Session): void;
+ /**
+ * Gets random index.
+ * @return {number}
+ * @private
+ */
+ _getRandomIndex(): number;
+ /**
+ * Creates new session without any extra behavior.
+ * @param sessionPool
+ * @return {Session} - New session.
+ * @private
+ */
+ _defaultCreateSessionFunction(sessionPool: any): Session;
+ /**
+ * Creates new session and adds it to the pool.
+ * @return {Promise<Session>} - Newly created `Session` instance.
+ * @private
+ */
+ _createSession(): Promise<Session>;
+ /**
+ * Decides whether there is enough space for creating new session.
+ * @return {boolean}
+ * @private
+ */
+ _hasSpaceForSession(): boolean;
+ /**
+ * Picks random session from the `SessionPool`.
+ * @return {Session} - Picked `Session`
+ * @private
+ */
+ _pickSession(): Session;
+ /**
+ * Potentially loads `SessionPool`.
+ * If the state was persisted it loads the `SessionPool` from the persisted state.
+ * @return {Promise<void>}
+ * @private
+ */
+ _maybeLoadSessionPool(): Promise<void>;
+ addListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ on(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ once(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ prependListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ prependOnceListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ removeListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ off(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ removeAllListeners(event?: string | symbol): SessionPool;
+ setMaxListeners(n: number): SessionPool;
+}
+export function openSessionPool(sessionPoolOptions: SessionPoolOptions): Promise<SessionPool>;
+/**
+ * Factory user-function which creates customized {@link Session} instances.
+ */
+export type CreateSession = (sessionPool: SessionPool) => any;
+export type SessionPoolOptions = {
+ /**
+ * - Maximum size of the pool.
+ * Indicates how many sessions are rotated.
+ */
+ maxPoolSize?: number;
+ /**
+ * The configuration options for {Session} instances.
+ */
+ sessionOptions?: SessionOptions;
+ /**
+ * - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
+ */
+ persistStateKeyValueStoreId?: string;
+ /**
+ * - Session pool persists it's state under this key in Key value store.
+ */
+ persistStateKey?: string;
+ /**
+ * - Custom function that should return `Session` instance.
+ * Function receives `SessionPool` instance as a parameter
+ */
+ createSessionFunction?: CreateSession;
+};
+import { EventEmitter } from "node/events";
+import { Session } from "./session";
+import { SessionOptions } from "./session";
diff --git a/types/session_pool/session_utils.d.ts b/types/session_pool/session_utils.d.ts
new file mode 100644
index 000000000..303d4905c
--- /dev/null
+++ b/types/session_pool/session_utils.d.ts
@@ -0,0 +1 @@
+export function getCookiesFromResponse(response: any): any[];
diff --git a/types/stealth/hiding_tricks.d.ts b/types/stealth/hiding_tricks.d.ts
new file mode 100644
index 000000000..cad783ad8
--- /dev/null
+++ b/types/stealth/hiding_tricks.d.ts
@@ -0,0 +1,23 @@
+declare namespace _default {
+ export { addPlugins };
+ export { emulateWindowFrame };
+ export { emulateWebGL };
+ export { emulateConsoleDebug };
+ export { addLanguage };
+ export { hideWebDriver };
+ export { hackPermissions };
+ export { mockChrome };
+ export { mockChromeInIframe };
+ export { mockDeviceMemory };
+}
+export default _default;
+declare function addPlugins(): void;
+declare function emulateWindowFrame(): void;
+declare function emulateWebGL(): void;
+declare function emulateConsoleDebug(): void;
+declare function addLanguage(): void;
+declare function hideWebDriver(): void;
+declare function hackPermissions(): void;
+declare function mockChrome(): void;
+declare function mockChromeInIframe(): void;
+declare function mockDeviceMemory(): void;
diff --git a/types/stealth/stealth.d.ts b/types/stealth/stealth.d.ts
new file mode 100644
index 000000000..2d185f69f
--- /dev/null
+++ b/types/stealth/stealth.d.ts
@@ -0,0 +1,54 @@
+/**
+ * The main purpose of this function is to override newPage function and attach selected tricks.
+ * @param {Browser} browser - puppeteer browser instance
+ * @param {StealthOptions} options
+ * @returns {Promise<Browser>} - Instance of Browser from puppeteer package
+ */
+export default function applyStealthToBrowser(browser: Browser, options: StealthOptions): Promise<Browser>;
+/**
+ * Configuration of stealth tricks for a proper hiding effect all of them should be set to true.
+ * These tricks are applied only when the `stealth` option is set to `true`.
+ */
+export type StealthOptions = {
+ /**
+ * - If plugins should be added to the navigator.
+ */
+ addPlugins?: boolean;
+ /**
+ * - Emulates window Iframe.
+ */
+ emulateWindowFrame?: boolean;
+ /**
+ * - Emulates graphic card.
+ */
+ emulateWebGL?: boolean;
+ /**
+ * - Emulates console.debug to return null.
+ */
+ emulateConsoleDebug?: boolean;
+ /**
+ * - Adds languages to the navigator.
+ */
+ addLanguage?: boolean;
+ /**
+ * - Hides the webdriver by changing the navigator proto.
+ */
+ hideWebDriver?: boolean;
+ /**
+ * - Fakes interaction with permissions.
+ */
+ hackPermissions?: boolean;
+ /**
+ * - Adds the chrome runtime properties.
+ */
+ mockChrome?: boolean;
+ /**
+ * - Adds the chrome runtime properties inside the every newly created iframe.
+ */
+ mockChromeInIframe?: boolean;
+ /**
+ * - Sets device memory to other value than 0.
+ */
+ mockDeviceMemory?: boolean;
+};
+import { Browser } from "puppeteer";
diff --git a/types/typedefs.d.ts b/types/typedefs.d.ts
new file mode 100644
index 000000000..cb26ac6a7
--- /dev/null
+++ b/types/typedefs.d.ts
@@ -0,0 +1,122 @@
+/**
+ * `CherioStatic`, the return type of `cheerio.load()`, is not exported from `@types/cheerio`, so it can not be imported and aliased, it is a global
+ * type. We reference the type and redefine it with a bettter name here.
+ */
+export type Cheerio = CheerioStatic;
+/**
+ * Represents information about an actor run, as returned by the
+ * [`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.
+ * The object is almost equivalent to the JSON response
+ * of the
+ * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>
+ * Apify API endpoint and extended with certain fields.
+ * For more details, see
+ * <a href="https://docs.apify.com/actor/run" target="_blank">Runs.</a>
+ */
+export type ActorRun = {
+ /**
+ * Actor run ID
+ */
+ id: string;
+ /**
+ * Actor ID
+ */
+ actId: string;
+ /**
+ * Time when the actor run started
+ */
+ startedAt: Date;
+ /**
+ * Time when the actor run finished. Contains `null` for running actors.
+ */
+ finishedAt: Date;
+ /**
+ * Status of the run. For possible values, see
+ * <a href="https://docs.apify.com/actor/run#lifecycle" target="_blank">Run lifecycle</a>
+ * in Apify actor documentation.
+ */
+ status: string;
+ /**
+ * Actor run meta-data. For example:
+ * ```
+ * {
+ * "origin": "API",
+ * "clientIp": "1.2.3.4",
+ * "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"
+ * }
+ * ```
+ */
+ meta: any;
+ /**
+ * An object containing various actor run statistics. For example:
+ * ```
+ * {
+ * "inputBodyLen": 22,
+ * "restartCount": 0,
+ * "workersUsed": 1,
+ * }
+ * ```
+ * Beware that object fields might change in future releases.
+ */
+ stats: any;
+ /**
+ * Actor run options. For example:
+ * ```
+ * {
+ * "build": "latest",
+ * "waitSecs": 0,
+ * "memoryMbytes": 256,
+ * "diskMbytes": 512
+ * }
+ * ```
+ */
+ options: any;
+ /**
+ * ID of the actor build used for the run. For details, see
+ * <a href="https://docs.apify.com/actor/build" target="_blank">Builds</a>
+ * in Apify actor documentation.
+ */
+ buildId: string;
+ /**
+ * Number of the actor build used for the run. For example, `0.0.10`.
+ */
+ buildNumber: string;
+ /**
+ * Exit code of the actor run process. It's `null` if actor is still running.
+ */
+ exitCode: number;
+ /**
+ * ID of the default key-value store associated with the actor run. See [`KeyValueStore`](../api/keyvaluestore) for details.
+ */
+ defaultKeyValueStoreId: string;
+ /**
+ * ID of the default dataset associated with the actor run. See [`Dataset`](../api/dataset) for details.
+ */
+ defaultDatasetId: string;
+ /**
+ * ID of the default request queue associated with the actor run. See [`RequestQueue`](../api/requestqueue) for details.
+ */
+ defaultRequestQueueId: string;
+ /**
+ * URL on which the web server running inside actor run's Docker container can be accessed.
+ * For more details, see
+ * <a href="https://docs.apify.com/actor/run#container-web-server" target="_blank">Container web server</a>
+ * in Apify actor documentation.
+ */
+ containerUrl: string;
+ /**
+ * Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,
+ * or if you pass `false` to the `fetchOutput` option of [`Apify.call()`](../api/apify#module_Apify.call).
+ *
+ * For example:
+ * ```
+ * {
+ * "contentType": "application/json; charset=utf-8",
+ * "body": {
+ * "message": "Hello world!"
+ * }
+ * }
+ * ```
+ */
+ output: any;
+};
diff --git a/types/utils.d.ts b/types/utils.d.ts
new file mode 100644
index 000000000..c2eb47fd8
--- /dev/null
+++ b/types/utils.d.ts
@@ -0,0 +1,110 @@
+export function newClient(): any;
+export function logSystemInfo(): void;
+/**
+ * Gets the default instance of the `ApifyClient` class provided
+ * <a href="https://docs.apify.com/api/apify-client-js/latest"
+ * target="_blank">apify-client</a> by the NPM package.
+ * The instance is created automatically by the Apify SDK and it is configured using the
+ * `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN` environment variables.
+ *
+ * The instance is used for all underlying calls to the Apify API in functions such as
+ * [`Apify.getValue()`](#module_Apify.getValue) or [`Apify.call()`](#module_Apify.call).
+ * The settings of the client can be globally altered by calling the
+ * <a href="https://docs.apify.com/api/apify-client-js/latest#ApifyClient-setOptions"
+ * target="_blank">`Apify.client.setOptions()`</a> function.
+ * Beware that altering these settings might have unintended effects on the entire Apify SDK package.
+ *
+ * @type {*}
+ *
+ * @memberof module:Apify
+ * @name client
+ */
+export const apifyClient: any;
+export function newPromise(): Promise<void>;
+export function addCharsetToContentType(contentType: string): string;
+export function isDocker(forceReset: boolean): Promise<boolean>;
+export function sum(arr: number[]): number;
+export function avg(arr: number[]): number;
+export function weightedAvg(arrValues: number[], arrWeights: number[]): number;
+export function getMemoryInfo(): Promise<MemoryInfo>;
+export function isPromise(maybePromise: any): boolean;
+export function isProduction(): boolean;
+export function ensureDirExists(dirPath: any): any;
+export function getFirstKey(dict: any): string;
+export function getTypicalChromeExecutablePath(): string;
+export function addTimeoutToPromise<T>(promise: Promise<T>, timeoutMillis: number, errorMessage: string): Promise<T>;
+export function isAtHome(): boolean;
+export function sleep(millis: number): Promise<void>;
+export function openLocalStorage(idOrName: any, defaultIdEnvVar: any, LocalClass: any, cache: any): Promise<any>;
+export function openRemoteStorage(idOrName: any, defaultIdEnvVar: any, RemoteClass: any, cache: any, getOrCreateFunction: any): Promise<any>;
+export function ensureTokenOrLocalStorageEnvExists(storageName: any): void;
+export function snakeCaseToCamelCase(snakeCaseStr: string): string;
+export function printOutdatedSdkWarning(): void;
+export function parseContentTypeFromResponse(response: any): {
+ type: string;
+ charset: string;
+};
+export namespace publicUtils {
+ export { isDocker };
+ export { sleep };
+ export { downloadListOfUrls };
+ export { extractUrls };
+ export { getRandomUserAgent };
+ export { htmlToText };
+ export { URL_NO_COMMAS_REGEX };
+ export { URL_WITH_COMMAS_REGEX };
+ export { createRequestDebugInfo };
+ export { parseContentTypeFromResponse };
+}
+/**
+ * Describes memory usage of an Actor.
+ */
+export type MemoryInfo = {
+ /**
+ * Total memory available in the system or container
+ */
+ totalBytes: number;
+ /**
+ * Amount of free memory in the system or container
+ */
+ freeBytes: number;
+ /**
+ * Amount of memory used (= totalBytes - freeBytes)
+ */
+ usedBytes: number;
+ /**
+ * Amount of memory used the current Node.js process
+ */
+ mainProcessBytes: number;
+ /**
+ * Amount of memory used by child processes of the current Node.js process
+ */
+ childProcessesBytes: number;
+};
+declare function downloadListOfUrls({ url, encoding, urlRegExp }: {
+ url: string;
+ encoding?: string;
+ urlRegExp?: RegExp;
+}): Promise<string[]>;
+declare function extractUrls({ string, urlRegExp }: {
+ string: string;
+ urlRegExp?: RegExp;
+}): string[];
+declare function getRandomUserAgent(): string;
+declare function htmlToText(html: string | CheerioStatic): string;
+/**
+ * Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters
+ * and does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
+ * @memberOf utils
+ */
+declare const URL_NO_COMMAS_REGEX: RegExp;
+/**
+ * Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.
+ * Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
+ * @memberOf utils
+ */
+declare const URL_WITH_COMMAS_REGEX: RegExp;
+declare function createRequestDebugInfo(request: any, response?: IncomingMessage | PuppeteerResponse, additionalFields?: any): any;
+import { IncomingMessage } from "http";
+import { Response as PuppeteerResponse } from "puppeteer";
+export {};
diff --git a/types/utils_request.d.ts b/types/utils_request.d.ts
new file mode 100644
index 000000000..db35acab0
--- /dev/null
+++ b/types/utils_request.d.ts
@@ -0,0 +1,58 @@
+export const FIREFOX_MOBILE_USER_AGENT: "Mozilla/5.0 (Android; Mobile; rv:14.0) Gecko/14.0 Firefox/14.0";
+export const FIREFOX_DESKTOP_USER_AGENT: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0";
+export namespace REQUEST_AS_BROWSER_DEFAULT_OPTIONS {
+ export const countryCode: string;
+ export const languageCode: string;
+ export const headers: {};
+ export const method: string;
+ export const useMobileVersion: boolean;
+ export const useBrotli: boolean;
+ export const json: boolean;
+ export function abortFunction(res: any): boolean;
+ export const useCaseSensitiveHeaders: boolean;
+ export const useStream: boolean;
+ export const proxyUrl: any;
+ export const timeoutSecs: number;
+}
+export function requestAsBrowser(options: RequestAsBrowserOptions): Promise<any>;
+/**
+ * (@apify/http-request)[https://www.npmjs.com/package/@apify/http-request] NPM package.
+ */
+export type RequestAsBrowserOptions = {
+ /**
+ * URL of the target endpoint. Supports both HTTP and HTTPS schemes.
+ */
+ url: string;
+ /**
+ * HTTP method.
+ */
+ method?: string;
+ /**
+ * Additional HTTP headers to add. It's only recommended to use this option,
+ * with headers that are typically added by websites, such as cookies. Overriding
+ * default browser headers will remove the masking this function provides.
+ */
+ headers?: any;
+ /**
+ * An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.
+ */
+ proxyUrl?: string;
+ /**
+ * Two-letter ISO 639 language code.
+ */
+ languageCode?: string;
+ /**
+ * Two-letter ISO 3166 country code.
+ */
+ countryCode?: string;
+ /**
+ * If `true`, the function uses User-Agent of a mobile browser.
+ */
+ isMobile?: boolean;
+ /**
+ * Function accepts `response` object as a single parameter and should return true or false.
+ * If function returns true request gets aborted. This function is passed to the
+ * (
+ */
+ abortFunction?: Function;
+};
diff --git a/types/utils_social.d.ts b/types/utils_social.d.ts
new file mode 100644
index 000000000..aee56f930
--- /dev/null
+++ b/types/utils_social.d.ts
@@ -0,0 +1,87 @@
+export namespace socialUtils {
+ export { emailsFromText };
+ export { emailsFromUrls };
+ export { phonesFromText };
+ export { phonesFromUrls };
+ export { parseHandlesFromHtml };
+ export { EMAIL_REGEX };
+ export { EMAIL_REGEX_GLOBAL };
+ export { LINKEDIN_REGEX };
+ export { LINKEDIN_REGEX_GLOBAL };
+ export { INSTAGRAM_REGEX };
+ export { INSTAGRAM_REGEX_GLOBAL };
+ export { TWITTER_REGEX };
+ export { TWITTER_REGEX_GLOBAL };
+ export { FACEBOOK_REGEX };
+ export { FACEBOOK_REGEX_GLOBAL };
+ export { YOUTUBE_REGEX };
+ export { YOUTUBE_REGEX_GLOBAL };
+}
+/**
+ * Representation of social handles parsed from a HTML page.
+ *
+ * The object has the following structure:
+ *
+ * ```
+ * {
+ * emails: String[],
+ * phones: String[],
+ * phonesUncertain: String[],
+ * linkedIns: String[],
+ * twitters: String[],
+ * instagrams: String[],
+ * facebooks: String[],
+ * youtubes: String[],
+ * }
+ * ```
+ */
+export type SocialHandles = {
+ emails: string[];
+ phones: string[];
+ phonesUncertain: string[];
+ linkedIns: string[];
+ twitters: string[];
+ instagrams: string[];
+ facebooks: string[];
+ youtubes: string[];
+};
+declare function emailsFromText(text: string): string[];
+declare function emailsFromUrls(urls: string[]): string[];
+declare function phonesFromText(text: string): string[];
+declare function phonesFromUrls(urls: string[]): string[];
+declare function parseHandlesFromHtml(html: string, data?: any): SocialHandles;
+/**
+ * Regular expression to exactly match a single email address.
+ * It has the following form: `/^...$/i`.
+ * @type {RegExp}
+ * @memberOf social
+ */
+declare const EMAIL_REGEX: RegExp;
+/**
+ * Regular expression to find multiple email addresses in a text.
+ * It has the following form: `/.../ig`.
+ * @type {RegExp}
+ * @memberOf social
+ */
+declare const EMAIL_REGEX_GLOBAL: RegExp;
+/** @type RegExp */
+declare let LINKEDIN_REGEX: RegExp;
+/** @type RegExp */
+declare let LINKEDIN_REGEX_GLOBAL: RegExp;
+/** @type RegExp */
+declare let INSTAGRAM_REGEX: RegExp;
+/** @type RegExp */
+declare let INSTAGRAM_REGEX_GLOBAL: RegExp;
+/** @type RegExp */
+declare let TWITTER_REGEX: RegExp;
+/** @type RegExp */
+declare let TWITTER_REGEX_GLOBAL: RegExp;
+/** @type RegExp */
+declare let FACEBOOK_REGEX: RegExp;
+/** @type RegExp */
+declare let FACEBOOK_REGEX_GLOBAL: RegExp;
+/** @type RegExp */
+declare let YOUTUBE_REGEX: RegExp;
+/** @type RegExp */
+declare let YOUTUBE_REGEX_GLOBAL: RegExp;
+export {};
diff --git a/website/i18n/en.json b/website/i18n/en.json
index 0c743a08b..aaa3f4171 100644
--- a/website/i18n/en.json
+++ b/website/i18n/en.json
@@ -122,6 +122,9 @@
"guides/whatisanactor": {
"title": "What is an Actor"
},
+ "guides/typescriptactor": {
+ "title": "Typescript Actors"
+ },
"readme/introduction": {
"title": "readme/introduction"
},
@@ -137,14 +140,125 @@
"typedefs/apifycallerror": {
"title": "ApifyCallError"
},
+ "typedefs/apifyenv": {
+ "title": "ApifyEnv"
+ },
+ "typedefs/autoscaledpooloptions": {
+ "title": "AutoscaledPoolOptions"
+ },
+ "typedefs/basiccrawleroptions": {
+ "title": "BasicCrawlerOptions"
+ },
+ "typedefs/cheeriocrawleroptions": {
+ "title": "CheerioCrawlerOptions"
+ },
+ "typedefs/cheeriohandlepage": {
+ "title": "CheerioHandlePage"
+ },
+ "typedefs/cheeriohandlepageinputs": {
+ "title": "CheerioHandlePageInputs"
+ },
+ "typedefs/datasetconsumer": {
+ "title": "DatasetConsumer"
+ },
+ "typedefs/datasetcontent": {
+ "title": "DatasetContent"
+ },
+ "typedefs/datasetmapper": {
+ "title": "DatasetMapper"
+ },
+ "typedefs/datasetreducer": {
+ "title": "DatasetReducer"
+ },
+ "typedefs/handlefailedrequest": {
+ "title": "HandleFailedRequest"
+ },
+ "typedefs/handlefailedrequestinput": {
+ "title": "HandleFailedRequestInput"
+ },
+ "typedefs/handlerequest": {
+ "title": "HandleRequest"
+ },
+ "typedefs/handlerequestinputs": {
+ "title": "HandleRequestInputs"
+ },
+ "typedefs/keyconsumer": {
+ "title": "KeyConsumer"
+ },
+ "typedefs/launchpuppeteer": {
+ "title": "LaunchPuppeteer"
+ },
"typedefs/launchpuppeteeroptions": {
"title": "LaunchPuppeteerOptions"
},
+ "typedefs/memoryinfo": {
+ "title": "MemoryInfo"
+ },
+ "typedefs/preparerequest": {
+ "title": "PrepareRequest"
+ },
+ "typedefs/preparerequestinputs": {
+ "title": "PrepareRequestInputs"
+ },
+ "typedefs/puppeteercrawleroptions": {
+ "title": "PuppeteerCrawlerOptions"
+ },
+ "typedefs/puppeteergoto": {
+ "title": "PuppeteerGoto"
+ },
+ "typedefs/puppeteergotoinputs": {
+ "title": "PuppeteerGotoInputs"
+ },
+ "typedefs/puppeteerhandlepage": {
+ "title": "PuppeteerHandlePage"
+ },
+ "typedefs/puppeteerhandlepageinputs": {
+ "title": "PuppeteerHandlePageInputs"
+ },
+ "typedefs/puppeteerpooloptions": {
+ "title": "PuppeteerPoolOptions"
+ },
"typedefs/queueoperationinfo": {
"title": "QueueOperationInfo"
},
+ "typedefs/requestasbrowseroptions": {
+ "title": "RequestAsBrowserOptions"
+ },
+ "typedefs/requestlistoptions": {
+ "title": "RequestListOptions"
+ },
+ "typedefs/requestliststate": {
+ "title": "RequestListState"
+ },
+ "typedefs/requestoptions": {
+ "title": "RequestOptions"
+ },
+ "typedefs/requesttransform": {
+ "title": "RequestTransform"
+ },
+ "typedefs/sessionoptions": {
+ "title": "SessionOptions"
+ },
+ "typedefs/sessionpooloptions": {
+ "title": "SessionPoolOptions"
+ },
+ "typedefs/sessionstate": {
+ "title": "SessionState"
+ },
+ "typedefs/snapshotteroptions": {
+ "title": "SnapshotterOptions"
+ },
+ "typedefs/socialhandles": {
+ "title": "SocialHandles"
+ },
"typedefs/stealthoptions": {
"title": "StealthOptions"
+ },
+ "typedefs/systeminfo": {
+ "title": "SystemInfo"
+ },
+ "typedefs/systemstatusoptions": {
+ "title": "SystemStatusOptions"
}
},
"links": {
@@ -158,7 +272,13 @@
"Reference": "Reference",
"Examples": "Examples",
"Autoscaling Tools": "Autoscaling Tools",
- "Type Definitions": "Type Definitions"
+ "Type Definitions": "Type Definitions",
+ "Constructor Options": "Constructor Options",
+ "Crawler User-Functions": "Crawler User-Functions",
+ "Puppeteer User-Functions": "Puppeteer User-Functions",
+ "Dataset User-Functions": "Dataset User-Functions",
+ "KeyValueStore User-Functions": "KeyValueStore User-Functions",
+ "Utility User-Functions": "Utility User-Functions"
}
},
"pages-strings": {
diff --git a/website/sidebars.json b/website/sidebars.json
index 944538c09..5152e7b27 100644
--- a/website/sidebars.json
+++ b/website/sidebars.json
@@ -5,6 +5,7 @@
"guides/gettingstarted",
"guides/quickstart",
"guides/whatisanactor",
+ "guides/typescriptactor",
"guides/environmentvariables",
"guides/datastorage",
"guides/puppeteerliveview",
@@ -54,8 +55,63 @@
"Type Definitions": [
"typedefs/actorrun",
"typedefs/apifycallerror",
+ "typedefs/apifyenv",
+ "typedefs/datasetcontent",
+ "typedefs/memoryinfo",
+ "typedefs/queueoperationinfo",
+ "typedefs/requestliststate",
+ "typedefs/sessionstate",
+ "typedefs/socialhandles",
+ "typedefs/systeminfo"
+ ],
+ "Constructor Options": [
+ "typedefs/autoscaledpooloptions",
+ "typedefs/basiccrawleroptions",
+ "typedefs/cheeriocrawleroptions",
+ "typedefs/puppeteercrawleroptions",
+ "typedefs/puppeteerpooloptions",
"typedefs/launchpuppeteeroptions",
- "typedefs/queueoperationinfo"
+ "typedefs/requestoptions",
+ "typedefs/requestlistoptions",
+ "typedefs/requestasbrowseroptions",
+ "typedefs/sessionoptions",
+ "typedefs/sessionpooloptions",
+ "typedefs/snapshotteroptions",
+ "typedefs/stealhtoptions",
+ "typedefs/systemstatusoptions"
+ ]
+ },
+ "User-Functions": {
+ "Crawler User-Functions": [
+ "typedefs/cheeriohandlepage",
+ "typedefs/cheeriohandlepageinputs",
+ "typedefs/handlefailedrequest",
+ "typedefs/handlefailedrequestinput",
+ "typedefs/handlerequest",
+ "typedefs/handlerequestinputs",
+ "typedefs/preparerequest",
+ "typedefs/preparerequestinputs"
+ ],
+ "Puppeteer User-Functions": [
+ "typedefs/puppeteerhandlepage",
+ "typedefs/puppeteerhandlepageinputs",
+ "typedefs/launchpuppeteer",
+ "typedefs/puppeteergoto",
+ "typedefs/puppeteergotoinputs"
+ ],
+ "Dataset User-Functions": [
+ "typedefs/datasetconsumer",
+ "typedefs/datasetmapper",
+ "typedefs/datasetreducer"
+ ],
+ "KeyValueStore User-Functions": [
+ "typedefs/keyconsumer"
+ ],
+ "SessionPool User-Functions": [
+ "typedefs/createsession"
+ ],
+ "Utility User-Functions": [
+ "typedefs/requesttransform"
]
}
}