Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Pocesar feature/tsc compile (#605)

Ondra Urban9a38f8cddea2
diff --git a/.babelrc b/.babelrc
index 64e62a5e9..98ed8e4e2 100644
--- a/.babelrc
+++ b/.babelrc
@@ -1,9 +1,10 @@
{
- "presets": [
- ["@babel/preset-env", {
- "targets": {
- "node": "10.17.0"
- }
- }]
- ]
+ "comments": false,
+ "presets": [
+ ["@babel/preset-env", {
+ "targets": {
+ "node": "10.17.0"
+ }
+ }]
+ ]
}
diff --git a/.gitignore b/.gitignore
index 201de97d4..2dc0d6076 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,3 +15,4 @@ pids
yarn.lock
package-lock.json
tmp
+jsconfig.json
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index 1d5dd7ada..8899304ea 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -15,7 +15,7 @@ separate, detailed, documentation pages accessible from the left sidebar. To lea
## `Apify.addWebhook(options)`
-**Returns**: `Promise<Object>` - The return value is the Webhook object. For more information, see the
+**Returns**: `Promise<object>` - The return value is the Webhook object. For more information, see the
[Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.
Creates an ad-hoc webhook for the current actor run, which lets you receive a notification when the actor run finished or failed. For more information
@@ -57,7 +57,7 @@ By passing the `waitSecs` option you can reduce the maximum amount of time to wa
the function returns immediately after the run is started.
The result of the function is an [`ActorRun`](/docs/typedefs/actor-run) object that contains details about the actor run and its output (if any). If
-the actor run fails, the function throws the [`ApifyCallError`](/docs/typedefs/apify-call-error) exception.
+the actor run fails, the function throws the [`ApifyCallError`](/docs/api/apify-call-error) exception.
If you want to run an actor task rather than an actor, please use the [`Apify.callTask()`](/docs/api/apify#calltask) function instead.
@@ -75,32 +75,33 @@ API endpoints to obtain the output.
**Throws**:
-- [`ApifyCallError`](/docs/typedefs/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.
+- [`ApifyCallError`](/docs/api/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.
**Params**
-- **`actId`**: `String` - Either `username/actor-name` or actor ID.
-- **`[input]`**: `Object` | `String` | `Buffer` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set
- to `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.
-- **`[options]`**: `Object` - Object with the settings below:
- - **`[.contentType]`**: `String` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified
+- **`actId`**: `string` - Either `username/actor-name` or actor ID.
+- **`[input]`**: `object` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set to
+ `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.
+- **`[options]`**: `Object` <code> = {}</code> - Object with the settings below:
+ - **`[.contentType]`**: `string` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified
to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or
`Buffer`.
- - **`[.token]`**: `String` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.
- - **`[.memoryMbytes]`**: `Number` - Memory in megabytes which will be allocated for the new actor run. If not provided, the run uses memory of
+ - **`[.token]`**: `string` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.
+ - **`[.memoryMbytes]`**: `number` - Memory in megabytes which will be allocated for the new actor run. If not provided, the run uses memory of
the default actor run configuration.
- - **`[.timeoutSecs]`**: `Number` - Timeout for the actor run in seconds. Zero value means there is no timeout. If not provided, the run uses
+ - **`[.timeoutSecs]`**: `number` - Timeout for the actor run in seconds. Zero value means there is no timeout. If not provided, the run uses
timeout of the default actor run configuration.
- - **`[.build]`**: `String` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or
+ - **`[.build]`**: `string` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or
number from the default actor run configuration (typically `latest`).
- - **`[.waitSecs]`**: `String` - Maximum time to wait for the actor run to finish, in seconds. If the limit is reached, the returned promise is
+ - **`[.waitSecs]`**: `string` - Maximum time to wait for the actor run to finish, in seconds. If the limit is reached, the returned promise is
resolved to a run object that will have status `READY` or `RUNNING` and it will not contain the actor run output. If `waitSecs` is null or
undefined, the function waits for the actor to finish (default behavior).
- - **`[.fetchOutput]`**: `Boolean` <code> = true</code> - If `false` then the function does not fetch output of the actor.
- - **`[.disableBodyParser]`**: `Boolean` <code> = false</code> - If `true` then the function will not attempt to parse the actor's output and
+ - **`[.fetchOutput]`**: `boolean` <code> = true</code> - If `false` then the function does not fetch output of the actor.
+ - **`[.disableBodyParser]`**: `boolean` <code> = false</code> - If `true` then the function will not attempt to parse the actor's output and
will return it in a raw `Buffer`.
- - **`[.webhooks]`**: `Array` - Specifies optional webhooks associated with the actor run, which can be used to receive a notification e.g. when
- the actor finished or failed, see [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.
+ - **`[.webhooks]`**: `Array<object>` - Specifies optional webhooks associated with the actor run, which can be used to receive a notification
+ e.g. when the actor finished or failed, see [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed
+ description.
---
@@ -117,7 +118,7 @@ By passing the `waitSecs` option you can reduce the maximum amount of time to wa
the function returns immediately after the run is started.
The result of the function is an [`ActorRun`](/docs/typedefs/actor-run) object that contains details about the actor run and its output (if any). If
-the actor run failed, the function fails with [`ApifyCallError`](/docs/typedefs/apify-call-error) exception.
+the actor run failed, the function fails with [`ApifyCallError`](/docs/api/apify-call-error) exception.
Note that an actor task is a saved input configuration and options for an actor. If you want to run an actor directly rather than an actor task,
please use the [`Apify.call()`](/docs/api/apify#call) function instead.
@@ -136,30 +137,31 @@ other API endpoints to obtain the output.
**Throws**:
-- [`ApifyCallError`](/docs/typedefs/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.
+- [`ApifyCallError`](/docs/api/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.
**Params**
-- **`taskId`**: `String` - Either `username/task-name` or task ID.
-- **`[input]`**: `Object` | `String` | `Buffer` - Input overrides for the actor task. If it is an object, it will be stringified to JSON and its
- content type set to `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided. Provided input will be
- merged with actor task input.
-- **`[options]`**: `Object` - Object with the settings below:
- - **`[.contentType]`**: `String` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified
+- **`taskId`**: `string` - Either `username/task-name` or task ID.
+- **`[input]`**: `object` - Input overrides for the actor task. If it is an object, it will be stringified to JSON and its content type set to
+ `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided. Provided input will be merged with actor task
+ input.
+- **`[options]`**: `Object` <code> = {}</code> - Object with the settings below:
+ - **`[.contentType]`**: `string` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified
to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or
`Buffer`.
- - **`[.token]`**: `String` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.
- - **`[.memoryMbytes]`**: `Number` - Memory in megabytes which will be allocated for the new actor task run. If not provided, the run uses memory
+ - **`[.token]`**: `string` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.
+ - **`[.memoryMbytes]`**: `number` - Memory in megabytes which will be allocated for the new actor task run. If not provided, the run uses memory
of the default actor run configuration.
- - **`[.timeoutSecs]`**: `Number` - Timeout for the actor task run in seconds. Zero value means there is no timeout. If not provided, the run
+ - **`[.timeoutSecs]`**: `number` - Timeout for the actor task run in seconds. Zero value means there is no timeout. If not provided, the run
uses timeout of the default actor run configuration.
- - **`[.build]`**: `String` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or
+ - **`[.build]`**: `string` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or
number from the default actor run configuration (typically `latest`).
- - **`[.waitSecs]`**: `String` - Maximum time to wait for the actor task run to finish, in seconds. If the limit is reached, the returned promise
+ - **`[.waitSecs]`**: `string` - Maximum time to wait for the actor task run to finish, in seconds. If the limit is reached, the returned promise
is resolved to a run object that will have status `READY` or `RUNNING` and it will not contain the actor run output. If `waitSecs` is null or
undefined, the function waits for the actor task to finish (default behavior).
- - **`[.webhooks]`**: `Array` - Specifies optional webhooks associated with the actor run, which can be used to receive a notification e.g. when
- the actor finished or failed, see [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.
+ - **`[.webhooks]`**: `Array<object>` - Specifies optional webhooks associated with the actor run, which can be used to receive a notification
+ e.g. when the actor finished or failed, see [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed
+ description.
---
@@ -248,9 +250,9 @@ The following table shows all currently emitted events:
<a name="getapifyproxyurl"></a>
-## `Apify.getApifyProxyUrl(options)`
+## `Apify.getApifyProxyUrl([options])`
-**Returns**: `String` - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
+**Returns**: `string` - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
Constructs an Apify Proxy URL using the specified settings. The proxy URL can be used from Apify actors, web browsers or any other HTTP proxy-enabled
applications.
@@ -259,20 +261,21 @@ For more information, see the [Apify Proxy](https://my.apify.com/proxy) page in
**Params**
-- **`options`**: `Object` - Object with the settings below: - **`[.password]`**: `String` - User's password for the proxy. By default, it is taken
- from the `APIFY_PROXY_PASSWORD` environment variable, which is automatically set by the system when running the actors on the Apify cloud, or when
- using the [Apify CLI](https://github.com/apifytech/apify-cli) package and the user previously logged in (called `apify login`). - **`[.groups]`**:
- `Array<String>` - Array of Apify Proxy groups to be used. If not provided, the proxy will select the groups automatically. - **`[.session]`**:
- `String` - Apify Proxy session identifier to be used by the Chrome browser. All HTTP requests going through the proxy with the same session
- identifier will use the same target proxy server (i.e. the same IP address), unless using Residential proxies. The identifier can only contain the
- following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`. - **`[.country]`**: `String` - If specified, all proxied requests will use IP
- addresses that are geolocated to the specified country. For example `GB` for IPs from Great Britain. Note that online services often have their
- own rules for handling geolocation and thus the country selection is a best attempt at geolocation, rather than a guaranteed hit. This parameter
- is optional, by default, each proxied request is assigned an IP address from a random country. The country code needs to be a two letter ISO
- country code \- see the <a href="https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements" target="_blank"> full list of
- available country codes </a>.
-
-This parameter is optional, by default, the proxy uses all available proxy servers from all countries.
+- **`[options]`**: `Object` - Object with the props below:
+ - **`[.password]`**: `string` - User's password for the proxy. By default, it is taken from the `APIFY_PROXY_PASSWORD` environment variable,
+ which is automatically set by the system when running the actors on the Apify cloud, or when using the
+ [Apify CLI](https://github.com/apifytech/apify-cli) package and the user previously logged in (called `apify login`).
+ - **`[.groups]`**: `Array<string>` - Array of Apify Proxy groups to be used. If not provided, the proxy will select the groups automatically.
+ - **`[.session]`**: `string` - Apify Proxy session identifier to be used by the Chrome browser. All HTTP requests going through the proxy with
+ the same session identifier will use the same target proxy server (i.e. the same IP address), unless using Residential proxies. The identifier
+ can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
+ - **`[.country]`**: `string` - If set and relevant proxies are available in your Apify account, all proxied requests will use IP addresses that
+ are geolocated to the specified country. For example `GB` for IPs from Great Britain. Note that online services often have their own rules for
+ handling geolocation and thus the country selection is a best attempt at geolocation, rather than a guaranteed hit. This parameter is
+ optional, by default, each proxied request is assigned an IP address from a random country. The country code needs to be a two letter ISO
+ country code. See the
+ [full list of available country codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements). This parameter is
+ optional, by default, the proxy uses all available proxy servers from all countries.
---
@@ -293,7 +296,7 @@ the variables are not defined or are invalid, the corresponding value in the res
## `Apify.getInput()`
-**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.
+**Returns**: `Promise<(object|null)>` - Returns a promise that resolves once the record is stored.
Gets the actor input value from the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.
@@ -334,7 +337,7 @@ second.
## `Apify.getValue(key)`
-**Returns**: `Promise<Object>` - Returns a promise that resolves once the record is stored.
+**Returns**: `Promise<(object|null)>` - Returns a promise that resolves once the record is stored.
Gets a value from the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.
@@ -358,7 +361,7 @@ For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyv
**Params**
-- **`key`**: `String` - Unique record key.
+- **`key`**: `string` - Unique record key.
---
@@ -366,7 +369,7 @@ For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyv
## `Apify.isAtHome()`
-**Returns**: `Boolean`
+**Returns**: `boolean`
Returns `true` when code is running on Apify platform and `false` otherwise (for example locally).
@@ -498,8 +501,8 @@ Apify.main(async () => {
**Params**
-- **`userFunc`**: `function` - User function to be executed. If it returns a promise, the promise will be awaited. The user function is called with
- no arguments.
+- **`userFunc`**: [`UserFunc`](/docs/typedefs/user-func) - User function to be executed. If it returns a promise, the promise will be awaited. The
+ user function is called with no arguments.
---
@@ -514,14 +517,14 @@ default storages are preserved and the new input is stored under the `INPUT-META
**Params**
-- **`targetActorId`**: `String` - Either `username/actor-name` or actor ID of an actor to which we want to metamorph.
-- **`[input]`**: `Object` | `String` | `Buffer` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set
- to `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.
-- **`[options]`**: `Object` - Object with the settings below:
- - **`[.contentType]`**: `String` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified
+- **`targetActorId`**: `string` - Either `username/actor-name` or actor ID of an actor to which we want to metamorph.
+- **`[input]`**: `object` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set to
+ `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.
+- **`[options]`**: `Object` <code> = {}</code> - Object with the settings below:
+ - **`[.contentType]`**: `string` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified
to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or
`Buffer`.
- - **`[.build]`**: `String` - Tag or number of the target actor build to metamorph into (e.g. `beta` or `1.2.345`). If not provided, the run uses
+ - **`[.build]`**: `string` - Tag or number of the target actor build to metamorph into (e.g. `beta` or `1.2.345`). If not provided, the run uses
build tag or number from the default actor run configuration (typically `latest`).
---
@@ -543,7 +546,7 @@ For more details and code examples, see the [`Dataset`](/docs/api/dataset) class
- **`[datasetIdOrName]`**: `string` - ID or name of the dataset to be opened. If `null` or `undefined`, the function returns the default dataset
associated with the actor run.
-- **`[options]`**: `object`
+- **`[options]`**: `Object`
- **`[.forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the
`APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.
@@ -605,7 +608,7 @@ const requestList = await Apify.openRequestList('my-name', sources);
If `null`, the list will not be persisted and will only be stored in memory. Process restart will then cause the list to be crawled again from the
beginning. We suggest always using a name.
-- **`sources`**: [`Array<(Request|RequestOptions|string)>`](/docs/api/request) - An array of sources of URLs for the
+- **`sources`**: [`SourceInput`](/docs/typedefs/source-input) | `Array<string>` - An array of sources of URLs for the
[`RequestList`](/docs/api/request-list). It can be either an array of plain objects that define at least the `url` property, or an array of
instances of the [`Request`](/docs/api/request) class.
@@ -666,7 +669,7 @@ For more details and code examples, see the [`SessionPool`](/docs/api/session-po
## `Apify.pushData(item)`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Stores an object or an array of objects to the default [`Dataset`](/docs/api/dataset) of the current actor run.
@@ -689,8 +692,8 @@ For more information, see [`Apify.openDataset()`](/docs/api/apify#opendataset) a
**Params**
-- **`item`**: `Object` | `Array` - Object or array of objects containing data to be stored in the default dataset. The objects must be serializable
- to JSON and the JSON representation of each object must be smaller than 9MB.
+- **`item`**: `object` - Object or array of objects containing data to be stored in the default dataset. The objects must be serializable to JSON
+ and the JSON representation of each object must be smaller than 9MB.
---
@@ -698,7 +701,7 @@ For more information, see [`Apify.openDataset()`](/docs/api/apify#opendataset) a
## `Apify.setValue(key, value, [options])`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Stores or deletes a value in the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.
@@ -722,8 +725,8 @@ For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyv
**Params**
-- **`key`**: `String` - Unique record key.
-- **`value`**: `Object` | `String` | `Buffer` - Record data, which can be one of the following values:
+- **`key`**: `string` - Unique record key.
+- **`value`**: `object` - Record data, which can be one of the following values:
<ul>
<li>If `null`, the record in the key-value store is deleted.</li>
<li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
@@ -732,6 +735,6 @@ For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyv
</ul>
For any other value an error will be thrown.
- **`[options]`**: `Object`
- - **`[.contentType]`**: `String` - Specifies a custom MIME content type of the record.
+ - **`[.contentType]`**: `string` - Specifies a custom MIME content type of the record.
---
diff --git a/docs/api/ApifyCallError.md b/docs/api/ApifyCallError.md
new file mode 100644
index 000000000..1d4b3b8f7
--- /dev/null
+++ b/docs/api/ApifyCallError.md
@@ -0,0 +1,45 @@
+---
+id: apify-call-error
+title: ApifyCallError
+---
+
+<a name="apifycallerror"></a>
+
+The class represents exceptions thrown by the [`Apify.call()`](/docs/api/apify#call) function.
+
+## Properties
+
+### `message`
+
+**Type**: `string`
+
+Error message
+
+---
+
+### `run`
+
+**Type**: [`ActorRun`](/docs/typedefs/actor-run)
+
+Object representing the failed actor run.
+
+---
+
+### `name`
+
+**Type**: `string`
+
+Contains `"ApifyCallError"`
+
+---
+
+<a name="exports.apifycallerror"></a>
+
+## `new ApifyCallError(run, [message])`
+
+**Params**
+
+- **`run`**: [`ActorRun`](/docs/typedefs/actor-run)
+- **`[message]`**: `string` <code> = &quot;The actor invoked by Apify.call() did not succeed&quot;</code>
+
+---
diff --git a/docs/api/AutoscaledPool.md b/docs/api/AutoscaledPool.md
index cd8b64236..b7db8d54a 100644
--- a/docs/api/AutoscaledPool.md
+++ b/docs/api/AutoscaledPool.md
@@ -144,7 +144,7 @@ Gets the the number of parallel tasks currently running in the pool.
## `autoscaledPool.run()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once all the tasks are finished or one of them fails.
@@ -154,7 +154,7 @@ Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once
## `autoscaledPool.abort()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Aborts the run of the auto-scaled pool and destroys it. The promise returned from the [`AutoscaledPool.run()`](/docs/api/autoscaled-pool#run) function
will immediately resolve, no more new tasks will be spawned and all running tasks will be left in their current state.
@@ -169,7 +169,7 @@ but that does not mean that some parts of their asynchronous chains of commands
## `autoscaledPool.pause([timeoutSecs])`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish (unlike abort, which terminates them). Used together with
[`AutoscaledPool.resume()`](/docs/api/autoscaled-pool#resume)
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index 9a55cd956..9350ad0d3 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -57,6 +57,17 @@ const crawler = new Apify.BasicCrawler({
await crawler.run();
```
+## Properties
+
+### `autoscaledPool`
+
+**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)
+
+A reference to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property
+is only initialized after calling the [`BasicCrawler.run()`](/docs/api/basic-crawler#run) function. You can use it to change the concurrency settings
+on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause) or to abort it by calling
+[`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort).
+
---
<a name="basiccrawler"></a>
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index aeac05560..d3df7ad1f 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -78,6 +78,17 @@ const crawler = new Apify.CheerioCrawler({
await crawler.run();
```
+## Properties
+
+### `autoscaledPool`
+
+**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)
+
+A reference to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property
+is only initialized after calling the [`CheerioCrawler.run()`](/docs/api/cheerio-crawler#run) function. You can use it to change the concurrency
+settings on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause) or to abort it by calling
+[`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort).
+
---
<a name="cheeriocrawler"></a>
@@ -95,7 +106,7 @@ await crawler.run();
## `cheerioCrawler.run()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Runs the crawler. Returns promise that gets resolved once all the requests got processed.
diff --git a/docs/api/CookieParseError.md b/docs/api/CookieParseError.md
new file mode 100644
index 000000000..73d38c734
--- /dev/null
+++ b/docs/api/CookieParseError.md
@@ -0,0 +1,18 @@
+---
+id: cookie-parse-error
+title: CookieParseError
+---
+
+<a name="cookieparseerror"></a>
+
+---
+
+<a name="exports.cookieparseerror"></a>
+
+## `new CookieParseError(cookieHeaderString)`
+
+**Params**
+
+- **`cookieHeaderString`**: `string`
+
+---
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.md
index 8cceb2097..c5c336e0e 100644
--- a/docs/api/Dataset.md
+++ b/docs/api/Dataset.md
@@ -79,8 +79,8 @@ the data from this item onwards to prevent duplicates.
**Params**
-- **`data`**: `Object` | `Array` - Object or array of objects containing data to be stored in the default dataset. The objects must be serializable
- to JSON and the JSON representation of each object must be smaller than 9MB.
+- **`data`**: `object` | `Array<object>` - Object or array of objects containing data to be stored in the default dataset. The objects must be
+ serializable to JSON and the JSON representation of each object must be smaller than 9MB.
---
@@ -98,37 +98,38 @@ Returns {DatasetContent} object holding the items in the dataset based on the pr
**Params**
- **`[options]`**: `Object` - All `getData()` parameters are passed via an options object with the following keys:
- - **`[.format]`**: `String` <code> = &#x27;json&#x27;</code> - Format of the `items` property, possible values are: `json`, `csv`, `xlsx`,
- `html`, `xml` and `rss`.
- - **`[.offset]`**: `Number` <code> = 0</code> - Number of array elements that should be skipped at the start.
- - **`[.limit]`**: `Number` <code> = 250000</code> - Maximum number of array elements to return.
- - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by `createdAt` in descending order. Otherwise they are
+ - **`[.format]`**: `string` <code> = &quot;&#x27;json&#x27;&quot;</code> - Format of the `items` property, possible values are: `json`, `csv`,
+ `xlsx`, `html`, `xml` and `rss`.
+ - **`[.offset]`**: `number` <code> = 0</code> - Number of array elements that should be skipped at the start.
+ - **`[.limit]`**: `number` <code> = 250000</code> - Maximum number of array elements to return.
+ - **`[.desc]`**: `boolean` <code> = false</code> - If `true` then the objects are sorted by `createdAt` in descending order. Otherwise they are
sorted in ascending order.
- - **`[.fields]`**: `Array` - An array of field names that will be included in the result. If omitted, all fields are included in the results.
- - **`[.unwind]`**: `String` - Specifies a name of the field in the result objects that will be used to unwind the resulting objects. By default,
+ - **`[.fields]`**: `Array<string>` - An array of field names that will be included in the result. If omitted, all fields are included in the
+ results.
+ - **`[.unwind]`**: `string` - Specifies a name of the field in the result objects that will be used to unwind the resulting objects. By default,
the results are returned as they are.
- - **`[.disableBodyParser]`**: `Boolean` <code> = false</code> - If `true` then response from API will not be parsed.
- - **`[.attachment]`**: `Boolean` <code> = false</code> - If `true` then the response will define the `Content-Disposition: attachment` HTTP
+ - **`[.disableBodyParser]`**: `boolean` <code> = false</code> - If `true` then response from API will not be parsed.
+ - **`[.attachment]`**: `boolean` <code> = false</code> - If `true` then the response will define the `Content-Disposition: attachment` HTTP
header, forcing a web browser to download the file rather than to display it. By default, this header is not present.
- - **`[.delimiter]`**: `String` <code> = &#x27;,&#x27;</code> - A delimiter character for CSV files, only used if `format` is `csv`.
- - **`[.bom]`**: `Boolean` - All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte Order Mark
+ - **`[.delimiter]`**: `string` <code> = &quot;&#x27;,&#x27;&quot;</code> - A delimiter character for CSV files, only used if `format` is `csv`.
+ - **`[.bom]`**: `boolean` - All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte Order Mark
(BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default behavior, set `bom` option to `true` to
include the BOM, or set `bom` to `false` to skip it.
- - **`[.xmlRoot]`**: `String` <code> = &#x27;results&#x27;</code> - Overrides the default root element name of the XML output. By default, the
- root element is `results`.
- - **`[.xmlRow]`**: `String` <code> = &#x27;page&#x27;</code> - Overrides the default element name that wraps each page or page function result
- object in XML output. By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
- - **`[.skipHeaderRow]`**: `Boolean` <code> = false</code> - If set to `true` then header row in CSV format is skipped.
- - **`[.clean]`**: `Boolean` <code> = false</code> - If `true` then the function returns only non-empty items and skips hidden fields (i.e.
+ - **`[.xmlRoot]`**: `string` <code> = &quot;&#x27;results&#x27;&quot;</code> - Overrides the default root element name of the XML output. By
+ default, the root element is `results`.
+ - **`[.xmlRow]`**: `string` <code> = &quot;&#x27;page&#x27;&quot;</code> - Overrides the default element name that wraps each page or page
+ function result object in XML output. By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
+ - **`[.skipHeaderRow]`**: `boolean` <code> = false</code> - If set to `true` then header row in CSV format is skipped.
+ - **`[.clean]`**: `boolean` <code> = false</code> - If `true` then the function returns only non-empty items and skips hidden fields (i.e.
fields starting with `#` character). Note that the `clean` parameter is a shortcut for `skipHidden: true` and `skipEmpty: true` options.
- - **`[.skipHidden]`**: `Boolean` <code> = false</code> - If `true` then the function doesn't return hidden fields (fields starting with "#"
+ - **`[.skipHidden]`**: `boolean` <code> = false</code> - If `true` then the function doesn't return hidden fields (fields starting with "#"
character).
- - **`[.skipEmpty]`**: `Boolean` <code> = false</code> - If `true` then the function doesn't return empty items. Note that in this case the
+ - **`[.skipEmpty]`**: `boolean` <code> = false</code> - If `true` then the function doesn't return empty items. Note that in this case the
returned number of items might be lower than limit parameter and pagination must be done using the `limit` value.
- - **`[.simplified]`**: `Boolean` - If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and
+ - **`[.simplified]`**: `boolean` - If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and
`unwind: 'pageFunctionResult'` options. This feature is used to emulate simplified results provided by Apify API version 1 used for the legacy
Apify Crawler and it's not recommended to use it in new integrations.
- - **`[.skipFailedPages]`**: `Boolean` - If `true` then, the all the items with errorInfo property will be skipped from the output. This feature
+ - **`[.skipFailedPages]`**: `boolean` - If `true` then, the all the items with errorInfo property will be skipped from the output. This feature
is here to emulate functionality of Apify API version 1 used for the legacy Apify Crawler product and it's not recommended to use it in new
integrations.
@@ -138,7 +139,7 @@ Returns {DatasetContent} object holding the items in the dataset based on the pr
## `dataset.getInfo()`
-**Returns**: `Promise<Object>`
+**Returns**: `Promise<object>`
Returns an object containing general information about the dataset.
@@ -188,18 +189,18 @@ await dataset.forEach(async (item, index) => {
- **`iteratee`**: [`DatasetConsumer`](/docs/typedefs/dataset-consumer) - A function that is called for every item in the dataset.
- **`[options]`**: `Object` - All `forEach()` parameters are passed via an options object with the following keys:
- - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by `createdAt` in descending order.
- - **`[.fields]`**: `Array` - If provided then returned objects will only contain specified keys.
- - **`[.unwind]`**: `String` - If provided then objects will be unwound based on provided field.
-- **`[index]`**: `Number` <code> = 0</code> - Specifies the initial index number passed to the `iteratee` function.
+ - **`[.desc]`**: `boolean` <code> = false</code> - If `true` then the objects are sorted by `createdAt` in descending order.
+ - **`[.fields]`**: `Array<string>` - If provided then returned objects will only contain specified keys.
+ - **`[.unwind]`**: `string` - If provided then objects will be unwound based on provided field.
+- **`[index]`**: `number` <code> = 0</code> - Specifies the initial index number passed to the `iteratee` function.
---
<a name="map"></a>
-## `dataset.map(iteratee, options)`
+## `dataset.map(iteratee, [options])`
-**Returns**: `Promise<Array<T>>`
+**Returns**: `Promise<Array<object>>`
Produces a new array of values by mapping each value in list through a transformation function `iteratee()`. Each invocation of `iteratee()` is called
with two arguments: `(element, index)`.
@@ -209,18 +210,18 @@ If `iteratee` returns a `Promise` then it's awaited before a next call.
**Params**
- **`iteratee`**: [`DatasetMapper`](/docs/typedefs/dataset-mapper)
-- **`options`**: `Object` - All `map()` parameters are passed via an options object with the following keys:
- - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by createdAt in descending order.
- - **`[.fields]`**: `Array` - If provided then returned objects will only contain specified keys
- - **`[.unwind]`**: `String` - If provided then objects will be unwound based on provided field.
+- **`[options]`**: `Object` - All `map()` parameters are passed via an options object with the following keys:
+ - **`[.desc]`**: `boolean` <code> = false</code> - If `true` then the objects are sorted by createdAt in descending order.
+ - **`[.fields]`**: `Array<string>` - If provided then returned objects will only contain specified keys
+ - **`[.unwind]`**: `string` - If provided then objects will be unwound based on provided field.
---
<a name="reduce"></a>
-## `dataset.reduce(iteratee, memo, options)`
+## `dataset.reduce(iteratee, memo, [options])`
-**Returns**: `Promise<T>`
+**Returns**: `Promise<object>`
Reduces a list of values down to a single value.
@@ -235,11 +236,11 @@ If `iteratee()` returns a `Promise` then it's awaited before a next call.
**Params**
- **`iteratee`**: [`DatasetReducer`](/docs/typedefs/dataset-reducer)
-- **`memo`**: `T` - Initial state of the reduction.
-- **`options`**: `Object` - All `reduce()` parameters are passed via an options object with the following keys:
- - **`[.desc]`**: `Boolean` <code> = false</code> - If `true` then the objects are sorted by createdAt in descending order.
- - **`[.fields]`**: `Array` - If provided then returned objects will only contain specified keys
- - **`[.unwind]`**: `String` - If provided then objects will be unwound based on provided field.
+- **`memo`**: `object` - Initial state of the reduction.
+- **`[options]`**: `Object` - All `reduce()` parameters are passed via an options object with the following keys:
+ - **`[.desc]`**: `boolean` <code> = false</code> - If `true` then the objects are sorted by createdAt in descending order.
+ - **`[.fields]`**: `Array<string>` - If provided then returned objects will only contain specified keys
+ - **`[.unwind]`**: `string` - If provided then objects will be unwound based on provided field.
---
diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.md
index f0702dec7..d110b2dd7 100644
--- a/docs/api/KeyValueStore.md
+++ b/docs/api/KeyValueStore.md
@@ -80,7 +80,7 @@ await store.drop();
## `keyValueStore.getValue(key)`
-**Returns**: `Promise<(Object|String|Buffer)>` - Returns a promise that resolves to an object, string or [](https://nodejs.org/api/buffer.html),
+**Returns**: `Promise<(object|Buffer|string|null)>` - Returns a promise that resolves to an object, string or [](https://nodejs.org/api/buffer.html),
depending on the MIME content type of the record.
Gets a value from the key-value store.
@@ -102,7 +102,7 @@ const buffer = await store.getValue('screenshot1.png');
**Params**
-- **`key`**: `String` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,
+- **`key`**: `string` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,
`A`-`Z`, `0`-`9` and `!-_.'()`
---
@@ -143,9 +143,9 @@ stored!
**Params**
-- **`key`**: `String` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,
+- **`key`**: `string` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,
`A`-`Z`, `0`-`9` and `!-_.'()`
-- **`value`**: `Object` | `String` | `Buffer` - Record data, which can be one of the following values:
+- **`value`**: `Object` | `string` | `Buffer` | `null` - Record data, which can be one of the following values:
<ul>
<li>If `null`, the record in the key-value store is deleted.</li>
<li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
@@ -154,7 +154,7 @@ stored!
</ul>
For any other value an error will be thrown.
- **`[options]`**: `Object`
- - **`[.contentType]`**: `String` - Specifies a custom MIME content type of the record.
+ - **`[.contentType]`**: `string` - Specifies a custom MIME content type of the record.
---
diff --git a/docs/api/LiveViewServer.md b/docs/api/LiveViewServer.md
index 26d19551a..6af51a82a 100644
--- a/docs/api/LiveViewServer.md
+++ b/docs/api/LiveViewServer.md
@@ -58,7 +58,7 @@ but `LiveViewServer` works too.
## `liveViewServer.start()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Starts the HTTP server with web socket connections enabled. Snapshots will not be created until a client has connected.
@@ -68,7 +68,7 @@ Starts the HTTP server with web socket connections enabled. Snapshots will not b
## `liveViewServer.stop()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Prevents the server from receiving more connections. Existing connections will not be terminated, but the server will not prevent a process exit.
@@ -78,7 +78,7 @@ Prevents the server from receiving more connections. Existing connections will n
## `liveViewServer.serve(page)`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Serves a snapshot to all connected clients. Screenshots are not served directly, only their index number which is used by client to retrieve the
screenshot.
diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.md
index 5592be50c..074cf2125 100644
--- a/docs/api/PseudoUrl.md
+++ b/docs/api/PseudoUrl.md
@@ -60,7 +60,7 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M
**Params**
-- **`purl`**: `String` | `RegExp` - A pseudo-URL string or a regular expression object. Using a `RegExp` instance enables more granular control,
+- **`purl`**: `string` | `RegExp` - A pseudo-URL string or a regular expression object. Using a `RegExp` instance enables more granular control,
such as making the matching case sensitive.
- **`requestTemplate`**: [`RequestOptions`](/docs/typedefs/request-options) - Options for the new [`Request`](/docs/api/request) instances created
for matching URLs by the [`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks) function.
@@ -71,13 +71,13 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M
## `pseudoUrl.matches(url)`
-**Returns**: `Boolean` - Returns `true` if given URL matches pseudo-URL.
+**Returns**: `boolean` - Returns `true` if given URL matches pseudo-URL.
Determines whether a URL matches this pseudo-URL pattern.
**Params**
-- **`url`**: `String` - URL to be matched.
+- **`url`**: `string` - URL to be matched.
---
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index 88ee0f50e..8d9faa281 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -66,6 +66,17 @@ const crawler = new Apify.PuppeteerCrawler({
await crawler.run();
```
+## Properties
+
+### `autoscaledPool`
+
+**Type**: [`AutoscaledPool`](/docs/api/autoscaled-pool)
+
+A reference to the underlying [`AutoscaledPool`](/docs/api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property
+is only initialized after calling the [`PuppeteerCrawler.run()`](/docs/api/puppeteer-crawler#run) function. You can use it to change the concurrency
+settings on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](/docs/api/autoscaled-pool#pause) or to abort it by calling
+[`AutoscaledPool.abort()`](/docs/api/autoscaled-pool#abort).
+
---
<a name="puppeteercrawler"></a>
diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.md
index 45f6d01d8..9b3395461 100644
--- a/docs/api/PuppeteerPool.md
+++ b/docs/api/PuppeteerPool.md
@@ -117,6 +117,6 @@ Tells the connected LiveViewServer to serve a snapshot when available.
**Params**
-- **`page`**:
+- **`page`**: `Page`
---
diff --git a/docs/api/Request.md b/docs/api/Request.md
index 445833842..a8f6a582b 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -34,7 +34,7 @@ const foo = request.userData.foo;
### `id`
-**Type**: `String`
+**Type**: `string`
Request ID
@@ -42,7 +42,7 @@ Request ID
### `url`
-**Type**: `String`
+**Type**: `string`
URL of the web page to crawl.
@@ -50,7 +50,7 @@ URL of the web page to crawl.
### `loadedUrl`
-**Type**: `String`
+**Type**: `string`
An actually loaded URL after redirects, if present. HTTP redirects are guaranteed to be included.
@@ -61,7 +61,7 @@ nature. This generally means that redirects, which happen immediately will most
### `uniqueKey`
-**Type**: `String`
+**Type**: `string`
A unique key identifying the request. Two requests with the same `uniqueKey` are considered as pointing to the same URL.
@@ -69,7 +69,7 @@ A unique key identifying the request. Two requests with the same `uniqueKey` are
### `method`
-**Type**: `String`
+**Type**: `string`
HTTP method, e.g. `GET` or `POST`.
@@ -77,7 +77,7 @@ HTTP method, e.g. `GET` or `POST`.
### `payload`
-**Type**: `String` | `Buffer`
+**Type**: `string` | `Buffer`
HTTP request payload, e.g. for POST requests.
@@ -85,7 +85,7 @@ HTTP request payload, e.g. for POST requests.
### `noRetry`
-**Type**: `Boolean`
+**Type**: `boolean`
The `true` value indicates that the request will not be automatically retried on error.
@@ -93,7 +93,7 @@ The `true` value indicates that the request will not be automatically retried on
### `retryCount`
-**Type**: `Number`
+**Type**: `number`
Indicates the number of times the crawling of the request has been retried on error.
@@ -101,7 +101,7 @@ Indicates the number of times the crawling of the request has been retried on er
### `errorMessages`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
An array of error messages from request processing.
@@ -117,7 +117,7 @@ Object with HTTP headers. Key is header name, value is the value.
### `userData`
-**Type**: `Object`
+**Type**: `object`
Custom user data assigned to the request.
@@ -155,8 +155,8 @@ debugging rather difficult.
**Params**
-- **`errorOrMessage`**: `Error` | `String` - Error object or error message to be stored in the request.
+- **`errorOrMessage`**: `Error` | `string` - Error object or error message to be stored in the request.
- **`[options]`**: `Object`
- - **`[.omitStack]`**: `Boolean` <code> = false</code> - Only push the error message without stack trace when true.
+ - **`[.omitStack]`**: `boolean` <code> = false</code> - Only push the error message without stack trace when true.
---
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index c14c6f55e..ea99b1655 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -134,7 +134,7 @@ Returns `true` if all requests were already handled and there are no more left.
## `requestList.fetchNextRequest()`
-**Returns**: [`Promise<Request>`](/docs/api/request)
+**Returns**: [`Promise<(Request|null)>`](/docs/api/request)
Gets the next [`Request`](/docs/api/request) to process. First, the function gets a request previously reclaimed using the
[`RequestList.reclaimRequest()`](/docs/api/request-list#reclaimrequest) function, if there is any. Otherwise it gets the next request from sources.
@@ -175,7 +175,7 @@ Reclaims request to the list if its processing failed. The request will become a
## `requestList.length()`
-**Returns**: `Number`
+**Returns**: `number`
Returns the total number of unique requests present in the `RequestList`.
@@ -185,7 +185,7 @@ Returns the total number of unique requests present in the `RequestList`.
## `requestList.handledCount()`
-**Returns**: `Number`
+**Returns**: `number`
Returns number of handled requests.
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index 90ba812b0..8615c5043 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -66,6 +66,18 @@ await queue.reclaimRequest(request2);
---
+<a name="exports.requestqueue"></a>
+
+## `new RequestQueue(queueId, [queueName], [clientKey])`
+
+**Params**
+
+- **`queueId`**: `string`
+- **`[queueName]`**: `string`
+- **`[clientKey]`**: `string`
+
+---
+
<a name="addrequest"></a>
## `requestQueue.addRequest(request, [options])`
@@ -85,7 +97,7 @@ function.
- **`request`**: [`Request`](/docs/api/request) | [`RequestOptions`](/docs/typedefs/request-options) - [`Request`](/docs/api/request) object or
vanilla object with request data. Note that the function sets the `uniqueKey` and `id` fields to the passed object.
- **`[options]`**: `Object`
- - **`[.forefront]`**: `Boolean` <code> = false</code> - If `true`, the request will be added to the foremost position in the queue.
+ - **`[.forefront]`**: `boolean` <code> = false</code> - If `true`, the request will be added to the foremost position in the queue.
---
@@ -93,13 +105,13 @@ function.
## `requestQueue.getRequest(requestId)`
-**Returns**: [`Promise<Request>`](/docs/api/request) - Returns the request object, or `null` if it was not found.
+**Returns**: [`Promise<(Request|null)>`](/docs/api/request) - Returns the request object, or `null` if it was not found.
Gets the request from the queue specified by ID.
**Params**
-- **`requestId`**: `String` - ID of the request.
+- **`requestId`**: `string` - ID of the request.
---
@@ -107,7 +119,7 @@ Gets the request from the queue specified by ID.
## `requestQueue.fetchNextRequest()`
-**Returns**: [`Promise<Request>`](/docs/api/request) - Returns the request object or `null` if there are no more pending requests.
+**Returns**: [`Promise<(Request|null)>`](/docs/api/request) - Returns the request object or `null` if there are no more pending requests.
Returns a next request in the queue to be processed, or `null` if there are no more pending requests.
@@ -149,7 +161,7 @@ parameter. For example, this lets you store the number of retries or error messa
**Params**
- **`request`**: [`Request`](/docs/api/request)
-- **`[options]`**: `Object` - **`[.forefront]`**: `Boolean` <code> = false</code> - If `true` then the request it placed to the beginning of the
+- **`[options]`**: `Object` - **`[.forefront]`**: `boolean` <code> = false</code> - If `true` then the request it placed to the beginning of the
queue, so that it's returned in the next call to [`RequestQueue.fetchNextRequest()`](/docs/api/request-queue#fetchnextrequest). By default, it's
put to the end of the queue.
@@ -159,7 +171,7 @@ parameter. For example, this lets you store the number of retries or error messa
## `requestQueue.isEmpty()`
-**Returns**: `Promise<Boolean>`
+**Returns**: `Promise<boolean>`
Resolves to `true` if the next call to [`RequestQueue.fetchNextRequest()`](/docs/api/request-queue#fetchnextrequest) would return `null`, otherwise it
resolves to `false`. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that
@@ -171,7 +183,7 @@ there is no activity in the queue, use [`RequestQueue.isFinished()`](/docs/api/r
## `requestQueue.isFinished()`
-**Returns**: `Promise<Boolean>`
+**Returns**: `Promise<boolean>`
Resolves to `true` if all requests were already handled and there are no more left. Due to the nature of distributed storage used by the queue, the
function might occasionally return a false negative, but it will never return a false positive.
@@ -182,7 +194,7 @@ function might occasionally return a false negative, but it will never return a
## `requestQueue.drop()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Removes the queue either from the Apify Cloud storage or from the local directory, depending on the mode of operation.
@@ -208,7 +220,7 @@ const { handledRequestCount } = await queue.getInfo();
## `requestQueue.getInfo()`
-**Returns**: `Promise<Object>`
+**Returns**: `Promise<object>`
Returns an object containing general information about the request queue.
diff --git a/docs/api/Session.md b/docs/api/Session.md
index d547689bc..934a3d57c 100644
--- a/docs/api/Session.md
+++ b/docs/api/Session.md
@@ -129,7 +129,7 @@ It then parses and saves the cookies from the `set-cookie` header, if available.
**Params**
-- **`response`**: `Object`
+- **`response`**: `PuppeteerResponse` | `IncomingMessage`
---
@@ -165,7 +165,7 @@ Returns cookies in a format compatible with puppeteer and ready to be used with
**Params**
-- **`url`**: `String` - website url. Only cookies stored for this url will be returned
+- **`url`**: `string` - website url. Only cookies stored for this url will be returned
---
diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.md
index 62b597286..3b453727d 100644
--- a/docs/api/SessionPool.md
+++ b/docs/api/SessionPool.md
@@ -108,8 +108,6 @@ the picked session is usable it is returned, otherwise it creates and returns a
## `sessionPool.getState()`
-**Returns**: `Object`
-
Returns an object representing the internal state of the `SessionPool` instance. Note that the object's fields can change in future releases.
---
@@ -118,7 +116,7 @@ Returns an object representing the internal state of the `SessionPool` instance.
## `sessionPool.persistState()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Persists the current state of the `SessionPool` into the default [`KeyValueStore`](/docs/api/key-value-store). The state is persisted automatically in
regular intervals.
diff --git a/docs/api/Snapshotter.md b/docs/api/Snapshotter.md
index 006442d8f..a76aacf75 100644
--- a/docs/api/Snapshotter.md
+++ b/docs/api/Snapshotter.md
@@ -39,7 +39,7 @@ set interval.
## `snapshotter.start()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Starts capturing snapshots at configured intervals.
@@ -49,7 +49,7 @@ Starts capturing snapshots at configured intervals.
## `snapshotter.stop()`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Stops all resource capturing.
@@ -59,14 +59,14 @@ Stops all resource capturing.
## `snapshotter.getMemorySample([sampleDurationMillis])`
-**Returns**: `Array`
+**Returns**: `Array<*>`
Returns a sample of latest memory snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full
snapshot history.
**Params**
-- **`[sampleDurationMillis]`**: `Number`
+- **`[sampleDurationMillis]`**: `number`
---
@@ -74,14 +74,14 @@ snapshot history.
## `snapshotter.getEventLoopSample([sampleDurationMillis])`
-**Returns**: `Array`
+**Returns**: `Array<*>`
Returns a sample of latest event loop snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a
full snapshot history.
**Params**
-- **`[sampleDurationMillis]`**: `Number`
+- **`[sampleDurationMillis]`**: `number`
---
@@ -89,14 +89,14 @@ full snapshot history.
## `snapshotter.getCpuSample([sampleDurationMillis])`
-**Returns**: `Array`
+**Returns**: `Array<*>`
Returns a sample of latest CPU snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full
snapshot history.
**Params**
-- **`[sampleDurationMillis]`**: `Number`
+- **`[sampleDurationMillis]`**: `number`
---
@@ -104,13 +104,13 @@ snapshot history.
## `snapshotter.getClientSample(sampleDurationMillis)`
-**Returns**: `Array`
+**Returns**: `Array<*>`
Returns a sample of latest Client snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full
snapshot history.
**Params**
-- **`sampleDurationMillis`**: `Number`
+- **`sampleDurationMillis`**: `number`
---
diff --git a/docs/api/SystemStatus.md b/docs/api/SystemStatus.md
index 0a503d5ee..e35db5b4c 100644
--- a/docs/api/SystemStatus.md
+++ b/docs/api/SystemStatus.md
@@ -56,7 +56,7 @@ otherwise.
## `systemStatus.getHistoricalStatus()`
-**Returns**: `Object`
+**Returns**: [`SystemInfo`](/docs/typedefs/system-info)
Returns an [`SystemInfo`](/docs/typedefs/system-info) object with the following structure:
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 9dc326b30..cca0c06a1 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -28,7 +28,7 @@ await puppeteer.injectJQuery(page);
## `puppeteer.addInterceptRequestHandler`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Adds request interception handler in similar to `page.on('request', handler);` but in addition to that supports multiple parallel handlers.
@@ -73,7 +73,7 @@ await page.goto('http://example.com');
**Params**
- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
-- **`handler`**: `function` - Request interception handler.
+- **`handler`**: [`InterceptHandler`](/docs/typedefs/intercept-handler) - Request interception handler.
---
@@ -81,14 +81,14 @@ await page.goto('http://example.com');
## `puppeteer.removeInterceptRequestHandler`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Removes request interception handler for given page.
**Params**
- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
-- **`handler`**: `function` - Request interception handler.
+- **`handler`**: [`InterceptHandler`](/docs/typedefs/intercept-handler) - Request interception handler.
---
@@ -96,7 +96,7 @@ Removes request interception handler for given page.
## `puppeteer.gotoExtended`
-**Returns**: `Promise<Response>`
+**Returns**: `Promise<(Response|null)>`
Extended version of Puppeteer's `page.goto()` allowing to perform requests with HTTP method other than GET, with custom headers and POST payload. URL,
method, headers and payload are taken from request parameter that must be an instance of Apify.Request class.
@@ -108,7 +108,7 @@ performance.
- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- **`request`**: [`Request`](/docs/api/request)
-- **`gotoOptions`**: `Object` - Custom options for `page.goto()`.
+- **`gotoOptions`**: `DirectNavigationOptions` - Custom options for `page.goto()`.
---
@@ -116,16 +116,16 @@ performance.
## `puppeteer.infiniteScroll`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.
**Params**
-- **`page`**: `Object` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
+- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- **`[options]`**: `Object`
- - **`[.timeoutSecs]`**: `Number` <code> = 0</code> - How many seconds to scroll for. If 0, will scroll until bottom of page.
- - **`[.waitForSecs]`**: `Number` <code> = 4</code> - How many seconds to wait for no new content to load before exit.
+ - **`[.timeoutSecs]`**: `number` <code> = 0</code> - How many seconds to scroll for. If 0, will scroll until bottom of page.
+ - **`[.waitForSecs]`**: `number` <code> = 4</code> - How many seconds to wait for no new content to load before exit.
---
@@ -133,24 +133,24 @@ Scrolls to the bottom of a page, or until it times out. Loads dynamic content wh
## `puppeteer.saveSnapshot`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Saves a full screenshot and HTML of the current page into a Key-Value store.
**Params**
-- **`page`**: `Object` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
+- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- **`[options]`**: `Object`
- - **`[.key]`**: `String` <code> = SNAPSHOT</code> - Key under which the screenshot and HTML will be saved. `.jpg` will be appended for
- screenshot and `.html` for HTML.
- - **`[.screenshotQuality]`**: `Number` <code> = 50</code> - The quality of the image, between 0-100. Higher quality images have bigger size and
+ - **`[.key]`**: `string` <code> = &quot;SNAPSHOT&quot;</code> - Key under which the screenshot and HTML will be saved. `.jpg` will be appended
+ for screenshot and `.html` for HTML.
+ - **`[.screenshotQuality]`**: `number` <code> = 50</code> - The quality of the image, between 0-100. Higher quality images have bigger size and
require more storage.
- - **`[.saveScreenshot]`**: `Boolean` <code> = true</code> - If true, it will save a full screenshot of the current page as a record with `key`
+ - **`[.saveScreenshot]`**: `boolean` <code> = true</code> - If true, it will save a full screenshot of the current page as a record with `key`
appended by `.jpg`.
- - **`[.saveHtml]`**: `Boolean` <code> = true</code> - If true, it will save a full HTML of the current page as a record with `key` appended by
+ - **`[.saveHtml]`**: `boolean` <code> = true</code> - If true, it will save a full HTML of the current page as a record with `key` appended by
`.html`.
- - **`[.keyValueStoreName]`**: `String` <code> = </code> - Name or id of the Key-Value store where snapshot is saved. By default it is saved to
- default Key-Value store.
+ - **`[.keyValueStoreName]`**: `string` <code> = null</code> - Name or id of the Key-Value store where snapshot is saved. By default it is saved
+ to default Key-Value store.
---
@@ -207,10 +207,10 @@ await Apify.utils.enqueueLinksByClickingElements({
- **`.page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- **`.requestQueue`**: [`RequestQueue`](/docs/api/request-queue) - A request queue to which the URLs will be enqueued.
- - **`.selector`**: `String` - A CSS selector matching elements to be clicked on. Unlike in
+ - **`.selector`**: `string` - A CSS selector matching elements to be clicked on. Unlike in
[`utils.enqueueLinks()`](/docs/api/utils#enqueuelinks), there is no default value. This is to prevent suboptimal use of this function by using
it too broadly.
- - **`[.pseudoUrls]`**: `Array<(String|RegExp|Object)>` - An array of [`PseudoUrl`](/docs/api/pseudo-url)s matching the URLs to be enqueued, or
+ - **`[.pseudoUrls]`**: `Array<(string|RegExp|Object)>` - An array of [`PseudoUrl`](/docs/api/pseudo-url)s matching the URLs to be enqueued, or
an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](/docs/api/pseudo-url)s can be constructed.
The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the
@@ -260,7 +260,7 @@ await Apify.utils.enqueueLinksByClickingElements({
## `puppeteer.injectFile(page, filePath, [options])`
-**Returns**: `Promise`
+**Returns**: `Promise<*>`
Injects a JavaScript file into a Puppeteer page. Unlike Puppeteer's `addScriptTag` function, this function works on pages with arbitrary Cross-Origin
Resource Sharing (CORS) policies.
@@ -270,7 +270,7 @@ File contents are cached for up to 10 files to limit file system access.
**Params**
- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
-- **`filePath`**: `String` - File path
+- **`filePath`**: `string` - File path
- **`[options]`**: `Object`
- **`[.surviveNavigations]`**: `boolean` - Enables the injected script to survive page navigations and reloads without need to be re-injected
manually. This does not mean, however, that internal state will be preserved. Just that it will be automatically re-injected on each
@@ -282,7 +282,7 @@ File contents are cached for up to 10 files to limit file system access.
## `puppeteer.injectJQuery(page)`
-**Returns**: `Promise`
+**Returns**: `Promise<*>`
Injects the [](https://jquery.com/) library into a Puppeteer page. jQuery is often useful for various web scraping and crawling tasks. For example, it
can help extract text from HTML elements using CSS selectors.
@@ -313,7 +313,7 @@ Note that `injectJQuery()` does not affect the Puppeteer's [](https://pptr.dev/#
## `puppeteer.injectUnderscore(page)`
-**Returns**: `Promise`
+**Returns**: `Promise<*>`
Injects the [](https://underscorejs.org/) library into a Puppeteer page.
@@ -341,7 +341,7 @@ const escapedHtml = await page.evaluate(() => {
## `puppeteer.blockRequests(page, [options])`
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
Forces the Puppeteer browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it
reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.
@@ -395,7 +395,7 @@ await page.goto('https://cnn.com');
**_Deprecated_**
-**Returns**: `Promise`
+**Returns**: `Promise<void>`
_NOTE:_ In recent versions of Puppeteer using this function entirely disables browser cache which resolves in sub-optimal performance. Until this
resolves, we suggest just relying on the in-browser cache unless absolutely necessary.
@@ -408,7 +408,7 @@ mitigated in future iterations of this feature.
- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- **`cache`**: `Object` - Object in which responses are stored
-- **`responseUrlRules`**: `Array<(String|RegExp)>` - List of rules that are used to check if the response should be cached. String rules are
+- **`responseUrlRules`**: `Array<(string|RegExp)>` - List of rules that are used to check if the response should be cached. String rules are
compared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).
---
@@ -442,7 +442,7 @@ context. Preferably making secured copies beforehand.
**Params**
-- **`scriptString`**: `String`
+- **`scriptString`**: `string`
- **`context`**: `Object`
---
diff --git a/docs/api/social.md b/docs/api/social.md
index 4b2a8a658..c99d1ac58 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -325,13 +325,13 @@ Regular expression to find multiple email addresses in a text. It has the follow
## `social.emailsFromText(text)`
-**Returns**: `Array<String>` - Array of emails addresses found. If no emails are found, the function returns an empty array.
+**Returns**: `Array<string>` - Array of emails addresses found. If no emails are found, the function returns an empty array.
The function extracts email addresses from a plain text. Note that the function preserves the order of emails and keep duplicates.
**Params**
-- **`text`**: `String` - Text to search in.
+- **`text`**: `string` - Text to search in.
---
@@ -339,14 +339,14 @@ The function extracts email addresses from a plain text. Note that the function
## `social.emailsFromUrls(urls)`
-**Returns**: `Array<String>` - Array of emails addresses found. If no emails are found, the function returns an empty array.
+**Returns**: `Array<string>` - Array of emails addresses found. If no emails are found, the function returns an empty array.
The function extracts email addresses from a list of URLs. Basically it looks for all `mailto:` URLs and returns valid email addresses from them. Note
that the function preserves the order of emails and keep duplicates.
**Params**
-- **`urls`**: `Array<String>` - Array of URLs.
+- **`urls`**: `Array<string>` - Array of URLs.
---
@@ -354,14 +354,14 @@ that the function preserves the order of emails and keep duplicates.
## `social.phonesFromText(text)`
-**Returns**: `Array<String>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.
+**Returns**: `Array<string>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.
The function attempts to extract phone numbers from a text. Please note that the results might not be accurate, since phone numbers appear in a large
variety of formats and conventions. If you encounter some problems, please [file an issue](https://github.com/apifytech/apify-js/issues).
**Params**
-- **`text`**: `String` - Text to search the phone numbers in.
+- **`text`**: `string` - Text to search the phone numbers in.
---
@@ -369,14 +369,14 @@ variety of formats and conventions. If you encounter some problems, please [file
## `social.phonesFromUrls(urls)`
-**Returns**: `Array<String>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.
+**Returns**: `Array<string>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.
Finds phone number links in an array of URLs and extracts the phone numbers from them. Note that the phone number links look like `tel://123456789`,
`tel:/123456789` or `tel:123456789`.
**Params**
-- **`urls`**: `Array<String>` - Array of URLs.
+- **`urls`**: `Array<string>` - Array of URLs.
---
@@ -410,7 +410,7 @@ console.dir(result);
**Params**
-- **`html`**: `String` - HTML text
+- **`html`**: `string` - HTML text
- **`data`**: `Object` <code> = </code> - Optional object which will receive the `text` and `$` properties that contain text content of the HTML and
`cheerio` object, respectively. This is an optimization so that the caller doesn't need to parse the HTML document again, if needed.
diff --git a/docs/api/utils.md b/docs/api/utils.md
index e3f436e22..5f60846cd 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -74,15 +74,15 @@ await Apify.utils.enqueueLinks({
- **`options`**: `Object` - All `enqueueLinks()` parameters are passed via an options object with the following keys:
- - **`.page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object. Either `page` or `$` option must be
- provided.
- - **`.$`**: `Cheerio` - [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML. Either `page` or `$` option must be
+ - **`[.page]`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object. Either `page` or `$` option must
+ be provided.
+ - **`[.$]`**: `CheerioStatic` - [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML. Either `page` or `$` option must be
provided.
- **`.requestQueue`**: [`RequestQueue`](/docs/api/request-queue) - A request queue to which the URLs will be enqueued.
- - **`[.selector]`**: `String` <code> = &#x27;a&#x27;</code> - A CSS selector matching links to be enqueued.
+ - **`[.selector]`**: `string` <code> = &quot;&#x27;a&#x27;&quot;</code> - A CSS selector matching links to be enqueued.
- **`[.baseUrl]`**: `string` - A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer, since the
relative URL resolution is done inside the browser automatically.
- - **`[.pseudoUrls]`**: `Array<Object&gt;` | `Array.&lt;String>` - An array of [`PseudoUrl`](/docs/api/pseudo-url)s matching the URLs to be
+ - **`[.pseudoUrls]`**: `Array<Object&gt;` | `Array.&lt;string>` - An array of [`PseudoUrl`](/docs/api/pseudo-url)s matching the URLs to be
enqueued, or an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](/docs/api/pseudo-url)s can be constructed.
The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the
@@ -116,7 +116,7 @@ await Apify.utils.enqueueLinks({
## `utils.requestAsBrowser(options)`
-**Returns**: `Promise<(http.IncomingMessage|stream.Readable)>` - This will typically be a
+**Returns**: `Promise<(IncomingMessage|Readable)>` - This will typically be a
[Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage), however, if returned from the cache it will be a
[response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.
@@ -124,7 +124,7 @@ await Apify.utils.enqueueLinks({
because it allows bypassing certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping scenarios, please set
`useInsecureHttpParser: false` and `ignoreSslErrors: false`.
-Sends an HTTP request that looks like a request sent by a web browser, fully emulating browser's HTTP headers.
+Sends a HTTP request that looks like a request sent by a web browser, fully emulating browser's HTTP headers.
This function is useful for web scraping of websites that send the full HTML in the first response. Thanks to this function, the target web server has
no simple way to find out the request hasn't been sent by a full web browser. Using a headless browser for such requests is an order of magnitude more
@@ -178,7 +178,7 @@ await Apify.utils.sleep(1500);
**Params**
-- **`millis`**: `Number` - Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
+- **`millis`**: `number` - Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
---
@@ -186,7 +186,7 @@ await Apify.utils.sleep(1500);
## `utils.downloadListOfUrls(options)`
-**Returns**: `Promise<Array<String>>`
+**Returns**: `Promise<Array<string>>`
Returns a promise that resolves to an array of urls parsed from the resource available at the provided url. Optionally, custom regular expression and
encoding may be provided.
@@ -194,8 +194,8 @@ encoding may be provided.
**Params**
- **`options`**: `Object`
- - **`.url`**: `String` - URL to the file
- - **`[.encoding]`**: `String` <code> = &#x27;utf8&#x27;</code> - The encoding of the file.
+ - **`.url`**: `string` - URL to the file
+ - **`[.encoding]`**: `string` <code> = &quot;&#x27;utf8&#x27;&quot;</code> - The encoding of the file.
- **`[.urlRegExp]`**: `RegExp` <code> = URL_NO_COMMAS_REGEX</code> - Custom regular expression to identify the URLs in the file to extract. The
regular expression should be case-insensitive and have global flag set (i.e. `/something/gi`).
@@ -205,14 +205,14 @@ encoding may be provided.
## `utils.extractUrls(options)`
-**Returns**: `Array<String>`
+**Returns**: `Array<string>`
Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.
**Params**
- **`options`**: `Object`
- - **`.string`**: `String`
+ - **`.string`**: `string`
- **`[.urlRegExp]`**: `RegExp` <code> = Apify.utils.URL_NO_COMMAS_REGEX</code>
---
@@ -221,7 +221,7 @@ Collects all URLs in an arbitrary string to an array, optionally using a custom
## `utils.getRandomUserAgent()`
-**Returns**: `String`
+**Returns**: `string`
Returns a randomly selected User-Agent header out of a list of the most common headers.
@@ -231,7 +231,7 @@ Returns a randomly selected User-Agent header out of a list of the most common h
## `utils.htmlToText(html)`
-**Returns**: `String` - Plain text
+**Returns**: `string` - Plain text
The function converts a HTML document to a plain text.
@@ -259,6 +259,6 @@ const text = htmlToText(cheerio.load(html, { decodeEntities: true }));
**Params**
-- **`html`**: `String` | `Cheerio` - HTML text or parsed HTML represented using a [cheerio](https://www.npmjs.com/package/cheerio) function.
+- **`html`**: `string` | `CheerioStatic` - HTML text or parsed HTML represented using a [cheerio](https://www.npmjs.com/package/cheerio) function.
---
diff --git a/docs/typedefs/AbortFunction.md b/docs/typedefs/AbortFunction.md
new file mode 100644
index 000000000..fa88a4f00
--- /dev/null
+++ b/docs/typedefs/AbortFunction.md
@@ -0,0 +1,14 @@
+---
+id: abort-function
+title: AbortFunction
+---
+
+<a name="abortfunction"></a>
+
+**Returns**: `boolean`
+
+**Params**
+
+- **`response`**: `IncomingMessage`
+
+---
diff --git a/docs/typedefs/ActorRun.md b/docs/typedefs/ActorRun.md
index a286cd18e..9b8a3a96a 100644
--- a/docs/typedefs/ActorRun.md
+++ b/docs/typedefs/ActorRun.md
@@ -14,7 +14,7 @@ details, see [Runs.](https://docs.apify.com/actor/run)
### `id`
-**Type**: `String`
+**Type**: `string`
Actor run ID
@@ -22,7 +22,7 @@ Actor run ID
### `actId`
-**Type**: `String`
+**Type**: `string`
Actor ID
@@ -46,7 +46,7 @@ Time when the actor run finished. Contains `null` for running actors.
### `status`
-**Type**: `String`
+**Type**: `string`
Status of the run. For possible values, see [Run lifecycle](https://docs.apify.com/actor/run#lifecycle) in Apify actor documentation.
@@ -54,7 +54,7 @@ Status of the run. For possible values, see [Run lifecycle](https://docs.apify.c
### `meta`
-**Type**: `Object`
+**Type**: `object`
Actor run meta-data. For example:
@@ -70,7 +70,7 @@ Actor run meta-data. For example:
### `stats`
-**Type**: `Object`
+**Type**: `object`
An object containing various actor run statistics. For example:
@@ -88,7 +88,7 @@ Beware that object fields might change in future releases.
### `options`
-**Type**: `Object`
+**Type**: `object`
Actor run options. For example:
@@ -105,7 +105,7 @@ Actor run options. For example:
### `buildId`
-**Type**: `String`
+**Type**: `string`
ID of the actor build used for the run. For details, see [Builds](https://docs.apify.com/actor/build) in Apify actor documentation.
@@ -113,7 +113,7 @@ ID of the actor build used for the run. For details, see [Builds](https://docs.a
### `buildNumber`
-**Type**: `String`
+**Type**: `string`
Number of the actor build used for the run. For example, `0.0.10`.
@@ -121,7 +121,7 @@ Number of the actor build used for the run. For example, `0.0.10`.
### `exitCode`
-**Type**: `Number`
+**Type**: `number`
Exit code of the actor run process. It's `null` if actor is still running.
@@ -129,7 +129,7 @@ Exit code of the actor run process. It's `null` if actor is still running.
### `defaultKeyValueStoreId`
-**Type**: `String`
+**Type**: `string`
ID of the default key-value store associated with the actor run. See [`KeyValueStore`](/docs/api/key-value-store) for details.
@@ -137,7 +137,7 @@ ID of the default key-value store associated with the actor run. See [`KeyValueS
### `defaultDatasetId`
-**Type**: `String`
+**Type**: `string`
ID of the default dataset associated with the actor run. See [`Dataset`](/docs/api/dataset) for details.
@@ -145,7 +145,7 @@ ID of the default dataset associated with the actor run. See [`Dataset`](/docs/a
### `defaultRequestQueueId`
-**Type**: `String`
+**Type**: `string`
ID of the default request queue associated with the actor run. See [`RequestQueue`](/docs/api/request-queue) for details.
@@ -153,7 +153,7 @@ ID of the default request queue associated with the actor run. See [`RequestQueu
### `containerUrl`
-**Type**: `String`
+**Type**: `string`
URL on which the web server running inside actor run's Docker container can be accessed. For more details, see
[Container web server](https://docs.apify.com/actor/run#container-web-server) in Apify actor documentation.
@@ -162,7 +162,7 @@ URL on which the web server running inside actor run's Docker container can be a
### `output`
-**Type**: `Object`
+**Type**: `object` | `null` | `undefined`
Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running, or if you pass `false` to the `fetchOutput`
option of [`Apify.call()`](/docs/api/apify#call).
diff --git a/docs/typedefs/ApifyEnv.md b/docs/typedefs/ApifyEnv.md
index ed5647609..169701461 100644
--- a/docs/typedefs/ApifyEnv.md
+++ b/docs/typedefs/ApifyEnv.md
@@ -11,7 +11,7 @@ Parsed representation of the `APIFY_XXX` environmental variables.
### `actorId`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
ID of the actor (APIFY_ACTOR_ID)
@@ -19,7 +19,7 @@ ID of the actor (APIFY_ACTOR_ID)
### `actorRunId`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
ID of the actor run (APIFY_ACTOR_RUN_ID)
@@ -27,7 +27,7 @@ ID of the actor run (APIFY_ACTOR_RUN_ID)
### `actorTaskId`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
ID of the actor task (APIFY_ACTOR_TASK_ID)
@@ -35,7 +35,7 @@ ID of the actor task (APIFY_ACTOR_TASK_ID)
### `userId`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
ID of the user who started the actor - note that it might be different than the owner ofthe actor (APIFY_USER_ID)
@@ -43,7 +43,7 @@ ID of the user who started the actor - note that it might be different than the
### `token`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
Authentication token representing privileges given to the actor run, it can be passed to various Apify APIs (APIFY_TOKEN)
@@ -67,7 +67,7 @@ Date when the actor will time out (APIFY_TIMEOUT_AT)
### `defaultKeyValueStoreId`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
ID of the key-value store where input and output data of this actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
@@ -75,7 +75,7 @@ ID of the key-value store where input and output data of this actor is stored (A
### `defaultDatasetId`
-**Type**: `String` | `null`
+**Type**: `string` | `null`
ID of the dataset where input and output data of this actor is stored (APIFY_DEFAULT_DATASET_ID)
@@ -83,7 +83,7 @@ ID of the dataset where input and output data of this actor is stored (APIFY_DEF
### `memoryMbytes`
-**Type**: `Number` | `null`
+**Type**: `number` | `null`
Amount of memory allocated for the actor, in megabytes (APIFY_MEMORY_MBYTES)
diff --git a/docs/typedefs/AutoscaledPoolOptions.md b/docs/typedefs/AutoscaledPoolOptions.md
index 651e9424b..2ba7ebfd6 100644
--- a/docs/typedefs/AutoscaledPoolOptions.md
+++ b/docs/typedefs/AutoscaledPoolOptions.md
@@ -37,7 +37,7 @@ called. To abort a run, use the [`AutoscaledPool.abort()`](/docs/api/autoscaled-
### `minConcurrency`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
The minimum number of tasks running in parallel.
@@ -48,7 +48,7 @@ not sure, just keep the default value and the concurrency will scale up automati
### `maxConcurrency`
-**Type**: `Number` <code> = 1000</code>
+**Type**: `number` <code> = 1000</code>
The maximum number of tasks running in parallel.
@@ -56,7 +56,7 @@ The maximum number of tasks running in parallel.
### `desiredConcurrency`
-**Type**: `Number`
+**Type**: `number`
The desired number of tasks that should be running parallel on the start of the pool, if there is a large enough supply of them. By default, it is
`minConcurrency`.
@@ -65,7 +65,7 @@ The desired number of tasks that should be running parallel on the start of the
### `desiredConcurrencyRatio`
-**Type**: `Number` <code> = 0.95</code>
+**Type**: `number` <code> = 0.95</code>
Minimum level of desired concurrency to reach before more scaling up is allowed.
@@ -73,7 +73,7 @@ Minimum level of desired concurrency to reach before more scaling up is allowed.
### `scaleUpStepRatio`
-**Type**: `Number` <code> = 0.05</code>
+**Type**: `number` <code> = 0.05</code>
Defines the fractional amount of desired concurrency to be added with each scaling up. The minimum scaling step is one.
@@ -81,7 +81,7 @@ Defines the fractional amount of desired concurrency to be added with each scali
### `scaleDownStepRatio`
-**Type**: `Number` <code> = 0.05</code>
+**Type**: `number` <code> = 0.05</code>
Defines the amount of desired concurrency to be subtracted with each scaling down. The minimum scaling step is one.
@@ -89,7 +89,7 @@ Defines the amount of desired concurrency to be subtracted with each scaling dow
### `maybeRunIntervalSecs`
-**Type**: `Number` <code> = 0.5</code>
+**Type**: `number` <code> = 0.5</code>
Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds. This has no effect on starting new tasks immediately
after a task completes.
@@ -98,7 +98,7 @@ after a task completes.
### `loggingIntervalSecs`
-**Type**: `Number` <code> = 60</code>
+**Type**: `number` <code> = 60</code>
Specifies a period in which the instance logs its state, in seconds. Set to `null` to disable periodic logging.
@@ -106,7 +106,7 @@ Specifies a period in which the instance logs its state, in seconds. Set to `nul
### `autoscaleIntervalSecs`
-**Type**: `Number` <code> = 10</code>
+**Type**: `number` <code> = 10</code>
Defines in seconds how often the pool should attempt to adjust the desired concurrency based on the latest system status. Setting it lower than 1
might have a severe impact on performance. We suggest using a value from 5 to 20.
diff --git a/docs/typedefs/BasicCrawlerOptions.md b/docs/typedefs/BasicCrawlerOptions.md
index 95f9354ec..45140b3f0 100644
--- a/docs/typedefs/BasicCrawlerOptions.md
+++ b/docs/typedefs/BasicCrawlerOptions.md
@@ -83,7 +83,7 @@ function.
### `maxRequestRetries`
-**Type**: `Number` <code> = 3</code>
+**Type**: `number` <code> = 3</code>
Indicates how many times the request is retried if
[`BasicCrawlerOptions.handleRequestFunction`](/docs/typedefs/basic-crawler-options#handlerequestfunction) fails.
@@ -92,7 +92,7 @@ Indicates how many times the request is retried if
### `maxRequestsPerCrawl`
-**Type**: `Number`
+**Type**: `number`
Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite
loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -111,7 +111,7 @@ Custom options passed to the underlying [`AutoscaledPool`](/docs/api/autoscaled-
### `minConcurrency`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.
@@ -122,7 +122,7 @@ you're not sure, just keep the default value and the concurrency will scale up a
### `maxConcurrency`
-**Type**: `Number` <code> = 1000</code>
+**Type**: `number` <code> = 1000</code>
Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.
@@ -130,7 +130,7 @@ Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corres
### `useSessionPool`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If set to true. Basic crawler will initialize the [`SessionPool`](/docs/api/session-pool) with the corresponding `sessionPoolOptions`. The session
instance will be than available in the `handleRequestFunction`.
diff --git a/docs/typedefs/CheerioCrawlerOptions.md b/docs/typedefs/CheerioCrawlerOptions.md
index c09f4299f..f0721b9c1 100644
--- a/docs/typedefs/CheerioCrawlerOptions.md
+++ b/docs/typedefs/CheerioCrawlerOptions.md
@@ -133,7 +133,7 @@ SDK would have access to a different [`Request`](/docs/api/request) instance.
### `handlePageTimeoutSecs`
-**Type**: `Number` <code> = 60</code>
+**Type**: `number` <code> = 60</code>
Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
@@ -141,7 +141,7 @@ Timeout in which the function passed as `handlePageFunction` needs to finish, gi
### `requestTimeoutSecs`
-**Type**: `Number` <code> = 30</code>
+**Type**: `number` <code> = 30</code>
Timeout in which the HTTP request to the resource needs to finish, given in seconds.
@@ -149,7 +149,7 @@ Timeout in which the HTTP request to the resource needs to finish, given in seco
### `ignoreSslErrors`
-**Type**: `Boolean` <code> = true</code>
+**Type**: `boolean` <code> = true</code>
If set to true, SSL certificate errors will be ignored.
@@ -157,7 +157,7 @@ If set to true, SSL certificate errors will be ignored.
### `useApifyProxy`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If set to `true`, `CheerioCrawler` will be configured to use [Apify Proxy](https://my.apify.com/proxy) for all connections. For more information, see
the [documentation](https://docs.apify.com/proxy)
@@ -166,7 +166,7 @@ the [documentation](https://docs.apify.com/proxy)
### `apifyProxyGroups`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/proxy). Only applied if the `useApifyProxy` option is `true`.
@@ -174,7 +174,7 @@ An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/
### `apifyProxySession`
-**Type**: `String`
+**Type**: `string`
Apify Proxy session identifier to be used with requests made by `CheerioCrawler`. All HTTP requests going through the proxy with the same session
identifier will use the same target proxy server (i.e. the same IP address). The identifier can only contain the following characters: `0-9`, `a-z`,
@@ -184,7 +184,7 @@ identifier will use the same target proxy server (i.e. the same IP address). The
### `proxyUrls`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
An array of custom proxy URLs to be used by the `CheerioCrawler` instance. The provided custom proxies' order will be randomized and the resulting
list rotated. Custom proxies are not compatible with Apify Proxy and an attempt to use both configuration options will cause an error to be thrown on
@@ -217,7 +217,7 @@ function.
### `additionalMimeTypes`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
target="_blank">MIME types</a> you want the crawler to load and process. By default, only `text/html` and `application/xhtml+xml` MIME types are
@@ -227,7 +227,7 @@ supported.
### `maxRequestRetries`
-**Type**: `Number` <code> = 3</code>
+**Type**: `number` <code> = 3</code>
Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
@@ -235,7 +235,7 @@ Indicates how many times the request is retried if either `requestFunction` or `
### `maxRequestsPerCrawl`
-**Type**: `Number`
+**Type**: `number`
Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite
loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -255,7 +255,7 @@ parses HTML synchronously and therefore blocks the event loop.
### `minConcurrency`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.
@@ -266,7 +266,7 @@ you're not sure, just keep the default value and the concurrency will scale up a
### `maxConcurrency`
-**Type**: `Number` <code> = 1000</code>
+**Type**: `number` <code> = 1000</code>
Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](/docs/api/autoscaled-pool) option.
@@ -274,7 +274,7 @@ Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corres
### `useSessionPool`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes. It also marks
Session as bad after a request timeout.
@@ -291,7 +291,7 @@ Custom options passed to the underlying [`SessionPool`](/docs/api/session-pool)
### `persistCookiesPerSession`
-**Type**: `Boolean`
+**Type**: `boolean`
Automatically saves cookies to Session. Works only if Session Pool is used.
diff --git a/docs/typedefs/CheerioHandlePageInputs.md b/docs/typedefs/CheerioHandlePageInputs.md
index 45c499032..f793969da 100644
--- a/docs/typedefs/CheerioHandlePageInputs.md
+++ b/docs/typedefs/CheerioHandlePageInputs.md
@@ -9,7 +9,7 @@ title: CheerioHandlePageInputs
### `$`
-**Type**: `Cheerio`
+**Type**: `CheerioSelector`
The [Cheerio](https://cheerio.js.org/) object with parsed HTML.
@@ -17,7 +17,7 @@ The [Cheerio](https://cheerio.js.org/) object with parsed HTML.
### `body`
-**Type**: `String` | `Buffer`
+**Type**: `string` | `Buffer`
The request body of the web page.
@@ -25,7 +25,7 @@ The request body of the web page.
### `json`
-**Type**: `Object`
+**Type**: `*`
The parsed object from JSON string if the response contains the content type application/json.
@@ -35,7 +35,7 @@ The parsed object from JSON string if the response contains the content type app
**Type**: [`Request`](/docs/api/request)
-The original {Request} object.
+The original [`Request`](/docs/api/request) object.
---
@@ -68,6 +68,6 @@ settings on the fly, to pause the crawler by calling [`AutoscaledPool.pause()`](
### `session`
-**Type**: `session`
+**Type**: [`Session`](/docs/api/session)
---
diff --git a/docs/typedefs/CreateSession.md b/docs/typedefs/CreateSession.md
index 34b48f4e3..8d0e38c86 100644
--- a/docs/typedefs/CreateSession.md
+++ b/docs/typedefs/CreateSession.md
@@ -5,6 +5,8 @@ title: CreateSession
<a name="createsession"></a>
+**Returns**: [`Session`](/docs/api/session)
+
Factory user-function which creates customized [`Session`](/docs/api/session) instances.
**Params**
diff --git a/docs/typedefs/DatasetConsumer.md b/docs/typedefs/DatasetConsumer.md
index 2bf5946cc..2fc83ee43 100644
--- a/docs/typedefs/DatasetConsumer.md
+++ b/docs/typedefs/DatasetConsumer.md
@@ -5,13 +5,13 @@ title: DatasetConsumer
<a name="datasetconsumer"></a>
-**Returns**: T
+**Returns**: `object`
User-function used in the `Dataset.forEach()` API.
**Params**
-- **`item`**: `Object` - Current [`Dataset`](/docs/api/dataset) entry being processed.
-- **`index`**: `Number` - Position of current {Dataset} entry.
+- **`item`**: `object` - Current [`Dataset`](/docs/api/dataset) entry being processed.
+- **`index`**: `number` - Position of current {Dataset} entry.
---
diff --git a/docs/typedefs/DatasetContent.md b/docs/typedefs/DatasetContent.md
index ece0e9189..e7e97dd87 100644
--- a/docs/typedefs/DatasetContent.md
+++ b/docs/typedefs/DatasetContent.md
@@ -9,7 +9,7 @@ title: DatasetContent
### `items`
-**Type**: `Array<Object&gt;` | `Array<String>` | `Array.&lt;Buffer>`
+**Type**: `Array<object>`
Dataset entries based on chosen format parameter.
@@ -17,7 +17,7 @@ Dataset entries based on chosen format parameter.
### `total`
-**Type**: `Number`
+**Type**: `number`
Total count of entries in the dataset.
@@ -25,7 +25,7 @@ Total count of entries in the dataset.
### `offset`
-**Type**: `Number`
+**Type**: `number`
Position of the first returned entry in the dataset.
@@ -33,7 +33,7 @@ Position of the first returned entry in the dataset.
### `count`
-**Type**: `Number`
+**Type**: `number`
Count of dataset entries returned in this set.
@@ -41,7 +41,7 @@ Count of dataset entries returned in this set.
### `limit`
-**Type**: `Number`
+**Type**: `number`
Maximum number of dataset entries requested.
diff --git a/docs/typedefs/DatasetMapper.md b/docs/typedefs/DatasetMapper.md
index 8b4602081..7518030cf 100644
--- a/docs/typedefs/DatasetMapper.md
+++ b/docs/typedefs/DatasetMapper.md
@@ -5,13 +5,13 @@ title: DatasetMapper
<a name="datasetmapper"></a>
-**Returns**: T
+**Returns**: `object`
User-function used in the `Dataset.map()` API.
**Params**
-- **`item`**: `Object` - Currect [`Dataset`](/docs/api/dataset) entry being processed.
-- **`index`**: `Number` - Position of current {Dataset} entry.
+- **`item`**: `object` - Currect [`Dataset`](/docs/api/dataset) entry being processed.
+- **`index`**: `number` - Position of current {Dataset} entry.
---
diff --git a/docs/typedefs/DatasetReducer.md b/docs/typedefs/DatasetReducer.md
index 3fe56aee2..1eaa506c3 100644
--- a/docs/typedefs/DatasetReducer.md
+++ b/docs/typedefs/DatasetReducer.md
@@ -5,14 +5,14 @@ title: DatasetReducer
<a name="datasetreducer"></a>
-**Returns**: T
+**Returns**: `object`
User-function used in the `Dataset.reduce()` API.
**Params**
-- **`memo`**: `T` - Previous state of the reduction.
-- **`item`**: `Object` - Currect [`Dataset`](/docs/api/dataset) entry being processed.
-- **`index`**: `Number` - Position of current {Dataset} entry.
+- **`memo`**: `object` - Previous state of the reduction.
+- **`item`**: `object` - Currect [`Dataset`](/docs/api/dataset) entry being processed.
+- **`index`**: `number` - Position of current {Dataset} entry.
---
diff --git a/docs/typedefs/InterceptHandler.md b/docs/typedefs/InterceptHandler.md
new file mode 100644
index 000000000..6aa19861f
--- /dev/null
+++ b/docs/typedefs/InterceptHandler.md
@@ -0,0 +1,12 @@
+---
+id: intercept-handler
+title: InterceptHandler
+---
+
+<a name="intercepthandler"></a>
+
+**Params**
+
+- **`request`**: [`PuppeteerRequest`](/docs/api/request)
+
+---
diff --git a/docs/typedefs/KeyConsumer.md b/docs/typedefs/KeyConsumer.md
index 4f47bc8ed..a02ae781a 100644
--- a/docs/typedefs/KeyConsumer.md
+++ b/docs/typedefs/KeyConsumer.md
@@ -9,9 +9,9 @@ User-function used in the [`KeyValueStore.forEachKey()`](/docs/api/key-value-sto
**Params**
-- **`key`**: `String` - Current {KeyValue} key being processed.
-- **`index`**: `Number` - Position of the current key in {KeyValuestore}.
-- **`info`**: `Object` - Information about the current {KeyValueStore} entry.
- - **`.size`**: `Number` - Size of the value associated with the current key in bytes.
+- **`key`**: `string` - Current {KeyValue} key being processed.
+- **`index`**: `number` - Position of the current key in {KeyValuestore}.
+- **`info`**: `object` - Information about the current {KeyValueStore} entry.
+ - **`.size`**: `number` - Size of the value associated with the current key in bytes.
---
diff --git a/docs/typedefs/LaunchPuppeteerFunction.md b/docs/typedefs/LaunchPuppeteerFunction.md
new file mode 100644
index 000000000..bd91cd0bb
--- /dev/null
+++ b/docs/typedefs/LaunchPuppeteerFunction.md
@@ -0,0 +1,14 @@
+---
+id: launch-puppeteer-function
+title: LaunchPuppeteerFunction
+---
+
+<a name="launchpuppeteerfunction"></a>
+
+**Returns**: `Promise<Browser>`
+
+**Params**
+
+- **`options`**: [`LaunchPuppeteerOptions`](/docs/typedefs/launch-puppeteer-options)
+
+---
diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.md
index 2574c4609..2f9d84847 100644
--- a/docs/typedefs/LaunchPuppeteerOptions.md
+++ b/docs/typedefs/LaunchPuppeteerOptions.md
@@ -13,7 +13,7 @@ Apify extends the launch options of Puppeteer. You can use any of the Puppeteer
### `proxyUrl`
-**Type**: `String`
+**Type**: `string`
URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password.
@@ -23,7 +23,7 @@ Example: `http://bob:[email protected]:1234`.
### `userAgent`
-**Type**: `String`
+**Type**: `string`
The `User-Agent` HTTP header used by the browser. If not provided, the function sets `User-Agent` to a reasonable default to reduce the chance of
detection of the crawler.
@@ -32,7 +32,7 @@ detection of the crawler.
### `useChrome`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If `true` and `executablePath` is not set, Puppeteer will launch full Google Chrome browser available on the machine rather than the bundled Chromium.
The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical Google
@@ -42,7 +42,7 @@ Chrome executable location specific for the operating system. By default, this o
### `useApifyProxy`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If set to `true`, Puppeteer will be configured to use [Apify Proxy](https://my.apify.com/proxy) for all connections. For more information, see the
[documentation](https://docs.apify.com/proxy)
@@ -51,7 +51,7 @@ If set to `true`, Puppeteer will be configured to use [Apify Proxy](https://my.a
### `apifyProxyGroups`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/proxy). Only applied if the `useApifyProxy` option is `true`.
@@ -59,7 +59,7 @@ An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/
### `apifyProxySession`
-**Type**: `String`
+**Type**: `string`
Apify Proxy session identifier to be used by all the Chrome browsers. All HTTP requests going through the proxy with the same session identifier will
use the same target proxy server (i.e. the same IP address). The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`,
diff --git a/docs/typedefs/MemoryInfo.md b/docs/typedefs/MemoryInfo.md
index f838c44ed..77a7070e8 100644
--- a/docs/typedefs/MemoryInfo.md
+++ b/docs/typedefs/MemoryInfo.md
@@ -11,7 +11,7 @@ Describes memory usage of an Actor.
### `totalBytes`
-**Type**: `Number`
+**Type**: `number`
Total memory available in the system or container
@@ -19,7 +19,7 @@ Total memory available in the system or container
### `freeBytes`
-**Type**: `Number`
+**Type**: `number`
Amount of free memory in the system or container
@@ -27,7 +27,7 @@ Amount of free memory in the system or container
### `usedBytes`
-**Type**: `Number`
+**Type**: `number`
Amount of memory used (= totalBytes - freeBytes)
@@ -35,7 +35,7 @@ Amount of memory used (= totalBytes - freeBytes)
### `mainProcessBytes`
-**Type**: `Number`
+**Type**: `number`
Amount of memory used the current Node.js process
@@ -43,7 +43,7 @@ Amount of memory used the current Node.js process
### `childProcessesBytes`
-**Type**: `Number`
+**Type**: `number`
Amount of memory used by child processes of the current Node.js process
diff --git a/docs/typedefs/PrepareRequestInputs.md b/docs/typedefs/PrepareRequestInputs.md
index ad3a43e8f..598576b57 100644
--- a/docs/typedefs/PrepareRequestInputs.md
+++ b/docs/typedefs/PrepareRequestInputs.md
@@ -14,3 +14,11 @@ title: PrepareRequestInputs
Original instance fo the {Request} object. Must be modified in-place.
---
+
+### `session`
+
+**Type**: [`Session`](/docs/api/session)
+
+The current session
+
+---
diff --git a/docs/typedefs/PuppeteerCrawlerOptions.md b/docs/typedefs/PuppeteerCrawlerOptions.md
index 628dc8846..c5b62e88f 100644
--- a/docs/typedefs/PuppeteerCrawlerOptions.md
+++ b/docs/typedefs/PuppeteerCrawlerOptions.md
@@ -57,7 +57,7 @@ provided (or both).
### `handlePageTimeoutSecs`
-**Type**: `Number` <code> = 60</code>
+**Type**: `number` <code> = 60</code>
Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
@@ -82,7 +82,7 @@ code on [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/
### `gotoTimeoutSecs`
-**Type**: `Number` <code> = 60</code>
+**Type**: `number` <code> = 60</code>
Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default function is overridden, this timeout
will not be used and needs to be configured in the new `gotoFunction()`.
@@ -114,7 +114,7 @@ function.
### `maxRequestRetries`
-**Type**: `Number` <code> = 3</code>
+**Type**: `number` <code> = 3</code>
Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
@@ -122,7 +122,7 @@ Indicates how many times the request is retried if either `handlePageFunction()`
### `maxRequestsPerCrawl`
-**Type**: `Number`
+**Type**: `number`
Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite
loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -139,7 +139,7 @@ Custom options passed to the underlying [`PuppeteerPool`](/docs/api/puppeteer-po
### `launchPuppeteerFunction`
-**Type**: `function`
+**Type**: [`LaunchPuppeteerFunction`](/docs/typedefs/launch-puppeteer-function)
Overrides the default function to launch a new Puppeteer instance. Shortcut to the corresponding [`PuppeteerPool`](/docs/api/puppeteer-pool) option.
See source code on [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28) for default behavior.
@@ -166,7 +166,7 @@ Custom options passed to the underlying [`AutoscaledPool`](/docs/api/autoscaled-
### `minConcurrency`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding
[`AutoscaledPoolOptions.minConcurrency`](/docs/typedefs/autoscaled-pool-options#minconcurrency) option.
@@ -178,7 +178,7 @@ you're not sure, just keep the default value and the concurrency will scale up a
### `maxConcurrency`
-**Type**: `Number` <code> = 1000</code>
+**Type**: `number` <code> = 1000</code>
Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding
[`AutoscaledPoolOptions.maxConcurrency`](/docs/typedefs/autoscaled-pool-options#maxconcurrency) option.
@@ -187,7 +187,7 @@ Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corres
### `useSessionPool`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes. It also marks
Session as bad after a request timeout.
@@ -204,7 +204,7 @@ Custom options passed to the underlying [`SessionPool`](/docs/api/session-pool)
### `persistCookiesPerSession`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
Automatically saves cookies to Session. Works only if Session Pool is used.
diff --git a/docs/typedefs/PuppeteerGoto.md b/docs/typedefs/PuppeteerGoto.md
index 09834686c..88d572c77 100644
--- a/docs/typedefs/PuppeteerGoto.md
+++ b/docs/typedefs/PuppeteerGoto.md
@@ -5,8 +5,9 @@ title: PuppeteerGoto
<a name="puppeteergoto"></a>
-**Returns**: `Promise<PuppeteerResponse>` - An instance of the Puppeteer [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),
-which is the main resource response as returned by `page.goto(request.url)`.
+**Returns**: `Promise<(PuppeteerResponse|null)>` - An instance of the Puppeteer
+[`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response), which is the main resource response as returned by
+`page.goto(request.url)`.
**Params**
diff --git a/docs/typedefs/PuppeteerPoolOptions.md b/docs/typedefs/PuppeteerPoolOptions.md
index 529c10ed0..32d5f34a6 100644
--- a/docs/typedefs/PuppeteerPoolOptions.md
+++ b/docs/typedefs/PuppeteerPoolOptions.md
@@ -62,7 +62,7 @@ closed even if there are pending open tabs.
### `launchPuppeteerFunction`
-**Type**: `function`
+**Type**: [`LaunchPuppeteerFunction`](/docs/typedefs/launch-puppeteer-function)
Overrides the default function to launch a new Puppeteer instance. The function must return a promise resolving to
[`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance. See the source code on
diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.md
index e85b7f267..91263bd2b 100644
--- a/docs/typedefs/QueueOperationInfo.md
+++ b/docs/typedefs/QueueOperationInfo.md
@@ -12,7 +12,7 @@ A helper class that is used to report results from various [`RequestQueue`](/doc
### `wasAlreadyPresent`
-**Type**: `Boolean`
+**Type**: `boolean`
Indicates if request was already present in the queue.
@@ -20,7 +20,7 @@ Indicates if request was already present in the queue.
### `wasAlreadyHandled`
-**Type**: `Boolean`
+**Type**: `boolean`
Indicates if request was already marked as handled.
@@ -28,7 +28,7 @@ Indicates if request was already marked as handled.
### `requestId`
-**Type**: `String`
+**Type**: `string`
The ID of the added request
@@ -36,7 +36,7 @@ The ID of the added request
### `request`
-**Type**: `Object`
+**Type**: [`Request`](/docs/api/request)
The original [`Request`](/docs/api/request) object passed to the `RequestQueue` function.
diff --git a/docs/typedefs/RequestAsBrowserOptions.md b/docs/typedefs/RequestAsBrowserOptions.md
index 8b5c2e056..55fb72ffc 100644
--- a/docs/typedefs/RequestAsBrowserOptions.md
+++ b/docs/typedefs/RequestAsBrowserOptions.md
@@ -9,7 +9,7 @@ title: RequestAsBrowserOptions
### `url`
-**Type**: `String`
+**Type**: `string`
URL of the target endpoint. Supports both HTTP and HTTPS schemes.
@@ -17,7 +17,7 @@ URL of the target endpoint. Supports both HTTP and HTTPS schemes.
### `method`
-**Type**: `String` <code> = GET</code>
+**Type**: `string` <code> = &quot;GET&quot;</code>
HTTP method.
@@ -34,7 +34,7 @@ Overriding default browser headers will remove the masking this function provide
### `proxyUrl`
-**Type**: `String`
+**Type**: `string`
An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.
@@ -42,7 +42,7 @@ An HTTP proxy to be passed down to the HTTP request. Supports proxy authenticati
### `languageCode`
-**Type**: `String` <code> = en</code>
+**Type**: `string` <code> = &quot;en&quot;</code>
Two-letter ISO 639 language code.
@@ -50,7 +50,7 @@ Two-letter ISO 639 language code.
### `countryCode`
-**Type**: `String` <code> = US</code>
+**Type**: `string` <code> = &quot;US&quot;</code>
Two-letter ISO 3166 country code.
@@ -58,7 +58,7 @@ Two-letter ISO 3166 country code.
### `useMobileVersion`
-**Type**: `Boolean`
+**Type**: `boolean`
If `true`, the function uses User-Agent of a mobile browser.
@@ -66,7 +66,7 @@ If `true`, the function uses User-Agent of a mobile browser.
### `abortFunction`
-**Type**: `function`
+**Type**: [`AbortFunction`](/docs/typedefs/abort-function)
Function accepts `response` object as a single parameter and should return true or false. If function returns true request gets aborted. This function
is passed to the [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.
diff --git a/docs/typedefs/RequestListInput.md b/docs/typedefs/RequestListInput.md
new file mode 100644
index 000000000..d5dbda7fc
--- /dev/null
+++ b/docs/typedefs/RequestListInput.md
@@ -0,0 +1,26 @@
+---
+id: request-list-input
+title: RequestListInput
+---
+
+<a name="requestlistinput"></a>
+
+## Properties
+
+### `method`
+
+**Type**: `string`
+
+---
+
+### `requestsFromUrl`
+
+**Type**: `string`
+
+---
+
+### `regex`
+
+**Type**: `RegExp`
+
+---
diff --git a/docs/typedefs/RequestListOptions.md b/docs/typedefs/RequestListOptions.md
index f16ba240c..f3a8d0367 100644
--- a/docs/typedefs/RequestListOptions.md
+++ b/docs/typedefs/RequestListOptions.md
@@ -9,7 +9,7 @@ title: RequestListOptions
### `sources`
-**Type**: [`Array<(RequestOptions|Request)>`](/docs/typedefs/request-options)
+**Type**: [`SourceInput`](/docs/typedefs/source-input)
An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that define the `url` property, or an array of instances
of the [`Request`](/docs/api/request) class. Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct
@@ -37,7 +37,7 @@ of the [`Request`](/docs/api/request) class. Additionally, the `requestsFromUrl`
### `persistStateKey`
-**Type**: `String`
+**Type**: `string`
Identifies the key in the default key-value store under which `RequestList` periodically stores its state (i.e. which URLs were crawled and which
not). If the actor is restarted, `RequestList` will read the state and continue where it left off.
@@ -48,7 +48,7 @@ If `persistStateKey` is not set, `RequestList` will always start from the beginn
### `persistSourcesKey`
-**Type**: `String`
+**Type**: `string`
Identifies the key in the default key-value store under which the `RequestList` persists its sources (i.e. the lists of URLs) during the
[`RequestList.initialize()`](/docs/api/request-list#initialize) call. This is necessary if `persistStateKey` is set and the source URLs might
@@ -82,7 +82,7 @@ Note that the preferred (and simpler) way to persist the state of crawling of th
### `keepDuplicateUrls`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based on the `uniqueKey` property of passed source
[`Request`](/docs/api/request) objects.
diff --git a/docs/typedefs/RequestListState.md b/docs/typedefs/RequestListState.md
index 907c2247c..c09cdfd63 100644
--- a/docs/typedefs/RequestListState.md
+++ b/docs/typedefs/RequestListState.md
@@ -23,7 +23,7 @@ Represents state of a {RequestList}. It can be used to resume a {RequestList} wh
### `nextIndex`
-**Type**: `Number`
+**Type**: `number`
Position of the next request to be processed.
@@ -31,7 +31,7 @@ Position of the next request to be processed.
### `nextUniqueKey`
-**Type**: `String`
+**Type**: `string`
Key of the next request to be processed.
@@ -39,7 +39,7 @@ Key of the next request to be processed.
### `inProgress`
-**Type**: `Object<String, Boolean>`
+**Type**: `Object<string, boolean>`
An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.
diff --git a/docs/typedefs/RequestOptions.md b/docs/typedefs/RequestOptions.md
index bb3b02ffb..3dc8dbc86 100644
--- a/docs/typedefs/RequestOptions.md
+++ b/docs/typedefs/RequestOptions.md
@@ -5,13 +5,13 @@ title: RequestOptions
<a name="requestoptions"></a>
-Specifies required and optional fields for constructing a [`Request.setValue()`](/docs/api/request#setvalue).
+Specifies required and optional fields for constructing a [`Request`](../api/request).
## Properties
### `url`
-**Type**: `String`
+**Type**: `string`
URL of the web page to crawl. It must be a non-empty string.
@@ -19,7 +19,7 @@ URL of the web page to crawl. It must be a non-empty string.
### `uniqueKey`
-**Type**: `String`
+**Type**: `string`
A unique key identifying the request. Two requests with the same `uniqueKey` are considered as pointing to the same URL.
@@ -38,13 +38,13 @@ Pass an arbitrary non-empty text value to the `uniqueKey` property to override t
### `method`
-**Type**: `String` <code> = &#x27;GET&#x27;</code>
+**Type**: `string` <code> = &quot;&#x27;GET&#x27;&quot;</code>
---
### `payload`
-**Type**: `String` | `Buffer`
+**Type**: `string` | `Buffer`
HTTP request payload, e.g. for POST requests.
@@ -67,7 +67,7 @@ HTTP headers in the following format:
### `userData`
-**Type**: `Object` <code> = {}</code>
+**Type**: `object` <code> = {}</code>
Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries,
failures etc.
@@ -76,7 +76,7 @@ failures etc.
### `keepUrlFragment`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If `false` then the hash part of a URL is removed when computing the `uniqueKey` property. For example, this causes the `http://www.example.com#foo`
and `http://www.example.com#bar` URLs to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal. Note that this
@@ -86,7 +86,7 @@ option only has an effect if `uniqueKey` is not set.
### `useExtendedUniqueKey`
-**Type**: `Boolean` <code> = false</code>
+**Type**: `boolean` <code> = false</code>
If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload properties. This is useful when making requests
to the same URL that are differentiated by method or payload, such as form submit navigations in browsers.
diff --git a/docs/typedefs/SessionOptions.md b/docs/typedefs/SessionOptions.md
index 4da2f1bd2..4a0b48c7e 100644
--- a/docs/typedefs/SessionOptions.md
+++ b/docs/typedefs/SessionOptions.md
@@ -25,7 +25,7 @@ Number of seconds after which the session is considered as expired.
### `userData`
-**Type**: `Object`
+**Type**: `object`
Object where custom user data can be stored. For example custom headers.
diff --git a/docs/typedefs/SessionPoolOptions.md b/docs/typedefs/SessionPoolOptions.md
index 0bf631e11..0beb142ff 100644
--- a/docs/typedefs/SessionPoolOptions.md
+++ b/docs/typedefs/SessionPoolOptions.md
@@ -9,7 +9,7 @@ title: SessionPoolOptions
### `maxPoolSize`
-**Type**: `Number` <code> = 1000</code>
+**Type**: `number` <code> = 1000</code>
Maximum size of the pool. Indicates how many sessions are rotated.
@@ -25,7 +25,7 @@ The configuration options for {Session} instances.
### `persistStateKeyValueStoreId`
-**Type**: `String`
+**Type**: `string`
Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
@@ -33,7 +33,7 @@ Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
### `persistStateKey`
-**Type**: `String` <code> = &quot;SESSION_POOL_STATE&quot;</code>
+**Type**: `string` <code> = &quot;\&quot;SESSION_POOL_STATE\&quot;&quot;</code>
Session pool persists it's state under this key in Key value store.
diff --git a/docs/typedefs/SessionState.md b/docs/typedefs/SessionState.md
index f29201834..81f2743e8 100644
--- a/docs/typedefs/SessionState.md
+++ b/docs/typedefs/SessionState.md
@@ -17,13 +17,13 @@ Persistable [`Session`](/docs/api/session) state.
### `cookieJar`
-**Type**: `Object`
+**Type**: `CookieJar`
---
### `userData`
-**Type**: `Object`
+**Type**: `object`
---
diff --git a/docs/typedefs/SnapshotterOptions.md b/docs/typedefs/SnapshotterOptions.md
index c28e72d9e..70963c766 100644
--- a/docs/typedefs/SnapshotterOptions.md
+++ b/docs/typedefs/SnapshotterOptions.md
@@ -9,7 +9,7 @@ title: SnapshotterOptions
### `eventLoopSnapshotIntervalSecs`
-**Type**: `Number` <code> = 0.5</code>
+**Type**: `number` <code> = 0.5</code>
Defines the interval of measuring the event loop response time.
@@ -17,7 +17,7 @@ Defines the interval of measuring the event loop response time.
### `clientSnapshotIntervalSecs`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Defines the interval of checking the current state of the remote API client.
@@ -25,7 +25,7 @@ Defines the interval of checking the current state of the remote API client.
### `maxBlockedMillis`
-**Type**: `Number` <code> = 50</code>
+**Type**: `number` <code> = 50</code>
Maximum allowed delay of the event loop in milliseconds. Exceeding this limit overloads the event loop.
@@ -33,7 +33,7 @@ Maximum allowed delay of the event loop in milliseconds. Exceeding this limit ov
### `cpuSnapshotIntervalSecs`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Defines the interval of measuring CPU usage. This is only used when running locally. On the Apify platform, the statistics are provided externally at
a fixed interval.
@@ -42,7 +42,7 @@ a fixed interval.
### `maxUsedCpuRatio`
-**Type**: `Number` <code> = 0.95</code>
+**Type**: `number` <code> = 0.95</code>
Defines the maximum usage of CPU. Exceeding this limit overloads the CPU.
@@ -50,7 +50,7 @@ Defines the maximum usage of CPU. Exceeding this limit overloads the CPU.
### `memorySnapshotIntervalSecs`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Defines the interval of measuring memory consumption. This is only used when running locally. On the Apify platform, the statistics are provided
externally at a fixed interval. The measurement itself is resource intensive (25 - 50ms async). Therefore, setting this interval below 1 second is not
@@ -60,7 +60,7 @@ recommended.
### `maxUsedMemoryRatio`
-**Type**: `Number` <code> = 0.7</code>
+**Type**: `number` <code> = 0.7</code>
Defines the maximum ratio of total memory that can be used. Exceeding this limit overloads the memory.
@@ -68,7 +68,7 @@ Defines the maximum ratio of total memory that can be used. Exceeding this limit
### `maxClientErrors`
-**Type**: `Number` <code> = 1</code>
+**Type**: `number` <code> = 1</code>
Defines the maximum number of new rate limit errors within the given interval.
@@ -76,7 +76,7 @@ Defines the maximum number of new rate limit errors within the given interval.
### `snapshotHistorySecs`
-**Type**: `Number` <code> = 60</code>
+**Type**: `number` <code> = 60</code>
Sets the interval in seconds for which a history of resource snapshots will be kept. Increasing this to very high numbers will affect performance.
diff --git a/docs/typedefs/SocialHandles.md b/docs/typedefs/SocialHandles.md
index 15aabb580..225b56f52 100644
--- a/docs/typedefs/SocialHandles.md
+++ b/docs/typedefs/SocialHandles.md
@@ -26,48 +26,48 @@ The object has the following structure:
### `emails`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `phones`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `phonesUncertain`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `linkedIns`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `twitters`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `instagrams`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `facebooks`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
### `youtubes`
-**Type**: `Array<String>`
+**Type**: `Array<string>`
---
diff --git a/docs/typedefs/SourceInput.md b/docs/typedefs/SourceInput.md
new file mode 100644
index 000000000..224a5fcdf
--- /dev/null
+++ b/docs/typedefs/SourceInput.md
@@ -0,0 +1,8 @@
+---
+id: source-input
+title: SourceInput
+---
+
+<a name="sourceinput"></a>
+
+---
diff --git a/docs/typedefs/SystemInfo.md b/docs/typedefs/SystemInfo.md
index feadca8c8..eebad09ea 100644
--- a/docs/typedefs/SystemInfo.md
+++ b/docs/typedefs/SystemInfo.md
@@ -11,7 +11,7 @@ Represents the current status of the system.
### `isSystemIdle`
-**Type**: `Boolean`
+**Type**: `boolean`
If true, system is being overloaded.
@@ -19,7 +19,7 @@ If true, system is being overloaded.
### `memInfo`
-**Type**: `Object`
+**Type**: `object`
Memory
@@ -27,12 +27,12 @@ Memory
### `eventLoopInfo`
-**Type**: `Object`
+**Type**: `object`
---
### `cpuInfo`
-**Type**: `Object`
+**Type**: `object`
---
diff --git a/docs/typedefs/SystemStatusOptions.md b/docs/typedefs/SystemStatusOptions.md
index 3cf34d8da..ffd10f479 100644
--- a/docs/typedefs/SystemStatusOptions.md
+++ b/docs/typedefs/SystemStatusOptions.md
@@ -9,7 +9,7 @@ title: SystemStatusOptions
### `currentHistorySecs`
-**Type**: `Number` <code> = 5</code>
+**Type**: `number` <code> = 5</code>
Defines max age of snapshots used in the [`SystemStatus.getCurrentStatus()`](/docs/api/system-status#getcurrentstatus) measurement.
@@ -17,7 +17,7 @@ Defines max age of snapshots used in the [`SystemStatus.getCurrentStatus()`](/do
### `maxMemoryOverloadedRatio`
-**Type**: `Number` <code> = 0.2</code>
+**Type**: `number` <code> = 0.2</code>
Sets the maximum ratio of overloaded snapshots in a memory sample. If the sample exceeds this ratio, the system will be overloaded.
@@ -25,7 +25,7 @@ Sets the maximum ratio of overloaded snapshots in a memory sample. If the sample
### `maxEventLoopOverloadedRatio`
-**Type**: `Number` <code> = 0.2</code>
+**Type**: `number` <code> = 0.2</code>
Sets the maximum ratio of overloaded snapshots in an event loop sample. If the sample exceeds this ratio, the system will be overloaded.
@@ -33,7 +33,7 @@ Sets the maximum ratio of overloaded snapshots in an event loop sample. If the s
### `maxCpuOverloadedRatio`
-**Type**: `Number` <code> = 0.4</code>
+**Type**: `number` <code> = 0.4</code>
Sets the maximum ratio of overloaded snapshots in a CPU sample. If the sample exceeds this ratio, the system will be overloaded.
@@ -41,7 +41,7 @@ Sets the maximum ratio of overloaded snapshots in a CPU sample. If the sample ex
### `maxClientOverloadedRatio`
-**Type**: `Number` <code> = 0.2</code>
+**Type**: `number` <code> = 0.2</code>
Sets the maximum ratio of overloaded snapshots in a Client sample. If the sample exceeds this ratio, the system will be overloaded.
diff --git a/docs/typedefs/UserFunc.md b/docs/typedefs/UserFunc.md
new file mode 100644
index 000000000..b959c093e
--- /dev/null
+++ b/docs/typedefs/UserFunc.md
@@ -0,0 +1,10 @@
+---
+id: user-func
+title: UserFunc
+---
+
+<a name="userfunc"></a>
+
+**Returns**: `Promise<void>`
+
+---
diff --git a/package.json b/package.json
index 011713e0d..d06ac0700 100644
--- a/package.json
+++ b/package.json
@@ -6,7 +6,7 @@
"node": ">=10.17.0"
},
"main": "build/index.js",
- "types": "types/index.esm.d.ts",
+ "types": "types/index.d.ts",
"keywords": [
"apify",
"headless",
@@ -39,26 +39,26 @@
"types"
],
"scripts": {
- "build": "rm -rf ./build && babel src --out-dir build --copy-files",
- "build-docs": "npm run clean && npm run build && node ./website/tools/build_docs.js && npm run build-types",
+ "build": "npm run clean && tsc -p tsconfig.json && node ./tools/typescript_fixes.js",
+ "build-docs": "npm run build && node ./website/tools/build_docs.js",
"build-readme": "node ./tools/build_readme.js",
- "build-types": "./node_modules/.bin/tsc -p tsconfig.json && node ./tools/typescript_fixes.js",
- "test": "npm run build && npm run build-types && jest --testTimeout=20000 --maxWorkers=3",
+ "test": "npm run build && jest --testTimeout=20000 --maxWorkers=3",
"prepare": "npm run build",
"prepublishOnly": "(test $TRAVIS || (echo \"Publishing is reserved to the Travis CI!\"; exit 1))",
- "clean": "rm -rf build",
- "lint": "npm run build && ./node_modules/.bin/eslint ./src ./test",
- "lint:fix": "./node_modules/.bin/eslint ./src ./test --ext .js,.jsx --fix"
+ "clean": "rm -rf ./build ./types",
+ "lint": "npm run build && eslint ./src ./test",
+ "lint:fix": "eslint ./src ./test --ext .js,.jsx --fix"
},
"dependencies": {
"@apify/http-request": "^1.1.5",
"@apify/ps-tree": "^1.1.3",
- "@types/cheerio": "^0.22.15",
- "@types/node": "^13.1.8",
+ "@types/cheerio": "^0.22.16",
+ "@types/node": "^12",
"@types/puppeteer": "^2.0.0",
"@types/tough-cookie": "^2.3.6",
+ "@types/socket.io": "^2.1.4",
"apify-client": "^0.5.25",
- "apify-shared": "^0.2.5",
+ "apify-shared": "^0.2.7",
"cheerio": "^1.0.0-rc.3",
"content-type": "^1.0.4",
"express": "^4.17.1",
@@ -74,7 +74,6 @@
"semver": "^6.3.0",
"socket.io": "^2.2.0",
"tough-cookie": "^3.0.1",
- "ts-jest": "^24.3.0",
"underscore": "^1.9.1",
"ws": "^6.2.1"
},
@@ -84,10 +83,16 @@
"@babel/core": "^7.0.0",
"@babel/preset-env": "^7.0.0",
"@babel/register": "^7.0.0",
+ "@types/content-type": "^1.1.3",
"@types/fs-extra": "^7.0.0",
+ "@types/htmlparser2": "^3.10.1",
"@types/jest": "^24.0.21",
+ "@types/request-promise-native": "^1.0.17",
+ "@types/rimraf": "^2.0.3",
+ "@types/semver": "^7.1.0",
"@types/sinon": "^7.0.13",
- "@types/socket.io": "^2.1.2",
+ "@types/underscore": "^1.9.4",
+ "@types/ws": "^7.2.1",
"basic-auth-parser": "^0.0.2",
"body-parser": "^1.19.0",
"csv-parse": "^4.4.3",
@@ -95,6 +100,7 @@
"fpcollect": "^1.0.4",
"fpscanner": "^0.1.5",
"jest": "^24.9.0",
+ "ts-jest": "^24.3.0",
"jsdoc": "^3.5.5",
"jsdoc-export-default-interop": "^0.3.1",
"portastic": "^1.0.1",
@@ -103,7 +109,7 @@
"sinon": "^7.4.1",
"sinon-stub-promise": "^4.0.0",
"socket.io-client": "^2.2.0",
- "typescript": "^3.7.4",
+ "typescript": "3.7.5",
"ua-parser-js": "^0.7.20",
"why-is-node-running": "^2.0.3"
},
diff --git a/src/actor.js b/src/actor.js
index 322eccf5e..4ffd650de 100644
--- a/src/actor.js
+++ b/src/actor.js
@@ -1,22 +1,20 @@
-import path from 'path';
-import _ from 'underscore';
-import log from 'apify-shared/log';
+import * as path from 'path';
+import * as _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { APIFY_PROXY_VALUE_REGEX } from 'apify-shared/regexs';
import { ENV_VARS, INTEGER_ENV_VARS, LOCAL_ENV_VARS, ACT_JOB_TERMINAL_STATUSES, ACT_JOB_STATUSES } from 'apify-shared/consts';
+import log from './utils_log';
import { EXIT_CODES, COUNTRY_CODE_REGEX } from './constants';
import { initializeEvents, stopEvents } from './events';
import { apifyClient, addCharsetToContentType, sleep, snakeCaseToCamelCase, isAtHome, logSystemInfo, printOutdatedSdkWarning } from './utils';
import { maybeStringify } from './key_value_store';
import { ApifyCallError } from './errors';
-// TYPE IMPORTS
-/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
-import { ActorRun } from './typedefs';
-/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
-
const METAMORPH_AFTER_SLEEP_MILLIS = 300000;
+// eslint-disable-next-line import/named,no-unused-vars,import/first
+import { ActorRun } from './typedefs';
+
/**
* Tries to parse a string with date.
* Returns either a Date object or undefined
@@ -99,21 +97,21 @@ const addInputOptionsOrThrow = (input, contentType, options) => {
/**
* Parsed representation of the `APIFY_XXX` environmental variables.
*
- * @typedef {Object} ApifyEnv
- * @property {String|null} actorId ID of the actor (APIFY_ACTOR_ID)
- * @property {String|null} actorRunId ID of the actor run (APIFY_ACTOR_RUN_ID)
- * @property {String|null} actorTaskId ID of the actor task (APIFY_ACTOR_TASK_ID)
- * @property {String|null} userId ID of the user who started the actor - note that it might be
+ * @typedef ApifyEnv
+ * @property {string|null} actorId ID of the actor (APIFY_ACTOR_ID)
+ * @property {string|null} actorRunId ID of the actor run (APIFY_ACTOR_RUN_ID)
+ * @property {string|null} actorTaskId ID of the actor task (APIFY_ACTOR_TASK_ID)
+ * @property {string|null} userId ID of the user who started the actor - note that it might be
* different than the owner ofthe actor (APIFY_USER_ID)
- * @property {String|null} token Authentication token representing privileges given to the actor run,
+ * @property {string|null} token Authentication token representing privileges given to the actor run,
* it can be passed to various Apify APIs (APIFY_TOKEN)
* @property {Date|null} startedAt Date when the actor was started (APIFY_STARTED_AT)
* @property {Date|null} timeoutAt Date when the actor will time out (APIFY_TIMEOUT_AT)
- * @property {String|null} defaultKeyValueStoreId ID of the key-value store where input and output data of this
+ * @property {string|null} defaultKeyValueStoreId ID of the key-value store where input and output data of this
* actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
- * @property {String|null} defaultDatasetId ID of the dataset where input and output data of this
+ * @property {string|null} defaultDatasetId ID of the dataset where input and output data of this
* actor is stored (APIFY_DEFAULT_DATASET_ID)
- * @property {Number|null} memoryMbytes Amount of memory allocated for the actor,
+ * @property {number|null} memoryMbytes Amount of memory allocated for the actor,
* in megabytes (APIFY_MEMORY_MBYTES)
*/
@@ -150,6 +148,11 @@ export const getEnv = () => {
return envVars;
};
+/**
+ * @callback UserFunc
+ * @return {Promise<void>}
+ */
+
/**
* Runs the main user function that performs the job of the actor
* and terminates the process when the user function finishes.
@@ -214,7 +217,7 @@ export const getEnv = () => {
* });
* ```
*
- * @param {Function} userFunc User function to be executed. If it returns a promise,
+ * @param {UserFunc} userFunc User function to be executed. If it returns a promise,
* the promise will be awaited. The user function is called with no arguments.
*
* @memberof module:Apify
@@ -309,41 +312,41 @@ let callMemoryWarningIssued = false;
* [Run actor](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor)
* and several other API endpoints to obtain the output.
*
- * @param {String} actId
+ * @param {string} actId
* Either `username/actor-name` or actor ID.
- * @param {Object|String|Buffer} [input]
+ * @param {object} [input]
* Input for the actor. If it is an object, it will be stringified to
* JSON and its content type set to `application/json; charset=utf-8`.
* Otherwise the `options.contentType` parameter must be provided.
- * @param {Object} [options]
+ * @param {Object} [options={}]
* Object with the settings below:
- * @param {String} [options.contentType]
+ * @param {string} [options.contentType]
* Content type for the `input`. If not specified,
* `input` is expected to be an object that will be stringified to JSON and content type set to
* `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a
* `String` or `Buffer`.
- * @param {String} [options.token]
+ * @param {string} [options.token]
* User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.
- * @param {Number} [options.memoryMbytes]
+ * @param {number} [options.memoryMbytes]
* Memory in megabytes which will be allocated for the new actor run.
* If not provided, the run uses memory of the default actor run configuration.
- * @param {Number} [options.timeoutSecs]
+ * @param {number} [options.timeoutSecs]
* Timeout for the actor run in seconds. Zero value means there is no timeout.
* If not provided, the run uses timeout of the default actor run configuration.
- * @param {String} [options.build]
+ * @param {string} [options.build]
* Tag or number of the actor build to run (e.g. `beta` or `1.2.345`).
* If not provided, the run uses build tag or number from the default actor run configuration (typically `latest`).
- * @param {String} [options.waitSecs]
+ * @param {string} [options.waitSecs]
* Maximum time to wait for the actor run to finish, in seconds.
* If the limit is reached, the returned promise is resolved to a run object that will have
* status `READY` or `RUNNING` and it will not contain the actor run output.
* If `waitSecs` is null or undefined, the function waits for the actor to finish (default behavior).
- * @param {Boolean} [options.fetchOutput=true]
+ * @param {boolean} [options.fetchOutput=true]
* If `false` then the function does not fetch output of the actor.
- * @param {Boolean} [options.disableBodyParser=false]
+ * @param {boolean} [options.disableBodyParser=false]
* If `true` then the function will not attempt to parse the
* actor's output and will return it in a raw `Buffer`.
- * @param {Array} [options.webhooks] Specifies optional webhooks associated with the actor run, which can be used
+ * @param {Array<object>} [options.webhooks] Specifies optional webhooks associated with the actor run, which can be used
* to receive a notification e.g. when the actor finished or failed, see
* [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.
* @returns {Promise<ActorRun>}
@@ -449,37 +452,37 @@ export const call = async (actId, input, options = {}) => {
* [Run task](https://apify.com/docs/api/v2#/reference/actor-tasks/run-collection/run-task)
* and several other API endpoints to obtain the output.
*
- * @param {String} taskId
+ * @param {string} taskId
* Either `username/task-name` or task ID.
- * @param {Object|String|Buffer} [input]
+ * @param {object} [input]
* Input overrides for the actor task. If it is an object, it will be stringified to
* JSON and its content type set to `application/json; charset=utf-8`.
* Otherwise the `options.contentType` parameter must be provided.
* Provided input will be merged with actor task input.
- * @param {Object} [options]
+ * @param {Object} [options={}]
* Object with the settings below:
- * @param {String} [options.contentType]
+ * @param {string} [options.contentType]
* Content type for the `input`. If not specified,
* `input` is expected to be an object that will be stringified to JSON and content type set to
* `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a
* `String` or `Buffer`.
- * @param {String} [options.token]
+ * @param {string} [options.token]
* User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.
- * @param {Number} [options.memoryMbytes]
+ * @param {number} [options.memoryMbytes]
* Memory in megabytes which will be allocated for the new actor task run.
* If not provided, the run uses memory of the default actor run configuration.
- * @param {Number} [options.timeoutSecs]
+ * @param {number} [options.timeoutSecs]
* Timeout for the actor task run in seconds. Zero value means there is no timeout.
* If not provided, the run uses timeout of the default actor run configuration.
- * @param {String} [options.build]
+ * @param {string} [options.build]
* Tag or number of the actor build to run (e.g. `beta` or `1.2.345`).
* If not provided, the run uses build tag or number from the default actor run configuration (typically `latest`).
- * @param {String} [options.waitSecs]
+ * @param {string} [options.waitSecs]
* Maximum time to wait for the actor task run to finish, in seconds.
* If the limit is reached, the returned promise is resolved to a run object that will have
* status `READY` or `RUNNING` and it will not contain the actor run output.
* If `waitSecs` is null or undefined, the function waits for the actor task to finish (default behavior).
- * @param {Array} [options.webhooks] Specifies optional webhooks associated with the actor run, which can be used
+ * @param {Array<object>} [options.webhooks] Specifies optional webhooks associated with the actor run, which can be used
* to receive a notification e.g. when the actor finished or failed, see
* [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.
* @returns {Promise<ActorRun>}
@@ -549,20 +552,20 @@ export const callTask = async (taskId, input, options = {}) => {
* Transforms this actor run to an actor run of a given actor. The system stops the current container and starts the new container
* instead. All the default storages are preserved and the new input is stored under the `INPUT-METAMORPH-1` key in the same default key-value store.
*
- * @param {String} targetActorId
+ * @param {string} targetActorId
* Either `username/actor-name` or actor ID of an actor to which we want to metamorph.
- * @param {Object|String|Buffer} [input]
+ * @param {object} [input]
* Input for the actor. If it is an object, it will be stringified to
* JSON and its content type set to `application/json; charset=utf-8`.
* Otherwise the `options.contentType` parameter must be provided.
- * @param {Object} [options]
+ * @param {Object} [options={}]
* Object with the settings below:
- * @param {String} [options.contentType]
+ * @param {string} [options.contentType]
* Content type for the `input`. If not specified,
* `input` is expected to be an object that will be stringified to JSON and content type set to
* `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a
* `String` or `Buffer`.
- * @param {String} [options.build]
+ * @param {string} [options.build]
* Tag or number of the target actor build to metamorph into (e.g. `beta` or `1.2.345`).
* If not provided, the run uses build tag or number from the default actor run configuration (typically `latest`).
* @returns {Promise<void>}
@@ -611,35 +614,35 @@ export const metamorph = async (targetActorId, input, options = {}) => {
* The proxy URL can be used from Apify actors, web browsers or any other HTTP
* proxy-enabled applications.
*
- * For more information, see
- * the [Apify Proxy](https://my.apify.com/proxy) page in the app
+ * For more information, see the [Apify Proxy](https://my.apify.com/proxy) page in the app
* or the [documentation](https://docs.apify.com/proxy).
*
- * @param {Object} options
- * Object with the settings below:
- * @param {String} [options.password] User's password for the proxy.
- * By default, it is taken from the `APIFY_PROXY_PASSWORD` environment variable,
- * which is automatically set by the system when running the actors on the Apify cloud,
- * or when using the [Apify CLI](https://github.com/apifytech/apify-cli)
+ * @param {Object} [options]
+ * Object with the props below:
+ * @param {string} [options.password]
+ * User's password for the proxy. By default, it is taken from the `APIFY_PROXY_PASSWORD`
+ * environment variable, which is automatically set by the system when running the actors
+ * on the Apify cloud, or when using the [Apify CLI](https://github.com/apifytech/apify-cli)
* package and the user previously logged in (called `apify login`).
- * @param {String[]} [options.groups] Array of Apify Proxy groups to be used.
- * If not provided, the proxy will select the groups automatically.
- * @param {String} [options.session] Apify Proxy session identifier to be used by the Chrome browser.
- * All HTTP requests going through the proxy with the same session identifier
- * will use the same target proxy server (i.e. the same IP address), unless using Residential proxies.
- * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
- * @param {String} [options.country] If specified, all proxied requests will use IP addresses that are geolocated to the specified country.
- * For example `GB` for IPs from Great Britain. Note that online services often have their own rules for handling geolocation and thus
- * the country selection is a best attempt at geolocation, rather than a guaranteed hit.
- * This parameter is optional, by default, each proxied request is assigned an IP address from a random country.
- * The country code needs to be a two letter ISO country code
- * \- see the <a href="https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements" target="_blank">
- * full list of available country codes
- * </a>.
- *
- * This parameter is optional, by default, the proxy uses all available proxy servers from all countries.
- *
- * @returns {String} Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
+ * @param {string[]} [options.groups]
+ * Array of Apify Proxy groups to be used. If not provided, the proxy will select
+ * the groups automatically.
+ * @param {string} [options.session]
+ * Apify Proxy session identifier to be used by the Chrome browser. All HTTP requests
+ * going through the proxy with the same session identifier will use the same target
+ * proxy server (i.e. the same IP address), unless using Residential proxies. The identifier
+ * can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
+ * @param {string} [options.country]
+ * If set and relevant proxies are available in your Apify account, all proxied requests will
+ * use IP addresses that are geolocated to the specified country. For example `GB` for IPs
+ * from Great Britain. Note that online services often have their own rules for handling
+ * geolocation and thus the country selection is a best attempt at geolocation, rather than
+ * a guaranteed hit. This parameter is optional, by default, each proxied request is assigned
+ * an IP address from a random country. The country code needs to be a two letter ISO country code. See the
+ * [full list of available country codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements).
+ * This parameter is optional, by default, the proxy uses all available proxy servers from all countries.
+ *
+ * @returns {string} Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
*
* @memberof module:Apify
* @function
@@ -740,7 +743,7 @@ export const getApifyProxyUrl = (options = {}) => {
* an actor restart or other situation that would cause the `addWebhook()` function to be called again.
* We suggest using the actor run ID as the idempotency key. You can get the run ID by calling
* {@link Apify#getEnv} function.
- * @return {Promise<Object>} The return value is the Webhook object.
+ * @return {Promise<object>} The return value is the Webhook object.
* For more information, see the [Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.
*
* @memberof module:Apify
diff --git a/src/autoscaling/autoscaled_pool.js b/src/autoscaling/autoscaled_pool.js
index cfaac1c8a..637de6975 100644
--- a/src/autoscaling/autoscaled_pool.js
+++ b/src/autoscaling/autoscaled_pool.js
@@ -1,7 +1,7 @@
-import _ from 'underscore';
+import * as _ from 'underscore';
import { betterSetInterval, betterClearInterval } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
+import log from '../utils_log';
import Snapshotter, { SnapshotterOptions } from './snapshotter'; // eslint-disable-line import/named,no-unused-vars
import SystemStatus, { SystemStatusOptions } from './system_status'; // eslint-disable-line import/named,no-unused-vars
@@ -18,7 +18,7 @@ const DEFAULT_OPTIONS = {
/**
- * @typedef {Object} AutoscaledPoolOptions
+ * @typedef AutoscaledPoolOptions
* @property {Function} runTaskFunction
* A function that performs an asynchronous resource-intensive task.
* The function must either be labeled `async` or return a promise.
@@ -36,32 +36,32 @@ const DEFAULT_OPTIONS = {
* keeps resolving to `true`, `isFinishedFunction()` will never be called.
* To abort a run, use the {@link AutoscaledPool#abort} method.
*
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* The minimum number of tasks running in parallel.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* The maximum number of tasks running in parallel.
- * @property {Number} [desiredConcurrency]
+ * @property {number} [desiredConcurrency]
* The desired number of tasks that should be running parallel on the start of the pool,
* if there is a large enough supply of them.
* By default, it is `minConcurrency`.
- * @property {Number} [desiredConcurrencyRatio=0.95]
+ * @property {number} [desiredConcurrencyRatio=0.95]
* Minimum level of desired concurrency to reach before more scaling up is allowed.
- * @property {Number} [scaleUpStepRatio=0.05]
+ * @property {number} [scaleUpStepRatio=0.05]
* Defines the fractional amount of desired concurrency to be added with each scaling up.
* The minimum scaling step is one.
- * @property {Number} [scaleDownStepRatio=0.05]
+ * @property {number} [scaleDownStepRatio=0.05]
* Defines the amount of desired concurrency to be subtracted with each scaling down.
* The minimum scaling step is one.
- * @property {Number} [maybeRunIntervalSecs=0.5]
+ * @property {number} [maybeRunIntervalSecs=0.5]
* Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds.
* This has no effect on starting new tasks immediately after a task completes.
- * @property {Number} [loggingIntervalSecs=60]
+ * @property {number} [loggingIntervalSecs=60]
* Specifies a period in which the instance logs its state, in seconds.
* Set to `null` to disable periodic logging.
- * @property {Number} [autoscaleIntervalSecs=10]
+ * @property {number} [autoscaleIntervalSecs=10]
* Defines in seconds how often the pool should attempt to adjust the desired concurrency
* based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
* We suggest using a value from 5 to 20.
@@ -281,7 +281,7 @@ class AutoscaledPool {
* Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once
* all the tasks are finished or one of them fails.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async run() {
this.poolPromise = new Promise((resolve, reject) => {
@@ -318,7 +318,7 @@ class AutoscaledPool {
* their state after the invocation of `.abort()`, but that does not mean that some
* parts of their asynchronous chains of commands will not execute.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async abort() {
this.isStopped = true;
@@ -340,7 +340,7 @@ class AutoscaledPool {
* when `.pause()` is invoked (unlike abort, which resolves it).
*
* @param {number} [timeoutSecs]
- * @return {Promise}
+ * @return {Promise<void>}
*/
async pause(timeoutSecs) {
if (this.isStopped) return;
diff --git a/src/autoscaling/snapshotter.js b/src/autoscaling/snapshotter.js
index db9bc6933..f87b2cab2 100644
--- a/src/autoscaling/snapshotter.js
+++ b/src/autoscaling/snapshotter.js
@@ -1,10 +1,10 @@
-import os from 'os';
-import _ from 'underscore';
+import * as os from 'os';
+import * as _ from 'underscore';
import { betterSetInterval, betterClearInterval } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
import { ACTOR_EVENT_NAMES, ENV_VARS } from 'apify-shared/consts';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { getMemoryInfo, isAtHome, apifyClient } from '../utils';
+import log from '../utils_log';
import events from '../events';
const DEFAULT_OPTIONS = {
@@ -24,34 +24,34 @@ const CRITICAL_OVERLOAD_RATE_LIMIT_MILLIS = 10000;
/**
* @typedef SnapshotterOptions
- * @property {Number} [eventLoopSnapshotIntervalSecs=0.5]
+ * @property {number} [eventLoopSnapshotIntervalSecs=0.5]
* Defines the interval of measuring the event loop response time.
- * @property {Number} [clientSnapshotIntervalSecs=1]
+ * @property {number} [clientSnapshotIntervalSecs=1]
* Defines the interval of checking the current state
* of the remote API client.
- * @property {Number} [maxBlockedMillis=50]
+ * @property {number} [maxBlockedMillis=50]
* Maximum allowed delay of the event loop in milliseconds.
* Exceeding this limit overloads the event loop.
- * @property {Number} [cpuSnapshotIntervalSecs=1]
+ * @property {number} [cpuSnapshotIntervalSecs=1]
* Defines the interval of measuring CPU usage.
* This is only used when running locally. On the Apify platform,
* the statistics are provided externally at a fixed interval.
- * @property {Number} [maxUsedCpuRatio=0.95]
+ * @property {number} [maxUsedCpuRatio=0.95]
* Defines the maximum usage of CPU.
* Exceeding this limit overloads the CPU.
- * @property {Number} [memorySnapshotIntervalSecs=1]
+ * @property {number} [memorySnapshotIntervalSecs=1]
* Defines the interval of measuring memory consumption.
* This is only used when running locally. On the Apify platform,
* the statistics are provided externally at a fixed interval.
* The measurement itself is resource intensive (25 - 50ms async).
* Therefore, setting this interval below 1 second is not recommended.
- * @property {Number} [maxUsedMemoryRatio=0.7]
+ * @property {number} [maxUsedMemoryRatio=0.7]
* Defines the maximum ratio of total memory that can be used.
* Exceeding this limit overloads the memory.
- * @property {Number} [maxClientErrors=1]
+ * @property {number} [maxClientErrors=1]
* Defines the maximum number of new rate limit errors within
* the given interval.
- * @property {Number} [snapshotHistorySecs=60]
+ * @property {number} [snapshotHistorySecs=60]
* Sets the interval in seconds for which a history of resource snapshots
* will be kept. Increasing this to very high numbers will affect performance.
*/
@@ -138,7 +138,7 @@ class Snapshotter {
/**
* Starts capturing snapshots at configured intervals.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async start() {
await this._ensureCorrectMaxMemory();
@@ -157,7 +157,7 @@ class Snapshotter {
/**
* Stops all resource capturing.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async stop() {
betterClearInterval(this.eventLoopInterval);
@@ -173,8 +173,8 @@ class Snapshotter {
/**
* Returns a sample of latest memory snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
*/
getMemorySample(sampleDurationMillis) {
return this._getSample(this.memorySnapshots, sampleDurationMillis);
@@ -183,8 +183,8 @@ class Snapshotter {
/**
* Returns a sample of latest event loop snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
*/
getEventLoopSample(sampleDurationMillis) {
return this._getSample(this.eventLoopSnapshots, sampleDurationMillis);
@@ -193,8 +193,8 @@ class Snapshotter {
/**
* Returns a sample of latest CPU snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
*/
getCpuSample(sampleDurationMillis) {
return this._getSample(this.cpuSnapshots, sampleDurationMillis);
@@ -203,8 +203,8 @@ class Snapshotter {
/**
* Returns a sample of latest Client snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} sampleDurationMillis
- * @return {Array}
+ * @param {number} sampleDurationMillis
+ * @return {Array<*>}
*/
getClientSample(sampleDurationMillis) {
return this._getSample(this.clientSnapshots, sampleDurationMillis);
@@ -212,9 +212,9 @@ class Snapshotter {
/**
* Finds the latest snapshots by sampleDurationMillis in the provided array.
- * @param {Array} snapshots
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {Array<*>} snapshots
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
* @ignore
*/
_getSample(snapshots, sampleDurationMillis) { // eslint-disable-line class-methods-use-this
@@ -236,7 +236,7 @@ class Snapshotter {
/**
* Creates a snapshot of current memory usage
* using the Apify platform `systemInfo` event.
- * @param {Object} systemInfo
+ * @param {*} systemInfo
* @ignore
*/
_snapshotMemoryOnPlatform(systemInfo) {
@@ -257,7 +257,7 @@ class Snapshotter {
* Creates a snapshot of current memory usage
* using the Apify platform `systemInfo` event.
* @param {Function} intervalCallback
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
async _snapshotMemoryOnLocal(intervalCallback) {
@@ -284,8 +284,8 @@ class Snapshotter {
/**
* Checks for critical memory overload and logs it to the console.
+ * @param {*} systemInfo
* @ignore
- * @param {Object} systemInfo
*/
_memoryOverloadWarning({ memCurrentBytes }) {
const now = new Date();
@@ -433,7 +433,7 @@ class Snapshotter {
/**
* Removes snapshots that are older than the snapshotHistorySecs option
* from the array (destructively - in place).
- * @param {Array} snapshots
+ * @param {Array<*>} snapshots
* @param {Date} now
* @ignore
*/
diff --git a/src/autoscaling/system_status.js b/src/autoscaling/system_status.js
index 12b1bc39e..8ed1dfbc7 100644
--- a/src/autoscaling/system_status.js
+++ b/src/autoscaling/system_status.js
@@ -1,14 +1,8 @@
import { checkParamOrThrow } from 'apify-client/build/utils';
-import _ from 'underscore';
+import * as _ from 'underscore';
import Snapshotter from './snapshotter'; // eslint-disable-line import/no-duplicates
import { weightedAvg } from '../utils';
-// TYPE IMPORTS
-/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
-import { SnapshotterOptions } from './snapshotter';
-/* eslint-anable no-unused-vars,import/named,import/no-duplicates,import/order */
-
-
const DEFAULT_OPTIONS = {
currentHistorySecs: 5,
maxMemoryOverloadedRatio: 0.2,
@@ -21,30 +15,30 @@ const DEFAULT_OPTIONS = {
/**
* Represents the current status of the system.
*
- * @typedef {Object} SystemInfo
- * @property {Boolean} isSystemIdle
+ * @typedef SystemInfo
+ * @property {boolean} isSystemIdle
* If true, system is being overloaded.
- * @property {Object} memInfo
+ * @property {object} memInfo
* Memory
- * @property {Object} eventLoopInfo
- * @property {Object} cpuInfo
+ * @property {object} eventLoopInfo
+ * @property {object} cpuInfo
*/
/**
- * @typedef {Object} SystemStatusOptions
- * @property {Number} [currentHistorySecs=5]
+ * @typedef SystemStatusOptions
+ * @property {number} [currentHistorySecs=5]
* Defines max age of snapshots used in the
* {@link SystemStatus#getCurrentStatus} measurement.
- * @property {Number} [maxMemoryOverloadedRatio=0.2]
+ * @property {number} [maxMemoryOverloadedRatio=0.2]
* Sets the maximum ratio of overloaded snapshots in a memory sample.
* If the sample exceeds this ratio, the system will be overloaded.
- * @property {Number} [maxEventLoopOverloadedRatio=0.2]
+ * @property {number} [maxEventLoopOverloadedRatio=0.2]
* Sets the maximum ratio of overloaded snapshots in an event loop sample.
* If the sample exceeds this ratio, the system will be overloaded.
- * @property {Number} [maxCpuOverloadedRatio=0.4]
+ * @property {number} [maxCpuOverloadedRatio=0.4]
* Sets the maximum ratio of overloaded snapshots in a CPU sample.
* If the sample exceeds this ratio, the system will be overloaded.
- * @property {Number} [maxClientOverloadedRatio=0.2]
+ * @property {number} [maxClientOverloadedRatio=0.2]
* Sets the maximum ratio of overloaded snapshots in a Client sample.
* If the sample exceeds this ratio, the system will be overloaded.
* @property {Snapshotter} [snapshotter]
@@ -137,7 +131,7 @@ class SystemStatus {
* Where the `isSystemIdle` property is set to `false` if the system
* has been overloaded in the full history of the {@link Snapshotter}
* (which is configurable in the {@link Snapshotter}) and `true` otherwise.
- * @return {Object}
+ * @return {SystemInfo}
*/
getHistoricalStatus() {
return this._isSystemIdle();
@@ -146,7 +140,7 @@ class SystemStatus {
/**
* Returns a system status object.
*
- * @param {Number} [sampleDurationMillis]
+ * @param {number} [sampleDurationMillis]
* @return {SystemInfo}
* @ignore
*/
@@ -168,8 +162,8 @@ class SystemStatus {
* Returns an object with an isOverloaded property set to true
* if the memory has been overloaded in the last sampleDurationMillis.
*
- * @param {Number} sampleDurationMillis
- * @return {Object}
+ * @param {number} sampleDurationMillis
+ * @return {object}
* @ignore
*/
_isMemoryOverloaded(sampleDurationMillis) {
@@ -181,8 +175,8 @@ class SystemStatus {
* Returns an object with an isOverloaded property set to true
* if the event loop has been overloaded in the last sampleDurationMillis.
*
- * @param {Number} sampleDurationMillis
- * @return {Object}
+ * @param {number} sampleDurationMillis
+ * @return {object}
* @ignore
*/
_isEventLoopOverloaded(sampleDurationMillis) {
@@ -194,8 +188,8 @@ class SystemStatus {
* Returns an object with an isOverloaded property set to true
* if the CPU has been overloaded in the last sampleDurationMillis.
*
- * @param {Number} sampleDurationMillis
- * @return {Object}
+ * @param {number} sampleDurationMillis
+ * @return {object}
* @ignore
*/
_isCpuOverloaded(sampleDurationMillis) {
@@ -206,8 +200,8 @@ class SystemStatus {
/**
* Returns an object with an isOverloaded property set to true
* if the client has been overloaded in the last sampleDurationMillis.
- * @param sampleDurationMillis
- * @return {{isOverloaded, maxOverloadedRatio, actualRatio}}
+ * @param {number} sampleDurationMillis
+ * @return {{isOverloaded: boolean, maxOverloadedRatio: number, actualRatio: number}}
* @private
*/
_isClientOverloaded(sampleDurationMillis) {
@@ -219,9 +213,9 @@ class SystemStatus {
* Returns an object with sample information and an isOverloaded property
* set to true if at least the ratio of snapshots in the sample are overloaded.
*
- * @param {Array} sample
- * @param {Number} ratio
- * @return {Object}
+ * @param {Array<*>} sample
+ * @param {number} ratio
+ * @return {object}
* @ignore
*/
_isSampleOverloaded(sample, ratio) { // eslint-disable-line class-methods-use-this
diff --git a/src/constants.js b/src/constants.js
index 83fcdb628..754750226 100644
--- a/src/constants.js
+++ b/src/constants.js
@@ -20,8 +20,7 @@ export const EXIT_CODES = {
/**
* These events are just internal for Apify package, so we don't need them in apify-shared package.
*
- * @type {{CPU_INFO: String, SYSTEM_INFO: String, MIGRATING: String, PERSIST_STATE: String}}
- *
+ * @type {{CPU_INFO: string, SYSTEM_INFO: string, MIGRATING: string, PERSIST_STATE: string}}
*/
export const ACTOR_EVENT_NAMES_EX = Object.assign({}, ACTOR_EVENT_NAMES, { PERSIST_STATE: 'persistState' });
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index 9bd37bf5a..6aee9a9c4 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -1,12 +1,12 @@
import { checkParamOrThrow } from 'apify-client/build/utils';
import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';
-import log from 'apify-shared/log';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
-import _ from 'underscore';
+import * as _ from 'underscore';
import AutoscaledPool from '../autoscaling/autoscaled_pool'; // eslint-disable-line import/no-duplicates
import { RequestList } from '../request_list';
-import { RequestQueue, RequestQueueLocal } from '../request_queue';
+import { RequestQueue, RequestQueueLocal } from '../request_queue'; // eslint-disable-line import/no-duplicates
import events from '../events';
+import log from '../utils_log';
import { openSessionPool } from '../session_pool/session_pool'; // eslint-disable-line import/no-duplicates
import Statistics from './statistics';
import { addTimeoutToPromise } from '../utils';
@@ -15,6 +15,7 @@ import { addTimeoutToPromise } from '../utils';
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
import { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool';
import Request from '../request';
+import { QueueOperationInfo } from '../request_queue';
import { Session } from '../session_pool/session';
import { SessionPoolOptions } from '../session_pool/session_pool';
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
@@ -45,7 +46,7 @@ const DEFAULT_OPTIONS = {
};
/**
- * @typedef {Object} BasicCrawlerOptions
+ * @typedef BasicCrawlerOptions
* @property {HandleRequest} handleRequestFunction
* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
*
@@ -92,9 +93,9 @@ const DEFAULT_OPTIONS = {
* See
* [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)
* for the default implementation of this function.
- * @property {Number} [maxRequestRetries=3]
+ * @property {number} [maxRequestRetries=3]
* Indicates how many times the request is retried if {@link BasicCrawlerOptions.handleRequestFunction} fails.
- * @property {Number} [maxRequestsPerCrawl]
+ * @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -103,14 +104,14 @@ const DEFAULT_OPTIONS = {
* Note that the `runTaskFunction` and `isTaskReadyFunction` options
* are provided by `BasicCrawler` and cannot be overridden.
* However, you can provide a custom implementation of `isFinishedFunction`.
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @property {Boolean} [useSessionPool=false]
+ * @property {boolean} [useSessionPool=false]
* If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
* The session instance will be than available in the `handleRequestFunction`.
* @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {SessionPool} to use.
@@ -173,6 +174,13 @@ const DEFAULT_OPTIONS = {
*
* await crawler.run();
* ```
+ *
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
class BasicCrawler {
/**
@@ -449,8 +457,8 @@ class BasicCrawler {
* Handles errors thrown by user provided handleRequestFunction()
* @param {Error} error
* @param {Request} request
- * @param {RequestList|RequestQueue} source
- * @return {Promise<Boolean>} willBeRetried
+ * @param {(RequestList|RequestQueue)} source
+ * @return {Promise<boolean|void|QueueOperationInfo>} willBeRetried
* @ignore
*/
async _requestFunctionErrorHandler(error, request, source) {
@@ -518,8 +526,9 @@ export default BasicCrawler;
/**
* @callback HandleFailedRequest
* @param {HandleFailedRequestInput} inputs Arguments passed to this callback.
- * @returns {void|Promise<void>}
+ * @returns {(void|Promise<void>)}
*/
+
/**
* @typedef HandleFailedRequestInput
* @property {Request} request The original {Request} object.
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 104bb6a24..bde86218c 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -1,31 +1,30 @@
/* eslint-disable class-methods-use-this */
-import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { readStreamToString, concatStreamToBuffer } from 'apify-shared/streams_utilities';
-import cheerio from 'cheerio';
-import contentTypeParser from 'content-type';
-import htmlparser from 'htmlparser2';
+import * as cheerio from 'cheerio';
+import * as contentTypeParser from 'content-type';
+import * as htmlparser from 'htmlparser2';
import * as iconv from 'iconv-lite';
-import _ from 'underscore';
-import util from 'util';
+import * as _ from 'underscore';
+import * as util from 'util';
import { getApifyProxyUrl } from '../actor';
import { BASIC_CRAWLER_TIMEOUT_MULTIPLIER } from '../constants';
import { TimeoutError } from '../errors';
import { addTimeoutToPromise, parseContentTypeFromResponse } from '../utils';
import * as utilsRequest from '../utils_request'; // eslint-disable-line import/no-duplicates
import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
+import log from '../utils_log';
// TYPE IMPORTS
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
import { IncomingMessage } from 'http';
import AutoscaledPool, { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool';
import { HandleFailedRequest } from './basic_crawler';
-import Request, { RequestOptions } from '../request';
+import Request from '../request';
import { RequestList } from '../request_list';
import { RequestQueue } from '../request_queue';
import { Session } from '../session_pool/session';
import { SessionPoolOptions } from '../session_pool/session_pool';
-import { Cheerio } from '../typedefs';
import { RequestAsBrowserOptions } from '../utils_request';
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
@@ -44,7 +43,7 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {
};
/**
- * @typedef {Object} CheerioCrawlerOptions
+ * @typedef CheerioCrawlerOptions
* @property {CheerioHandlePage} handlePageFunction
* User-provided function that performs the logic of the crawler. It is called for each page
* loaded and parsed by the crawler.
@@ -145,27 +144,27 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {
* this function is therefore not supported, because it would create inconsistencies where
* different parts of SDK would have access to a different {@link Request} instance.
*
- * @property {Number} [handlePageTimeoutSecs=60]
+ * @property {number} [handlePageTimeoutSecs=60]
* Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
- * @property {Number} [requestTimeoutSecs=30]
+ * @property {number} [requestTimeoutSecs=30]
* Timeout in which the HTTP request to the resource needs to finish, given in seconds.
- * @property {Boolean} [ignoreSslErrors=true]
+ * @property {boolean} [ignoreSslErrors=true]
* If set to true, SSL certificate errors will be ignored.
- * @property {Boolean} [useApifyProxy=false]
+ * @property {boolean} [useApifyProxy=false]
* If set to `true`, `CheerioCrawler` will be configured to use
* [Apify Proxy](https://my.apify.com/proxy) for all connections.
* For more information, see the [documentation](https://docs.apify.com/proxy)
- * @property {String[]} [apifyProxyGroups]
+ * @property {string[]} [apifyProxyGroups]
* An array of proxy groups to be used
* by the [Apify Proxy](https://docs.apify.com/proxy).
* Only applied if the `useApifyProxy` option is `true`.
- * @property {String} [apifyProxySession]
+ * @property {string} [apifyProxySession]
* Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.
* All HTTP requests going through the proxy with the same session identifier
* will use the same target proxy server (i.e. the same IP address).
* The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
* Only applied if the `useApifyProxy` option is `true`.
- * @property {String[]} [proxyUrls]
+ * @property {string[]} [proxyUrls]
* An array of custom proxy URLs to be used by the `CheerioCrawler` instance.
* The provided custom proxies' order will be randomized and the resulting list rotated.
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
@@ -185,13 +184,13 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {
*
* See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13)
* for the default implementation of this function.
- * @property {String[]} [additionalMimeTypes]
+ * @property {string[]} [additionalMimeTypes]
* An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
* target="_blank">MIME types</a> you want the crawler to load and process.
* By default, only `text/html` and `application/xhtml+xml` MIME types are supported.
- * @property {Number} [maxRequestRetries=3]
+ * @property {number} [maxRequestRetries=3]
* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
- * @property {Number} [maxRequestsPerCrawl]
+ * @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -201,19 +200,19 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {
* are provided by `CheerioCrawler` and cannot be overridden. Reasonable {@link Snapshotter}
* and {@link SystemStatus} defaults are provided to account for the fact that `cheerio`
* parses HTML synchronously and therefore blocks the event loop.
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @property {Boolean} [useSessionPool=false]
+ * @property {boolean} [useSessionPool=false]
* If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
* It also marks Session as bad after a request timeout.
* @property {SessionPoolOptions} [sessionPoolOptions]
* Custom options passed to the underlying {@link SessionPool} constructor.
- * @property {Boolean} [persistCookiesPerSession]
+ * @property {boolean} [persistCookiesPerSession]
* Automatically saves cookies to Session. Works only if Session Pool is used.
*
* It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.
@@ -296,6 +295,12 @@ const DEFAULT_AUTOSCALED_POOL_OPTIONS = {
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
class CheerioCrawler {
/**
@@ -374,6 +379,7 @@ class CheerioCrawler {
this.persistCookiesPerSession = persistCookiesPerSession;
this.useSessionPool = useSessionPool;
+ /** @ignore */
this.basicCrawler = new BasicCrawler({
// Basic crawler options.
requestList,
@@ -400,7 +406,7 @@ class CheerioCrawler {
/**
* Runs the crawler. Returns promise that gets resolved once all the requests got processed.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async run() {
if (this.isRunningPromise) return this.isRunningPromise;
@@ -537,7 +543,7 @@ class CheerioCrawler {
/**
* Combines the provided `requestOptions` with mandatory (non-overridable) values.
* @param {Request} request
- * @param {Session?} session
+ * @param {Session} [session]
* @ignore
*/
_getRequestOptions(request, session) {
@@ -577,8 +583,8 @@ class CheerioCrawler {
/**
* Enables the use of a proxy by returning a proxy URL
* based on configured options or null if no proxy is used.
- * @param {Session?} session
- * @returns {string|null}
+ * @param {Session} [session]
+ * @returns {(string|null)}
* @ignore
*/
_getProxyUrl(session = {}) {
@@ -630,7 +636,7 @@ class CheerioCrawler {
/**
* Checks and extends supported mime types
- * @param {Array<String|Object>} additionalMimeTypes
+ * @param {Array<(string|Object)>} additionalMimeTypes
* @ignore
*/
_extendSupportedMimeTypes(additionalMimeTypes) {
@@ -646,8 +652,8 @@ class CheerioCrawler {
/**
* Handles blocked request
- * @param session {Session}
- * @param statusCode {Number}
+ * @param {Session} session
+ * @param {number} statusCode
* @private
*/
_throwOnBlockedRequest(session, statusCode) {
@@ -660,7 +666,7 @@ class CheerioCrawler {
/**
* Handles timeout request
- * @param session {Session}
+ * @param {Session} session
* @private
*/
_handleRequestTimeout(session) {
@@ -686,35 +692,38 @@ export default CheerioCrawler;
/**
* @typedef PrepareRequestInputs
* @property {Request} request Original instance fo the {Request} object. Must be modified in-place.
+ * @property {Session} [session] The current session
*/
+
/**
* @callback PrepareRequest
* @param {PrepareRequestInputs} inputs Arguments passed to this callback.
- * @returns {void|Promise<void>}
+ * @returns {(void|Promise<void>)}
*/
/**
* @typedef CheerioHandlePageInputs
- * @property {Cheerio} [$]
+ * @property {CheerioSelector} [$]
* The [Cheerio](https://cheerio.js.org/) object with parsed HTML.
- * @property {String|Buffer} body
+ * @property {(string|Buffer)} body
* The request body of the web page.
- * @property {Object} [json]
+ * @property {*} [json]
* The parsed object from JSON string if the response contains the content type application/json.
* @property {Request} request
- * The original {Request} object.
+ * The original {@link Request} object.
* @property {{ type: string, encoding: string }} contentType
* Parsed `Content-Type header: { type, encoding }`.
* @property {IncomingMessage} response
- * An instance of Node's http.IncomingMessage object,
+ * An instance of Node's http.IncomingMessage object,
* @property {AutoscaledPool} autoscaledPool
* A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
* Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
* You can use it to change the concurrency settings on the fly,
* to pause the crawler by calling {@link AutoscaledPool#pause}
* or to abort it by calling {@link AutoscaledPool#abort}.
- * @property {session} [session]
+ * @property {Session} [session]
*/
+
/**
* @callback CheerioHandlePage
* @param {CheerioHandlePageInputs} inputs Arguments passed to this callback.
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 6aa95d688..92d361a1e 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -1,18 +1,18 @@
import { checkParamOrThrow } from 'apify-client/build/utils';
-import log from 'apify-shared/log';
-import _ from 'underscore';
+import * as _ from 'underscore';
import PuppeteerPool, { BROWSER_SESSION_KEY_NAME } from '../puppeteer_pool'; // eslint-disable-line import/no-duplicates
import { BASIC_CRAWLER_TIMEOUT_MULTIPLIER } from '../constants';
import { gotoExtended } from '../puppeteer_utils';
import { openSessionPool } from '../session_pool/session_pool'; // eslint-disable-line import/no-duplicates
import { addTimeoutToPromise } from '../utils';
import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
+import log from '../utils_log';
// TYPE IMPORTS
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
import { Browser, Page as PuppeteerPage, Response as PuppeteerResponse } from 'puppeteer';
import { HandleFailedRequest } from './basic_crawler';
-import { PuppeteerPoolOptions } from '../puppeteer_pool';
+import { PuppeteerPoolOptions, LaunchPuppeteerFunction } from '../puppeteer_pool';
import Request from '../request'; // eslint-disable-line no-unused-vars
import { RequestList } from '../request_list'; // eslint-disable-line no-unused-vars
import { RequestQueue } from '../request_queue'; // eslint-disable-line no-unused-vars
@@ -23,7 +23,7 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
// eslint-enable-line import/no-duplicates
/**
- * @typedef {Object} PuppeteerCrawlerOptions
+ * @typedef PuppeteerCrawlerOptions
* @property {PuppeteerHandlePage} handlePageFunction
* Function that is called to process each request.
* It is passed an object with the following fields:
@@ -63,7 +63,7 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
* @property {RequestQueue} [requestQueue]
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
* Either `requestList` or `requestQueue` option must be provided (or both).
- * @property {Number} [handlePageTimeoutSecs=60]
+ * @property {number} [handlePageTimeoutSecs=60]
* Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
* @property {PuppeteerGoto} [gotoFunction]
* Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's
@@ -78,7 +78,7 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
* By default, the function invokes {@link puppeteer#gotoExtended} with a timeout of 60 seconds.
* For details, see source code on
* [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292).
- * @property {Number} [gotoTimeoutSecs=60]
+ * @property {number} [gotoTimeoutSecs=60]
* Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
* function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
* @property {HandleFailedRequest} [handleFailedRequestFunction]
@@ -97,16 +97,16 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
* See
* [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301)
* for the default implementation of this function.
- * @property {Number} [maxRequestRetries=3]
+ * @property {number} [maxRequestRetries=3]
* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
- * @property {Number} [maxRequestsPerCrawl]
+ * @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
* @property {PuppeteerPoolOptions} [puppeteerPoolOptions]
* Custom options passed to the underlying {@link PuppeteerPool} constructor.
* You can tweak those to fine-tune browser management.
- * @property {Function} [launchPuppeteerFunction]
+ * @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* Shortcut to the corresponding {@link PuppeteerPool} option.
* See source code on
@@ -119,22 +119,22 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
* Custom options passed to the underlying {@link AutoscaledPool} instance constructor.
* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
* are provided by `PuppeteerCrawler` and should not be overridden.
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the
* corresponding {@link AutoscaledPoolOptions.minConcurrency} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU,
* your crawler will run extremely slow or crash. If you're not sure, just keep the default value
* and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the
* corresponding {@link AutoscaledPoolOptions.maxConcurrency} option.
- * @property {Boolean} [useSessionPool=false]
+ * @property {boolean} [useSessionPool=false]
* If set to true Crawler will automatically use Session Pool. It will automatically retire
* sessions on 403, 401 and 429 status codes. It also marks Session as bad after a request timeout.
* @property {SessionPoolOptions} [sessionPoolOptions]
* Custom options passed to the underlying {@link SessionPool} constructor.
- * @property {Boolean} [persistCookiesPerSession=false]
+ * @property {boolean} [persistCookiesPerSession=false]
* Automatically saves cookies to Session. Works only if Session Pool is used.
*/
@@ -199,6 +199,13 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
+ *
*/
class PuppeteerCrawler {
/**
@@ -266,6 +273,7 @@ class PuppeteerCrawler {
this.sessionPoolOptions = sessionPoolOptions;
this.persistCookiesPerSession = persistCookiesPerSession;
+ /** @ignore */
this.basicCrawler = new BasicCrawler({
// Basic crawler options.
requestList,
@@ -399,6 +407,7 @@ export default PuppeteerCrawler;
* or to abort it by calling {@link AutoscaledPool#abort}.
* @property {Session} [session]
*/
+
/**
* @callback PuppeteerHandlePage
* @param {PuppeteerHandlePageInputs} inputs Arguments passed to this callback.
@@ -409,15 +418,17 @@ export default PuppeteerCrawler;
* @typedef PuppeteerGotoInputs
* @property {PuppeteerPage} page is an instance of the Puppeteer
* [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)
- * @property {Request} request An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * @property {Request} request
+ * An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
* @property {AutoscaledPool} autoscaledPool An instance of the `AutoscaledPool`.
* @property {PuppeteerPool} puppeteerPool An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
* @property {Session} [session] `Session` object for this request.
*/
+
/**
* @callback PuppeteerGoto
* @param {PuppeteerGotoInputs} inputs Arguments passed to this callback.
- * @return {Promise<PuppeteerResponse>} An instance of the Puppeteer
+ * @return {Promise<(PuppeteerResponse | null)>} An instance of the Puppeteer
* [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),
* which is the main resource response as returned by `page.goto(request.url)`.
*/
diff --git a/src/crawlers/statistics.js b/src/crawlers/statistics.js
index a29aef534..231540731 100644
--- a/src/crawlers/statistics.js
+++ b/src/crawlers/statistics.js
@@ -1,5 +1,5 @@
-import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
+import log from '../utils_log';
class Job {
constructor() {
diff --git a/src/dataset.js b/src/dataset.js
index e1c563002..53340b0d8 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -1,12 +1,12 @@
-import path from 'path';
+import * as path from 'path';
import { promisify } from 'util';
-import fs from 'fs-extra';
-import _ from 'underscore';
+import * as fs from 'fs-extra';
+import * as _ from 'underscore';
import { leftpad } from 'apify-shared/utilities';
-import LruCache from 'apify-shared/lru_cache';
-import log from 'apify-shared/log';
+import * as LruCache from 'apify-shared/lru_cache';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { ENV_VARS, LOCAL_STORAGE_SUBDIRS, MAX_PAYLOAD_SIZE_BYTES } from 'apify-shared/consts';
+import log from './utils_log';
import { apifyClient, ensureDirExists, openRemoteStorage, openLocalStorage, ensureTokenOrLocalStorageEnvExists } from './utils';
export const DATASET_ITERATORS_DEFAULT_LIMIT = 10000;
@@ -33,8 +33,8 @@ const datasetsCache = new LruCache({ maxLength: MAX_OPENED_STORES }); // Open Da
* in an array to provide better error messages. Returns serialized object.
*
* @param {Object} item
- * @param {Number} limitBytes
- * @param {Number} [index]
+ * @param {number} limitBytes
+ * @param {number} [index]
* @returns {string}
* @ignore
*/
@@ -63,9 +63,9 @@ export const checkAndSerialize = (item, limitBytes, index) => {
*
* The function assumes that none of the items is larger than limitBytes and does not validate.
*
- * @param {Array} items
- * @param {Number} limitBytes
- * @returns {Array}
+ * @param {Array<*>} items
+ * @param {number} limitBytes
+ * @returns {Array<*>}
* @ignore
*/
export const chunkBySize = (items, limitBytes) => {
@@ -178,8 +178,7 @@ export class Dataset {
* the items have already been saved to the dataset while other items from the source array were not.
* To overcome this limitation, the developer may, for example, read the last item saved in the dataset
* and re-attempt the save of the data from this item onwards to prevent duplicates.
- *
- * @param {Object|Array} data Object or array of objects containing data to be stored in the default dataset.
+ * @param {object|Array<object>} data Object or array of objects containing data to be stored in the default dataset.
* The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
* @return {Promise<void>}
*/
@@ -214,51 +213,51 @@ export class Dataset {
*
* @param {Object} [options] All `getData()` parameters are passed
* via an options object with the following keys:
- * @param {String} [options.format='json']
+ * @param {string} [options.format='json']
* Format of the `items` property, possible values are: `json`, `csv`, `xlsx`, `html`, `xml` and `rss`.
- * @param {Number} [options.offset=0]
+ * @param {number} [options.offset=0]
* Number of array elements that should be skipped at the start.
- * @param {Number} [options.limit=250000]
+ * @param {number} [options.limit=250000]
* Maximum number of array elements to return.
- * @param {Boolean} [options.desc=false]
+ * @param {boolean} [options.desc=false]
* If `true` then the objects are sorted by `createdAt` in descending order.
* Otherwise they are sorted in ascending order.
- * @param {Array} [options.fields]
+ * @param {string[]} [options.fields]
* An array of field names that will be included in the result. If omitted, all fields are included in the results.
- * @param {String} [options.unwind]
+ * @param {string} [options.unwind]
* Specifies a name of the field in the result objects that will be used to unwind the resulting objects.
* By default, the results are returned as they are.
- * @param {Boolean} [options.disableBodyParser=false]
+ * @param {boolean} [options.disableBodyParser=false]
* If `true` then response from API will not be parsed.
- * @param {Boolean} [options.attachment=false]
+ * @param {boolean} [options.attachment=false]
* If `true` then the response will define the `Content-Disposition: attachment` HTTP header, forcing a web
* browser to download the file rather than to display it. By default, this header is not present.
- * @param {String} [options.delimiter=',']
+ * @param {string} [options.delimiter=',']
* A delimiter character for CSV files, only used if `format` is `csv`.
- * @param {Boolean} [options.bom]
+ * @param {boolean} [options.bom]
* All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte
* Order Mark (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default
* behavior, set `bom` option to `true` to include the BOM, or set `bom` to `false` to skip it.
- * @param {String} [options.xmlRoot='results']
+ * @param {string} [options.xmlRoot='results']
* Overrides the default root element name of the XML output. By default, the root element is `results`.
- * @param {String} [options.xmlRow='page']
+ * @param {string} [options.xmlRow='page']
* Overrides the default element name that wraps each page or page function result object in XML output.
* By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
- * @param {Boolean} [options.skipHeaderRow=false]
+ * @param {boolean} [options.skipHeaderRow=false]
* If set to `true` then header row in CSV format is skipped.
- * @param {Boolean} [options.clean=false]
+ * @param {boolean} [options.clean=false]
* If `true` then the function returns only non-empty items and skips hidden fields (i.e. fields starting with `#` character).
* Note that the `clean` parameter is a shortcut for `skipHidden: true` and `skipEmpty: true` options.
- * @param {Boolean} [options.skipHidden=false]
+ * @param {boolean} [options.skipHidden=false]
* If `true` then the function doesn't return hidden fields (fields starting with "#" character).
- * @param {Boolean} [options.skipEmpty=false]
+ * @param {boolean} [options.skipEmpty=false]
* If `true` then the function doesn't return empty items.
* Note that in this case the returned number of items might be lower than limit parameter and pagination must be done using the `limit` value.
- * @param {Boolean} [options.simplified]
+ * @param {boolean} [options.simplified]
* If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and `unwind: 'pageFunctionResult'` options.
* This feature is used to emulate simplified results provided by Apify API version 1 used for
* the legacy Apify Crawler and it's not recommended to use it in new integrations.
- * @param {Boolean} [options.skipFailedPages]
+ * @param {boolean} [options.skipFailedPages]
* If `true` then, the all the items with errorInfo property will be skipped from the output.
* This feature is here to emulate functionality of Apify API version 1 used for
* the legacy Apify Crawler product and it's not recommended to use it in new integrations.
@@ -307,7 +306,7 @@ export class Dataset {
* }
* ```
*
- * @returns {Promise<Object>}
+ * @returns {Promise<object>}
*/
async getInfo() {
return datasets.getDataset({ datasetId: this.datasetId });
@@ -331,10 +330,10 @@ export class Dataset {
* @param {DatasetConsumer} iteratee A function that is called for every item in the dataset.
* @param {Object} [options] All `forEach()` parameters are passed
* via an options object with the following keys:
- * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by `createdAt` in descending order.
- * @param {Array} [options.fields] If provided then returned objects will only contain specified keys.
- * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @param {Number} [index=0] Specifies the initial index number passed to the `iteratee` function.
+ * @param {boolean} [options.desc=false] If `true` then the objects are sorted by `createdAt` in descending order.
+ * @param {string[]} [options.fields] If provided then returned objects will only contain specified keys.
+ * @param {string} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @param {number} [index=0] Specifies the initial index number passed to the `iteratee` function.
* @return {Promise<void>}
*/
async forEach(iteratee, options = {}, index = 0) {
@@ -363,14 +362,13 @@ export class Dataset {
*
* If `iteratee` returns a `Promise` then it's awaited before a next call.
*
- * @template T
* @param {DatasetMapper} iteratee
- * @param {Object} options All `map()` parameters are passed
+ * @param {Object} [options] All `map()` parameters are passed
* via an options object with the following keys:
- * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
- * @param {Array} [options.fields] If provided then returned objects will only contain specified keys
- * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @return {Promise<T[]>}
+ * @param {boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
+ * @param {string[]} [options.fields] If provided then returned objects will only contain specified keys
+ * @param {string} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @return {Promise<Array<object>>}
*/
map(iteratee, options) {
const result = [];
@@ -398,15 +396,14 @@ export class Dataset {
*
* If `iteratee()` returns a `Promise` then it's awaited before a next call.
*
- * @template T
* @param {DatasetReducer} iteratee
- * @param {T} memo Initial state of the reduction.
- * @param {Object} options All `reduce()` parameters are passed
+ * @param {object} memo Initial state of the reduction.
+ * @param {Object} [options] All `reduce()` parameters are passed
* via an options object with the following keys:
- * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
- * @param {Array} [options.fields] If provided then returned objects will only contain specified keys
- * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @return {Promise<T>}
+ * @param {boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
+ * @param {string[]} [options.fields] If provided then returned objects will only contain specified keys
+ * @param {string} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @return {Promise<object>}
*/
reduce(iteratee, memo, options) {
let currentMemo = memo;
@@ -670,7 +667,7 @@ const getOrCreateDataset = (datasetIdOrName) => {
* @param {string} [datasetIdOrName]
* ID or name of the dataset to be opened. If `null` or `undefined`,
* the function returns the default dataset associated with the actor run.
- * @param {object} [options]
+ * @param {Object} [options]
* @param {boolean} [options.forceCloud=false]
* If set to `true` then the function uses cloud storage usage even if the `APIFY_LOCAL_STORAGE_DIR`
* environment variable is set. This way it is possible to combine local and cloud storage.
@@ -712,9 +709,9 @@ export const openDataset = (datasetIdOrName, options = {}) => {
* **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,
* otherwise the actor process might finish before the data are stored!
*
- * @param {Object|Array} item Object or array of objects containing data to be stored in the default dataset.
+ * @param {object} item Object or array of objects containing data to be stored in the default dataset.
* The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
- * @returns {Promise}
+ * @returns {Promise<void>}
*
* @memberof module:Apify
* @name pushData
@@ -724,37 +721,36 @@ export const pushData = item => openDataset().then(dataset => dataset.pushData(i
/**
* @typedef DatasetContent
- * @property {Object[]|String[]|Buffer[]} items Dataset entries based on chosen format parameter.
- * @property {Number} total Total count of entries in the dataset.
- * @property {Number} offset Position of the first returned entry in the dataset.
- * @property {Number} count Count of dataset entries returned in this set.
- * @property {Number} limit Maximum number of dataset entries requested.
+ * @property {Array<object>} items Dataset entries based on chosen format parameter.
+ * @property {number} total Total count of entries in the dataset.
+ * @property {number} offset Position of the first returned entry in the dataset.
+ * @property {number} count Count of dataset entries returned in this set.
+ * @property {number} limit Maximum number of dataset entries requested.
*/
-// TODO yin: Typescript candoes not understand `@callback` with generic `@template T`. Change after this is fixed
/**
* User-function used in the `Dataset.forEach()` API.
+ *
* @callback DatasetConsumer
- * @param {Object} item Current {@link Dataset} entry being processed.
- * @param {Number} index Position of current {Dataset} entry.
- * @returns T
+ * @param {object} item Current {@link Dataset} entry being processed.
+ * @param {number} index Position of current {Dataset} entry.
+ * @returns {object}
*/
-// TODO yin: Typescript candoes not understand `@callback` with generic `@template T`. Change after this is fixed
/**
* User-function used in the `Dataset.map()` API.
+ *
* @callback DatasetMapper
- * @param {Object} item Currect {@link Dataset} entry being processed.
- * @param {Number} index Position of current {Dataset} entry.
- * @returns T
+ * @param {object} item Currect {@link Dataset} entry being processed.
+ * @param {number} index Position of current {Dataset} entry.
+ * @returns {object}
*/
-// TODO yin: Typescript candoes not understand `@callback` with generic `@template T`. Change after this is fixed
/**
* User-function used in the `Dataset.reduce()` API.
* @callback DatasetReducer
- * @param {T} memo Previous state of the reduction.
- * @param {Object} item Currect {@link Dataset} entry being processed.
- * @param {Number} index Position of current {Dataset} entry.
- * @returns T
+ * @param {object} memo Previous state of the reduction.
+ * @param {object} item Currect {@link Dataset} entry being processed.
+ * @param {number} index Position of current {Dataset} entry.
+ * @returns {object}
*/
diff --git a/src/enqueue_links/click_elements.js b/src/enqueue_links/click_elements.js
index 275ec0ba1..486206d19 100644
--- a/src/enqueue_links/click_elements.js
+++ b/src/enqueue_links/click_elements.js
@@ -1,11 +1,19 @@
import { URL } from 'url';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
+import log from '../utils_log';
import { RequestQueue, RequestQueueLocal, QueueOperationInfo } from '../request_queue'; // eslint-disable-line import/named,no-unused-vars
import { addInterceptRequestHandler, removeInterceptRequestHandler } from '../puppeteer_request_interception';
-// eslint-disable-next-line import/named,no-unused-vars
-import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatches, createRequestOptions, RequestTransform } from './shared';
+/* eslint-disable import/named,no-unused-vars,import/order */
+import { Page, Request as PuppeteerRequest, Target } from 'puppeteer';
+import {
+ constructPseudoUrlInstances,
+ createRequests,
+ addRequestsToQueueInBatches,
+ createRequestOptions,
+ RequestTransform,
+} from './shared';
+/* eslint-enable import/named,no-unused-vars,import/order */
const STARTING_Z_INDEX = 2147400000;
@@ -53,7 +61,6 @@ const STARTING_Z_INDEX = 2147400000;
* ],
* });
* ```
- *
* @param {Object} options
* All `enqueueLinksByClickingElements()` parameters are passed
* via an options object with the following keys:
@@ -61,10 +68,10 @@ const STARTING_Z_INDEX = 2147400000;
* Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* @param {RequestQueue} options.requestQueue
* A request queue to which the URLs will be enqueued.
- * @param {String} options.selector
+ * @param {string} options.selector
* A CSS selector matching elements to be clicked on. Unlike in {@link utils#enqueueLinks}, there is no default
* value. This is to prevent suboptimal use of this function by using it too broadly.
- * @param {Array<String|RegExp|Object>} [options.pseudoUrls]
+ * @param {Array<(string|RegExp|Object)>} [options.pseudoUrls]
* An array of {@link PseudoUrl}s matching the URLs to be enqueued,
* or an array of strings or RegExps or plain Objects from which the {@link PseudoUrl}s can be constructed.
*
@@ -111,7 +118,7 @@ const STARTING_Z_INDEX = 2147400000;
* This is the maximum period for which the function will keep tracking events, even if more events keep coming.
* Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself.
* See `waitForPageIdleSecs` above for an explanation.
- * @return {Promise<QueueOperationInfo[]>}
+ * @return {Promise<Array<QueueOperationInfo>>}
* Promise that resolves to an array of {@link QueueOperationInfo} objects.
* @memberOf puppeteer
* @name enqueueLinksByClickingElements
@@ -162,7 +169,7 @@ export async function enqueueLinksByClickingElements(options = {}) {
* @param {Object} options
* @param {Page} options.page
* @param {string} options.selector
- * @return {Promise<Object[]>}
+ * @return {Promise<Array<object>>}
* @ignore
*/
export async function clickElementsAndInterceptNavigationRequests(options) {
@@ -201,7 +208,7 @@ export async function clickElementsAndInterceptNavigationRequests(options) {
/**
* @param {Page} page
- * @param {Set} requests
+ * @param {Set<*>} requests
* @return {Function}
* @ignore
*/
@@ -226,7 +233,7 @@ function createInterceptRequestHandler(page, requests) {
/**
* @param {Page} page
- * @param {Request} req
+ * @param {PuppeteerRequest} req
* @return {boolean}
* @ignore
*/
@@ -237,7 +244,7 @@ function isTopFrameNavigationRequest(page, req) {
/**
* @param {Page} page
- * @param {Set} requests
+ * @param {Set<*>} requests
* @return {Function}
* @ignore
*/
@@ -272,7 +279,7 @@ export function isTargetRelevant(page, target) {
/**
* @param {Page} page
- * @param {Set} requests
+ * @param {Set<*>} requests
* @return {Function}
* @ignore
*/
@@ -286,7 +293,7 @@ function createFrameNavigatedHandler(page, requests) {
/**
* @param {Page} page
- * @return {Promise}
+ * @return {Promise<*>}
* @ignore
*/
async function preventHistoryNavigation(page) {
@@ -320,7 +327,7 @@ async function preventHistoryNavigation(page) {
*
* @param {Page} page
* @param {string} selector
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
export async function clickElements(page, selector) {
@@ -382,7 +389,7 @@ function updateElementCssToEnableMouseClick(el, zIndex) {
* @param {Page} options.page
* @param {number} options.waitForPageIdleMillis
* @param {number} options.maxWaitForPageIdleMillis
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
async function waitForPageIdle({ page, waitForPageIdleMillis, maxWaitForPageIdleMillis }) {
@@ -425,8 +432,8 @@ async function waitForPageIdle({ page, waitForPageIdleMillis, maxWaitForPageIdle
/**
* @param {Page} page
- * @param {Set} requests
- * @return {Promise}
+ * @param {Set<*>} requests
+ * @return {Promise<void>}
* @ignore
*/
async function restoreHistoryNavigationAndSaveCapturedUrls(page, requests) {
diff --git a/src/enqueue_links/enqueue_links.js b/src/enqueue_links/enqueue_links.js
index 8dec5e63f..da1ed1c9c 100644
--- a/src/enqueue_links/enqueue_links.js
+++ b/src/enqueue_links/enqueue_links.js
@@ -1,7 +1,7 @@
import { URL } from 'url';
-import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
+import log from '../utils_log';
/* eslint-disable import/no-duplicates */
import { RequestQueue, RequestQueueLocal } from '../request_queue';
import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatches, createRequestOptions } from './shared';
@@ -10,10 +10,8 @@ import { constructPseudoUrlInstances, createRequests, addRequestsToQueueInBatche
// TYPE IMPORTS
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
import { Page } from 'puppeteer';
-import { RequestOptions } from '../request';
import { QueueOperationInfo } from '../request_queue';
import { RequestTransform } from './shared';
-import { Cheerio } from '../typedefs';
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
@@ -51,20 +49,20 @@ import { Cheerio } from '../typedefs';
* @param {Object} options
* All `enqueueLinks()` parameters are passed
* via an options object with the following keys:
- * @param {Page} options.page
+ * @param {Page} [options.page]
* Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* Either `page` or `$` option must be provided.
- * @param {Cheerio} options.$
+ * @param {CheerioStatic} [options.$]
* [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML.
* Either `page` or `$` option must be provided.
* @param {RequestQueue} options.requestQueue
* A request queue to which the URLs will be enqueued.
- * @param {String} [options.selector='a']
+ * @param {string} [options.selector='a']
* A CSS selector matching links to be enqueued.
* @param {string} [options.baseUrl]
* A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer,
* since the relative URL resolution is done inside the browser automatically.
- * @param {Object[]|String[]} [options.pseudoUrls]
+ * @param {Array<Object>|Array<string>} [options.pseudoUrls]
* An array of {@link PseudoUrl}s matching the URLs to be enqueued,
* or an array of strings or RegExps or plain Objects from which the {@link PseudoUrl}s can be constructed.
*
@@ -93,7 +91,7 @@ import { Cheerio } from '../typedefs';
* }
* }
* ```
- * @return {Promise<QueueOperationInfo[]>}
+ * @return {Promise<Array<QueueOperationInfo>>}
* Promise that resolves to an array of {@link QueueOperationInfo} objects.
* @memberOf utils
* @name enqueueLinks
@@ -148,7 +146,7 @@ export async function enqueueLinks(options = {}) {
*
* @param {Page} page
* @param {string} selector
- * @return {string[]}
+ * @return {Promise<Array<string>>}
* @ignore
*/
export async function extractUrlsFromPage(page, selector) {
@@ -159,7 +157,7 @@ export async function extractUrlsFromPage(page, selector) {
/**
* Extracts URLs from a given Cheerio object.
*
- * @param {Cheerio} $
+ * @param {CheerioStatic} $
* @param {string} selector
* @param {string} baseUrl
* @return {string[]}
diff --git a/src/enqueue_links/shared.js b/src/enqueue_links/shared.js
index 4234f80ec..f08d73d00 100644
--- a/src/enqueue_links/shared.js
+++ b/src/enqueue_links/shared.js
@@ -1,9 +1,14 @@
import { URL } from 'url';
-import _ from 'underscore';
+import * as _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
import PseudoUrl from '../pseudo_url';
-import Request from '../request';
+import Request from '../request'; // eslint-disable-line import/no-duplicates
+// TYPES IMPORT
+/* eslint-disable import/order,no-unused-vars,import/named,import/no-duplicates */
+import { RequestQueue, QueueOperationInfo } from '../request_queue';
+import { RequestOptions } from '../request';
+/* eslint-enable */
const MAX_ENQUEUE_LINKS_CACHE_SIZE = 1000;
@@ -17,8 +22,8 @@ const enqueueLinksPseudoUrlCache = new Map();
/**
* Helper factory used in the `enqueueLinks()` and enqueueLinksByClickingElements() function.
- * @param {string[]|Object[]} pseudoUrls
- * @return {PseudoUrl[]}
+ * @param {Array<(string|Object)>} pseudoUrls
+ * @return {Array<PseudoUrl>}
* @ignore
*/
export function constructPseudoUrlInstances(pseudoUrls) {
@@ -46,9 +51,9 @@ export function constructPseudoUrlInstances(pseudoUrls) {
});
}
/**
- * @param {string[]|Object[]} requestOptions
- * @param {PseudoUrl[]} pseudoUrls
- * @return {Request[]}
+ * @param {Array<(string|Object)>} requestOptions
+ * @param {Array<PseudoUrl>} pseudoUrls
+ * @return {Array<Request>}
* @ignore
*/
export function createRequests(requestOptions, pseudoUrls) {
@@ -69,7 +74,7 @@ export function createRequests(requestOptions, pseudoUrls) {
}
/**
- * @param {string[]|Object[]} sources
+ * @param {Array<(string|Object)>} sources
* @param {Object} [userData]
* @ignore
*/
@@ -90,10 +95,10 @@ export function createRequestOptions(sources, userData = {}) {
}
/**
- * @param {Request[]} requests
+ * @param {Array<Request>} requests
* @param {RequestQueue} requestQueue
* @param {number} batchSize
- * @return {Promise<QueueOperationInfo[]>}
+ * @return {Promise<Array<QueueOperationInfo>>}
* @ignore
*/
export async function addRequestsToQueueInBatches(requests, requestQueue, batchSize = 5) {
diff --git a/src/errors.js b/src/errors.js
index 280fca823..eabd934e5 100644
--- a/src/errors.js
+++ b/src/errors.js
@@ -1,23 +1,24 @@
-// TYPE IMPORTS
-/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
-import { ActorRun } from './typedefs';
-/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
-
export const APIFY_CALL_ERROR_NAME = 'ApifyCallError';
+// eslint-disable-next-line import/named,no-unused-vars,import/first
+import { ActorRun } from './typedefs';
+
/**
* The class represents exceptions thrown
* by the {@link Apify#call} function.
*
- * @typedef {Object} ApifyCallError
- * @property {String} message
+ * @property {string} message
* Error message
- * @property {String} name
- * Contains `"ApifyCallError"`
* @property {ActorRun} run
* Object representing the failed actor run.
+ * @property {string} name
+ * Contains `"ApifyCallError"`
*/
export class ApifyCallError extends Error {
+ /**
+ * @param {ActorRun} run
+ * @param {string} [message]
+ */
constructor(run, message = 'The actor invoked by Apify.call() did not succeed') {
super(`${message} (run ID: ${run.id})`);
this.name = APIFY_CALL_ERROR_NAME;
@@ -26,6 +27,7 @@ export class ApifyCallError extends Error {
Error.captureStackTrace(this, ApifyCallError);
}
}
+
/**
* TimeoutError class.
* This error should be thrown after request timeout from `requestAsBrowser`.
diff --git a/src/events.js b/src/events.js
index 295e42b90..071961582 100644
--- a/src/events.js
+++ b/src/events.js
@@ -1,7 +1,7 @@
import { EventEmitter } from 'events';
-import WebSocket from 'ws';
-import log from 'apify-shared/log';
+import * as WebSocket from 'ws';
import { ENV_VARS, ACTOR_EVENT_NAMES } from 'apify-shared/consts';
+import log from './utils_log';
import { ACTOR_EVENT_NAMES_EX } from './constants';
// NOTE: This value is mentioned below in docs, if you update it here, update it there too.
diff --git a/src/index.js b/src/index.js
index d88667e23..66dce3aef 100644
--- a/src/index.js
+++ b/src/index.js
@@ -1,4 +1,4 @@
-import * as Apify from './index.esm';
+import * as Apify from './main';
export default Apify;
-export * from './index.esm';
+export * from './main';
diff --git a/src/key_value_store.js b/src/key_value_store.js
index a5ec102ce..441e033fe 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -1,14 +1,14 @@
-import fs from 'fs-extra';
-import path from 'path';
+import * as fs from 'fs-extra';
+import * as path from 'path';
import { promisify } from 'util';
-import contentTypeParser from 'content-type';
-import mime from 'mime-types';
-import LruCache from 'apify-shared/lru_cache';
+import * as contentTypeParser from 'content-type';
+import * as mime from 'mime-types';
+import * as LruCache from 'apify-shared/lru_cache';
import { KEY_VALUE_STORE_KEY_REGEX } from 'apify-shared/regexs';
import { ENV_VARS, LOCAL_STORAGE_SUBDIRS, KEY_VALUE_STORE_KEYS } from 'apify-shared/consts';
import { jsonStringifyExtended } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
import { checkParamOrThrow, parseBody } from 'apify-client/build/utils';
+import log from './utils_log';
import {
addCharsetToContentType, apifyClient, ensureDirExists, openRemoteStorage, openLocalStorage, ensureTokenOrLocalStorageEnvExists,
} from './utils';
@@ -195,11 +195,10 @@ export class KeyValueStore {
* const store = await Apify.openKeyValueStore();
* const buffer = await store.getValue('screenshot1.png');
* ```
- *
- * @param {String} key
+ * @param {string} key
* Unique key of the record. It can be at most 256 characters long and only consist
* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
- * @returns {Promise<Object|String|Buffer>}
+ * @returns {Promise<(object|Buffer|string|null)>}
* Returns a promise that resolves to an object, string
* or [](https://nodejs.org/api/buffer.html), depending
* on the MIME content type of the record.
@@ -247,10 +246,10 @@ export class KeyValueStore {
* **IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`,
* otherwise the actor process might finish before the value is stored!
*
- * @param {String} key
+ * @param {string} key
* Unique key of the record. It can be at most 256 characters long and only consist
* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
- * @param {Object|String|Buffer} value
+ * @param {(Object|string|Buffer|null)} value
* Record data, which can be one of the following values:
* <ul>
* <li>If `null`, the record in the key-value store is deleted.</li>
@@ -260,7 +259,7 @@ export class KeyValueStore {
* </ul>
* For any other value an error will be thrown.
* @param {Object} [options]
- * @param {String} [options.contentType]
+ * @param {string} [options.contentType]
* Specifies a custom MIME content type of the record.
* @returns {Promise<void>}
*
@@ -358,7 +357,7 @@ export class KeyValueStore {
/**
* Helper to create a file-matching RegExp from a KeyValueStore key.
- * @param {String} key
+ * @param {string} key
* @returns {RegExp}
* @ignore
*/
@@ -488,9 +487,9 @@ export class KeyValueStoreLocal {
*
* Returns an object when a file is found and handler executes successfully, null otherwise.
*
- * @param {String} key
+ * @param {string} key
* @param {Function} handler
- * @returns {Promise} null or object in the following format:
+ * @returns {Promise<*>} null or object in the following format:
* {
* returnValue: return value of the handler function,
* fileName: name of the file including found extension
@@ -514,9 +513,9 @@ export class KeyValueStoreLocal {
/**
* Performs a lookup for a file in the local emulation directory's file list.
- * @param {String} key
+ * @param {string} key
* @param {Function} handler
- * @returns {Promise}
+ * @returns {Promise<*>}
* @ignore
*/
_fullDirectoryLookup(key, handler) {
@@ -532,8 +531,8 @@ export class KeyValueStoreLocal {
/**
* Helper function to resolve file paths.
- * @param {String} fileName
- * @returns {String}
+ * @param {string} fileName
+ * @returns {string}
* @ignore
*/
_getPath(fileName) {
@@ -629,9 +628,9 @@ export const openKeyValueStore = (storeIdOrName, options = {}) => {
* For more information, see {@link Apify#openKeyValueStore}
* and {@link KeyValueStore#getValue}.
*
- * @param {String} key
+ * @param {string} key
* Unique record key.
- * @returns {Promise<Object>}
+ * @returns {Promise<(object|null)>}
* Returns a promise that resolves once the record is stored.
*
* @memberof module:Apify
@@ -664,9 +663,9 @@ export const getValue = async (key) => {
* For more information, see {@link Apify#openKeyValueStore}
* and {@link KeyValueStore#getValue}.
*
- * @param {String} key
+ * @param {string} key
* Unique record key.
- * @param {Object|String|Buffer} value
+ * @param {object} value
* Record data, which can be one of the following values:
* <ul>
* <li>If `null`, the record in the key-value store is deleted.</li>
@@ -676,9 +675,9 @@ export const getValue = async (key) => {
* </ul>
* For any other value an error will be thrown.
* @param {Object} [options]
- * @param {String} [options.contentType]
+ * @param {string} [options.contentType]
* Specifies a custom MIME content type of the record.
- * @return {Promise}
+ * @return {Promise<void>}
* @memberof module:Apify
* @name setValue
* @function
@@ -707,7 +706,7 @@ export const setValue = async (key, value, options) => {
* For more information, see {@link Apify#openKeyValueStore}
* and {@link KeyValueStore#getValue}.
*
- * @returns {Promise<Object>}
+ * @returns {Promise<(object|null)>}
* Returns a promise that resolves once the record is stored.
* @memberof module:Apify
* @name getInput
@@ -719,12 +718,12 @@ export const getInput = async () => getValue(process.env[ENV_VARS.INPUT_KEY] ||
/**
* User-function used in the {@link KeyValueStore#forEachKey} method.
* @callback KeyConsumer
- * @param {String} key
+ * @param {string} key
* Current {KeyValue} key being processed.
- * @param {Number} index
+ * @param {number} index
* Position of the current key in {KeyValuestore}.
- * @param {Object} info
+ * @param {object} info
* Information about the current {KeyValueStore} entry.
- * @param {Number} info.size
+ * @param {number} info.size
* Size of the value associated with the current key in bytes.
*/
diff --git a/src/live_view/live_view_server.js b/src/live_view/live_view_server.js
index d8e2e206c..28799142f 100644
--- a/src/live_view/live_view_server.js
+++ b/src/live_view/live_view_server.js
@@ -1,14 +1,14 @@
-import http from 'http';
-import fs from 'fs-extra';
-import path from 'path';
+import * as http from 'http';
+import * as fs from 'fs-extra';
+import * as path from 'path';
import { promisify } from 'util';
-import express from 'express';
-import socketio from 'socket.io';
-import log from 'apify-shared/log';
+import * as express from 'express';
+import * as socketio from 'socket.io';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { promisifyServerListen } from 'apify-shared/utilities';
import { ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';
import { Page } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import log from '../utils_log';
import { addTimeoutToPromise } from '../utils';
import Snapshot from './snapshot';
@@ -109,7 +109,7 @@ class LiveViewServer {
/**
* Starts the HTTP server with web socket connections enabled.
* Snapshots will not be created until a client has connected.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async start() {
this._isRunning = true;
@@ -126,7 +126,7 @@ class LiveViewServer {
/**
* Prevents the server from receiving more connections. Existing connections
* will not be terminated, but the server will not prevent a process exit.
- * @return {Promise}
+ * @return {Promise<void>}
*/
async stop() {
this.httpServer.unref();
@@ -148,7 +148,7 @@ class LiveViewServer {
* Will time out and throw in `options.snapshotTimeoutSecs`.
*
* @param {Page} page
- * @return {Promise}
+ * @return {Promise<void>}
*/
async serve(page) {
if (!this.hasClients()) return;
diff --git a/src/index.esm.js b/src/main.js
similarity index 98%
rename from src/index.esm.js
rename to src/main.js
index 019d5a9f2..f8577f214 100644
--- a/src/index.esm.js
+++ b/src/main.js
@@ -1,5 +1,5 @@
import { EventEmitter } from 'events';
-import log from 'apify-shared/log';
+import log from './utils_log';
import { main, getEnv, call, callTask, getApifyProxyUrl, metamorph, addWebhook } from './actor';
import AutoscaledPool from './autoscaling/autoscaled_pool';
import BasicCrawler from './crawlers/basic_crawler';
diff --git a/src/pseudo_url.js b/src/pseudo_url.js
index bb52b5f7e..8033f0c7d 100644
--- a/src/pseudo_url.js
+++ b/src/pseudo_url.js
@@ -1,7 +1,7 @@
-import _ from 'underscore';
-import log from 'apify-shared/log';
+import * as _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import Request from './request';
+import log from './utils_log';
+import Request, { RequestOptions } from './request'; // eslint-disable-line import/named,no-unused-vars
/**
* Parses PURL into Regex string.
@@ -101,7 +101,7 @@ const parsePurl = (purl) => {
*/
class PseudoUrl {
/**
- * @param {String|RegExp} purl
+ * @param {(string|RegExp)} purl
* A pseudo-URL string or a regular expression object.
* Using a `RegExp` instance enables more granular control,
* such as making the matching case sensitive.
@@ -127,8 +127,8 @@ class PseudoUrl {
/**
* Determines whether a URL matches this pseudo-URL pattern.
*
- * @param {String} url URL to be matched.
- * @return {Boolean} Returns `true` if given URL matches pseudo-URL.
+ * @param {string} url URL to be matched.
+ * @return {boolean} Returns `true` if given URL matches pseudo-URL.
*/
matches(url) {
return _.isString(url) && url.match(this.regex) !== null;
@@ -141,7 +141,7 @@ class PseudoUrl {
* be merged together, with the `userData` property having preference over the template.
* This enables dynamic overriding of the template.
*
- * @param {string|Object} urlOrProps
+ * @param {(string|Object)} urlOrProps
* @return {Request}
*/
createRequest(urlOrProps) {
diff --git a/src/puppeteer.js b/src/puppeteer.js
index e68f5c9fb..4b44feb69 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -1,10 +1,10 @@
-import _ from 'underscore';
+import * as _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { anonymizeProxy, closeAnonymizedProxy, redactUrl, parseUrl } from 'proxy-chain';
-import log from 'apify-shared/log';
import { ENV_VARS } from 'apify-shared/consts';
-import { LaunchOptions } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import { Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
import { DEFAULT_USER_AGENT } from './constants';
+import log from './utils_log';
import { getTypicalChromeExecutablePath, isAtHome } from './utils';
import { getApifyProxyUrl } from './actor';
import applyStealthToBrowser, { StealthOptions } from './stealth/stealth'; // eslint-disable-line no-unused-vars,import/named
@@ -80,7 +80,7 @@ const launchPuppeteerWithProxy = async (puppeteer, opts) => {
/**
* Requires `puppeteer` package, uses a replacement or throws meaningful error if not installed.
*
- * @param {string|Object} puppeteerModule
+ * @param {(string|Object)} puppeteerModule
* @ignore
*/
const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
@@ -110,38 +110,38 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
* options in the {@link Apify#launchPuppeteer}
* function and in addition, all the options available below.
*
- * @typedef {Object} LaunchPuppeteerOptions
- * @property {String} [proxyUrl]
+ * @typedef LaunchPuppeteerOptions
+ * @property {string} [proxyUrl]
* URL to a HTTP proxy server. It must define the port number,
* and it may also contain proxy username and password.
*
* Example: `http://bob:[email protected]:1234`.
- * @property {String} [userAgent]
+ * @property {string} [userAgent]
* The `User-Agent` HTTP header used by the browser.
* If not provided, the function sets `User-Agent` to a reasonable default
* to reduce the chance of detection of the crawler.
- * @property {Boolean} [useChrome=false]
+ * @property {boolean} [useChrome=false]
* If `true` and `executablePath` is not set,
* Puppeteer will launch full Google Chrome browser available on the machine
* rather than the bundled Chromium. The path to Chrome executable
* is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
* or defaults to the typical Google Chrome executable location specific for the operating system.
* By default, this option is `false`.
- * @property {Boolean} [useApifyProxy=false]
+ * @property {boolean} [useApifyProxy=false]
* If set to `true`, Puppeteer will be configured to use
* [Apify Proxy](https://my.apify.com/proxy) for all connections.
* For more information, see the [documentation](https://docs.apify.com/proxy)
- * @property {String[]} [apifyProxyGroups]
+ * @property {string[]} [apifyProxyGroups]
* An array of proxy groups to be used
* by the [Apify Proxy](https://docs.apify.com/proxy).
* Only applied if the `useApifyProxy` option is `true`.
- * @property {String} [apifyProxySession]
+ * @property {string} [apifyProxySession]
* Apify Proxy session identifier to be used by all the Chrome browsers.
* All HTTP requests going through the proxy with the same session identifier
* will use the same target proxy server (i.e. the same IP address).
* The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
* Only applied if the `useApifyProxy` option is `true`.
- * @property {string|Object} [puppeteerModule]
+ * @property {(string|Object)} [puppeteerModule]
* Either a require path (`string`) to a package to be used instead of default `puppeteer`,
* or an already required module (`Object`). This enables usage of various Puppeteer
* wrappers such as `puppeteer-extra`.
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index 921123024..b8d40ebbc 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -1,19 +1,24 @@
-import _ from 'underscore';
-import fs from 'fs';
-import os from 'os';
-import path from 'path';
-import util from 'util';
-import log from 'apify-shared/log';
-import LinkedList from 'apify-shared/linked_list';
-import rimraf from 'rimraf';
+import * as _ from 'underscore';
+import * as fs from 'fs';
+import * as os from 'os';
+import * as path from 'path';
+import * as util from 'util';
+import * as LinkedList from 'apify-shared/linked_list';
+import * as rimraf from 'rimraf';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import { Page, Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import log from './utils_log';
-import { launchPuppeteer, LaunchPuppeteerOptions } from './puppeteer'; // eslint-disable-line no-unused-vars,import/named
import { addTimeoutToPromise } from './utils';
import LiveViewServer from './live_view/live_view_server';
import EVENTS from './session_pool/events';
+// TYPE IMPORTS
+/* eslint-disable no-unused-vars,import/named,import/order */
+import { Page, Browser } from 'puppeteer';
+import { launchPuppeteer, LaunchPuppeteerOptions } from './puppeteer';
+import { SessionPool } from './session_pool/session_pool';
+/* eslint-enable no-unused-vars */
+
export const BROWSER_SESSION_KEY_NAME = 'APIFY_SESSION';
const PROCESS_KILL_TIMEOUT_MILLIS = 5000;
@@ -43,7 +48,7 @@ const DISK_CACHE_DIR = path.join(os.tmpdir(), 'puppeteer_disk_cache-');
/**
* Deletes Chrome's user data directory
- * @param {String} diskCacheDir
+ * @param {string} diskCacheDir
* @ignore
*/
const deleteDiskCacheDir = (diskCacheDir) => {
@@ -73,7 +78,13 @@ class PuppeteerInstance {
}
/**
- * @typedef {Object} PuppeteerPoolOptions
+ * @callback LaunchPuppeteerFunction
+ * @param {LaunchPuppeteerOptions} options
+ * @returns {Promise<Browser>}
+ */
+
+/**
+ * @typedef PuppeteerPoolOptions
* @property {boolean} [useLiveView]
* Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
* just before a page would be recycled by `PuppeteerPool`. If there are no clients
@@ -95,7 +106,7 @@ class PuppeteerInstance {
* it is considered retired and no more tabs will be opened. After the last tab is closed the
* whole browser is closed too. This parameter defines a time limit between the last tab was opened and
* before the browser is closed even if there are pending open tabs.
- * @property {Function} [launchPuppeteerFunction]
+ * @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* The function must return a promise resolving to
* [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.
@@ -215,13 +226,17 @@ class PuppeteerPool {
checkParamOrThrow(sessionPool, 'options.sessionPool', 'Maybe Object');
// Config.
-
this.sessionPool = sessionPool;
this.reusePages = reusePages;
this.maxOpenPagesPerInstance = maxOpenPagesPerInstance;
this.retireInstanceAfterRequestCount = retireInstanceAfterRequestCount;
this.puppeteerOperationTimeoutMillis = puppeteerOperationTimeoutSecs * 1000;
this.killInstanceAfterMillis = killInstanceAfterMillis || killInstanceAfterSecs * 1000;
+ // this is needed to avoid TS typings trying to link to the .d.ts
+ /**
+ * @type {*}
+ * @ignore
+ */
this.recycledDiskCacheDirs = recycleDiskCache ? new LinkedList() : null;
this.useIncognitoPages = useIncognitoPages;
this.proxyUrls = proxyUrls ? _.shuffle(proxyUrls) : null;
@@ -631,7 +646,7 @@ class PuppeteerPool {
/**
* Finds a PuppeteerInstance given a Puppeteer Browser running in the instance.
* @param {Browser} browser
- * @return {Promise}
+ * @return {Promise<*>}
* @ignore
*/
async _findInstanceByBrowser(browser) {
@@ -699,7 +714,7 @@ class PuppeteerPool {
/**
* Tells the connected LiveViewServer to serve a snapshot when available.
*
- * @param page
+ * @param {Page} page
* @return {Promise<void>}
*/
async serveLiveViewSnapshot(page) {
diff --git a/src/puppeteer_request_interception.js b/src/puppeteer_request_interception.js
index 2c17ea194..75311d609 100644
--- a/src/puppeteer_request_interception.js
+++ b/src/puppeteer_request_interception.js
@@ -1,15 +1,21 @@
-import _ from 'underscore';
+import * as _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
+import { Request as PuppeteerRequest, Page } from 'puppeteer'; // eslint-disable-line no-unused-vars
// We use weak maps here so that the content gets discarted after page gets closed.
const pageInterceptRequestHandlersMap = new WeakMap(); // Maps page to an array of request interception handlers.
const pageInterceptRequestMasterHandlerMap = new WeakMap(); // Maps page to master request interception handler.
+/**
+ * @callback InterceptHandler
+ * @param {PuppeteerRequest} request
+ */
+
/**
* Executes an array for given intercept request handlers for a given request object.
*
- * @param {Request} request Puppeteer's Request object.
- * @param {Array} interceptRequestHandlers An array of intercept request handlers.
+ * @param {PuppeteerRequest} request Puppeteer's Request object.
+ * @param {Array<InterceptHandler>} interceptRequestHandlers An array of intercept request handlers.
* @ignore
*/
const handleRequest = (request, interceptRequestHandlers) => {
@@ -108,8 +114,8 @@ const handleRequest = (request, interceptRequestHandlers) => {
*
* @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- * @param {Function} handler Request interception handler.
- * @return {Promise}
+ * @param {InterceptHandler} handler Request interception handler.
+ * @return {Promise<void>}
* @memberOf puppeteer
* @name addInterceptRequestHandler
*/
@@ -143,8 +149,8 @@ export const addInterceptRequestHandler = async (page, handler) => {
*
* @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- * @param {Function} handler Request interception handler.
- * @return {Promise}
+ * @param {InterceptHandler} handler Request interception handler.
+ * @return {Promise<void>}
* @memberOf puppeteer
* @name removeInterceptRequestHandler
*/
diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.js
index 72bb85346..f25d62be1 100644
--- a/src/puppeteer_utils.js
+++ b/src/puppeteer_utils.js
@@ -1,12 +1,12 @@
-import fs from 'fs';
-import vm from 'vm';
-import util from 'util';
-import _ from 'underscore';
-import log from 'apify-shared/log';
+import * as fs from 'fs';
+import * as vm from 'vm';
+import * as util from 'util';
+import * as _ from 'underscore';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { checkParamPrototypeOrThrow } from 'apify-shared/utilities';
-import LruCache from 'apify-shared/lru_cache';
-import { Page, Response } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import * as LruCache from 'apify-shared/lru_cache';
+import { Page, Response, DirectNavigationOptions } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import log from './utils_log';
import { RequestQueue, RequestQueueLocal } from './request_queue';
import Request from './request';
@@ -28,7 +28,7 @@ const DEFAULT_BLOCK_REQUEST_URL_PATTERNS = ['.css', '.jpg', '.jpeg', '.png', '.s
*
* @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- * @return {Promise}
+ * @return {Promise<void>}
* @memberOf puppeteer
* @ignore
*/
@@ -79,13 +79,13 @@ const injectedFilesCache = new LruCache({ maxLength: MAX_INJECT_FILE_CACHE_SIZE
*
* @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- * @param {String} filePath File path
+ * @param {string} filePath File path
* @param {Object} [options]
* @param {boolean} [options.surviveNavigations]
* Enables the injected script to survive page navigations and reloads without need to be re-injected manually.
* This does not mean, however, that internal state will be preserved. Just that it will be automatically
* re-injected on each navigation before any other scripts get the chance to execute.
- * @return {Promise}
+ * @return {Promise<*>}
* @memberOf puppeteer
*/
const injectFile = async (page, filePath, options = {}) => {
@@ -129,7 +129,7 @@ const injectFile = async (page, filePath, options = {}) => {
*
* @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
- * @return {Promise}
+ * @return {Promise<*>}
* @memberOf puppeteer
*/
const injectJQuery = (page) => {
@@ -155,7 +155,7 @@ const injectJQuery = (page) => {
* ```
*
* @param {Page} page Puppeteer [Page](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page) object.
- * @return {Promise}
+ * @return {Promise<*>}
* @memberOf puppeteer
*/
const injectUnderscore = (page) => {
@@ -239,7 +239,7 @@ const enqueueRequestsFromClickableElements = async (page, selector, purls, reque
* `.png` is the same as `*.png*`.
* @param {string[]} [options.extraUrlPatterns]
* If you just want to append to the default blocked patterns, use this property.
- * @return {Promise}
+ * @return {Promise<void>}
* @memberOf puppeteer
*/
const blockRequests = async (page, options = {}) => {
@@ -285,10 +285,10 @@ const blockResources = async (page, resourceTypes = ['stylesheet', 'font', 'imag
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* @param {Object} cache
* Object in which responses are stored
- * @param {Array<String|RegExp>} responseUrlRules
+ * @param {Array<(string|RegExp)>} responseUrlRules
* List of rules that are used to check if the response should be cached.
* String rules are compared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).
- * @return {Promise}
+ * @return {Promise<void>}
* @memberOf puppeteer
* @deprecated
*/
@@ -366,7 +366,7 @@ const cacheResponses = async (page, cache, responseUrlRules) => {
* make sure to only pass the really necessary objects to the context. Preferably making
* secured copies beforehand.
*
- * @param {String} scriptString
+ * @param {string} scriptString
* @param {Object} context
* @return {Function} `async ({ page, request }) => { scriptString }`
* @memberOf puppeteer
@@ -398,8 +398,8 @@ const compileScript = (scriptString, context = Object.create(null)) => {
* @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* @param {Request} request
- * @param {Object} gotoOptions Custom options for `page.goto()`.
- * @return {Promise<Response>}
+ * @param {DirectNavigationOptions} gotoOptions Custom options for `page.goto()`.
+ * @return {Promise<(Response | null)>}
*
* @memberOf puppeteer
* @name gotoExtended
@@ -440,14 +440,14 @@ export const gotoExtended = async (page, request, gotoOptions = {}) => {
/**
* Scrolls to the bottom of a page, or until it times out.
* Loads dynamic content when it hits the bottom of a page, and then continues scrolling.
- * @param {Object} page
+ * @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* @param {Object} [options]
- * @param {Number} [options.timeoutSecs=0]
+ * @param {number} [options.timeoutSecs=0]
* How many seconds to scroll for. If 0, will scroll until bottom of page.
- * @param {Number} [options.waitForSecs=4]
+ * @param {number} [options.waitForSecs=4]
* How many seconds to wait for no new content to load before exit.
- * @returns {Promise}
+ * @returns {Promise<void>}
* @memberOf puppeteer
* @name infiniteScroll
*/
@@ -509,20 +509,20 @@ export const infiniteScroll = async (page, options = {}) => {
/**
* Saves a full screenshot and HTML of the current page into a Key-Value store.
- * @param {Object} page
+ * @param {Page} page
* Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* @param {Object} [options]
- * @param {String} [options.key=SNAPSHOT]
+ * @param {string} [options.key=SNAPSHOT]
* Key under which the screenshot and HTML will be saved. `.jpg` will be appended for screenshot and `.html` for HTML.
- * @param {Number} [options.screenshotQuality=50]
+ * @param {number} [options.screenshotQuality=50]
* The quality of the image, between 0-100. Higher quality images have bigger size and require more storage.
- * @param {Boolean} [options.saveScreenshot=true]
+ * @param {boolean} [options.saveScreenshot=true]
* If true, it will save a full screenshot of the current page as a record with `key` appended by `.jpg`.
- * @param {Boolean} [options.saveHtml=true]
+ * @param {boolean} [options.saveHtml=true]
* If true, it will save a full HTML of the current page as a record with `key` appended by `.html`.
- * @param {String} [options.keyValueStoreName=null]
+ * @param {string} [options.keyValueStoreName=null]
* Name or id of the Key-Value store where snapshot is saved. By default it is saved to default Key-Value store.
- * @returns {Promise}
+ * @returns {Promise<void>}
* @memberOf puppeteer
* @name saveSnapshot
*/
diff --git a/src/request.js b/src/request.js
index 6fa9be832..9ca54dadc 100644
--- a/src/request.js
+++ b/src/request.js
@@ -1,9 +1,9 @@
-import _ from 'underscore';
-import util from 'util';
-import crypto from 'crypto';
+import * as _ from 'underscore';
+import * as util from 'util';
+import * as crypto from 'crypto';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { normalizeUrl } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
+import log from './utils_log';
export function computeUniqueKey({ url, method, payload, keepUrlFragment, useExtendedUniqueKey }) {
const normalizedMethod = method.toUpperCase();
@@ -30,51 +30,6 @@ export function hashPayload(payload) {
.substr(0, 8);
}
-/**
- * Specifies required and optional fields for constructing a {@link Request#setValue}.
- *
- * @typedef {Object} RequestOptions
- * @property {String} url URL of the web page to crawl. It must be a non-empty string.
- * @property {String} [uniqueKey] A unique key identifying the request.
- * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
- *
- * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
- * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
- * of `http://www.example.com/something`.
- *
- * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
- *
- * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
- * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
- * when requests point to the same URL, but with different methods and payloads. For example: form submits.
- *
- * Pass an arbitrary non-empty text value to the `uniqueKey` property
- * to override the default behavior and specify which URLs shall be considered equal.
- * @property {String} [method='GET']
- * @property {String|Buffer} [payload]
- * HTTP request payload, e.g. for POST requests.
- * @property {Object} [headers={}]
- * HTTP headers in the following format:
- * ```
- * {
- * Accept: 'text/html',
- * 'Content-Type': 'application/json'
- * }
- * ```
- * @property {Object} [userData={}]
- * Custom user data assigned to the request. Use this to save any request related data to the
- * request's scope, keeping them accessible on retries, failures etc.
- * @property {Boolean} [keepUrlFragment=false]
- * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
- * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
- * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
- * Note that this option only has an effect if `uniqueKey` is not set.
- * @property {Boolean} [useExtendedUniqueKey=false]
- * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
- * properties. This is useful when making requests to the same URL that are differentiated by method
- * or payload, such as form submit navigations in browsers.
- */
-
/**
* Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
* The `Request` object also stores information about errors that occurred during processing of the request.
@@ -101,41 +56,44 @@ export function hashPayload(payload) {
*
* const foo = request.userData.foo;
* ```
- * @param {RequestOptions} options `Request` parameters including the URL, HTTP method and headers, and others.
*
- * @property {String} id
+ * @property {string} id
* Request ID
- * @property {String} url
+ * @property {string} url
* URL of the web page to crawl.
- * @property {String} loadedUrl
+ * @property {string} loadedUrl
* An actually loaded URL after redirects, if present. HTTP redirects are guaranteed
* to be included.
*
* When using {@link PuppeteerCrawler}, meta tag and JavaScript redirects may,
* or may not be included, depending on their nature. This generally means that redirects,
* which happen immediately will most likely be included, but delayed redirects will not.
- * @property {String} uniqueKey
+ * @property {string} uniqueKey
* A unique key identifying the request.
* Two requests with the same `uniqueKey` are considered as pointing to the same URL.
- * @property {String} method
+ * @property {string} method
* HTTP method, e.g. `GET` or `POST`.
- * @property {String|Buffer} payload
+ * @property {(string|Buffer)} payload
* HTTP request payload, e.g. for POST requests.
- * @property {Boolean} noRetry
+ * @property {boolean} noRetry
* The `true` value indicates that the request will not be automatically retried on error.
- * @property {Number} retryCount
+ * @property {number} retryCount
* Indicates the number of times the crawling of the request has been retried on error.
- * @property {String[]} errorMessages
+ * @property {string[]} errorMessages
* An array of error messages from request processing.
* @property {Object} headers
* Object with HTTP headers. Key is header name, value is the value.
- * @property {Object} userData
+ * @property {object} userData
* Custom user data assigned to the request.
* @property {Date} handledAt
* Indicates the time when the request has been processed.
* Is `null` if the request has not been crawled yet.
*/
class Request {
+ /**
+ * @param {RequestOptions} options
+ * `Request` parameters including the URL, HTTP method and headers, and others.
+ */
constructor(options = {}) {
checkParamOrThrow(options, 'options', 'Object');
@@ -206,9 +164,9 @@ class Request {
* inspection of the passed argument and attempts to extract as much information
* as possible, since just throwing a bad type error makes any debugging rather difficult.
*
- * @param {Error|String} errorOrMessage Error object or error message to be stored in the request.
+ * @param {(Error|string)} errorOrMessage Error object or error message to be stored in the request.
* @param {Object} [options]
- * @param {Boolean} [options.omitStack=false] Only push the error message without stack trace when true.
+ * @param {boolean} [options.omitStack=false] Only push the error message without stack trace when true.
*/
pushErrorMessage(errorOrMessage, options = {}) {
const { omitStack } = options;
@@ -251,7 +209,7 @@ class Request {
* Optionally accepts a message that will be used to construct
* and throw an Error.
*
- * @param {String} [message]
+ * @param {string} [message]
* @deprecated 2019/06/26
* @ignore
*/
@@ -263,3 +221,48 @@ class Request {
}
export default Request;
+
+/**
+ * Specifies required and optional fields for constructing a [`Request`](../api/request).
+ *
+ * @typedef RequestOptions
+ * @property {string} url URL of the web page to crawl. It must be a non-empty string.
+ * @property {string} [uniqueKey] A unique key identifying the request.
+ * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
+ *
+ * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
+ * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
+ * of `http://www.example.com/something`.
+ *
+ * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
+ *
+ * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
+ * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
+ * when requests point to the same URL, but with different methods and payloads. For example: form submits.
+ *
+ * Pass an arbitrary non-empty text value to the `uniqueKey` property
+ * to override the default behavior and specify which URLs shall be considered equal.
+ * @property {string} [method='GET']
+ * @property {(string|Buffer)} [payload]
+ * HTTP request payload, e.g. for POST requests.
+ * @property {Object} [headers={}]
+ * HTTP headers in the following format:
+ * ```
+ * {
+ * Accept: 'text/html',
+ * 'Content-Type': 'application/json'
+ * }
+ * ```
+ * @property {object} [userData={}]
+ * Custom user data assigned to the request. Use this to save any request related data to the
+ * request's scope, keeping them accessible on retries, failures etc.
+ * @property {boolean} [keepUrlFragment=false]
+ * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
+ * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
+ * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
+ * Note that this option only has an effect if `uniqueKey` is not set.
+ * @property {boolean} [useExtendedUniqueKey=false]
+ * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
+ * properties. This is useful when making requests to the same URL that are differentiated by method
+ * or payload, such as form submit navigations in browsers.
+ */
diff --git a/src/request_list.js b/src/request_list.js
index 98b34bee7..60c50a86e 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -1,8 +1,8 @@
import { checkParamOrThrow } from 'apify-client/build/utils';
-import log from 'apify-shared/log';
-import _ from 'underscore';
+import * as _ from 'underscore';
import { ACTOR_EVENT_NAMES_EX } from './constants';
-import Request from './request';
+import Request, { RequestOptions } from './request'; // eslint-disable-line no-unused-vars,import/named
+import log from './utils_log';
import events from './events';
import { getFirstKey, publicUtils } from './utils';
import { getValue, setValue } from './key_value_store';
@@ -11,8 +11,19 @@ export const STATE_PERSISTENCE_KEY = 'REQUEST_LIST_STATE';
export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
/**
- * @typedef {Object} RequestListOptions
- * @property {Array<RequestOptions|Request>} sources
+ * @typedef RequestListInput
+ * @property {string} [method]
+ * @property {string} [requestsFromUrl]
+ * @property {RegExp} [regex]
+ */
+
+/**
+ * @typedef {Array<(RequestListInput|RequestOptions|Request)>} SourceInput
+ */
+
+/**
+ * @typedef RequestListOptions
+ * @property {SourceInput} sources
* An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
* define the `url` property, or an array of instances of the {@link Request} class.
* Additionally, the `requestsFromUrl` property may be used instead of `url`,
@@ -36,7 +47,7 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
* { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }
* ]
* ```
- * @property {String} [persistStateKey]
+ * @property {string} [persistStateKey]
* Identifies the key in the default key-value store under which `RequestList` periodically stores its
* state (i.e. which URLs were crawled and which not).
* If the actor is restarted, `RequestList` will read the state
@@ -44,7 +55,7 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
*
* If `persistStateKey` is not set, `RequestList` will always start from the beginning,
* and all the source URLs will be crawled again.
- * @property {String} [persistSourcesKey]
+ * @property {string} [persistSourcesKey]
* Identifies the key in the default key-value store under which the `RequestList` persists its
* sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.
* This is necessary if `persistStateKey` is set and the source URLs might potentially change,
@@ -70,7 +81,7 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
*
* Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
* is to use the `stateKeyPrefix` parameter instead.
- * @property {Boolean} [keepDuplicateUrls=false]
+ * @property {boolean} [keepDuplicateUrls=false]
* By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
* on the `uniqueKey` property of passed source {@link Request} objects.
*
@@ -341,7 +352,7 @@ export class RequestList {
* Attempts to load state and sources using the `RequestList` configuration
* and returns a tuple of [state, sources] where each may be null if not loaded.
*
- * @return {Promise<Array>}
+ * @return {Promise<Array<(RequestListState|null)>>}
* @ignore
*/
async _loadStateAndSources() {
@@ -410,7 +421,7 @@ export class RequestList {
* The function's `Promise` resolves to `null` if there are no more
* requests to process.
*
- * @returns {Promise<Request>}
+ * @returns {Promise<(Request|null)>}
*/
async fetchNextRequest() {
this._ensureIsInitialized();
@@ -439,7 +450,6 @@ export class RequestList {
* Marks request as handled after successful processing.
*
* @param {Request} request
- *
* @returns {Promise<void>}
*/
async markRequestHandled(request) {
@@ -458,7 +468,6 @@ export class RequestList {
* The request will become available in the next `this.fetchNextRequest()`.
*
* @param {Request} request
- *
* @returns {Promise<void>}
*/
async reclaimRequest(request) {
@@ -498,7 +507,7 @@ export class RequestList {
/**
* Fetches URLs from requestsFromUrl and returns them in format of list of requests
* @param source
- * @return {Promise<RequestOptions[]>}
+ * @return {Promise<Array<RequestOptions>>}
* @ignore
*/
async _fetchRequestsFromUrl(source) {
@@ -594,7 +603,7 @@ export class RequestList {
/**
* Returns the total number of unique requests present in the `RequestList`.
*
- * @returns {Number}
+ * @returns {number}
*/
length() {
this._ensureIsInitialized();
@@ -605,7 +614,7 @@ export class RequestList {
/**
* Returns number of handled requests.
*
- * @returns {Number}
+ * @returns {number}
*/
handledCount() {
this._ensureIsInitialized();
@@ -648,7 +657,7 @@ export class RequestList {
*
* If `null`, the list will not be persisted and will only be stored in memory. Process restart
* will then cause the list to be crawled again from the beginning. We suggest always using a name.
- * @param {Array<Request|RequestOptions|string>} sources
+ * @param {(SourceInput|string[])} sources
* An array of sources of URLs for the {@link RequestList}. It can be either an array of plain objects
* that define at least the `url` property, or an array of instances of the {@link Request} class.
*
@@ -702,11 +711,11 @@ export const openRequestList = async (listName, sources, options = {}) => {
* }
* ```
*
- * @typedef {Object} RequestListState
- * @property {Number} nextIndex
+ * @typedef RequestListState
+ * @property {number} nextIndex
* Position of the next request to be processed.
- * @property {String} nextUniqueKey
+ * @property {string} nextUniqueKey
* Key of the next request to be processed.
- * @property {Object<String,Boolean>} inProgress
+ * @property {Object<string,boolean>} inProgress
* An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.
*/
diff --git a/src/request_queue.js b/src/request_queue.js
index d50f2b022..b6d833184 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -1,12 +1,12 @@
-import path from 'path';
-import crypto from 'crypto';
-import fs from 'fs-extra';
+import * as path from 'path';
+import * as crypto from 'crypto';
+import * as fs from 'fs-extra';
import { checkParamOrThrow } from 'apify-client/build/utils';
-import LruCache from 'apify-shared/lru_cache';
-import ListDictionary from 'apify-shared/list_dictionary';
+import * as LruCache from 'apify-shared/lru_cache';
+import * as ListDictionary from 'apify-shared/list_dictionary';
import { ENV_VARS, LOCAL_STORAGE_SUBDIRS, REQUEST_QUEUE_HEAD_MAX_LIMIT } from 'apify-shared/consts';
import { checkParamPrototypeOrThrow, cryptoRandomObjectId } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
+import log from './utils_log';
import Request, { RequestOptions } from './request'; // eslint-disable-line import/named,no-unused-vars
import { ensureDirExists, apifyClient, openRemoteStorage, openLocalStorage, ensureTokenOrLocalStorageEnvExists, sleep } from './utils';
@@ -114,11 +114,11 @@ const getRequestId = (uniqueKey) => {
* {@link RequestQueue} functions as well as
* {@link utils#enqueueLinks}.
*
- * @typedef {Object} QueueOperationInfo
- * @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.
- * @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.
- * @property {String} requestId The ID of the added request
- * @property {Object} request The original {@link Request} object passed to the `RequestQueue` function.
+ * @typedef QueueOperationInfo
+ * @property {boolean} wasAlreadyPresent Indicates if request was already present in the queue.
+ * @property {boolean} wasAlreadyHandled Indicates if request was already marked as handled.
+ * @property {string} requestId The ID of the added request
+ * @property {Request} request The original {@link Request} object passed to the `RequestQueue` function.
*/
/**
@@ -187,6 +187,11 @@ const getRequestId = (uniqueKey) => {
* @hideconstructor
*/
export class RequestQueue {
+ /**
+ * @param {string} queueId
+ * @param {string} [queueName]
+ * @param {string} [clientKey]
+ */
constructor(queueId, queueName, clientKey = cryptoRandomObjectId()) {
checkParamOrThrow(queueId, 'queueId', 'String');
checkParamOrThrow(queueName, 'queueName', 'Maybe String');
@@ -200,6 +205,11 @@ export class RequestQueue {
// Contains a cached list of request IDs from the head of the queue,
// as obtained in the last query. Both key and value is the request ID.
+ // Need to apply a type here to the generated TS types don't try to use types-apify
+ /**
+ * @type {*}
+ * @ignore
+ */
this.queueHeadDict = new ListDictionary();
this.queryQueueHeadPromise = null;
@@ -241,10 +251,10 @@ export class RequestQueue {
* To add multiple requests to the queue by extracting links from a webpage,
* see the {@link utils#enqueueLinks} helper function.
*
- * @param {Request|RequestOptions} request {@link Request} object or vanilla object with request data.
+ * @param {(Request|RequestOptions)} request {@link Request} object or vanilla object with request data.
* Note that the function sets the `uniqueKey` and `id` fields to the passed object.
* @param {Object} [options]
- * @param {Boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.
+ * @param {boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.
* @return {Promise<QueueOperationInfo>}
*/
async addRequest(request, options = {}) {
@@ -294,8 +304,8 @@ export class RequestQueue {
/**
* Gets the request from the queue specified by ID.
*
- * @param {String} requestId ID of the request.
- * @return {Promise<Request>} Returns the request object, or `null` if it was not found.
+ * @param {string} requestId ID of the request.
+ * @return {Promise<(Request | null)>} Returns the request object, or `null` if it was not found.
*/
async getRequest(requestId) {
validateGetRequestParams(requestId);
@@ -324,7 +334,7 @@ export class RequestQueue {
* To check whether all requests in queue were finished,
* use {@link RequestQueue#isFinished} instead.
*
- * @returns {Promise<Request>}
+ * @returns {Promise<(Request|null)>}
* Returns the request object or `null` if there are no more pending requests.
*/
async fetchNextRequest() {
@@ -431,7 +441,7 @@ export class RequestQueue {
*
* @param {Request} request
* @param {Object} [options]
- * @param {Boolean} [options.forefront=false]
+ * @param {boolean} [options.forefront=false]
* If `true` then the request it placed to the beginning of the queue, so that it's returned
* in the next call to {@link RequestQueue#fetchNextRequest}.
* By default, it's put to the end of the queue.
@@ -481,7 +491,7 @@ export class RequestQueue {
* Note that even if the queue is empty, there might be some pending requests currently being processed.
* If you need to ensure that there is no activity in the queue, use {@link RequestQueue#isFinished}.
*
- * @returns {Promise<Boolean>}
+ * @returns {Promise<boolean>}
*/
async isEmpty() {
await this._ensureHeadIsNonEmpty();
@@ -494,7 +504,7 @@ export class RequestQueue {
* the function might occasionally return a false negative,
* but it will never return a false positive.
*
- * @returns {Promise<Boolean>}
+ * @returns {Promise<boolean>}
*/
async isFinished() {
if (this.queueHeadDict.length() > 0 || this.inProgressCount() > 0) return false;
@@ -523,12 +533,12 @@ export class RequestQueue {
/**
* We always request more items than is in progress to ensure that something falls into head.
*
- * @param {Boolean} [ensureConsistency=false] If true then query for queue head is retried until queueModifiedAt
+ * @param {boolean} [ensureConsistency=false] If true then query for queue head is retried until queueModifiedAt
* is older than queryStartedAt by at least API_PROCESSED_REQUESTS_DELAY_MILLIS to ensure that queue
* head is consistent.
- * @param {Number} [limit] How many queue head items will be fetched.
- * @param {Number} [iteration] Used when this function is called recursively to limit the recursion.
- * @return {Boolean} Indicates if queue head is consistent (true) or inconsistent (false).
+ * @param {number} [limit] How many queue head items will be fetched.
+ * @param {number} [iteration] Used when this function is called recursively to limit the recursion.
+ * @return {Promise<boolean>} Indicates if queue head is consistent (true) or inconsistent (false).
* @ignore
*/
async _ensureHeadIsNonEmpty(
@@ -637,7 +647,7 @@ export class RequestQueue {
* Removes the queue either from the Apify Cloud storage or from the local directory,
* depending on the mode of operation.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async drop() {
await requestQueues.deleteQueue({
@@ -696,7 +706,7 @@ export class RequestQueue {
* }
* ```
*
- * @returns {Promise<Object>}
+ * @returns {Promise<object>}
*/
async getInfo() {
return requestQueues.getQueue({ queueId: this.queueId });
diff --git a/src/session_pool/errors.js b/src/session_pool/errors.js
index ebee49a4a..5fcefc3a1 100644
--- a/src/session_pool/errors.js
+++ b/src/session_pool/errors.js
@@ -1,4 +1,7 @@
export class CookieParseError extends Error {
+ /**
+ * @param {string} cookieHeaderString
+ */
constructor(cookieHeaderString) {
super(`Could not parse cookie header string: ${cookieHeaderString}`);
this.cookieHeaderString = cookieHeaderString;
diff --git a/src/session_pool/session.js b/src/session_pool/session.js
index 0f6e12ce5..632594643 100644
--- a/src/session_pool/session.js
+++ b/src/session_pool/session.js
@@ -1,8 +1,8 @@
import { cryptoRandomObjectId } from 'apify-shared/utilities';
-import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { Cookie, CookieJar } from 'tough-cookie';
-import EVENTS from './events'; // eslint-disable-line import/named,no-unused-vars
+import log from '../utils_log';
+import EVENTS from './events';
import { STATUS_CODES_BLOCKED } from '../constants';
import { getCookiesFromResponse } from './session_utils';
@@ -10,14 +10,16 @@ import { getCookiesFromResponse } from './session_utils';
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order,import/no-cycle */
import { SessionPool } from './session_pool';
import { Cookie as PuppeteerCookie } from 'puppeteer';
+import { IncomingMessage } from 'http';
+import { Response as PuppeteerResponse } from 'puppeteer';
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order,import/no-cycle */
/**
* Persistable {@link Session} state.
- * @typedef {Object} SessionState
+ * @typedef SessionState
* @property {string} id
- * @property {Object} cookieJar
- * @property {Object} userData
+ * @property {CookieJar} cookieJar
+ * @property {object} userData
* @property {number} errorScore
* @property {number} maxErrorScore
* @property {number} errorScoreDecrement
@@ -27,10 +29,10 @@ import { Cookie as PuppeteerCookie } from 'puppeteer';
*/
/**
- * @typedef {Object} SessionOptions
+ * @typedef SessionOptions
* @property {string} [id] - Id of session used for generating fingerprints. It is used as proxy session name.
* @property {number} [maxAgeSecs=3000] - Number of seconds after which the session is considered as expired.
- * @property {Object} userData - Object where custom user data can be stored. For example custom headers.
+ * @property {object} [userData] - Object where custom user data can be stored. For example custom headers.
* @property {number} [maxErrorScore=3] - Maximum number of marking session as blocked usage.
* If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
* It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
@@ -46,14 +48,17 @@ import { Cookie as PuppeteerCookie } from 'puppeteer';
* This number indicates how many times the session is going to be used, before it is thrown away.
* @property {SessionPool} sessionPool - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
*/
+
/**
* Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions.
* You can imagine each session as a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint.
* Session internal state can be enriched with custom user data for example some authorization tokens and specific headers in general.
+ *
*/
export class Session {
/**
* Session configuration.
+ *
* @param {SessionOptions} options
*/
constructor(options = {}) {
@@ -90,11 +95,10 @@ export class Session {
throw new Error('Session: sessionPool must be instance of SessionPool');
}
- // Configurable
/**
- * @type CookieJar
+ * @type {CookieJar}
* @private
- * */
+ */
this.cookieJar = cookieJar.setCookie ? cookieJar : CookieJar.fromJSON(JSON.stringify(cookieJar));
this.id = id;
this.maxAgeSecs = maxAgeSecs;
@@ -167,7 +171,6 @@ export class Session {
getState() {
return {
id: this.id,
- cookies: this.cookies,
cookieJar: this.cookieJar.toJSON(),
userData: this.userData,
maxErrorScore: this.maxErrorScore,
@@ -227,7 +230,8 @@ export class Session {
* (typical Node.js responses) or a `Function` (Puppeteer Response).
*
* It then parses and saves the cookies from the `set-cookie` header, if available.
- * @param {{ headers }} response
+
+ * @param {(PuppeteerResponse|IncomingMessage)} response
*/
setCookiesFromResponse(response) {
try {
@@ -253,8 +257,8 @@ export class Session {
* ]
* ```
*
- * @param cookies {PuppeteerCookie[]}
- * @param url {string}
+ * @param {PuppeteerCookie[]} cookies
+ * @param {string} url
*/
setPuppeteerCookies(cookies, url) {
try {
@@ -267,7 +271,7 @@ export class Session {
/**
* Returns cookies in a format compatible with puppeteer and ready to be used with `page.setCookie`.
- * @param url {String} - website url. Only cookies stored for this url will be returned
+ * @param {string} url website url. Only cookies stored for this url will be returned
* @return {PuppeteerCookie[]}
*/
getPuppeteerCookies(url) {
@@ -287,10 +291,9 @@ export class Session {
return this.cookieJar.getCookieStringSync(url, {});
}
-
/**
* Transforms puppeteer cookie to tough-cookie.
- * @param puppeteerCookie {PuppeteerCookie} - Cookie from puppeteer `page.cookies method.
+ * @param {PuppeteerCookie} puppeteerCookie Cookie from puppeteer `page.cookies method.
* @return {Cookie}
* @private
*/
diff --git a/src/session_pool/session_pool.js b/src/session_pool/session_pool.js
index 6f2deaf47..b72bb453b 100644
--- a/src/session_pool/session_pool.js
+++ b/src/session_pool/session_pool.js
@@ -1,6 +1,6 @@
import { EventEmitter } from 'events';
-import log from 'apify-shared/log';
import { checkParamOrThrow } from 'apify-client/build/utils';
+import log from '../utils_log';
import { openKeyValueStore } from '../key_value_store';
import { Session, SessionOptions } from './session'; // eslint-disable-line no-unused-vars,import/named,import/no-cycle
@@ -11,20 +11,20 @@ import { ACTOR_EVENT_NAMES_EX } from '../constants';
* Factory user-function which creates customized {@link Session} instances.
* @callback CreateSession
* @param {SessionPool} sessionPool Pool requesting the new session.
+ * @returns {Session}
*/
/**
- * @typedef {Object} SessionPoolOptions
- * @property {Number} [maxPoolSize=1000] - Maximum size of the pool.
+ * @typedef SessionPoolOptions
+ * @property {number} [maxPoolSize=1000] - Maximum size of the pool.
* Indicates how many sessions are rotated.
* @property {SessionOptions} [sessionOptions] The configuration options for {Session} instances.
- * @property {String} [persistStateKeyValueStoreId] - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
- * @property {String} [persistStateKey="SESSION_POOL_STATE"] - Session pool persists it's state under this key in Key value store.
+ * @property {string} [persistStateKeyValueStoreId] - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
+ * @property {string} [persistStateKey="SESSION_POOL_STATE"] - Session pool persists it's state under this key in Key value store.
* @property {CreateSession} [createSessionFunction] - Custom function that should return `Session` instance.
* Function receives `SessionPool` instance as a parameter
*/
-// TODO yin: `tsc` generates a class declaration containing EventEmitter methods with wrong return type (`:SessionPool instead of `:this`).
/**
* Handles the sessions rotation, creation and persistence.
* Creates a pool of {@link Session} instances, that are randomly rotated.
@@ -174,8 +174,6 @@ export class SessionPool extends EventEmitter {
/**
* Returns an object representing the internal state of the `SessionPool` instance.
* Note that the object's fields can change in future releases.
- *
- * @returns {Object}
*/
getState() {
return {
@@ -189,7 +187,7 @@ export class SessionPool extends EventEmitter {
* Persists the current state of the `SessionPool` into the default {@link KeyValueStore}.
* The state is persisted automatically in regular intervals.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
async persistState() {
log.debug('SessionPool: Persisting state',
@@ -210,7 +208,7 @@ export class SessionPool extends EventEmitter {
/**
* Removes `Session` instance from `SessionPool`.
- * @param session {Session} - Session to be removed
+ * @param {Session} session - Session to be removed
* @private
*/
_removeSession(session) {
@@ -222,7 +220,7 @@ export class SessionPool extends EventEmitter {
/**
* Adds `Session` instance to `SessionPool`.
- * @param newSession {Session} - `Session` instance to be added.
+ * @param {Session} newSession `Session` instance to be added.
* @private
*/
_addSession(newSession) {
@@ -240,7 +238,7 @@ export class SessionPool extends EventEmitter {
/**
* Creates new session without any extra behavior.
- * @param sessionPool
+ * @param {SessionPool} sessionPool
* @return {Session} - New session.
* @private
*/
diff --git a/src/session_pool/session_utils.js b/src/session_pool/session_utils.js
index 802974868..ec4324f9b 100644
--- a/src/session_pool/session_utils.js
+++ b/src/session_pool/session_utils.js
@@ -1,10 +1,14 @@
+/* eslint-disable no-unused-vars */
+import { IncomingMessage } from 'http';
+import { Response as PuppeteerResponse } from 'puppeteer';
+/* eslint-enable no-unused-vars */
+
import { Cookie } from 'tough-cookie';
import { CookieParseError } from './errors';
/**
- *
- * @param response
- * @return {undefined|Array}
+ * @param {(IncomingMessage|PuppeteerResponse)} response
+ * @return {undefined|Array<*>}
*/
export const getCookiesFromResponse = (response) => {
const headers = typeof response.headers === 'function' ? response.headers() : response.headers;
diff --git a/src/stealth/stealth.js b/src/stealth/stealth.js
index d474bf462..c3519ca26 100644
--- a/src/stealth/stealth.js
+++ b/src/stealth/stealth.js
@@ -1,11 +1,11 @@
-import _ from 'underscore';
-import { Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import * as _ from 'underscore';
+import { Page, Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
import hidingTricks from './hiding_tricks';
/**
* Configuration of stealth tricks for a proper hiding effect all of them should be set to true.
* These tricks are applied only when the `stealth` option is set to `true`.
- * @typedef {Object} StealthOptions
+ * @typedef StealthOptions
* @property {boolean} [addPlugins=true] - If plugins should be added to the navigator.
* @property {boolean} [emulateWindowFrame=true] - Emulates window Iframe.
* @property {boolean} [emulateWebGL=true] - Emulates graphic card.
@@ -37,7 +37,6 @@ const DEFAULT_STEALTH_OPTIONS = {
* @param {StealthOptions} options
* @returns {Promise<Browser>} - Instance of Browser from puppeteer package
*/
-
export default function applyStealthToBrowser(browser, options) {
const modifiedBrowser = browser;
const opts = _.defaults(options, DEFAULT_STEALTH_OPTIONS);
@@ -61,11 +60,10 @@ export default function applyStealthToBrowser(browser, options) {
* Applies stealth tricks to the puppeteer page
* @param {Page} page
* @param {StealthOptions} options
- * @returns {Promise}
+ * @returns {Promise<void>}
* @private
* @ignore
*/
-
function applyStealthTricks(page, options) {
const functions = Object.keys(options)
.filter((key) => {
diff --git a/src/typedefs.js b/src/typedefs.js
index 7f26b3d46..d6ad3a1e3 100644
--- a/src/typedefs.js
+++ b/src/typedefs.js
@@ -1,10 +1,3 @@
-/**
- * `CherioStatic`, the return type of `cheerio.load()`, is not exported from `@types/cheerio`, so it can not be imported and aliased, it is a global
- * type. We reference the type and redefine it with a bettter name here.
- * @typedef {CheerioStatic} Cheerio
- * @ignore
- */
-
/**
* Represents information about an actor run, as returned by the
* {@link Apify#call} or {@link Apify#callTask} function.
@@ -15,20 +8,20 @@
* For more details, see
* [Runs.](https://docs.apify.com/actor/run)
*
- * @typedef {Object} ActorRun
- * @property {String} id
+ * @typedef ActorRun
+ * @property {string} id
* Actor run ID
- * @property {String} actId
+ * @property {string} actId
* Actor ID
* @property {Date} startedAt
* Time when the actor run started
* @property {Date} finishedAt
* Time when the actor run finished. Contains `null` for running actors.
- * @property {String} status
+ * @property {string} status
* Status of the run. For possible values, see
* [Run lifecycle](https://docs.apify.com/actor/run#lifecycle)
* in Apify actor documentation.
- * @property {Object} meta
+ * @property {object} meta
* Actor run meta-data. For example:
* ```
* {
@@ -37,7 +30,7 @@
* "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"
* }
* ```
- * @property {Object} stats
+ * @property {object} stats
* An object containing various actor run statistics. For example:
* ```
* {
@@ -47,7 +40,7 @@
* }
* ```
* Beware that object fields might change in future releases.
- * @property {Object} options
+ * @property {object} options
* Actor run options. For example:
* ```
* {
@@ -57,26 +50,26 @@
* "diskMbytes": 512
* }
* ```
- * @property {String} buildId
+ * @property {string} buildId
* ID of the actor build used for the run. For details, see
* [Builds](https://docs.apify.com/actor/build)
* in Apify actor documentation.
- * @property {String} buildNumber
+ * @property {string} buildNumber
* Number of the actor build used for the run. For example, `0.0.10`.
- * @property {Number} exitCode
+ * @property {number} exitCode
* Exit code of the actor run process. It's `null` if actor is still running.
- * @property {String} defaultKeyValueStoreId
+ * @property {string} defaultKeyValueStoreId
* ID of the default key-value store associated with the actor run. See {@link KeyValueStore} for details.
- * @property {String} defaultDatasetId
+ * @property {string} defaultDatasetId
* ID of the default dataset associated with the actor run. See {@link Dataset} for details.
- * @property {String} defaultRequestQueueId
+ * @property {string} defaultRequestQueueId
* ID of the default request queue associated with the actor run. See {@link RequestQueue} for details.
- * @property {String} containerUrl
+ * @property {string} containerUrl
* URL on which the web server running inside actor run's Docker container can be accessed.
* For more details, see
* [Container web server](https://docs.apify.com/actor/run#container-web-server)
* in Apify actor documentation.
- * @property {Object} output
+ * @property {(object|null|undefined)} output
* Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,
* or if you pass `false` to the `fetchOutput` option of {@link Apify#call}.
*
diff --git a/src/utils.js b/src/utils.js
index 02a1dac87..725dc0fae 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -1,30 +1,30 @@
-import psTree from '@apify/ps-tree';
-import ApifyClient from 'apify-client';
+import * as psTree from '@apify/ps-tree';
+import * as ApifyClient from 'apify-client';
import { checkParamOrThrow } from 'apify-client/build/utils';
import { version as apifyClientVersion } from 'apify-client/package.json';
import { ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';
-import log from 'apify-shared/log';
import { getRandomInt } from 'apify-shared/utilities';
-import cheerio from 'cheerio';
-import contentTypeParser from 'content-type';
-import fs from 'fs';
-import fsExtra from 'fs-extra';
-import mime from 'mime-types';
-import os from 'os';
-import path from 'path';
-import requestPromise from 'request-promise-native';
-import semver from 'semver';
-import _ from 'underscore';
+import * as cheerio from 'cheerio';
+import * as contentTypeParser from 'content-type';
+import * as fs from 'fs';
+import * as fsExtra from 'fs-extra';
+import * as mime from 'mime-types';
+import * as os from 'os';
+import * as path from 'path';
+import * as requestPromise from 'request-promise-native';
+import * as semver from 'semver';
+import * as _ from 'underscore';
import { URL } from 'url';
-import util from 'util';
+import * as util from 'util';
import { USER_AGENT_LIST } from './constants';
+import log from './utils_log';
import { version as apifyVersion } from '../package.json';
// TYPE IMPORTS
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
import { IncomingMessage } from 'http';
import { Response as PuppeteerResponse } from 'puppeteer';
-import { Cheerio } from './typedefs';
+import Request, { RequestOptions } from './request';
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
/**
@@ -111,8 +111,8 @@ export const newPromise = () => {
/**
* Adds charset=utf-8 to given content type if this parameter is missing.
*
- * @param {String} contentType
- * @returns {String}
+ * @param {string} contentType
+ * @returns {string}
*
* @ignore
*/
@@ -165,8 +165,8 @@ export const isDocker = (forceReset) => {
/**
* Sums an array of numbers.
*
- * @param {Number[]} arr An array of numbers.
- * @return {Number} Sum of the numbers.
+ * @param {number[]} arr An array of numbers.
+ * @return {number} Sum of the numbers.
*
* @ignore
*/
@@ -175,8 +175,8 @@ export const sum = arr => arr.reduce((total, c) => total + c, 0);
/**
* Computes an average of an array of numbers.
*
- * @param {Number[]} arr An array of numbers.
- * @return {Number} Average value.
+ * @param {number[]} arr An array of numbers.
+ * @return {number} Average value.
*
* @ignore
*/
@@ -185,9 +185,9 @@ export const avg = arr => sum(arr) / arr.length;
/**
* Computes a weighted average of an array of numbers, complemented by an array of weights.
*
- * @param {Number[]} arrValues
- * @param {Number[]} arrWeights
- * @return {Number}
+ * @param {number[]} arrValues
+ * @param {number[]} arrWeights
+ * @return {number}
*
* @ignore
*/
@@ -205,12 +205,12 @@ export const weightedAvg = (arrValues, arrWeights) => {
/**
* Describes memory usage of an Actor.
*
- * @typedef {Object} MemoryInfo
- * @property {Number} totalBytes Total memory available in the system or container
- * @property {Number} freeBytes Amount of free memory in the system or container
- * @property {Number} usedBytes Amount of memory used (= totalBytes - freeBytes)
- * @property {Number} mainProcessBytes Amount of memory used the current Node.js process
- * @property {Number} childProcessesBytes Amount of memory used by child processes of the current Node.js process
+ * @typedef MemoryInfo
+ * @property {number} totalBytes Total memory available in the system or container
+ * @property {number} freeBytes Amount of free memory in the system or container
+ * @property {number} usedBytes Amount of memory used (= totalBytes - freeBytes)
+ * @property {number} mainProcessBytes Amount of memory used the current Node.js process
+ * @property {number} childProcessesBytes Amount of memory used by child processes of the current Node.js process
*/
/**
@@ -336,11 +336,10 @@ export const getTypicalChromeExecutablePath = () => {
* Wraps the provided Promise with another one that rejects with the given errorMessage
* after the given timeoutMillis, unless the original promise resolves or rejects earlier.
*
- * @template T
- * @param {Promise<T>} promise
+ * @param {Promise<object>} promise
* @param {number} timeoutMillis
* @param {string} errorMessage
- * @return {Promise<T>}
+ * @return {Promise<object>}
* @ignore
*/
export const addTimeoutToPromise = (promise, timeoutMillis, errorMessage) => {
@@ -364,7 +363,7 @@ export const addTimeoutToPromise = (promise, timeoutMillis, errorMessage) => {
/**
* Returns `true` when code is running on Apify platform and `false` otherwise (for example locally).
*
- * @returns {Boolean}
+ * @returns {boolean}
*
* @memberof module:Apify
* @name isAtHome
@@ -386,7 +385,7 @@ export const isAtHome = () => !!process.env[ENV_VARS.IS_AT_HOME];
* // Sleep 1.5 seconds
* await Apify.utils.sleep(1500);
* ```
- * @param {Number} millis Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
+ * @param {number} millis Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
* @memberof utils
* @name sleep
* @function
@@ -401,12 +400,12 @@ export const sleep = (millis) => {
* Optionally, custom regular expression and encoding may be provided.
*
* @param {Object} options
- * @param {String} options.url URL to the file
- * @param {String} [options.encoding='utf8'] The encoding of the file.
+ * @param {string} options.url URL to the file
+ * @param {string} [options.encoding='utf8'] The encoding of the file.
* @param {RegExp} [options.urlRegExp=URL_NO_COMMAS_REGEX]
* Custom regular expression to identify the URLs in the file to extract.
* The regular expression should be case-insensitive and have global flag set (i.e. `/something/gi`).
- * @returns {Promise<String[]>}
+ * @returns {Promise<Array<string>>}
* @memberOf utils
*/
const downloadListOfUrls = async ({ url, encoding = 'utf8', urlRegExp = URL_NO_COMMAS_REGEX }) => {
@@ -421,9 +420,9 @@ const downloadListOfUrls = async ({ url, encoding = 'utf8', urlRegExp = URL_NO_C
/**
* Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.
* @param {Object} options
- * @param {String} options.string
+ * @param {string} options.string
* @param {RegExp} [options.urlRegExp=Apify.utils.URL_NO_COMMAS_REGEX]
- * @returns {String[]}
+ * @returns {string[]}
* @memberOf utils
*/
const extractUrls = ({ string, urlRegExp = URL_NO_COMMAS_REGEX }) => {
@@ -434,7 +433,7 @@ const extractUrls = ({ string, urlRegExp = URL_NO_COMMAS_REGEX }) => {
/**
* Returns a randomly selected User-Agent header out of a list of the most common headers.
- * @returns {String}
+ * @returns {string}
* @memberOf utils
*/
const getRandomUserAgent = () => {
@@ -528,9 +527,9 @@ const BLOCK_TAGS_REGEX = /^(p|h1|h2|h3|h4|h5|h6|ol|ul|li|pre|address|blockquote|
* const html = '<html><body>Some text</body></html>';
* const text = htmlToText(cheerio.load(html, { decodeEntities: true }));
* ```
- * @param {String|Cheerio} html HTML text or parsed HTML represented using a
+ * @param {(string|CheerioStatic)} html HTML text or parsed HTML represented using a
* [cheerio](https://www.npmjs.com/package/cheerio) function.
- * @return {String} Plain text
+ * @return {string} Plain text
* @memberOf utils
* @function
*/
@@ -543,7 +542,7 @@ const htmlToText = (html) => {
// produces really text with a lot of HTML elements in it. Let's just deprecate this sort of usage,
// and make the parameter "htmlOrCheerioElement"
/**
- * @type {Cheerio}
+ * @type {CheerioStatic}
* @ignore
*/
const $ = typeof html === 'function' ? html : cheerio.load(html, { decodeEntities: true });
@@ -588,13 +587,13 @@ const htmlToText = (html) => {
/**
* Creates a standardized debug info from request and response. This info is usually added to dataset under the hidden `#debug` field.
*
- * @param {Request|RequestOptions} request [Apify.Request](https://sdk.apify.com/docs/api/request) object.
- * @param {IncomingMessage|PuppeteerResponse} [response]
+ * @param {(Request|RequestOptions)} request [Apify.Request](https://sdk.apify.com/docs/api/request) object.
+ * @param {(IncomingMessage|PuppeteerResponse)} [response]
* Puppeteer [`Response`](https://pptr.dev/#?product=Puppeteer&version=v1.11.0&show=api-class-response)
* or NodeJS [`http.IncomingMessage`](https://nodejs.org/api/http.html#http_class_http_serverresponse).
* @param {Object} [additionalFields] Object containing additional fields to be added.
- * @return {Object}
+ * @return {object}
*/
const createRequestDebugInfo = (request, response = {}, additionalFields = {}) => {
checkParamOrThrow(request, 'request', 'Object');
@@ -619,8 +618,8 @@ const createRequestDebugInfo = (request, response = {}, additionalFields = {}) =
/**
* Converts SNAKE_CASE to camelCase.
*
- * @param {String} snakeCaseStr
- * @return {String}
+ * @param {string} snakeCaseStr
+ * @return {string}
* @ignore
*/
export const snakeCaseToCamelCase = (snakeCaseStr) => {
@@ -651,7 +650,7 @@ export const printOutdatedSdkWarning = () => {
/**
* Gets parsed content type from response object
- * @param {Object} response - HTTP response object
+ * @param {IncomingMessage} response - HTTP response object
* @return {{ type: string, charset: string }}
* @ignore
*/
diff --git a/src/utils_log.d.ts b/src/utils_log.d.ts
deleted file mode 100644
index e554c1f7c..000000000
--- a/src/utils_log.d.ts
+++ /dev/null
@@ -1,29 +0,0 @@
-declare module log {
- const LEVELS: {
- // Turns off logging completely
- OFF: number,
- // For unexpected errors in Apify system
- ERROR: number,
- WARNING: number,
- INFO: number,
- DEBUG: number,
- // for performance stats
- PERF: number
- };
-
- // Sets log level
- function setLevel(level: number): void;
-
- function getLevel(): number;
-
- // Helper functions for common usage
- function warning(message: string, data?: any): void;
-
- function info(message: string, data?: any): void;
-
- function debug(message: string, data?: any): void;
-
- function error(message: string, data?: any): void;
-
- function exception(exception: Error, message?: string, data?: any): void;
-}
diff --git a/src/utils_log.js b/src/utils_log.js
index 8ee209d95..c9304df6d 100644
--- a/src/utils_log.js
+++ b/src/utils_log.js
@@ -1,7 +1,7 @@
-import log from 'apify-shared/log';
-
-// Add docs for log separately, as it's imported from apify-shared.
-// Adding them directly to the log object in utils in `index.js` breaks JSDoc.
+// Changes to the JsDoc here need to be replicated to types-apify/apify-shared/log.d.ts
+// This file is mainly for building the docs, not to provide typings
+// For the output of "main" typings to be correct, this file should be references
+// instead of 'apify-shared/log'
/**
* @typedef {Object} LoggerOptions
@@ -179,4 +179,7 @@ import log from 'apify-shared/log';
* @memberOf log
*/
-export default log;
+// variable can't be the same from the namespace above
+import * as Log from 'apify-shared/log';
+
+export default Log;
diff --git a/src/utils_request.js b/src/utils_request.js
index bb0a12ec6..796297e9b 100644
--- a/src/utils_request.js
+++ b/src/utils_request.js
@@ -1,7 +1,11 @@
import { URL } from 'url';
-import httpRequest from '@apify/http-request';
-import errors from '@apify/http-request/src/errors';
+import * as httpRequest from '@apify/http-request';
+import * as errors from '@apify/http-request/src/errors';
+/* eslint-disable no-unused-vars,import/named,import/order */
import { TimeoutError } from './errors';
+import { IncomingMessage } from 'http';
+import { Readable } from 'stream';
+/* eslint-enable no-unused-vars,import/named,import/order */
export const FIREFOX_MOBILE_USER_AGENT = 'Mozilla/5.0 (Android; Mobile; rv:14.0) Gecko/14.0 Firefox/14.0';
export const FIREFOX_DESKTOP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0';
@@ -16,24 +20,24 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
};
/**
- * @typedef {Object} RequestAsBrowserOptions
- * @property {String} url
+ * @typedef RequestAsBrowserOptions
+ * @property {string} url
* URL of the target endpoint. Supports both HTTP and HTTPS schemes.
- * @property {String} [method=GET]
+ * @property {string} [method=GET]
* HTTP method.
* @property {Object} [headers]
* Additional HTTP headers to add. It's only recommended to use this option,
* with headers that are typically added by websites, such as cookies. Overriding
* default browser headers will remove the masking this function provides.
- * @property {String} [proxyUrl]
+ * @property {string} [proxyUrl]
* An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.
- * @property {String} [languageCode=en]
+ * @property {string} [languageCode=en]
* Two-letter ISO 639 language code.
- * @property {String} [countryCode=US]
+ * @property {string} [countryCode=US]
* Two-letter ISO 3166 country code.
- * @property {Boolean} [useMobileVersion]
+ * @property {boolean} [useMobileVersion]
* If `true`, the function uses User-Agent of a mobile browser.
- * @property {Function} [abortFunction]
+ * @property {AbortFunction} [abortFunction]
* Function accepts `response` object as a single parameter and should return true or false.
* If function returns true request gets aborted. This function is passed to the
* [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.
@@ -49,13 +53,19 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
* Learn more in this [blog post](https://snyk.io/blog/node-js-release-fixes-a-critical-http-security-vulnerability/).
*/
+/**
+ * @callback AbortFunction
+ * @param {IncomingMessage} response
+ * @returns {boolean}
+ */
+
/**
* **IMPORTANT:** This function uses an insecure version of HTTP parser by default
* and also ignores SSL/TLS errors. This is very useful in scraping, because it allows bypassing
* certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping
* scenarios, please set `useInsecureHttpParser: false` and `ignoreSslErrors: false`.
*
- * Sends an HTTP request that looks like a request sent by a web browser,
+ * Sends a HTTP request that looks like a request sent by a web browser,
* fully emulating browser's HTTP headers.
*
* This function is useful for web scraping of websites that send the full HTML in the first response.
@@ -75,7 +85,7 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
*
* @param {RequestAsBrowserOptions} options All `requestAsBrowser` configuration options.
*
- * @return {Promise<http.IncomingMessage|stream.Readable>} This will typically be a
+ * @return {Promise<(IncomingMessage|Readable)>} This will typically be a
* [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage),
* however, if returned from the cache it will be a [response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.
* @memberOf utils
diff --git a/src/utils_social.js b/src/utils_social.js
index 18d168ec8..617b2b4e7 100644
--- a/src/utils_social.js
+++ b/src/utils_social.js
@@ -1,7 +1,7 @@
/* eslint-disable no-continue */
-import _ from 'underscore';
-import cheerio from 'cheerio';
-import log from 'apify-shared/log';
+import * as _ from 'underscore';
+import * as cheerio from 'cheerio';
+import log from './utils_log';
import { publicUtils } from './utils';
// TODO: We could support URLs like https://www.linkedin.com/company/some-company-inc
@@ -33,8 +33,8 @@ const EMAIL_URL_PREFIX_REGEX = /^mailto:/i;
/**
* The function extracts email addresses from a plain text.
* Note that the function preserves the order of emails and keep duplicates.
- * @param {String} text Text to search in.
- * @return {String[]} Array of emails addresses found.
+ * @param {string} text Text to search in.
+ * @return {string[]} Array of emails addresses found.
* If no emails are found, the function returns an empty array.
* @memberOf social
*/
@@ -48,8 +48,8 @@ const emailsFromText = (text) => {
* The function extracts email addresses from a list of URLs.
* Basically it looks for all `mailto:` URLs and returns valid email addresses from them.
* Note that the function preserves the order of emails and keep duplicates.
- * @param {String[]} urls Array of URLs.
- * @return {String[]} Array of emails addresses found.
+ * @param {string[]} urls Array of URLs.
+ * @return {string[]} Array of emails addresses found.
* If no emails are found, the function returns an empty array.
* @memberOf social
*/
@@ -133,8 +133,8 @@ const SKIP_PHONE_REGEX = new RegExp(`^(${SKIP_PHONE_REGEXS.join('|')})$`, 'i');
* The function attempts to extract phone numbers from a text. Please note that
* the results might not be accurate, since phone numbers appear in a large variety of formats and conventions.
* If you encounter some problems, please [file an issue](https://github.com/apifytech/apify-js/issues).
- * @param {String} text Text to search the phone numbers in.
- * @return {String[]} Array of phone numbers found.
+ * @param {string} text Text to search the phone numbers in.
+ * @return {string[]} Array of phone numbers found.
* If no phone numbers are found, the function returns an empty array.
* @memberOf social
*/
@@ -161,8 +161,8 @@ const phonesFromText = (text) => {
/**
* Finds phone number links in an array of URLs and extracts the phone numbers from them.
* Note that the phone number links look like `tel://123456789`, `tel:/123456789` or `tel:123456789`.
- * @param {String[]} urls Array of URLs.
- * @return {String[]} Array of phone numbers found.
+ * @param {string[]} urls Array of URLs.
+ * @return {string[]} Array of phone numbers found.
* If no phone numbers are found, the function returns an empty array.
* @memberOf social
*/
@@ -509,14 +509,14 @@ try {
* }
* ```
* @typedef SocialHandles
- * @property {String[]} emails
- * @property {String[]} phones
- * @property {String[]} phonesUncertain
- * @property {String[]} linkedIns
- * @property {String[]} twitters
- * @property {String[]} instagrams
- * @property {String[]} facebooks
- * @property {String[]} youtubes
+ * @property {string[]} emails
+ * @property {string[]} phones
+ * @property {string[]} phonesUncertain
+ * @property {string[]} linkedIns
+ * @property {string[]} twitters
+ * @property {string[]} instagrams
+ * @property {string[]} facebooks
+ * @property {string[]} youtubes
*/
/**
@@ -543,7 +543,7 @@ try {
* console.dir(result);
* ```
*
- * @param {String} html HTML text
+ * @param {string} html HTML text
* @param {Object} data Optional object which will receive the `text` and `$` properties
* that contain text content of the HTML and `cheerio` object, respectively. This is an optimization
* so that the caller doesn't need to parse the HTML document again, if needed.
diff --git a/test/autoscaling/autoscaled_pool.test.js b/test/autoscaling/autoscaled_pool.test.js
index 61c5b92fe..7746b716a 100644
--- a/test/autoscaling/autoscaled_pool.test.js
+++ b/test/autoscaling/autoscaled_pool.test.js
@@ -1,5 +1,5 @@
import _ from 'underscore';
-import log from 'apify-shared/log';
+import log from '../../build/utils_log';
import { sleep } from '../../build/utils';
import AutoscaledPool from '../../build/autoscaling/autoscaled_pool';
diff --git a/test/autoscaling/snapshotter.test.js b/test/autoscaling/snapshotter.test.js
index 83492db77..30dcff499 100644
--- a/test/autoscaling/snapshotter.test.js
+++ b/test/autoscaling/snapshotter.test.js
@@ -2,8 +2,8 @@
import os from 'os';
import sinon from 'sinon';
-import log from 'apify-shared/log';
import { ACTOR_EVENT_NAMES, ENV_VARS } from 'apify-shared/consts';
+import log from '../../build/utils_log';
import * as Apify from '../../build/index';
import events from '../../build/events';
import Snapshotter from '../../build/autoscaling/snapshotter';
diff --git a/test/autoscaling/system_status.test.js b/test/autoscaling/system_status.test.js
index 559f258a3..6d4193ae6 100644
--- a/test/autoscaling/system_status.test.js
+++ b/test/autoscaling/system_status.test.js
@@ -1,4 +1,4 @@
-import log from 'apify-shared/log';
+import log from '../../build/utils_log';
import SystemStatus from '../../build/autoscaling/system_status';
import Snapshotter from '../../build/autoscaling/snapshotter';
diff --git a/test/crawlers/basic_crawler.test.js b/test/crawlers/basic_crawler.test.js
index b95122207..e48cfd91e 100644
--- a/test/crawlers/basic_crawler.test.js
+++ b/test/crawlers/basic_crawler.test.js
@@ -1,7 +1,7 @@
import _ from 'underscore';
import sinon from 'sinon';
-import log from 'apify-shared/log';
import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';
+import log from '../../build/utils_log';
import * as Apify from '../../build';
import * as keyValueStore from '../../build/key_value_store';
import { RequestQueue, RequestQueueLocal } from '../../build/request_queue';
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 3154dd619..8aa80a339 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -1,10 +1,10 @@
import fs from 'fs';
import path from 'path';
-import log from 'apify-shared/log';
import { ENV_VARS } from 'apify-shared/consts';
import express from 'express';
import bodyParser from 'body-parser';
import sinon from 'sinon';
+import log from '../../build/utils_log';
import Apify from '../../build';
import { sleep } from '../../build/utils';
import { Session } from '../../build/session_pool/session';
diff --git a/test/crawlers/puppeteer_crawler.test.js b/test/crawlers/puppeteer_crawler.test.js
index 7627a795a..2e6f0db57 100644
--- a/test/crawlers/puppeteer_crawler.test.js
+++ b/test/crawlers/puppeteer_crawler.test.js
@@ -1,5 +1,5 @@
-import log from 'apify-shared/log';
import { ENV_VARS } from 'apify-shared/consts';
+import log from '../../build/utils_log';
import * as Apify from '../../build';
import LocalStorageDirEmulator from '../local_storage_dir_emulator';
diff --git a/test/crawlers/statistics.test.js b/test/crawlers/statistics.test.js
index b56c1cd7a..b10272fad 100644
--- a/test/crawlers/statistics.test.js
+++ b/test/crawlers/statistics.test.js
@@ -1,5 +1,5 @@
import sinon from 'sinon';
-import log from 'apify-shared/log';
+import log from '../../build/utils_log';
import Statistics from '../../build/crawlers/statistics';
describe('Statistics', () => {
diff --git a/test/enqueue_links/click_elements.test.js b/test/enqueue_links/click_elements.test.js
index 19e843d93..5478721fb 100644
--- a/test/enqueue_links/click_elements.test.js
+++ b/test/enqueue_links/click_elements.test.js
@@ -30,7 +30,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<a href="https://example.com">link</div>
- </body>
+ </body>
</html>
`;
@@ -58,7 +58,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="window.clicked = true;">div</div>
- </body>
+ </body>
</html>
`;
await page.setContent(html);
@@ -72,7 +72,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="window.clicked = true;"></div>
- </body>
+ </body>
</html>
`;
await page.setContent(html);
@@ -92,7 +92,7 @@ describe('enqueueLinksByClickingElements()', () => {
<div onclick="return window.handleClick(event)"></div>
<p onclick="return window.handleClick(event)"></p>
<footer onclick="return window.handleClick(event)"></footer>
- </body>
+ </body>
</html>
`;
await page.setContent(html);
@@ -106,7 +106,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="window.clicked = true;" style="visibility: hidden; display: none"></div>
- </body>
+ </body>
</html>
`;
await page.setContent(html);
@@ -127,7 +127,7 @@ describe('enqueueLinksByClickingElements()', () => {
<div onclick="return window.handleClick(event)" style="position: absolute; z-index: 1">div</div>
<main onclick="return window.handleClick(event)" style="position: absolute; z-index: 2">main</main>
<footer onclick="return window.handleClick(event)" style="position: absolute; z-index: 3">footer</footer>
- </body>
+ </body>
</html>
`;
await page.setContent(html);
@@ -153,7 +153,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<a href="https://example.com">link</div>
- </body>
+ </body>
</html>
`;
await page.setContent(html);
@@ -170,7 +170,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="return window.location = 'https://example.com'">div</div>
- </body>
+ </body>
</html>
`;
await page.goto('https://example.com');
@@ -189,7 +189,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="return window.location = '#foo'">div</div>
- </body>
+ </body>
</html>
`;
await page.goto('https://example.com');
@@ -207,7 +207,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="return window.location.reload()">div</div>
- </body>
+ </body>
</html>
`;
await page.goto('https://example.com');
@@ -224,7 +224,7 @@ describe('enqueueLinksByClickingElements()', () => {
<html>
<body>
<div onclick="return window.location.reload(true)">div</div>
- </body>
+ </body>
</html>
`;
await page.goto('https://example.com');
diff --git a/test/local_storage_dir_emulator.js b/test/local_storage_dir_emulator.js
index 10b4b3073..7ceedf3c4 100644
--- a/test/local_storage_dir_emulator.js
+++ b/test/local_storage_dir_emulator.js
@@ -2,7 +2,7 @@ import { cryptoRandomObjectId } from 'apify-shared/utilities';
import { LOCAL_STORAGE_SUBDIRS, LOCAL_ENV_VARS, ENV_VARS } from 'apify-shared/consts';
import fs from 'fs-extra';
import path from 'path';
-import log from 'apify-shared/log';
+import log from '../build/utils_log';
import { LOCAL_STORAGE_DIR } from './_helper';
diff --git a/test/index.esm.test.js b/test/main.test.js
similarity index 100%
rename from test/index.esm.test.js
rename to test/main.test.js
diff --git a/test/puppeteer_pool.test.js b/test/puppeteer_pool.test.js
index 492328ea6..5f16110de 100644
--- a/test/puppeteer_pool.test.js
+++ b/test/puppeteer_pool.test.js
@@ -1,12 +1,12 @@
import fs from 'fs';
import _ from 'underscore';
-import log from 'apify-shared/log';
import { ENV_VARS } from 'apify-shared/consts';
+import log from '../build/utils_log';
import * as Apify from '../build/index';
import { launchPuppeteer } from '../build/puppeteer';
import { SessionPool } from '../build/session_pool/session_pool';
import { BROWSER_SESSION_KEY_NAME } from '../build/puppeteer_pool';
-import { sleep } from '../src/utils';
+import { sleep } from '../build/utils';
import LocalStorageDirEmulator from './local_storage_dir_emulator';
diff --git a/test/request_list.test.js b/test/request_list.test.js
index ed01e6d8a..760bc5f15 100644
--- a/test/request_list.test.js
+++ b/test/request_list.test.js
@@ -1,7 +1,7 @@
import _ from 'underscore';
import request from 'request-promise-native';
import sinon from 'sinon';
-import log from 'apify-shared/log';
+import log from '../build/utils_log';
import { ACTOR_EVENT_NAMES_EX } from '../build/constants';
import Apify from '../build/index';
import * as keyValueStore from '../build/key_value_store';
diff --git a/test/session_pool/session_pool.test.js b/test/session_pool/session_pool.test.js
index 64bb69687..37bc76772 100644
--- a/test/session_pool/session_pool.test.js
+++ b/test/session_pool/session_pool.test.js
@@ -3,7 +3,7 @@ import Apify from '../../build';
import events from '../../build/events';
import { ACTOR_EVENT_NAMES_EX } from '../../build/constants';
-import { Session } from '../../src/session_pool/session';
+import { Session } from '../../build/session_pool/session';
import LocalStorageDirEmulator from '../local_storage_dir_emulator';
describe('SessionPool - testing session pool', () => {
diff --git a/test/typescript/crawlers/basic_crawler.test.ts b/test/typescript/crawlers/basic_crawler.test.ts
new file mode 100644
index 000000000..224fce23b
--- /dev/null
+++ b/test/typescript/crawlers/basic_crawler.test.ts
@@ -0,0 +1,50 @@
+import { BasicCrawler, BasicCrawlerOptions } from "../../..";
+
+interface RequestData {
+ myValue: string;
+ myMaybeValue?: boolean;
+}
+
+interface SessionData {
+ userAgent: string;
+}
+
+describe('BasicCrawler TS', () => {
+ describe('generics', () => {
+ test('options', () => {
+ const requestQueue: any = {
+ addRequest: () => {}
+ };
+
+ const options: BasicCrawlerOptions = {
+ handleRequestFunction: null as any,
+ requestQueue,
+ sessionPoolOptions: {
+ sessionOptions: {
+ sessionPool: null as any,
+ userData: {
+ userAgent: 'user-agent'
+ }
+ }
+ }
+ };
+
+ options.requestQueue!.addRequest({
+ url: '',
+ userData: {
+ myValue: 'asdf'
+ }
+ });
+
+ options.requestQueue!.addRequest({
+ url: '',
+ userData: {
+ myValue: 'asdf',
+ myMaybeValue: false
+ }
+ }, { forefront: true });
+
+ options.sessionPoolOptions!.sessionOptions!.userData!.userAgent === 'user-agent';
+ })
+ });
+});
diff --git a/test/typescript/crawlers/cheerio_crawler.test.ts b/test/typescript/crawlers/cheerio_crawler.test.ts
index fcb78e10c..43a5c4e11 100644
--- a/test/typescript/crawlers/cheerio_crawler.test.ts
+++ b/test/typescript/crawlers/cheerio_crawler.test.ts
@@ -1,71 +1,82 @@
-/// <reference types="jest"/>
-import Apify, {CheerioCrawler, CheerioHandlePage, CheerioHandlePageInputs, RequestList} from "../../..";
-import cheerio from "cheerio";
+import Apify, {
+ CheerioCrawler,
+ CheerioHandlePage,
+ CheerioHandlePageInputs,
+ RequestList
+} from "../../..";
+import * as cheerio from "cheerio";
-describe('CheerioHandlePage', () => {
- let testInputs: CheerioHandlePageInputs;
+describe('CheerioCrawler TS', () => {
+ describe('generics', () => {
+ test('options', () => {
- beforeEach(() => {
- const body = '<a href="#">';
- testInputs = {
- $: cheerio.load(body),
- body: body,
- request: new Apify.Request({url: '#'}),
- contentType: {type: 'text/html', encoding: 'utf-8'},
- response: null,
- autoscaledPool: null
- };
+ });
});
- test('Can pass around and call `handler(var: CheerioHandlePageInputs)`', async () => {
- // This form can be easily reused returning arbitrary type by passing `inputs` from another handler function and
- // processing the return value there. If returning anything, this function can't be used directly in a crawler.
- // Auto-completion works on all input variables in parameter `inputs`.
- const x = async (inputs: CheerioHandlePageInputs) => {
- expect(inputs.$('a').attr('href')).toEqual('#');
- };
+ describe('CheerioHandlePage', () => {
+ let testInputs: CheerioHandlePageInputs;
- const crawlerX = new CheerioCrawler({
- handlePageFunction: x,
- requestList: new RequestList({ sources: [] }),
+ beforeEach(() => {
+ const body = '<a href="#">';
+ testInputs = {
+ $: cheerio.load(body),
+ body: body,
+ request: new Apify.Request({url: '#'}),
+ contentType: {type: 'text/html', encoding: 'utf-8'},
+ response: null as any,
+ autoscaledPool: null as any
+ };
});
- await x(testInputs);
- });
+ test('Can pass around and call `handler(var: CheerioHandlePageInputs)`', async () => {
+ // This form can be easily reused returning arbitrary type by passing `inputs` from another handler function and
+ // processing the return value there. If returning anything, this function can't be used directly in a crawler.
+ // Auto-completion works on all input variables in parameter `inputs`.
+ const x = async (inputs: CheerioHandlePageInputs) => {
+ expect(inputs.$!('a').attr('href')).toEqual('#');
+ };
- test('Can pass around and call `handler({ var }: { var: Type})`', async () => {
- // This form can also be easily reused as above.
- // Auto-completion works on defined input variables in parameter list.
- const y = async ({$}: { $?: CheerioStatic }) => {
- expect($('a').attr('href')).toEqual('#');
- };
+ const crawlerX = new CheerioCrawler({
+ handlePageFunction: x,
+ requestList: new RequestList({ sources: [] }),
+ });
+
+ await x(testInputs);
+ });
- const crawler = new CheerioCrawler({
- handlePageFunction: y,
- requestList: new RequestList({ sources: [] }),
+ test('Can pass around and call `handler({ var }: { var: Type})`', async () => {
+ // This form can also be easily reused as above.
+ // Auto-completion works on defined input variables in parameter list.
+ const y = async ({$}: { $?: CheerioSelector }) => {
+ expect($!('a').attr('href')).toEqual('#');
+ };
+ const crawler = new CheerioCrawler({
+ handlePageFunction: y,
+ requestList: new RequestList({ sources: [] }),
+ });
+
+ await y(testInputs);
});
- await y(testInputs);
- });
+ test('Can pass around and call `handler: CheerioHandlePage`', async () => {
+ // In this form, the return type of the function is strictly enforced to be `Promise<void>`. This form can be reused
+ // if it produces the desired side-effects (or nothing) without returning a value. On the other hand, we can describe
+ // the parameter list in any of the above styles without attaching a typing to input variables.
+ // Auto-completion works in both cases.
+ // Type-checking guards from potential errors in logic, be raising an error if return type does not match what is
+ // expected by the crawler.
+ const z: CheerioHandlePage = async ({$ = null}) => {
+ expect($!('a').attr('href')).toEqual('#');
+ };
- test('Can pass around and call `handler: CheerioHandlePage`', async () => {
- // In this form, the return type of the function is strictly enforced to be `Promise<void>`. This form can be reused
- // if it produces the desired side-effects (or nothing) without returning a value. On the other hand, we can describe
- // the parameter list in any of the above styles without attaching a typing to input variables.
- // Auto-completion works in both cases.
- // Type-checking guards from potential errors in logic, be raising an error if return type does not match what is
- // expected by the crawler.
- const z: CheerioHandlePage = async ({$ = null}) => {
- expect($('a').attr('href')).toEqual('#');
- };
+ const crawler = new CheerioCrawler({
+ handlePageFunction: z,
+ requestList: new RequestList({ sources: [] }),
+ });
- const crawler = new CheerioCrawler({
- handlePageFunction: z,
- requestList: new RequestList({ sources: [] }),
+ await z(testInputs);
});
-
- await z(testInputs);
});
});
diff --git a/test/typescript/crawlers/puppeteer_crawler.test.ts b/test/typescript/crawlers/puppeteer_crawler.test.ts
new file mode 100644
index 000000000..e43add5c8
--- /dev/null
+++ b/test/typescript/crawlers/puppeteer_crawler.test.ts
@@ -0,0 +1,15 @@
+import { BasicCrawler, BasicCrawlerOptions } from "../../..";
+
+interface RequestData {
+ myValue: string;
+}
+
+interface SessionData {
+
+}
+
+describe('PuppeteerCrawler TS', () => {
+ test('generics', () => {
+
+ });
+});
diff --git a/test/typescript/dataset.test.ts b/test/typescript/dataset.test.ts
new file mode 100644
index 000000000..e961e104c
--- /dev/null
+++ b/test/typescript/dataset.test.ts
@@ -0,0 +1,7 @@
+import Apify from "../..";
+
+describe('Dataset TS', () => {
+ test('generics', () => {
+
+ });
+});
diff --git a/test/typescript/key_value_store.test.ts b/test/typescript/key_value_store.test.ts
new file mode 100644
index 000000000..344e9fdef
--- /dev/null
+++ b/test/typescript/key_value_store.test.ts
@@ -0,0 +1,7 @@
+import Apify from "../..";
+
+describe('KeyValueStore TS', () => {
+ test('generics', () => {
+
+ });
+});
diff --git a/test/typescript/utils_log.test.ts b/test/typescript/utils_log.test.ts
index a27551ea7..0159c3e4a 100644
--- a/test/typescript/utils_log.test.ts
+++ b/test/typescript/utils_log.test.ts
@@ -1,7 +1,6 @@
-/// <reference types="jest"/>
import Apify from "../..";
-describe('Typescript with Apify.utils.log', () => {
+describe('utils.log TS', () => {
const log = Apify.utils.log;
test('log.debug()', () => {
diff --git a/test/utils.test.js b/test/utils.test.js
index 580b00871..7a0148f15 100644
--- a/test/utils.test.js
+++ b/test/utils.test.js
@@ -7,8 +7,8 @@ import cheerio from 'cheerio';
import semver from 'semver';
import requestPromise from 'request-promise-native';
import LruCache from 'apify-shared/lru_cache';
-import log from 'apify-shared/log';
import { ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';
+import log from '../build/utils_log';
import * as utils from '../build/utils';
import Apify from '../build/index';
diff --git a/test/utils_social.test.js b/test/utils_social.test.js
index 841043b06..c00867dac 100644
--- a/test/utils_social.test.js
+++ b/test/utils_social.test.js
@@ -36,7 +36,7 @@ describe('utils.social', () => {
testEmailsFromText(`
this'is'also'valid'[email protected]
not @ an.email.com
@also.not.an.email
`, [
@@ -124,10 +124,10 @@ describe('utils.social', () => {
test('extracts phones correctly', () => {
testPhonesFromText(`
+420775123456 +420775123456
-
+
+420 775 123 456
-
- 775123456 775123456 \n\n
+
+ 775123456 775123456 \n\n
00420775123456
1234567 1234567890
+44 7911 123456
@@ -216,7 +216,7 @@ describe('utils.social', () => {
000-0000
000 0000
000.0000
-
+
0000000
0000000000
(000)0000000
@@ -332,26 +332,26 @@ describe('utils.social', () => {
expect(social.parseHandlesFromHtml(`
<html>
<head>
- <title>Bla</title>
+ <title>Bla</title>
</head>
<a>
- [email protected] testing skipping duplicates xxx@blabla
+ [email protected] testing skipping duplicates xxx@blabla
<p>carl&#64;example.com</p>
<a href="mailto:[email protected]"></a>
- <a href="mailto:david&#64;example.com"></a>
-
- <a href="[email protected]"></a>
+ <a href="mailto:david&#64;example.com"></a>
+
+ <a href="[email protected]"></a>
<img src="http://somewhere.com/ [email protected] " />
<a href="http://somewhere.com/ [email protected] "></a>
-
- &#43;420775222222
+
+ &#43;420775222222
+4207751111111
+4207751111111 test duplicate
1 2 3 4 5 6 7 8 9 0
<a href="skip.this.one: +42099999999"></a>
<a href="tel://+42077533333"></a>
-
+
https://www.linkedin.com/in/bobnewman
https://www.linkedin.com/in/alicenewman/
https://www.linkedin.com/in/alicenewman/ duplicate
@@ -361,27 +361,27 @@ describe('utils.social', () => {
https://www.linkedin.com/in/first-last-123456a
<a href="https://www.linkedin.com/in/jancurn">Profile</a>
<a href="https://www.linkedin.com/in/carl-newman-5555555a/detail/recent-activity/">Sub-link</a>
-
+
https://www.instagram.com/old_prague/
https://www.instagram.com/old_prague/ duplicate
- instagram.com/old_prague
- https://www.instagram.com/newyorkarea/
+ instagram.com/old_prague
+ https://www.instagram.com/newyorkarea/
<a href="https://www.instagram.com/york">link</a>
<a href="https://www.instagram.com/york2/something">sub-link</a>
-
+
<a href="twitter.com/betasomething">link</a>
https://www.twitter.com/apify
- https://www.twitter.com/apify duplicate
+ https://www.twitter.com/apify duplicate
<a href="twitter.com/cblabla/sub-dir/">link</a>
-
- <a href="facebook.com/carl.username123/sub-dir/">link</a>
+
+ <a href="facebook.com/carl.username123/sub-dir/">link</a>
https://www.facebook.com/bob.username123/
https://www.facebook.com/bob.username123/ duplicate
http://www.facebook.com/alice.username123
<a href="https://www.facebook.com/profile.php?id=1155802082&xxx=5">link x</a>
<a href="https://youtu.be/kM7YfhfkiEE">Youtube</a>
<a href="fb.com/dada5678?query=1">link</a>
-
+
</body>
</html>
`)).toEqual({
@@ -428,7 +428,7 @@ describe('utils.social', () => {
social.parseHandlesFromHtml(`
<html>
<head>
- <title>Bla</title>
+ <title>Bla</title>
</head>
<body>
Body content
@@ -516,7 +516,7 @@ describe('utils.social', () => {
expect(social.LINKEDIN_REGEX_GLOBAL.test('https://www.linkedin.com/in/bobnewman')).toBe(true);
expect(`
- https://www.linkedin.com/in/bobnewman
+ https://www.linkedin.com/in/bobnewman
"http://ie.linkedin.com/in/alicenewman"
https://www.linkedin.com/in/someverylongnamesomeverylongnamesomeverylongnamesomeverylongnamesomeverylongnamesomeverylongname
linkedin.com/in/carlnewman
@@ -583,7 +583,7 @@ describe('utils.social', () => {
expect(social.INSTAGRAM_REGEX_GLOBAL.test('https://www.instagram.com/old_prague')).toBe(true);
expect(`
https://www.instagram.com/old_prague
- https://www.instagram.com/someverylongusernamethatisnotgood
+ https://www.instagram.com/someverylongusernamethatisnotgood
"instagram.com/old_brno"
http://instagr.am/old_plzen
`.match(social.INSTAGRAM_REGEX_GLOBAL)).toEqual([
diff --git a/tools/typescript_fixes.js b/tools/typescript_fixes.js
index 1f6ceab5a..606d66ea7 100644
--- a/tools/typescript_fixes.js
+++ b/tools/typescript_fixes.js
@@ -1,23 +1,26 @@
const { readdirSync, readFileSync, writeFileSync, realpathSync, statSync } = require('fs');
const path = require('path');
-console.debugFixLog = false ? console.debug : () => {
-};
-console.debugReadTypes = false ? console.debug : () => {
-};
-console.debugRemoveReferences = false ? console.debug : () => {
+const debug = (namespace, ...args) => {
+ if (
+ process.env.DEBUG_TYPEFIXES
+ && (process.env.DEBUG_TYPEFIXES === '*' || process.env.DEBUG_TYPEFIXES.includes(namespace))
+ ) {
+ console.log(...args);
+ }
};
console.log('Currect directory:', realpathSync('./'));
const typesPath = './types';
const paths = {
- 'index.esm': path.join(typesPath, 'index.esm.d.ts'),
+ main: path.join(typesPath, 'main.d.ts'),
+ session_pool: path.join(typesPath, 'session_pool', 'session_pool.d.ts'),
};
-console.log('Processing', paths['index.esm']);
+console.log('Processing', paths['main']);
/**
- * Removes `namespace log` and adds appropriate import and reference to index.esm.d.ts.
+ * Removes `namespace log` and adds appropriate import and reference to main.d.ts.
*
* @param {string[]} input
* @return {string[]}
@@ -35,24 +38,24 @@ const fixLog = (input) => {
for (const line of input) {
try {
if (mode === modes.base) {
- if (index === 0) {
- output.push('/// <reference path="../src/utils_log.d.ts" />');
- console.debugFixLog('line #1', mode);
- }
+ // if (index === 0) {
+ // output.push('/// <reference path="../src/utils_log.d.ts" />');
+ // debug('fixLog', 'line #1', mode);
+ // }
if (line.match(/^declare namespace log {/)) {
mode = modes['log-ns-replace'];
- console.debugFixLog('switch to', mode, line);
+ debug('fixLog', 'switch to', mode, line);
} else {
output.push(line);
- console.debugFixLog('out', line);
+ debug('fixLog', 'out', line);
}
} else if (mode === modes['log-ns-replace']) {
if (line.match(/^\}/)) {
mode = modes.base;
- console.debugFixLog('switch to', mode, line);
+ debug('fixLog', 'switch to', mode, line);
} else {
// nothing
- console.debugFixLog('remove ', line);
+ debug('fixLog', 'remove ', line);
}
}
} finally {
@@ -62,16 +65,49 @@ const fixLog = (input) => {
return output;
};
+const fixSessionPool = async () => {
+ const input = readFileSync(paths.session_pool, { encoding: 'utf8' })
+ .split('\n');
+
+ const matchLines = [
+ /addListener\(event/,
+ /on\(event/,
+ /once\(event/,
+ /prependListener\(event/,
+ /prependOnceListener\(event/,
+ /removeListener\(event/,
+ /off\(event/,
+ /removeAllListeners\(event/,
+ /setMaxListeners\(n/,
+ ];
+ const output = [];
+ let changed = false;
+
+ for (const line of input) {
+ if (!matchLines.some(s => s.test(line))) {
+ output.push(line);
+ } else {
+ changed = true;
+ debug('fixSessionPool', 'removing line', line);
+ }
+ }
+
+ if (changed) {
+ console.log('Writing file', paths.session_pool);
+ writeFileSync(paths.session_pool, output.join('\n'), { encoding: 'utf8' });
+ }
+};
+
/**
* @callback FilenameFilter
* @param {String} path Relative path to file considered.
- * @returns {Boolen} `true` is the path should be processed, `false` otherwise.
+ * @returns {Boolean} `true` is the path should be processed, `false` otherwise.
*/
/**
* @callback FileHandler
* @param {String} content Relative path to file.
- * @returns {String} Content after transformation.
+ * @returns {{ [index: string]: object|array }} Content after transformation.
*/
/**
@@ -80,14 +116,14 @@ const fixLog = (input) => {
* @param {String} dir Starting directory.
* @param {FilenameFilter} filter Pathname filter function.
* @param {FileHandler} handleFile File processing function.
- * @returns {object<string,object|array>} Hierarchical collection of `handleFile`'s return values.
+ * @returns {Promise<{ [index: string]: object|array }>} Hierarchical collection of `handleFile`'s return values.
*/
const traverseDirs = async (dir, filter, handleFile) => {
console.log('Reading directory', dir);
const types = {};
const dirContent = readdirSync(dir);
- dirContent.map(async (entry) => {
+ for (const entry of dirContent) {
const entryPath = path.join(dir, entry);
const entryStat = statSync(entryPath);
@@ -98,23 +134,25 @@ const traverseDirs = async (dir, filter, handleFile) => {
} else if (entryStat.isDirectory()) {
types[entry] = await traverseDirs(entryPath, filter, handleFile);
}
- });
+ }
return types;
};
+// const EXCLUDED_FROM_EXPORT = [/Session/, /RequestList/, /Local$/];
+
/**
* Blindly extracts exported typenames from a `*.d.ts` file.
* @param {string} filepath
* @return {string[]}
*/
const readTypes = (filepath) => {
- const input = readFileSync(filepath)
- .toString();
+ const input = readFileSync(filepath, { encoding: 'utf8' });
const inputByLine = input.split('\n');
const types = [];
for (const line of inputByLine) {
- const matches = line.match(/^export type (\S+)/);
- console.debugReadTypes(!!matches, line);
+ const matches = line.match(/^export type ([^<\s]+)/); // get exported types without generic Export<T>
+ debug('readTypes', !!matches, line);
+ // && !EXCLUDED_FROM_EXPORT.some(s => s.test(matches[1]))
if (matches) {
types.push(matches[1]);
}
@@ -124,10 +162,10 @@ const readTypes = (filepath) => {
/**
* Flattens hierarchy of type exports into a mapping from relative paths to exported types.
- * @param {object<string,object|array>} types
+ * @param {{ [index: string] : object|array }} types
* @param {string} prefix
- * @param {object<string,string[]>} output
- * @return {object<string,string[]>} Dictionary mapping module declaration paths to exported types.
+ * @param {{ [index: string]: string[] }} output
+ * @return {{ [index: string]: string[] }} Dictionary mapping module declaration paths to exported types.
*/
const typeHierarchyToExports = (types, prefix = null, output = {}) => {
for (const key of Object.keys(types)) {
@@ -152,7 +190,7 @@ const fixTypes = (input, types = {}) => {
const exports = typeHierarchyToExports(types, './');
for (const file of Object.keys(exports)) {
const exportedTypes = exports[file];
- const line = `export { ${exportedTypes.join(', ')} } from '${file}'`;
+ const line = `export { ${exportedTypes.join(', ')} } from "${file}";`;
if (outputCache[line] === undefined) {
output.push(line);
outputCache[line] = true;
@@ -169,36 +207,42 @@ const processIndexEsm = async () => {
},
readTypes,
);
- const input = readFileSync(paths['index.esm'])
- .toString()
+ const input = readFileSync(paths['main'], { encoding: 'utf8' })
.split('\n');
- console.log('Fixing log module', paths['index.esm']);
- const fixedLog = fixLog(input);
- console.log('Fixing type exports', paths['index.esm']);
- const fixedTypes = fixTypes(fixedLog, types);
- console.log('Writing', paths['index.esm']);
- writeFileSync(paths['index.esm'], fixedTypes.join('\n'));
+ console.log('Fixing log module', paths['main']);
+ // const fixedLog = fixLog(input);
+ console.log('Fixing type exports', paths['main']);
+ const fixedTypes = fixTypes(input, types);
+ console.log('Writing', paths['main']);
+ writeFileSync(paths['main'], fixedTypes.join('\n'));
};
-const removeAllNodeReferences = async () => {
+const fixTypesReferences = async () => {
await traverseDirs(
typesPath,
(filename) => {
return filename.endsWith('.d.ts');
},
(filepath) => {
- const input = readFileSync(filepath)
- .toString()
+ const input = readFileSync(filepath, { encoding: 'utf8' })
.split('\n');
const output = [];
let changed = false;
+ let match;
for (const line of input) {
- if (line.match(/^\/\/\/\s*<reference\s*types="node"\s*\/>/)) {
- console.debugRemoveReferences('removing node types reference from file', filepath);
+ /* eslint-disable no-cond-assign */
+ if (match = line.match(/\/\/\/\s*<reference\s*types="types-apify\/([^"]+)"\s*\/>/)) {
+ debug('fixTypesReferences', 'fixing types-apify from file', filepath);
+ output.push(`/// <reference path="../types-apify/${match[1]}.d.ts" />`);
+ changed = true;
+ } else if (match = line.match(/^([^"]+)"node\/([^$]+)/)) {
+ debug('fixTypesReferences', 'fixing "node/" from file', filepath);
+ output.push(`${match[1]} "${match[2]}`);
changed = true;
} else {
output.push(line);
}
+ /* eslint-enable no-cond-assign */
}
if (changed === true) {
@@ -211,5 +255,6 @@ const removeAllNodeReferences = async () => {
(async () => {
await processIndexEsm();
- await removeAllNodeReferences();
-})().then();
+ await fixTypesReferences();
+ await fixSessionPool();
+})();
diff --git a/tsconfig.json b/tsconfig.json
index 3283c410d..805fddcfd 100644
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -4,29 +4,31 @@
"module": "commonjs",
"moduleResolution": "node",
"lib": [
- "es2018",
- "dom"
- ],
- "types": [
- "node",
- "cheerio"
+ "ES2018",
+ "ES2019.Object",
+ "ES2018.AsyncIterable"
],
+ "baseUrl": "./",
"allowJs": true,
"checkJs": false,
"noEmit": false,
+ "newLine": "lf",
"declaration": true,
- "emitDeclarationOnly": true,
- "strict": false,
+ "emitDeclarationOnly": false,
+ "strict": true,
+ "noImplicitAny": true,
+ "noImplicitReturns": true,
"noImplicitThis": true,
"alwaysStrict": true,
- "esModuleInterop": true,
- "outDir": "types/"
+ "noStrictGenericChecks": true,
+ "strictNullChecks": true,
+ "allowSyntheticDefaultImports": false,
+ "esModuleInterop": false,
+ "declarationDir": "types/",
+ "outDir": "build/"
},
"include": [
- "src/index.esm.js",
- "src/utils_log.d.ts"
- ],
- "exclude": [
- "src/index.js"
+ "./types-apify/**/*.d.ts",
+ "./src/**/*.js"
]
}
diff --git a/types-apify/apify-client.d.ts b/types-apify/apify-client.d.ts
new file mode 100644
index 000000000..ba4acc927
--- /dev/null
+++ b/types-apify/apify-client.d.ts
@@ -0,0 +1,10 @@
+declare module 'apify-client' {
+ class ApifyClient {
+ constructor(opts: any);
+ }
+ export = ApifyClient;
+}
+declare module 'apify-client/build/utils' {
+ export function checkParamOrThrow(...args: any): any;
+ export function parseBody(body: any, mime: any): any;
+}
diff --git a/types-apify/apify-http-request.d.ts b/types-apify/apify-http-request.d.ts
new file mode 100644
index 000000000..8ed3a6efb
--- /dev/null
+++ b/types-apify/apify-http-request.d.ts
@@ -0,0 +1,8 @@
+declare module '@apify/http-request' {
+ const EXPORTED: (...args: any) => any;
+ export = EXPORTED;
+}
+
+declare module '@apify/http-request/src/errors' {
+ export const TimeoutError: Error;
+}
diff --git a/types-apify/apify-shared/consts.d.ts b/types-apify/apify-shared/consts.d.ts
new file mode 100644
index 000000000..bd7216b6f
--- /dev/null
+++ b/types-apify/apify-shared/consts.d.ts
@@ -0,0 +1,49 @@
+declare module 'apify-shared/consts' {
+ export const ACTOR_EVENT_NAMES: {
+ MIGRATING: string;
+ SYSTEM_INFO: string;
+ CPU_INFO: string;
+ };
+ export const REQUEST_QUEUE_HEAD_MAX_LIMIT: number;
+ export const ENV_VARS: {
+ MEMORY_MBYTES: string;
+ API_BASE_URL: string;
+ IS_AT_HOME: string;
+ SDK_LATEST_VERSION: string;
+ TOKEN: string;
+ ACTOR_RUN_ID: string;
+ CONTAINER_PORT: string;
+ CONTAINER_URL: string;
+ LOCAL_STORAGE_DIR: string;
+ DEFAULT_DATASET_ID: string;
+ USER_ID: string;
+ DEFAULT_KEY_VALUE_STORE_ID: string;
+ ACTOR_EVENTS_WS_URL: string;
+ INPUT_KEY: string;
+ };
+ export const KEY_VALUE_STORE_KEYS: {
+ INPUT: string;
+ };
+ export const LOCAL_STORAGE_SUBDIRS: {
+ datasets: string;
+ requestQueues: string;
+ keyValueStores: string;
+ };
+ export const MAX_PAYLOAD_SIZE_BYTES: {
+
+ };
+ export const INTEGER_ENV_VARS: {
+
+ };
+ export const ACT_JOB_TERMINAL_STATUSES: {
+
+ };
+ export const ACT_JOB_STATUSES: {
+
+ };
+ export const LOCAL_ENV_VARS: {
+ [index: string]: string;
+ CONTAINER_PORT: string;
+ CONTAINER_URL: string;
+ };
+}
diff --git a/types-apify/apify-shared/linked_list.d.ts b/types-apify/apify-shared/linked_list.d.ts
new file mode 100644
index 000000000..af351b409
--- /dev/null
+++ b/types-apify/apify-shared/linked_list.d.ts
@@ -0,0 +1,7 @@
+declare module 'apify-shared/linked_list' {
+ class LinkedList {
+
+ }
+
+ export = LinkedList;
+}
diff --git a/types-apify/apify-shared/list_dictionary.d.ts b/types-apify/apify-shared/list_dictionary.d.ts
new file mode 100644
index 000000000..6ad372c2c
--- /dev/null
+++ b/types-apify/apify-shared/list_dictionary.d.ts
@@ -0,0 +1,7 @@
+declare module 'apify-shared/list_dictionary' {
+ class ListDictionary {
+
+ }
+
+ export = ListDictionary;
+}
diff --git a/types-apify/apify-shared/log.d.ts b/types-apify/apify-shared/log.d.ts
index 193e0cd52..dfd40d793 100644
--- a/types-apify/apify-shared/log.d.ts
+++ b/types-apify/apify-shared/log.d.ts
@@ -1,36 +1,195 @@
-declare module 'apify-log/log' {
- const LEVELS: {
- // Turns off logging completely
- OFF: number,
- // For unexpected errors in Apify system
- ERROR: number,
- // For situations where error is caused by user (e.g. Meteor.Error), i.e. when the error is not
- // caused by Apify system, avoid the word "ERROR" to simplify searching in log
- SOFT_FAIL: number,
- WARNING: number,
- INFO: number,
- DEBUG: number,
- // for performance stats
- PERF: number
- };
+declare module 'apify-shared/log' {
+ export interface LoggerOptions {
+ /**
+ * Sets the log level to the given value, preventing messages from less important log levels
+ * from being printed to the console. Use in conjunction with the `log.LEVELS` constants.
+ */
+ level: number;
+ /**
+ * Max depth of data object that will be logged. Anything deeper than the limit will be stripped off.
+ */
+ maxDepth: number;
+ /**
+ * Max length of the string to be logged. Longer strings will be truncated.
+ */
+ maxStringLength: number;
+ /**
+ * Prefix to be prepended the each logged line.
+ */
+ prefix: string;
+ /**
+ * Suffix that will be appended the each logged line.
+ */
+ suffix: string;
+ /**
+ * Logger implementation to be used. Default one is log.LoggerText to log messages as easily readable
+ * strings. Optionally you can use `log.LoggerJson` that formats each log line as a JSON.
+ */
+ logger: any;
+ /**
+ * Additional data to be added to each log line.
+ */
+ data: any;
+ }
- // Indicates whether DEBUG messages will be printed or not
- let isDebugMode: boolean;
+ /**
+ * The log instance enables level aware logging of messages and we advise
+ * to use it instead of `console.log()` and its aliases in most development
+ * scenarios.
+ *
+ * A very useful use case for `log` is using `log.debug` liberally throughout
+ * the codebase to get useful logging messages only when appropriate log level is set
+ * and keeping the console tidy in production environments.
+ *
+ * The available logging levels are, in this order: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `OFF`
+ * and can be referenced from the `log.LEVELS` constant, such as `log.LEVELS.ERROR`.
+ *
+ * To log messages to the system console, use the `log.level(message)` invocation,
+ * such as `log.debug('this is a debug message')`.
+ *
+ * To prevent writing of messages above a certain log level to the console, simply
+ * set the appropriate level. The default log level is `INFO`, which means that
+ * `DEBUG` messages will not be printed, unless enabled.
+ *
+ * **Example:**
+ * ```
+ * const Apify = require('apify');
+ * const { log } = Apify.utils;
+ *
+ * log.info('Information message', { someData: 123 }); // prints message
+ * log.debug('Debug message', { debugData: 'hello' }); // doesn't print anything
+ *
+ * log.setLevel(log.LEVELS.DEBUG);
+ * log.debug('Debug message'); // prints message
+ *
+ * log.setLevel(log.LEVELS.ERROR);
+ * log.debug('Debug message'); // doesn't print anything
+ * log.info('Info message'); // doesn't print anything
+ *
+ * log.error('Error message', { errorDetails: 'This is bad!' }); // prints message
+ * try {
+ * throw new Error('Not good!');
+ * } catch (e) {
+ * log.exception(e, 'Exception occurred', { errorDetails: 'This is really bad!' }); // prints message
+ * }
+ *
+ * log.setOptions({ prefix: 'My actor' });
+ * log.info('I am running!'); // prints "My actor: I am running"
+ *
+ * const childLog = log.child({ prefix: 'Crawler' });
+ * log.info('I am crawling!'); // prints "My actor:Crawler: I am crawling"
+ * ```
+ *
+ * Another very useful way of setting the log level is by setting the `APIFY_LOG_LEVEL`
+ * environment variable, such as `APIFY_LOG_LEVEL=DEBUG`. This way, no code changes
+ * are necessary to turn on your debug messages and start debugging right away.
+ */
+ export class Log {
+ deprecationsReported: any;
+ options: LoggerOptions;
+ constructor(options: Partial<LoggerOptions>);
+ getLevel(): LEVELS;
+ setLevel(level: LEVELS): void;
+ internal(level: LEVELS, message: string, data: any, exception: any): void;
+ setOptions(options: Partial<LoggerOptions>): void;
+ getOptions(): LoggerOptions;
+ child(options: Partial<LoggerOptions>): Log;
+ error(message: string, data?: any): void;
+ exception(exception: Error, message?: string, data?: any): void;
+ softFail(message: string, data?: any): void;
+ warning(message: string, data?: any): void;
+ info(message: string, data?: any): void;
+ debug(message: string, data?: any): void;
+ perf(message: string, data?: any): void;
+ deprecated(message: string): void;
+ }
- // Sets log level
- function setLevel(level: number): void;
+ export const LoggerText: any;
+ export const LoggerJson: any;
- function getLevel(): number;
+ /**
+ * Map of available log levels that's useful for easy setting of appropriate log levels.
+ * Each log level is represented internally by a number. Eg. `log.LEVELS.DEBUG === 5`.
+ */
+ export enum LEVELS {
+ // Turns off logging completely
+ OFF = 0,
+ // For unexpected errors in Apify system
+ ERROR = 1,
+ // For situations where error is caused by user (e.g. Meteor.Error), i.e. when the error is not
+ // caused by Apify system, avoid the word "ERROR" to simplify searching in log
+ SOFT_FAIL = 2,
+ WARNING = 3,
+ INFO = 4,
+ DEBUG = 5,
+ // for performance stats
+ PERF = 6
+ }
- // Helper functions for common usage
- function warning(message: string, data?: any): void;
+ // The shape of the Log class can't be easily typed in TS because it
+ // dynamically attaches values to the log instance, so we need to document
+ // only the "inner" functions of the Log class, and not the class itself
+ // That's why the export consts below:
- function info(message: string, data?: any): void;
-
- function debug(message: string, data?: any): void;
-
- function error(message: string, data?: any): void;
-
- function exception(exception: Error, message?: string, data?: any): void;
+ /**
+ * Returns the currently selected logging level. This is useful for checking whether a message
+ * will actually be printed to the console before one actually performs a resource intensive operation
+ * to construct the message, such as querying a DB for some metadata that need to be added. If the log
+ * level is not high enough at the moment, it doesn't make sense to execute the query.
+ */
+ export const getLevel: Log['getLevel'];
+ /**
+ * Sets the log level to the given value, preventing messages from less important log levels
+ * from being printed to the console. Use in conjunction with the `log.LEVELS` constants such as
+ *
+ * ```
+ * log.setLevel(log.LEVELS.DEBUG);
+ * ```
+ *
+ * Default log level is INFO.
+ */
+ export const setLevel: Log['setLevel'];
+ /**
+ * Configures logger.
+ */
+ export const setOptions: Log['setOptions'];
+ /**
+ * Returns the logger configuration.
+ */
+ export const getOptions: Log['getOptions'];
+ /**
+ * Creates a new instance of logger that inherits settings from a parent logger.
+ */
+ export const child: Log['child'];
+ /**
+ * Logs an `ERROR` message. Use this method to log error messages that are not directly connected
+ * to an exception. For logging exceptions, use the `log.exception` method.
+ */
+ export const error: Log['error'];
+ /**
+ * Logs an `ERROR` level message with a nicely formatted exception. Note that the exception is the first parameter
+ * here and an additional message is only optional.
+ */
+ export const exception: Log['exception'];
+ /**
+ * Logs a `WARNING` level message. Data are stringified and appended to the message.
+ */
+ export const warning: Log['warning'];
+ /**
+ * Logs an `INFO` message. `INFO` is the default log level so info messages will be always logged,
+ * unless the log level is changed. Data are stringified and appended to the message.
+ */
+ export const info: Log['info'];
+ /**
+ * Logs a `DEBUG` message. By default, it will not be written to the console. To see `DEBUG`
+ * messages in the console, set the log level to `DEBUG` either using the `log.setLevel(log.LEVELS.DEBUG)`
+ * method or using the environment variable `APIFY_LOG_LEVEL=DEBUG`. Data are stringified and appended
+ * to the message.
+ */
+ export const debug: Log['debug'];
+ /**
+ * Logs given message only once as WARNING. It's used to warn user that some feature he is using
+ * has been deprecated.
+ */
+ export const deprecated: Log['deprecated'];
}
-export {}
diff --git a/types-apify/apify-shared/lru_cache.d.ts b/types-apify/apify-shared/lru_cache.d.ts
new file mode 100644
index 000000000..924b3322a
--- /dev/null
+++ b/types-apify/apify-shared/lru_cache.d.ts
@@ -0,0 +1,4 @@
+declare module 'apify-shared/lru_cache' {
+ const LruCache: any;
+ export = LruCache;
+}
diff --git a/types-apify/apify-shared/regexs.d.ts b/types-apify/apify-shared/regexs.d.ts
new file mode 100644
index 000000000..12393d056
--- /dev/null
+++ b/types-apify/apify-shared/regexs.d.ts
@@ -0,0 +1,4 @@
+declare module 'apify-shared/regexs' {
+ export const APIFY_PROXY_VALUE_REGEX: RegExp;
+ export const KEY_VALUE_STORE_KEY_REGEX: RegExp;
+}
diff --git a/types-apify/apify-shared/streams_utilities.d.ts b/types-apify/apify-shared/streams_utilities.d.ts
new file mode 100644
index 000000000..98ca50890
--- /dev/null
+++ b/types-apify/apify-shared/streams_utilities.d.ts
@@ -0,0 +1,4 @@
+declare module 'apify-shared/streams_utilities' {
+ export function readStreamToString(response: any, encoding: any): any;
+ export function concatStreamToBuffer(response: any): any;
+}
diff --git a/types-apify/apify-shared/utilities.d.ts b/types-apify/apify-shared/utilities.d.ts
new file mode 100644
index 000000000..c1e4ce0cb
--- /dev/null
+++ b/types-apify/apify-shared/utilities.d.ts
@@ -0,0 +1,11 @@
+declare module 'apify-shared/utilities' {
+ export function checkParamPrototypeOrThrow(...args: any): any;
+ export function betterSetInterval(f: Function, timeout: number): any;
+ export function betterClearInterval(intervalId: any): any;
+ export function promisifyServerListen(server: any): (port: any) => Promise<any>;
+ export function cryptoRandomObjectId(len?: number): string;
+ export function leftpad(val: number, len: number, char: number): string;
+ export const jsonStringifyExtended: typeof JSON.stringify;
+ export function normalizeUrl(url: any, keep: boolean): any;
+ export function getRandomInt(from: number): number;
+}
diff --git a/types-apify/ps-tree.d.ts b/types-apify/ps-tree.d.ts
new file mode 100644
index 000000000..70dee81a3
--- /dev/null
+++ b/types-apify/ps-tree.d.ts
@@ -0,0 +1,4 @@
+declare module '@apify/ps-tree' {
+ const EXPORTED: (pid: any, includeRoot: any, callback: any) => void;
+ export = EXPORTED;
+}
diff --git a/types/actor.d.ts b/types/actor.d.ts
index b1e58ac9a..3fd116837 100644
--- a/types/actor.d.ts
+++ b/types/actor.d.ts
@@ -1,5 +1,5 @@
export function getEnv(): ApifyEnv;
-export function main(userFunc: Function): void;
+export function main(userFunc: UserFunc): void;
export function call(actId: string, input?: any, options?: {
contentType?: string;
token?: string;
@@ -10,7 +10,7 @@ export function call(actId: string, input?: any, options?: {
fetchOutput?: boolean;
disableBodyParser?: boolean;
webhooks?: any[];
-}): Promise<ActorRun>;
+} | undefined): Promise<ActorRun>;
export function callTask(taskId: string, input?: any, options?: {
contentType?: string;
token?: string;
@@ -19,17 +19,17 @@ export function callTask(taskId: string, input?: any, options?: {
build?: string;
waitSecs?: string;
webhooks?: any[];
-}): Promise<ActorRun>;
+} | undefined): Promise<ActorRun>;
export function metamorph(targetActorId: string, input?: any, options?: {
contentType?: string;
build?: string;
-}): Promise<void>;
+} | undefined): Promise<void>;
export function getApifyProxyUrl(options?: {
password?: string;
groups?: string[];
session?: string;
country?: string;
-}): string;
+} | undefined): string;
export function addWebhook({ eventTypes, requestUrl, payloadTemplate, idempotencyKey }: {
eventTypes: string[];
requestUrl: string;
@@ -43,47 +43,48 @@ export type ApifyEnv = {
/**
* ID of the actor (APIFY_ACTOR_ID)
*/
- actorId: string;
+ actorId: string | null;
/**
* ID of the actor run (APIFY_ACTOR_RUN_ID)
*/
- actorRunId: string;
+ actorRunId: string | null;
/**
* ID of the actor task (APIFY_ACTOR_TASK_ID)
*/
- actorTaskId: string;
+ actorTaskId: string | null;
/**
* ID of the user who started the actor - note that it might be
* different than the owner ofthe actor (APIFY_USER_ID)
*/
- userId: string;
+ userId: string | null;
/**
* Authentication token representing privileges given to the actor run,
* it can be passed to various Apify APIs (APIFY_TOKEN)
*/
- token: string;
+ token: string | null;
/**
* Date when the actor was started (APIFY_STARTED_AT)
*/
- startedAt: Date;
+ startedAt: Date | null;
/**
* Date when the actor will time out (APIFY_TIMEOUT_AT)
*/
- timeoutAt: Date;
+ timeoutAt: Date | null;
/**
* ID of the key-value store where input and output data of this
* actor is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)
*/
- defaultKeyValueStoreId: string;
+ defaultKeyValueStoreId: string | null;
/**
* ID of the dataset where input and output data of this
* actor is stored (APIFY_DEFAULT_DATASET_ID)
*/
- defaultDatasetId: string;
+ defaultDatasetId: string | null;
/**
* Amount of memory allocated for the actor,
* in megabytes (APIFY_MEMORY_MBYTES)
*/
- memoryMbytes: number;
+ memoryMbytes: number | null;
};
+export type UserFunc = () => Promise<void>;
import { ActorRun } from "./typedefs";
diff --git a/types/autoscaling/autoscaled_pool.d.ts b/types/autoscaling/autoscaled_pool.d.ts
index 87cc43943..f321bb6f2 100644
--- a/types/autoscaling/autoscaled_pool.d.ts
+++ b/types/autoscaling/autoscaled_pool.d.ts
@@ -80,7 +80,7 @@ export type AutoscaledPoolOptions = {
systemStatusOptions?: SystemStatusOptions;
};
/**
- * @typedef {Object} AutoscaledPoolOptions
+ * @typedef AutoscaledPoolOptions
* @property {Function} runTaskFunction
* A function that performs an asynchronous resource-intensive task.
* The function must either be labeled `async` or return a promise.
@@ -98,32 +98,32 @@ export type AutoscaledPoolOptions = {
* keeps resolving to `true`, `isFinishedFunction()` will never be called.
* To abort a run, use the {@link AutoscaledPool#abort} method.
*
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* The minimum number of tasks running in parallel.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* The maximum number of tasks running in parallel.
- * @property {Number} [desiredConcurrency]
+ * @property {number} [desiredConcurrency]
* The desired number of tasks that should be running parallel on the start of the pool,
* if there is a large enough supply of them.
* By default, it is `minConcurrency`.
- * @property {Number} [desiredConcurrencyRatio=0.95]
+ * @property {number} [desiredConcurrencyRatio=0.95]
* Minimum level of desired concurrency to reach before more scaling up is allowed.
- * @property {Number} [scaleUpStepRatio=0.05]
+ * @property {number} [scaleUpStepRatio=0.05]
* Defines the fractional amount of desired concurrency to be added with each scaling up.
* The minimum scaling step is one.
- * @property {Number} [scaleDownStepRatio=0.05]
+ * @property {number} [scaleDownStepRatio=0.05]
* Defines the amount of desired concurrency to be subtracted with each scaling down.
* The minimum scaling step is one.
- * @property {Number} [maybeRunIntervalSecs=0.5]
+ * @property {number} [maybeRunIntervalSecs=0.5]
* Indicates how often the pool should call the `runTaskFunction()` to start a new task, in seconds.
* This has no effect on starting new tasks immediately after a task completes.
- * @property {Number} [loggingIntervalSecs=60]
+ * @property {number} [loggingIntervalSecs=60]
* Specifies a period in which the instance logs its state, in seconds.
* Set to `null` to disable periodic logging.
- * @property {Number} [autoscaleIntervalSecs=10]
+ * @property {number} [autoscaleIntervalSecs=10]
* Defines in seconds how often the pool should attempt to adjust the desired concurrency
* based on the latest system status. Setting it lower than 1 might have a severe impact on performance.
* We suggest using a value from 5 to 20.
@@ -202,8 +202,8 @@ declare class AutoscaledPool {
_currentConcurrency: number;
isStopped: boolean;
lastLoggingTime: number;
- resolve: (value?: any) => void;
- reject: (reason?: any) => void;
+ resolve: ((value?: any) => void) | null;
+ reject: ((reason?: any) => void) | null;
/**
* Gets called every autoScaleIntervalSecs and evaluates the current system status.
* If the system IS NOT overloaded and the settings allow it, it scales up.
@@ -284,10 +284,10 @@ declare class AutoscaledPool {
* Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once
* all the tasks are finished or one of them fails.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
- run(): Promise<any>;
- poolPromise: Promise<any>;
+ run(): Promise<void>;
+ poolPromise: Promise<any> | undefined;
autoscaleInterval: any;
maybeRunInterval: any;
/**
@@ -301,9 +301,9 @@ declare class AutoscaledPool {
* their state after the invocation of `.abort()`, but that does not mean that some
* parts of their asynchronous chains of commands will not execute.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
- abort(): Promise<any>;
+ abort(): Promise<void>;
/**
* Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish
* (unlike abort, which terminates them). Used together with {@link AutoscaledPool#resume}
@@ -316,9 +316,9 @@ declare class AutoscaledPool {
* when `.pause()` is invoked (unlike abort, which resolves it).
*
* @param {number} [timeoutSecs]
- * @return {Promise}
+ * @return {Promise<void>}
*/
- pause(timeoutSecs?: number): Promise<any>;
+ pause(timeoutSecs?: number | undefined): Promise<void>;
/**
* Resumes the operation of the autoscaled-pool by allowing more tasks to be run.
* Used together with {@link AutoscaledPool#pause}
@@ -326,7 +326,7 @@ declare class AutoscaledPool {
* Tasks will automatically start running again in `options.maybeRunIntervalSecs`.
*/
resume(): void;
- queryingIsTaskReady: boolean;
+ queryingIsTaskReady: boolean | undefined;
/**
* Scales the pool up by increasing
* the desired concurrency by the scaleUpStepRatio.
@@ -334,7 +334,7 @@ declare class AutoscaledPool {
* @param {Object} systemStatus for logging
* @ignore
*/
- _scaleUp(systemStatus: any): void;
+ _scaleUp(systemStatus: Object): void;
/**
* Scales the pool down by decreasing
* the desired concurrency by the scaleDownStepRatio.
@@ -342,7 +342,7 @@ declare class AutoscaledPool {
* @param {Object} systemStatus for logging
* @ignore
*/
- _scaleDown(systemStatus: any): void;
+ _scaleDown(systemStatus: Object): void;
/**
* If there are no running tasks and this.isFinishedFunction() returns true then closes
* the pool and resolves the pool's promise returned by the run() method.
@@ -352,7 +352,7 @@ declare class AutoscaledPool {
* @ignore
*/
_maybeFinish(): Promise<void>;
- queryingIsFinished: boolean;
+ queryingIsFinished: boolean | undefined;
/**
* Cleans up resources.
*
diff --git a/types/autoscaling/snapshotter.d.ts b/types/autoscaling/snapshotter.d.ts
index 66967ce08..378adcc9c 100644
--- a/types/autoscaling/snapshotter.d.ts
+++ b/types/autoscaling/snapshotter.d.ts
@@ -51,34 +51,34 @@ export type SnapshotterOptions = {
};
/**
* @typedef SnapshotterOptions
- * @property {Number} [eventLoopSnapshotIntervalSecs=0.5]
+ * @property {number} [eventLoopSnapshotIntervalSecs=0.5]
* Defines the interval of measuring the event loop response time.
- * @property {Number} [clientSnapshotIntervalSecs=1]
+ * @property {number} [clientSnapshotIntervalSecs=1]
* Defines the interval of checking the current state
* of the remote API client.
- * @property {Number} [maxBlockedMillis=50]
+ * @property {number} [maxBlockedMillis=50]
* Maximum allowed delay of the event loop in milliseconds.
* Exceeding this limit overloads the event loop.
- * @property {Number} [cpuSnapshotIntervalSecs=1]
+ * @property {number} [cpuSnapshotIntervalSecs=1]
* Defines the interval of measuring CPU usage.
* This is only used when running locally. On the Apify platform,
* the statistics are provided externally at a fixed interval.
- * @property {Number} [maxUsedCpuRatio=0.95]
+ * @property {number} [maxUsedCpuRatio=0.95]
* Defines the maximum usage of CPU.
* Exceeding this limit overloads the CPU.
- * @property {Number} [memorySnapshotIntervalSecs=1]
+ * @property {number} [memorySnapshotIntervalSecs=1]
* Defines the interval of measuring memory consumption.
* This is only used when running locally. On the Apify platform,
* the statistics are provided externally at a fixed interval.
* The measurement itself is resource intensive (25 - 50ms async).
* Therefore, setting this interval below 1 second is not recommended.
- * @property {Number} [maxUsedMemoryRatio=0.7]
+ * @property {number} [maxUsedMemoryRatio=0.7]
* Defines the maximum ratio of total memory that can be used.
* Exceeding this limit overloads the memory.
- * @property {Number} [maxClientErrors=1]
+ * @property {number} [maxClientErrors=1]
* Defines the maximum number of new rate limit errors within
* the given interval.
- * @property {Number} [snapshotHistorySecs=60]
+ * @property {number} [snapshotHistorySecs=60]
* Sets the interval in seconds for which a history of resource snapshots
* will be kept. Increasing this to very high numbers will affect performance.
*/
@@ -110,7 +110,7 @@ declare class Snapshotter {
/**
* @param {SnapshotterOptions} [options] All `Snapshotter` configuration options.
*/
- constructor(options?: SnapshotterOptions);
+ constructor(options?: SnapshotterOptions | undefined);
eventLoopSnapshotIntervalMillis: number;
memorySnapshotIntervalMillis: number;
clientSnapshotIntervalMillis: number;
@@ -120,7 +120,7 @@ declare class Snapshotter {
maxUsedMemoryRatio: any;
maxUsedCpuRatio: any;
maxClientErrors: any;
- maxMemoryBytes: number;
+ maxMemoryBytes: number | null;
cpuSnapshots: any[];
eventLoopSnapshots: any[];
memorySnapshots: any[];
@@ -129,79 +129,79 @@ declare class Snapshotter {
memoryInterval: any;
clientInterval: any;
cpuInterval: any;
- lastLoggedCriticalMemoryOverloadAt: Date;
+ lastLoggedCriticalMemoryOverloadAt: Date | null;
/**
* Creates a snapshot of current CPU usage
* using the Apify platform `systemInfo` event.
* @param {Object} systemInfo
* @ignore
*/
- _snapshotCpuOnPlatform(systemInfo: any): void;
+ _snapshotCpuOnPlatform(systemInfo: Object): void;
/**
* Creates a snapshot of current memory usage
* using the Apify platform `systemInfo` event.
- * @param {Object} systemInfo
+ * @param {*} systemInfo
* @ignore
*/
_snapshotMemoryOnPlatform(systemInfo: any): void;
/**
* Starts capturing snapshots at configured intervals.
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async start(): Promise<any>;
+ start(): Promise<void>;
/**
* Stops all resource capturing.
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async stop(): Promise<any>;
+ stop(): Promise<void>;
/**
* Returns a sample of latest memory snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
*/
- getMemorySample(sampleDurationMillis?: number): any[];
+ getMemorySample(sampleDurationMillis?: number | undefined): any[];
/**
* Returns a sample of latest event loop snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
*/
- getEventLoopSample(sampleDurationMillis?: number): any[];
+ getEventLoopSample(sampleDurationMillis?: number | undefined): any[];
/**
* Returns a sample of latest CPU snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
*/
- getCpuSample(sampleDurationMillis?: number): any[];
+ getCpuSample(sampleDurationMillis?: number | undefined): any[];
/**
* Returns a sample of latest Client snapshots, with the size of the sample defined
* by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
- * @param {Number} sampleDurationMillis
- * @return {Array}
+ * @param {number} sampleDurationMillis
+ * @return {Array<*>}
*/
getClientSample(sampleDurationMillis: number): any[];
/**
* Finds the latest snapshots by sampleDurationMillis in the provided array.
- * @param {Array} snapshots
- * @param {Number} [sampleDurationMillis]
- * @return {Array}
+ * @param {Array<*>} snapshots
+ * @param {number} [sampleDurationMillis]
+ * @return {Array<*>}
* @ignore
*/
- _getSample(snapshots: any[], sampleDurationMillis?: number): any[];
+ _getSample(snapshots: any[], sampleDurationMillis?: number | undefined): any[];
/**
* Creates a snapshot of current memory usage
* using the Apify platform `systemInfo` event.
* @param {Function} intervalCallback
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
- async _snapshotMemoryOnLocal(intervalCallback: Function): Promise<any>;
+ _snapshotMemoryOnLocal(intervalCallback: Function): Promise<void>;
/**
* Checks for critical memory overload and logs it to the console.
+ * @param {*} systemInfo
* @ignore
- * @param {Object} systemInfo
*/
_memoryOverloadWarning({ memCurrentBytes }: any): void;
/**
@@ -232,11 +232,11 @@ declare class Snapshotter {
* @param intervalCallback
* @private
*/
- private _snapshotClient;
+ _snapshotClient(intervalCallback: any): void;
/**
* Removes snapshots that are older than the snapshotHistorySecs option
* from the array (destructively - in place).
- * @param {Array} snapshots
+ * @param {Array<*>} snapshots
* @param {Date} now
* @ignore
*/
@@ -245,5 +245,5 @@ declare class Snapshotter {
* Calculate max memory for platform or local usage.
* @ignore
*/
- async _ensureCorrectMaxMemory(): Promise<void>;
+ _ensureCorrectMaxMemory(): Promise<void>;
}
diff --git a/types/autoscaling/system_status.d.ts b/types/autoscaling/system_status.d.ts
index f61333845..01bdd488e 100644
--- a/types/autoscaling/system_status.d.ts
+++ b/types/autoscaling/system_status.d.ts
@@ -48,29 +48,29 @@ export type SystemStatusOptions = {
/**
* Represents the current status of the system.
*
- * @typedef {Object} SystemInfo
- * @property {Boolean} isSystemIdle
+ * @typedef SystemInfo
+ * @property {boolean} isSystemIdle
* If true, system is being overloaded.
- * @property {Object} memInfo
+ * @property {object} memInfo
* Memory
- * @property {Object} eventLoopInfo
- * @property {Object} cpuInfo
+ * @property {object} eventLoopInfo
+ * @property {object} cpuInfo
*/
/**
- * @typedef {Object} SystemStatusOptions
- * @property {Number} [currentHistorySecs=5]
+ * @typedef SystemStatusOptions
+ * @property {number} [currentHistorySecs=5]
* Defines max age of snapshots used in the
* {@link SystemStatus#getCurrentStatus} measurement.
- * @property {Number} [maxMemoryOverloadedRatio=0.2]
+ * @property {number} [maxMemoryOverloadedRatio=0.2]
* Sets the maximum ratio of overloaded snapshots in a memory sample.
* If the sample exceeds this ratio, the system will be overloaded.
- * @property {Number} [maxEventLoopOverloadedRatio=0.2]
+ * @property {number} [maxEventLoopOverloadedRatio=0.2]
* Sets the maximum ratio of overloaded snapshots in an event loop sample.
* If the sample exceeds this ratio, the system will be overloaded.
- * @property {Number} [maxCpuOverloadedRatio=0.4]
+ * @property {number} [maxCpuOverloadedRatio=0.4]
* Sets the maximum ratio of overloaded snapshots in a CPU sample.
* If the sample exceeds this ratio, the system will be overloaded.
- * @property {Number} [maxClientOverloadedRatio=0.2]
+ * @property {number} [maxClientOverloadedRatio=0.2]
* Sets the maximum ratio of overloaded snapshots in a Client sample.
* If the sample exceeds this ratio, the system will be overloaded.
* @property {Snapshotter} [snapshotter]
@@ -101,7 +101,7 @@ declare class SystemStatus {
/**
* @param {SystemStatusOptions} [options] All `SystemStatus` configuration options.
*/
- constructor(options?: SystemStatusOptions);
+ constructor(options?: SystemStatusOptions | undefined);
currentHistorySecs: number;
maxMemoryOverloadedRatio: any;
maxEventLoopOverloadedRatio: any;
@@ -141,23 +141,23 @@ declare class SystemStatus {
* Where the `isSystemIdle` property is set to `false` if the system
* has been overloaded in the full history of the {@link Snapshotter}
* (which is configurable in the {@link Snapshotter}) and `true` otherwise.
- * @return {Object}
+ * @return {SystemInfo}
*/
- getHistoricalStatus(): any;
+ getHistoricalStatus(): SystemInfo;
/**
* Returns a system status object.
*
- * @param {Number} [sampleDurationMillis]
+ * @param {number} [sampleDurationMillis]
* @return {SystemInfo}
* @ignore
*/
- _isSystemIdle(sampleDurationMillis?: number): SystemInfo;
+ _isSystemIdle(sampleDurationMillis?: number | undefined): SystemInfo;
/**
* Returns an object with an isOverloaded property set to true
* if the memory has been overloaded in the last sampleDurationMillis.
*
- * @param {Number} sampleDurationMillis
- * @return {Object}
+ * @param {number} sampleDurationMillis
+ * @return {object}
* @ignore
*/
_isMemoryOverloaded(sampleDurationMillis: number): any;
@@ -165,8 +165,8 @@ declare class SystemStatus {
* Returns an object with an isOverloaded property set to true
* if the event loop has been overloaded in the last sampleDurationMillis.
*
- * @param {Number} sampleDurationMillis
- * @return {Object}
+ * @param {number} sampleDurationMillis
+ * @return {object}
* @ignore
*/
_isEventLoopOverloaded(sampleDurationMillis: number): any;
@@ -174,26 +174,30 @@ declare class SystemStatus {
* Returns an object with an isOverloaded property set to true
* if the CPU has been overloaded in the last sampleDurationMillis.
*
- * @param {Number} sampleDurationMillis
- * @return {Object}
+ * @param {number} sampleDurationMillis
+ * @return {object}
* @ignore
*/
_isCpuOverloaded(sampleDurationMillis: number): any;
/**
* Returns an object with an isOverloaded property set to true
* if the client has been overloaded in the last sampleDurationMillis.
- * @param sampleDurationMillis
- * @return {{isOverloaded, maxOverloadedRatio, actualRatio}}
+ * @param {number} sampleDurationMillis
+ * @return {{isOverloaded: boolean, maxOverloadedRatio: number, actualRatio: number}}
* @private
*/
- private _isClientOverloaded;
+ _isClientOverloaded(sampleDurationMillis: number): {
+ isOverloaded: boolean;
+ maxOverloadedRatio: number;
+ actualRatio: number;
+ };
/**
* Returns an object with sample information and an isOverloaded property
* set to true if at least the ratio of snapshots in the sample are overloaded.
*
- * @param {Array} sample
- * @param {Number} ratio
- * @return {Object}
+ * @param {Array<*>} sample
+ * @param {number} ratio
+ * @return {object}
* @ignore
*/
_isSampleOverloaded(sample: any[], ratio: number): any;
diff --git a/types/constants.d.ts b/types/constants.d.ts
index 23c4e1255..f1ba61070 100644
--- a/types/constants.d.ts
+++ b/types/constants.d.ts
@@ -11,14 +11,13 @@ export namespace EXIT_CODES {
/**
* These events are just internal for Apify package, so we don't need them in apify-shared package.
*
- * @type {{CPU_INFO: String, SYSTEM_INFO: String, MIGRATING: String, PERSIST_STATE: String}}
- *
+ * @type {{CPU_INFO: string, SYSTEM_INFO: string, MIGRATING: string, PERSIST_STATE: string}}
*/
export const ACTOR_EVENT_NAMES_EX: {
- CPU_INFO: String;
- SYSTEM_INFO: String;
- MIGRATING: String;
- PERSIST_STATE: String;
+ CPU_INFO: string;
+ SYSTEM_INFO: string;
+ MIGRATING: string;
+ PERSIST_STATE: string;
};
/**
* Most common user agents from https://techblog.willshouse.com/2012/01/03/most-common-user-agents/
diff --git a/types/crawlers/basic_crawler.d.ts b/types/crawlers/basic_crawler.d.ts
index ca9214c80..c8fe99012 100644
--- a/types/crawlers/basic_crawler.d.ts
+++ b/types/crawlers/basic_crawler.d.ts
@@ -122,7 +122,7 @@ export type HandleFailedRequestInput = {
error: Error;
};
/**
- * @typedef {Object} BasicCrawlerOptions
+ * @typedef BasicCrawlerOptions
* @property {HandleRequest} handleRequestFunction
* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
*
@@ -169,9 +169,9 @@ export type HandleFailedRequestInput = {
* See
* [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)
* for the default implementation of this function.
- * @property {Number} [maxRequestRetries=3]
+ * @property {number} [maxRequestRetries=3]
* Indicates how many times the request is retried if {@link BasicCrawlerOptions.handleRequestFunction} fails.
- * @property {Number} [maxRequestsPerCrawl]
+ * @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -180,14 +180,14 @@ export type HandleFailedRequestInput = {
* Note that the `runTaskFunction` and `isTaskReadyFunction` options
* are provided by `BasicCrawler` and cannot be overridden.
* However, you can provide a custom implementation of `isFinishedFunction`.
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @property {Boolean} [useSessionPool=false]
+ * @property {boolean} [useSessionPool=false]
* If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
* The session instance will be than available in the `handleRequestFunction`.
* @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {SessionPool} to use.
@@ -249,6 +249,13 @@ export type HandleFailedRequestInput = {
*
* await crawler.run();
* ```
+ *
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
declare class BasicCrawler {
/**
@@ -267,51 +274,51 @@ declare class BasicCrawler {
sessionPoolOptions: any;
useSessionPool: any;
autoscaledPoolOptions: any;
- isRunningPromise: Promise<any>;
+ isRunningPromise: Promise<void> | null;
/**
* Runs the crawler. Returns a promise that gets resolved once all the requests are processed.
*
* @return {Promise<void>}
*/
- async run(): Promise<void>;
- autoscaledPool: AutoscaledPool;
- sessionPool: import("../session_pool/session_pool").SessionPool;
- async _pauseOnMigration(): Promise<void>;
+ run(): Promise<void>;
+ autoscaledPool: AutoscaledPool | undefined;
+ sessionPool: import("../session_pool/session_pool").SessionPool | undefined;
+ _pauseOnMigration(): Promise<void>;
/**
* Fetches request from either RequestList or RequestQueue. If request comes from a RequestList
* and RequestQueue is present then enqueues it to the queue first.
*
* @ignore
*/
- async _fetchNextRequest(): Promise<any>;
+ _fetchNextRequest(): Promise<any>;
/**
* Wrapper around handleRequestFunction that fetches requests from RequestList/RequestQueue
* then retries them in a case of an error, etc.
*
* @ignore
*/
- async _runTaskFunction(): Promise<void>;
+ _runTaskFunction(): Promise<void>;
/**
* Returns true if either RequestList or RequestQueue have a request ready for processing.
*
* @ignore
*/
- async _isTaskReadyFunction(): Promise<boolean>;
+ _isTaskReadyFunction(): Promise<boolean>;
/**
* Returns true if both RequestList and RequestQueue have all requests finished.
*
* @ignore
*/
- async _defaultIsFinishedFunction(): Promise<any>;
+ _defaultIsFinishedFunction(): Promise<any>;
/**
* Handles errors thrown by user provided handleRequestFunction()
* @param {Error} error
* @param {Request} request
- * @param {RequestList|RequestQueue} source
- * @return {Promise<Boolean>} willBeRetried
+ * @param {(RequestList|RequestQueue)} source
+ * @return {Promise<boolean|void|QueueOperationInfo>} willBeRetried
* @ignore
*/
- async _requestFunctionErrorHandler(error: Error, request: Request, source: RequestList | RequestQueue): Promise<boolean>;
+ _requestFunctionErrorHandler(error: Error, request: Request, source: RequestList | RequestQueue): Promise<boolean | void | QueueOperationInfo>;
/**
* Updates handledRequestsCount from possibly stored counts,
* usually after worker migration. Since one of the stores
@@ -323,7 +330,7 @@ declare class BasicCrawler {
* @return {Promise<void>}
* @ignore
*/
- async _loadHandledRequestCount(): Promise<void>;
+ _loadHandledRequestCount(): Promise<void>;
}
import { RequestList } from "../request_list";
import { RequestQueue } from "../request_queue";
@@ -333,3 +340,4 @@ import Request from "../request";
import AutoscaledPool from "../autoscaling/autoscaled_pool";
import { Session } from "../session_pool/session";
import Statistics from "./statistics";
+import { QueueOperationInfo } from "../request_queue";
diff --git a/types/crawlers/cheerio_crawler.d.ts b/types/crawlers/cheerio_crawler.d.ts
index 963483881..95f523ba2 100644
--- a/types/crawlers/cheerio_crawler.d.ts
+++ b/types/crawlers/cheerio_crawler.d.ts
@@ -1,3 +1,4 @@
+/// <reference types="node" />
export default CheerioCrawler;
export type CheerioCrawlerOptions = {
/**
@@ -222,13 +223,17 @@ export type PrepareRequestInputs = {
* Original instance fo the {Request} object. Must be modified in-place.
*/
request: Request;
+ /**
+ * The current session
+ */
+ session?: Session;
};
export type PrepareRequest = (inputs: PrepareRequestInputs) => void | Promise<void>;
export type CheerioHandlePageInputs = {
/**
* The [Cheerio](https://cheerio.js.org/) object with parsed HTML.
*/
- $?: CheerioStatic;
+ $?: CheerioSelector;
/**
* The request body of the web page.
*/
@@ -238,7 +243,7 @@ export type CheerioHandlePageInputs = {
*/
json?: any;
/**
- * The original {Request} object.
+ * The original {@link Request} object.
*/
request: Request;
/**
@@ -260,11 +265,11 @@ export type CheerioHandlePageInputs = {
* or to abort it by calling {@link AutoscaledPool#abort}.
*/
autoscaledPool: AutoscaledPool;
- session?: any;
+ session?: Session;
};
export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<void>;
/**
- * @typedef {Object} CheerioCrawlerOptions
+ * @typedef CheerioCrawlerOptions
* @property {CheerioHandlePage} handlePageFunction
* User-provided function that performs the logic of the crawler. It is called for each page
* loaded and parsed by the crawler.
@@ -365,27 +370,27 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi
* this function is therefore not supported, because it would create inconsistencies where
* different parts of SDK would have access to a different {@link Request} instance.
*
- * @property {Number} [handlePageTimeoutSecs=60]
+ * @property {number} [handlePageTimeoutSecs=60]
* Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
- * @property {Number} [requestTimeoutSecs=30]
+ * @property {number} [requestTimeoutSecs=30]
* Timeout in which the HTTP request to the resource needs to finish, given in seconds.
- * @property {Boolean} [ignoreSslErrors=true]
+ * @property {boolean} [ignoreSslErrors=true]
* If set to true, SSL certificate errors will be ignored.
- * @property {Boolean} [useApifyProxy=false]
+ * @property {boolean} [useApifyProxy=false]
* If set to `true`, `CheerioCrawler` will be configured to use
* [Apify Proxy](https://my.apify.com/proxy) for all connections.
* For more information, see the [documentation](https://docs.apify.com/proxy)
- * @property {String[]} [apifyProxyGroups]
+ * @property {string[]} [apifyProxyGroups]
* An array of proxy groups to be used
* by the [Apify Proxy](https://docs.apify.com/proxy).
* Only applied if the `useApifyProxy` option is `true`.
- * @property {String} [apifyProxySession]
+ * @property {string} [apifyProxySession]
* Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.
* All HTTP requests going through the proxy with the same session identifier
* will use the same target proxy server (i.e. the same IP address).
* The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.
* Only applied if the `useApifyProxy` option is `true`.
- * @property {String[]} [proxyUrls]
+ * @property {string[]} [proxyUrls]
* An array of custom proxy URLs to be used by the `CheerioCrawler` instance.
* The provided custom proxies' order will be randomized and the resulting list rotated.
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
@@ -405,13 +410,13 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi
*
* See [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13)
* for the default implementation of this function.
- * @property {String[]} [additionalMimeTypes]
+ * @property {string[]} [additionalMimeTypes]
* An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
* target="_blank">MIME types</a> you want the crawler to load and process.
* By default, only `text/html` and `application/xhtml+xml` MIME types are supported.
- * @property {Number} [maxRequestRetries=3]
+ * @property {number} [maxRequestRetries=3]
* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.
- * @property {Number} [maxRequestsPerCrawl]
+ * @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
@@ -421,19 +426,19 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi
* are provided by `CheerioCrawler` and cannot be overridden. Reasonable {@link Snapshotter}
* and {@link SystemStatus} defaults are provided to account for the fact that `cheerio`
* parses HTML synchronously and therefore blocks the event loop.
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
- * @property {Boolean} [useSessionPool=false]
+ * @property {boolean} [useSessionPool=false]
* If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
* It also marks Session as bad after a request timeout.
* @property {SessionPoolOptions} [sessionPoolOptions]
* Custom options passed to the underlying {@link SessionPool} constructor.
- * @property {Boolean} [persistCookiesPerSession]
+ * @property {boolean} [persistCookiesPerSession]
* Automatically saves cookies to Session. Works only if Session Pool is used.
*
* It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.
@@ -515,6 +520,12 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
declare class CheerioCrawler {
/**
@@ -531,19 +542,20 @@ declare class CheerioCrawler {
useApifyProxy: boolean;
apifyProxyGroups: string[];
apifyProxySession: string;
- proxyUrls: any;
+ proxyUrls: any[];
lastUsedProxyUrlIndex: number;
prepareRequestFunction: PrepareRequest;
persistCookiesPerSession: boolean;
useSessionPool: boolean;
+ /** @ignore */
basicCrawler: any;
isRunningPromise: any;
/**
* Runs the crawler. Returns promise that gets resolved once all the requests got processed.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async run(): Promise<any>;
+ run(): Promise<void>;
autoscaledPool: any;
/**
* Wrapper around handlePageFunction that opens and closes pages etc.
@@ -554,11 +566,11 @@ declare class CheerioCrawler {
* @param {Session} options.session
* @ignore
*/
- async _handleRequestFunction({ request, autoscaledPool, session }: {
+ _handleRequestFunction({ request, autoscaledPool, session }: {
request: Request;
autoscaledPool: AutoscaledPool;
session: Session;
- }): Promise<void>;
+ }): Promise<any>;
/**
* Function to make the HTTP request. It performs optimizations
* on the request such as only downloading the request body if the
@@ -569,7 +581,7 @@ declare class CheerioCrawler {
* @param {Session} options.session
* @ignore
*/
- async _requestFunction({ request, session }: {
+ _requestFunction({ request, session }: {
request: Request;
session: Session;
}): Promise<{
@@ -594,15 +606,15 @@ declare class CheerioCrawler {
/**
* Combines the provided `requestOptions` with mandatory (non-overridable) values.
* @param {Request} request
- * @param {Session?} session
+ * @param {Session} [session]
* @ignore
*/
- _getRequestOptions(request: Request, session: Session): utilsRequest.RequestAsBrowserOptions & {
- url: any;
- method: any;
+ _getRequestOptions(request: Request, session?: Session | undefined): utilsRequest.RequestAsBrowserOptions & {
+ url: string;
+ method: string;
headers: any;
ignoreSslErrors: boolean;
- proxyUrl: string;
+ proxyUrl: string | null;
stream: boolean;
useCaseSensitiveHeaders: boolean;
abortFunction: (res: any) => boolean;
@@ -611,35 +623,35 @@ declare class CheerioCrawler {
/**
* Enables the use of a proxy by returning a proxy URL
* based on configured options or null if no proxy is used.
- * @param {Session?} session
- * @returns {string|null}
+ * @param {Session} [session]
+ * @returns {(string|null)}
* @ignore
*/
- _getProxyUrl(session?: Session): string;
+ _getProxyUrl(session?: Session | undefined): string | null;
_encodeResponse(request: any, response: any, encoding: any): {
response: any;
encoding: any;
};
- async _parseHtmlToDom(response: any): Promise<any>;
+ _parseHtmlToDom(response: any): Promise<any>;
/**
* Checks and extends supported mime types
- * @param {Array<String|Object>} additionalMimeTypes
+ * @param {Array<(string|Object)>} additionalMimeTypes
* @ignore
*/
- _extendSupportedMimeTypes(additionalMimeTypes: any[]): void;
+ _extendSupportedMimeTypes(additionalMimeTypes: (string | Object)[]): void;
/**
* Handles blocked request
- * @param session {Session}
- * @param statusCode {Number}
+ * @param {Session} session
+ * @param {number} statusCode
* @private
*/
- private _throwOnBlockedRequest;
+ _throwOnBlockedRequest(session: Session, statusCode: number): void;
/**
* Handles timeout request
- * @param session {Session}
+ * @param {Session} session
* @private
*/
- private _handleRequestTimeout;
+ _handleRequestTimeout(session: Session): void;
/**
* @param {Object} options
* @param {Error} options.error
@@ -647,7 +659,7 @@ declare class CheerioCrawler {
* @return {Promise<void>}
* @ignore
*/
- async _defaultHandleFailedRequestFunction({ error, request }: {
+ _defaultHandleFailedRequestFunction({ error, request }: {
error: Error;
request: Request;
}): Promise<void>;
@@ -659,6 +671,6 @@ import { HandleFailedRequest } from "./basic_crawler";
import { AutoscaledPoolOptions } from "../autoscaling/autoscaled_pool";
import { SessionPoolOptions } from "../session_pool/session_pool";
import Request from "../request";
+import { Session } from "../session_pool/session";
import { IncomingMessage } from "http";
import AutoscaledPool from "../autoscaling/autoscaled_pool";
-import { Session } from "../session_pool/session";
diff --git a/types/crawlers/puppeteer_crawler.d.ts b/types/crawlers/puppeteer_crawler.d.ts
index cb16b668e..b423c2e34 100644
--- a/types/crawlers/puppeteer_crawler.d.ts
+++ b/types/crawlers/puppeteer_crawler.d.ts
@@ -109,7 +109,7 @@ export type PuppeteerCrawlerOptions = {
* [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)
* for default behavior.
*/
- launchPuppeteerFunction?: Function;
+ launchPuppeteerFunction?: LaunchPuppeteerFunction;
/**
* Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.
* Shortcut to the corresponding {@link PuppeteerPool} option.
@@ -203,10 +203,10 @@ export type PuppeteerGotoInputs = {
*/
session?: Session;
};
-export type PuppeteerGoto = (inputs: PuppeteerGotoInputs) => Promise<PuppeteerResponse>;
+export type PuppeteerGoto = (inputs: PuppeteerGotoInputs) => Promise<PuppeteerResponse | null>;
export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browser>;
/**
- * @typedef {Object} PuppeteerCrawlerOptions
+ * @typedef PuppeteerCrawlerOptions
* @property {PuppeteerHandlePage} handlePageFunction
* Function that is called to process each request.
* It is passed an object with the following fields:
@@ -246,7 +246,7 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse
* @property {RequestQueue} [requestQueue]
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
* Either `requestList` or `requestQueue` option must be provided (or both).
- * @property {Number} [handlePageTimeoutSecs=60]
+ * @property {number} [handlePageTimeoutSecs=60]
* Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
* @property {PuppeteerGoto} [gotoFunction]
* Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's
@@ -261,7 +261,7 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse
* By default, the function invokes {@link puppeteer#gotoExtended} with a timeout of 60 seconds.
* For details, see source code on
* [GitHub](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L292).
- * @property {Number} [gotoTimeoutSecs=60]
+ * @property {number} [gotoTimeoutSecs=60]
* Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
* function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
* @property {HandleFailedRequest} [handleFailedRequestFunction]
@@ -280,16 +280,16 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse
* See
* [source code](https://github.com/apifytech/apify-js/blob/master/src/crawlers/puppeteer_crawler.js#L301)
* for the default implementation of this function.
- * @property {Number} [maxRequestRetries=3]
+ * @property {number} [maxRequestRetries=3]
* Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
- * @property {Number} [maxRequestsPerCrawl]
+ * @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
* @property {PuppeteerPoolOptions} [puppeteerPoolOptions]
* Custom options passed to the underlying {@link PuppeteerPool} constructor.
* You can tweak those to fine-tune browser management.
- * @property {Function} [launchPuppeteerFunction]
+ * @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* Shortcut to the corresponding {@link PuppeteerPool} option.
* See source code on
@@ -302,22 +302,22 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse
* Custom options passed to the underlying {@link AutoscaledPool} instance constructor.
* Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
* are provided by `PuppeteerCrawler` and should not be overridden.
- * @property {Number} [minConcurrency=1]
+ * @property {number} [minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the
* corresponding {@link AutoscaledPoolOptions.minConcurrency} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU,
* your crawler will run extremely slow or crash. If you're not sure, just keep the default value
* and the concurrency will scale up automatically.
- * @property {Number} [maxConcurrency=1000]
+ * @property {number} [maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the
* corresponding {@link AutoscaledPoolOptions.maxConcurrency} option.
- * @property {Boolean} [useSessionPool=false]
+ * @property {boolean} [useSessionPool=false]
* If set to true Crawler will automatically use Session Pool. It will automatically retire
* sessions on 403, 401 and 429 status codes. It also marks Session as bad after a request timeout.
* @property {SessionPoolOptions} [sessionPoolOptions]
* Custom options passed to the underlying {@link SessionPool} constructor.
- * @property {Boolean} [persistCookiesPerSession=false]
+ * @property {boolean} [persistCookiesPerSession=false]
* Automatically saves cookies to Session. Works only if Session Pool is used.
*/
/**
@@ -381,6 +381,13 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
+ *
*/
declare class PuppeteerCrawler {
/**
@@ -393,7 +400,7 @@ declare class PuppeteerCrawler {
handlePageTimeoutMillis: number;
gotoTimeoutMillis: number;
puppeteerPoolOptions: {
- launchPuppeteerFunction: Function;
+ launchPuppeteerFunction: LaunchPuppeteerFunction;
launchPuppeteerOptions: LaunchPuppeteerOptions;
/**
* Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
@@ -455,20 +462,21 @@ declare class PuppeteerCrawler {
*/
proxyUrls?: string[];
};
- puppeteerPool: PuppeteerPool;
+ puppeteerPool: PuppeteerPool | null;
useSessionPool: boolean;
sessionPoolOptions: SessionPoolOptions;
persistCookiesPerSession: boolean;
+ /** @ignore */
basicCrawler: BasicCrawler;
/**
* Runs the crawler. Returns promise that gets resolved once all the requests got processed.
*
* @return {Promise<void>}
*/
- async run(): Promise<void>;
- sessionPool: import("../session_pool/session_pool").SessionPool;
- isRunningPromise: Promise<void>;
- autoscaledPool: AutoscaledPool;
+ run(): Promise<void>;
+ sessionPool: import("../session_pool/session_pool").SessionPool | undefined;
+ isRunningPromise: Promise<void> | undefined;
+ autoscaledPool: AutoscaledPool | undefined;
/**
* Wrapper around handlePageFunction that opens and closes pages etc.
*
@@ -477,7 +485,7 @@ declare class PuppeteerCrawler {
* @param {AutoscaledPool} options.autoscaledPool
* @ignore
*/
- async _handleRequestFunction({ request, autoscaledPool }: {
+ _handleRequestFunction({ request, autoscaledPool }: {
request: Request;
autoscaledPool: AutoscaledPool;
}): Promise<void>;
@@ -488,7 +496,7 @@ declare class PuppeteerCrawler {
* @return {Promise<PuppeteerResponse>}
* @ignore
*/
- async _defaultGotoFunction({ page, request }: {
+ _defaultGotoFunction({ page, request }: {
page: PuppeteerPage;
request: Request;
}): Promise<PuppeteerResponse>;
@@ -499,7 +507,7 @@ declare class PuppeteerCrawler {
* @return {Promise<void>}
* @ignore
*/
- async _defaultHandleFailedRequestFunction({ error, request }: {
+ _defaultHandleFailedRequestFunction({ error, request }: {
error: Error;
request: Request;
}): Promise<void>;
@@ -508,6 +516,7 @@ import { RequestList } from "../request_list";
import { RequestQueue } from "../request_queue";
import { HandleFailedRequest } from "./basic_crawler";
import { PuppeteerPoolOptions } from "../puppeteer_pool";
+import { LaunchPuppeteerFunction } from "../puppeteer_pool";
import { LaunchPuppeteerOptions } from "../puppeteer";
import { AutoscaledPoolOptions } from "../autoscaling/autoscaled_pool";
import { SessionPoolOptions } from "../session_pool/session_pool";
diff --git a/types/crawlers/statistics.d.ts b/types/crawlers/statistics.d.ts
index 2fdf691a2..78ec18369 100644
--- a/types/crawlers/statistics.d.ts
+++ b/types/crawlers/statistics.d.ts
@@ -21,8 +21,8 @@ export default class Statistics {
minJobDurationMillis: number;
maxJobDurationMillis: number;
totalJobDurationMillis: number;
- startedAt: Date;
- logInterval: NodeJS.Timeout;
+ startedAt: Date | null;
+ logInterval: NodeJS.Timeout | null;
startJob(id: any): void;
finishJob(id: any): void;
failJob(id: any): void;
diff --git a/types/dataset.d.ts b/types/dataset.d.ts
index 048e0ac40..bf69b3675 100644
--- a/types/dataset.d.ts
+++ b/types/dataset.d.ts
@@ -1,8 +1,8 @@
export const DATASET_ITERATORS_DEFAULT_LIMIT: 10000;
-export const LOCAL_STORAGE_SUBDIR: any;
+export const LOCAL_STORAGE_SUBDIR: string;
export const LOCAL_FILENAME_DIGITS: 9;
export const LOCAL_GET_ITEMS_DEFAULT_LIMIT: 250000;
-export function checkAndSerialize(item: any, limitBytes: number, index?: number): string;
+export function checkAndSerialize(item: Object, limitBytes: number, index?: number | undefined): string;
export function chunkBySize(items: any[], limitBytes: number): any[];
/**
* The `Dataset` class represents a store for structured data where each object stored has the same attributes,
@@ -81,12 +81,11 @@ export class Dataset {
* the items have already been saved to the dataset while other items from the source array were not.
* To overcome this limitation, the developer may, for example, read the last item saved in the dataset
* and re-attempt the save of the data from this item onwards to prevent duplicates.
- *
- * @param {Object|Array} data Object or array of objects containing data to be stored in the default dataset.
+ * @param {object|Array<object>} data Object or array of objects containing data to be stored in the default dataset.
* The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
* @return {Promise<void>}
*/
- async pushData(data: any): Promise<void>;
+ pushData(data: any): Promise<void>;
/**
* Returns {DatasetContent} object holding the items in the dataset based on the provided parameters.
*
@@ -97,62 +96,62 @@ export class Dataset {
*
* @param {Object} [options] All `getData()` parameters are passed
* via an options object with the following keys:
- * @param {String} [options.format='json']
+ * @param {string} [options.format='json']
* Format of the `items` property, possible values are: `json`, `csv`, `xlsx`, `html`, `xml` and `rss`.
- * @param {Number} [options.offset=0]
+ * @param {number} [options.offset=0]
* Number of array elements that should be skipped at the start.
- * @param {Number} [options.limit=250000]
+ * @param {number} [options.limit=250000]
* Maximum number of array elements to return.
- * @param {Boolean} [options.desc=false]
+ * @param {boolean} [options.desc=false]
* If `true` then the objects are sorted by `createdAt` in descending order.
* Otherwise they are sorted in ascending order.
- * @param {Array} [options.fields]
+ * @param {string[]} [options.fields]
* An array of field names that will be included in the result. If omitted, all fields are included in the results.
- * @param {String} [options.unwind]
+ * @param {string} [options.unwind]
* Specifies a name of the field in the result objects that will be used to unwind the resulting objects.
* By default, the results are returned as they are.
- * @param {Boolean} [options.disableBodyParser=false]
+ * @param {boolean} [options.disableBodyParser=false]
* If `true` then response from API will not be parsed.
- * @param {Boolean} [options.attachment=false]
+ * @param {boolean} [options.attachment=false]
* If `true` then the response will define the `Content-Disposition: attachment` HTTP header, forcing a web
* browser to download the file rather than to display it. By default, this header is not present.
- * @param {String} [options.delimiter=',']
+ * @param {string} [options.delimiter=',']
* A delimiter character for CSV files, only used if `format` is `csv`.
- * @param {Boolean} [options.bom]
+ * @param {boolean} [options.bom]
* All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte
* Order Mark (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default
* behavior, set `bom` option to `true` to include the BOM, or set `bom` to `false` to skip it.
- * @param {String} [options.xmlRoot='results']
+ * @param {string} [options.xmlRoot='results']
* Overrides the default root element name of the XML output. By default, the root element is `results`.
- * @param {String} [options.xmlRow='page']
+ * @param {string} [options.xmlRow='page']
* Overrides the default element name that wraps each page or page function result object in XML output.
* By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
- * @param {Boolean} [options.skipHeaderRow=false]
+ * @param {boolean} [options.skipHeaderRow=false]
* If set to `true` then header row in CSV format is skipped.
- * @param {Boolean} [options.clean=false]
+ * @param {boolean} [options.clean=false]
* If `true` then the function returns only non-empty items and skips hidden fields (i.e. fields starting with `#` character).
* Note that the `clean` parameter is a shortcut for `skipHidden: true` and `skipEmpty: true` options.
- * @param {Boolean} [options.skipHidden=false]
+ * @param {boolean} [options.skipHidden=false]
* If `true` then the function doesn't return hidden fields (fields starting with "#" character).
- * @param {Boolean} [options.skipEmpty=false]
+ * @param {boolean} [options.skipEmpty=false]
* If `true` then the function doesn't return empty items.
* Note that in this case the returned number of items might be lower than limit parameter and pagination must be done using the `limit` value.
- * @param {Boolean} [options.simplified]
+ * @param {boolean} [options.simplified]
* If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and `unwind: 'pageFunctionResult'` options.
* This feature is used to emulate simplified results provided by Apify API version 1 used for
* the legacy Apify Crawler and it's not recommended to use it in new integrations.
- * @param {Boolean} [options.skipFailedPages]
+ * @param {boolean} [options.skipFailedPages]
* If `true` then, the all the items with errorInfo property will be skipped from the output.
* This feature is here to emulate functionality of Apify API version 1 used for
* the legacy Apify Crawler product and it's not recommended to use it in new integrations.
* @return {Promise<DatasetContent>}
*/
- async getData(options?: {
+ getData(options?: {
format?: string;
offset?: number;
limit?: number;
desc?: boolean;
- fields?: any[];
+ fields?: string[];
unwind?: string;
disableBodyParser?: boolean;
attachment?: boolean;
@@ -166,7 +165,7 @@ export class Dataset {
skipEmpty?: boolean;
simplified?: boolean;
skipFailedPages?: boolean;
- }): Promise<DatasetContent>;
+ } | undefined): Promise<DatasetContent>;
/**
* Returns an object containing general information about the dataset.
*
@@ -190,9 +189,9 @@ export class Dataset {
* }
* ```
*
- * @returns {Promise<Object>}
+ * @returns {Promise<object>}
*/
- async getInfo(): Promise<any>;
+ getInfo(): Promise<any>;
/**
* Iterates over dataset items, yielding each in turn to an `iteratee` function.
* Each invocation of `iteratee` is called with two arguments: `(item, index)`.
@@ -211,37 +210,36 @@ export class Dataset {
* @param {DatasetConsumer} iteratee A function that is called for every item in the dataset.
* @param {Object} [options] All `forEach()` parameters are passed
* via an options object with the following keys:
- * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by `createdAt` in descending order.
- * @param {Array} [options.fields] If provided then returned objects will only contain specified keys.
- * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @param {Number} [index=0] Specifies the initial index number passed to the `iteratee` function.
+ * @param {boolean} [options.desc=false] If `true` then the objects are sorted by `createdAt` in descending order.
+ * @param {string[]} [options.fields] If provided then returned objects will only contain specified keys.
+ * @param {string} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @param {number} [index=0] Specifies the initial index number passed to the `iteratee` function.
* @return {Promise<void>}
*/
- async forEach(iteratee: DatasetConsumer, options?: {
+ forEach(iteratee: DatasetConsumer, options?: {
desc?: boolean;
- fields?: any[];
+ fields?: string[];
unwind?: string;
- }, index?: number): Promise<void>;
+ } | undefined, index?: number | undefined): Promise<void>;
/**
* Produces a new array of values by mapping each value in list through a transformation function `iteratee()`.
* Each invocation of `iteratee()` is called with two arguments: `(element, index)`.
*
* If `iteratee` returns a `Promise` then it's awaited before a next call.
*
- * @template T
* @param {DatasetMapper} iteratee
- * @param {Object} options All `map()` parameters are passed
+ * @param {Object} [options] All `map()` parameters are passed
* via an options object with the following keys:
- * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
- * @param {Array} [options.fields] If provided then returned objects will only contain specified keys
- * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @return {Promise<T[]>}
+ * @param {boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
+ * @param {string[]} [options.fields] If provided then returned objects will only contain specified keys
+ * @param {string} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @return {Promise<Array<object>>}
*/
- map<T>(iteratee: DatasetMapper, options: {
+ map(iteratee: DatasetMapper, options?: {
desc?: boolean;
- fields?: any[];
+ fields?: string[];
unwind?: string;
- }): Promise<T[]>;
+ } | undefined): Promise<any[]>;
/**
* Reduces a list of values down to a single value.
*
@@ -253,30 +251,29 @@ export class Dataset {
*
* If `iteratee()` returns a `Promise` then it's awaited before a next call.
*
- * @template T
* @param {DatasetReducer} iteratee
- * @param {T} memo Initial state of the reduction.
- * @param {Object} options All `reduce()` parameters are passed
+ * @param {object} memo Initial state of the reduction.
+ * @param {Object} [options] All `reduce()` parameters are passed
* via an options object with the following keys:
- * @param {Boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
- * @param {Array} [options.fields] If provided then returned objects will only contain specified keys
- * @param {String} [options.unwind] If provided then objects will be unwound based on provided field.
- * @return {Promise<T>}
+ * @param {boolean} [options.desc=false] If `true` then the objects are sorted by createdAt in descending order.
+ * @param {string[]} [options.fields] If provided then returned objects will only contain specified keys
+ * @param {string} [options.unwind] If provided then objects will be unwound based on provided field.
+ * @return {Promise<object>}
*/
- reduce<T_1>(iteratee: DatasetReducer, memo: T_1, options: {
+ reduce(iteratee: DatasetReducer, memo: any, options?: {
desc?: boolean;
- fields?: any[];
+ fields?: string[];
unwind?: string;
- }): Promise<T_1>;
+ } | undefined): Promise<any>;
/**
* Removes the dataset either from the Apify cloud storage or from the local directory,
* depending on the mode of operation.
*
* @return {Promise<void>}
*/
- async drop(): Promise<void>;
+ drop(): Promise<void>;
/** @ignore */
- async delete(): Promise<void>;
+ delete(): Promise<void>;
}
/**
* This is a local emulation of a dataset.
@@ -286,7 +283,7 @@ export class Dataset {
export class DatasetLocal {
constructor(datasetId: any, localStorageDir: any);
localStoragePath: string;
- counter: number;
+ counter: number | null;
datasetId: any;
createdAt: any;
modifiedAt: any;
@@ -294,47 +291,47 @@ export class DatasetLocal {
initializationPromise: any;
_initialize(): any;
pushData(data: any): any;
- async getData(opts?: {}): Promise<{
+ getData(opts?: {}): Promise<{
items: any[];
- total: number;
+ total: number | null;
offset: any;
count: number;
limit: any;
}>;
- async getInfo(): Promise<{
+ getInfo(): Promise<{
id: any;
name: any;
- userId: string;
+ userId: string | null;
createdAt: any;
modifiedAt: any;
accessedAt: any;
- itemCount: number;
- cleanItemCount: number;
+ itemCount: number | null;
+ cleanItemCount: number | null;
}>;
- async forEach(iteratee: any): Promise<void>;
- async map(iteratee: any): Promise<any[]>;
- async reduce(iteratee: any, memo: any): Promise<any>;
- async drop(): Promise<void>;
- async delete(): Promise<void>;
+ forEach(iteratee: any): Promise<void>;
+ map(iteratee: any): Promise<any[]>;
+ reduce(iteratee: any, memo: any): Promise<any>;
+ drop(): Promise<void>;
+ delete(): Promise<void>;
/**
* Returns an array of item indexes for given offset and limit.
*/
- _getItemIndexes(offset?: number, limit?: number): any;
+ _getItemIndexes(offset?: number, limit?: number | null): number[];
/**
* Reads and parses file for given index.
*/
_readAndParseFile(index: any): Promise<any>;
_updateMetadata(isModified: any): void;
}
-export function openDataset(datasetIdOrName?: string, options?: {
+export function openDataset(datasetIdOrName?: string | undefined, options?: {
forceCloud?: boolean;
-}): Promise<Dataset>;
-export function pushData(item: any): Promise<any>;
+} | undefined): Promise<Dataset>;
+export function pushData(item: any): Promise<void>;
export type DatasetContent = {
/**
* Dataset entries based on chosen format parameter.
*/
- items: any[] | string[] | Buffer[];
+ items: any[];
/**
* Total count of entries in the dataset.
*/
diff --git a/types/enqueue_links/click_elements.d.ts b/types/enqueue_links/click_elements.d.ts
index f4addf78d..01cd2d711 100644
--- a/types/enqueue_links/click_elements.d.ts
+++ b/types/enqueue_links/click_elements.d.ts
@@ -42,7 +42,6 @@
* ],
* });
* ```
- *
* @param {Object} options
* All `enqueueLinksByClickingElements()` parameters are passed
* via an options object with the following keys:
@@ -50,10 +49,10 @@
* Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* @param {RequestQueue} options.requestQueue
* A request queue to which the URLs will be enqueued.
- * @param {String} options.selector
+ * @param {string} options.selector
* A CSS selector matching elements to be clicked on. Unlike in {@link utils#enqueueLinks}, there is no default
* value. This is to prevent suboptimal use of this function by using it too broadly.
- * @param {Array<String|RegExp|Object>} [options.pseudoUrls]
+ * @param {Array<(string|RegExp|Object)>} [options.pseudoUrls]
* An array of {@link PseudoUrl}s matching the URLs to be enqueued,
* or an array of strings or RegExps or plain Objects from which the {@link PseudoUrl}s can be constructed.
*
@@ -100,17 +99,17 @@
* This is the maximum period for which the function will keep tracking events, even if more events keep coming.
* Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself.
* See `waitForPageIdleSecs` above for an explanation.
- * @return {Promise<QueueOperationInfo[]>}
+ * @return {Promise<Array<QueueOperationInfo>>}
* Promise that resolves to an array of {@link QueueOperationInfo} objects.
* @memberOf puppeteer
* @name enqueueLinksByClickingElements
* @function
*/
export function enqueueLinksByClickingElements(options?: {
- page: any;
+ page: Page;
requestQueue: RequestQueue;
selector: string;
- pseudoUrls?: any[];
+ pseudoUrls?: (string | Object | RegExp)[];
transformRequestFunction?: RequestTransform;
waitForPageIdleSecs?: number;
maxWaitForPageIdleSecs?: number;
@@ -123,11 +122,11 @@ export function enqueueLinksByClickingElements(options?: {
* @param {Object} options
* @param {Page} options.page
* @param {string} options.selector
- * @return {Promise<Object[]>}
+ * @return {Promise<Array<object>>}
* @ignore
*/
export function clickElementsAndInterceptNavigationRequests(options: {
- page: any;
+ page: Page;
selector: string;
}): Promise<any[]>;
/**
@@ -137,7 +136,7 @@ export function clickElementsAndInterceptNavigationRequests(options: {
* @param {Target} target
* @return {boolean}
*/
-export function isTargetRelevant(page: any, target: any): boolean;
+export function isTargetRelevant(page: Page, target: Target): boolean;
/**
* Click all elements matching the given selector. To be able to do this using
* Puppeteer's `.click()` we need to make sure the elements are reachable by mouse,
@@ -147,10 +146,12 @@ export function isTargetRelevant(page: any, target: any): boolean;
*
* @param {Page} page
* @param {string} selector
- * @return {Promise}
+ * @return {Promise<void>}
* @ignore
*/
-export function clickElements(page: any, selector: string): Promise<any>;
+export function clickElements(page: Page, selector: string): Promise<void>;
+import { Page } from "puppeteer";
import { RequestQueue } from "../request_queue";
import { RequestTransform } from "./shared";
import { QueueOperationInfo } from "../request_queue";
+import { Target } from "puppeteer";
diff --git a/types/enqueue_links/enqueue_links.d.ts b/types/enqueue_links/enqueue_links.d.ts
index 9edfa7b50..7219d8008 100644
--- a/types/enqueue_links/enqueue_links.d.ts
+++ b/types/enqueue_links/enqueue_links.d.ts
@@ -32,20 +32,20 @@
* @param {Object} options
* All `enqueueLinks()` parameters are passed
* via an options object with the following keys:
- * @param {Page} options.page
+ * @param {Page} [options.page]
* Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.
* Either `page` or `$` option must be provided.
- * @param {Cheerio} options.$
+ * @param {CheerioStatic} [options.$]
* [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML.
* Either `page` or `$` option must be provided.
* @param {RequestQueue} options.requestQueue
* A request queue to which the URLs will be enqueued.
- * @param {String} [options.selector='a']
+ * @param {string} [options.selector='a']
* A CSS selector matching links to be enqueued.
* @param {string} [options.baseUrl]
* A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer,
* since the relative URL resolution is done inside the browser automatically.
- * @param {Object[]|String[]} [options.pseudoUrls]
+ * @param {Array<Object>|Array<string>} [options.pseudoUrls]
* An array of {@link PseudoUrl}s matching the URLs to be enqueued,
* or an array of strings or RegExps or plain Objects from which the {@link PseudoUrl}s can be constructed.
*
@@ -74,19 +74,19 @@
* }
* }
* ```
- * @return {Promise<QueueOperationInfo[]>}
+ * @return {Promise<Array<QueueOperationInfo>>}
* Promise that resolves to an array of {@link QueueOperationInfo} objects.
* @memberOf utils
* @name enqueueLinks
* @function
*/
export function enqueueLinks(options?: {
- page: Page;
- $: CheerioStatic;
+ page?: Page;
+ $?: CheerioStatic;
requestQueue: RequestQueue;
selector?: string;
baseUrl?: string;
- pseudoUrls?: any[] | string[];
+ pseudoUrls?: string[] | Object[];
transformRequestFunction?: RequestTransform;
}): Promise<QueueOperationInfo[]>;
/**
@@ -94,14 +94,14 @@ export function enqueueLinks(options?: {
*
* @param {Page} page
* @param {string} selector
- * @return {string[]}
+ * @return {Promise<Array<string>>}
* @ignore
*/
-export function extractUrlsFromPage(page: Page, selector: string): string[];
+export function extractUrlsFromPage(page: Page, selector: string): Promise<string[]>;
/**
* Extracts URLs from a given Cheerio object.
*
- * @param {Cheerio} $
+ * @param {CheerioStatic} $
* @param {string} selector
* @param {string} baseUrl
* @return {string[]}
diff --git a/types/enqueue_links/shared.d.ts b/types/enqueue_links/shared.d.ts
index ad7dfad6f..901f3b971 100644
--- a/types/enqueue_links/shared.d.ts
+++ b/types/enqueue_links/shared.d.ts
@@ -1,35 +1,40 @@
/**
* Helper factory used in the `enqueueLinks()` and enqueueLinksByClickingElements() function.
- * @param {string[]|Object[]} pseudoUrls
- * @return {PseudoUrl[]}
+ * @param {Array<(string|Object)>} pseudoUrls
+ * @return {Array<PseudoUrl>}
* @ignore
*/
-export function constructPseudoUrlInstances(pseudoUrls: any[] | string[]): PseudoUrl[];
+export function constructPseudoUrlInstances(pseudoUrls: (string | Object)[]): PseudoUrl[];
/**
- * @param {string[]|Object[]} requestOptions
- * @param {PseudoUrl[]} pseudoUrls
- * @return {Request[]}
+ * @param {Array<(string|Object)>} requestOptions
+ * @param {Array<PseudoUrl>} pseudoUrls
+ * @return {Array<Request>}
* @ignore
*/
-export function createRequests(requestOptions: any[] | string[], pseudoUrls: PseudoUrl[]): Request[];
+export function createRequests(requestOptions: (string | Object)[], pseudoUrls: PseudoUrl[]): Request[];
/**
- * @param {string[]|Object[]} sources
+ * @param {Array<(string|Object)>} sources
* @param {Object} [userData]
* @ignore
*/
-export function createRequestOptions(sources: any[] | string[], userData?: any): any;
+export function createRequestOptions(sources: (string | Object)[], userData?: Object | undefined): (Object | {
+ url: string;
+})[];
/**
- * @param {Request[]} requests
+ * @param {Array<Request>} requests
* @param {RequestQueue} requestQueue
* @param {number} batchSize
- * @return {Promise<QueueOperationInfo[]>}
+ * @return {Promise<Array<QueueOperationInfo>>}
* @ignore
*/
-export function addRequestsToQueueInBatches(requests: Request[], requestQueue: any, batchSize?: number): Promise<any[]>;
+export function addRequestsToQueueInBatches(requests: Request[], requestQueue: RequestQueue, batchSize?: number): Promise<QueueOperationInfo[]>;
/**
* Takes an Apify {RequestOptions} object and changes it's attributes in a desired way. This user-function is used
* {@link utils#enqueueLinks} to modify requests before enqueuing them.
*/
-export type RequestTransform = (original: any) => any;
+export type RequestTransform = (original: RequestOptions) => RequestOptions;
import PseudoUrl from "../pseudo_url";
import Request from "../request";
+import { RequestQueue } from "../request_queue";
+import { QueueOperationInfo } from "../request_queue";
+import { RequestOptions } from "../request";
diff --git a/types/errors.d.ts b/types/errors.d.ts
index 4610947e8..d5833ef48 100644
--- a/types/errors.d.ts
+++ b/types/errors.d.ts
@@ -3,17 +3,20 @@ export const APIFY_CALL_ERROR_NAME: "ApifyCallError";
* The class represents exceptions thrown
* by the {@link Apify#call} function.
*
- * @typedef {Object} ApifyCallError
- * @property {String} message
+ * @property {string} message
* Error message
- * @property {String} name
- * Contains `"ApifyCallError"`
* @property {ActorRun} run
* Object representing the failed actor run.
+ * @property {string} name
+ * Contains `"ApifyCallError"`
*/
export class ApifyCallError extends Error {
- constructor(run: any, message?: string);
- run: any;
+ /**
+ * @param {ActorRun} run
+ * @param {string} [message]
+ */
+ constructor(run: ActorRun, message?: string | undefined);
+ run: ActorRun;
}
/**
* TimeoutError class.
@@ -21,5 +24,6 @@ export class ApifyCallError extends Error {
* @ignore
*/
export class TimeoutError extends Error {
- constructor(message?: string);
+ constructor(message?: string | undefined);
}
+import { ActorRun } from "./typedefs";
diff --git a/types/events.d.ts b/types/events.d.ts
index 1b60b4408..fd46571f5 100644
--- a/types/events.d.ts
+++ b/types/events.d.ts
@@ -6,4 +6,4 @@ export function stopEvents(): void;
* @ignore
*/
declare const events: EventEmitter;
-import { EventEmitter } from "node/events";
+import { EventEmitter } from "events";
diff --git a/types/index.d.ts b/types/index.d.ts
new file mode 100644
index 000000000..28db16310
--- /dev/null
+++ b/types/index.d.ts
@@ -0,0 +1,3 @@
+export default Apify;
+export * from "./main";
+import * as Apify from "./main";
diff --git a/types/key_value_store.d.ts b/types/key_value_store.d.ts
index 08b3c316b..f5956c618 100644
--- a/types/key_value_store.d.ts
+++ b/types/key_value_store.d.ts
@@ -1,4 +1,4 @@
-export const LOCAL_STORAGE_SUBDIR: any;
+export const LOCAL_STORAGE_SUBDIR: string;
export function maybeStringify(value: any, options: any): any;
/**
* The `KeyValueStore` class represents a key-value store, a simple data storage that is used
@@ -94,11 +94,10 @@ export class KeyValueStore {
* const store = await Apify.openKeyValueStore();
* const buffer = await store.getValue('screenshot1.png');
* ```
- *
- * @param {String} key
+ * @param {string} key
* Unique key of the record. It can be at most 256 characters long and only consist
* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
- * @returns {Promise<Object|String|Buffer>}
+ * @returns {Promise<(object|Buffer|string|null)>}
* Returns a promise that resolves to an object, string
* or [](https://nodejs.org/api/buffer.html), depending
* on the MIME content type of the record.
@@ -136,10 +135,10 @@ export class KeyValueStore {
* **IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`,
* otherwise the actor process might finish before the value is stored!
*
- * @param {String} key
+ * @param {string} key
* Unique key of the record. It can be at most 256 characters long and only consist
* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
- * @param {Object|String|Buffer} value
+ * @param {(Object|string|Buffer|null)} value
* Record data, which can be one of the following values:
* <ul>
* <li>If `null`, the record in the key-value store is deleted.</li>
@@ -149,23 +148,23 @@ export class KeyValueStore {
* </ul>
* For any other value an error will be thrown.
* @param {Object} [options]
- * @param {String} [options.contentType]
+ * @param {string} [options.contentType]
* Specifies a custom MIME content type of the record.
* @returns {Promise<void>}
*
*/
- setValue(key: string, value: any, options?: {
+ setValue(key: string, value: string | Object | Buffer | null, options?: {
contentType?: string;
- }): Promise<void>;
+ } | undefined): Promise<void>;
/**
* Removes the key-value store either from the Apify cloud storage or from the local directory,
* depending on the mode of operation.
*
* @return {Promise<void>}
*/
- async drop(): Promise<void>;
+ drop(): Promise<void>;
/** @ignore */
- async delete(): Promise<void>;
+ delete(): Promise<void>;
/**
* Returns a URL for the given key that may be used to publicly
* access the value in the remote key value store.
@@ -198,9 +197,9 @@ export class KeyValueStore {
* @param {string} [options.exclusiveStartKey] All keys up to this one (including) are skipped from the result.
* @return {Promise<void>}
*/
- async forEachKey(iteratee: KeyConsumer, options?: {
+ forEachKey(iteratee: KeyConsumer, options?: {
exclusiveStartKey?: string;
- }, index?: number): Promise<void>;
+ } | undefined, index?: number): Promise<void>;
}
export function getFileNameRegexp(key: string): RegExp;
/**
@@ -213,11 +212,11 @@ export class KeyValueStoreLocal {
localStoragePath: string;
storeId: any;
initializationPromise: any;
- async getValue(key: any): Promise<any>;
- async setValue(key: any, value: any, options?: {}): Promise<void>;
- async delete(): Promise<void>;
- async drop(): Promise<void>;
- async forEachKey(iteratee: any, options?: {}, index?: number): Promise<void>;
+ getValue(key: any): Promise<any>;
+ setValue(key: any, value: any, options?: {}): Promise<void>;
+ delete(): Promise<void>;
+ drop(): Promise<void>;
+ forEachKey(iteratee: any, options?: {}, index?: number): Promise<void>;
/**
* Helper function to handle files. Accepts a promisified 'fs' function as a second parameter
* which will be executed against the file saved under the key. Since the file's extension and thus
@@ -226,28 +225,28 @@ export class KeyValueStoreLocal {
*
* Returns an object when a file is found and handler executes successfully, null otherwise.
*
- * @param {String} key
+ * @param {string} key
* @param {Function} handler
- * @returns {Promise} null or object in the following format:
+ * @returns {Promise<*>} null or object in the following format:
* {
* returnValue: return value of the handler function,
* fileName: name of the file including found extension
* }
* @ignore
*/
- async _handleFile(key: string, handler: Function): Promise<any>;
+ _handleFile(key: string, handler: Function): Promise<any>;
/**
* Performs a lookup for a file in the local emulation directory's file list.
- * @param {String} key
+ * @param {string} key
* @param {Function} handler
- * @returns {Promise}
+ * @returns {Promise<*>}
* @ignore
*/
_fullDirectoryLookup(key: string, handler: Function): Promise<any>;
/**
* Helper function to resolve file paths.
- * @param {String} fileName
- * @returns {String}
+ * @param {string} fileName
+ * @returns {string}
* @ignore
*/
_getPath(fileName: string): string;
@@ -264,13 +263,13 @@ export class KeyValueStoreLocal {
*/
getPublicUrl(fileName: string): string;
}
-export function openKeyValueStore(storeIdOrName?: string, options?: {
+export function openKeyValueStore(storeIdOrName?: string | undefined, options?: {
forceCloud?: boolean;
-}): Promise<KeyValueStore>;
+} | undefined): Promise<KeyValueStore>;
export function getValue(key: string): Promise<any>;
export function setValue(key: string, value: any, options?: {
contentType?: string;
-}): Promise<any>;
+} | undefined): Promise<void>;
export function getInput(): Promise<any>;
/**
* User-function used in the {@link KeyValueStore#forEachKey} method.
diff --git a/types/live_view/live_view_server.d.ts b/types/live_view/live_view_server.d.ts
index 14624a06b..b48791cdd 100644
--- a/types/live_view/live_view_server.d.ts
+++ b/types/live_view/live_view_server.d.ts
@@ -1,3 +1,4 @@
+/// <reference types="node" />
export default LiveViewServer;
/**
* `LiveViewServer` enables serving of browser snapshots via web sockets. It includes its own client
@@ -57,7 +58,7 @@ declare class LiveViewServer {
maxScreenshotFiles?: number;
snapshotTimeoutSecs?: number;
maxSnapshotFrequencySecs?: number;
- });
+ } | undefined);
screenshotDirectoryPath: string;
maxScreenshotFiles: number;
snapshotTimeoutMillis: number;
@@ -66,25 +67,25 @@ declare class LiveViewServer {
* @type {?Snapshot}
* @private
*/
- private lastSnapshot;
+ lastSnapshot: Snapshot | null;
lastScreenshotIndex: number;
clientCount: number;
_isRunning: boolean;
- httpServer: http.Server;
- socketio: socketio.Server;
+ httpServer: http.Server | null;
+ socketio: socketio.Server | null;
servingSnapshot: boolean;
/**
* Starts the HTTP server with web socket connections enabled.
* Snapshots will not be created until a client has connected.
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async start(): Promise<any>;
+ start(): Promise<void>;
/**
* Prevents the server from receiving more connections. Existing connections
* will not be terminated, but the server will not prevent a process exit.
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async stop(): Promise<any>;
+ stop(): Promise<void>;
/**
* Serves a snapshot to all connected clients.
* Screenshots are not served directly, only their index number
@@ -93,9 +94,9 @@ declare class LiveViewServer {
* Will time out and throw in `options.snapshotTimeoutSecs`.
*
* @param {Page} page
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async serve(page: Page): Promise<any>;
+ serve(page: Page): Promise<void>;
/**
* @return {boolean}
*/
@@ -110,33 +111,34 @@ declare class LiveViewServer {
* @return {string}
* @private
*/
- private _getScreenshotPath;
+ _getScreenshotPath(screenshotIndex: number): string;
/**
* @param {Page} page
* @return {Promise<Snapshot>}
* @private
*/
- private async _makeSnapshot;
+ _makeSnapshot(page: Page): Promise<Snapshot>;
/**
* @param {Snapshot} snapshot
* @private
*/
- private _pushSnapshot;
+ _pushSnapshot(snapshot: Snapshot): void;
/**
* Initiates an async delete and does not wait for it to complete.
* @param {number} screenshotIndex
* @private
*/
- private _deleteScreenshot;
+ _deleteScreenshot(screenshotIndex: number): void;
_setupHttpServer(): void;
- port: number;
- liveViewUrl: any;
+ port: number | undefined;
+ liveViewUrl: string | undefined;
/**
* @param {socketio.Socket} socket
* @private
*/
- private _socketConnectionHandler;
+ _socketConnectionHandler(socket: socketio.Socket): void;
}
-import http from "http";
-import socketio from "socket.io";
+import Snapshot from "./snapshot";
+import * as http from "http";
+import * as socketio from "socket.io";
import { Page } from "puppeteer";
diff --git a/types/index.esm.d.ts b/types/main.d.ts
similarity index 62%
rename from types/index.esm.d.ts
rename to types/main.d.ts
index 08b03bd07..b065ad420 100644
--- a/types/index.esm.d.ts
+++ b/types/main.d.ts
@@ -1,4 +1,4 @@
-/// <reference path="../src/utils_log.d.ts" />
+/// <reference types="node" />
import { main } from "./actor";
import { getEnv } from "./actor";
import { call } from "./actor";
@@ -48,50 +48,50 @@ declare const exportedUtils: {
htmlToText: (html: string | CheerioStatic) => string;
URL_NO_COMMAS_REGEX: RegExp;
URL_WITH_COMMAS_REGEX: RegExp;
- createRequestDebugInfo: (request: any, response?: import("http").IncomingMessage | import("puppeteer").Response, additionalFields?: any) => any;
- parseContentTypeFromResponse: (response: any) => {
+ createRequestDebugInfo: (request: Request | import("./request").RequestOptions, response?: import("http").IncomingMessage | import("puppeteer").Response | undefined, additionalFields?: Object | undefined) => any;
+ parseContentTypeFromResponse: (response: import("http").IncomingMessage) => {
type: string;
charset: string;
};
} & {
puppeteer: {
- hideWebDriver: (page: import("puppeteer").Page) => Promise<any>;
+ hideWebDriver: (page: import("puppeteer").Page) => Promise<void>;
injectFile: (page: import("puppeteer").Page, filePath: string, options?: {
surviveNavigations?: boolean;
- }) => Promise<any>;
+ } | undefined) => Promise<any>;
injectJQuery: (page: import("puppeteer").Page) => Promise<any>;
injectUnderscore: (page: import("puppeteer").Page) => Promise<any>;
enqueueRequestsFromClickableElements: (page: any, selector: any, purls: any, requestQueue: any, requestOpts?: {}) => Promise<any[]>;
- enqueueLinks: (...args: any[]) => Promise<import("./request_queue").QueueOperationInfo[]>;
+ enqueueLinks: (...args: any[]) => Promise<import("./request_queue").QueueOperationInfo[] | undefined>;
enqueueLinksByClickingElements: typeof import("./enqueue_links/click_elements").enqueueLinksByClickingElements;
blockRequests: (page: import("puppeteer").Page, options?: {
urlPatterns?: string[];
extraUrlPatterns?: string[];
- }) => Promise<any>;
+ } | undefined) => Promise<void>;
blockResources: (page: any, resourceTypes?: string[]) => Promise<void>;
- cacheResponses: (page: import("puppeteer").Page, cache: any, responseUrlRules: (string | RegExp)[]) => Promise<any>;
- compileScript: (scriptString: string, context?: any) => Function;
- gotoExtended: (page: import("puppeteer").Page, request: Request, gotoOptions?: any) => Promise<import("puppeteer").Response>;
- addInterceptRequestHandler: (page: any, handler: Function) => Promise<any>;
- removeInterceptRequestHandler: (page: any, handler: Function) => Promise<any>;
- infiniteScroll: (page: any, options?: {
+ cacheResponses: (page: import("puppeteer").Page, cache: Object, responseUrlRules: (string | RegExp)[]) => Promise<void>;
+ compileScript: (scriptString: string, context?: Object) => Function;
+ gotoExtended: (page: import("puppeteer").Page, request: Request, gotoOptions?: import("puppeteer").DirectNavigationOptions) => Promise<import("puppeteer").Response | null>;
+ addInterceptRequestHandler: (page: import("puppeteer").Page, handler: import("./puppeteer_request_interception").InterceptHandler) => Promise<void>;
+ removeInterceptRequestHandler: (page: import("puppeteer").Page, handler: import("./puppeteer_request_interception").InterceptHandler) => Promise<void>;
+ infiniteScroll: (page: import("puppeteer").Page, options?: {
timeoutSecs?: number;
waitForSecs?: number;
- }) => Promise<any>;
- saveSnapshot: (page: any, options?: {
+ } | undefined) => Promise<void>;
+ saveSnapshot: (page: import("puppeteer").Page, options?: {
key?: string;
screenshotQuality?: number;
saveScreenshot?: boolean;
saveHtml?: boolean;
keyValueStoreName?: string;
- }) => Promise<any>;
+ } | undefined) => Promise<void>;
};
social: {
emailsFromText: (text: string) => string[];
emailsFromUrls: (urls: string[]) => string[];
phonesFromText: (text: string) => string[];
phonesFromUrls: (urls: string[]) => string[];
- parseHandlesFromHtml: (html: string, data?: any) => import("./utils_social").SocialHandles;
+ parseHandlesFromHtml: (html: string, data?: Object) => import("./utils_social").SocialHandles;
EMAIL_REGEX: RegExp;
EMAIL_REGEX_GLOBAL: RegExp;
LINKEDIN_REGEX: RegExp;
@@ -105,32 +105,35 @@ declare const exportedUtils: {
YOUTUBE_REGEX: RegExp;
YOUTUBE_REGEX_GLOBAL: RegExp;
};
- log: any;
+ log: typeof log;
enqueueLinks: typeof enqueueLinks;
- requestAsBrowser: (options: import("./utils_request").RequestAsBrowserOptions) => Promise<any>;
+ requestAsBrowser: (options: import("./utils_request").RequestAsBrowserOptions) => Promise<import("stream").Readable | import("http").IncomingMessage>;
};
+import log from "./utils_log";
import { enqueueLinks } from "./enqueue_links/enqueue_links";
export { main, getEnv, call, callTask, metamorph, getMemoryInfo, getApifyProxyUrl, isAtHome, apifyClient as client, addWebhook, AutoscaledPool, BasicCrawler, CheerioCrawler, pushData, openDataset, events, initializeEvents, stopEvents, getValue, setValue, getInput, openKeyValueStore, launchPuppeteer, PuppeteerPool, PuppeteerCrawler, PseudoUrl, Request, RequestList, openRequestList, openRequestQueue, openSessionPool, LiveViewServer, Session, exportedUtils as utils };
-export { ApifyEnv } from './actor'
-export { DatasetContent, DatasetConsumer, DatasetMapper, DatasetReducer } from './dataset'
-export { KeyConsumer } from './key_value_store'
-export { LaunchPuppeteerOptions } from './puppeteer'
-export { PuppeteerPoolOptions } from './puppeteer_pool'
-export { RequestOptions } from './request'
-export { RequestListOptions, RequestListState } from './request_list'
-export { QueueOperationInfo } from './request_queue'
-export { Cheerio, ActorRun } from './typedefs'
-export { MemoryInfo } from './utils'
-export { RequestAsBrowserOptions } from './utils_request'
-export { SocialHandles } from './utils_social'
-export { AutoscaledPoolOptions } from './autoscaling/autoscaled_pool'
-export { SnapshotterOptions } from './autoscaling/snapshotter'
-export { SystemInfo, SystemStatusOptions } from './autoscaling/system_status'
-export { BasicCrawlerOptions, HandleRequest, HandleRequestInputs, HandleFailedRequest, HandleFailedRequestInput } from './crawlers/basic_crawler'
-export { CheerioCrawlerOptions, PrepareRequestInputs, PrepareRequest, CheerioHandlePageInputs, CheerioHandlePage } from './crawlers/cheerio_crawler'
-export { PuppeteerCrawlerOptions, PuppeteerHandlePageInputs, PuppeteerHandlePage, PuppeteerGotoInputs, PuppeteerGoto, LaunchPuppeteer } from './crawlers/puppeteer_crawler'
-export { RequestTransform } from './enqueue_links/shared'
-export { SessionState, SessionOptions } from './session_pool/session'
-export { CreateSession, SessionPoolOptions } from './session_pool/session_pool'
-export { StealthOptions } from './stealth/stealth'
\ No newline at end of file
+export { ApifyEnv, UserFunc } from "./actor";
+export { AutoscaledPoolOptions } from "./autoscaling/autoscaled_pool";
+export { SnapshotterOptions } from "./autoscaling/snapshotter";
+export { SystemInfo, SystemStatusOptions } from "./autoscaling/system_status";
+export { BasicCrawlerOptions, HandleRequest, HandleRequestInputs, HandleFailedRequest, HandleFailedRequestInput } from "./crawlers/basic_crawler";
+export { CheerioCrawlerOptions, PrepareRequestInputs, PrepareRequest, CheerioHandlePageInputs, CheerioHandlePage } from "./crawlers/cheerio_crawler";
+export { PuppeteerCrawlerOptions, PuppeteerHandlePageInputs, PuppeteerHandlePage, PuppeteerGotoInputs, PuppeteerGoto, LaunchPuppeteer } from "./crawlers/puppeteer_crawler";
+export { DatasetContent, DatasetConsumer, DatasetMapper, DatasetReducer } from "./dataset";
+export { RequestTransform } from "./enqueue_links/shared";
+export { KeyConsumer } from "./key_value_store";
+export { LaunchPuppeteerOptions } from "./puppeteer";
+export { LaunchPuppeteerFunction, PuppeteerPoolOptions } from "./puppeteer_pool";
+export { InterceptHandler } from "./puppeteer_request_interception";
+export { RequestOptions } from "./request";
+export { RequestListInput, SourceInput, RequestListOptions, RequestListState } from "./request_list";
+export { QueueOperationInfo } from "./request_queue";
+export { SessionState, SessionOptions } from "./session_pool/session";
+export { CreateSession, SessionPoolOptions } from "./session_pool/session_pool";
+export { StealthOptions } from "./stealth/stealth";
+export { ActorRun } from "./typedefs";
+export { MemoryInfo } from "./utils";
+export { LoggerOptions } from "./utils_log";
+export { RequestAsBrowserOptions, AbortFunction } from "./utils_request";
+export { SocialHandles } from "./utils_social";
\ No newline at end of file
diff --git a/types/pseudo_url.d.ts b/types/pseudo_url.d.ts
index ef29f477c..ecc2e36a8 100644
--- a/types/pseudo_url.d.ts
+++ b/types/pseudo_url.d.ts
@@ -51,7 +51,7 @@ export default PseudoUrl;
*/
declare class PseudoUrl {
/**
- * @param {String|RegExp} purl
+ * @param {(string|RegExp)} purl
* A pseudo-URL string or a regular expression object.
* Using a `RegExp` instance enables more granular control,
* such as making the matching case sensitive.
@@ -59,14 +59,14 @@ declare class PseudoUrl {
* Options for the new {@link Request} instances created for matching URLs
* by the {@link utils#enqueueLinks} function.
*/
- constructor(purl: string | RegExp, requestTemplate?: any);
+ constructor(purl: string | RegExp, requestTemplate?: RequestOptions);
regex: RegExp;
- requestTemplate: any;
+ requestTemplate: RequestOptions;
/**
* Determines whether a URL matches this pseudo-URL pattern.
*
- * @param {String} url URL to be matched.
- * @return {Boolean} Returns `true` if given URL matches pseudo-URL.
+ * @param {string} url URL to be matched.
+ * @return {boolean} Returns `true` if given URL matches pseudo-URL.
*/
matches(url: string): boolean;
/**
@@ -76,9 +76,10 @@ declare class PseudoUrl {
* be merged together, with the `userData` property having preference over the template.
* This enables dynamic overriding of the template.
*
- * @param {string|Object} urlOrProps
+ * @param {(string|Object)} urlOrProps
* @return {Request}
*/
- createRequest(urlOrProps: any): Request;
+ createRequest(urlOrProps: string | Object): Request;
}
+import { RequestOptions } from "./request";
import Request from "./request";
diff --git a/types/puppeteer.d.ts b/types/puppeteer.d.ts
index f7ff7f528..3817d259d 100644
--- a/types/puppeteer.d.ts
+++ b/types/puppeteer.d.ts
@@ -1,4 +1,4 @@
-export function launchPuppeteer(options?: LaunchPuppeteerOptions): Promise<any>;
+export function launchPuppeteer(options?: LaunchPuppeteerOptions | undefined): Promise<Browser>;
/**
* Apify extends the launch options of Puppeteer.
* You can use any of the Puppeteer compatible
@@ -57,7 +57,7 @@ export type LaunchPuppeteerOptions = {
* Take caution, because it can cause all kinds of unexpected errors and weird behavior.
* Apify SDK is not tested with any other library besides `puppeteer` itself.
*/
- puppeteerModule?: any;
+ puppeteerModule?: string | Object;
/**
* This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.
* It is recommended to use it together with the `useChrome` set to `true`.
@@ -69,4 +69,5 @@ export type LaunchPuppeteerOptions = {
*/
stealthOptions?: StealthOptions;
};
+import { Browser } from "puppeteer";
import { StealthOptions } from "./stealth/stealth";
diff --git a/types/puppeteer_pool.d.ts b/types/puppeteer_pool.d.ts
index 811e50910..b7b368a24 100644
--- a/types/puppeteer_pool.d.ts
+++ b/types/puppeteer_pool.d.ts
@@ -1,5 +1,6 @@
export const BROWSER_SESSION_KEY_NAME: "APIFY_SESSION";
export default PuppeteerPool;
+export type LaunchPuppeteerFunction = (options: LaunchPuppeteerOptions) => Promise<Browser>;
export type PuppeteerPoolOptions = {
/**
* Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
@@ -42,7 +43,7 @@ export type PuppeteerPoolOptions = {
* [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)
* for the default implementation.
*/
- launchPuppeteerFunction?: Function;
+ launchPuppeteerFunction?: LaunchPuppeteerFunction;
/**
* Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.
*/
@@ -75,7 +76,12 @@ export type PuppeteerPoolOptions = {
proxyUrls?: string[];
};
/**
- * @typedef {Object} PuppeteerPoolOptions
+ * @callback LaunchPuppeteerFunction
+ * @param {LaunchPuppeteerOptions} options
+ * @returns {Promise<Browser>}
+ */
+/**
+ * @typedef PuppeteerPoolOptions
* @property {boolean} [useLiveView]
* Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
* just before a page would be recycled by `PuppeteerPool`. If there are no clients
@@ -97,7 +103,7 @@ export type PuppeteerPoolOptions = {
* it is considered retired and no more tabs will be opened. After the last tab is closed the
* whole browser is closed too. This parameter defines a time limit between the last tab was opened and
* before the browser is closed even if there are pending open tabs.
- * @property {Function} [launchPuppeteerFunction]
+ * @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* The function must return a promise resolving to
* [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.
@@ -170,17 +176,21 @@ declare class PuppeteerPool {
* All `PuppeteerPool` parameters are passed
* via an options object.
*/
- constructor(options?: PuppeteerPoolOptions);
+ constructor(options?: PuppeteerPoolOptions | undefined);
sessionPool: any;
reusePages: boolean;
maxOpenPagesPerInstance: any;
retireInstanceAfterRequestCount: any;
puppeteerOperationTimeoutMillis: number;
killInstanceAfterMillis: any;
+ /**
+ * @type {*}
+ * @ignore
+ */
recycledDiskCacheDirs: any;
useIncognitoPages: any;
- proxyUrls: any;
- liveViewServer: LiveViewServer;
+ proxyUrls: any[] | null;
+ liveViewServer: LiveViewServer | null;
launchPuppeteerFunction: () => Promise<any>;
browserCounter: number;
activeInstances: {};
@@ -192,7 +202,7 @@ declare class PuppeteerPool {
pagesToInstancesMap: WeakMap<object, any>;
liveViewSnapshotsInProgress: WeakMap<object, any>;
sigintListener: () => void;
- async _retireBrowserWithSession(session: any): Promise<void>;
+ _retireBrowserWithSession(session: any): Promise<void>;
/**
* Launches new browser instance.
*
@@ -206,13 +216,13 @@ declare class PuppeteerPool {
* @returns {Promise<void>}
* @ignore
*/
- async _initBrowser(browserPromise: Promise<Browser>, instance: PuppeteerInstance): Promise<void>;
+ _initBrowser(browserPromise: Promise<Browser>, instance: PuppeteerInstance): Promise<void>;
/**
* Retires some of the instances for example due to many uses.
*
* @ignore
*/
- _retireInstance(instance: any): any;
+ _retireInstance(instance: any): void;
/**
* Kills all the retired instances that:
* - have all tabs closed
@@ -226,7 +236,7 @@ declare class PuppeteerPool {
*
* @ignore
*/
- async _killInstance(instance: any): Promise<void>;
+ _killInstance(instance: any): Promise<void>;
/**
* Kills all running PuppeteerInstances.
* @ignore
@@ -248,7 +258,7 @@ declare class PuppeteerPool {
*
* @return {Promise<Page>}
*/
- async newPage(): Promise<Page>;
+ newPage(): Promise<Page>;
/**
* Opens new tab in one of the browsers in the pool and returns a `Promise`
* that resolves to an instance of a Puppeteer
@@ -257,7 +267,7 @@ declare class PuppeteerPool {
* @return {Promise<Page>}
* @ignore
*/
- async _openNewTab(): Promise<Page>;
+ _openNewTab(): Promise<Page>;
/**
* Adds the necessary boilerplate to allow page reuse and also
* captures page.close() errors to prevent meaningless log clutter.
@@ -272,19 +282,19 @@ declare class PuppeteerPool {
* @param {Browser} browser
* @ignore
*/
- async _focusOldestTab(browser: Browser): Promise<void>;
+ _focusOldestTab(browser: Browser): Promise<void>;
/**
* Closes all open browsers.
* @return {Promise<void>}
*/
- async destroy(): Promise<void>;
+ destroy(): Promise<void>;
/**
* Finds a PuppeteerInstance given a Puppeteer Browser running in the instance.
* @param {Browser} browser
- * @return {Promise}
+ * @return {Promise<*>}
* @ignore
*/
- async _findInstanceByBrowser(browser: Browser): Promise<any>;
+ _findInstanceByBrowser(browser: Browser): Promise<any>;
/**
* Manually retires a Puppeteer
* [`Browser`](https://pptr.dev/#?product=Puppeteer&show=api-class-browser)
@@ -293,7 +303,7 @@ declare class PuppeteerPool {
* @param {Browser} browser
* @return {Promise<void>}
*/
- async retire(browser: Browser): Promise<void>;
+ retire(browser: Browser): Promise<void>;
/**
* Closes the page, unless the `reuseTabs` option is set to true.
* Then it would only flag the page for a future reuse, without actually closing it.
@@ -305,18 +315,19 @@ declare class PuppeteerPool {
* @param {Page} page
* @return {Promise<void>}
*/
- async recyclePage(page: Page): Promise<void>;
+ recyclePage(page: Page): Promise<void>;
/**
* Tells the connected LiveViewServer to serve a snapshot when available.
*
- * @param page
+ * @param {Page} page
* @return {Promise<void>}
*/
- async serveLiveViewSnapshot(page: any): Promise<void>;
+ serveLiveViewSnapshot(page: Page): Promise<void>;
_findInstancesBySession(session: any): any[];
_killInstanceWithNoPages(instance: any): void;
}
import { LaunchPuppeteerOptions } from "./puppeteer";
+import { Browser } from "puppeteer";
import LiveViewServer from "./live_view/live_view_server";
/**
* Internal representation of Puppeteer instance.
@@ -334,5 +345,4 @@ declare class PuppeteerInstance {
childProcess: any;
recycleDiskCacheDir: any;
}
-import { Browser } from "puppeteer";
import { Page } from "puppeteer";
diff --git a/types/puppeteer_request_interception.d.ts b/types/puppeteer_request_interception.d.ts
index bc97d3ca6..8cb356bec 100644
--- a/types/puppeteer_request_interception.d.ts
+++ b/types/puppeteer_request_interception.d.ts
@@ -1,2 +1,5 @@
-export function addInterceptRequestHandler(page: any, handler: Function): Promise<any>;
-export function removeInterceptRequestHandler(page: any, handler: Function): Promise<any>;
+export function addInterceptRequestHandler(page: Page, handler: InterceptHandler): Promise<void>;
+export function removeInterceptRequestHandler(page: Page, handler: InterceptHandler): Promise<void>;
+export type InterceptHandler = (request: PuppeteerRequest) => any;
+import { Page } from "puppeteer";
+import { Request as PuppeteerRequest } from "puppeteer";
diff --git a/types/puppeteer_utils.d.ts b/types/puppeteer_utils.d.ts
index 700e68738..e25ce83da 100644
--- a/types/puppeteer_utils.d.ts
+++ b/types/puppeteer_utils.d.ts
@@ -1,15 +1,15 @@
-export function gotoExtended(page: Page, request: Request, gotoOptions?: any): Promise<Response>;
-export function infiniteScroll(page: any, options?: {
+export function gotoExtended(page: Page, request: Request, gotoOptions?: DirectNavigationOptions): Promise<Response | null>;
+export function infiniteScroll(page: Page, options?: {
timeoutSecs?: number;
waitForSecs?: number;
-}): Promise<any>;
+} | undefined): Promise<void>;
export namespace puppeteerUtils {
export { hideWebDriver };
export { injectFile };
export { injectJQuery };
export { injectUnderscore };
export { enqueueRequestsFromClickableElements };
- export function enqueueLinks(...args: any[]): Promise<import("./request_queue").QueueOperationInfo[]>;
+ export function enqueueLinks(...args: any[]): Promise<import("./request_queue").QueueOperationInfo[] | undefined>;
export { enqueueLinksByClickingElements };
export { blockRequests };
export { blockResources };
@@ -23,11 +23,12 @@ export namespace puppeteerUtils {
}
import { Page } from "puppeteer";
import Request from "./request";
+import { DirectNavigationOptions } from "puppeteer";
import { Response } from "puppeteer";
-declare function hideWebDriver(page: Page): Promise<any>;
+declare function hideWebDriver(page: Page): Promise<void>;
declare function injectFile(page: Page, filePath: string, options?: {
surviveNavigations?: boolean;
-}): Promise<any>;
+} | undefined): Promise<any>;
declare function injectJQuery(page: Page): Promise<any>;
declare function injectUnderscore(page: Page): Promise<any>;
declare function enqueueRequestsFromClickableElements(page: any, selector: any, purls: any, requestQueue: any, requestOpts?: {}): Promise<any[]>;
@@ -35,17 +36,17 @@ import { enqueueLinksByClickingElements } from "./enqueue_links/click_elements";
declare function blockRequests(page: Page, options?: {
urlPatterns?: string[];
extraUrlPatterns?: string[];
-}): Promise<any>;
+} | undefined): Promise<void>;
declare function blockResources(page: any, resourceTypes?: string[]): Promise<void>;
-declare function cacheResponses(page: Page, cache: any, responseUrlRules: (string | RegExp)[]): Promise<any>;
-declare function compileScript(scriptString: string, context?: any): Function;
+declare function cacheResponses(page: Page, cache: Object, responseUrlRules: (string | RegExp)[]): Promise<void>;
+declare function compileScript(scriptString: string, context?: Object): Function;
import { addInterceptRequestHandler } from "./puppeteer_request_interception";
import { removeInterceptRequestHandler } from "./puppeteer_request_interception";
-declare function saveSnapshot(page: any, options?: {
+declare function saveSnapshot(page: Page, options?: {
key?: string;
screenshotQuality?: number;
saveScreenshot?: boolean;
saveHtml?: boolean;
keyValueStoreName?: string;
-}): Promise<any>;
+} | undefined): Promise<void>;
export {};
diff --git a/types/request.d.ts b/types/request.d.ts
index 39956d8da..f60805b91 100644
--- a/types/request.d.ts
+++ b/types/request.d.ts
@@ -8,7 +8,7 @@ export function computeUniqueKey({ url, method, payload, keepUrlFragment, useExt
export function hashPayload(payload: any): string;
export default Request;
/**
- * Specifies required and optional fields for constructing a {@link Request#setValue}.
+ * Specifies required and optional fields for constructing a [`Request`](../api/request).
*/
export type RequestOptions = {
/**
@@ -47,7 +47,7 @@ export type RequestOptions = {
* }
* ```
*/
- headers?: any;
+ headers?: Object;
/**
* Custom user data assigned to the request. Use this to save any request related data to the
* request's scope, keeping them accessible on retries, failures etc.
@@ -67,50 +67,6 @@ export type RequestOptions = {
*/
useExtendedUniqueKey?: boolean;
};
-/**
- * Specifies required and optional fields for constructing a {@link Request#setValue}.
- *
- * @typedef {Object} RequestOptions
- * @property {String} url URL of the web page to crawl. It must be a non-empty string.
- * @property {String} [uniqueKey] A unique key identifying the request.
- * Two requests with the same `uniqueKey` are considered as pointing to the same URL.
- *
- * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.
- * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `uniqueKey`
- * of `http://www.example.com/something`.
- *
- * The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.
- *
- * The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`,
- * producing a `uniqueKey` in the following format: `METHOD(payloadHash):normalizedUrl`. This is useful
- * when requests point to the same URL, but with different methods and payloads. For example: form submits.
- *
- * Pass an arbitrary non-empty text value to the `uniqueKey` property
- * to override the default behavior and specify which URLs shall be considered equal.
- * @property {String} [method='GET']
- * @property {String|Buffer} [payload]
- * HTTP request payload, e.g. for POST requests.
- * @property {Object} [headers={}]
- * HTTP headers in the following format:
- * ```
- * {
- * Accept: 'text/html',
- * 'Content-Type': 'application/json'
- * }
- * ```
- * @property {Object} [userData={}]
- * Custom user data assigned to the request. Use this to save any request related data to the
- * request's scope, keeping them accessible on retries, failures etc.
- * @property {Boolean} [keepUrlFragment=false]
- * If `false` then the hash part of a URL is removed when computing the `uniqueKey` property.
- * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs
- * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.
- * Note that this option only has an effect if `uniqueKey` is not set.
- * @property {Boolean} [useExtendedUniqueKey=false]
- * If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload
- * properties. This is useful when making requests to the same URL that are differentiated by method
- * or payload, such as form submit navigations in browsers.
- */
/**
* Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
* The `Request` object also stores information about errors that occurred during processing of the request.
@@ -137,54 +93,57 @@ export type RequestOptions = {
*
* const foo = request.userData.foo;
* ```
- * @param {RequestOptions} options `Request` parameters including the URL, HTTP method and headers, and others.
*
- * @property {String} id
+ * @property {string} id
* Request ID
- * @property {String} url
+ * @property {string} url
* URL of the web page to crawl.
- * @property {String} loadedUrl
+ * @property {string} loadedUrl
* An actually loaded URL after redirects, if present. HTTP redirects are guaranteed
* to be included.
*
* When using {@link PuppeteerCrawler}, meta tag and JavaScript redirects may,
* or may not be included, depending on their nature. This generally means that redirects,
* which happen immediately will most likely be included, but delayed redirects will not.
- * @property {String} uniqueKey
+ * @property {string} uniqueKey
* A unique key identifying the request.
* Two requests with the same `uniqueKey` are considered as pointing to the same URL.
- * @property {String} method
+ * @property {string} method
* HTTP method, e.g. `GET` or `POST`.
- * @property {String|Buffer} payload
+ * @property {(string|Buffer)} payload
* HTTP request payload, e.g. for POST requests.
- * @property {Boolean} noRetry
+ * @property {boolean} noRetry
* The `true` value indicates that the request will not be automatically retried on error.
- * @property {Number} retryCount
+ * @property {number} retryCount
* Indicates the number of times the crawling of the request has been retried on error.
- * @property {String[]} errorMessages
+ * @property {string[]} errorMessages
* An array of error messages from request processing.
* @property {Object} headers
* Object with HTTP headers. Key is header name, value is the value.
- * @property {Object} userData
+ * @property {object} userData
* Custom user data assigned to the request.
* @property {Date} handledAt
* Indicates the time when the request has been processed.
* Is `null` if the request has not been crawled yet.
*/
declare class Request {
- constructor(options?: {});
+ /**
+ * @param {RequestOptions} options
+ * `Request` parameters including the URL, HTTP method and headers, and others.
+ */
+ constructor(options?: RequestOptions);
id: any;
- url: any;
+ url: string;
loadedUrl: any;
uniqueKey: any;
- method: any;
+ method: string;
payload: any;
noRetry: any;
retryCount: any;
errorMessages: any;
headers: any;
userData: any;
- handledAt: any;
+ handledAt: Date | null;
/**
* Stores information about an error that occurred during processing of this request.
*
@@ -195,13 +154,13 @@ declare class Request {
* inspection of the passed argument and attempts to extract as much information
* as possible, since just throwing a bad type error makes any debugging rather difficult.
*
- * @param {Error|String} errorOrMessage Error object or error message to be stored in the request.
+ * @param {(Error|string)} errorOrMessage Error object or error message to be stored in the request.
* @param {Object} [options]
- * @param {Boolean} [options.omitStack=false] Only push the error message without stack trace when true.
+ * @param {boolean} [options.omitStack=false] Only push the error message without stack trace when true.
*/
pushErrorMessage(errorOrMessage: string | Error, options?: {
omitStack?: boolean;
- }): void;
+ } | undefined): void;
/**
* Flags the request with no retry which prevents {@link BasicCrawler}
* (as well as {@PuppeteerCrawler} and {@CheerioCrawler}, since they use {@BasicCrawler} internally)
@@ -210,9 +169,9 @@ declare class Request {
* Optionally accepts a message that will be used to construct
* and throw an Error.
*
- * @param {String} [message]
+ * @param {string} [message]
* @deprecated 2019/06/26
* @ignore
*/
- doNotRetry(message?: string): void;
+ doNotRetry(message?: string | undefined): void;
}
diff --git a/types/request_list.d.ts b/types/request_list.d.ts
index 86296499b..9c1a0502b 100644
--- a/types/request_list.d.ts
+++ b/types/request_list.d.ts
@@ -1,8 +1,17 @@
export const STATE_PERSISTENCE_KEY: "REQUEST_LIST_STATE";
export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";
/**
- * @typedef {Object} RequestListOptions
- * @property {Array<RequestOptions|Request>} sources
+ * @typedef RequestListInput
+ * @property {string} [method]
+ * @property {string} [requestsFromUrl]
+ * @property {RegExp} [regex]
+ */
+/**
+ * @typedef {Array<(RequestListInput|RequestOptions|Request)>} SourceInput
+ */
+/**
+ * @typedef RequestListOptions
+ * @property {SourceInput} sources
* An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
* define the `url` property, or an array of instances of the {@link Request} class.
* Additionally, the `requestsFromUrl` property may be used instead of `url`,
@@ -26,7 +35,7 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";
* { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }
* ]
* ```
- * @property {String} [persistStateKey]
+ * @property {string} [persistStateKey]
* Identifies the key in the default key-value store under which `RequestList` periodically stores its
* state (i.e. which URLs were crawled and which not).
* If the actor is restarted, `RequestList` will read the state
@@ -34,7 +43,7 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";
*
* If `persistStateKey` is not set, `RequestList` will always start from the beginning,
* and all the source URLs will be crawled again.
- * @property {String} [persistSourcesKey]
+ * @property {string} [persistSourcesKey]
* Identifies the key in the default key-value store under which the `RequestList` persists its
* sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.
* This is necessary if `persistStateKey` is set and the source URLs might potentially change,
@@ -60,7 +69,7 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";
*
* Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
* is to use the `stateKeyPrefix` parameter instead.
- * @property {Boolean} [keepDuplicateUrls=false]
+ * @property {boolean} [keepDuplicateUrls=false]
* By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
* on the `uniqueKey` property of passed source {@link Request} objects.
*
@@ -164,7 +173,7 @@ export class RequestList {
*
* @returns {Promise<void>}
*/
- async initialize(): Promise<void>;
+ initialize(): Promise<void>;
/**
* Persists the current state of the `RequestList` into the default {@link KeyValueStore}.
* The state is persisted automatically in regular intervals, but calling this method manually
@@ -174,7 +183,7 @@ export class RequestList {
*
* @return {Promise<void>}
*/
- async persistState(): Promise<void>;
+ persistState(): Promise<void>;
/**
* Unlike persistState(), this is used only internally, since the sources
* are automatically persisted at RequestList initialization (if the persistSourcesKey is set),
@@ -183,7 +192,7 @@ export class RequestList {
* @return {Promise<void>}
* @ignore
*/
- async _persistSources(): Promise<void>;
+ _persistSources(): Promise<void>;
/**
* Restores RequestList state from a state object.
*
@@ -195,10 +204,10 @@ export class RequestList {
* Attempts to load state and sources using the `RequestList` configuration
* and returns a tuple of [state, sources] where each may be null if not loaded.
*
- * @return {Promise<Array>}
+ * @return {Promise<Array<(RequestListState|null)>>}
* @ignore
*/
- async _loadStateAndSources(): Promise<any[]>;
+ _loadStateAndSources(): Promise<(RequestListState | null)[]>;
/**
* Returns an object representing the internal state of the `RequestList` instance.
* Note that the object's fields can change in future releases.
@@ -213,13 +222,13 @@ export class RequestList {
*
* @returns {Promise<Boolean>}
*/
- async isEmpty(): Promise<boolean>;
+ isEmpty(): Promise<boolean>;
/**
* Returns `true` if all requests were already handled and there are no more left.
*
* @returns {Promise<Boolean>}
*/
- async isFinished(): Promise<boolean>;
+ isFinished(): Promise<boolean>;
/**
* Gets the next {@link Request} to process. First, the function gets a request previously reclaimed
* using the {@link RequestList#reclaimRequest} function, if there is any.
@@ -228,39 +237,37 @@ export class RequestList {
* The function's `Promise` resolves to `null` if there are no more
* requests to process.
*
- * @returns {Promise<Request>}
+ * @returns {Promise<(Request|null)>}
*/
- async fetchNextRequest(): Promise<Request>;
+ fetchNextRequest(): Promise<Request | null>;
/**
* Marks request as handled after successful processing.
*
* @param {Request} request
- *
* @returns {Promise<void>}
*/
- async markRequestHandled(request: Request): Promise<void>;
+ markRequestHandled(request: Request): Promise<void>;
/**
* Reclaims request to the list if its processing failed.
* The request will become available in the next `this.fetchNextRequest()`.
*
* @param {Request} request
- *
* @returns {Promise<void>}
*/
- async reclaimRequest(request: Request): Promise<void>;
+ reclaimRequest(request: Request): Promise<void>;
/**
* Adds all fetched requests from a URL from a remote resource.
*
* @ignore
*/
- async _addFetchedRequests(source: any, fetchedRequests: any): Promise<void>;
+ _addFetchedRequests(source: any, fetchedRequests: any): Promise<void>;
/**
* Fetches URLs from requestsFromUrl and returns them in format of list of requests
* @param source
- * @return {Promise<RequestOptions[]>}
+ * @return {Promise<Array<RequestOptions>>}
* @ignore
*/
- async _fetchRequestsFromUrl(source: any): Promise<any[]>;
+ _fetchRequestsFromUrl(source: any): Promise<RequestOptions[]>;
/**
* Adds given request.
* If the `opts` parameter is a plain object and not an instance of a `Request`, then the function
@@ -291,17 +298,23 @@ export class RequestList {
/**
* Returns the total number of unique requests present in the `RequestList`.
*
- * @returns {Number}
+ * @returns {number}
*/
length(): number;
/**
* Returns number of handled requests.
*
- * @returns {Number}
+ * @returns {number}
*/
handledCount(): number;
}
-export function openRequestList(listName: string, sources: any[], options?: RequestListOptions): Promise<RequestList>;
+export function openRequestList(listName: string | null, sources: string[] | (Request | RequestOptions | RequestListInput)[], options?: RequestListOptions | undefined): Promise<RequestList>;
+export type RequestListInput = {
+ method?: string;
+ requestsFromUrl?: string;
+ regex?: RegExp;
+};
+export type SourceInput = (Request | RequestOptions | RequestListInput)[];
export type RequestListOptions = {
/**
* An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that
@@ -328,7 +341,7 @@ export type RequestListOptions = {
* ]
* ```
*/
- sources: any[];
+ sources: (Request | RequestOptions | RequestListInput)[];
/**
* Identifies the key in the default key-value store under which `RequestList` periodically stores its
* state (i.e. which URLs were crawled and which not).
@@ -413,6 +426,9 @@ export type RequestListState = {
/**
* An object mapping request keys to a boolean value respresenting whether they are being processed at the moment.
*/
- inProgress: any;
+ inProgress: {
+ [x: string]: boolean;
+ };
};
import Request from "./request";
+import { RequestOptions } from "./request";
diff --git a/types/request_queue.d.ts b/types/request_queue.d.ts
index ea3071d7e..54543be4a 100644
--- a/types/request_queue.d.ts
+++ b/types/request_queue.d.ts
@@ -1,4 +1,4 @@
-export const LOCAL_STORAGE_SUBDIR: any;
+export const LOCAL_STORAGE_SUBDIR: string;
export const QUERY_HEAD_MIN_LENGTH: 100;
export const QUERY_HEAD_BUFFER: 3;
export const API_PROCESSED_REQUESTS_DELAY_MILLIS: number;
@@ -9,11 +9,11 @@ export const STORAGE_CONSISTENCY_DELAY_MILLIS: 3000;
* {@link RequestQueue} functions as well as
* {@link utils#enqueueLinks}.
*
- * @typedef {Object} QueueOperationInfo
- * @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.
- * @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.
- * @property {String} requestId The ID of the added request
- * @property {Object} request The original {@link Request} object passed to the `RequestQueue` function.
+ * @typedef QueueOperationInfo
+ * @property {boolean} wasAlreadyPresent Indicates if request was already present in the queue.
+ * @property {boolean} wasAlreadyHandled Indicates if request was already marked as handled.
+ * @property {string} requestId The ID of the added request
+ * @property {Request} request The original {@link Request} object passed to the `RequestQueue` function.
*/
/**
* Represents a queue of URLs to crawl, which is used for deep crawling of websites
@@ -81,10 +81,19 @@ export const STORAGE_CONSISTENCY_DELAY_MILLIS: 3000;
* @hideconstructor
*/
export class RequestQueue {
- constructor(queueId: any, queueName: any, clientKey?: any);
- clientKey: any;
- queueId: any;
- queueName: any;
+ /**
+ * @param {string} queueId
+ * @param {string} [queueName]
+ * @param {string} [clientKey]
+ */
+ constructor(queueId: string, queueName?: string | undefined, clientKey?: string | undefined);
+ clientKey: string;
+ queueId: string;
+ queueName: string | undefined;
+ /**
+ * @type {*}
+ * @ignore
+ */
queueHeadDict: any;
queryQueueHeadPromise: any;
inProgress: Set<any>;
@@ -106,22 +115,22 @@ export class RequestQueue {
* To add multiple requests to the queue by extracting links from a webpage,
* see the {@link utils#enqueueLinks} helper function.
*
- * @param {Request|RequestOptions} request {@link Request} object or vanilla object with request data.
+ * @param {(Request|RequestOptions)} request {@link Request} object or vanilla object with request data.
* Note that the function sets the `uniqueKey` and `id` fields to the passed object.
* @param {Object} [options]
- * @param {Boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.
+ * @param {boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.
* @return {Promise<QueueOperationInfo>}
*/
- async addRequest(request: RequestOptions | Request, options?: {
+ addRequest(request: Request | RequestOptions, options?: {
forefront?: boolean;
- }): Promise<QueueOperationInfo>;
+ } | undefined): Promise<QueueOperationInfo>;
/**
* Gets the request from the queue specified by ID.
*
- * @param {String} requestId ID of the request.
- * @return {Promise<Request>} Returns the request object, or `null` if it was not found.
+ * @param {string} requestId ID of the request.
+ * @return {Promise<(Request | null)>} Returns the request object, or `null` if it was not found.
*/
- async getRequest(requestId: string): Promise<Request>;
+ getRequest(requestId: string): Promise<Request | null>;
/**
* Returns a next request in the queue to be processed, or `null` if there are no more pending requests.
*
@@ -136,10 +145,10 @@ export class RequestQueue {
* To check whether all requests in queue were finished,
* use {@link RequestQueue#isFinished} instead.
*
- * @returns {Promise<Request>}
+ * @returns {Promise<(Request|null)>}
* Returns the request object or `null` if there are no more pending requests.
*/
- async fetchNextRequest(): Promise<Request>;
+ fetchNextRequest(): Promise<Request | null>;
/**
* Marks a request that was previously returned by the
* {@link RequestQueue#fetchNextRequest}
@@ -149,7 +158,7 @@ export class RequestQueue {
* @param {Request} request
* @return {Promise<QueueOperationInfo>}
*/
- async markRequestHandled(request: Request): Promise<QueueOperationInfo>;
+ markRequestHandled(request: Request): Promise<QueueOperationInfo>;
/**
* Reclaims a failed request back to the queue, so that it can be returned for processed later again
* by another call to {@link RequestQueue#fetchNextRequest}.
@@ -158,33 +167,33 @@ export class RequestQueue {
*
* @param {Request} request
* @param {Object} [options]
- * @param {Boolean} [options.forefront=false]
+ * @param {boolean} [options.forefront=false]
* If `true` then the request it placed to the beginning of the queue, so that it's returned
* in the next call to {@link RequestQueue#fetchNextRequest}.
* By default, it's put to the end of the queue.
* @return {Promise<QueueOperationInfo>}
*/
- async reclaimRequest(request: Request, options?: {
+ reclaimRequest(request: Request, options?: {
forefront?: boolean;
- }): Promise<QueueOperationInfo>;
+ } | undefined): Promise<QueueOperationInfo>;
/**
* Resolves to `true` if the next call to {@link RequestQueue#fetchNextRequest}
* would return `null`, otherwise it resolves to `false`.
* Note that even if the queue is empty, there might be some pending requests currently being processed.
* If you need to ensure that there is no activity in the queue, use {@link RequestQueue#isFinished}.
*
- * @returns {Promise<Boolean>}
+ * @returns {Promise<boolean>}
*/
- async isEmpty(): Promise<boolean>;
+ isEmpty(): Promise<boolean>;
/**
* Resolves to `true` if all requests were already handled and there are no more left.
* Due to the nature of distributed storage used by the queue,
* the function might occasionally return a false negative,
* but it will never return a false positive.
*
- * @returns {Promise<Boolean>}
+ * @returns {Promise<boolean>}
*/
- async isFinished(): Promise<boolean>;
+ isFinished(): Promise<boolean>;
/**
* Caches information about request to beware of unneeded addRequest() calls.
*
@@ -194,29 +203,29 @@ export class RequestQueue {
/**
* We always request more items than is in progress to ensure that something falls into head.
*
- * @param {Boolean} [ensureConsistency=false] If true then query for queue head is retried until queueModifiedAt
+ * @param {boolean} [ensureConsistency=false] If true then query for queue head is retried until queueModifiedAt
* is older than queryStartedAt by at least API_PROCESSED_REQUESTS_DELAY_MILLIS to ensure that queue
* head is consistent.
- * @param {Number} [limit] How many queue head items will be fetched.
- * @param {Number} [iteration] Used when this function is called recursively to limit the recursion.
- * @return {Boolean} Indicates if queue head is consistent (true) or inconsistent (false).
+ * @param {number} [limit] How many queue head items will be fetched.
+ * @param {number} [iteration] Used when this function is called recursively to limit the recursion.
+ * @return {Promise<boolean>} Indicates if queue head is consistent (true) or inconsistent (false).
* @ignore
*/
- async _ensureHeadIsNonEmpty(ensureConsistency?: boolean, limit?: number, iteration?: number): boolean;
+ _ensureHeadIsNonEmpty(ensureConsistency?: boolean | undefined, limit?: number | undefined, iteration?: number | undefined): Promise<boolean>;
/**
* Adds a request straight to the queueHeadDict, to improve performance.
* @private
*/
- private _maybeAddRequestToQueueHead;
+ _maybeAddRequestToQueueHead(requestId: any, forefront: any): void;
/**
* Removes the queue either from the Apify Cloud storage or from the local directory,
* depending on the mode of operation.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async drop(): Promise<any>;
+ drop(): Promise<void>;
/** @ignore */
- async delete(): Promise<void>;
+ delete(): Promise<void>;
/**
* Returns the number of handled requests.
*
@@ -228,7 +237,7 @@ export class RequestQueue {
*
* @return {Promise<number>}
*/
- async handledCount(): Promise<number>;
+ handledCount(): Promise<number>;
/**
* Returns an object containing general information about the request queue.
*
@@ -253,9 +262,9 @@ export class RequestQueue {
* }
* ```
*
- * @returns {Promise<Object>}
+ * @returns {Promise<object>}
*/
- async getInfo(): Promise<any>;
+ getInfo(): Promise<any>;
}
/**
* Local directory-based implementation of the `RequestQueue` class.
@@ -278,56 +287,56 @@ export class RequestQueueLocal {
requestIdToQueueOrderNo: {};
queueOrderNoInProgress: {};
requestsBeingWrittenToFile: Map<any, any>;
- createdAt: Date;
- modifiedAt: Date;
- accessedAt: Date;
+ createdAt: Date | null;
+ modifiedAt: Date | null;
+ accessedAt: Date | null;
initializationPromise: Promise<void>;
- async _initialize(): Promise<void>;
- async _saveRequestIdToQueueOrderNo(filepath: any): Promise<void>;
+ _initialize(): Promise<void>;
+ _saveRequestIdToQueueOrderNo(filepath: any): Promise<void>;
_getFilePath(queueOrderNo: any, isHandled?: boolean): string;
_getQueueOrderNo(forefront?: boolean): number;
- async _getRequestByQueueOrderNo(queueOrderNo: any): Promise<Request>;
- async addRequest(request: any, opts?: {}): Promise<{
+ _getRequestByQueueOrderNo(queueOrderNo: any): Promise<Request>;
+ addRequest(request: any, opts?: {}): Promise<{
requestId: any;
wasAlreadyHandled: any;
wasAlreadyPresent: boolean;
request: any;
}>;
- async getRequest(requestId: any): Promise<any>;
- async fetchNextRequest(): Promise<Request>;
- async markRequestHandled(request: any): Promise<{
+ getRequest(requestId: any): Promise<any>;
+ fetchNextRequest(): Promise<Request | null>;
+ markRequestHandled(request: any): Promise<{
requestId: any;
wasAlreadyHandled: boolean;
wasAlreadyPresent: boolean;
request: any;
}>;
- async reclaimRequest(request: any, opts?: {}): Promise<{
+ reclaimRequest(request: any, opts?: {}): Promise<{
requestId: any;
wasAlreadyHandled: boolean;
wasAlreadyPresent: boolean;
request: any;
}>;
- async isEmpty(): Promise<boolean>;
- async isFinished(): Promise<boolean>;
- async drop(): Promise<void>;
- async delete(): Promise<void>;
- async handledCount(): Promise<number>;
- async getInfo(): Promise<{
+ isEmpty(): Promise<boolean>;
+ isFinished(): Promise<boolean>;
+ drop(): Promise<void>;
+ delete(): Promise<void>;
+ handledCount(): Promise<number>;
+ getInfo(): Promise<{
id: any;
name: any;
- userId: string;
- createdAt: Date;
- modifiedAt: Date;
- accessedAt: Date;
+ userId: string | null;
+ createdAt: Date | null;
+ modifiedAt: Date | null;
+ accessedAt: Date | null;
totalRequestCount: number;
handledRequestCount: number;
pendingRequestCount: number;
}>;
_updateMetadata(isModified: any): void;
}
-export function openRequestQueue(queueIdOrName?: string, options?: {
+export function openRequestQueue(queueIdOrName?: string | undefined, options?: {
forceCloud?: boolean;
-}): Promise<RequestQueue>;
+} | undefined): Promise<RequestQueue>;
/**
* A helper class that is used to report results from various
* {@link RequestQueue} functions as well as
@@ -349,7 +358,7 @@ export type QueueOperationInfo = {
/**
* The original {@link Request} object passed to the `RequestQueue` function.
*/
- request: any;
+ request: Request;
};
-import { RequestOptions } from "./request";
import Request from "./request";
+import { RequestOptions } from "./request";
diff --git a/types/session_pool/errors.d.ts b/types/session_pool/errors.d.ts
index fd7363e81..115a6952b 100644
--- a/types/session_pool/errors.d.ts
+++ b/types/session_pool/errors.d.ts
@@ -1,4 +1,7 @@
export class CookieParseError extends Error {
- constructor(cookieHeaderString: any);
- cookieHeaderString: any;
+ /**
+ * @param {string} cookieHeaderString
+ */
+ constructor(cookieHeaderString: string);
+ cookieHeaderString: string;
}
diff --git a/types/session_pool/session.d.ts b/types/session_pool/session.d.ts
index 023b67ba0..2b451e8f7 100644
--- a/types/session_pool/session.d.ts
+++ b/types/session_pool/session.d.ts
@@ -1,9 +1,10 @@
+/// <reference types="node" />
/**
* Persistable {@link Session} state.
- * @typedef {Object} SessionState
+ * @typedef SessionState
* @property {string} id
- * @property {Object} cookieJar
- * @property {Object} userData
+ * @property {CookieJar} cookieJar
+ * @property {object} userData
* @property {number} errorScore
* @property {number} maxErrorScore
* @property {number} errorScoreDecrement
@@ -12,10 +13,10 @@
* @property {string} createdAt
*/
/**
- * @typedef {Object} SessionOptions
+ * @typedef SessionOptions
* @property {string} [id] - Id of session used for generating fingerprints. It is used as proxy session name.
* @property {number} [maxAgeSecs=3000] - Number of seconds after which the session is considered as expired.
- * @property {Object} userData - Object where custom user data can be stored. For example custom headers.
+ * @property {object} [userData] - Object where custom user data can be stored. For example custom headers.
* @property {number} [maxErrorScore=3] - Maximum number of marking session as blocked usage.
* If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
* It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
@@ -35,18 +36,20 @@
* Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions.
* You can imagine each session as a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint.
* Session internal state can be enriched with custom user data for example some authorization tokens and specific headers in general.
+ *
*/
export class Session {
/**
* Session configuration.
+ *
* @param {SessionOptions} options
*/
constructor(options?: SessionOptions);
/**
- * @type CookieJar
+ * @type {CookieJar}
* @private
- * */
- private cookieJar;
+ */
+ cookieJar: CookieJar;
id: string;
maxAgeSecs: number;
userData: any;
@@ -115,18 +118,17 @@ export class Session {
* @param [blockedStatusCodes] {number[]} - Custom HTTP status codes that means blocking on particular website.
* @return {boolean} whether the session was retired.
*/
- retireOnBlockedStatusCodes(statusCode: number, blockedStatusCodes?: number[]): boolean;
+ retireOnBlockedStatusCodes(statusCode: number, blockedStatusCodes?: number[] | undefined): boolean;
/**
* Saves cookies from an HTTP response to be used with the session.
* It expects an object with a `headers` property that's either an `Object`
* (typical Node.js responses) or a `Function` (Puppeteer Response).
*
* It then parses and saves the cookies from the `set-cookie` header, if available.
- * @param {{ headers }} response
+
+ * @param {(PuppeteerResponse|IncomingMessage)} response
*/
- setCookiesFromResponse(response: {
- headers: any;
- }): void;
+ setCookiesFromResponse(response: IncomingMessage | PuppeteerResponse): void;
/**
* Saves an array with cookie objects to be used with the session.
* The objects should be in the format that
@@ -140,13 +142,13 @@ export class Session {
* ]
* ```
*
- * @param cookies {PuppeteerCookie[]}
- * @param url {string}
+ * @param {PuppeteerCookie[]} cookies
+ * @param {string} url
*/
setPuppeteerCookies(cookies: PuppeteerCookie[], url: string): void;
/**
* Returns cookies in a format compatible with puppeteer and ready to be used with `page.setCookie`.
- * @param url {String} - website url. Only cookies stored for this url will be returned
+ * @param {string} url website url. Only cookies stored for this url will be returned
* @return {PuppeteerCookie[]}
*/
getPuppeteerCookies(url: string): PuppeteerCookie[];
@@ -160,32 +162,32 @@ export class Session {
getCookieString(url: string): string;
/**
* Transforms puppeteer cookie to tough-cookie.
- * @param puppeteerCookie {PuppeteerCookie} - Cookie from puppeteer `page.cookies method.
+ * @param {PuppeteerCookie} puppeteerCookie Cookie from puppeteer `page.cookies method.
* @return {Cookie}
* @private
*/
- private _puppeteerCookieToTough;
+ _puppeteerCookieToTough(puppeteerCookie: PuppeteerCookie): Cookie;
/**
* Transforms tough-cookie to puppeteer cookie .
* @param {Cookie} toughCookie - Cookie from CookieJar
* @return {PuppeteerCookie} - Cookie from Puppeteer
* @private
*/
- private _toughCookieToPuppeteer;
+ _toughCookieToPuppeteer(toughCookie: Cookie): PuppeteerCookie;
/**
* Sets cookies.
* @param {Cookie[]} cookies
* @param {string} url
* @private
*/
- private _setCookies;
+ _setCookies(cookies: Cookie[], url: string): void;
}
/**
* Persistable {@link Session} state.
*/
export type SessionState = {
id: string;
- cookieJar: any;
+ cookieJar: CookieJar;
userData: any;
errorScore: number;
maxErrorScore: number;
@@ -206,7 +208,7 @@ export type SessionOptions = {
/**
* - Object where custom user data can be stored. For example custom headers.
*/
- userData: any;
+ userData?: any;
/**
* - Maximum number of marking session as blocked usage.
* If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
@@ -246,5 +248,9 @@ export type SessionOptions = {
*/
sessionPool: SessionPool;
};
+import { CookieJar } from "tough-cookie";
import { SessionPool } from "./session_pool";
+import { IncomingMessage } from "http";
+import { Response as PuppeteerResponse } from "puppeteer";
import { Cookie as PuppeteerCookie } from "puppeteer";
+import { Cookie } from "tough-cookie";
diff --git a/types/session_pool/session_pool.d.ts b/types/session_pool/session_pool.d.ts
index 487b3b09f..353c12ca8 100644
--- a/types/session_pool/session_pool.d.ts
+++ b/types/session_pool/session_pool.d.ts
@@ -2,14 +2,15 @@
* Factory user-function which creates customized {@link Session} instances.
* @callback CreateSession
* @param {SessionPool} sessionPool Pool requesting the new session.
+ * @returns {Session}
*/
/**
- * @typedef {Object} SessionPoolOptions
- * @property {Number} [maxPoolSize=1000] - Maximum size of the pool.
+ * @typedef SessionPoolOptions
+ * @property {number} [maxPoolSize=1000] - Maximum size of the pool.
* Indicates how many sessions are rotated.
* @property {SessionOptions} [sessionOptions] The configuration options for {Session} instances.
- * @property {String} [persistStateKeyValueStoreId] - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
- * @property {String} [persistStateKey="SESSION_POOL_STATE"] - Session pool persists it's state under this key in Key value store.
+ * @property {string} [persistStateKeyValueStoreId] - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
+ * @property {string} [persistStateKey="SESSION_POOL_STATE"] - Session pool persists it's state under this key in Key value store.
* @property {CreateSession} [createSessionFunction] - Custom function that should return `Session` instance.
* Function receives `SessionPool` instance as a parameter
*/
@@ -66,13 +67,13 @@ export class SessionPool extends EventEmitter {
* Session pool configuration.
* @param {SessionPoolOptions} [options] All `SessionPool` configuration options.
*/
- constructor(options?: SessionPoolOptions);
+ constructor(options?: SessionPoolOptions | undefined);
maxPoolSize: number;
- createSessionFunction: CreateSession;
+ createSessionFunction: any;
sessionOptions: {};
- persistStateKeyValueStoreId: string;
+ persistStateKeyValueStoreId: any;
persistStateKey: string;
- keyValueStore: import("../key_value_store").KeyValueStore;
+ keyValueStore: import("../key_value_store").KeyValueStore | null;
sessions: any[];
/**
* Gets count of usable sessions in the pool.
@@ -90,8 +91,8 @@ export class SessionPool extends EventEmitter {
*
* @return {Promise<void>}
*/
- async initialize(): Promise<void>;
- _listener: any;
+ initialize(): Promise<void>;
+ _listener: (() => Promise<void>) | undefined;
/**
* Gets session.
* If there is space for new session, it creates and return new session.
@@ -100,21 +101,23 @@ export class SessionPool extends EventEmitter {
*
* @return {Promise<Session>}
*/
- async getSession(): Promise<Session>;
+ getSession(): Promise<Session>;
/**
* Returns an object representing the internal state of the `SessionPool` instance.
* Note that the object's fields can change in future releases.
- *
- * @returns {Object}
*/
- getState(): any;
+ getState(): {
+ usableSessionsCount: number;
+ retiredSessionsCount: number;
+ sessions: any[];
+ };
/**
* Persists the current state of the `SessionPool` into the default {@link KeyValueStore}.
* The state is persisted automatically in regular intervals.
*
- * @return {Promise}
+ * @return {Promise<void>}
*/
- async persistState(): Promise<any>;
+ persistState(): Promise<void>;
/**
* Removes listener from `persistState` event.
* This function should be called after you are done with using the `SessionPool` instance.
@@ -122,69 +125,60 @@ export class SessionPool extends EventEmitter {
teardown(): void;
/**
* Removes `Session` instance from `SessionPool`.
- * @param session {Session} - Session to be removed
+ * @param {Session} session - Session to be removed
* @private
*/
- private _removeSession;
+ _removeSession(session: Session): void;
/**
* Adds `Session` instance to `SessionPool`.
- * @param newSession {Session} - `Session` instance to be added.
+ * @param {Session} newSession `Session` instance to be added.
* @private
*/
- private _addSession;
+ _addSession(newSession: Session): void;
/**
* Gets random index.
* @return {number}
* @private
*/
- private _getRandomIndex;
+ _getRandomIndex(): number;
/**
* Creates new session without any extra behavior.
- * @param sessionPool
+ * @param {SessionPool} sessionPool
* @return {Session} - New session.
* @private
*/
- private _defaultCreateSessionFunction;
+ _defaultCreateSessionFunction(sessionPool: SessionPool): Session;
/**
* Creates new session and adds it to the pool.
* @return {Promise<Session>} - Newly created `Session` instance.
* @private
*/
- private async _createSession;
+ _createSession(): Promise<Session>;
/**
* Decides whether there is enough space for creating new session.
* @return {boolean}
* @private
*/
- private _hasSpaceForSession;
+ _hasSpaceForSession(): boolean;
/**
* Picks random session from the `SessionPool`.
* @return {Session} - Picked `Session`
* @private
*/
- private _pickSession;
+ _pickSession(): Session;
/**
* Potentially loads `SessionPool`.
* If the state was persisted it loads the `SessionPool` from the persisted state.
* @return {Promise<void>}
* @private
*/
- private async _maybeLoadSessionPool;
- addListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
- on(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
- once(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
- removeListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
- off(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
- removeAllListeners(event?: string | symbol): SessionPool;
- setMaxListeners(n: number): SessionPool;
- prependListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
- prependOnceListener(event: string | symbol, listener: (...args: any[]) => void): SessionPool;
+ _maybeLoadSessionPool(): Promise<void>;
}
export function openSessionPool(sessionPoolOptions: SessionPoolOptions): Promise<SessionPool>;
/**
* Factory user-function which creates customized {@link Session} instances.
*/
-export type CreateSession = (sessionPool: SessionPool) => any;
+export type CreateSession = (sessionPool: SessionPool) => Session;
export type SessionPoolOptions = {
/**
* - Maximum size of the pool.
@@ -209,6 +203,6 @@ export type SessionPoolOptions = {
*/
createSessionFunction?: CreateSession;
};
-import { EventEmitter } from "node/events";
+import { EventEmitter } from "events";
import { Session } from "./session";
import { SessionOptions } from "./session";
diff --git a/types/session_pool/session_utils.d.ts b/types/session_pool/session_utils.d.ts
index 303d4905c..2e79425a2 100644
--- a/types/session_pool/session_utils.d.ts
+++ b/types/session_pool/session_utils.d.ts
@@ -1 +1,4 @@
-export function getCookiesFromResponse(response: any): any[];
+/// <reference types="node" />
+export function getCookiesFromResponse(response: IncomingMessage | PuppeteerResponse): any[] | undefined;
+import { IncomingMessage } from "http";
+import { Response as PuppeteerResponse } from "puppeteer";
diff --git a/types/typedefs.d.ts b/types/typedefs.d.ts
index b4f4b8884..3c48b3d81 100644
--- a/types/typedefs.d.ts
+++ b/types/typedefs.d.ts
@@ -1,8 +1,3 @@
-/**
- * `CherioStatic`, the return type of `cheerio.load()`, is not exported from `@types/cheerio`, so it can not be imported and aliased, it is a global
- * type. We reference the type and redefine it with a bettter name here.
- */
-export type Cheerio = CheerioStatic;
/**
* Represents information about an actor run, as returned by the
* {@link Apify#call} or {@link Apify#callTask} function.
diff --git a/types/utils.d.ts b/types/utils.d.ts
index 5a304a667..896ee376d 100644
--- a/types/utils.d.ts
+++ b/types/utils.d.ts
@@ -1,3 +1,4 @@
+/// <reference types="node" />
export function newClient(): any;
export function logSystemInfo(): void;
/**
@@ -30,9 +31,9 @@ export function getMemoryInfo(): Promise<MemoryInfo>;
export function isPromise(maybePromise: any): boolean;
export function isProduction(): boolean;
export function ensureDirExists(dirPath: any): any;
-export function getFirstKey(dict: any): string;
+export function getFirstKey(dict: any): string | undefined;
export function getTypicalChromeExecutablePath(): string;
-export function addTimeoutToPromise<T>(promise: Promise<T>, timeoutMillis: number, errorMessage: string): Promise<T>;
+export function addTimeoutToPromise(promise: Promise<any>, timeoutMillis: number, errorMessage: string): Promise<any>;
export function isAtHome(): boolean;
export function sleep(millis: number): Promise<void>;
export function openLocalStorage(idOrName: any, defaultIdEnvVar: any, LocalClass: any, cache: any): Promise<any>;
@@ -40,7 +41,7 @@ export function openRemoteStorage(idOrName: any, defaultIdEnvVar: any, RemoteCla
export function ensureTokenOrLocalStorageEnvExists(storageName: any): void;
export function snakeCaseToCamelCase(snakeCaseStr: string): string;
export function printOutdatedSdkWarning(): void;
-export function parseContentTypeFromResponse(response: any): {
+export function parseContentTypeFromResponse(response: IncomingMessage): {
type: string;
charset: string;
};
@@ -81,6 +82,7 @@ export type MemoryInfo = {
*/
childProcessesBytes: number;
};
+import { IncomingMessage } from "http";
declare function downloadListOfUrls({ url, encoding, urlRegExp }: {
url: string;
encoding?: string;
@@ -104,7 +106,8 @@ declare const URL_NO_COMMAS_REGEX: RegExp;
* @memberOf utils
*/
declare const URL_WITH_COMMAS_REGEX: RegExp;
-declare function createRequestDebugInfo(request: any, response?: IncomingMessage | PuppeteerResponse, additionalFields?: any): any;
-import { IncomingMessage } from "http";
+declare function createRequestDebugInfo(request: Request | RequestOptions, response?: IncomingMessage | PuppeteerResponse | undefined, additionalFields?: Object | undefined): any;
+import Request from "./request";
+import { RequestOptions } from "./request";
import { Response as PuppeteerResponse } from "puppeteer";
export {};
diff --git a/types/utils_log.d.ts b/types/utils_log.d.ts
new file mode 100644
index 000000000..e85489005
--- /dev/null
+++ b/types/utils_log.d.ts
@@ -0,0 +1,35 @@
+/// <reference path="../types-apify/apify-shared/log.d.ts" />
+export default Log;
+export type LoggerOptions = {
+ /**
+ * Sets the log level to the given value, preventing messages from less important log levels
+ * from being printed to the console. Use in conjunction with the `log.LEVELS` constants.
+ */
+ level?: number;
+ /**
+ * Max depth of data object that will be logged. Anything deeper than the limit will be stripped off.
+ */
+ maxDepth?: number;
+ /**
+ * Max length of the string to be logged. Longer strings will be truncated.
+ */
+ maxStringLength?: number;
+ /**
+ * Prefix to be prepended the each logged line.
+ */
+ prefix?: string;
+ /**
+ * Suffix that will be appended the each logged line.
+ */
+ suffix?: string;
+ /**
+ * Logger implementation to be used. Default one is log.LoggerText to log messages as easily readable
+ * strings. Optionally you can use `log.LoggerJson` that formats each log line as a JSON.
+ */
+ logger?: Object;
+ /**
+ * Additional data to be added to each log line.
+ */
+ data?: Object;
+};
+import * as Log from "apify-shared/log";
diff --git a/types/utils_request.d.ts b/types/utils_request.d.ts
index f2bc3e54a..d6a61ba79 100644
--- a/types/utils_request.d.ts
+++ b/types/utils_request.d.ts
@@ -1,6 +1,7 @@
+/// <reference types="node" />
export const FIREFOX_MOBILE_USER_AGENT: "Mozilla/5.0 (Android; Mobile; rv:14.0) Gecko/14.0 Firefox/14.0";
export const FIREFOX_DESKTOP_USER_AGENT: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0";
-export function requestAsBrowser(options: RequestAsBrowserOptions): Promise<any>;
+export function requestAsBrowser(options: RequestAsBrowserOptions): Promise<Readable | IncomingMessage>;
/**
* [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.
*/
@@ -18,7 +19,7 @@ export type RequestAsBrowserOptions = {
* with headers that are typically added by websites, such as cookies. Overriding
* default browser headers will remove the masking this function provides.
*/
- headers?: any;
+ headers?: Object;
/**
* An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.
*/
@@ -40,5 +41,8 @@ export type RequestAsBrowserOptions = {
* If function returns true request gets aborted. This function is passed to the
* [
*/
- abortFunction?: Function;
+ abortFunction?: AbortFunction;
};
+export type AbortFunction = (response: IncomingMessage) => boolean;
+import { Readable } from "stream";
+import { IncomingMessage } from "http";
diff --git a/types/utils_social.d.ts b/types/utils_social.d.ts
index aee56f930..5ec254edb 100644
--- a/types/utils_social.d.ts
+++ b/types/utils_social.d.ts
@@ -49,7 +49,7 @@ declare function emailsFromText(text: string): string[];
declare function emailsFromUrls(urls: string[]): string[];
declare function phonesFromText(text: string): string[];
declare function phonesFromUrls(urls: string[]): string[];
-declare function parseHandlesFromHtml(html: string, data?: any): SocialHandles;
+declare function parseHandlesFromHtml(html: string, data?: Object): SocialHandles;
/**
* Regular expression to exactly match a single email address.
* It has the following form: `/^...$/i`.
diff --git a/website/i18n/en.json b/website/i18n/en.json
index 2eca8a47d..a245c3873 100644
--- a/website/i18n/en.json
+++ b/website/i18n/en.json
@@ -8,6 +8,9 @@
"api/apify": {
"title": "Apify"
},
+ "api/apify-call-error": {
+ "title": "ApifyCallError"
+ },
"api/autoscaled-pool": {
"title": "AutoscaledPool"
},
@@ -17,6 +20,9 @@
"api/cheerio-crawler": {
"title": "CheerioCrawler"
},
+ "api/cookie-parse-error": {
+ "title": "CookieParseError"
+ },
"api/dataset": {
"title": "Dataset"
},
@@ -134,6 +140,9 @@
"readme/support": {
"title": "readme/support"
},
+ "typedefs/abort-function": {
+ "title": "AbortFunction"
+ },
"typedefs/actor-run": {
"title": "ActorRun"
},
@@ -185,12 +194,18 @@
"typedefs/handle-request-inputs": {
"title": "HandleRequestInputs"
},
+ "typedefs/intercept-handler": {
+ "title": "InterceptHandler"
+ },
"typedefs/key-consumer": {
"title": "KeyConsumer"
},
"typedefs/launch-puppeteer": {
"title": "LaunchPuppeteer"
},
+ "typedefs/launch-puppeteer-function": {
+ "title": "LaunchPuppeteerFunction"
+ },
"typedefs/launch-puppeteer-options": {
"title": "LaunchPuppeteerOptions"
},
@@ -230,6 +245,9 @@
"typedefs/request-as-browser-options": {
"title": "RequestAsBrowserOptions"
},
+ "typedefs/request-list-input": {
+ "title": "RequestListInput"
+ },
"typedefs/request-list-options": {
"title": "RequestListOptions"
},
@@ -257,6 +275,9 @@
"typedefs/social-handles": {
"title": "SocialHandles"
},
+ "typedefs/source-input": {
+ "title": "SourceInput"
+ },
"typedefs/stealth-options": {
"title": "StealthOptions"
},
@@ -265,6 +286,9 @@
},
"typedefs/system-status-options": {
"title": "SystemStatusOptions"
+ },
+ "typedefs/user-func": {
+ "title": "UserFunc"
}
},
"links": {