Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated docs and README

Jan Curn334cd8754992
diff --git a/README.md b/README.md
index 2d38b7be9..21a244971 100644
--- a/README.md
+++ b/README.md
@@ -718,7 +718,7 @@ Bridge, takes a screenshot of the page and saves it as output.
This actor
can be invoked synchronously using a single HTTP request to directly obtain its output as a reponse,
using the [Run actor synchronously](https://www.apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input)
-API endpoint.
+Apify API endpoint.
The example is also shared as the [apify/example-golden-gate-webcam](https://www.apify.com/apify/example-golden-gate-webcam)
actor in the Apify library, so you can test it directly there simply by sending a POST request to
https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]
diff --git a/examples/synchronous_run.js b/examples/synchronous_run.js
index e1716e2e0..8c9ad2edb 100644
--- a/examples/synchronous_run.js
+++ b/examples/synchronous_run.js
@@ -6,7 +6,7 @@
* This actor
* can be invoked synchronously using a single HTTP request to directly obtain its output as a reponse,
* using the [Run actor synchronously](https://www.apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input)
- * API endpoint.
+ * Apify API endpoint.
* The example is also shared as the [apify/example-golden-gate-webcam](https://www.apify.com/apify/example-golden-gate-webcam)
* actor in the Apify library, so you can test it directly there simply by sending a POST request to
* https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]
diff --git a/src/actor.js b/src/actor.js
index a7ce648e1..90706ef6c 100644
--- a/src/actor.js
+++ b/src/actor.js
@@ -148,7 +148,7 @@ export const getEnv = () => {
* });
* ```
*
- * @param userFunc {Function} User function to be executed. If it returns a promise,
+ * @param {Function} userFunc User function to be executed. If it returns a promise,
* the promise will be awaited. The user function is called with no arguments.
*
* @memberof module:Apify
@@ -216,7 +216,7 @@ export const main = (userFunc) => {
*
* The result of the function is an object that contains details about the actor run and its output (if any).
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* const run = await Apify.call('apify/hello-world', { myInput: 123 });
@@ -266,7 +266,7 @@ export const main = (userFunc) => {
* ```
* Internally, the `Apify.call` function calls the
* <a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor</a>
- * API endpoint and few others to obtain the output.
+ * Apify API endpoint and few others to obtain the output.
*
* @param {String} actId
* Either `username/actor-name` or actor ID.
diff --git a/src/autoscaling/autoscaled_pool.js b/src/autoscaling/autoscaled_pool.js
index 13267f77a..300be9257 100644
--- a/src/autoscaling/autoscaled_pool.js
+++ b/src/autoscaling/autoscaled_pool.js
@@ -42,7 +42,7 @@ const DEFAULT_OPTIONS = {
* The pool evaluates whether it should start a new task every time one of the tasks finishes
* and also in the interval set by the `options.maybeRunIntervalSecs` parameter.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* const pool = new Apify.AutoscaledPool({
diff --git a/src/basic_crawler.js b/src/basic_crawler.js
index d0d4bc3c8..c4331c7df 100644
--- a/src/basic_crawler.js
+++ b/src/basic_crawler.js
@@ -37,7 +37,7 @@ const DEFAULT_OPTIONS = {
* of the `CheerioCrawler` constructor.
* For user convenience, the `minConcurrency` and `maxConcurrency` options are available directly in the constructor.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* const rp = require('request-promise');
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.js
index 0929e428c..a16b61795 100644
--- a/src/cheerio_crawler.js
+++ b/src/cheerio_crawler.js
@@ -45,7 +45,7 @@ const DEFAULT_OPTIONS = {
* of the `CheerioCrawler` constructor.
* For user convenience, the `minConcurrency` and `maxConcurrency` options are available directly.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* // Prepare a list of URLs to crawl
diff --git a/src/dataset.js b/src/dataset.js
index 49b4e45bb..d37d96920 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -129,7 +129,7 @@ export const chunkBySize = (items, limitBytes) => {
* If the `APIFY_TOKEN` environment variable is provided instead, the data is stored
* in the [Apify Dataset](https://www.apify.com/docs/storage#dataset) cloud storage.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* // Write a single row to the default dataset
@@ -272,7 +272,7 @@ export class Dataset {
* "itemsCount": 0
* }
*
- * @param opts
+ * @param {Object} opts
* @returns {Promise}
*/
getInfo(opts = {}) {
diff --git a/src/events.js b/src/events.js
index 3bfa9657d..30d23c71e 100644
--- a/src/events.js
+++ b/src/events.js
@@ -29,7 +29,7 @@ let persistStateInterval = null;
* that emits various events from the SDK or the Apify platform.
* The event emitter is initialized by calling <a href="#module-Apify-main"><code>Apify.main()</code></a> function.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* Apify.main(async () => {
diff --git a/src/key_value_store.js b/src/key_value_store.js
index 1f31930e4..43c2e6ea8 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -92,17 +92,39 @@ export const maybeStringify = (value, options) => {
return value;
};
-
/**
- * The `KeyValueStore` class provides a simple interface to the [Apify Key-value stores](https://www.apify.com/docs/storage#key-value-store).
- * You should not instantiate this class directly, use the
- * [Apify.openKeyValueStore()](#module-Apify-openKeyValueStore) function.
+ * The `KeyValueStore` class represents a key-value store, a simple data storage that is used
+ * for saving and reading data records or files. Each data record is
+ * represented by a unique key and associated with a MIME content type. Key-value stores are ideal
+ * for saving screenshots, actor inputs and outputs, web pages, PDFs or to persist the state of crawlers.
+ *
+ * Do not instantiate this class directly, use the
+ * {@link Apify#openKeyValueStore|`Apify.openKeyValueStore()`} function instead.
+ *
+ * Each actor run is associated with a default key-value store, which is created exclusively
+ * for the run. By convention, the actor input and output is stored into the
+ * default key-value store under the `INPUT` and `OUTPUT` key, respectively.
+ * Typically the input and output is a JSON file, although it can be any other format.
+ * To access the default key-value store directly, you can use the
+ * {@linkcode Apify.getValue|Apify.getValue()}
+ * and {@linkcode Apify.getValue|Apify.setValue()} convenience functions.
+ *
+ * `KeyValueStore` stores its data either on local disk or in the Apify cloud,
+ * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable is set.
+ *
+ * If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in
+ * the local directory in the following files:
+ * ```
+ * [APIFY_LOCAL_STORAGE_DIR]/key_value_stores/[STORE_ID]/[KEY].[EXT]
+ * ```
+ * Note that `[STORE_ID]` is the name or ID of the key-value store. The default key value store has ID `default`,
+ * unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.
+ * The `[KEY]` is the key of the record and `[EXT]` corresponds to the MIME content type of the data value.
*
- * The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
- * in the Apify cloud (see [Key-value stores documentation](https://www.apify.com/docs/storage#key-value-store) when actor is running on Apify
- * platform or if `APIFY_TOKEN` environment variable is set.
+ * If the `APIFY_TOKEN` environment variable is provided instead, the data is stored
+ * in the [Apify Dataset](https://www.apify.com/docs/storage#dataset) cloud storage.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* // Get actor input from the default key-value store
@@ -114,20 +136,20 @@ export const maybeStringify = (value, options) => {
* // Open a named key-value store
* const store = await Apify.openKeyValueStore('some-name');
*
- * // Write record. JavaScript object is automatically converted to JSON,
+ * // Write a record. JavaScript object is automatically converted to JSON,
* // strings and binary buffers are stored as they are
* await store.setValue('some-key', { foo: 'bar' });
*
- * // Read record. Note that JSON is automatically parsed to a JavaScript object,
+ * // Read a record. Note that JSON is automatically parsed to a JavaScript object,
* // text data returned as a string and other data is returned as binary buffer
* const value = await store.getValue('some-key');
*
* // Delete record
* await store.delete('some-key');
* ```
- *
- * @param {String} storeId - ID of the key-value store.
* @hideconstructor
+ * @see {@linkcode Apify.getValue|Apify.getValue()}
+ * @see {@linkcode Apify.setValue|Apify.setValue()}
*/
export class KeyValueStore {
constructor(storeId) {
@@ -136,33 +158,93 @@ export class KeyValueStore {
this.storeId = storeId;
}
- // TODO: Move here the Apify.getValue()/setValue() documentation, and link it from there.
- // This place should be the main source of information.
-
/**
- * Gets a record from the current key-value store using its key.
- * For more details, see [Apify.getValue](#module-Apify-getValue).
+ * Gets a value from the key-value store.
*
- * @param {String} key Record key.
- * @return {Promise}
+ * The function returns a promise that resolves to the record value,
+ * whose JavaScript type depends on the MIME content type of the record.
+ * Records with the `application/json`
+ * content type are automatically parsed and returned as a JavaScript object.
+ * Similarly, records with `text/plain` content types are returned as a string.
+ * For all other content types, the value is returned as a raw
+ * <a href="https://nodejs.org/api/buffer.html"><code>Buffer</code></a> instance.
+ *
+ * If the record does not exist, the functions resolves to `null`.
+ *
+ * To save or delete a value in the key-value store, use the
+ * {@linkcode KeyValueStore#setValue|KeyValueStore.setValue()} function.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const store = await Apify.openKeyValueStore('my-screenshots');
+ * const buffer = await store.getValue('screenshot1.png');
+ * ```
+ *
+ * @param {String} key
+ * Key of the record.
+ * @returns {Promise}
+ * Returns a promise that resolves to an object, string
+ * or <a href="https://nodejs.org/api/buffer.html"><code>Buffer</code></a>, depending
+ * on the MIME content type of the record.
*/
getValue(key) {
validateGetValueParams(key);
+ // TODO: Perhaps we should add options.contentType or options.asBuffer/asString
+ // to enforce the representation of value
+
return keyValueStores
.getRecord({ storeId: this.storeId, key })
.then(output => (output ? output.body : null));
}
/**
- * Stores a record to the key-value stores.
- * The function has no result, but throws on invalid arguments or other errors.
+ * Saves or deleted a record to the key-value store.
+ * The function returns a promise that resolves once the record has been saved or deleted.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const store = await Apify.openKeyValueStore('my-store');
+ * await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });
+ * ```
+ *
+ * By default, `value` is converted to JSON and stored with the
+ * `application/json; charset=utf-8` MIME content type.
+ * To store the value with another content type, pass it in the options as follows:
+ * ```javascript
+ * const store = await Apify.openKeyValueStore('my-store');
+ * await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });
+ * ```
+ * If you set custom content type, `value` must be either a string or
+ * <a href="https://nodejs.org/api/buffer.html"><code>Buffer</code></a>, otherwise an error will be thrown.
+ *
+ * If `value` is null, the record is deleted instead. Note that the `setValue()` function succeeds
+ * regardless whether the record existed or not.
+ *
+ * To retrieve a value from the key-value store, use the
+ * {@linkcode KeyValueStore#setValue|KeyValueStore.getValue()} function.
+ *
+ * **IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`,
+ * otherwise the actor process might finish before the value is stored!
+ *
+ * @param {String} key
+ * Unique record key.
+ * @param {Object|String|Buffer} value
+ * Record data, which can be one of the following values:
+ * <ul>
+ * <li>If `null`, the record in the key-value store is deleted.</li>
+ * <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
+ * <li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`
+ * or <a href="https://nodejs.org/api/buffer.html"><code>Buffer</code></a>.</li>
+ * </ul>
+ * For any other value an error will be thrown.
+ * @param {Object} [options]
+ * @param {String} [options.contentType]
+ * Specifies a custom MIME content type of the record.
+ * @returns {Promise} Returns a promise that resolves once the value is stored or deleted.
*
- * @param {String} key Record key.
- * @param {Object|String|Buffer} value Record value. If content type is not provided then the value is stringified to JSON.
- * @param {Object} [Options]
- * @param {Object} [Options.contentType] Content type of the record.
- * @return {Promise}
*/
setValue(key, value, options = {}) {
validateSetValueParams(key, value, options);
@@ -203,7 +285,7 @@ export class KeyValueStore {
/**
* Helper to create a file-matching RegExp from a KeyValueStore key.
- * @param key
+ * @param {String} key
* @returns {RegExp}
* @ignore
*/
@@ -357,28 +439,21 @@ const getOrCreateKeyValueStore = (storeIdOrName) => {
});
};
+
/**
- * Opens a key-value store and returns a promise resolving to an instance
- * of the [KeyValueStore](#KeyValueStore) class.
- *
- * Key-value store is a simple storage for records, where each record has a unique key.
- * For more information, see [Key-value store documentation](https://www.apify.com/docs/storage#dataset).
- *
- * The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
- * in the Apify cloud (see [Key-value stores documentation](https://www.apify.com/docs/storage#key-value-store) when actor is running on Apify
- * platform or if `APIFY_TOKEN` environment variable is set.
- *
- * Example usage:
+ * Opens a key-value store and returns a promise resolving to an instance of the {@linkcode KeyValueStore} class.
*
- * ```javascript
- * const store = await Apify.openKeyValueStore('my-store-id');
- * await store.setValue('some-key', { foo: 'bar' });
- * ```
+ * Key-value stores are used to store records or files, along with their MIME content type.
+ * The records are stored and retrieved using a unique key.
+ * The actual data is stored either on a local filesystem or in the Apify cloud.
*
- * @param {string} storeIdOrName ID or name of the key-value store to be opened. If no value is
- * provided then the function opens the default key-value store associated with the actor run.
- * @returns {Promise<KeyValueStore>} Returns a promise that resolves to a KeyValueStore object.
+ * For more details and code examples, see the {@linkcode KeyValueStore} class.
*
+ * @param {string} [storeIdOrName]
+ * ID or name of the key-value store to be opened. If `null` or `undefined`,
+ * the function returns the default key-value store associated with the actor run.
+ * @returns {Promise<KeyValueStore>}
+ * Returns a promise that resolves to an instance of the `KeyValueStore` class.
* @memberof module:Apify
* @name openKeyValueStore
* @instance
@@ -393,36 +468,32 @@ export const openKeyValueStore = (storeIdOrName) => {
: openRemoteStorage(storeIdOrName, ENV_VARS.DEFAULT_KEY_VALUE_STORE_ID, KeyValueStore, storesCache, getOrCreateKeyValueStore);
};
+
/**
- * Gets a value from the default key-value store for the current actor run using the Apify API.
- * The key-value store is created automatically for each actor run
- * and its ID is passed by the Actor platform in the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.
- * It is used to store input and output of the actor under keys named `INPUT` and `OUTPUT`, respectively.
- * However, the store can be used for storage of any other values under arbitrary keys.
- *
- * Example usage:
+ * Gets a value from the default {@linkcode KeyValueStore} associated with the current actor run.
*
+ * This is just a convenient shortcut for {@link KeyValueStore#getValue|`KeyValueStore.getValue()`}.
+ * For example, calling the following code:
* ```javascript
* const input = await Apify.getValue('INPUT');
+ * ```
*
- * console.log('My input:');
- * console.dir(input);
+ * is equivalent to:
+ * ```javascript
+ * const store = await Apify.openKeyValueStore();
+ * await store.getValue('INPUT');
* ```
*
- * The result of the function is the body of the record. Bodies with the `application/json`
- * content type are automatically parsed to an object.
- * Similarly, for `text/plain` content types the body is parsed as `String`.
- * For all other content types, the body is a raw `Buffer`.
- * If the record cannot be found, the result is null.
+ * To store the value to the default-key value store, you can use the {@link Apify#setValue|`Apify.setValue()`} function.
*
- * If the actor is not running on Apify platform and `APIFY_LOCAL_STORAGE_DIR` environment variable is set
- * the value is read from a local directory rather than the Apify key-value store cloud storage, specifically
- * from a file that has the key as a name. If file does not exists, the returned value is `null`. The file
- * will get extension based on it's content type. This feature is useful for local development and debugging
- * of your actors.
+ * For more information, see {@link Apify.openKeyValueStore|`Apify.openKeyValueStore()`}
+ * and {@link KeyValueStore#getValue|`KeyValueStore.getValue()`}.
*
- * @param {String} key Key of the record.
- * @returns {Promise} Returns a promise.
+ * @param {String} key
+ * Unique record key.
+ * @returns {Promise}
+ * Returns a promise that resolves once the record is stored.
+ * @see {@link KeyValueStore}
*
* @memberof module:Apify
* @name getValue
@@ -431,47 +502,46 @@ export const openKeyValueStore = (storeIdOrName) => {
*/
export const getValue = key => openKeyValueStore().then(store => store.getValue(key));
+
/**
- * Stores a value in the default key-value store for the current actor run using the Apify API.
- * The data is stored in the key-value store created specifically for the actor run,
- * whose ID is defined in the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.
- * The function has no result, but throws on invalid args or other errors.
+ * Stores or deletes a value in the default {@linkcode KeyValueStore} associated with the current actor run.
*
+ * This is just a convenient shortcut for {@link KeyValueStore#setValue|`KeyValueStore.setValue()`}.
+ * For example, calling the following code:
* ```javascript
- * await Apify.setValue('OUTPUT', { someValue: 123 });
+ * await Apify.setValue('OUTPUT', { foo: "bar" });
* ```
*
- * By default, `value` is converted to JSON and stored with the `application/json; charset=utf-8` content type.
- * To store a value with another content type, pass it in the options as follows:
+ * is equivalent to:
* ```javascript
- * await Apify.setValue('OUTPUT', 'my text data', { contentType: 'text/plain' });
+ * const store = await Apify.openKeyValueStore();
+ * await store.setValue('OUTPUT', { foo: "bar" });
* ```
- * In this case, the value must be a string or Buffer.
- *
- * If the actor is not running on Apify platform and `APIFY_LOCAL_STORAGE_DIR` environment variable is set
- * the value is written a local directory rather than the Apify key-value store cloud storage, specifically
- * from a file that has the key as a name. If file does not exists, the returned value is `null`. The file
- * will get extension based on it's content type. This feature is useful for local development and debugging
- * of your actors.
- *
- * **IMPORTANT:** Make sure to use the `await` keyword when calling `Apify.setValue()`,
- * otherwise the actor process might finish before the value is stored!
- *
- * @param key Key of the record
- * @param value Value of the record:
- * <ul>
- * <li>If `null`, the record in the key-value store is deleted.</li>
- * <li>If no `options.contentType` is specified, `value` can be any object and it will be stringified to JSON.</li>
- * <li>If `options.contentType` is specified, `value` is considered raw data and it must be a String or Buffer.</li>
- * </ul>
- * For any other value an error will be thrown.
*
+ * To get a value from the default-key value store, you can use the {@link Apify#getValue|`Apify.getValue()`} function.
+ *
+ * For more information, see {@link Apify.openKeyValueStore|`Apify.openKeyValueStore()`}
+ * and {@link KeyValueStore#setValue|`KeyValueStore.setValue()`}.
+ *
+ * @param {String} key
+ * Unique record key.
+ * @param {Object|String|Buffer} value
+ * Record data, which can be one of the following values:
+ * <ul>
+ * <li>If `null`, the record in the key-value store is deleted.</li>
+ * <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
+ * <li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`
+ * or <a href="https://nodejs.org/api/buffer.html"><code>Buffer</code></a>.</li>
+ * </ul>
+ * For any other value an error will be thrown.
* @param {Object} [options]
- * @param {String} [options.contentType] - Sets the MIME content type of the value.
- * @returns {Promise} Returns a promise that resolves to the value.
+ * @param {String} [options.contentType]
+ * Specifies a custom MIME content type of the record.
+ * @returns {Promise} Returns a promise that resolves once the value is stored or deleted.
+ * @see {@link KeyValueStore}
*
* @memberof module:Apify
- * @name setValue
+ * @name getValue
* @instance
* @function
*/
diff --git a/src/pseudo_url.js b/src/pseudo_url.js
index 817db29d6..6f8193910 100644
--- a/src/pseudo_url.js
+++ b/src/pseudo_url.js
@@ -61,7 +61,7 @@ const parsePurl = (purl) => {
* <li>`http://www.example.com/pages/something`</li>
* </ul>
*
- * Example use:
+ * **Example usage:**
*
* ```javascript
* const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]');
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index ce5956ebc..570978452 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -27,7 +27,7 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;
* New tasks are only started if there is enough free CPU and memory available,
* using the `AutoscaledPool` class internally.
*
- * Basic usage:
+ * **Example usage:**
*
* ```javascript
* const crawler = new Apify.PuppeteerCrawler({
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index 483b03a9a..be016e675 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -36,7 +36,8 @@ const DISK_CACHE_DIR = path.join(os.tmpdir(), 'puppeteer_disk_cache-');
/**
* Deletes Chrome's user data directory
- * @param diskCacheDir
+ * @param {String} diskCacheDir
+ * @ignore
*/
const deleteDiskCacheDir = (diskCacheDir) => {
log.debug('PuppeteerPool: Deleting disk cache directory', { diskCacheDir });
@@ -76,7 +77,7 @@ class PuppeteerInstance {
*
* `PuppeteerPool` is internally used by the {@link PuppeteerCrawler|`PuppeteerCrawler`} class.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* const puppeteerPool = new PuppeteerPool({
diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.js
index 01f7a82e9..f72a5ede8 100644
--- a/src/puppeteer_utils.js
+++ b/src/puppeteer_utils.js
@@ -134,13 +134,18 @@ const enqueueRequestsFromClickableElements = async (page, selector, purls, reque
* To create a Request object function uses `requestTemplate` from a matching Pseudo-URL.
*
* *WARNING*: This is work in progress. Currently the function doesn't click elements and only takes their `href` attribute and so
- * is working only for link (`a`) elements and not for buttons or javascript links.
+ * is working only for link (`a`) elements, but not for buttons or JavaScript links.
*
- * @param {Page} page Puppeteer [Page](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page) object.
- * @param {String} selector CSS selector matching elements to be clicked.
- * @param {Array} pseudoUrls An array of `Apify.PseudoUrl` objects matching URL to be enqueued.
- * @param {RequestQueue} requestQueue `Apify.RequestQueue` object where URLs will be enqueued.
- * @return {Promise} Promise resolves to array of RequestOperationInfo objects.
+ * @param {Page} page
+ * Puppeteer [Page](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page) object.
+ * @param {String} selector
+ * CSS selector matching elements to be clicked.
+ * @param {PseudoUrl[]} pseudoUrls
+ * An array of pseudo-URLs matching the URLs to be enqueued.
+ * @param {RequestQueue} requestQueue
+ * Request queue object where URLs will be enqueued.
+ * @return {Promise<RequestOperationInfo[]>}
+ * Promise that resolves to an array of {@linkcode RequestOperationInfo} objects.
* @memberof utils.puppeteer
*/
const enqueueLinks = async (page, selector, purls, requestQueue) => {
@@ -181,6 +186,7 @@ const enqueueLinks = async (page, selector, purls, requestQueue) => {
* @param {Page} page Puppeteer's `Page` object
* @param {Array<String>} resourceTypes Array of resource types to block.
* @return {Promise<void>}
+ * @memberof utils.puppeteer
*/
const blockResources = async (page, resourceTypes = ['stylesheet', 'font', 'image', 'media']) => {
await page.setRequestInterception(true);
@@ -212,6 +218,7 @@ const blockResources = async (page, resourceTypes = ['stylesheet', 'font', 'imag
*
* @param {String} scriptString
* @return {Function} async ({ page, request }) => { scriptString }
+ * @memberof utils.puppeteer
*/
const compileScript = (scriptString) => {
const funcString = `async ({ page, request }) => {${scriptString}}`;
@@ -233,7 +240,8 @@ const compileScript = (scriptString) => {
/**
* A namespace that contains various Puppeteer utilities.
*
- * Example usage:
+ * **Example usage:**
+ *
* ```javascript
* const Apify = require('apify');
*
diff --git a/src/request_list.js b/src/request_list.js
index 4135fd9bc..bc6b467ae 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -44,7 +44,7 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* so that if the Node.js process is restarted,
* the crawling can continue where it left off. For more details, see {@link KeyValueStore|`KeyValueStore`}.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* const requestList = new Apify.RequestList({
diff --git a/src/request_queue.js b/src/request_queue.js
index 5280c3731..a32ec6c07 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -151,7 +151,7 @@ const getRequestId = (uniqueKey) => {
* If the `APIFY_TOKEN` environment variable is provided instead, the data is stored
* in the [Apify Request Queue](https://www.apify.com/docs/storage#queue) cloud storage.
*
- * Example usage:
+ * **Example usage:**
*
* ```javascript
* // Open the default request queue associated with the actor run
@@ -242,7 +242,7 @@ export class RequestQueue {
/**
* Gets the request from the queue specified by ID.
*
- * @param {String} requestId Request ID
+ * @param {String} requestId Request ID
* @return {Promise<Request>}
*/
getRequest(requestId) {
diff --git a/src/utils.js b/src/utils.js
index 06a972781..2b431cf5f 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -86,8 +86,8 @@ export const newPromise = () => {
/**
* Adds charset=utf-8 to given content type if this parameter is missing.
*
- * @param contentType
- * @returns {string}
+ * @param {String} contentType
+ * @returns {String}
*
* @ignore
*/
@@ -140,7 +140,7 @@ export const isDocker = (forceReset) => {
/**
* Sums an array of numbers.
*
- * @param {Array} arr An array of numbers.
+ * @param {Array} arr An array of numbers.
* @return {Number} Sum of the numbers.
*
* @ignore
@@ -150,7 +150,7 @@ export const sum = arr => arr.reduce((total, c) => total + c, 0);
/**
* Computes an average of an array of numbers.
*
- * @param {Array} arr An array of numbers.
+ * @param {Array} arr An array of numbers.
* @return {Number} Average value.
*
* @ignore
@@ -163,6 +163,8 @@ export const avg = arr => sum(arr) / arr.length;
* @param {Array} arrValues
* @param {Array} arrWeights
* @return {Number}
+ *
+ * @ignore
*/
export const weightedAvg = (arrValues, arrWeights) => {
const result = arrValues.map((value, i) => {
@@ -340,7 +342,8 @@ export const isAtHome = () => !!process.env[ENV_VARS.IS_AT_HOME];
* Returns a promise that resolves after a specific period of time. This is useful to implement waiting
* in your code, e.g. to prevent overloading of target website or to avoid bot detection.
*
- * Example usage:
+ * **Example usage:**
+ *
* ```javascript
* const Apify = require('apify);
*
@@ -349,7 +352,7 @@ export const isAtHome = () => !!process.env[ENV_VARS.IS_AT_HOME];
* // Sleep 1.5 seconds
* await Apify.utils.sleep(1500);
* ```
- * @param millis Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
+ * @param {Number} millis Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
* @memberof utils
* @return {Promise}
*/
@@ -450,6 +453,7 @@ export const openRemoteStorage = async (idOrName, defaultIdEnvVar, RemoteClass,
/**
* Checks if at least one of APIFY_LOCAL_STORAGE_DIR and APIFY_TOKEN environment variables is set.
+ * @ignore
*/
export const ensureTokenOrLocalStorageEnvExists = (storageName) => {
if (!process.env[ENV_VARS.LOCAL_STORAGE_DIR] && !process.env[ENV_VARS.TOKEN]) {
@@ -460,7 +464,8 @@ export const ensureTokenOrLocalStorageEnvExists = (storageName) => {
/**
* A namespace that contains various utilities.
*
- * Example usage:
+ * **Example usage:**
+ *
* ```javascript
* const Apify = require('apify);
*