Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated docs and README
Jan Curn8dd4561d49fb
diff --git a/README.md b/README.mdindex 43ccb74ed..a5e39c1fa 100644--- a/README.md+++ b/README.md@@ -1,272 +1,15 @@-# apify: Apify runtime for JavaScript+# apify: Apify Actor runtime for JavaScript[](http://badge.fury.io/js/apify)[](https://travis-ci.org/Apifier/apify-runtime-js)-Helper package that simplifies development of Apify acts. See https://www.apify.com/docs#acts for details.-This is still work in progress, things might change and break.+Helper package that simplifies development of acts on the Apify Actor platform.-If you're looking for the API documentation package previously published as **apify**,-please go to [jsdocify](https://www.npmjs.com/package/jsdocify).-Big kudos to [jbalsas](https://www.npmjs.com/~jbalsas) for giving away this package name!---## Installation--```bash-npm install apify --save-```--This package requires Node.js 6 or higher.-It might work with lower versions too, but they are neither tested nor supported.--## Usage inside acts--Import the package to your act.--```javascript-const Apify = require('apify');-```--### Main function--To simplify development of acts, the runtime provides the `Apify.main(func)` function which does the following:--1) Invokes the user function `func`--2) If the function returned a promise, waits for it to resolve--3) Exits the process--If the user function throws an exception or some other error is encountered,-then `Apify.main()` prints the details to console so that they are stored to the log file.--`Apify.main()` accepts a single argument - the user function that performs the operation of the act.-In the simplest case, the user function is synchronous:--```javascript-Apify.main(() => {- // my synchronous function that returns immediately-});-```--If the user function returns a promise, it is considered as asynchronous:--```javascript-const request = require('request-promise');--Apify.main(() => {- // my asynchronous function that returns a promise- return Promise.resolve()- .then(() => {- return request('http://www.example.com');- })- .then((html) => {- console.log(html);- });-});-```--To simplify your code, you can take advantage of the `async`/`await` keywords:--```javascript-const request = require('request-promise');--Apify.main(async () => {- const html = await request('http://www.example.com');- console.log(html);-});-```--Note that the `Apify.main()` function does not need to be used at all,-it is provided merely for user convenience.-----### Input and output--Each act can have an input and output data record, which is raw data-with a specific MIME content type.-Both input and output is stored in the Apify key-value store created specifically for the act run,-under keys named `INPUT` and `OUTPUT`, respectively.-The ID of the key-value store is provided by the Actor runtime as the `APIFY_DEFAULT_KEY_VALUE_STORE_ID`-environment variable.--Use the `Apify.getValue(key, [, callback])` function to obtain the input of your act:--```javascript-const input = await Apify.getValue('INPUT');-console.log('My input:');-console.dir(input);-```--If the input data has the `application/json` content type, it is automatically parsed into a JavaScript object.-For the `text/plain` content type the result is a string.-For other content types, the result is raw Buffer.--Similarly, the output can be stored using the `Apify.setValue(key, value [, options] [, callback])` function as follows:--```javascript-const output = {- someValue: 123-};-await Apify.setValue('OUTPUT', output);-```--By default, the value is converted to JSON and stored with the `application/json` content type.-If you want to store your data with another content type, pass it in the options as follows:--```javascript-await Apify.setValue('OUTPUT', 'my text data', { contentType: 'text/plain' });-```--In this case, the value must be a string or Buffer.--**IMPORTANT: Do not forget to use the `await` keyword when calling `Apify.setValue()`,-otherwise the act process might finish before the output is stored and/or storage errors will not be reported!**--Besides the key `INPUT` and `OUTPUT`, you can use arbitrary keys-to store any data from your act, such as its state or larger results.+Complete documentation is available at https://www.apify.com/docs/js/apify-runtime-js/latest-During local development of the act, you can emulate the key-value store locally using the-`DEV_KEY_VALUE_STORE_DIR` environment variable. Simply set the variable to a path of some local directory,-and the runtime will read and store values in that directory rather than Apify servers.-By default, the values are considered to have the `application/json; charset=utf-8`-content type, but you can override it by setting the `DEV_KEY_VALUE_STORE_CONTENT_TYPE` environment variable.+For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor-### Browser--Apify runtime optionally depends on-the [selenium-webdriver](https://www.npmjs.com/package/selenium-webdriver) package that enables-automation of a web browser.-The simplest way to launch a new web browser is using the `Apify.browse([url,] [options,] [callback])`-function. For example:--```javascript-const browser = await Apify.browse('https://www.example.com/');-```--or--```javascript-const browser = await Apify.browse({- url: 'https://www.example.com/',- userAgent: 'MyCrawlingBot/1.23',-});-```--The `options` parameter controls settings of the web browser and it has the following properties:--```javascript-{- // Initial URL to open. Note that the url argument in Apify.browse() overrides this value.- // The default value is 'about:blank'- url: String,-- // The type of the web browser to use.- // See https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities for possible options.- // The default value is 'chrome', which is currently the only fully-supported browser.- browserName: String,-- // Indicates whether the browser should be opened in headless mode (i.e. without windows).- // By default, this value is based on the APIFY_HEADLESS environment variable.- headless: Boolean,-- // URL of the proxy server, e.g. 'http://username:[email protected]:55555'.- // Currently only the 'http' proxy type is supported.- // By default it is null, which means no proxy server is used.- proxyUrl: String,-- // Overrides the User-Agent HTTP header of the web browser.- // By default it is null, which means the browser uses its default User-Agent.- userAgent: String,-}-```--The result of the `Apify.browse()` is a new instance of the `Browser` class,-which represents a web browser instance (possibly with multiple windows or tabs).-If you pass a Node.js-style callback the `Browser` instance is passed to it,-otherwise the `Apify.browse()` function returns a promise that resolves to the `Browser` instance.--The `Browser` class has the following properties:--```javascript-{- // An instance of the Selenium's WebDriver class.- webDriver: Object,-- // A method that closes the web browser and releases associated resources.- // The method has no arguments and returns a promise that resolves when the browser was closed.- close: Function,-}-```--The `webDriver` property can be used to manipulate the web browser:--```javascript-const url = await browser.webDriver.getCurrentUrl();-```--For more information, see [WebDriver documentation](http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_WebDriver.html).--When the web browser is no longer needed, it should be closed:--```javascript-await browser.close();-```---### Promises--By default, the `getValue`, `setValue` and `browse` functions return a promise.-However, they also accept a Node.js-style callback parameter.-If the callback is provided, the return value of the functions is not defined-and the functions only invoke the callback upon completion or error.--To set a promise dependency from an external library, use a code such as:--```javascript-const Promise = require('bluebird');-Apify.setPromisesDependency(Promise);-```--If `Apify.setPromisesDependency()` is not called, the runtime defaults to-native promises if they are available, or it throws an error.---### Miscellaneous--The `Apify.client` property contains a reference to the `ApifyClient` instance-(from the [apify-client](https://www.npmjs.com/package/apify-client) NPM package),-that is used for all underlying calls to the Apify API.-The instance is created when the `apify` package is first imported-and it is configured using the `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN`-environment variables.-The default settings of the instance can be overridden by calling `Apify.client.setOptions()` function.--`Apify.events` property contains a reference to an `EventEmitter` instance-that is used by Actor runtime to notify your process about various events.-This will be used in the future.--`Apify.call` function can be used to quickly execute other act and get it's output. Example use:--```javascript-const run = await Apify.call('john23/my-favourite-act', { myInput: 123 });-console.dir(run.output);-```---## Package maintenance--* `npm run test` to run tests-* `npm run test-cov` to generate test coverage-* `npm run build` to transform ES6/ES7 to ES5 by Babel-* `npm run clean` to clean `build/` directory-* `npm run lint` to lint js using ESLint in Airbnb's Javascript style-* `npm publish` to run Babel, run tests and publish the package to NPM--## License--Apache 2.0+If you're looking for the API documentation package previously published as **apify**,+please go to [jsdocify](https://www.npmjs.com/package/jsdocify).+Big kudos to [jbalsas](https://www.npmjs.com/~jbalsas) for giving away this package name!diff --git a/src/actor.js b/src/actor.jsindex 73b85fccc..533bbceae 100644--- a/src/actor.js+++ b/src/actor.js@@ -14,12 +14,18 @@ import { getPromisePrototype, newPromise, nodeifyPromise, newClient, addCharsetT* @memberof module:Apify* @name client* @instance- * @description A default instance of the `ApifyClient` class provided+ * @description <p>A default instance of the `ApifyClient` class provided* by the {@link https://www.apify.com/docs/js/apify-client-js/latest|apify-client} NPM package.- * This instance is used to access the Apify API- * and its settings can be altered by calling the `Apify.client.setOptions()` function.- * Be careful, by changing the settings you might alter behavior of functions such as- * [Apify.getValue]{@linkcode getValue} or [Apify.setValue]{@linkcode setValue}.+ * The instance is created when the `apify` package is first imported+ * and it is configured using the `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN`+ * environment variables.+ * After that, the instance is used for all underlying calls to the Apify API+ * in functions such as <a href="#module-Apify-getValue">Apify.getValue()</a>+ * or <a href="#module-Apify-call">Apify.call()</a>.+ * The settings of the client can be globally altered by calling the+ * <a href="https://www.apify.com/docs/js/apify-client-js/latest#ApifyClient-setOptions"><code>Apify.client.setOptions()</code></a> function.+ * Just be careful, it might have undesired effects on other functions provided by this package.+ * </p>*/export const apifyClient = newClient();@@ -49,25 +55,37 @@ const getDefaultStoreIdOrThrow = () => {/*** @memberof module:Apify* @function- * @description Gets a value from the default key-value store for the current act run.- * This store is created automatically for this run- * and its ID is passed by the Apify platform as the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.- * The result of the function is the body of the record. For records with the 'application/json'- * content type, the body is the already parsed object- * and for 'text/plain' content types it is parsed as String.- * For other content types, the body is raw Buffer.- * If the record cannot be found, the result is null.+ * @description <p>Gets a value from the default key-value store for the current act run using the Apify API.+ * The key-value store is created automatically for each act run+ * and its ID is passed by the Actor platform in the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.+ * It is used to store input and output of the act under keys named `INPUT` and `OUTPUT`, respectively.+ * However, the store can be used for storage of any other values under arbitrary keys.+ * </p>+ * <p>Example usage</p>+ * <pre><code class="language-javascript">+ * const input = await Apify.getValue('INPUT');*+ * console.log('My input:');+ * console.dir(input);+ * </code></pre>+ * <p>+ * The result of the function is the body of the record. Bodies with the `application/json`+ * content type are automatically parsed to an object.+ * Similarly, for `text/plain` content types the body is parsed as `String`.+ * For all other content types, the body is a raw `Buffer`.+ * If the record cannot be found, the result is null.+ * </p>+ * <p>* If the `APIFY_DEV_KEY_VALUE_STORE_DIR` environment variable is defined,* the value is read from a that directory rather than the key-value store,- * from a file that has the key as a name.- * The directory must exist or an error is thrown. The file might not exist, in which case the value is `null`.+ * specifically from a file that has the key as a name.+ * The directory must exist or an error is thrown. If the file does not exists, the returned value is `null`.* The file is assumed to have a content type specified in the `APIFY_DEV_KEY_VALUE_STORE_CONTENT_TYPE`* environment variable, or `application/json` if not set.- * This is useful for local development of the act.- *- * @param callback Optional callback.- * @returns Returns a promise if no callback was provided, otherwise the return value is not defined.+ * This feature is useful for local development and debugging of your acts.+ * </p>+ * @param {Function} callback Optional callback.+ * @returns {Promise} Returns a promise if no callback was provided.*/export const getValue = (key, callback = null) => {if (!key || !_.isString(key)) throw new Error('The "key" parameter must be a non-empty string');@@ -141,25 +159,48 @@ export const getValue = (key, callback = null) => {return nodeifyPromise(promise, callback);};+/*** @memberof module:Apify* @function- * @description Stores a value in the default key-value store for the current act run.- * This data is stored in the key-value store created specifically for this run,+ * @description <p>Stores a value in the default key-value store for the current act run using the Apify API.+ * The data is stored in the key-value store created specifically for the act run,* whose ID is defined in the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.- * The function has no result, but throws on invalid args or other errors.- *+ * The function has no result, but throws on invalid args or other errors.</p>+ * <pre><code class="language-javascript">+ * await Apify.setValue('OUTPUT', { someValue: 123 });+ * </code></pre>+ * <p>+ * By default, `value` is converted to JSON and stored with the `application/json; charset=utf-8` content type.+ * To store a value with another content type, pass it in the options as follows:+ * </p>+ * <pre><code class="language-javascript">+ * await Apify.setValue('OUTPUT', 'my text data', { contentType: 'text/plain' });+ * </code></pre>+ * <p>+ * In this case, the value must be a string or Buffer.+ * </p>+ * <p>* If the `APIFY_DEV_KEY_VALUE_STORE_DIR` environment variable is defined,- * the value is written to that directory rather than the key-value store,- * to a file named as the key. This is useful for local development of the act.- * @param value- * If null, the record in the key-value store is deleted.- * If no contentType is specified, the value can be any object and it will be stringified to JSON.- * If contentType is specified, value is considered raw data and it must be a String or Buffer.+ * the value is written to that local directory rather than the key-value store on Apify cloud,+ * to a file named as the key. This is useful for local development and debugging of your acts.+ * </p>+ * <p>+ * **IMPORTANT: Do not forget to use the `await` keyword when calling `Apify.setValue()`,+ * otherwise the act process might finish before the value is stored!**+ * </p>+ * @param key Key of the record+ * @param value Value of the record:+ * <ul>+ * <li>If `null`, the record in the key-value store is deleted.</li>+ * <li>If no `options.contentType` is specified, `value` can be any object and it will be stringified to JSON.</li>+ * <li>If `options.contentType` is specified, `value` is considered raw data and it must be a String or Buffer.</li>+ * </ul>* For any other value an error will be thrown.- * @param options Optional settings, currently only \{ contentType: String \} is supported to set MIME content type of the value.- * @param callback Optional callback.- * @returns Returns a promise if no callback was provided, otherwise the return value is not defined.+ * @param {Object} [options]+ * @param {String} [options.contentType] - Sets the MIME content type of the value.+ * @param {Function} [callback] Optional callback. Function returns a promise if not provided.+ * @returns {Promise} Returns a promise if `callback` was not provided.*/export const setValue = (key, value, options, callback = null) => {if (!key || !_.isString(key)) throw new Error('The "key" parameter must be a non-empty string');@@ -257,52 +298,43 @@ export const setValue = (key, value, options, callback = null) => {/*** @memberof module:Apify* @function- * @description Returns a new object which contains information parsed from the `APIFY_XXX` environment variables.- * It has the following properties:- * ```javascript+ * @description <p>Returns a new object which contains information parsed from the `APIFY_XXX` environment variables.+ * It has the following properties:</p>+ * <pre><code class="language-javascript">* {- * // ID of the act.- * // Environment variable: APIFY_ACT_ID- * actId: String,- *- * // ID of the act run- * // Environment variable: APIFY_ACT_RUN_ID- * actRunId: String,- *- * // ID of the user who started the act (might be different than the owner of the act)- * // Environment variable: APIFY_USER_ID- * userId: String,- *- * // Authentication token representing privileges given to the act run,- * // it can be passed to various Apify APIs.- * // Environment variable: APIFY_TOKEN- * token: String,- *- * // Date when the act was started- * // Environment variable: APIFY_STARTED_AT- * startedAt: Date,- *- * // Date when the act will time out- * // Environment variable: APIFY_TIMEOUT_AT- * timeoutAt: Date,- *- * // ID of the key-value store where input and output data of this act is stored- * // Environment variable: APIFY_DEFAULT_KEY_VALUE_STORE_ID- * defaultKeyValueStoreId: String,- *- * // The amount of memory allocated for the act run, in megabytes.- * // It can be used by acts to optimize their memory usage.- * // Environment variable: APIFY_MEMORY_MBYTES- * memoryMbytes: Number,+ * // ID of the act (APIFY_ACT_ID)+ * actId: String,+ * + * // ID of the act run (APIFY_ACT_RUN_ID)+ * actRunId: String,+ * + * // ID of the user who started the act - note that it might be different than the owner of the act (APIFY_USER_ID)+ * userId: String,+ * + * // Authentication token representing privileges given to the act run,+ * // it can be passed to various Apify APIs (APIFY_TOKEN).+ * token: String,+ * + * // Date when the act was started (APIFY_STARTED_AT)+ * startedAt: Date,+ * + * // Date when the act will time out (APIFY_TIMEOUT_AT)+ * timeoutAt: Date,+ * + * // ID of the key-value store where input and output data of this act is stored (APIFY_DEFAULT_KEY_VALUE_STORE_ID)+ * defaultKeyValueStoreId: String,+ * + * // Amount of memory allocated for the act run, in megabytes (APIFY_MEMORY_MBYTES)+ * memoryMbytes: Number,* }- * ```+ * </code></pre>* For the list of the `APIFY_XXX` environment variables, see* {@link http://localhost/docs/actor.php#run-env-vars|Actor documentation}.* If some of the variables is not defined or is invalid, the corresponding value in the resulting object will be null.* @returns {Object}*/export const getEnv = () => {- // NOTE: don't throw if env vars are invalid to simplify local development and debugging of acts+ // NOTE: Don't throw if env vars are invalid to simplify local development and debugging of actsconst env = process.env || {};return {actId: env[ENV_VARS.ACT_ID] || null,@@ -320,13 +352,13 @@ export const getEnv = () => {/*** @memberof module:Apify* @function- * @description <p>Runs a user function that executes the logic of the act.- * It performs the following actions:</p>+ * @description <p>Runs a user function that performs the logic of the act.+ * The `Apify.main(userFunct)` function does the following actions:</p>* <ol>* <li>Invokes the user function passed as the `userFunc` parameter</li>* <li>If the user function returned a promise, waits for it to resolve</li>* <li>If the user function throws an exception or some other error is encountered,- * prints the error details to console so that they are stored to the log file</li>+ * prints error details to console so that they are stored to the log file</li>* <li>Exits the process</li>* </ol>* <p>@@ -334,14 +366,14 @@ export const getEnv = () => {* </p>* ```javascript* Apify.main(() => {- * // my synchronous function that returns immediately+ * // My synchronous function that returns immediately* });* ```* <p>If the user function returns a promise, it is considered as asynchronous:</p>* ```javascript* const request = require('request-promise');* Apify.main(() => {- * // my asynchronous function that returns a promise+ * // My asynchronous function that returns a promise* return Promise.resolve()* .then(() => {* return request('http://www.example.com');@@ -359,7 +391,7 @@ export const getEnv = () => {* console.log(html);* });* ```- * Note that the use of `Apify.main()` in acts is optional,+ * Note that the use of `Apify.main()` in acts is optional;* the function is provided merely for user convenience and acts don't need to use it.* @param userFunc {Function} User function to be executed*/@@ -430,18 +462,58 @@ export const readyFreddy = () => {/*** @memberof module:Apify* @function- * @description Executes another act, waits for it to finish and fetches its output.+ * @description <p>Executes another act under the current user account, waits for the act finish and fetches its output.</p>+ * <p>The result of the function is an object describing the act run, which looks something like this:</p>+ * ```json+ * {+ * "id": "ErYkuTTsmKiXccNGT",+ * "actId": "E2jjCZBezvAZnX8Rb",+ * "userId": "mb7q2dycFBHDhae6A",+ * "startedAt": "2017-10-25T14:23:44.376Z",+ * "finishedAt": "2017-10-25T14:23:46.723Z",+ * "status": "SUCCEEDED",+ * "meta": { "origin": "API", "clientIp": "1.2.3.4", "userAgent": null },+ * "stats": {+ * "netRxBytes": 180,+ * "netTxBytes": 0,+ * ...+ * },+ * "options": { "build": "latest", "timeoutSecs": 0, "memoryMbytes": 512, "diskMbytes": 1024 },+ * "buildId": "Bwkqk59MCkdexDP34",+ * "exitCode": 0,+ * "defaultKeyValueStoreId": "ccFfRptZru2uqdQHP",+ * "buildNumber": "0.1.2",+ * "output": {+ * "contentType": "application/json; charset=utf-8",+ * "body": { "message": "Hello world!" }+ * }+ * }+ * ```+ * <p>Internally, the function calls the {@link https://www.apify.com/docs/api/v2#/reference/acts/runs-collection/run-act|Run act} API endpoint+ * and few others.</p>+ * <p>Example usage:</p>+ * ```javascript+ * const run = await Apify.call('apify/hello-world', { myInput: 123 });+ * console.log(`Received message: ${run.output.body.message}`);+ * ```+ ** @param {String} actId - Either `username/act-name` or act ID.* @param {Object|String|Buffer} [input] - Act input body. If it is an object, it is stringified to- * JSON and its content type set to ``.+ * JSON and the content type set to `application/json; charset=utf-8`.* @param {Object} [opts]- * @param {String} [opts.token] - User API token.- * @param {String} [opts.build] - Build tag or number to be executed.- * @param {String} [opts.contentType] - Content type of the act input.- * @param {String} [opts.timeoutSecs] - Time limit for act to finish. If limit is reached then run in RUNNING status is returned.- Default is unlimited.- * @param {String} [opts.fetchOutput] - If false then doesn't fetch the OUTPUT from key-value store. Default is true.- * @param {String} [opts.disableBodyParser] - If true then doesn't parse the body - ie. JSON to object. Default is false.+ * @param {String} [opts.token] - User API token. By default, it is taken from the `APIFY_TOKEN` environment variable.+ * @param {String} [opts.build] - Tag or number of act build to be run (e.g. `beta` or `1.2.345`).+ * If not provided, the default build tag or number from act configuration is used (typically `latest`).+ * @param {String} [opts.contentType] - Content type for the `input`. If not specified,+ * `input` is expected to be an object that will be stringified to JSON and content type set to+ * `application/json; charset=utf-8`. If `opts.contentType` is specified, then `input` must be a `String` or `Buffer`.+ * @param {String} [opts.timeoutSecs] - Time limit for act to finish, in seconds.+ * If the limit is reached the resulting run will have the `RUNNING` status.+ * By default, there is no timeout.+ * @param {String} [opts.fetchOutput] - If `false` then the function does not fetch output of the act. Default is `true`.+ * @param {String} [opts.disableBodyParser] - If `true` then the function will not attempt to parse the+ * act's output and will return it in a raw `Buffer`. Default is `false`.+ * @param {Function} [callback] - Optional callback. Function returns a promise if not provided.* @returns {Promise} Returns a promise unless `callback` was supplied.*/export const call = (actId, input, opts = {}, callback) => {diff --git a/src/browser.js b/src/browser.jsindex c78ed1502..bb17e31a1 100644--- a/src/browser.js+++ b/src/browser.js@@ -35,6 +35,7 @@ export const getDefaultBrowseOptions = () => {/*** Represents a single web browser process.* Currently it is just a thin wrapper of Selenium's WebDriver instance.+ * @ignore*/export class Browser {@@ -203,6 +204,90 @@ export const processBrowseArgs = (url, options, callback) => {return { options, callback };};+/*+OLD INFO FROM README:+### Browser++Apify runtime optionally depends on+the [selenium-webdriver](https://www.npmjs.com/package/selenium-webdriver) package that enables+automation of a web browser.+The simplest way to launch a new web browser is using the `Apify.browse([url,] [options,] [callback])`+function. For example:++```javascript+const browser = await Apify.browse('https://www.example.com/');+```++or++```javascript+const browser = await Apify.browse({+ url: 'https://www.example.com/',+ userAgent: 'MyCrawlingBot/1.23',+});+```++The `options` parameter controls settings of the web browser and it has the following properties:++```javascript+{+ // Initial URL to open. Note that the url argument in Apify.browse() overrides this value.+ // The default value is 'about:blank'+ url: String,++ // The type of the web browser to use.+ // See https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities for possible options.+ // The default value is 'chrome', which is currently the only fully-supported browser.+ browserName: String,++ // Indicates whether the browser should be opened in headless mode (i.e. without windows).+ // By default, this value is based on the APIFY_HEADLESS environment variable.+ headless: Boolean,++ // URL of the proxy server, e.g. 'http://username:[email protected]:55555'.+ // Currently only the 'http' proxy type is supported.+ // By default it is null, which means no proxy server is used.+ proxyUrl: String,++ // Overrides the User-Agent HTTP header of the web browser.+ // By default it is null, which means the browser uses its default User-Agent.+ userAgent: String,+}+```++The result of the `Apify.browse()` is a new instance of the `Browser` class,+which represents a web browser instance (possibly with multiple windows or tabs).+If you pass a Node.js-style callback the `Browser` instance is passed to it,+otherwise the `Apify.browse()` function returns a promise that resolves to the `Browser` instance.++The `Browser` class has the following properties:++```javascript+{+ // An instance of the Selenium's WebDriver class.+ webDriver: Object,++ // A method that closes the web browser and releases associated resources.+ // The method has no arguments and returns a promise that resolves when the browser was closed.+ close: Function,+}+```++The `webDriver` property can be used to manipulate the web browser:++```javascript+const url = await browser.webDriver.getCurrentUrl();+```++For more information, see [WebDriver documentation](http://seleniumhq.github.io/selenium/docs/api/+javascript/module/selenium-webdriver/index_exports_WebDriver.html).++When the web browser is no longer needed, it should be closed:++```javascript+await browser.close();+```+ *//*** @memberof module:Apify@@ -215,6 +300,7 @@ export const processBrowseArgs = (url, options, callback) => {* @param options Optional settings, their defaults are provided by the getDefaultBrowseOptions() function.* @param callback Optional callback.* @returns Returns a promise if no callback was provided, otherwise the return value is not defined.+ * @ignore*/export const browse = (url, options, callback) => {const args = processBrowseArgs(url, options, callback);diff --git a/src/index.js b/src/index.jsindex 3c1434926..b3edeef8f 100644--- a/src/index.js+++ b/src/index.js@@ -25,15 +25,66 @@ const Apify = {};/**- * Helper package that simplifies development of Apify acts.* @module Apify- * @description Basic usage of Apify module in Apify acts:- * ```javascript- * const Apify = require('apify');+ * @description+ * <p>+ * The `apify` NPM package simplifies development of acts in Apify Actor -+ * a serverless computing platform that enables execution of arbitrary pieces of code in the cloud.+ * The package provides helper functions to launch web browsers with proxies,+ * access the storage etc. Note that the usage of the package is optional,+ * you can create acts without it.+ * </p>+ * <p>+ * For more information about the Apify Actor platform, please go to+ * {@link https://www.apify.com/docs/actor|Actor documentation}.+ * </p>*- * Apify.main(() => {- * // my synchronous function that returns immediately- * });+ * <h2>Example usage</h2>+ * <pre><code class="language-javascript">const Apify = require('apify');+ * + * Apify.main(async () => {+ * // Get input of the act+ * const input = await Apify.getValue('INPUT');+ * console.dir(input);+ * + * // Do something useful here...+ * const browser = await Apify.launchPuppeteer();+ * const page = await browser.newPage();+ * await page.goto('http://www.example.com');+ * const pageTitle = await page.title();+ * + * // Save output+ * await Apify.setValue('OUTPUT', { pageTitle });+ * });</code></pre>+ *+ * <h2>Installation</h2>+ * <p>This package requires Node.js 6 or higher. It might work with lower versions too,+ * but they are neither tested nor supported.+ * The package is primarily used+ * </p>+ * <p>The package also be used for local development. To install it on your machine,+ * run the following command in your Node.js project:+ * To install the package locally, run the following command:</p>+ * ```bash+ * npm install apify --save* ```+ * <p>+ * <h2>Promises vs. callbacks</h2>+ * <p>+ * By default, all asynchronous functions return a promise.+ * However, they also accept an optional Node.js-style callback as the last parameter.+ * If the callback is provided, the return value of the functions is not defined+ * and the functions only invoke the callback upon completion or error.+ * </p>+ * <p>+ * To set a promise dependency from an external library,+ * use the <a href="#module-Apify-setPromisesDependency"><code>Apify.setPromisesDependency()</code></a> function.+ * If this function is not called, the runtime defaults to+ * native promises if they are available, or it throws an error.+ * </p>+ * <h2>Full reference</h2>+ * <p>The following sections describe all functions and properties provided by the `apify` package.+ * All of them are instance members exported directly by the main module.+ * </p>*/module.exports = Apify;diff --git a/src/puppeteer.js b/src/puppeteer.jsindex 92e695584..0b71f701b 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -9,18 +9,19 @@ import { ENV_VARS } from './constants';* @memberof module:Apify* @function* @description <p>Launches headless Chrome using Puppeteer pre-configured to work with the Apify Actor platform.- * The function has the same argument and return values as `puppeteer.launch()`.+ * The function has the same argument and return value as `puppeteer.launch()`.* See {@link https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions|Puppeteer documentation} for more details.</p>* <p>The `launchPuppeteer()` function alters the following Puppeteer options:* <ul>- * <li>Passes the value defined in the `APIFY_HEADLESS` environment variables to the `headless` option,+ * <li>Passes the value of the `APIFY_HEADLESS` environment variable to the `headless` option,* unless it was already defined by the caller.</li>* <li>Takes the `proxyUrl` option and adds it to `env` under the `HTTPS_PROXY` or `HTTP_PROXY` key.</li>- * <li>Adds `--no-sandbox` to `args` to enable running headless Chrome in a Docker container.</li>+ * <li>Adds `--no-sandbox` to `args` to enable running headless Chrome in a Docker container on the Actor platform.</li>* </ul>* </p>* @param options Optional settings passed to `puppeteer.launch()`.- * Additionally, the options can contain the `proxyUrl` property to specify a proxy server.+ * Additionally, the options can contain the `proxyUrl` property to specify a proxy server+ * (e.g. `http://example.com:1234`).* @returns {Promise} Promise object returned by `puppeteer.launch()`*/export const launchPuppeteer = (opts = {}) => {diff --git a/src/utils.js b/src/utils.jsindex aab054326..862623ac5 100644--- a/src/utils.js+++ b/src/utils.js@@ -14,9 +14,16 @@ let PromisesDependency = typeof Promise === 'function' ? Promise : null;/*** @memberof module:Apify* @function- * @description Sets the promise dependency the SDK will use wherever Promises are returned.- * Passing `null` will force the SDK to use native Promises if they are available.- * @param [Constructor] dep A reference to a Promise constructor+ * @description <p>Sets the promise dependency that the package will use wherever promises are returned.+ * Passing `null` will force the SDK to use native Promises if they are available.</p>+ * <p>Example usage</p>+ * <pre><code>+ * const Promise = require('bluebird');+ * const Apify = require('apify');+ * + * Apify.setPromisesDependency(Promise);+ * </code></pre>+ * @param [Constructor] dep Reference to a Promise constructor*/export const setPromisesDependency = (dep) => {if (dep !== null && typeof dep !== 'function') throw new Error('The "dep" parameter must be a function');@@ -26,7 +33,8 @@ export const setPromisesDependency = (dep) => {/*** @memberof module:Apify* @function- * @description Gets the promise dependency set by `Apify.setPromisesDependency`.+ * @description Gets the promise dependency set by <a href="#module-Apify-setPromisesDependency"><code>Apify.setPromisesDependency</code></a>.+ * @returns {Constructor} Reference to a Promise constructor*/export const getPromisesDependency = () => {return PromisesDependency;