Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog, build docs
Ondra Urban0baf71ecdcd7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e6f20f18f..e68695b01 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,9 @@+1.0.2 / 2021/03/05+====================+- Add the ability to override `ProxyConfiguration` status check URL with the `APIFY_PROXY_STATUS_URL` env var.+- Fix inconsistencies in cookie handling when `SessionPool` was used.+- Fix TS types in multiple places. TS is still not a first class citizen, but this should improve the experience.+1.0.1 / 2021/02/03====================- Fix `dataset.pushData()` validation which would not allow other than plain objects.diff --git a/docs/guides/docker_images.md b/docs/guides/docker_images.mdindex 16878b644..068779461 100644--- a/docs/guides/docker_images.md+++ b/docs/guides/docker_images.md@@ -73,24 +73,14 @@ with v1.7.1.Installing `apify/actor-node-puppeteer-chrome` (without a tag) will install the latest available version.-We recommend reflecting this in your `package.json` files. Either by providing the same version of-the library as the selected image:+We recommend using asterisk `*` as the version in your `package.json` files. It makes sure the pre-installed+version of Puppeteer or Playwright is not reinstalled. This is important, because those libraries are only+guaranteed to work with specific versions of browsers, and those browsers come pre-installed in the image.```dockerfile-FROM apify/actor-node-playwright-chrome:1.7.1-```--```json-{- "dependencies": {- "playwright": "1.7.1"- }-}+FROM apify/actor-node-playwright-chrome```-Or by using an asterisk as your version. This will make sure the library version pre-installed in the-docker image is left untouched.-```json{"dependencies": {@@ -107,7 +97,7 @@ Be careful when installing new dependencies. Nothing prevents you from installin`actor-node-puppeteer-chrome` image, but the resulting image will be about 3 times larger and extremelyslow to download and build.-Use only what you need and you'll be rewarded with reasonable build and start times.+Use only what you need, and you'll be rewarded with reasonable build and start times.## actor-nodeThis is the smallest image we have based on Alpine Linux. It does not include any browsers, and it's thereforediff --git a/website/i18n/en.json b/website/i18n/en.jsonindex 273b65df9..605eec344 100644--- a/website/i18n/en.json+++ b/website/i18n/en.json@@ -17,6 +17,9 @@"api/basic-crawler": {"title": "BasicCrawler"},+ "api/browser-plugin": {+ "title": "BrowserPlugin"+ },"api/cheerio-crawler": {"title": "CheerioCrawler"},@@ -188,6 +191,9 @@"typedefs/actor-run": {"title": "ActorRun"},+ "typedefs/adhoc-webhook": {+ "title": "AdhocWebhook"+ },"typedefs/apify-env": {"title": "ApifyEnv"},@@ -197,6 +203,12 @@"typedefs/basic-crawler-options": {"title": "BasicCrawlerOptions"},+ "typedefs/browser-crawling-context": {+ "title": "BrowserCrawlingContext"+ },+ "typedefs/browser-handle-page-function": {+ "title": "BrowserHandlePageFunction"+ },"typedefs/browser-launch-context": {"title": "BrowserLaunchContext"},@@ -209,6 +221,15 @@"typedefs/cheerio-handle-page-inputs": {"title": "CheerioHandlePageInputs"},+ "typedefs/compiled-script-function": {+ "title": "CompiledScriptFunction"+ },+ "typedefs/compiled-script-params": {+ "title": "CompiledScriptParams"+ },+ "typedefs/crawling-context": {+ "title": "CrawlingContext"+ },"typedefs/create-session": {"title": "CreateSession"},@@ -224,6 +245,15 @@"typedefs/dataset-reducer": {"title": "DatasetReducer"},+ "typedefs/direct-navigation-options": {+ "title": "DirectNavigationOptions"+ },+ "typedefs/event-types": {+ "title": "EventTypes"+ },+ "typedefs/goto-function": {+ "title": "GotoFunction"+ },"typedefs/handle-failed-request": {"title": "HandleFailedRequest"},@@ -236,12 +266,18 @@"typedefs/handle-request-inputs": {"title": "HandleRequestInputs"},+ "typedefs/hook": {+ "title": "Hook"+ },"typedefs/intercept-handler": {"title": "InterceptHandler"},"typedefs/key-consumer": {"title": "KeyConsumer"},+ "typedefs/key-value-store-value-types": {+ "title": "KeyValueStoreValueTypes"+ },"typedefs/logger-options": {"title": "LoggerOptions"},@@ -251,6 +287,18 @@"typedefs/playwright-crawler-options": {"title": "PlaywrightCrawlerOptions"},+ "typedefs/playwright-goto-options": {+ "title": "PlaywrightGotoOptions"+ },+ "typedefs/playwright-handle-page-function": {+ "title": "PlaywrightHandlePageFunction"+ },+ "typedefs/playwright-handle-page-function-param": {+ "title": "PlaywrightHandlePageFunctionParam"+ },+ "typedefs/playwright-hook": {+ "title": "PlaywrightHook"+ },"typedefs/playwright-launch-context": {"title": "PlaywrightLaunchContext"},@@ -266,6 +314,9 @@"typedefs/prepare-request-inputs": {"title": "PrepareRequestInputs"},+ "typedefs/proxy-configuration-function": {+ "title": "ProxyConfigurationFunction"+ },"typedefs/proxy-configuration-options": {"title": "ProxyConfigurationOptions"},@@ -275,6 +326,15 @@"typedefs/puppeteer-crawler-options": {"title": "PuppeteerCrawlerOptions"},+ "typedefs/puppeteer-handle-page": {+ "title": "PuppeteerHandlePage"+ },+ "typedefs/puppeteer-handle-page-function-param": {+ "title": "PuppeteerHandlePageFunctionParam"+ },+ "typedefs/puppeteer-hook": {+ "title": "PuppeteerHook"+ },"typedefs/puppeteer-launch-context": {"title": "PuppeteerLaunchContext"},@@ -284,9 +344,15 @@"typedefs/request-as-browser-options": {"title": "RequestAsBrowserOptions"},+ "typedefs/request-as-browser-result": {+ "title": "RequestAsBrowserResult"+ },"typedefs/request-list-options": {"title": "RequestListOptions"},+ "typedefs/request-list-sources-function": {+ "title": "RequestListSourcesFunction"+ },"typedefs/request-list-state": {"title": "RequestListState"},@@ -332,6 +398,9 @@"typedefs/user-func": {"title": "UserFunc"},+ "typedefs/webhook-run": {+ "title": "WebhookRun"+ },"version-0.22.4/api/version-0.22.4-apify": {"title": "Apify"},@@ -796,6 +865,177 @@},"version-1.0.1/typedefs/version-1.0.1-puppeteer-crawler-options": {"title": "PuppeteerCrawlerOptions"+ },+ "version-1.0.2/api/version-1.0.2-apify": {+ "title": "Apify"+ },+ "version-1.0.2/api/version-1.0.2-autoscaled-pool": {+ "title": "AutoscaledPool"+ },+ "version-1.0.2/api/version-1.0.2-basic-crawler": {+ "title": "BasicCrawler"+ },+ "version-1.0.2/api/version-1.0.2-browser-plugin": {+ "title": "BrowserPlugin"+ },+ "version-1.0.2/api/version-1.0.2-key-value-store": {+ "title": "KeyValueStore"+ },+ "version-1.0.2/api/version-1.0.2-log": {+ "title": "utils.log"+ },+ "version-1.0.2/api/version-1.0.2-playwright": {+ "title": "utils.playwright"+ },+ "version-1.0.2/api/version-1.0.2-pseudo-url": {+ "title": "PseudoUrl"+ },+ "version-1.0.2/api/version-1.0.2-puppeteer": {+ "title": "utils.puppeteer"+ },+ "version-1.0.2/api/version-1.0.2-request": {+ "title": "Request"+ },+ "version-1.0.2/api/version-1.0.2-request-list": {+ "title": "RequestList"+ },+ "version-1.0.2/api/version-1.0.2-request-queue": {+ "title": "RequestQueue"+ },+ "version-1.0.2/api/version-1.0.2-session": {+ "title": "Session"+ },+ "version-1.0.2/api/version-1.0.2-session-pool": {+ "title": "SessionPool"+ },+ "version-1.0.2/api/version-1.0.2-snapshotter": {+ "title": "Snapshotter"+ },+ "version-1.0.2/api/version-1.0.2-social": {+ "title": "utils.social"+ },+ "version-1.0.2/api/version-1.0.2-utils": {+ "title": "utils"+ },+ "version-1.0.2/typedefs/version-1.0.2-actor-run": {+ "title": "ActorRun"+ },+ "version-1.0.2/typedefs/version-1.0.2-adhoc-webhook": {+ "title": "AdhocWebhook"+ },+ "version-1.0.2/typedefs/version-1.0.2-browser-crawling-context": {+ "title": "BrowserCrawlingContext"+ },+ "version-1.0.2/typedefs/version-1.0.2-browser-handle-page-function": {+ "title": "BrowserHandlePageFunction"+ },+ "version-1.0.2/typedefs/version-1.0.2-browser-launch-context": {+ "title": "BrowserLaunchContext"+ },+ "version-1.0.2/typedefs/version-1.0.2-cheerio-crawler-options": {+ "title": "CheerioCrawlerOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-cheerio-handle-page-inputs": {+ "title": "CheerioHandlePageInputs"+ },+ "version-1.0.2/typedefs/version-1.0.2-compiled-script-function": {+ "title": "CompiledScriptFunction"+ },+ "version-1.0.2/typedefs/version-1.0.2-compiled-script-params": {+ "title": "CompiledScriptParams"+ },+ "version-1.0.2/typedefs/version-1.0.2-crawling-context": {+ "title": "CrawlingContext"+ },+ "version-1.0.2/typedefs/version-1.0.2-create-session": {+ "title": "CreateSession"+ },+ "version-1.0.2/typedefs/version-1.0.2-direct-navigation-options": {+ "title": "DirectNavigationOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-event-types": {+ "title": "EventTypes"+ },+ "version-1.0.2/typedefs/version-1.0.2-goto-function": {+ "title": "GotoFunction"+ },+ "version-1.0.2/typedefs/version-1.0.2-handle-failed-request": {+ "title": "HandleFailedRequest"+ },+ "version-1.0.2/typedefs/version-1.0.2-hook": {+ "title": "Hook"+ },+ "version-1.0.2/typedefs/version-1.0.2-key-consumer": {+ "title": "KeyConsumer"+ },+ "version-1.0.2/typedefs/version-1.0.2-key-value-store-value-types": {+ "title": "KeyValueStoreValueTypes"+ },+ "version-1.0.2/typedefs/version-1.0.2-logger-options": {+ "title": "LoggerOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-playwright-crawler-options": {+ "title": "PlaywrightCrawlerOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-playwright-goto-options": {+ "title": "PlaywrightGotoOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-playwright-handle-page-function": {+ "title": "PlaywrightHandlePageFunction"+ },+ "version-1.0.2/typedefs/version-1.0.2-playwright-handle-page-function-param": {+ "title": "PlaywrightHandlePageFunctionParam"+ },+ "version-1.0.2/typedefs/version-1.0.2-playwright-hook": {+ "title": "PlaywrightHook"+ },+ "version-1.0.2/typedefs/version-1.0.2-playwright-launch-context": {+ "title": "PlaywrightLaunchContext"+ },+ "version-1.0.2/typedefs/version-1.0.2-proxy-configuration-function": {+ "title": "ProxyConfigurationFunction"+ },+ "version-1.0.2/typedefs/version-1.0.2-proxy-configuration-options": {+ "title": "ProxyConfigurationOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-puppeteer-crawler-options": {+ "title": "PuppeteerCrawlerOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-puppeteer-handle-page": {+ "title": "PuppeteerHandlePage"+ },+ "version-1.0.2/typedefs/version-1.0.2-puppeteer-handle-page-function-param": {+ "title": "PuppeteerHandlePageFunctionParam"+ },+ "version-1.0.2/typedefs/version-1.0.2-puppeteer-hook": {+ "title": "PuppeteerHook"+ },+ "version-1.0.2/typedefs/version-1.0.2-puppeteer-launch-context": {+ "title": "PuppeteerLaunchContext"+ },+ "version-1.0.2/typedefs/version-1.0.2-request-as-browser-options": {+ "title": "RequestAsBrowserOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-request-as-browser-result": {+ "title": "RequestAsBrowserResult"+ },+ "version-1.0.2/typedefs/version-1.0.2-request-list-options": {+ "title": "RequestListOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-request-list-sources-function": {+ "title": "RequestListSourcesFunction"+ },+ "version-1.0.2/typedefs/version-1.0.2-request-options": {+ "title": "RequestOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-session-options": {+ "title": "SessionOptions"+ },+ "version-1.0.2/typedefs/version-1.0.2-system-info": {+ "title": "SystemInfo"+ },+ "version-1.0.2/typedefs/version-1.0.2-webhook-run": {+ "title": "WebhookRun"}},"links": {diff --git a/website/package.json b/website/package.jsonindex 5d1962d9d..97862026d 100644--- a/website/package.json+++ b/website/package.json@@ -15,12 +15,12 @@"decamelize": "^4.0.0","docusaurus": "^1.13.0","fs-extra": "^8.1.0",+ "got": "^11.8.2","jsdoc-plugin-intersection": "^1.0.4","jsdoc-to-markdown": "^7.0.0","prettier": "^1.18.2"},"dependencies": {- "@apify/http-request": "^1.1.5","remarkable-extlink": "^1.1.0"}}diff --git a/website/tools/build_docs.js b/website/tools/build_docs.jsindex 39127e448..9fa1a9eba 100644--- a/website/tools/build_docs.js+++ b/website/tools/build_docs.js@@ -3,7 +3,7 @@ const fs = require('fs-extra');const jsdoc2md = require('jsdoc-to-markdown'); // eslint-disable-lineconst path = require('path');const prettier = require('prettier'); // eslint-disable-line-const httpRequest = require('@apify/http-request');+const got = require('got');const prettierConfig = require('./prettier.config');const sidebars = require('../sidebars.json');const { readStreamToString } = require('apify-shared/streams_utilities'); // eslint-disable-line@@ -123,10 +123,7 @@ const generateFinalMarkdown = (title, text, entityMap) => {async function getExamplesFromRepo() {await fs.emptyDir(EXAMPLES_DIR_NAME);process.chdir(EXAMPLES_DIR_NAME);- const { body } = await httpRequest({- url: EXAMPLES_REPO,- json: true,- });+ const body = await got(EXAMPLES_REPO).json();const builtExamples = await buildExamples(body);await addExamplesToSidebars(builtExamples);}@@ -134,20 +131,12 @@ async function getExamplesFromRepo() {async function buildExamples(exampleLinks) {const examples = [];for (const example of exampleLinks) {- const responseStream = await httpRequest({- url: example.download_url,- stream: true,- });- try {- console.log(`Rendering example ${example.name}`);- const fileContent = await readStreamToString(responseStream);- const markdown = prettier.format(fileContent, prettierConfig);- fs.writeFileSync(example.name, markdown);- const exampleName = example.name.split('.')[0];- examples.push(`examples/${exampleName.replace(/_/g, '-')}`);- } catch (err) {- throw err;- }+ const fileContent = await got(example.download_url).text();+ console.log(`Rendering example ${example.name}`);+ const markdown = prettier.format(fileContent, prettierConfig);+ fs.writeFileSync(example.name, markdown);+ const exampleName = example.name.split('.')[0];+ examples.push(`examples/${exampleName.replace(/_/g, '-')}`);}return examples;}diff --git a/website/versioned_docs/version-0.22.4/guides/migration_to_v1.md b/website/versioned_docs/version-0.22.4/guides/migration_to_v1.mdindex 0b2f9a5d7..cf6d2401d 100644--- a/website/versioned_docs/version-0.22.4/guides/migration_to_v1.md+++ b/website/versioned_docs/version-0.22.4/guides/migration_to_v1.md@@ -9,21 +9,21 @@ original_id: migration-to-v1- [Installation](#installation)- [Running on Apify Platform](#running-on-apify-platform)- [Handler arguments are now Crawling Context](#handler-arguments-are-now-crawling-context)- * [`Map` of crawling contexts and their IDs](#map-of-crawling-contexts-and-their-ids)- * [`autoscaledPool` was moved under `crawlingContext.crawler`](#autoscaledpool-was-moved-under-crawlingcontextcrawler)+ - [`Map` of crawling contexts and their IDs](#map-of-crawling-contexts-and-their-ids)+ - [`autoscaledPool` was moved under `crawlingContext.crawler`](#autoscaledpool-was-moved-under-crawlingcontextcrawler)- [Replacement of `PuppeteerPool` with `BrowserPool`](#replacement-of-puppeteerpool-with-browserpool)- * [Access to running `BrowserPool`](#access-to-running-browserpool)- * [Pages now have IDs](#pages-now-have-ids)- * [Configuration and lifecycle hooks](#configuration-and-lifecycle-hooks)- * [Introduction of `BrowserController`](#introduction-of-browsercontroller)- * [`BrowserPool` methods vs `PuppeteerPool`](#browserpool-methods-vs-puppeteerpool)+ - [Access to running `BrowserPool`](#access-to-running-browserpool)+ - [Pages now have IDs](#pages-now-have-ids)+ - [Configuration and lifecycle hooks](#configuration-and-lifecycle-hooks)+ - [Introduction of `BrowserController`](#introduction-of-browsercontroller)+ - [`BrowserPool` methods vs `PuppeteerPool`](#browserpool-methods-vs-puppeteerpool)- [Updated `PuppeteerCrawlerOptions`](#updated-puppeteercrawleroptions)- * [Removal of `gotoFunction`](#removal-of-gotofunction)- * [`launchPuppeteerOptions` => `launchContext`](#launchpuppeteeroptions--launchcontext)- * [Removal of `launchPuppeteerFunction`](#removal-of-launchpuppeteerfunction)+ - [Removal of `gotoFunction`](#removal-of-gotofunction)+ - [`launchPuppeteerOptions` => `launchContext`](#launchpuppeteeroptions--launchcontext)+ - [Removal of `launchPuppeteerFunction`](#removal-of-launchpuppeteerfunction)- [Launch functions](#launch-functions)- * [Updated arguments](#updated-arguments)- * [Custom modules](#custom-modules)+ - [Updated arguments](#updated-arguments)+ - [Custom modules](#custom-modules)## SummaryAfter 3.5 years of rapid development and a lot of breaking changes and deprecations,@@ -229,17 +229,17 @@ you should use a `browserController` to do it. You can find it in the handle pag```jsconst handlePageFunction = async ({ page, browserController }) => {// Wrong usage. Could backfire because it bypasses BrowserPool.- page.browser().close();+ await page.browser().close();// Correct usage. Allows graceful shutdown.- browserController.close();+ await browserController.close();const cookies = [/* some cookie objects */];// Wrong usage. Will only work in Puppeteer and not Playwright.- page.setCookies(...cookies);+ await page.setCookies(...cookies);// Correct usage. Will work in both.- browserController.setCookies(page, cookies);+ await browserController.setCookies(page, cookies);}```@@ -261,15 +261,15 @@ Some functions were removed (in line with earlier deprecations), and some were c```js// OLD-puppeteerPool.recyclePage(page);+await puppeteerPool.recyclePage(page);// NEW-page.close();+await page.close();``````js// OLD-puppeteerPool.retire(page.browser());+await puppeteerPool.retire(page.browser());// NEWbrowserPool.retireBrowserByPage(page);@@ -277,7 +277,7 @@ browserPool.retireBrowserByPage(page);```js// OLD-puppeteerPool.serveLiveViewSnapshot();+await puppeteerPool.serveLiveViewSnapshot();// NEW// There's no LiveView in BrowserPool@@ -299,7 +299,7 @@ const gotoFunction = async ({ request, page }) => {await makePageStealthy(page);// Have to remember how to do this:- const response = gotoExtended(page, request, {/* have to remember the defaults */});+ const response = await gotoExtended(page, request, {/* have to remember the defaults */});// post-processingawait page.evaluate(() => {diff --git a/website/versioned_docs/version-1.0.0/guides/docker_images.md b/website/versioned_docs/version-1.0.0/guides/docker_images.mdindex c23b19c76..c34d3e250 100644--- a/website/versioned_docs/version-1.0.0/guides/docker_images.md+++ b/website/versioned_docs/version-1.0.0/guides/docker_images.md@@ -74,24 +74,14 @@ with v1.7.1.Installing `apify/actor-node-puppeteer-chrome` (without a tag) will install the latest available version.-We recommend reflecting this in your `package.json` files. Either by providing the same version of-the library as the selected image:+We recommend using asterisk `*` as the version in your `package.json` files. It makes sure the pre-installed+version of Puppeteer or Playwright is not reinstalled. This is important, because those libraries are only+guaranteed to work with specific versions of browsers, and those browsers come pre-installed in the image.```dockerfile-FROM apify/actor-node-playwright-chrome:1.7.1-```--```json-{- "dependencies": {- "playwright": "1.7.1"- }-}+FROM apify/actor-node-playwright-chrome```-Or by using an asterisk as your version. This will make sure the library version pre-installed in the-docker image is left untouched.-```json{"dependencies": {@@ -108,7 +98,7 @@ Be careful when installing new dependencies. Nothing prevents you from installin`actor-node-puppeteer-chrome` image, but the resulting image will be about 3 times larger and extremelyslow to download and build.-Use only what you need and you'll be rewarded with reasonable build and start times.+Use only what you need, and you'll be rewarded with reasonable build and start times.## actor-nodeThis is the smallest image we have based on Alpine Linux. It does not include any browsers, and it's thereforediff --git a/website/versioned_docs/version-1.0.2/api/Apify.md b/website/versioned_docs/version-1.0.2/api/Apify.mdnew file mode 100644index 000000000..b9f3e7982--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/Apify.md@@ -0,0 +1,770 @@+---+id: version-1.0.2-apify+title: Apify+original_id: apify+---++<a name="apify"></a>++The following section describes all functions and properties provided by the `apify` package, except individual classes and namespaces that have their+separate, detailed, documentation pages accessible from the left sidebar. To learn how Apify SDK works, we suggest following the+[Getting Started](../guides/getting-started) tutorial.++**Important:**++> The following functions: `addWebhook`, `call`, `callTask` and `newClient` invoke features of the [Apify platform](../guides/apify-platform) and+> require your scripts to be authenticated. See the [authentication guide](../guides/apify-platform#logging-into-apify-platform-from-apify-sdk) for+> instructions.++---++<a name="addwebhook"></a>++## `Apify.addWebhook(options)`++Creates an ad-hoc webhook for the current actor run, which lets you receive a notification when the actor run finished or failed. For more information+about Apify actor webhooks, please see the [documentation](https://docs.apify.com/webhooks).++Note that webhooks are only supported for actors running on the Apify platform. In local environment, the function will print a warning and have no+effect.++**Parameters**:++- **`options`**: `object`+ - **`eventTypes`**: [`EventTypes`](../typedefs/event-types) - Array of event types, which you can set for actor run, see the+ [actor run events](https://docs.apify.com/webhooks/events#actor-run) in the Apify doc.+ - **`requestUrl`**: `string` - URL which will be requested using HTTP POST request, when actor run will reach the set event type.+ - **`[payloadTemplate]`**: `string` - Payload template is a JSON-like string that describes the structure of the webhook POST request payload.+ It uses JSON syntax, extended with a double curly braces syntax for injecting variables `{{variable}}`. Those variables are resolved at the+ time of the webhook's dispatch, and a list of available variables with their descriptions is available in the+ [Apify webhook documentation](https://docs.apify.com/webhooks). If `payloadTemplate` is omitted, the default payload template is used+ ([view docs](https://docs.apify.com/webhooks/actions#payload-template)).+ - **`[idempotencyKey]`**: `string` - Idempotency key enables you to ensure that a webhook will not be added multiple times in case of an actor+ restart or other situation that would cause the `addWebhook()` function to be called again. We suggest using the actor run ID as the+ idempotency key. You can get the run ID by calling [`Apify.getEnv()`](../api/apify#getenv) function.++**Returns**:++[`Promise<(WebhookRun|undefined)>`](../typedefs/webhook-run) - The return value is the Webhook object. For more information, see the+[Get webhook](https://apify.com/docs/api/v2#/reference/webhooks/webhook-object/get-webhook) API endpoint.++---++<a name="call"></a>++## `Apify.call(actId, [input], [options])`++Runs an actor on the Apify platform using the current user account (determined by the `APIFY_TOKEN` environment variable), waits for the actor to+finish and fetches its output.++By passing the `waitSecs` option you can reduce the maximum amount of time to wait for the run to finish. If the value is less than or equal to zero,+the function returns immediately after the run is started.++The result of the function is an [`ActorRun`](../typedefs/actor-run) object that contains details about the actor run and its output (if any). If the+actor run fails, the function throws the [`ApifyCallError`](../api/apify-call-error) exception.++If you want to run an actor task rather than an actor, please use the [`Apify.callTask()`](../api/apify#calltask) function instead.++For more information about actors, read the [documentation](https://docs.apify.com/actor).++**Example usage:**++```javascript+const run = await Apify.call('apify/hello-world', { myInput: 123 });+console.log(`Received message: ${run.output.body.message}`);+```++Internally, the `call()` function invokes the [Run actor](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor) and several other+API endpoints to obtain the output.++**Throws**:++- [`ApifyCallError`](../api/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.++**Parameters**:++- **`actId`**: `string` - Allowed formats are `username/actor-name`, `userId/actor-name` or actor ID.+- **`[input]`**: `Object<string, *>` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set to+ `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.+- **`[options]`**: `object` - Object with the settings below:+ - **`[contentType]`**: `string` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified+ to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or+ `Buffer`.+ - **`[token]`**: `string` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.+ - **`[memoryMbytes]`**: `number` - Memory in megabytes which will be allocated for the new actor run. If not provided, the run uses memory of+ the default actor run configuration.+ - **`[timeoutSecs]`**: `number` - Timeout for the actor run in seconds. Zero value means there is no timeout. If not provided, the run uses+ timeout of the default actor run configuration.+ - **`[build]`**: `string` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or+ number from the default actor run configuration (typically `latest`).+ - **`[waitSecs]`**: `number` - Maximum time to wait for the actor run to finish, in seconds. If the limit is reached, the returned promise is+ resolved to a run object that will have status `READY` or `RUNNING` and it will not contain the actor run output. If `waitSecs` is null or+ undefined, the function waits for the actor to finish (default behavior).+ - **`[fetchOutput]`**: `boolean` <code> = true</code> - If `false` then the function does not fetch output of the actor.+ - **`[disableBodyParser]`**: `boolean` <code> = false</code> - If `true` then the function will not attempt to parse the actor's output and will+ return it in a raw `Buffer`.+ - **`[webhooks]`**: [`Array<AdhocWebhook>`](../typedefs/adhoc-webhook) - Specifies optional webhooks associated with the actor run, which can be+ used to receive a notification e.g. when the actor finished or failed, see+ [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.++**Returns**:++[`Promise<ActorRun>`](../typedefs/actor-run)++---++<a name="calltask"></a>++## `Apify.callTask(taskId, [input], [options])`++Runs an actor task on the Apify platform using the current user account (determined by the `APIFY_TOKEN` environment variable), waits for the task to+finish and fetches its output.++By passing the `waitSecs` option you can reduce the maximum amount of time to wait for the run to finish. If the value is less than or equal to zero,+the function returns immediately after the run is started.++The result of the function is an [`ActorRun`](../typedefs/actor-run) object that contains details about the actor run and its output (if any). If the+actor run failed, the function fails with [`ApifyCallError`](../api/apify-call-error) exception.++Note that an actor task is a saved input configuration and options for an actor. If you want to run an actor directly rather than an actor task,+please use the [`Apify.call()`](../api/apify#call) function instead.++For more information about actor tasks, read the [documentation](https://docs.apify.com/tasks).++**Example usage:**++```javascript+const run = await Apify.callTask('bob/some-task');+console.log(`Received message: ${run.output.body.message}`);+```++Internally, the `callTask()` function calls the [Run task](https://apify.com/docs/api/v2#/reference/actor-tasks/run-collection/run-task) and several+other API endpoints to obtain the output.++**Throws**:++- [`ApifyCallError`](../api/apify-call-error) If the run did not succeed, e.g. if it failed or timed out.++**Parameters**:++- **`taskId`**: `string` - Allowed formats are `username/task-name`, `userId/task-name` or task ID.+- **`[input]`**: `Object<string, *>` - Input overrides for the actor task. If it is an object, it will be stringified to JSON and its content type+ set to `application/json; charset=utf-8`. Provided input will be merged with actor task input.+- **`[options]`**: `object` - Object with the settings below:+ - **`[token]`**: `string` - User API token that is used to run the actor. By default, it is taken from the `APIFY_TOKEN` environment variable.+ - **`[memoryMbytes]`**: `number` - Memory in megabytes which will be allocated for the new actor task run. If not provided, the run uses memory+ of the default actor run configuration.+ - **`[timeoutSecs]`**: `number` - Timeout for the actor task run in seconds. Zero value means there is no timeout. If not provided, the run uses+ timeout of the default actor run configuration.+ - **`[build]`**: `string` - Tag or number of the actor build to run (e.g. `beta` or `1.2.345`). If not provided, the run uses build tag or+ number from the default actor run configuration (typically `latest`).+ - **`[waitSecs]`**: `number` - Maximum time to wait for the actor task run to finish, in seconds. If the limit is reached, the returned promise+ is resolved to a run object that will have status `READY` or `RUNNING` and it will not contain the actor run output. If `waitSecs` is null or+ undefined, the function waits for the actor task to finish (default behavior).+ - **`[webhooks]`**: [`Array<AdhocWebhook>`](../typedefs/adhoc-webhook) - Specifies optional webhooks associated with the actor run, which can be+ used to receive a notification e.g. when the actor finished or failed, see+ [ad hook webhooks documentation](https://docs.apify.com/webhooks/ad-hoc-webhooks) for detailed description.++**Returns**:++[`Promise<ActorRun>`](../typedefs/actor-run)++---++<a name="createproxyconfiguration"></a>++## `Apify.createProxyConfiguration([proxyConfigurationOptions])`++Creates a proxy configuration and returns a promise resolving to an instance of the [`ProxyConfiguration`](../api/proxy-configuration) class that is+already initialized.++Configures connection to a proxy server with the provided options. Proxy servers are used to prevent target websites from blocking your crawlers based+on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures them to use the selected proxies for+all connections.++For more details and code examples, see the [`ProxyConfiguration`](../api/proxy-configuration) class.++```javascript++// Returns initialized proxy configuration class+const proxyConfiguration = await Apify.createProxyConfiguration({+ groups: ['GROUP1', 'GROUP2'] // List of Apify proxy groups+ countryCode: 'US'+});++const crawler = new Apify.CheerioCrawler({+ // ...+ proxyConfiguration,+ handlePageFunction: ({ proxyInfo }) => {+ const usedProxyUrl = proxyInfo.url; // Getting the proxy URL+ }+})++```++For compatibility with existing Actor Input UI (Input Schema), this function returns `undefined` when the following object is passed as+`proxyConfigurationOptions`.++```+{ useApifyProxy: false }+```++**Parameters**:++- **`[proxyConfigurationOptions]`**: [`ProxyConfigurationOptions`](../typedefs/proxy-configuration-options)++**Returns**:++[`Promise<(ProxyConfiguration|undefined)>`](../api/proxy-configuration)++---++<a name="events"></a>++## `Apify.events`++Gets an instance of a Node.js' [EventEmitter](https://nodejs.org/api/events.html#events_class_eventemitter) class that emits various events from the+SDK or the Apify platform. The event emitter is initialized by calling the [`Apify.main()`](../api/apify#main) function.++**Example usage:**++```javascript+Apify.events.on('cpuInfo', data => {+ if (data.isCpuOverloaded) console.log('Oh no, the CPU is overloaded!');+});+```++The following events are emitted:++- `cpuInfo`: `{ "isCpuOverloaded": Boolean }` The event is emitted approximately every second and it indicates whether the actor is using the+ maximum of available CPU resources. If that's the case, the actor should not add more workload. For example, this event is used by the+ [`AutoscaledPool`](../api/autoscaled-pool) class.+- `migrating`: `void` Emitted when the actor running on the Apify platform is going to be migrated to another worker server soon. You can use it to+ persist the state of the actor and abort the run, to speed up migration. For example, this is used by the [`RequestList`](../api/request-list)+ class.+- `persistState`: `{ "isMigrating": Boolean }` Emitted in regular intervals (by default 60 seconds) to notify all components of Apify SDK that it is+ time to persist their state, in order to avoid repeating all work when the actor restarts. This event is automatically emitted together with the+ `migrating` event, in which case the `isMigrating` flag is set to `true`. Otherwise the flag is `false`. Note that the `persistState` event is+ provided merely for user convenience, you can achieve the same effect using `setInterval()` and listening for the `migrating` event.++---++<a name="getenv"></a>++## `Apify.getEnv()`++Returns a new [`ApifyEnv`](../typedefs/apify-env) object which contains information parsed from all the `APIFY_XXX` environment variables.++For the list of the `APIFY_XXX` environment variables, see [Actor documentation](https://docs.apify.com/actor/run#environment-variables). If some of+the variables are not defined or are invalid, the corresponding value in the resulting object will be null.++**Returns**:++[`ApifyEnv`](../typedefs/apify-env)++---++<a name="getinput"></a>++## `Apify.getInput()`++Gets the actor input value from the default [`KeyValueStore`](../api/key-value-store) associated with the current actor run.++This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](key-value-store#getvalue). For example, calling the following code:++```javascript+const input = await Apify.getInput();+```++is equivalent to:++```javascript+const store = await Apify.openKeyValueStore();+await store.getValue('INPUT');+```++Note that the `getInput()` function does not cache the value read from the key-value store. If you need to use the input multiple times in your actor,+it is far more efficient to read it once and store it locally.++For more information, see [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) and+[`KeyValueStore.getValue()`](../api/key-value-store#getvalue).++**Returns**:++`Promise<(Object<string, *>|string|Buffer|null)>` - Returns a promise that resolves to an object, string or+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record, or `null` if the record is missing.++---++<a name="getmemoryinfo"></a>++## `Apify.getMemoryInfo()`++Returns memory statistics of the process and the system, see [`MemoryInfo`](../typedefs/memory-info).++If the process runs inside of Docker, the `getMemoryInfo` gets container memory limits, otherwise it gets system memory limits.++Beware that the function is quite inefficient because it spawns a new process. Therefore you shouldn't call it too often, like more than once per+second.++**Returns**:++[`Promise<MemoryInfo>`](../typedefs/memory-info)++---++<a name="getvalue"></a>++## `Apify.getValue(key)`++Gets a value from the default [`KeyValueStore`](../api/key-value-store) associated with the current actor run.++This is just a convenient shortcut for [`KeyValueStore.getValue()`](../api/key-value-store#getvalue). For example, calling the following code:++```javascript+const value = await Apify.getValue('my-key');+```++is equivalent to:++```javascript+const store = await Apify.openKeyValueStore();+const value = await store.getValue('my-key');+```++To store the value to the default key-value store, you can use the [`Apify.setValue()`](../api/apify#setvalue) function.++For more information, see [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) and+[`KeyValueStore.getValue()`](../api/key-value-store#getvalue).++**Parameters**:++- **`key`**: `string` - Unique record key.++**Returns**:++`Promise<(Object<string, *>|string|Buffer|null)>` - Returns a promise that resolves to an object, string or+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record, or `null` if the record is missing.++---++<a name="isathome"></a>++## `Apify.isAtHome()`++Returns `true` when code is running on Apify platform and `false` otherwise (for example locally).++**Returns**:++`boolean`++---++<a name="launchplaywright"></a>++## `Apify.launchPlaywright([launchContext])`++Launches headless browsers using Playwright pre-configured to work within the Apify platform. The function has the same return value as+`browserType.launch()`. See <a href="https://playwright.dev/docs/api/class-browsertype" target="_blank"> Playwright documentation</a> for more+details.++The `launchPlaywright()` function alters the following Playwright options:++- Passes the setting from the `APIFY_HEADLESS` environment variable to the `headless` option, unless it was already defined by the caller or+ `APIFY_XVFB` environment variable is set to `1`. Note that Apify Actor cloud platform automatically sets `APIFY_HEADLESS=1` to all running actors.+- Takes the `proxyUrl` option, validates it and adds it to `launchOptions` in a proper format. The proxy URL must define a port number and have one+ of the following schemes: `http://`, `https://`, `socks4://` or `socks5://`. If the proxy is HTTP (i.e. has the `http://` scheme) and contains+ username or password, the `launchPlaywright` functions sets up an anonymous proxy HTTP to make the proxy work with headless Chrome. For more+ information, read the+ <a href="https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212"+ target="_blank">blog post about proxy-chain library</a>.++To use this function, you need to have the [Playwright](https://www.npmjs.com/package/playwright) NPM package installed in your project. When running+on the Apify Platform, you can achieve that simply by using the `apify/actor-node-playwright-*` base Docker image for your actor - see+[Apify Actor documentation](https://docs.apify.com/actor/build#base-images) for details.++**Parameters**:++- **`[launchContext]`**: [`PlaywrightLaunchContext`](../typedefs/playwright-launch-context) - Optional settings passed to `browserType.launch()`. In+ addition to [Playwright's options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions) the object may+ contain our own [`PlaywrightLaunchContext`](../typedefs/playwright-launch-context) that enable additional features.++**Returns**:++`Promise<*>` - Promise that resolves to Playwright's `Browser` instance.++---++<a name="launchpuppeteer"></a>++## `Apify.launchPuppeteer([launchContext])`++Launches headless Chrome using Puppeteer pre-configured to work within the Apify platform. The function has the same argument and the return value as+`puppeteer.launch()`. See <a href="https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions" target="_blank"> Puppeteer+documentation</a> for more details.++The `launchPuppeteer()` function alters the following Puppeteer options:++- Passes the setting from the `APIFY_HEADLESS` environment variable to the `headless` option, unless it was already defined by the caller or+ `APIFY_XVFB` environment variable is set to `1`. Note that Apify Actor cloud platform automatically sets `APIFY_HEADLESS=1` to all running actors.+- Takes the `proxyUrl` option, validates it and adds it to `args` as `--proxy-server=XXX`. The proxy URL must define a port number and have one of+ the following schemes: `http://`, `https://`, `socks4://` or `socks5://`. If the proxy is HTTP (i.e. has the `http://` scheme) and contains+ username or password, the `launchPuppeteer` functions sets up an anonymous proxy HTTP to make the proxy work with headless Chrome. For more+ information, read the+ <a href="https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212"+ target="_blank">blog post about proxy-chain library</a>.++To use this function, you need to have the [puppeteer](https://www.npmjs.com/package/puppeteer) NPM package installed in your project. When running on+the Apify cloud, you can achieve that simply by using the `apify/actor-node-chrome` base Docker image for your actor - see+[Apify Actor documentation](https://docs.apify.com/actor/build#base-images) for details.++For an example of usage, see the [Synchronous run Example](../examples/synchronous-run) or the+[Puppeteer proxy Example](../examples/puppeteer-with-proxy)++**Parameters**:++- **`[launchContext]`**: [`PuppeteerLaunchContext`](../typedefs/puppeteer-launch-context) - All `PuppeteerLauncher` parameters are passed via an+ launchContext object. If you want to pass custom `puppeteer.launch(options)` options you can use the `PuppeteerLaunchContext.launchOptions`+ property.++**Returns**:++`Promise<*>` - Promise that resolves to Puppeteer's `Browser` instance.++---++<a name="main"></a>++## `Apify.main(userFunc)`++Runs the main user function that performs the job of the actor and terminates the process when the user function finishes.++**The `Apify.main()` function is optional** and is provided merely for your convenience. It is mainly useful when you're running your code as an actor+on the [Apify platform](https://apify.com/actors). However, if you want to use Apify SDK tools directly inside your existing projects, e.g. running in+an [Express](https://expressjs.com/) server, on [Google Cloud functions](https://cloud.google.com/functions) or+[AWS Lambda](https://aws.amazon.com/lambda/), it's better to avoid it since the function terminates the main process when it finishes!++The `Apify.main()` function performs the following actions:++- When running on the Apify platform (i.e. <code>APIFY_IS_AT_HOME</code> environment variable is set), it sets up a connection to listen for+ platform events. For example, to get a notification about an imminent migration to another server. See [`Apify.events`](../api/apify#events) for+ details.+- It checks that either <code>APIFY_TOKEN</code> or <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable is defined. If not, the functions sets+ <code>APIFY_LOCAL_STORAGE_DIR</code> to <code>./apify_storage</code> inside the current working directory. This is to simplify running code+ examples.+- It invokes the user function passed as the <code>userFunc</code> parameter.+- If the user function returned a promise, waits for it to resolve.+- If the user function throws an exception or some other error is encountered, prints error details to console so that they are stored to the log.+- Exits the Node.js process, with zero exit code on success and non-zero on errors.++The user function can be synchronous:++```javascript+Apify.main(() => {+ // My synchronous function that returns immediately+ console.log('Hello world from actor!');+});+```++If the user function returns a promise, it is considered asynchronous:++```javascript+const { requestAsBrowser } = require('some-request-library');++Apify.main(() => {+ // My asynchronous function that returns a promise+ return request('http://www.example.com').then(html => {+ console.log(html);+ });+});+```++To simplify your code, you can take advantage of the `async`/`await` keywords:++```javascript+const request = require('some-request-library');++Apify.main(async () => {+ // My asynchronous function+ const html = await request('http://www.example.com');+ console.log(html);+});+```++**Parameters**:++- **`userFunc`**: [`UserFunc`](../typedefs/user-func) - User function to be executed. If it returns a promise, the promise will be awaited. The user+ function is called with no arguments.++---++<a name="metamorph"></a>++## `Apify.metamorph(targetActorId, [input], [options])`++Transforms this actor run to an actor run of a given actor. The system stops the current container and starts the new container instead. All the+default storages are preserved and the new input is stored under the `INPUT-METAMORPH-1` key in the same default key-value store.++**Parameters**:++- **`targetActorId`**: `string` - Either `username/actor-name` or actor ID of an actor to which we want to metamorph.+- **`[input]`**: `Object<string, *>` - Input for the actor. If it is an object, it will be stringified to JSON and its content type set to+ `application/json; charset=utf-8`. Otherwise the `options.contentType` parameter must be provided.+- **`[options]`**: `object` - Object with the settings below:+ - **`[contentType]`**: `string` - Content type for the `input`. If not specified, `input` is expected to be an object that will be stringified+ to JSON and content type set to `application/json; charset=utf-8`. If `options.contentType` is specified, then `input` must be a `String` or+ `Buffer`.+ - **`[build]`**: `string` - Tag or number of the target actor build to metamorph into (e.g. `beta` or `1.2.345`). If not provided, the run uses+ build tag or number from the default actor run configuration (typically `latest`).++**Returns**:++`Promise<void>`++---++<a name="newclient"></a>++## `Apify.newClient([options])`++Returns a new instance of the Apify API client. The `ApifyClient` class is provided by the+<a href="https://www.npmjs.com/package/apify-client" target="_blank">apify-client</a> NPM package, and it is automatically configured using the+`APIFY_API_BASE_URL`, and `APIFY_TOKEN` environment variables. You can override the token via the available options. That's useful if you want to use+the client as a different Apify user than the SDK internals are using.++**Parameters**:++- **`[options]`**: `object`+ - **`[token]`**: `string`+ - **`[maxRetries]`**: `string`+ - **`[minDelayBetweenRetriesMillis]`**: `string`++**Returns**:++[`ApifyClient`](../api/apify)++---++<a name="opendataset"></a>++## `Apify.openDataset([datasetIdOrName], [options])`++Opens a dataset and returns a promise resolving to an instance of the [`Dataset`](../api/dataset) class.++Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. The+actual data is stored either on the local filesystem or in the cloud.++For more details and code examples, see the [`Dataset`](../api/dataset) class.++**Parameters**:++- **`[datasetIdOrName]`**: `string` - ID or name of the dataset to be opened. If `null` or `undefined`, the function returns the default dataset+ associated with the actor run.+- **`[options]`**: `Object`+ - **`[forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the+ `APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.++**Returns**:++[`Promise<Dataset>`](../api/dataset)++---++<a name="openkeyvaluestore"></a>++## `Apify.openKeyValueStore([storeIdOrName], [options])`++Opens a key-value store and returns a promise resolving to an instance of the [`KeyValueStore`](../api/key-value-store) class.++Key-value stores are used to store records or files, along with their MIME content type. The records are stored and retrieved using a unique key. The+actual data is stored either on a local filesystem or in the Apify cloud.++For more details and code examples, see the [`KeyValueStore`](../api/key-value-store) class.++**Parameters**:++- **`[storeIdOrName]`**: `string` - ID or name of the key-value store to be opened. If `null` or `undefined`, the function returns the default+ key-value store associated with the actor run.+- **`[options]`**: `object`+ - **`[forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the+ `APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.++**Returns**:++[`Promise<KeyValueStore>`](../api/key-value-store)++---++<a name="openrequestlist"></a>++## `Apify.openRequestList(listName, sources, [options])`++Opens a request list and returns a promise resolving to an instance of the [`RequestList`](../api/request-list) class that is already initialized.++[`RequestList`](../api/request-list) represents a list of URLs to crawl, which is always stored in memory. To enable picking up where left off after a+process restart, the request list sources are persisted to the key-value store at initialization of the list. Then, while crawling, a small state+object is regularly persisted to keep track of the crawling status.++For more details and code examples, see the [`RequestList`](../api/request-list) class.++**Example usage:**++```javascript+const sources = ['https://www.example.com', 'https://www.google.com', 'https://www.bing.com'];++const requestList = await Apify.openRequestList('my-name', sources);+```++**Parameters**:++- **`listName`**: `string` | `null` - Name of the request list to be opened. Setting a name enables the `RequestList`'s state to be persisted in the+ key-value store. This is useful in case of a restart or migration. Since `RequestList` is only stored in memory, a restart or migration wipes it+ clean. Setting a name will enable the `RequestList`'s state to survive those situations and continue where it left off.++ The name will be used as a prefix in key-value store, producing keys such as `NAME-REQUEST_LIST_STATE` and `NAME-REQUEST_LIST_SOURCES`.++ If `null`, the list will not be persisted and will only be stored in memory. Process restart will then cause the list to be crawled again from the+ beginning. We suggest always using a name.++- **`sources`**: [`Array<(RequestOptions|Request|string)>`](../typedefs/request-options) - An array of sources of URLs for the+ [`RequestList`](../api/request-list). It can be either an array of strings, plain objects that define at least the `url` property, or an array of+ [`Request`](../api/request) instances.++ **IMPORTANT:** The `sources` array will be consumed (left empty) after [`RequestList`](../api/request-list) initializes. This is a measure to+ prevent memory leaks in situations when millions of sources are added.++Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct [`RequestList`](../api/request-list) to download the+source URLs from a given remote location. The URLs will be parsed from the received response. In this case you can limit the URLs using `regex`+parameter containing regular expression pattern for URLs to be included.++For details, see the [`RequestListOptions.sources`](../typedefs/request-list-options#sources)++- **`[options]`**: [`RequestListOptions`](../typedefs/request-list-options) - The [`RequestList`](../api/request-list) options. Note that the+ `listName` parameter supersedes the [`RequestListOptions.persistStateKey`](../typedefs/request-list-options#persiststatekey) and+ [`RequestListOptions.persistRequestsKey`](../typedefs/request-list-options#persistrequestskey) options and the `sources` parameter supersedes the+ [`RequestListOptions.sources`](../typedefs/request-list-options#sources) option.++**Returns**:++[`Promise<RequestList>`](../api/request-list)++---++<a name="openrequestqueue"></a>++## `Apify.openRequestQueue([queueIdOrName], [options])`++Opens a request queue and returns a promise resolving to an instance of the [`RequestQueue`](../api/request-queue) class.++[`RequestQueue`](../api/request-queue) represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud. The queue is+used for deep crawling of websites, where you start with several URLs and then recursively follow links to other pages. The data structure supports+both breadth-first and depth-first crawling orders.++For more details and code examples, see the [`RequestQueue`](../api/request-queue) class.++**Parameters**:++- **`[queueIdOrName]`**: `string` - ID or name of the request queue to be opened. If `null` or `undefined`, the function returns the default request+ queue associated with the actor run.+- **`[options]`**: `object`+ - **`[forceCloud]`**: `boolean` <code> = false</code> - If set to `true` then the function uses cloud storage usage even if the+ `APIFY_LOCAL_STORAGE_DIR` environment variable is set. This way it is possible to combine local and cloud storage.++**Returns**:++[`Promise<RequestQueue>`](../api/request-queue)++---++<a name="opensessionpool"></a>++## `Apify.openSessionPool(sessionPoolOptions)`++Opens a SessionPool and returns a promise resolving to an instance of the [`SessionPool`](../api/session-pool) class that is already initialized.++For more details and code examples, see the [`SessionPool`](../api/session-pool) class.++**Parameters**:++- **`sessionPoolOptions`**: [`SessionPoolOptions`](../typedefs/session-pool-options)++**Returns**:++[`Promise<SessionPool>`](../api/session-pool)++---++<a name="pushdata"></a>++## `Apify.pushData(item)`++Stores an object or an array of objects to the default [`Dataset`](../api/dataset) of the current actor run.++This is just a convenient shortcut for [`Dataset.pushData()`](../api/dataset#pushdata). For example, calling the following code:++```javascript+await Apify.pushData({ myValue: 123 });+```++is equivalent to:++```javascript+const dataset = await Apify.openDataset();+await dataset.pushData({ myValue: 123 });+```++For more information, see [`Apify.openDataset()`](../api/apify#opendataset) and [`Dataset.pushData()`](../api/dataset#pushdata)++**IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`, otherwise the actor process might finish before the data are stored!++**Parameters**:++- **`item`**: `object` - Object or array of objects containing data to be stored in the default dataset. The objects must be serializable to JSON+ and the JSON representation of each object must be smaller than 9MB.++**Returns**:++`Promise<void>`++---++<a name="setvalue"></a>++## `Apify.setValue(key, value, [options])`++Stores or deletes a value in the default [`KeyValueStore`](../api/key-value-store) associated with the current actor run.++This is just a convenient shortcut for [`KeyValueStore.setValue()`](../api/key-value-store#setvalue). For example, calling the following code:++```javascript+await Apify.setValue('OUTPUT', { foo: 'bar' });+```++is equivalent to:++```javascript+const store = await Apify.openKeyValueStore();+await store.setValue('OUTPUT', { foo: 'bar' });+```++To get a value from the default key-value store, you can use the [`Apify.getValue()`](../api/apify#getvalue) function.++For more information, see [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) and+[`KeyValueStore.getValue()`](../api/key-value-store#getvalue).++**Parameters**:++- **`key`**: `string` - Unique record key.+- **`value`**: `*` - Record data, which can be one of the following values:+ - If `null`, the record in the key-value store is deleted.+ - If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.+ - If `options.contentType` is set, `value` is taken as is and it must be a `String` or [`Buffer`](https://nodejs.org/api/buffer.html). For any+ other value an error will be thrown.+- **`[options]`**: `object`+ - **`[contentType]`**: `string` - Specifies a custom MIME content type of the record.++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/api/AutoscaledPool.md b/website/versioned_docs/version-1.0.2/api/AutoscaledPool.mdnew file mode 100644index 000000000..7fc1dfe2d--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/AutoscaledPool.md@@ -0,0 +1,219 @@+---+id: version-1.0.2-autoscaled-pool+title: AutoscaledPool+original_id: autoscaled-pool+---++<a name="autoscaledpool"></a>++Manages a pool of asynchronous resource-intensive tasks that are executed in parallel. The pool only starts new tasks if there is enough free CPU and+memory available and the Javascript event loop is not blocked.++The information about the CPU and memory usage is obtained by the [`Snapshotter`](../api/snapshotter) class, which makes regular snapshots of system+resources that may be either local or from the Apify cloud infrastructure in case the process is running on the Apify platform. Meaningful data+gathered from these snapshots is provided to `AutoscaledPool` by the [`SystemStatus`](../api/system-status) class.++Before running the pool, you need to implement the following three functions:+[`AutoscaledPoolOptions.runTaskFunction()`](../typedefs/autoscaled-pool-options#runtaskfunction),+[`AutoscaledPoolOptions.isTaskReadyFunction()`](../typedefs/autoscaled-pool-options#istaskreadyfunction) and+[`AutoscaledPoolOptions.isFinishedFunction()`](../typedefs/autoscaled-pool-options#isfinishedfunction).++The auto-scaled pool is started by calling the [`AutoscaledPool.run()`](../api/autoscaled-pool#run) function. The pool periodically queries the+[`AutoscaledPoolOptions.isTaskReadyFunction()`](../typedefs/autoscaled-pool-options#istaskreadyfunction) function for more tasks, managing optimal+concurrency, until the function resolves to `false`. The pool then queries the+[`AutoscaledPoolOptions.isFinishedFunction()`](../typedefs/autoscaled-pool-options#isfinishedfunction). If it resolves to `true`, the run finishes+after all running tasks complete. If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for+tasks. If any of the tasks throws then the [`AutoscaledPool.run()`](../api/autoscaled-pool#run) function rejects the promise with an error.++The pool evaluates whether it should start a new task every time one of the tasks finishes and also in the interval set by the+`options.maybeRunIntervalSecs` parameter.++**Example usage:**++```javascript+const pool = new Apify.AutoscaledPool({+ maxConcurrency: 50,+ runTaskFunction: async () => {+ // Run some resource-intensive asynchronous operation here.+ },+ isTaskReadyFunction: async () => {+ // Tell the pool whether more tasks are ready to be processed.+ // Return true or false+ },+ isFinishedFunction: async () => {+ // Tell the pool whether it should finish+ // or wait for more tasks to become available.+ // Return true or false+ },+});++await pool.run();+```++---++<a name="autoscaledpool"></a>++## `new AutoscaledPool(options)`++**Parameters**:++- **`options`**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options) - All `AutoscaledPool` configuration options.++---++<a name="log"></a>++## `autoscaledPool.log`++---++<a name="minconcurrency"></a>++## `autoscaledPool.minConcurrency`++Gets the minimum number of tasks running in parallel.++**Returns**:++`number`++---++<a name="minconcurrency"></a>++## `autoscaledPool.minConcurrency`++Sets the minimum number of tasks running in parallel.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash. If you're+not sure, just keep the default value and the concurrency will scale up automatically.++**Parameters**:++- **`value`**: `number`++---++<a name="maxconcurrency"></a>++## `autoscaledPool.maxConcurrency`++Gets the maximum number of tasks running in parallel.++**Returns**:++`number`++---++<a name="maxconcurrency"></a>++## `autoscaledPool.maxConcurrency`++Sets the maximum number of tasks running in parallel.++**Parameters**:++- **`value`**: `number`++---++<a name="desiredconcurrency"></a>++## `autoscaledPool.desiredConcurrency`++Gets the desired concurrency for the pool, which is an estimated number of parallel tasks that the system can currently support.++**Returns**:++`number`++---++<a name="desiredconcurrency"></a>++## `autoscaledPool.desiredConcurrency`++Sets the desired concurrency for the pool, i.e. the number of tasks that should be running in parallel if there's large enough supply of tasks.++**Parameters**:++- **`value`**: `number`++---++<a name="currentconcurrency"></a>++## `autoscaledPool.currentConcurrency`++Gets the the number of parallel tasks currently running in the pool.++**Returns**:++`number`++---++<a name="run"></a>++## `autoscaledPool.run()`++Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once all the tasks are finished or one of them fails.++**Returns**:++`Promise<void>`++---++<a name="abort"></a>++## `autoscaledPool.abort()`++Aborts the run of the auto-scaled pool and destroys it. The promise returned from the [`AutoscaledPool.run()`](../api/autoscaled-pool#run) function+will immediately resolve, no more new tasks will be spawned and all running tasks will be left in their current state.++Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion of all the running tasks, therefore, no abortion is attempted and+some of the tasks may finish, while others may not. Essentially, auto-scaled pool doesn't care about their state after the invocation of `.abort()`,+but that does not mean that some parts of their asynchronous chains of commands will not execute.++**Returns**:++`Promise<void>`++---++<a name="pause"></a>++## `autoscaledPool.pause([timeoutSecs])`++Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish (unlike abort, which terminates them). Used together with+[`AutoscaledPool.resume()`](../api/autoscaled-pool#resume)++The function's promise will resolve once all running tasks have completed and the pool is effectively idle. If the `timeoutSecs` argument is provided,+the promise will reject with a timeout error after the `timeoutSecs` seconds.++The promise returned from the [`AutoscaledPool.run()`](../api/autoscaled-pool#run) function will not resolve when `.pause()` is invoked (unlike abort,+which resolves it).++**Parameters**:++- **`[timeoutSecs]`**: `number`++**Returns**:++`Promise<void>`++---++<a name="resume"></a>++## `autoscaledPool.resume()`++Resumes the operation of the autoscaled-pool by allowing more tasks to be run. Used together with+[`AutoscaledPool.pause()`](../api/autoscaled-pool#pause)++Tasks will automatically start running again in `options.maybeRunIntervalSecs`.++---diff --git a/website/versioned_docs/version-1.0.2/api/BasicCrawler.md b/website/versioned_docs/version-1.0.2/api/BasicCrawler.mdnew file mode 100644index 000000000..8a31ae5ec--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/BasicCrawler.md@@ -0,0 +1,140 @@+---+id: version-1.0.2-basic-crawler+title: BasicCrawler+original_id: basic-crawler+---++<a name="basiccrawler"></a>++Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of+URLs enabling recursive crawling of websites.++`BasicCrawler` is a low-level tool that requires the user to implement the page download and data extraction functionality themselves. If you want a+crawler that already facilitates this functionality, please consider using [`CheerioCrawler`](../api/cheerio-crawler),+[`PuppeteerCrawler`](../api/puppeteer-crawler) or [`PlaywrightCrawler`](../api/playwright-crawler).++`BasicCrawler` invokes the user-provided [`BasicCrawlerOptions.handleRequestFunction`](../typedefs/basic-crawler-options#handlerequestfunction) for+each [`Request`](../api/request) object, which represents a single URL to crawl. The [`Request`](../api/request) objects are fed from the+[`RequestList`](../api/request-list) or the [`RequestQueue`](../api/request-queue) instances provided by the+[`BasicCrawlerOptions.requestList`](../typedefs/basic-crawler-options#requestlist) or+[`BasicCrawlerOptions.requestQueue`](../typedefs/basic-crawler-options#requestqueue) constructor options, respectively.++If both [`BasicCrawlerOptions.requestList`](../typedefs/basic-crawler-options#requestlist) and+[`BasicCrawlerOptions.requestQueue`](../typedefs/basic-crawler-options#requestqueue) options are used, the instance first processes URLs from the+[`RequestList`](../api/request-list) and automatically enqueues all of them to [`RequestQueue`](../api/request-queue) before it starts their+processing. This ensures that a single URL is not crawled multiple times.++The crawler finishes if there are no more [`Request`](../api/request) objects to crawl.++New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the+[`AutoscaledPool`](../api/autoscaled-pool) class. All [`AutoscaledPool`](../api/autoscaled-pool) configuration options can be passed to the+`autoscaledPoolOptions` parameter of the `BasicCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`+[`AutoscaledPool`](../api/autoscaled-pool) options are available directly in the `BasicCrawler` constructor.++**Example usage:**++```javascript+// Prepare a list of URLs to crawl+const requestList = new Apify.RequestList({+ sources: [{ url: 'http://www.example.com/page-1' }, { url: 'http://www.example.com/page-2' }],+});+await requestList.initialize();++// Crawl the URLs+const crawler = new Apify.BasicCrawler({+ requestList,+ handleRequestFunction: async ({ request }) => {+ // 'request' contains an instance of the Request class+ // Here we simply fetch the HTML of the page and store it to a dataset+ const { body } = await Apify.utils.requestAsBrowser(request);+ await Apify.pushData({+ url: request.url,+ html: body,+ });+ },+});++await crawler.run();+```++## Properties++### `stats`++**Type**: [`Statistics`](../api/statistics)++Contains statistics about the current run.++---++### `requestList`++**Type**: [`RequestList`](../api/request-list)++A reference to the underlying [`RequestList`](../api/request-list) class that manages the crawler's [`Request`](../api/request)s. Only available if+used by the crawler.++---++### `requestQueue`++**Type**: [`RequestQueue`](../api/request-queue)++A reference to the underlying [`RequestQueue`](../api/request-queue) class that manages the crawler's [`Request`](../api/request)s. Only available if+used by the crawler.++---++### `sessionPool`++**Type**: [`SessionPool`](../api/session-pool)++A reference to the underlying [`SessionPool`](../api/session-pool) class that manages the crawler's [`Session`](../api/session)s. Only available if+used by the crawler.++---++### `autoscaledPool`++**Type**: [`AutoscaledPool`](../api/autoscaled-pool)++A reference to the underlying [`AutoscaledPool`](../api/autoscaled-pool) class that manages the concurrency of the crawler. Note that this property is+only initialized after calling the [`BasicCrawler.run()`](../api/basic-crawler#run) function. You can use it to change the concurrency settings on the+fly, to pause the crawler by calling [`AutoscaledPool.pause()`](../api/autoscaled-pool#pause) or to abort it by calling+[`AutoscaledPool.abort()`](../api/autoscaled-pool#abort).++---++<a name="basiccrawler"></a>++## `new BasicCrawler(options)`++**Parameters**:++- **`options`**: [`BasicCrawlerOptions`](../typedefs/basic-crawler-options) - All `BasicCrawler` parameters are passed via an options object.++---++<a name="log"></a>++## `basicCrawler.log`++---++<a name="sessionpooloptions"></a>++## `basicCrawler.sessionPoolOptions`++---++<a name="run"></a>++## `basicCrawler.run()`++Runs the crawler. Returns a promise that gets resolved once all the requests are processed.++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/api/BrowserPlugin.md b/website/versioned_docs/version-1.0.2/api/BrowserPlugin.mdnew file mode 100644index 000000000..7756f07c8--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/BrowserPlugin.md@@ -0,0 +1,20 @@+---+id: version-1.0.2-browser-plugin+title: BrowserPlugin+original_id: browser-plugin+---++<a name="browserplugin"></a>++---++<a name="exports.browserplugin"></a>++## `new BrowserPlugin(launcher, context)`++**Parameters**:++- **`launcher`**: `*`+- **`context`**: `Object<string, *>`++---diff --git a/website/versioned_docs/version-1.0.2/api/KeyValueStore.md b/website/versioned_docs/version-1.0.2/api/KeyValueStore.mdnew file mode 100644index 000000000..a119948ac--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/KeyValueStore.md@@ -0,0 +1,210 @@+---+id: version-1.0.2-key-value-store+title: KeyValueStore+original_id: key-value-store+---++<a name="keyvaluestore"></a>++The `KeyValueStore` class represents a key-value store, a simple data storage that is used for saving and reading data records or files. Each data+record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots, actor inputs and+outputs, web pages, PDFs or to persist the state of crawlers.++Do not instantiate this class directly, use the [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) function instead.++Each actor run is associated with a default key-value store, which is created exclusively for the run. By convention, the actor input and output are+stored into the default key-value store under the `INPUT` and `OUTPUT` key, respectively. Typically, input and output are JSON files, although it can+be any other format. To access the default key-value store directly, you can use the [`Apify.getValue()`](../api/apify#getvalue) and+[`Apify.setValue()`](../api/apify#setvalue) convenience functions.++To access the input, you can also use the [`Apify.getInput()`](../api/apify#getinput) convenience function.++`KeyValueStore` stores its data either on local disk or in the Apify cloud, depending on whether the+[`APIFY_LOCAL_STORAGE_DIR`](../guides/environment-variables#apify_local_storage_dir) or [`APIFY_TOKEN`](../guides/environment-variables#apify_token)+environment variables are set.++If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in the local directory in the following files:++```+{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}+```++Note that `{STORE_ID}` is the name or ID of the key-value store. The default key-value store has ID: `default`, unless you override it by setting the+`APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the+data value.++If the [`APIFY_TOKEN`](../guides/environment-variables#apify_token) environment variable is set but+[`APIFY_LOCAL_STORAGE_DIR`](../guides/environment-variables#apify_local_storage_dir) not, the data is stored in the+[Apify Key-value store](https://docs.apify.com/storage/key-value-store) cloud storage. Note that you can force usage of the cloud storage also by+passing the `forceCloud` option to [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) function, even if the+[`APIFY_LOCAL_STORAGE_DIR`](../guides/environment-variables#apify_local_storage_dir) variable is set.++**Example usage:**++```javascript+// Get actor input from the default key-value store+const input = await Apify.getInput();+const otherValue = Apify.getValue('my-key');++// Write actor output to the default key-value store.+await Apify.setValue('OUTPUT', { myResult: 123 });++// Open a named key-value store+const store = await Apify.openKeyValueStore('some-name');++// Write a record. JavaScript object is automatically converted to JSON,+// strings and binary buffers are stored as they are+await store.setValue('some-key', { foo: 'bar' });++// Read a record. Note that JSON is automatically parsed to a JavaScript object,+// text data returned as a string and other data is returned as binary buffer+const value = await store.getValue('some-key');++// Drop (delete) the store+await store.drop();+```++---++<a name="getvalue"></a>++## `keyValueStore.getValue(key)`++Gets a value from the key-value store.++The function returns a `Promise` that resolves to the record value, whose JavaScript type depends on the MIME content type of the record. Records with+the `application/json` content type are automatically parsed and returned as a JavaScript object. Similarly, records with `text/plain` content types+are returned as a string. For all other content types, the value is returned as a raw [`Buffer`](https://nodejs.org/api/buffer.html) instance.++If the record does not exist, the function resolves to `null`.++To save or delete a value in the key-value store, use the [`KeyValueStore.setValue()`](../api/key-value-store#setvalue) function.++**Example usage:**++```javascript+const store = await Apify.openKeyValueStore();+const buffer = await store.getValue('screenshot1.png');+```++**Parameters**:++- **`key`**: `string` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,+ `A`-`Z`, `0`-`9` and `!-_.'()`++**Returns**:++[`Promise<KeyValueStoreValueTypes>`](../typedefs/key-value-store-value-types) - Returns a promise that resolves to an object, string or+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record.++---++<a name="setvalue"></a>++## `keyValueStore.setValue(key, value, [options])`++Saves or deletes a record in the key-value store. The function returns a promise that resolves once the record has been saved or deleted.++**Example usage:**++```javascript+const store = await Apify.openKeyValueStore();+await store.setValue('OUTPUT', { foo: 'bar' });+```++Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`++By default, `value` is converted to JSON and stored with the `application/json; charset=utf-8` MIME content type. To store the value with another+content type, pass it in the options as follows:++```javascript+const store = await Apify.openKeyValueStore('my-text-store');+await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });+```++If you set custom content type, `value` must be either a string or [`Buffer`](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.++If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds regardless whether the record existed or not.++To retrieve a value from the key-value store, use the [`KeyValueStore.getValue()`](../api/key-value-store#getvalue) function.++**IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`, otherwise the actor process might finish before the value is+stored!++**Parameters**:++- **`key`**: `string` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,+ `A`-`Z`, `0`-`9` and `!-_.'()`+- **`value`**: [`KeyValueStoreValueTypes`](../typedefs/key-value-store-value-types) - Record data, which can be one of the following values:+ - If `null`, the record in the key-value store is deleted.+ - If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.+ - If `options.contentType` is set, `value` is taken as is and it must be a `String` or [`Buffer`](https://nodejs.org/api/buffer.html). For any+ other value an error will be thrown.+- **`[options]`**: `object`+ - **`[contentType]`**: `string` - Specifies a custom MIME content type of the record.++**Returns**:++`Promise<void>`++---++<a name="drop"></a>++## `keyValueStore.drop()`++Removes the key-value store either from the Apify cloud storage or from the local directory, depending on the mode of operation.++**Returns**:++`Promise<void>`++---++<a name="getpublicurl"></a>++## `keyValueStore.getPublicUrl(key)`++Returns a URL for the given key that may be used to publicly access the value in the remote key-value store.++**Parameters**:++- **`key`**: `string`++**Returns**:++`string`++---++<a name="foreachkey"></a>++## `keyValueStore.forEachKey(iteratee, [options])`++Iterates over key-value store keys, yielding each in turn to an `iteratee` function. Each invocation of `iteratee` is called with three arguments:+`(key, index, info)`, where `key` is the record key, `index` is a zero-based index of the key in the current iteration (regardless of+`options.exclusiveStartKey`) and `info` is an object that contains a single property `size` indicating size of the record in bytes.++If the `iteratee` function returns a Promise then it is awaited before the next call. If it throws an error, the iteration is aborted and the+`forEachKey` function throws the error.++**Example usage**++```javascript+const keyValueStore = await Apify.openKeyValueStore();+await keyValueStore.forEachKey(async (key, index, info) => {+ console.log(`Key at ${index}: ${key} has size ${info.size}`);+});+```++**Parameters**:++- **`iteratee`**: [`KeyConsumer`](../typedefs/key-consumer) - A function that is called for every key in the key-value store.+- **`[options]`**: `object` - All `forEachKey()` parameters are passed via an options object with the following keys:+ - **`[exclusiveStartKey]`**: `string` - All keys up to this one (including) are skipped from the result.++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/api/PseudoUrl.md b/website/versioned_docs/version-1.0.2/api/PseudoUrl.mdnew file mode 100644index 000000000..caab5381d--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/PseudoUrl.md@@ -0,0 +1,101 @@+---+id: version-1.0.2-pseudo-url+title: PseudoUrl+original_id: pseudo-url+---++<a name="pseudourl"></a>++Represents a pseudo-URL (PURL) - an URL pattern used by web crawlers to specify which URLs should the crawler visit. This class is used by the+[`utils.enqueueLinks()`](../api/utils#enqueuelinks) function.++A PURL is simply a URL with special directives enclosed in `[]` brackets. Currently, the only supported directive is `[RegExp]`, which defines a+JavaScript-style regular expression to match against the URL.++The `PseudoUrl` class can be constructed either using a pseudo-URL string or a regular expression (an instance of the `RegExp` object). With a+pseudo-URL string, the matching is always case-insensitive. If you need case-sensitive matching, use an appropriate `RegExp` object.++For example, a PURL `http://www.example.com/pages/[(\w|-)*]` will match all of the following URLs:++- `http://www.example.com/pages/`+- `http://www.example.com/pages/my-awesome-page`+- `http://www.example.com/pages/something`++Be careful to correctly escape special characters in the pseudo-URL string. If either `[` or `]` is part of the normal query string, it must be+encoded as `[\x5B]` or `[\x5D]`, respectively. For example, the following PURL:++```http+http://www.example.com/search?do[\x5B]load[\x5D]=1+```++will match the URL:++```http+http://www.example.com/search?do[load]=1+```++If the regular expression in the pseudo-URL contains a backslash character (\), you need to escape it with another back backslash, as shown in the+example below.++**Example usage:**++```javascript+// Using a pseudo-URL string+const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\\w|-)+]', {+ userData: { foo: 'bar' },+});++// Using a regular expression+const purl2 = new Apify.PseudoUrl(/http:\/\/www\.example\.com\/pages\/(\w|-)+/);++if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');+```++---++<a name="pseudourl"></a>++## `new PseudoUrl(purl, requestTemplate)`++**Parameters**:++- **`purl`**: `string` | `RegExp` - A pseudo-URL string or a regular expression object. Using a `RegExp` instance enables more granular control,+ such as making the matching case sensitive.+- **`requestTemplate`**: [`RequestOptions`](../typedefs/request-options) - Options for the new [`Request`](../api/request) instances created for+ matching URLs by the [`utils.enqueueLinks()`](../api/utils#enqueuelinks) function.++---++<a name="matches"></a>++## `pseudoUrl.matches(url)`++Determines whether a URL matches this pseudo-URL pattern.++**Parameters**:++- **`url`**: `string` - URL to be matched.++**Returns**:++`boolean` - Returns `true` if given URL matches pseudo-URL.++---++<a name="createrequest"></a>++## `pseudoUrl.createRequest(urlOrProps)`++Creates a Request object from a provided `requestTemplate` and a given URL or an object that specifies \$[`Request`](../api/request) properties. In+case of a collision the properties will override the template, except for `userData`, which will be merged together, with the `userData` property+having preference over the template. This enables dynamic overriding of the template.++**Parameters**:++- **`urlOrProps`**: `string` | `Object<string, *>`++**Returns**:++[`Request`](../api/request)++---diff --git a/website/versioned_docs/version-1.0.2/api/Request.md b/website/versioned_docs/version-1.0.2/api/Request.mdnew file mode 100644index 000000000..0111eb83b--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/Request.md@@ -0,0 +1,177 @@+---+id: version-1.0.2-request+title: Request+original_id: request+---++<a name="request"></a>++Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata. The `Request` object also stores information+about errors that occurred during processing of the request.++Each `Request` instance has the `uniqueKey` property, which can be either specified manually in the constructor or generated automatically from the+URL. Two requests with the same `uniqueKey` are considered as pointing to the same web resource. This behavior applies to all Apify SDK classes, such+as [`RequestList`](../api/request-list), [`RequestQueue`](../api/request-queue), [`PuppeteerCrawler`](../api/puppeteer-crawler) or+[`PlaywrightCrawler`](../api/playwright-crawler).++Example use:++```javascript+const request = new Apify.Request({+ url: 'http://www.example.com',+ headers: { Accept: 'application/json' },+});++...++request.userData.foo = 'bar';+request.pushErrorMessage(new Error('Request failed!'));++...++const foo = request.userData.foo;+```++## Properties++### `id`++**Type**: `string`++Request ID++---++### `url`++**Type**: `string`++URL of the web page to crawl.++---++### `loadedUrl`++**Type**: `string`++An actually loaded URL after redirects, if present. HTTP redirects are guaranteed to be included.++When using [`PuppeteerCrawler`](../api/puppeteer-crawler) or [`PlaywrightCrawler`](../api/playwright-crawler), meta tag and JavaScript redirects may,+or may not be included, depending on their nature. This generally means that redirects, which happen immediately will most likely be included, but+delayed redirects will not.++---++### `uniqueKey`++**Type**: `string`++A unique key identifying the request. Two requests with the same `uniqueKey` are considered as pointing to the same URL.++---++### `method`++**Type**: `string`++HTTP method, e.g. `GET` or `POST`.++---++### `payload`++**Type**: `string` | `Buffer`++HTTP request payload, e.g. for POST requests.++---++### `noRetry`++**Type**: `boolean`++The `true` value indicates that the request will not be automatically retried on error.++---++### `retryCount`++**Type**: `number`++Indicates the number of times the crawling of the request has been retried on error.++---++### `errorMessages`++**Type**: `Array<string>`++An array of error messages from request processing.++---++### `headers`++**Type**: `Object<string, string>`++Object with HTTP headers. Key is header name, value is the value.++---++### `userData`++**Type**: `Object<string, *>`++Custom user data assigned to the request.++---++### `handledAt`++**Type**: `Date`++Indicates the time when the request has been processed. Is `null` if the request has not been crawled yet.++---++<a name="request"></a>++## `new Request(options)`++**Parameters**:++- **`options`**: [`RequestOptions`](../typedefs/request-options) - `Request` parameters including the URL, HTTP method and headers, and others.++---++<a name="headers"></a>++## `request.headers`++---++<a name="userdata"></a>++## `request.userData`++---++<a name="pusherrormessage"></a>++## `request.pushErrorMessage(errorOrMessage, [options])`++Stores information about an error that occurred during processing of this request.++You should always use Error instances when throwing errors in JavaScript.++Nevertheless, to improve the debugging experience when using third party libraries that may not always throw an Error instance, the function performs+a type inspection of the passed argument and attempts to extract as much information as possible, since just throwing a bad type error makes any+debugging rather difficult.++**Parameters**:++- **`errorOrMessage`**: `Error` | `string` - Error object or error message to be stored in the request.+- **`[options]`**: `object`+ - **`[omitStack]`**: `boolean` <code> = false</code> - Only push the error message without stack trace when true.++---diff --git a/website/versioned_docs/version-1.0.2/api/RequestList.md b/website/versioned_docs/version-1.0.2/api/RequestList.mdnew file mode 100644index 000000000..72f83ebb4--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/RequestList.md@@ -0,0 +1,223 @@+---+id: version-1.0.2-request-list+title: RequestList+original_id: request-list+---++<a name="requestlist"></a>++Represents a static list of URLs to crawl. The URLs can be provided either in code or parsed from a text file hosted on the web. `RequestList` is used+by [`BasicCrawler`](../api/basic-crawler), [`CheerioCrawler`](../api/cheerio-crawler), [`PuppeteerCrawler`](../api/puppeteer-crawler) and+[`PlaywrightCrawler`](../api/playwright-crawler) as a source of URLs to crawl.++Each URL is represented using an instance of the [`Request`](../api/request) class. The list can only contain unique URLs. More precisely, it can only+contain `Request` instances with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden. To+add a single URL to the list multiple times, corresponding [`Request`](../api/request) objects will need to have different `uniqueKey` properties. You+can use the `keepDuplicateUrls` option to do this for you when initializing the `RequestList` from sources.++Once you create an instance of `RequestList`, you need to call the [`RequestList.initialize()`](../api/request-list#initialize) function before the+instance can be used. After that, no more URLs can be added to the list. Unlike [`RequestQueue`](../api/request-queue), `RequestList` is static but it+can contain even millions of URLs.++> Note that `RequestList` can be used together with `RequestQueue` by the same crawler. In such cases, each request from `RequestList` is enqueued+> into `RequestQueue` first and then consumed from the latter. This is necessary to avoid the same URL being processed more than once (from the list+> first and then possibly from the queue). In practical terms, such a combination can be useful when there is a large number of initial URLs, but more+> URLs would be added dynamically by the crawler.++`RequestList` has an internal state where it stores information about which requests were already handled, which are in progress and which were+reclaimed. The state may be automatically persisted to the default [`KeyValueStore`](../api/key-value-store) by setting the `persistStateKey` option+so that if the Node.js process is restarted, the crawling can continue where it left off. The automated persisting is launched upon receiving the+`persistState` event that is periodically emitted by [`Apify.events`](../api/apify#events).++The internal state is closely tied to the provided sources (URLs). If the sources change on actor restart, the state will become corrupted and+`RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web. In such case, use the+`persistRequestsKey` option in conjunction with `persistStateKey`, to make the `RequestList` store the initial sources to the default key-value store+and load them after restart, which will prevent any issues that a live list of URLs might cause.++**Basic usage:**++```javascript+// Use a helper function to simplify request list initialization.+// State and sources are automatically persisted. This is a preferred usage.+const requestList = await Apify.openRequestList('my-request-list', [+ 'http://www.example.com/page-1',+ { url: 'http://www.example.com/page-2', method: 'POST', userData: { foo: 'bar' } },+ { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },+]);+```++**Advanced usage:**++```javascript+// Use the constructor to get more control over the initialization.+const requestList = new Apify.RequestList({+ sources: [+ // Separate requests+ { url: 'http://www.example.com/page-1', method: 'GET', headers: { ... } },+ { url: 'http://www.example.com/page-2', userData: { foo: 'bar' }},++ // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`+ // Note that all URLs must start with http:// or https://+ { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },+ ],++ // Persist the state to avoid re-crawling which can lead to data duplications.+ // Keep in mind that the sources have to be immutable or this will throw an error.+ persistStateKey: 'my-state',+});++await requestList.initialize();+```++---++<a name="exports.requestlist"></a>++## `new RequestList(options)`++**Parameters**:++- **`options`**: [`RequestListOptions`](../typedefs/request-list-options) - All `RequestList` configuration options++---++<a name="requests"></a>++## `requestList.requests`++---++<a name="initialize"></a>++## `requestList.initialize()`++Loads all remote sources of URLs and potentially starts periodic state persistence. This function must be called before you can start using the+instance in a meaningful way.++**Returns**:++`Promise<void>`++---++<a name="persiststate"></a>++## `requestList.persistState()`++Persists the current state of the `RequestList` into the default [`KeyValueStore`](../api/key-value-store). The state is persisted automatically in+regular intervals, but calling this method manually is useful in cases where you want to have the most current state available after you pause or stop+fetching its requests. For example after you pause or abort a crawl. Or just before a server migration.++**Returns**:++`Promise<void>`++---++<a name="getstate"></a>++## `requestList.getState()`++Returns an object representing the internal state of the `RequestList` instance. Note that the object's fields can change in future releases.++**Returns**:++[`RequestListState`](../typedefs/request-list-state)++---++<a name="isempty"></a>++## `requestList.isEmpty()`++Resolves to `true` if the next call to [`RequestList.fetchNextRequest()`](../api/request-list#fetchnextrequest) function would return `null`,+otherwise it resolves to `false`. Note that even if the list is empty, there might be some pending requests currently being processed.++**Returns**:++`Promise<boolean>`++---++<a name="isfinished"></a>++## `requestList.isFinished()`++Returns `true` if all requests were already handled and there are no more left.++**Returns**:++`Promise<boolean>`++---++<a name="fetchnextrequest"></a>++## `requestList.fetchNextRequest()`++Gets the next [`Request`](../api/request) to process. First, the function gets a request previously reclaimed using the+[`RequestList.reclaimRequest()`](../api/request-list#reclaimrequest) function, if there is any. Otherwise it gets the next request from sources.++The function's `Promise` resolves to `null` if there are no more requests to process.++**Returns**:++[`Promise<(Request|null)>`](../api/request)++---++<a name="markrequesthandled"></a>++## `requestList.markRequestHandled(request)`++Marks request as handled after successful processing.++**Parameters**:++- **`request`**: [`Request`](../api/request)++**Returns**:++`Promise<void>`++---++<a name="reclaimrequest"></a>++## `requestList.reclaimRequest(request)`++Reclaims request to the list if its processing failed. The request will become available in the next `this.fetchNextRequest()`.++**Parameters**:++- **`request`**: [`Request`](../api/request)++**Returns**:++`Promise<void>`++---++<a name="length"></a>++## `requestList.length()`++Returns the total number of unique requests present in the `RequestList`.++**Returns**:++`number`++---++<a name="handledcount"></a>++## `requestList.handledCount()`++Returns number of handled requests.++**Returns**:++`number`++---diff --git a/website/versioned_docs/version-1.0.2/api/RequestQueue.md b/website/versioned_docs/version-1.0.2/api/RequestQueue.mdnew file mode 100644index 000000000..9f242ed75--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/RequestQueue.md@@ -0,0 +1,235 @@+---+id: version-1.0.2-request-queue+title: RequestQueue+original_id: request-queue+---++<a name="requestqueue"></a>++Represents a queue of URLs to crawl, which is used for deep crawling of websites where you start with several URLs and then recursively follow links+to other pages. The data structure supports both breadth-first and depth-first crawling orders.++Each URL is represented using an instance of the [`Request`](../api/request) class. The queue can only contain unique URLs. More precisely, it can+only contain [`Request`](../api/request) instances with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can+also be overridden. To add a single URL multiple times to the queue, corresponding [`Request`](../api/request) objects will need to have different+`uniqueKey` properties.++Do not instantiate this class directly, use the [`Apify.openRequestQueue()`](../api/apify#openrequestqueue) function instead.++`RequestQueue` is used by [`BasicCrawler`](../api/basic-crawler), [`CheerioCrawler`](../api/cheerio-crawler),+[`PuppeteerCrawler`](../api/puppeteer-crawler) and [`PlaywrightCrawler`](../api/playwright-crawler) as a source of URLs to crawl. Unlike+[`RequestList`](../api/request-list), `RequestQueue` supports dynamic adding and removing of requests. On the other hand, the queue is not optimized+for operations that add or remove a large number of URLs in a batch.++`RequestQueue` stores its data either on local disk or in the Apify Cloud, depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN`+environment variable is set.++If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the queue data is stored in that directory in an SQLite database file.++If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` is not, the data is stored in the+[Apify Request Queue](https://docs.apify.com/storage/request-queue) cloud storage. Note that you can force usage of the cloud storage also by passing+the `forceCloud` option to [`Apify.openRequestQueue()`](../api/apify#openrequestqueue) function, even if the `APIFY_LOCAL_STORAGE_DIR` variable is+set.++**Example usage:**++```javascript+// Open the default request queue associated with the actor run+const queue = await Apify.openRequestQueue();++// Open a named request queue+const queueWithName = await Apify.openRequestQueue('some-name');++// Enqueue few requests+await queue.addRequest({ url: 'http://example.com/aaa' });+await queue.addRequest({ url: 'http://example.com/bbb' });+await queue.addRequest({ url: 'http://example.com/foo/bar' }, { forefront: true });+```++---++<a name="addrequest"></a>++## `requestQueue.addRequest(requestLike, [options])`++Adds a request to the queue.++If a request with the same `uniqueKey` property is already present in the queue, it will not be updated. You can find out whether this happened from+the resulting [`QueueOperationInfo`](../typedefs/queue-operation-info) object.++To add multiple requests to the queue by extracting links from a webpage, see the [`utils.enqueueLinks()`](../api/utils#enqueuelinks) helper function.++**Parameters**:++- **`requestLike`**: [`Request`](../api/request) | [`RequestOptions`](../typedefs/request-options) - [`Request`](../api/request) object or vanilla+ object with request data. Note that the function sets the `uniqueKey` and `id` fields to the passed Request.+- **`[options]`**: `Object`+ - **`[forefront]`**: `boolean` <code> = false</code> - If `true`, the request will be added to the foremost position in the queue.++**Returns**:++[`Promise<QueueOperationInfo>`](../typedefs/queue-operation-info)++---++<a name="getrequest"></a>++## `requestQueue.getRequest(id)`++Gets the request from the queue specified by ID.++**Parameters**:++- **`id`**: `string` - ID of the request.++**Returns**:++[`Promise<(Request|null)>`](../api/request) - Returns the request object, or `null` if it was not found.++---++<a name="fetchnextrequest"></a>++## `requestQueue.fetchNextRequest()`++Returns a next request in the queue to be processed, or `null` if there are no more pending requests.++Once you successfully finish processing of the request, you need to call+[`RequestQueue.markRequestHandled()`](../api/request-queue#markrequesthandled) to mark the request as handled in the queue. If there was some error in+processing the request, call [`RequestQueue.reclaimRequest()`](../api/request-queue#reclaimrequest) instead, so that the queue will give the request+to some other consumer in another call to the `fetchNextRequest` function.++Note that the `null` return value doesn't mean the queue processing finished, it means there are currently no pending requests. To check whether all+requests in queue were finished, use [`RequestQueue.isFinished()`](../api/request-queue#isfinished) instead.++**Returns**:++[`Promise<(Request|null)>`](../api/request) - Returns the request object or `null` if there are no more pending requests.++---++<a name="markrequesthandled"></a>++## `requestQueue.markRequestHandled(request)`++Marks a request that was previously returned by the [`RequestQueue.fetchNextRequest()`](../api/request-queue#fetchnextrequest) function as handled+after successful processing. Handled requests will never again be returned by the `fetchNextRequest` function.++**Parameters**:++- **`request`**: [`Request`](../api/request)++**Returns**:++[`Promise<QueueOperationInfo>`](../typedefs/queue-operation-info)++---++<a name="reclaimrequest"></a>++## `requestQueue.reclaimRequest(request, [options])`++Reclaims a failed request back to the queue, so that it can be returned for processed later again by another call to+[`RequestQueue.fetchNextRequest()`](../api/request-queue#fetchnextrequest). The request record in the queue is updated using the provided `request`+parameter. For example, this lets you store the number of retries or error messages for the request.++**Parameters**:++- **`request`**: [`Request`](../api/request)+- **`[options]`**: `object` - **`[forefront]`**: `boolean` <code> = false</code> - If `true` then the request it placed to the beginning of the+ queue, so that it's returned in the next call to [`RequestQueue.fetchNextRequest()`](../api/request-queue#fetchnextrequest). By default, it's put+ to the end of the queue.++**Returns**:++[`Promise<QueueOperationInfo>`](../typedefs/queue-operation-info)++---++<a name="isempty"></a>++## `requestQueue.isEmpty()`++Resolves to `true` if the next call to [`RequestQueue.fetchNextRequest()`](../api/request-queue#fetchnextrequest) would return `null`, otherwise it+resolves to `false`. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that+there is no activity in the queue, use [`RequestQueue.isFinished()`](../api/request-queue#isfinished).++**Returns**:++`Promise<boolean>`++---++<a name="isfinished"></a>++## `requestQueue.isFinished()`++Resolves to `true` if all requests were already handled and there are no more left. Due to the nature of distributed storage used by the queue, the+function might occasionally return a false negative, but it will never return a false positive.++**Returns**:++`Promise<boolean>`++---++<a name="drop"></a>++## `requestQueue.drop()`++Removes the queue either from the Apify Cloud storage or from the local database, depending on the mode of operation.++**Returns**:++`Promise<void>`++---++<a name="handledcount"></a>++## `requestQueue.handledCount()`++Returns the number of handled requests.++This function is just a convenient shortcut for:++```javascript+const { handledRequestCount } = await queue.getInfo();+```++**Returns**:++`Promise<number>`++---++<a name="getinfo"></a>++## `requestQueue.getInfo()`++Returns an object containing general information about the request queue.++The function returns the same object as the Apify API Client's [getQueue](https://docs.apify.com/api/apify-client-js/latest#ApifyClient-requestQueues)+function, which in turn calls the [Get request queue](https://apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue) API endpoint.++**Example:**++```+{+ id: "WkzbQMuFYuamGv3YF",+ name: "my-queue",+ userId: "wRsJZtadYvn4mBZmm",+ createdAt: new Date("2015-12-12T07:34:14.202Z"),+ modifiedAt: new Date("2015-12-13T08:36:13.202Z"),+ accessedAt: new Date("2015-12-14T08:36:13.202Z"),+ totalRequestCount: 25,+ handledRequestCount: 5,+ pendingRequestCount: 20,+}+```++**Returns**:++[`Promise<RequestQueueInfo>`](../typedefs/request-queue-info)++---diff --git a/website/versioned_docs/version-1.0.2/api/Session.md b/website/versioned_docs/version-1.0.2/api/Session.mdnew file mode 100644index 000000000..4a2c7fa6b--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/Session.md@@ -0,0 +1,207 @@+---+id: version-1.0.2-session+title: Session+original_id: session+---++<a name="session"></a>++Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions. You can imagine each session as+a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint. Session internal state can be enriched with custom+user data for example some authorization tokens and specific headers in general.++---++<a name="exports.session"></a>++## `new Session(options)`++Session configuration.++**Parameters**:++- **`options`**: [`SessionOptions`](../typedefs/session-options)++---++<a name="userdata"></a>++## `session.userData`++---++<a name="isblocked"></a>++## `session.isBlocked()`++indicates whether the session is blocked. Session is blocked once it reaches the `maxErrorScore`.++**Returns**:++`boolean`++---++<a name="isexpired"></a>++## `session.isExpired()`++Indicates whether the session is expired. Session expiration is determined by the `maxAgeSecs`. Once the session is older than+`createdAt + maxAgeSecs` the session is considered expired.++**Returns**:++`boolean`++---++<a name="ismaxusagecountreached"></a>++## `session.isMaxUsageCountReached()`++Indicates whether the session is used maximum number of times. Session maximum usage count can be changed by `maxUsageCount` parameter.++**Returns**:++`boolean`++---++<a name="isusable"></a>++## `session.isUsable()`++Indicates whether the session can be used for next requests. Session is usable when it is not expired, not blocked and the maximum usage count has not+be reached.++**Returns**:++`boolean`++---++<a name="markgood"></a>++## `session.markGood()`++This method should be called after a successful session usage. It increases `usageCount` and potentially lowers the `errorScore` by the+`errorScoreDecrement`.++---++<a name="getstate"></a>++## `session.getState()`++Gets session state for persistence in KeyValueStore.++**Returns**:++[`SessionState`](../typedefs/session-state) - represents session internal state.++---++<a name="retire"></a>++## `session.retire()`++Marks session as blocked and emits event on the `SessionPool` This method should be used if the session usage was unsuccessful and you are sure that+it is because of the session configuration and not any external matters. For example when server returns 403 status code. If the session does not work+due to some external factors as server error such as 5XX you probably want to use `markBad` method.++---++<a name="markbad"></a>++## `session.markBad()`++Increases usage and error count. Should be used when the session has been used unsuccessfully. For example because of timeouts.++---++<a name="retireonblockedstatuscodes"></a>++## `session.retireOnBlockedStatusCodes(statusCode, [blockedStatusCodes])`++With certain status codes: `401`, `403` or `429` we can be certain that the target website is blocking us. This function helps to do this conveniently+by retiring the session when such code is received. Optionally the default status codes can be extended in the second parameter.++**Parameters**:++- **`statusCode`**: `number` - HTTP status code+- **`[blockedStatusCodes]`**: `Array<number>` - Custom HTTP status codes that means blocking on particular website.++**Returns**:++`boolean` - whether the session was retired.++---++<a name="setcookiesfromresponse"></a>++## `session.setCookiesFromResponse(response)`++Saves cookies from an HTTP response to be used with the session. It expects an object with a `headers` property that's either an `Object` (typical+Node.js responses) or a `Function` (Puppeteer Response).++It then parses and saves the cookies from the `set-cookie` header, if available.++**Parameters**:++- **`response`**: `PuppeteerResponse` | `IncomingMessage`++---++<a name="setpuppeteercookies"></a>++## `session.setPuppeteerCookies(cookies, url)`++Saves an array with cookie objects to be used with the session. The objects should be in the format that+[Puppeteer uses](https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-pagecookiesurls), but you can also use this function to set cookies+manually:++```+[+ { name: 'cookie1', value: 'my-cookie' },+ { name: 'cookie2', value: 'your-cookie' }+]+```++**Parameters**:++- **`cookies`**: `Array<PuppeteerCookie>`+- **`url`**: `string`++---++<a name="getpuppeteercookies"></a>++## `session.getPuppeteerCookies(url)`++Returns cookies in a format compatible with puppeteer and ready to be used with `page.setCookie`.++**Parameters**:++- **`url`**: `string` - website url. Only cookies stored for this url will be returned++**Returns**:++`Array<PuppeteerCookie>`++---++<a name="getcookiestring"></a>++## `session.getCookieString(url)`++Returns cookies saved with the session in the typical key1=value1; key2=value2 format, ready to be used in a cookie header or elsewhere.++**Parameters**:++- **`url`**: `string`++**Returns**:++`string` - - represents `Cookie` header.++---diff --git a/website/versioned_docs/version-1.0.2/api/SessionPool.md b/website/versioned_docs/version-1.0.2/api/SessionPool.mdnew file mode 100644index 000000000..f26b486f0--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/SessionPool.md@@ -0,0 +1,146 @@+---+id: version-1.0.2-session-pool+title: SessionPool+original_id: session-pool+---++<a name="sessionpool"></a>++Handles the rotation, creation and persistence of user-like sessions. Creates a pool of [`Session`](../api/session) instances, that are randomly+rotated. When some session is marked as blocked. It is removed and new one is created instead. Learn more in the+[`Session management guide`](../guides/session-management).++You can create one by calling the [`Apify.openSessionPool`](../api/apify#opensessionpool) function.++Session pool is already integrated into crawlers, and it can significantly improve your scraper performance with just 2 lines of code.++**Example usage:**++```javascript+const crawler = new Apify.CheerioCrawler({+ useSessionPool: true,+ persistCookiesPerSession: true,+ // ...+});+```++You can configure the pool with many options. See the [`SessionPoolOptions`](../typedefs/session-pool-options). Session pool is by default persisted+in default [`KeyValueStore`](../api/key-value-store). If you want to have one pool for all runs you have to specify+[`SessionPoolOptions.persistStateKeyValueStoreId`](../typedefs/session-pool-options#persiststatekeyvaluestoreid).++**Advanced usage:**++```javascript+const sessionPool = await Apify.openSessionPool({+ maxPoolSize: 25,+ sessionOptions: {+ maxAgeSecs: 10,+ maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.+ },+ persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',+ persistStateKey: 'my-session-pool',+});++// Get random session from the pool+const session1 = await sessionPool.getSession();+const session2 = await sessionPool.getSession();+const session3 = await sessionPool.getSession();++// Now you can mark the session either failed or successful++// Marks session as bad after unsuccessful usage -> it increases error count (soft retire)+session1.markBad();++// Marks as successful.+session2.markGood();++// Retires session -> session is removed from the pool+session3.retire();+```++---++<a name="sessions"></a>++## `sessionPool.sessions`++---++<a name="usablesessionscount"></a>++## `sessionPool.usableSessionsCount`++Gets count of usable sessions in the pool.++**Returns**:++`number`++---++<a name="retiredsessionscount"></a>++## `sessionPool.retiredSessionsCount`++Gets count of retired sessions in the pool.++**Returns**:++`number`++---++<a name="initialize"></a>++## `sessionPool.initialize()`++Starts periodic state persistence and potentially loads SessionPool state from [`KeyValueStore`](../api/key-value-store). It is called automatically+by the [`Apify.openSessionPool`](../api/apify#opensessionpool) function.++**Returns**:++`Promise<void>`++---++<a name="getsession"></a>++## `sessionPool.getSession()`++Gets session. If there is space for new session, it creates and return new session. If the session pool is full, it picks a session from the pool, If+the picked session is usable it is returned, otherwise it creates and returns a new one.++**Returns**:++[`Promise<Session>`](../api/session)++---++<a name="getstate"></a>++## `sessionPool.getState()`++Returns an object representing the internal state of the `SessionPool` instance. Note that the object's fields can change in future releases.++---++<a name="persiststate"></a>++## `sessionPool.persistState()`++Persists the current state of the `SessionPool` into the default [`KeyValueStore`](../api/key-value-store). The state is persisted automatically in+regular intervals.++**Returns**:++`Promise<void>`++---++<a name="teardown"></a>++## `sessionPool.teardown()`++Removes listener from `persistState` event. This function should be called after you are done with using the `SessionPool` instance.++---diff --git a/website/versioned_docs/version-1.0.2/api/Snapshotter.md b/website/versioned_docs/version-1.0.2/api/Snapshotter.mdnew file mode 100644index 000000000..10cb9c5e3--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/Snapshotter.md@@ -0,0 +1,135 @@+---+id: version-1.0.2-snapshotter+title: Snapshotter+original_id: snapshotter+---++<a name="snapshotter"></a>++Creates snapshots of system resources at given intervals and marks the resource as either overloaded or not during the last interval. Keeps a history+of the snapshots. It tracks the following resources: Memory, EventLoop, API and CPU. The class is used by the+[`AutoscaledPool`](../api/autoscaled-pool) class.++When running on the Apify platform, the CPU and memory statistics are provided by the platform, as collected from the running Docker container. When+running locally, `Snapshotter` makes its own statistics by querying the OS.++CPU becomes overloaded locally when its current use exceeds the `maxUsedCpuRatio` option or when Apify platform marks it as overloaded.++Memory becomes overloaded if its current use exceeds the `maxUsedMemoryRatio` option. It's computed using the total memory available to the container+when running on the Apify platform and a quarter of total system memory when running locally. Max total memory when running locally may be overridden+by using the `APIFY_MEMORY_MBYTES` environment variable.++Event loop becomes overloaded if it slows down by more than the `maxBlockedMillis` option.++Client becomes overloaded when rate limit errors (429 - Too Many Requests), typically received from the request queue, exceed the set limit within the+set interval.++---++<a name="snapshotter"></a>++## `new Snapshotter([options])`++**Parameters**:++- **`[options]`**: [`SnapshotterOptions`](../typedefs/snapshotter-options) - All `Snapshotter` configuration options.++---++<a name="log"></a>++## `snapshotter.log`++---++<a name="start"></a>++## `snapshotter.start()`++Starts capturing snapshots at configured intervals.++**Returns**:++`Promise<void>`++---++<a name="stop"></a>++## `snapshotter.stop()`++Stops all resource capturing.++**Returns**:++`Promise<void>`++---++<a name="getmemorysample"></a>++## `snapshotter.getMemorySample([sampleDurationMillis])`++Returns a sample of latest memory snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full+snapshot history.++**Parameters**:++- **`[sampleDurationMillis]`**: `number`++**Returns**:++`Array<*>`++---++<a name="geteventloopsample"></a>++## `snapshotter.getEventLoopSample([sampleDurationMillis])`++Returns a sample of latest event loop snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a+full snapshot history.++**Parameters**:++- **`[sampleDurationMillis]`**: `number`++**Returns**:++`Array<*>`++---++<a name="getcpusample"></a>++## `snapshotter.getCpuSample([sampleDurationMillis])`++Returns a sample of latest CPU snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full+snapshot history.++**Parameters**:++- **`[sampleDurationMillis]`**: `number`++**Returns**:++`Array<*>`++---++<a name="getclientsample"></a>++## `snapshotter.getClientSample(sampleDurationMillis)`++Returns a sample of latest Client snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full+snapshot history.++**Parameters**:++- **`sampleDurationMillis`**: `number`++**Returns**:++`Array<*>`++---diff --git a/website/versioned_docs/version-1.0.2/api/log.md b/website/versioned_docs/version-1.0.2/api/log.mdnew file mode 100644index 000000000..dcd0f4de3--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/log.md@@ -0,0 +1,211 @@+---+id: version-1.0.2-log+title: utils.log+original_id: log+---++<a name="log"></a>++The log instance enables level aware logging of messages and we advise to use it instead of `console.log()` and its aliases in most development+scenarios.++A very useful use case for `log` is using `log.debug` liberally throughout the codebase to get useful logging messages only when appropriate log level+is set and keeping the console tidy in production environments.++The available logging levels are, in this order: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `OFF` and can be referenced from the `log.LEVELS` constant, such+as `log.LEVELS.ERROR`.++To log messages to the system console, use the `log.level(message)` invocation, such as `log.debug('this is a debug message')`.++To prevent writing of messages above a certain log level to the console, simply set the appropriate level. The default log level is `INFO`, which+means that `DEBUG` messages will not be printed, unless enabled.++**Example:**++```js+const Apify = require('apify');+const { log } = Apify.utils;++log.info('Information message', { someData: 123 }); // prints message+log.debug('Debug message', { debugData: 'hello' }); // doesn't print anything++log.setLevel(log.LEVELS.DEBUG);+log.debug('Debug message'); // prints message++log.setLevel(log.LEVELS.ERROR);+log.debug('Debug message'); // doesn't print anything+log.info('Info message'); // doesn't print anything++log.error('Error message', { errorDetails: 'This is bad!' }); // prints message+try {+ throw new Error('Not good!');+} catch (e) {+ log.exception(e, 'Exception occurred', { errorDetails: 'This is really bad!' }); // prints message+}++log.setOptions({ prefix: 'My actor' });+log.info('I am running!'); // prints "My actor: I am running"++const childLog = log.child({ prefix: 'Crawler' });+log.info('I am crawling!'); // prints "My actor:Crawler: I am crawling"+```++Another very useful way of setting the log level is by setting the `APIFY_LOG_LEVEL` environment variable, such as `APIFY_LOG_LEVEL=DEBUG`. This way,+no code changes are necessary to turn on your debug messages and start debugging right away.++To add timestamps to your logs, you can override the default logger settings:++```js+log.setOptions({+ logger: new log.LoggerText({ skipTime: false }),+});+```++You can customize your logging further by extending or replacing the default logger instances with your own implementations.++---++<a name="levels"></a>++## `log.LEVELS`++Map of available log levels that's useful for easy setting of appropriate log levels. Each log level is represented internally by a number. Eg.+`log.LEVELS.DEBUG === 5`.++---++<a name="setlevel"></a>++## `log.setLevel(level)`++Sets the log level to the given value, preventing messages from less important log levels from being printed to the console. Use in conjunction with+the `log.LEVELS` constants such as++```+log.setLevel(log.LEVELS.DEBUG);+```++Default log level is INFO.++**Parameters**:++- **`level`**: `number`++---++<a name="getlevel"></a>++## `log.getLevel()`++Returns the currently selected logging level. This is useful for checking whether a message will actually be printed to the console before one+actually performs a resource intensive operation to construct the message, such as querying a DB for some metadata that need to be added. If the log+level is not high enough at the moment, it doesn't make sense to execute the query.++---++<a name="setoptions"></a>++## `log.setOptions(options)`++Configures logger.++**Parameters**:++- **`options`**: [`LoggerOptions`](../typedefs/logger-options)++---++<a name="child"></a>++## `log.child([options])`++Creates a new instance of logger that inherits settings from a parent logger.++**Parameters**:++- **`[options]`**: [`LoggerOptions`](../typedefs/logger-options) - Supports the same options as the `setOptions()` method.++---++<a name="getoptions"></a>++## `log.getOptions()`++Returns the logger configuration.++**Returns**:++[`LoggerOptions`](../typedefs/logger-options)++---++<a name="debug"></a>++## `log.debug(message, [data])`++Logs a `DEBUG` message. By default, it will not be written to the console. To see `DEBUG` messages in the console, set the log level to `DEBUG` either+using the `log.setLevel(log.LEVELS.DEBUG)` method or using the environment variable `APIFY_LOG_LEVEL=DEBUG`. Data are stringified and appended to the+message.++**Parameters**:++- **`message`**: `string`+- **`[data]`**: `object`++---++<a name="info"></a>++## `log.info(message, [data])`++Logs an `INFO` message. `INFO` is the default log level so info messages will be always logged, unless the log level is changed. Data are stringified+and appended to the message.++**Parameters**:++- **`message`**: `string`+- **`[data]`**: `object`++---++<a name="warning"></a>++## `log.warning(message, [data])`++Logs a `WARNING` level message. Data are stringified and appended to the message.++**Parameters**:++- **`message`**: `string`+- **`[data]`**: `object`++---++<a name="error"></a>++## `log.error(message, [data])`++Logs an `ERROR` message. Use this method to log error messages that are not directly connected to an exception. For logging exceptions, use the+`log.exception` method.++**Parameters**:++- **`message`**: `string`+- **`[data]`**: `object`++---++<a name="exception"></a>++## `log.exception(exception, [message], [data])`++Logs an `ERROR` level message with a nicely formatted exception. Note that the exception is the first parameter here and an additional message is only+optional.++**Parameters**:++- **`exception`**: `Error`+- **`[message]`**: `string`+- **`[data]`**: `object`++---diff --git a/website/versioned_docs/version-1.0.2/api/playwright.md b/website/versioned_docs/version-1.0.2/api/playwright.mdnew file mode 100644index 000000000..ddd630b2f--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/playwright.md@@ -0,0 +1,48 @@+---+id: version-1.0.2-playwright+title: utils.playwright+original_id: playwright+---++<a name="playwright"></a>++A namespace that contains various utilities for [Playwright](https://github.com/microsoft/playwright) - the headless Chrome Node API.++**Example usage:**++```javascript+const Apify = require('apify');+const { playwright } = Apify.utils;++// Navigate to https://www.example.com in Playwright with a POST request+const browser = await Apify.launchPlaywright();+const page = await browser.newPage();+await playwright.gotoExtended(page, {+ url: 'https://example.com,+ method: 'POST',+});+```++---++<a name="gotoextended"></a>++## `playwright.gotoExtended`++Extended version of Playwright's `page.goto()` allowing to perform requests with HTTP method other than GET, with custom headers and POST payload.+URL, method, headers and payload are taken from request parameter that must be an instance of Apify.Request class.++_NOTE:_ In recent versions of Playwright using requests other than GET, overriding headers and adding payloads disables browser cache which degrades+performance.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://playwright.dev/docs/api/class-page) object.+- **`request`**: [`Request`](../api/request)+- **`[gotoOptions]`**: [`DirectNavigationOptions`](../typedefs/direct-navigation-options) - Custom options for `page.goto()`.++**Returns**:++`Promise<(Response|null)>`++---diff --git a/website/versioned_docs/version-1.0.2/api/puppeteer.md b/website/versioned_docs/version-1.0.2/api/puppeteer.mdnew file mode 100644index 000000000..38592f9b9--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/puppeteer.md@@ -0,0 +1,471 @@+---+id: version-1.0.2-puppeteer+title: utils.puppeteer+original_id: puppeteer+---++<a name="puppeteer"></a>++A namespace that contains various utilities for [Puppeteer](https://github.com/puppeteer/puppeteer) - the headless Chrome Node API.++**Example usage:**++```javascript+const Apify = require('apify');+const { puppeteer } = Apify.utils;++// Open https://www.example.com in Puppeteer+const browser = await Apify.launchPuppeteer();+const page = await browser.newPage();+await page.goto('https://www.example.com');++// Inject jQuery into a page+await puppeteer.injectJQuery(page);+```++---++<a name="addinterceptrequesthandler"></a>++## `puppeteer.addInterceptRequestHandler`++Adds request interception handler in similar to `page.on('request', handler);` but in addition to that supports multiple parallel handlers.++All the handlers are executed sequentially in the order as they were added. Each of the handlers must call one of `request.continue()`,+`request.abort()` and `request.respond()`. In addition to that any of the handlers may modify the request object (method, postData, headers) by+passing its overrides to `request.continue()`. If multiple handlers modify same property then the last one wins. Headers are merged separately so you+can override only a value of specific header.++If one the handlers calls `request.abort()` or `request.respond()` then request is not propagated further to any of the remaining handlers.++**Example usage:**++```javascript+// Replace images with placeholder.+await addInterceptRequestHandler(page, request => {+ if (request.resourceType() === 'image') {+ return request.respond({+ statusCode: 200,+ contentType: 'image/jpeg',+ body: placeholderImageBuffer,+ });+ }+ return request.continue();+});++// Abort all the scripts.+await addInterceptRequestHandler(page, request => {+ if (request.resourceType() === 'script') return request.abort();+ return request.continue();+});++// Change requests to post.+await addInterceptRequestHandler(page, request => {+ return request.continue({+ method: 'POST',+ });+});++await page.goto('http://example.com');+```++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`handler`**: [`InterceptHandler`](../typedefs/intercept-handler) - Request interception handler.++**Returns**:++`Promise<void>`++---++<a name="removeinterceptrequesthandler"></a>++## `puppeteer.removeInterceptRequestHandler`++Removes request interception handler for given page.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`handler`**: [`InterceptHandler`](../typedefs/intercept-handler) - Request interception handler.++**Returns**:++`Promise<void>`++---++<a name="gotoextended"></a>++## `puppeteer.gotoExtended`++Extended version of Puppeteer's `page.goto()` allowing to perform requests with HTTP method other than GET, with custom headers and POST payload. URL,+method, headers and payload are taken from request parameter that must be an instance of Apify.Request class.++_NOTE:_ In recent versions of Puppeteer using requests other than GET, overriding headers and adding payloads disables browser cache which degrades+performance.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`request`**: [`Request`](../api/request)+- **`[gotoOptions]`**: [`DirectNavigationOptions`](../typedefs/direct-navigation-options) - Custom options for `page.goto()`.++**Returns**:++`Promise<(Response|null)>`++---++<a name="infinitescroll"></a>++## `puppeteer.infiniteScroll`++Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`[options]`**: `object`+ - **`[timeoutSecs]`**: `number` <code> = 0</code> - How many seconds to scroll for. If 0, will scroll until bottom of page.+ - **`[waitForSecs]`**: `number` <code> = 4</code> - How many seconds to wait for no new content to load before exit.+ - **`[scrollDownAndUp]`**: `boolean` <code> = false</code> - If true, it will scroll up a bit after each scroll down. This is required on some+ websites for the scroll to work.+ - **`[buttonSelector]`**: `string` - Optionally checks and clicks a button if it appears while scrolling. This is required on some websites for+ the scroll to work.++**Returns**:++`Promise<void>`++---++<a name="savesnapshot"></a>++## `puppeteer.saveSnapshot`++Saves a full screenshot and HTML of the current page into a Key-Value store.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`[options]`**: `object`+ - **`[key]`**: `string` <code> = "SNAPSHOT"</code> - Key under which the screenshot and HTML will be saved. `.jpg` will be appended+ for screenshot and `.html` for HTML.+ - **`[screenshotQuality]`**: `number` <code> = 50</code> - The quality of the image, between 0-100. Higher quality images have bigger size and+ require more storage.+ - **`[saveScreenshot]`**: `boolean` <code> = true</code> - If true, it will save a full screenshot of the current page as a record with `key`+ appended by `.jpg`.+ - **`[saveHtml]`**: `boolean` <code> = true</code> - If true, it will save a full HTML of the current page as a record with `key` appended by+ `.html`.+ - **`[keyValueStoreName]`**: `string` | `null` <code> = null</code> - Name or id of the Key-Value store where snapshot is saved. By default it+ is saved to default Key-Value store.++**Returns**:++`Promise<void>`++---++<a name="enqueuelinksbyclickingelements"></a>++## `puppeteer.enqueueLinksByClickingElements(options)`++The function finds elements matching a specific CSS selector in a Puppeteer page, clicks all those elements using a mouse move and a left mouse button+click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods,+headers and payloads are then enqueued to a provided [`RequestQueue`](../api/request-queue). This is useful to crawl JavaScript heavy pages where+links are not available in `href` elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in `href`+attributes of the page, see [`utils.enqueueLinks()`](../api/utils#enqueuelinks).++Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](../api/pseudo-url) objects and override settings+of the enqueued [`Request`](../api/request) objects.++**IMPORTANT**: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and+their visibility. Therefore, it is recommended to only use this function as the last operation in the page.++**USING HEADFUL BROWSER**: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting+concurrency to 1. In headless mode, full concurrency can be achieved.++**PERFORMANCE**: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU+intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets+only the elements that you assume or know will produce a navigation. You can certainly click everything by using the `*` selector, but be prepared to+wait minutes to get results on a large and complex page.++**Example usage**++```javascript+await Apify.utils.puppeteer.enqueueLinksByClickingElements({+ page,+ requestQueue,+ selector: 'a.product-detail',+ pseudoUrls: [+ 'https://www.example.com/handbags/[.*]'+ 'https://www.example.com/purses/[.*]'+ ],+});+```++**Parameters**:++- **`options`**: `object` - All `enqueueLinksByClickingElements()` parameters are passed via an options object with the following keys:++ - **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ - **`requestQueue`**: [`RequestQueue`](../api/request-queue) - A request queue to which the URLs will be enqueued.+ - **`selector`**: `string` - A CSS selector matching elements to be clicked on. Unlike in [`utils.enqueueLinks()`](../api/utils#enqueuelinks),+ there is no default value. This is to prevent suboptimal use of this function by using it too broadly.+ - **`[pseudoUrls]`**: `Array<(string|RegExp|Object<string, *>)>` - An array of [`PseudoUrl`](../api/pseudo-url)s matching the URLs to be+ enqueued, or an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](../api/pseudo-url)s can be constructed.++ The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the+ `requestTemplate` argument of the [`PseudoUrl`](../api/pseudo-url) constructor, which lets you specify special properties for the enqueued+ [`Request`](../api/request) objects.++ If `pseudoUrls` is an empty array, `null` or `undefined`, then the function enqueues all links found on the page.++ - **`[transformRequestFunction]`**: [`RequestTransform`](../typedefs/request-transform) - Just before a new [`Request`](../api/request) is+ constructed and enqueued to the [`RequestQueue`](../api/request-queue), this function can be used to remove it or modify its contents such as+ `userData`, `payload` or, most importantly `uniqueKey`. This is useful when you need to enqueue multiple `Requests` to the queue that share+ the same URL, but differ in methods or payloads, or to dynamically update or create `userData`.++ For example: by adding `useExtendedUniqueKey: true` to the `request` object, `uniqueKey` will be computed from a combination of `url`, `method`+ and `payload` which enables crawling of websites that navigate using form submits (POST requests).++ **Example:**++ ```javascript+ {+ transformRequestFunction: request => {+ request.userData.foo = 'bar';+ request.useExtendedUniqueKey = true;+ return request;+ };+ }+ ```++ - **`[waitForPageIdleSecs]`**: `number` <code> = 1</code> - Clicking in the page triggers various asynchronous operations that lead to new URLs+ being shown by the browser. It could be a simple JavaScript redirect or opening of a new tab in the browser. These events often happen only+ some time after the actual click. Requests typically take milliseconds while new tabs open in hundreds of milliseconds.++ To be able to capture all those events, the `enqueueLinksByClickingElements()` function repeatedly waits for the `waitForPageIdleSecs`. By+ repeatedly we mean that whenever a relevant event is triggered, the timer is restarted. As long as new events keep coming, the function will not+ return, unless the below `maxWaitForPageIdleSecs` timeout is reached.++ You may want to reduce this for example when you're sure that your clicks do not open new tabs, or increase when you're not getting all the+ expected URLs.++ - **`[maxWaitForPageIdleSecs]`**: `number` <code> = 5</code> - This is the maximum period for which the function will keep tracking events, even+ if more events keep coming. Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself. See+ `waitForPageIdleSecs` above for an explanation.++**Returns**:++[`Promise<Array<QueueOperationInfo>>`](../typedefs/queue-operation-info) - Promise that resolves to an array of+[`QueueOperationInfo`](../typedefs/queue-operation-info) objects.++---++<a name="injectfile"></a>++## `puppeteer.injectFile(page, filePath, [options])`++Injects a JavaScript file into a Puppeteer page. Unlike Puppeteer's `addScriptTag` function, this function works on pages with arbitrary Cross-Origin+Resource Sharing (CORS) policies.++File contents are cached for up to 10 files to limit file system access.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`filePath`**: `string` - File path+- **`[options]`**: `object`+ - **`[surviveNavigations]`**: `boolean` - Enables the injected script to survive page navigations and reloads without need to be re-injected+ manually. This does not mean, however, that internal state will be preserved. Just that it will be automatically re-injected on each+ navigation before any other scripts get the chance to execute.++**Returns**:++`Promise<*>`++---++<a name="injectjquery"></a>++## `puppeteer.injectJQuery(page)`++Injects the [jQuery](https://jquery.com/) library into a Puppeteer page. jQuery is often useful for various web scraping and crawling tasks. For+example, it can help extract text from HTML elements using CSS selectors.++Beware that the injected jQuery object will be set to the `window.$` variable and thus it might cause conflicts with other libraries included by the+page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.++The injected jQuery will survive page navigations and reloads.++**Example usage:**++```javascript+await Apify.utils.puppeteer.injectJQuery(page);+const title = await page.evaluate(() => {+ return $('head title').text();+});+```++Note that `injectJQuery()` does not affect the Puppeteer's [`page.$()`](https://pptr.dev/#?product=Puppeteer&show=api-pageselector) function in any+way.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.++**Returns**:++`Promise<*>`++---++<a name="injectunderscore"></a>++## `puppeteer.injectUnderscore(page)`++Injects the [Underscore](https://underscorejs.org/) library into a Puppeteer page.++Beware that the injected Underscore object will be set to the `window._` variable and thus it might cause conflicts with libraries included by the+page that use the same variable name. This can affect functionality of page's scripts.++The injected Underscore will survive page navigations and reloads.++**Example usage:**++```javascript+await Apify.utils.puppeteer.injectUnderscore(page);+const escapedHtml = await page.evaluate(() => {+ return _.escape('<h1>Hello</h1>');+});+```++**Parameters**:++- **`page`**: `Page` - Puppeteer [Page](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#class-page) object.++**Returns**:++`Promise<*>`++---++<a name="blockrequests"></a>++## `puppeteer.blockRequests(page, [options])`++Forces the Puppeteer browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it+reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.++By default, the function will block all URLs including the following patterns:++```json+[".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]+```++If you want to extend this list further, use the `extraUrlPatterns` option, which will keep blocking the default patterns, as well as add your custom+ones. If you would like to block only specific patterns, use the `urlPatterns` option, which will override the defaults and block only URLs with your+custom patterns.++This function does not use Puppeteer's request interception and therefore does not interfere with browser cache. It's also faster than blocking+requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra+benefits of request interception.++The function will never block main document loads and their respective redirects.++**Example usage**++```javascript+const Apify = require('apify');++const browser = await Apify.launchPuppeteer();+const page = await browser.newPage();++// Block all requests to URLs that include `adsbygoogle.js` and also all defaults.+await Apify.utils.puppeteer.blockRequests(page, {+ extraUrlPatterns: ['adsbygoogle.js'],+});++await page.goto('https://cnn.com');+```++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`[options]`**: `object`+ - **`[urlPatterns]`**: `Array<string>` - The patterns of URLs to block from being loaded by the browser. Only `*` can be used as a wildcard. It+ is also automatically added to the beginning and end of the pattern. This limitation is enforced by the DevTools protocol. `.png` is the same+ as `*.png*`.+ - **`[extraUrlPatterns]`**: `Array<string>` - If you just want to append to the default blocked patterns, use this property.++**Returns**:++`Promise<void>`++---++<a name="cacheresponses"></a>++## ~~`puppeteer.cacheResponses(page, cache, responseUrlRules)`~~++**_Deprecated_**++_NOTE:_ In recent versions of Puppeteer using this function entirely disables browser cache which resolves in sub-optimal performance. Until this+resolves, we suggest just relying on the in-browser cache unless absolutely necessary.++Enables caching of intercepted responses into a provided object. Automatically enables request interception in Puppeteer. _IMPORTANT_: Caching+responses stores them to memory, so too loose rules could cause memory leaks for longer running crawlers. This issue should be resolved or atleast+mitigated in future iterations of this feature.++**Parameters**:++- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`cache`**: `Object<string, *>` - Object in which responses are stored+- **`responseUrlRules`**: `Array<(string|RegExp)>` - List of rules that are used to check if the response should be cached. String rules are+ compared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).++**Returns**:++`Promise<void>`++---++<a name="compilescript"></a>++## `puppeteer.compileScript(scriptString, context)`++Compiles a Puppeteer script into an async function that may be executed at any time by providing it with the following object:++```+{+ page: Page,+ request: Request,+}+```++Where `page` is a Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) and `request` is a [`Request`](../api/request).++The function is compiled by using the `scriptString` parameter as the function's body, so any limitations to function bodies apply. Return value of+the compiled function is the return value of the function body = the `scriptString` parameter.++As a security measure, no globals such as `process` or `require` are accessible from within the function body. Note that the function does not provide+a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation.+Therefore you should only use this function to execute sanitized or safe code.++Custom context may also be provided using the `context` parameter. To improve security, make sure to only pass the really necessary objects to the+context. Preferably making secured copies beforehand.++**Parameters**:++- **`scriptString`**: `string`+- **`context`**: `Object<string, *>`++**Returns**:++[`CompiledScriptFunction`](../typedefs/compiled-script-function)++---diff --git a/website/versioned_docs/version-1.0.2/api/social.md b/website/versioned_docs/version-1.0.2/api/social.mdnew file mode 100644index 000000000..3515c5156--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/social.md@@ -0,0 +1,428 @@+---+id: version-1.0.2-social+title: utils.social+original_id: social+---++<a name="social"></a>++A namespace that contains various utilities to help you extract social handles from text, URLs and and HTML documents.++**Example usage:**++```javascript+const Apify = require('apify');++const emails = Apify.utils.social.emailsFromText('[email protected] [email protected]');+```++---++<a name="linkedin_regex"></a>++## `social.LINKEDIN_REGEX`++Regular expression to exactly match a single LinkedIn profile URL. It has the following form: `/^...$/i` and matches URLs such as:++```+https://www.linkedin.com/in/alan-turing+en.linkedin.com/in/alan-turing+linkedin.com/in/alan-turing+```++The regular expression does NOT match URLs with additional subdirectories or query parameters, such as:++```+https://www.linkedin.com/in/linus-torvalds/latest-activity+```++Example usage:++```+if (Apify.utils.social.LINKEDIN_REGEX.test('https://www.linkedin.com/in/alan-turing')) {+ console.log('Match!');+}+```++---++<a name="linkedin_regex_global"></a>++## `social.LINKEDIN_REGEX_GLOBAL`++Regular expression to find multiple LinkedIn profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:++```+https://www.linkedin.com/in/alan-turing+en.linkedin.com/in/alan-turing+linkedin.com/in/alan-turing+```++If the profile URL contains subdirectories or query parameters, the regular expression extracts just the base part of the profile URL. For example,+from text such as:++```+https://www.linkedin.com/in/linus-torvalds/latest-activity+```++the expression extracts just the following base URL:++```+https://www.linkedin.com/in/linus-torvalds+```++Example usage:++```+const matches = text.match(Apify.utils.social.LINKEDIN_REGEX_GLOBAL);+if (matches) console.log(`${matches.length} LinkedIn profiles found!`);+```++---++<a name="instagram_regex"></a>++## `social.INSTAGRAM_REGEX`++Regular expression to exactly match a single Instagram profile URL. It has the following form: `/^...$/i` and matches URLs such as:++```+https://www.instagram.com/old_prague+www.instagram.com/old_prague/+instagr.am/old_prague+```++The regular expression does NOT match URLs with additional subdirectories or query parameters, such as:++```+https://www.instagram.com/cristiano/followers+```++Example usage:++```+if (Apify.utils.social.INSTAGRAM_REGEX.test('https://www.instagram.com/old_prague')) {+ console.log('Match!');+}+```++---++<a name="instagram_regex_global"></a>++## `social.INSTAGRAM_REGEX_GLOBAL`++Regular expression to find multiple Instagram profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:++```+https://www.instagram.com/old_prague+www.instagram.com/old_prague/+instagr.am/old_prague+```++If the profile URL contains subdirectories or query parameters, the regular expression extracts just the base part of the profile URL. For example,+from text such as:++```+https://www.instagram.com/cristiano/followers+```++the expression extracts just the following base URL:++```+https://www.instagram.com/cristiano+```++Example usage:++```+const matches = text.match(Apify.utils.social.INSTAGRAM_REGEX_GLOBAL);+if (matches) console.log(`${matches.length} Instagram profiles found!`);+```++---++<a name="twitter_regex"></a>++## `social.TWITTER_REGEX`++Regular expression to exactly match a single Twitter profile URL. It has the following form: `/^...$/i` and matches URLs such as:++```+https://www.twitter.com/apify+twitter.com/apify+```++The regular expression does NOT match URLs with additional subdirectories or query parameters, such as:++```+https://www.twitter.com/realdonaldtrump/following+```++Example usage:++```+if (Apify.utils.social.TWITTER_REGEX.test('https://www.twitter.com/apify')) {+ console.log('Match!');+}+```++---++<a name="twitter_regex_global"></a>++## `social.TWITTER_REGEX_GLOBAL`++Regular expression to find multiple Twitter profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:++```+https://www.twitter.com/apify+twitter.com/apify+```++If the profile URL contains subdirectories or query parameters, the regular expression extracts just the base part of the profile URL. For example,+from text such as:++```+https://www.twitter.com/realdonaldtrump/following+```++the expression extracts only the following base URL:++```+https://www.twitter.com/realdonaldtrump+```++Example usage:++```+const matches = text.match(Apify.utils.social.TWITTER_REGEX_STRING);+if (matches) console.log(`${matches.length} Twitter profiles found!`);+```++---++<a name="facebook_regex"></a>++## `social.FACEBOOK_REGEX`++Regular expression to exactly match a single Facebook profile URL. It has the following form: `/^...$/i` and matches URLs such as:++```+https://www.facebook.com/apifytech+facebook.com/apifytech+fb.com/apifytech+https://www.facebook.com/profile.php?id=123456789+```++The regular expression does NOT match URLs with additional subdirectories or query parameters, such as:++```+https://www.facebook.com/apifytech/photos+```++Example usage:++```+if (Apify.utils.social.FACEBOOK_REGEX.test('https://www.facebook.com/apifytech')) {+ console.log('Match!');+}+```++---++<a name="facebook_regex_global"></a>++## `social.FACEBOOK_REGEX_GLOBAL`++Regular expression to find multiple Facebook profile URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:++```+https://www.facebook.com/apifytech+facebook.com/apifytech+fb.com/apifytech+```++If the profile URL contains subdirectories or query parameters, the regular expression extracts just the base part of the profile URL. For example,+from text such as:++```+https://www.facebook.com/apifytech/photos+```++the expression extracts only the following base URL:++```+https://www.facebook.com/apifytech+```++Example usage:++```+const matches = text.match(Apify.utils.social.FACEBOOK_REGEX_GLOBAL);+if (matches) console.log(`${matches.length} Facebook profiles found!`);+```++---++<a name="youtube_regex"></a>++## `social.YOUTUBE_REGEX`++Regular expression to exactly match a single Youtube video URL. It has the following form: `/^...$/i` and matches URLs such as:++```+https://www.youtube.com/watch?v=kM7YfhfkiEE+https://youtu.be/kM7YfhfkiEE+```++Example usage:++```+if (Apify.utils.social.YOUTUBE_REGEX.test('https://www.youtube.com/watch?v=kM7YfhfkiEE')) {+ console.log('Match!');+}+```++---++<a name="youtube_regex_global"></a>++## `social.YOUTUBE_REGEX_GLOBAL`++Regular expression to find multiple Youtube video URLs in a text or HTML. It has the following form: `/.../ig` and matches URLs such as:++```+https://www.youtube.com/watch?v=kM7YfhfkiEE+https://youtu.be/kM7YfhfkiEE+```++Example usage:++```+const matches = text.match(Apify.utils.social.YOUTUBE_REGEX_GLOBAL);+if (matches) console.log(`${matches.length} Youtube videos found!`);+```++---++<a name="email_regex"></a>++## `social.EMAIL_REGEX`++Regular expression to exactly match a single email address. It has the following form: `/^...$/i`.++---++<a name="email_regex_global"></a>++## `social.EMAIL_REGEX_GLOBAL`++Regular expression to find multiple email addresses in a text. It has the following form: `/.../ig`.++---++<a name="emailsfromtext"></a>++## `social.emailsFromText(text)`++The function extracts email addresses from a plain text. Note that the function preserves the order of emails and keep duplicates.++**Parameters**:++- **`text`**: `string` - Text to search in.++**Returns**:++`Array<string>` - Array of emails addresses found. If no emails are found, the function returns an empty array.++---++<a name="emailsfromurls"></a>++## `social.emailsFromUrls(urls)`++The function extracts email addresses from a list of URLs. Basically it looks for all `mailto:` URLs and returns valid email addresses from them. Note+that the function preserves the order of emails and keep duplicates.++**Parameters**:++- **`urls`**: `Array<string>` - Array of URLs.++**Returns**:++`Array<string>` - Array of emails addresses found. If no emails are found, the function returns an empty array.++---++<a name="phonesfromtext"></a>++## `social.phonesFromText(text)`++The function attempts to extract phone numbers from a text. Please note that the results might not be accurate, since phone numbers appear in a large+variety of formats and conventions. If you encounter some problems, please [file an issue](https://github.com/apify/apify-js/issues).++**Parameters**:++- **`text`**: `string` - Text to search the phone numbers in.++**Returns**:++`Array<string>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.++---++<a name="phonesfromurls"></a>++## `social.phonesFromUrls(urls)`++Finds phone number links in an array of URLs and extracts the phone numbers from them. Note that the phone number links look like `tel://123456789`,+`tel:/123456789` or `tel:123456789`.++**Parameters**:++- **`urls`**: `Array<string>` - Array of URLs.++**Returns**:++`Array<string>` - Array of phone numbers found. If no phone numbers are found, the function returns an empty array.++---++<a name="parsehandlesfromhtml"></a>++## `social.parseHandlesFromHtml(html, [data])`++The function attempts to extract emails, phone numbers and social profile URLs from a HTML document, specifically LinkedIn, Twitter, Instagram and+Facebook profile URLs. The function removes duplicates from the resulting arrays and sorts the items alphabetically.++Note that the `phones` field contains phone numbers extracted from the special phone links such as `[call us](tel:+1234556789)` (see+[`social.phonesFromUrls()`](../api/social#phonesfromurls)) and potentially other sources with high certainty, while `phonesUncertain` contains phone+numbers extracted from the plain text, which might be very inaccurate.++**Example usage:**++```javascript+const Apify = require('apify');++const browser = await Apify.launchPuppeteer();+const page = await browser.newPage();+await page.goto('http://www.example.com');+const html = await page.content();++const result = Apify.utils.social.parseHandlesFromHtml(html);+console.log('Social handles:');+console.dir(result);+```++**Parameters**:++- **`html`**: `string` - HTML text+- **`[data]`**: `*` | `null` <code> = </code> - Optional object which will receive the `text` and `$` properties that contain text content of the+ HTML and `cheerio` object, respectively. This is an optimization so that the caller doesn't need to parse the HTML document again, if needed.++**Returns**:++[`SocialHandles`](../typedefs/social-handles) - An object with the social handles.++---diff --git a/website/versioned_docs/version-1.0.2/api/utils.md b/website/versioned_docs/version-1.0.2/api/utils.mdnew file mode 100644index 000000000..4970360cb--- /dev/null+++ b/website/versioned_docs/version-1.0.2/api/utils.md@@ -0,0 +1,307 @@+---+id: version-1.0.2-utils+title: utils+original_id: utils+---++<a name="utils"></a>++A namespace that contains various utilities.++**Example usage:**++```javascript+const Apify = require('apify');++...++// Sleep 1.5 seconds+await Apify.utils.sleep(1500);+```++---++<a name="url_no_commas_regex"></a>++## `utils.URL_NO_COMMAS_REGEX`++Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters and does not+support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).++---++<a name="url_with_commas_regex"></a>++## `utils.URL_WITH_COMMAS_REGEX`++Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query. Note,+however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.++---++<a name="enqueuelinks"></a>++## `utils.enqueueLinks(options)`++The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default) either in a Puppeteer page, or in a Cheerio object+(parsed HTML), and enqueues the URLs in their `href` attributes to the provided [`RequestQueue`](../api/request-queue). If you're looking to find URLs+in JavaScript heavy pages where links are not available in `href` elements, but rather navigations are triggered in click handlers see+[`puppeteer.enqueueLinksByClickingElements()`](../api/puppeteer#enqueuelinksbyclickingelements).++Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](../api/pseudo-url) objects and override settings+of the enqueued [`Request`](../api/request) objects.++**Example usage**++```javascript+await Apify.utils.enqueueLinks({+ page,+ requestQueue,+ selector: 'a.product-detail',+ pseudoUrls: ['https://www.example.com/handbags/[.*]', 'https://www.example.com/purses/[.*]'],+});+```++**Parameters**:++- **`options`**: `object` - All `enqueueLinks()` parameters are passed via an options object with the following keys:++ - **`[page]`**: `PuppeteerPage` | `PlaywrightPage` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object. Either+ `page` or `$` option must be provided.+ - **`[limit]`**: `Number` - Limit the count of actually enqueued URLs to this number. Useful for testing across the entire crawling scope.+ - **`[$]`**: `cheerio.Root` | `cheerio.Selector` - [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML. Either `page` or+ `$` option must be provided.+ - **`requestQueue`**: [`RequestQueue`](../api/request-queue) - A request queue to which the URLs will be enqueued.+ - **`[selector]`**: `string` <code> = "'a'"</code> - A CSS selector matching links to be enqueued.+ - **`[baseUrl]`**: `string` - A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer, since the+ relative URL resolution is done inside the browser automatically.+ - **`[pseudoUrls]`**: `Array<Object<string, *>>` | `Array.<string>` - An array of [`PseudoUrl`](../api/pseudo-url)s matching the URLs to+ be enqueued, or an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](../api/pseudo-url)s can be constructed.++ The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the+ `requestTemplate` argument of the [`PseudoUrl`](../api/pseudo-url) constructor, which lets you specify special properties for the enqueued+ [`Request`](../api/request) objects.++ If `pseudoUrls` is an empty array, `null` or `undefined`, then the function enqueues all links found on the page.++ - **`[transformRequestFunction]`**: [`RequestTransform`](../typedefs/request-transform) - Just before a new [`Request`](../api/request) is+ constructed and enqueued to the [`RequestQueue`](../api/request-queue), this function can be used to remove it or modify its contents such as+ `userData`, `payload` or, most importantly `uniqueKey`. This is useful when you need to enqueue multiple `Requests` to the queue that share+ the same URL, but differ in methods or payloads, or to dynamically update or create `userData`.++ For example: by adding `keepUrlFragment: true` to the `request` object, URL fragments will not be removed when `uniqueKey` is computed.++ **Example:**++ ```javascript+ {+ transformRequestFunction: request => {+ request.userData.foo = 'bar';+ request.keepUrlFragment = true;+ return request;+ };+ }+ ```++**Returns**:++[`Promise<Array<QueueOperationInfo>>`](../typedefs/queue-operation-info) - Promise that resolves to an array of+[`QueueOperationInfo`](../typedefs/queue-operation-info) objects.++---++<a name="requestasbrowser"></a>++## `utils.requestAsBrowser(options)`++**IMPORTANT:** This function uses an insecure version of HTTP parser by default and also ignores SSL/TLS errors. This is very useful in scraping,+because it allows bypassing certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping scenarios, please set+`useInsecureHttpParser: false` and `ignoreSslErrors: false`.++Sends a HTTP request that looks like a request sent by a web browser, fully emulating browser's HTTP headers.++This function is useful for web scraping of websites that send the full HTML in the first response. Thanks to this function, the target web server has+no simple way to find out the request hasn't been sent by a full web browser. Using a headless browser for such requests is an order of magnitude more+resource-intensive than this function. By default it aborts all requests that returns 406 status codes or non-HTML content-types. You can override+this behavior by passing custom `abortFunction`.++Currently, the function sends requests the same way as Firefox web browser does. In the future, it might add support for other browsers too.++Internally, the function uses `httpRequest` function from the [@apify/http-request](https://github.com/apify/http-request) NPM package to perform the+request. All `options` not recognized by this function are passed to it, so see it for more details.++**Example usage:**++```js+const Apify = require('apify');++const { utils: { requestAsBrowser } } = Apify;++...++const response = await requestAsBrowser({ url: 'https://www.example.com/' });++const html = response.body;+const status = response.statusCode;+const contentType = response.headers['content-type'];+```++**Parameters**:++- **`options`**: [`RequestAsBrowserOptions`](../typedefs/request-as-browser-options) - All `requestAsBrowser` configuration options.++**Returns**:++[`Promise<RequestAsBrowserResult>`](../typedefs/request-as-browser-result) - This will typically be a+[Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage), however, if returned from the cache it will be a+[response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.++---++<a name="isdocker"></a>++## `utils.isDocker(forceReset)`++Returns a `Promise` that resolves to true if the code is running in a Docker container.++**Parameters**:++- **`forceReset`**: `boolean`++**Returns**:++`Promise<boolean>`++---++<a name="sleep"></a>++## `utils.sleep(millis)`++Returns a `Promise` that resolves after a specific period of time. This is useful to implement waiting in your code, e.g. to prevent overloading of+target website or to avoid bot detection.++**Example usage:**++```+const Apify = require('apify');++...++// Sleep 1.5 seconds+await Apify.utils.sleep(1500);+```++**Parameters**:++- **`millis`**: `number` - Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.++**Returns**:++`Promise<void>`++---++<a name="downloadlistofurls"></a>++## `utils.downloadListOfUrls(options)`++Returns a promise that resolves to an array of urls parsed from the resource available at the provided url. Optionally, custom regular expression and+encoding may be provided.++**Parameters**:++- **`options`**: `object`+ - **`url`**: `string` - URL to the file+ - **`[encoding]`**: `string` <code> = "'utf8'"</code> - The encoding of the file.+ - **`[urlRegExp]`**: `RegExp` <code> = URL_NO_COMMAS_REGEX</code> - Custom regular expression to identify the URLs in the file to extract. The+ regular expression should be case-insensitive and have global flag set (i.e. `/something/gi`).++**Returns**:++`Promise<Array<string>>`++---++<a name="extracturls"></a>++## `utils.extractUrls(options)`++Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.++**Parameters**:++- **`options`**: `object`+ - **`string`**: `string`+ - **`[urlRegExp]`**: `RegExp` <code> = Apify.utils.URL_NO_COMMAS_REGEX</code>++**Returns**:++`Array<string>`++---++<a name="htmltotext"></a>++## `utils.htmlToText(html)`++The function converts a HTML document to a plain text.++The plain text generated by the function is similar to a text captured by pressing Ctrl+A and Ctrl+C on a page when loaded in a web browser. The+function doesn't aspire to preserve the formatting or to be perfectly correct with respect to HTML specifications. However, it attempts to generate+newlines and whitespaces in and around HTML elements to avoid merging distinct parts of text and thus enable extraction of data from the text (e.g.+phone numbers).++**Example usage**++```javascript+const text = htmlToText('<html><body>Some text</body></html>');+console.log(text);+```++Note that the function uses [cheerio](https://www.npmjs.com/package/cheerio) to parse the HTML. Optionally, to avoid duplicate parsing of HTML and+thus improve performance, you can pass an existing Cheerio object to the function instead of the HTML text. The HTML should be parsed with the+`decodeEntities` option set to `true`. For example:++```javascript+const cheerio = require('cheerio');+const html = '<html><body>Some text</body></html>';+const text = htmlToText(cheerio.load(html, { decodeEntities: true }));+```++**Parameters**:++- **`html`**: `string` | `cheerio.Root` - HTML text or parsed HTML represented using a [cheerio](https://www.npmjs.com/package/cheerio) function.++**Returns**:++`string` - Plain text++---++<a name="waitforruntofinish"></a>++## `utils.waitForRunToFinish(options)`++Returns a promise that resolves with the finished Run object when the provided actor run finishes or with the unfinished Run object when the+`waitSecs` timeout lapses. The promise is NOT rejected based on run status. You can inspect the `status` property of the Run object to find out its+status.++This is useful when you need to chain actor executions. Similar effect can be achieved by using webhooks, so be sure to review which technique fits+your use-case better.++**Parameters**:++- **`options`**: `object`+ - **`actorId`**: `string` - ID of the actor that started the run.+ - **`runId`**: `string` - ID of the run itself.+ - **`[waitSecs]`**: `string` - Maximum time to wait for the run to finish, in seconds. If the limit is reached, the returned promise is resolved+ to a run object that will have status `READY` or `RUNNING`. If `waitSecs` omitted, the function waits indefinitely.+ - **`[token]`**: `string` - You can supply an Apify token to override the default one that's used by the default ApifyClient instance. E.g. you+ can track other users' runs.++**Returns**:++[`Promise<ActorRun>`](../typedefs/actor-run)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/ActorRun.md b/website/versioned_docs/version-1.0.2/typedefs/ActorRun.mdnew file mode 100644index 000000000..024699d57--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/ActorRun.md@@ -0,0 +1,182 @@+---+id: version-1.0.2-actor-run+title: ActorRun+original_id: actor-run+---++<a name="actorrun"></a>++Represents information about an actor run, as returned by the [`Apify.call()`](../api/apify#call) or [`Apify.callTask()`](../api/apify#calltask)+function. The object is almost equivalent to the JSON response of the+[Actor run](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor) Apify API endpoint and extended with certain fields. For more+details, see [Runs.](https://docs.apify.com/actor/run)++## Properties++### `id`++**Type**: `string`++Actor run ID++---++### `actId`++**Type**: `string`++Actor ID++---++### `startedAt`++**Type**: `Date`++Time when the actor run started++---++### `finishedAt`++**Type**: `Date`++Time when the actor run finished. Contains `null` for running actors.++---++### `status`++**Type**: `string`++Status of the run. For possible values, see [Run lifecycle](https://docs.apify.com/actor/run#lifecycle) in Apify actor documentation.++---++### `meta`++**Type**: `Object<string, string>`++Actor run meta-data. For example:++```+{+ "origin": "API",+ "clientIp": "1.2.3.4",+ "userAgent": "ApifyClient/0.2.13 (Linux; Node/v8.11.3)"+}+```++---++### `stats`++**Type**: `Object<string, number>`++An object containing various actor run statistics. For example:++```+{+ "inputBodyLen": 22,+ "restartCount": 0,+ "workersUsed": 1,+}+```++Beware that object fields might change in future releases.++---++### `options`++**Type**: `Object<string, *>`++Actor run options. For example:++```+{+ "build": "latest",+ "waitSecs": 0,+ "memoryMbytes": 256,+ "diskMbytes": 512+}+```++---++### `buildId`++**Type**: `string`++ID of the actor build used for the run. For details, see [Builds](https://docs.apify.com/actor/build) in Apify actor documentation.++---++### `buildNumber`++**Type**: `string`++Number of the actor build used for the run. For example, `0.0.10`.++---++### `exitCode`++**Type**: `number`++Exit code of the actor run process. It's `null` if actor is still running.++---++### `defaultKeyValueStoreId`++**Type**: `string`++ID of the default key-value store associated with the actor run. See [`KeyValueStore`](../api/key-value-store) for details.++---++### `defaultDatasetId`++**Type**: `string`++ID of the default dataset associated with the actor run. See [`Dataset`](../api/dataset) for details.++---++### `defaultRequestQueueId`++**Type**: `string`++ID of the default request queue associated with the actor run. See [`RequestQueue`](../api/request-queue) for details.++---++### `containerUrl`++**Type**: `string`++URL on which the web server running inside actor run's Docker container can be accessed. For more details, see+[Container web server](https://docs.apify.com/actor/run#container-web-server) in Apify actor documentation.++---++### `output`++**Type**: `Object<string, *>` | `null` | `undefined`++Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running, or if you pass `false` to the `fetchOutput`+option of [`Apify.call()`](../api/apify#call).++For example:++```+{+ "contentType": "application/json; charset=utf-8",+ "body": {+ "message": "Hello world!"+ }+}+```++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/AdhocWebhook.md b/website/versioned_docs/version-1.0.2/typedefs/AdhocWebhook.mdnew file mode 100644index 000000000..ce50a7e3d--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/AdhocWebhook.md@@ -0,0 +1,33 @@+---+id: version-1.0.2-adhoc-webhook+title: AdhocWebhook+original_id: adhoc-webhook+---++<a name="adhocwebhook"></a>++## Properties++### `eventTypes`++**Type**: [`EventTypes`](../typedefs/event-types)++---++### `requestUrl`++**Type**: `string`++---++### `idempotencyKey`++**Type**: `string`++---++### `payloadTemplate`++**Type**: `string`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/BrowserCrawlingContext.md b/website/versioned_docs/version-1.0.2/typedefs/BrowserCrawlingContext.mdnew file mode 100644index 000000000..32d7d4a41--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/BrowserCrawlingContext.md@@ -0,0 +1,15 @@+---+id: version-1.0.2-browser-crawling-context+title: BrowserCrawlingContext+original_id: browser-crawling-context+---++<a name="browsercrawlingcontext"></a>++## Properties++### `browserController`++**Type**: `BrowserController`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/BrowserHandlePageFunction.md b/website/versioned_docs/version-1.0.2/typedefs/BrowserHandlePageFunction.mdnew file mode 100644index 000000000..9b41352c5--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/BrowserHandlePageFunction.md@@ -0,0 +1,17 @@+---+id: version-1.0.2-browser-handle-page-function+title: BrowserHandlePageFunction+original_id: browser-handle-page-function+---++<a name="browserhandlepagefunction"></a>++**Parameters**:++- **`context`**: [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) | [`CrawlingContext`](../typedefs/crawling-context)++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/BrowserLaunchContext.md b/website/versioned_docs/version-1.0.2/typedefs/BrowserLaunchContext.mdnew file mode 100644index 000000000..93f3d2583--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/BrowserLaunchContext.md@@ -0,0 +1,46 @@+---+id: version-1.0.2-browser-launch-context+title: BrowserLaunchContext+original_id: browser-launch-context+---++<a name="browserlaunchcontext"></a>++## Properties++### `launchOptions`++**Type**: `Object<string, *>`++`Options passed to the browser launcher function. Options are based on underlying library.++---++### `proxyUrl`++**Type**: `string`++URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password.++Example: `http://bob:[email protected]:1234`.++---++### `useChrome`++**Type**: `boolean` <code> = false</code>++If `true` and `executablePath` is not set, Playwright will launch full Google Chrome browser available on the machine rather than the bundled+Chromium. The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical+Google Chrome executable location specific for the operating system. By default, this option is `false`.++---++### `launcher`++**Type**: `*`++By default this function uses+require("playwright").chromium`. If you want to use a different browser you can pass it by this property as`require("playwright").firefox++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/CheerioCrawlerOptions.md b/website/versioned_docs/version-1.0.2/typedefs/CheerioCrawlerOptions.mdnew file mode 100644index 000000000..bf745a491--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/CheerioCrawlerOptions.md@@ -0,0 +1,328 @@+---+id: version-1.0.2-cheerio-crawler-options+title: CheerioCrawlerOptions+original_id: cheerio-crawler-options+---++<a name="cheeriocrawleroptions"></a>++## Properties++### `handlePageFunction`++**Type**: [`CheerioHandlePage`](../typedefs/cheerio-handle-page)++User-provided function that performs the logic of the crawler. It is called for each page loaded and parsed by the crawler.++The function receives the following object as an argument:++```+{+ // The Cheerio object's function with the parsed HTML.+ $: Cheerio,++ // The request body of the web page, whose type depends on the content type.+ body: String|Buffer,++ // The parsed object from JSON for responses with the "application/json" content types.+ // For other content types it's null.+ json: Object,++ // Apify.Request object with details of the requested web page+ request: Request,++ // Parsed Content-Type HTTP header: { type, encoding }+ contentType: Object,++ // An instance of Node's http.IncomingMessage object,+ response: Object,++ // Session object, useful to work around anti-scraping protections+ session: Session++ // ProxyInfo object with information about currently used proxy+ proxyInfo: ProxyInfo++ // The running cheerio crawler instance.+ crawler: CheerioCrawler+}+```++Type of `body` depends on the `Content-Type` header of the web page:++- String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types+- Buffer for others MIME content types++Parsed `Content-Type` header using [content-type package](https://www.npmjs.com/package/content-type) is stored in `contentType`.++Cheerio is available only for HTML and XML content types.++With the [`Request`](../api/request) object representing the URL to crawl.++If the function returns, the returned promise is awaited by the crawler.++If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your+function throw exceptions rather than catch them. The exceptions are logged to the request using the+[`Request.pushErrorMessage()`](../api/request#pusherrormessage) function.++---++### `requestList`++**Type**: [`RequestList`](../api/request-list)++Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).++---++### `requestQueue`++**Type**: [`RequestQueue`](../api/request-queue)++Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be+provided (or both).++---++### `prepareRequestFunction`++**Type**: [`PrepareRequest`](../typedefs/prepare-request)++A function that executes before the HTTP request is made to the target resource. This function is suitable for setting dynamic properties such as+cookies to the [`Request`](../api/request).++The function receives the following object as an argument:++```+{+ request: Request,+ session: Session,+ proxyInfo: ProxyInfo,+ crawler: CheerioCrawler,+}+```++where the [`Request`](../api/request) instance corresponds to the initialized request and the [`Session`](../api/session) instance corresponds to used+session.++The function should modify the properties of the passed [`Request`](../api/request) instance in place because there are already earlier references to+it. Making a copy and returning it from this function is therefore not supported, because it would create inconsistencies where different parts of SDK+would have access to a different [`Request`](../api/request) instance.++---++### `postResponseFunction`++**Type**: [`PostResponse`](../typedefs/post-response)++A function that executes right after the HTTP request is made to the target resource and response is returned. This function is suitable for+overriding custom properties of response e.g. setting headers because of response parsing.++**Example usage:**++```javascript+const cheerioCrawlerOptions = {+ // ...+ postResponseFunction: ({ request, response }) => {+ if (request.userData.parseAsJSON) {+ response.headers['content-type'] = 'application/json; charset=utf-8';+ }+ },+};+```++The function receives the following object as an argument:++```+{+ response: Object,+ request: Request,+ session: Session,+ proxyInfo: ProxyInfo,+ crawler: CheerioCrawler,+}+```++The response is an instance of Node's http.IncomingMessage object.++---++### `handlePageTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.++---++### `requestTimeoutSecs`++**Type**: `number` <code> = 30</code>++Timeout in which the HTTP request to the resource needs to finish, given in seconds.++---++### `ignoreSslErrors`++**Type**: `boolean` <code> = true</code>++If set to true, SSL certificate errors will be ignored.++---++### `proxyConfiguration`++**Type**: [`ProxyConfiguration`](../api/proxy-configuration)++If set, `CheerioCrawler` will be configured for all connections to use [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and+rotated according to the configuration. For more information, see the [documentation](https://docs.apify.com/proxy).++---++### `handleFailedRequestFunction`++**Type**: [`HandleFailedRequest`](../typedefs/handle-failed-request)++A function to handle requests that failed more than `option.maxRequestRetries` times.++The function receives the following object as an argument:++```+{+ error: Error,+ request: Request,+ session: Session,+ $: Cheerio,+ body: String|Buffer,+ json: Object,+ contentType: Object,+ response: Object,+ proxyInfo: ProxyInfo,+ crawler: CheerioCrawler,+}+```++where the [`Request`](../api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during+processing of the request.++See [source code](https://github.com/apify/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13) for the default implementation of this function.++---++### `additionalMimeTypes`++**Type**: `Array<string>`++An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"+ target="_blank">MIME types</a> you want the crawler to load and process. By default, only `text/html` and `application/xhtml+xml` MIME types are+supported.++---++### `suggestResponseEncoding`++**Type**: `string`++By default `CheerioCrawler` will extract correct encoding from the HTTP response headers. Sadly, there are some websites which use invalid headers.+Those are encoded using the UTF-8 encoding. If those sites actually use a different encoding, the response will be corrupted. You can use+`suggestResponseEncoding` to fall back to a certain encoding, if you know that your target website uses it. To force a certain encoding, disregarding+the response headers, use [`CheerioCrawlerOptions.forceResponseEncoding`](../typedefs/cheerio-crawler-options#forceresponseencoding)++```+// Will fall back to windows-1250 encoding if none found+suggestResponseEncoding: 'windows-1250'+```++---++### `forceResponseEncoding`++**Type**: `string`++By default `CheerioCrawler` will extract correct encoding from the HTTP response headers. Use `forceResponseEncoding` to force a certain encoding,+disregarding the response headers. To only provide a default for missing encodings, use+[`CheerioCrawlerOptions.suggestResponseEncoding`](../typedefs/cheerio-crawler-options#suggestresponseencoding)++```+// Will force windows-1250 encoding even if headers say otherwise+forceResponseEncoding: 'windows-1250'+```++---++### `maxRequestRetries`++**Type**: `number` <code> = 3</code>++Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.++---++### `maxRequestsPerCrawl`++**Type**: `number`++Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.++---++### `autoscaledPoolOptions`++**Type**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options)++Custom options passed to the underlying [`AutoscaledPool`](../api/autoscaled-pool) constructor. Note that the `runTaskFunction`, `isTaskReadyFunction`+and `isFinishedFunction` options are provided by `CheerioCrawler` and cannot be overridden. Reasonable [`Snapshotter`](../api/snapshotter) and+[`SystemStatus`](../api/system-status) defaults are provided to account for the fact that `cheerio` parses HTML synchronously and therefore blocks the+event loop.++---++### `minConcurrency`++**Type**: `number` <code> = 1</code>++Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If+you're not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `number` <code> = 1000</code>++Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.++---++### `useSessionPool`++**Type**: `boolean` <code> = true</code>++If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes. It also marks+Session as bad after a request timeout.++---++### `sessionPoolOptions`++**Type**: [`SessionPoolOptions`](../typedefs/session-pool-options)++Custom options passed to the underlying [`SessionPool`](../api/session-pool) constructor.++---++### `persistCookiesPerSession`++**Type**: `boolean`++Automatically saves cookies to Session. Works only if Session Pool is used.++It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request. It passes the+"Cookie" header to the request with the session cookies.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/CheerioHandlePageInputs.md b/website/versioned_docs/version-1.0.2/typedefs/CheerioHandlePageInputs.mdnew file mode 100644index 000000000..3b02087d8--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/CheerioHandlePageInputs.md@@ -0,0 +1,77 @@+---+id: version-1.0.2-cheerio-handle-page-inputs+title: CheerioHandlePageInputs+original_id: cheerio-handle-page-inputs+---++<a name="cheeriohandlepageinputs"></a>++## Properties++### `$`++**Type**: `cheerio.Root`++The [Cheerio](https://cheerio.js.org/) object with parsed HTML.++---++### `body`++**Type**: `string` | `Buffer`++The request body of the web page.++---++### `json`++**Type**: `*`++The parsed object from JSON string if the response contains the content type application/json.++---++### `request`++**Type**: [`Request`](../api/request)++The original [`Request`](../api/request) object.++---++### `contentType`++**Type**: `Object`++Parsed `Content-Type header: { type, encoding }`.++---++### `response`++**Type**: `IncomingMessage`++An instance of Node's http.IncomingMessage object,++---++### `session`++**Type**: [`Session`](../api/session)++---++### `proxyInfo`++**Type**: [`ProxyInfo`](../typedefs/proxy-info)++An object with information about currently used proxy by the crawler and configured by the [`ProxyConfiguration`](../api/proxy-configuration) class.++---++### `crawler`++**Type**: [`CheerioCrawler`](../api/cheerio-crawler)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/CompiledScriptFunction.md b/website/versioned_docs/version-1.0.2/typedefs/CompiledScriptFunction.mdnew file mode 100644index 000000000..2ca7e9e75--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/CompiledScriptFunction.md@@ -0,0 +1,17 @@+---+id: version-1.0.2-compiled-script-function+title: CompiledScriptFunction+original_id: compiled-script-function+---++<a name="compiledscriptfunction"></a>++**Parameters**:++- **`params`**: [`CompiledScriptParams`](../typedefs/compiled-script-params)++**Returns**:++`Promise<*>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/CompiledScriptParams.md b/website/versioned_docs/version-1.0.2/typedefs/CompiledScriptParams.mdnew file mode 100644index 000000000..4d0834375--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/CompiledScriptParams.md@@ -0,0 +1,21 @@+---+id: version-1.0.2-compiled-script-params+title: CompiledScriptParams+original_id: compiled-script-params+---++<a name="compiledscriptparams"></a>++## Properties++### `params.page`++**Type**: `Page`++---++### `params.request`++**Type**: [`Request`](../api/request)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/CrawlingContext.md b/website/versioned_docs/version-1.0.2/typedefs/CrawlingContext.mdnew file mode 100644index 000000000..9e4b28bea--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/CrawlingContext.md@@ -0,0 +1,39 @@+---+id: version-1.0.2-crawling-context+title: CrawlingContext+original_id: crawling-context+---++<a name="crawlingcontext"></a>++## Properties++### `id`++**Type**: `string`++---++### `request`++**Type**: [`Request`](../api/request)++---++### `session`++**Type**: [`Session`](../api/session)++---++### `proxyInfo`++**Type**: [`ProxyInfo`](../typedefs/proxy-info)++---++### `response`++**Type**: `*`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/CreateSession.md b/website/versioned_docs/version-1.0.2/typedefs/CreateSession.mdnew file mode 100644index 000000000..1fcc44c35--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/CreateSession.md@@ -0,0 +1,19 @@+---+id: version-1.0.2-create-session+title: CreateSession+original_id: create-session+---++<a name="createsession"></a>++Factory user-function which creates customized [`Session`](../api/session) instances.++**Parameters**:++- **`sessionPool`**: [`SessionPool`](../api/session-pool) - Pool requesting the new session.++**Returns**:++[`Promise<Session>`](../api/session)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/DirectNavigationOptions.md b/website/versioned_docs/version-1.0.2/typedefs/DirectNavigationOptions.mdnew file mode 100644index 000000000..94d5959b7--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/DirectNavigationOptions.md@@ -0,0 +1,37 @@+---+id: version-1.0.2-direct-navigation-options+title: DirectNavigationOptions+original_id: direct-navigation-options+---++<a name="directnavigationoptions"></a>++## Properties++### `timeout`++**Type**: `number`++Maximum operation time in milliseconds, defaults to 30 seconds, pass `0` to disable timeout. The default value can be changed by using the+browserContext.setDefaultNavigationTimeout(timeout), browserContext.setDefaultTimeout(timeout), page.setDefaultNavigationTimeout(timeout) or+page.setDefaultTimeout(timeout) methods.++---++### `waitUntil`++**Type**: `"domcontentloaded"` | `"load"` | `"networkidle"`++When to consider operation succeeded, defaults to `load`. Events can be either: - `'domcontentloaded'` - consider operation to be finished when the+`DOMContentLoaded` event is fired. - `'load'` - consider operation to be finished when the `load` event is fired. - `'networkidle'` - consider+operation to be finished when there are no network connections for at least `500` ms.++---++### `referer`++**Type**: `string`++Referer header value. If provided it will take preference over the referer header value set by page.setExtraHTTPHeaders(headers).++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/EventTypes.md b/website/versioned_docs/version-1.0.2/typedefs/EventTypes.mdnew file mode 100644index 000000000..d5efabc51--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/EventTypes.md@@ -0,0 +1,9 @@+---+id: version-1.0.2-event-types+title: EventTypes+original_id: event-types+---++<a name="eventtypes"></a>++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/GotoFunction.md b/website/versioned_docs/version-1.0.2/typedefs/GotoFunction.mdnew file mode 100644index 000000000..dedc9d39d--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/GotoFunction.md@@ -0,0 +1,18 @@+---+id: version-1.0.2-goto-function+title: GotoFunction+original_id: goto-function+---++<a name="gotofunction"></a>++**Parameters**:++- **`context`**: [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) | [`CrawlingContext`](../typedefs/crawling-context)+- **`gotoOptions`**: `Object<string, *>`++**Returns**:++`Promise<*>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/HandleFailedRequest.md b/website/versioned_docs/version-1.0.2/typedefs/HandleFailedRequest.mdnew file mode 100644index 000000000..a3746c873--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/HandleFailedRequest.md@@ -0,0 +1,17 @@+---+id: version-1.0.2-handle-failed-request+title: HandleFailedRequest+original_id: handle-failed-request+---++<a name="handlefailedrequest"></a>++**Parameters**:++- **`inputs`**: [`HandleFailedRequestInput`](../typedefs/handle-failed-request-input) - Arguments passed to this callback.++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/Hook.md b/website/versioned_docs/version-1.0.2/typedefs/Hook.mdnew file mode 100644index 000000000..44c469279--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/Hook.md@@ -0,0 +1,18 @@+---+id: version-1.0.2-hook+title: Hook+original_id: hook+---++<a name="hook"></a>++**Parameters**:++- **`crawlingContext`**: [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) | [`CrawlingContext`](../typedefs/crawling-context)+- **`gotoOptions`**: `Object<string, *>`++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/KeyConsumer.md b/website/versioned_docs/version-1.0.2/typedefs/KeyConsumer.mdnew file mode 100644index 000000000..53a04bcd3--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/KeyConsumer.md@@ -0,0 +1,18 @@+---+id: version-1.0.2-key-consumer+title: KeyConsumer+original_id: key-consumer+---++<a name="keyconsumer"></a>++User-function used in the [`KeyValueStore.forEachKey()`](../api/key-value-store#foreachkey) method.++**Parameters**:++- **`key`**: `string` - Current {KeyValue} key being processed.+- **`index`**: `number` - Position of the current key in [`KeyValueStore`](../api/key-value-store).+- **`info`**: `*` - Information about the current [`KeyValueStore`](../api/key-value-store) entry.+ - **`size`**: `number` - Size of the value associated with the current key in bytes.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/KeyValueStoreValueTypes.md b/website/versioned_docs/version-1.0.2/typedefs/KeyValueStoreValueTypes.mdnew file mode 100644index 000000000..eb97df3be--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/KeyValueStoreValueTypes.md@@ -0,0 +1,9 @@+---+id: version-1.0.2-key-value-store-value-types+title: KeyValueStoreValueTypes+original_id: key-value-store-value-types+---++<a name="keyvaluestorevaluetypes"></a>++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/LoggerOptions.md b/website/versioned_docs/version-1.0.2/typedefs/LoggerOptions.mdnew file mode 100644index 000000000..77683a43b--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/LoggerOptions.md@@ -0,0 +1,67 @@+---+id: version-1.0.2-logger-options+title: LoggerOptions+original_id: logger-options+---++<a name="loggeroptions"></a>++## Properties++### `level`++**Type**: `number` <code> = 4</code>++Sets the log level to the given value, preventing messages from less important log levels from being printed to the console. Use in conjunction with+the `log.LEVELS` constants.++---++### `maxDepth`++**Type**: `number` <code> = 4</code>++Max depth of data object that will be logged. Anything deeper than the limit will be stripped off.++---++### `maxStringLength`++**Type**: `number` <code> = 2000</code>++Max length of the string to be logged. Longer strings will be truncated.++---++### `prefix`++**Type**: `string`++Prefix to be prepended the each logged line.++---++### `suffix`++**Type**: `string`++Suffix that will be appended the each logged line.++---++### `logger`++**Type**: `object`++Logger implementation to be used. Default one is log.LoggerText to log messages as easily readable strings. Optionally you can use `log.LoggerJson`+that formats each log line as a JSON.++---++### `data`++**Type**: `object`++Additional data to be added to each log line.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PlaywrightCrawlerOptions.md b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightCrawlerOptions.mdnew file mode 100644index 000000000..da7f1d972--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightCrawlerOptions.md@@ -0,0 +1,243 @@+---+id: version-1.0.2-playwright-crawler-options+title: PlaywrightCrawlerOptions+original_id: playwright-crawler-options+---++<a name="playwrightcrawleroptions"></a>++## Properties++### `handlePageFunction`++**Type**: [`PlaywrightHandlePageFunction`](../typedefs/playwright-handle-page-function)++Function that is called to process each request. It is passed an object with the following fields:++```+{+ request: Request,+ response: Response,+ page: Page,+ session: Session,+ browserController: BrowserController,+ proxyInfo: ProxyInfo,+ crawler: PlaywrightCrawler,+}+```++`request` is an instance of the [`Request`](../api/request) object with details about the URL to open, HTTP method etc. `page` is an instance of the+`Playwright` [`Page`](https://playwright.dev/docs/api/class-page) `browserController` is an instance of the+[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller), `response` is an instance of the `Playwright`+[`Response`](https://playwright.dev/docs/api/class-response), which is the main resource response as returned by `page.goto(request.url)`. The+function must return a promise, which is then awaited by the crawler.++If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your+function throw exceptions rather than catch them. The exceptions are logged to the request using the+[`Request.pushErrorMessage()`](../api/request#pusherrormessage) function.++---++### `navigationTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default function is overridden, this timeout+will not be used and needs to be configured in the new `gotoFunction()`.++---++### `handleFailedRequestFunction`++**Type**: [`HandleFailedRequest`](../typedefs/handle-failed-request)++A function to handle requests that failed more than `option.maxRequestRetries` times.++The function receives the following object as an argument:++```+{+ request: Request,+ response: Response,+ page: Page,+ session: Session,+ browserController: BrowserController,+ proxyInfo: ProxyInfo,+ crawler: PlaywrightCrawler,+}+```++Where the [`Request`](../api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during+processing of the request.++---++### `preNavigationHooks`++**Type**: [`Array<PlaywrightHook>`](../typedefs/playwright-hook)++Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation.+The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `gotoFunction` the crawler calls to navigate.+Example:++```+preNavigationHooks: [+ async (crawlingContext, gotoOptions) => {+ await page.evaluate((attr) => { window.foo = attr; }, 'bar');+ }+]+```++---++### `postNavigationHooks`++**Type**: [`Array<PlaywrightHook>`](../typedefs/playwright-hook)++Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts+`crawlingContext` as an only parameter. Example:++```+postNavigationHooks: [+ async (crawlingContext) => {+ const { page } = crawlingContext;+ if (hasCaptcha(page)) {+ await solveCaptcha (page);+ }+ };+]+```++---++### `launchContext`++**Type**: [`PlaywrightLaunchContext`](../typedefs/playwright-launch-context)++The same options as used by [`Apify.launchPlaywright()`](../api/apify#launchplaywright).++---++### `handlePageTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.++---++### `browserPoolOptions`++**Type**: `BrowserPoolOptions`++Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor. You can tweak those to+fine-tune browser management.++---++### `persistCookiesPerSession`++**Type**: `boolean` <code> = true</code>++Automatically saves cookies to Session. Works only if Session Pool is used.++---++### `proxyConfiguration`++**Type**: [`ProxyConfiguration`](../api/proxy-configuration)++If set, `PlaywrightCrawler` will be configured for all connections to use [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided+and rotated according to the configuration. For more information, see the [documentation](https://docs.apify.com/proxy).++---++### `requestList`++**Type**: [`RequestList`](../api/request-list)++Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).++---++### `requestQueue`++**Type**: [`RequestQueue`](../api/request-queue)++Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be+provided (or both).++---++### `handleRequestTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.++---++### `maxRequestRetries`++**Type**: `number` <code> = 3</code>++Indicates how many times the request is retried if+[`PlaywrightCrawlerOptions.handlePageFunction`](../typedefs/playwright-crawler-options#handlepagefunction) fails.++---++### `maxRequestsPerCrawl`++**Type**: `number`++Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.++---++### `autoscaledPoolOptions`++**Type**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options)++Custom options passed to the underlying [`AutoscaledPool`](../api/autoscaled-pool) constructor. Note that the `runTaskFunction` and+`isTaskReadyFunction` options are provided by `BasicCrawler` and cannot be overridden. However, you can provide a custom implementation of+`isFinishedFunction`.++---++### `minConcurrency`++**Type**: `number` <code> = 1</code>++Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If+you're not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `number` <code> = 1000</code>++Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.++---++### `useSessionPool`++**Type**: `boolean` <code> = true</code>++Playwright crawler will initialize the [`SessionPool`](../api/session-pool) with the corresponding `sessionPoolOptions`. The session instance will be+than available in the `handleRequestFunction`.++---++### `sessionPoolOptions`++**Type**: [`SessionPoolOptions`](../typedefs/session-pool-options)++The configuration options for [`SessionPool`](../api/session-pool) to use.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PlaywrightGotoOptions.md b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightGotoOptions.mdnew file mode 100644index 000000000..6bc6d3527--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightGotoOptions.md@@ -0,0 +1,37 @@+---+id: version-1.0.2-playwright-goto-options+title: PlaywrightGotoOptions+original_id: playwright-goto-options+---++<a name="playwrightgotooptions"></a>++## Properties++### `timeout`++**Type**: `number`++Maximum operation time in milliseconds, defaults to 30 seconds, pass `0` to disable timeout. The default value can be changed by using the+browserContext.setDefaultNavigationTimeout(timeout), browserContext.setDefaultTimeout(timeout), page.setDefaultNavigationTimeout(timeout) or+page.setDefaultTimeout(timeout) methods.++---++### `waitUntil`++**Type**: `"domcontentloaded"` | `"load"` | `"networkidle"`++When to consider operation succeeded, defaults to `load`. Events can be either: - `'domcontentloaded'` - consider operation to be finished when the+`DOMContentLoaded` event is fired. - `'load'` - consider operation to be finished when the `load` event is fired. - `'networkidle'` - consider+operation to be finished when there are no network connections for at least `500` ms.++---++### `referer`++**Type**: `string`++Referer header value. If provided it will take preference over the referer header value set by page.setExtraHTTPHeaders(headers).++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHandlePageFunction.md b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHandlePageFunction.mdnew file mode 100644index 000000000..7471a63bf--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHandlePageFunction.md@@ -0,0 +1,18 @@+---+id: version-1.0.2-playwright-handle-page-function+title: PlaywrightHandlePageFunction+original_id: playwright-handle-page-function+---++<a name="playwrighthandlepagefunction"></a>++**Parameters**:++- **`context`**: [`PlaywrightHandlePageFunctionParam`](../typedefs/playwright-handle-page-function-param) |+ [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) | [`CrawlingContext`](../typedefs/crawling-context)++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHandlePageFunctionParam.md b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHandlePageFunctionParam.mdnew file mode 100644index 000000000..778f5bf33--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHandlePageFunctionParam.md@@ -0,0 +1,21 @@+---+id: version-1.0.2-playwright-handle-page-function-param+title: PlaywrightHandlePageFunctionParam+original_id: playwright-handle-page-function-param+---++<a name="playwrighthandlepagefunctionparam"></a>++## Properties++### `page`++**Type**: `Page`++---++### `crawler`++**Type**: [`PlaywrightCrawler`](../api/playwright-crawler)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHook.md b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHook.mdnew file mode 100644index 000000000..f1ad12b8e--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightHook.md@@ -0,0 +1,19 @@+---+id: version-1.0.2-playwright-hook+title: PlaywrightHook+original_id: playwright-hook+---++<a name="playwrighthook"></a>++**Parameters**:++- **`crawlingContext`**: `Object` | [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) |+ [`CrawlingContext`](../typedefs/crawling-context)+- **`gotoOptions`**: [`PlaywrightGotoOptions`](../typedefs/playwright-goto-options)++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PlaywrightLaunchContext.md b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightLaunchContext.mdnew file mode 100644index 000000000..fbc1f3db9--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PlaywrightLaunchContext.md@@ -0,0 +1,65 @@+---+id: version-1.0.2-playwright-launch-context+title: PlaywrightLaunchContext+original_id: playwright-launch-context+---++<a name="playwrightlaunchcontext"></a>++Apify extends the launch options of Playwright. You can use any of the Playwright compatible+[`LaunchOptions`](https://playwright.dev/docs/api/class-browsertype#browsertypelaunchoptions) options by providing the `launchOptions` property.++**Example:**++```js+// launch a headless Chrome (not Chromium)+const launchContext = {+ // Apify helpers+ useChrome: true,+ proxyUrl: 'http://user:[email protected]'+ // Native Playwright options+ launchOptions: {+ headless: true,+ args: ['--some-flag'],+ }+}+```++## Properties++### `launchOptions`++**Type**: `Object<string, *>`++`browserType.launch` [options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions)++---++### `proxyUrl`++**Type**: `string`++URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password.++Example: `http://bob:[email protected]:1234`.++---++### `useChrome`++**Type**: `boolean` <code> = false</code>++If `true` and `executablePath` is not set, Playwright will launch full Google Chrome browser available on the machine rather than the bundled+Chromium. The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical+Google Chrome executable location specific for the operating system. By default, this option is `false`.++---++### `launcher`++**Type**: `*`++By default this function uses `require("playwright").chromium`. If you want to use a different browser you can pass it by this property as e.g.+`require("playwright").firefox`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/ProxyConfigurationFunction.md b/website/versioned_docs/version-1.0.2/typedefs/ProxyConfigurationFunction.mdnew file mode 100644index 000000000..1d77fb0b2--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/ProxyConfigurationFunction.md@@ -0,0 +1,17 @@+---+id: version-1.0.2-proxy-configuration-function+title: ProxyConfigurationFunction+original_id: proxy-configuration-function+---++<a name="proxyconfigurationfunction"></a>++**Parameters**:++- **`sessionId`**: `string` | `number`++**Returns**:++`string`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/ProxyConfigurationOptions.md b/website/versioned_docs/version-1.0.2/typedefs/ProxyConfigurationOptions.mdnew file mode 100644index 000000000..b119c3d74--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/ProxyConfigurationOptions.md@@ -0,0 +1,77 @@+---+id: version-1.0.2-proxy-configuration-options+title: ProxyConfigurationOptions+original_id: proxy-configuration-options+---++<a name="proxyconfigurationoptions"></a>++## Properties++### `password`++**Type**: `string`++User's password for the proxy. By default, it is taken from the `APIFY_PROXY_PASSWORD` environment variable, which is automatically set by the system+when running the actors.++---++### `groups`++**Type**: `Array<string>`++An array of proxy groups to be used by the [Apify Proxy](https://docs.apify.com/proxy). If not provided, the proxy will select the groups+automatically.++---++### `countryCode`++**Type**: `string`++If set and relevant proxies are available in your Apify account, all proxied requests will use IP addresses that are geolocated to the specified+country. For example `GB` for IPs from Great Britain. Note that online services often have their own rules for handling geolocation and thus the+country selection is a best attempt at geolocation, rather than a guaranteed hit. This parameter is optional, by default, each proxied request is+assigned an IP address from a random country. The country code needs to be a two letter ISO country code. See the+[full list of available country codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements). This parameter is+optional, by default, the proxy uses all available proxy servers from all countries. on the Apify cloud, or when using the+[Apify CLI](https://github.com/apify/apify-cli).++---++### `apifyProxyGroups`++**Type**: `Array<string>`++Same option as `groups` which can be used to configurate the proxy by UI input schema. You should use the `groups` option in your crawler code.++---++### `apifyProxyCountry`++**Type**: `string`++Same option as `countryCode` which can be used to configurate the proxy by UI input schema. You should use the `countryCode` option in your crawler+code.++---++### `proxyUrls`++**Type**: `Array<string>`++An array of custom proxy URLs to be rotated. Custom proxies are not compatible with Apify Proxy and an attempt to use both configuration options will+cause an error to be thrown on initialize.++---++### `newUrlFunction`++**Type**: [`ProxyConfigurationFunction`](../typedefs/proxy-configuration-function)++Custom function that allows you to generate the new proxy URL dynamically. It gets the `sessionId` as a parameter and should always return stringified+proxy URL. This function is used to generate the URL when [`ProxyConfiguration.newUrl`](../api/proxy-configuration#newurl) or+[`ProxyConfiguration.newProxyInfo`](../api/proxy-configuration#newproxyinfo) is called.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PuppeteerCrawlerOptions.md b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerCrawlerOptions.mdnew file mode 100644index 000000000..83ce72d7b--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerCrawlerOptions.md@@ -0,0 +1,245 @@+---+id: version-1.0.2-puppeteer-crawler-options+title: PuppeteerCrawlerOptions+original_id: puppeteer-crawler-options+---++<a name="puppeteercrawleroptions"></a>++## Properties++### `handlePageFunction`++**Type**: [`PuppeteerHandlePage`](../typedefs/puppeteer-handle-page)++Function that is called to process each request. It is passed an object with the following fields:++```+{+ request: Request,+ response: Response,+ page: Page,+ session: Session,+ browserController: BrowserController,+ proxyInfo: ProxyInfo,+ crawler: PuppeteerCrawler,+}+```++`request` is an instance of the [`Request`](../api/request) object with details about the URL to open, HTTP method etc. `page` is an instance of the+`Puppeteer` [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) `browserPool` is an instance of the+[`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool), `browserController` is an instance of the+[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller), `response` is an instance of the `Puppeteer`+[`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response), which is the main resource response as returned by+`page.goto(request.url)`. The function must return a promise, which is then awaited by the crawler.++If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your+function throw exceptions rather than catch them. The exceptions are logged to the request using the+[`Request.pushErrorMessage()`](../api/request#pusherrormessage) function.++---++### `navigationTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default function is overridden, this timeout+will not be used and needs to be configured in the new `gotoFunction()`.++---++### `handleFailedRequestFunction`++**Type**: [`HandleFailedRequest`](../typedefs/handle-failed-request)++A function to handle requests that failed more than `option.maxRequestRetries` times.++The function receives the following object as an argument:++```+{+ request: Request,+ response: Response,+ page: Page,+ session: Session,+ browserController: BrowserController,+ proxyInfo: ProxyInfo,+ crawler: PuppeteerCrawler,+}+```++Where the [`Request`](../api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during+processing of the request.++---++### `launchContext`++**Type**: [`PuppeteerLaunchContext`](../typedefs/puppeteer-launch-context)++Options used by [`Apify.launchPuppeteer()`](../api/apify#launchpuppeteer) to start new Puppeteer instances.++---++### `handlePageTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.++---++### `browserPoolOptions`++**Type**: `BrowserPoolOptions`++Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor. You can tweak those to+fine-tune browser management.++---++### `persistCookiesPerSession`++**Type**: `boolean` <code> = true</code>++Automatically saves cookies to Session. Works only if Session Pool is used.++---++### `proxyConfiguration`++**Type**: [`ProxyConfiguration`](../api/proxy-configuration)++If set, `PuppeteerCrawler` will be configured for all connections to use [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and+rotated according to the configuration. For more information, see the [documentation](https://docs.apify.com/proxy).++---++### `preNavigationHooks`++**Type**: [`Array<PuppeteerHook>`](../typedefs/puppeteer-hook)++Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation.+The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `gotoFunction` the crawler calls to navigate.+Example:++```+preNavigationHooks: [+ async (crawlingContext, gotoOptions) => {+ const { page } = crawlingContext;+ await page.evaluate((attr) => { window.foo = attr; }, 'bar');+ }+]+```++---++### `postNavigationHooks`++**Type**: [`Array<PuppeteerHook>`](../typedefs/puppeteer-hook)++Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts+`crawlingContext` as an only parameter. Example:++```+postNavigationHooks: [+ async (crawlingContext) => {+ const { page } = crawlingContext;+ if (hasCaptcha(page)) {+ await solveCaptcha (page);+ }+ };+]+```++---++### `requestList`++**Type**: [`RequestList`](../api/request-list)++Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).++---++### `requestQueue`++**Type**: [`RequestQueue`](../api/request-queue)++Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be+provided (or both).++---++### `handleRequestTimeoutSecs`++**Type**: `number` <code> = 60</code>++Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.++---++### `maxRequestRetries`++**Type**: `number` <code> = 3</code>++Indicates how many times the request is retried if+[`PuppeteerCrawlerOptions.handlePageFunction`](../typedefs/puppeteer-crawler-options#handlepagefunction) fails.++---++### `maxRequestsPerCrawl`++**Type**: `number`++Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.++---++### `autoscaledPoolOptions`++**Type**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options)++Custom options passed to the underlying [`AutoscaledPool`](../api/autoscaled-pool) constructor. Note that the `runTaskFunction` and+`isTaskReadyFunction` options are provided by `BasicCrawler` and cannot be overridden. However, you can provide a custom implementation of+`isFinishedFunction`.++---++### `minConcurrency`++**Type**: `number` <code> = 1</code>++Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.++_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If+you're not sure, just keep the default value and the concurrency will scale up automatically.++---++### `maxConcurrency`++**Type**: `number` <code> = 1000</code>++Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.++---++### `useSessionPool`++**Type**: `boolean` <code> = true</code>++Puppeteer crawler will initialize the [`SessionPool`](../api/session-pool) with the corresponding `sessionPoolOptions`. The session instance will be+than available in the `handleRequestFunction`.++---++### `sessionPoolOptions`++**Type**: [`SessionPoolOptions`](../typedefs/session-pool-options)++The configuration options for [`SessionPool`](../api/session-pool) to use.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHandlePage.md b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHandlePage.mdnew file mode 100644index 000000000..9cd446804--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHandlePage.md@@ -0,0 +1,17 @@+---+id: version-1.0.2-puppeteer-handle-page+title: PuppeteerHandlePage+original_id: puppeteer-handle-page+---++<a name="puppeteerhandlepage"></a>++**Parameters**:++- **`context`**: [`CrawlingContext`](../typedefs/crawling-context) | [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) | `Object`++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHandlePageFunctionParam.md b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHandlePageFunctionParam.mdnew file mode 100644index 000000000..afa93a882--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHandlePageFunctionParam.md@@ -0,0 +1,21 @@+---+id: version-1.0.2-puppeteer-handle-page-function-param+title: PuppeteerHandlePageFunctionParam+original_id: puppeteer-handle-page-function-param+---++<a name="puppeteerhandlepagefunctionparam"></a>++## Properties++### `page`++**Type**: `Page`++---++### `crawler`++**Type**: [`PuppeteerCrawler`](../api/puppeteer-crawler)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHook.md b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHook.mdnew file mode 100644index 000000000..674281af2--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerHook.md@@ -0,0 +1,19 @@+---+id: version-1.0.2-puppeteer-hook+title: PuppeteerHook+original_id: puppeteer-hook+---++<a name="puppeteerhook"></a>++**Parameters**:++- **`crawlingContext`**: `Object` | [`BrowserCrawlingContext`](../typedefs/browser-crawling-context) |+ [`CrawlingContext`](../typedefs/crawling-context)+- **`gotoOptions`**: [`DirectNavigationOptions`](../typedefs/direct-navigation-options)++**Returns**:++`Promise<void>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/PuppeteerLaunchContext.md b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerLaunchContext.mdnew file mode 100644index 000000000..58183996a--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/PuppeteerLaunchContext.md@@ -0,0 +1,93 @@+---+id: version-1.0.2-puppeteer-launch-context+title: PuppeteerLaunchContext+original_id: puppeteer-launch-context+---++<a name="puppeteerlaunchcontext"></a>++Apify extends the launch options of Puppeteer. You can use any of the Puppeteer compatible+[`LaunchOptions`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions) options by providing the `launchOptions` property.++**Example:**++```js+// launch a headless Chrome (not Chromium)+const launchContext = {+ // Apify helpers+ useChrome: true,+ proxyUrl: 'http://user:[email protected]'+ // Native Puppeteer options+ launchOptions: {+ headless: true,+ args: ['--some-flag'],+ }+}+```++## Properties++### `launchOptions`++**Type**: `Object<string, *>`++`puppeteer.launch` [options](https://pptr.dev/#?product=Puppeteer&version=v5.5.0&show=api-puppeteerlaunchoptions)++---++### `proxyUrl`++**Type**: `string`++URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password.++Example: `http://bob:[email protected]:1234`.++---++### `userAgent`++**Type**: `string`++The `User-Agent` HTTP header used by the browser. If not provided, the function sets `User-Agent` to a reasonable default to reduce the chance of+detection of the crawler.++---++### `useChrome`++**Type**: `boolean` <code> = false</code>++If `true` and `executablePath` is not set, Puppeteer will launch full Google Chrome browser available on the machine rather than the bundled Chromium.+The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical Google+Chrome executable location specific for the operating system. By default, this option is `false`.++---++### `launcher`++**Type**: `*`++Already required module (`Object`). This enables usage of various Puppeteer wrappers such as `puppeteer-extra`.++Take caution, because it can cause all kinds of unexpected errors and weird behavior. Apify SDK is not tested with any other library besides+`puppeteer` itself.++---++### `stealth`++**Type**: `boolean`++This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable. It is recommended to use it together+with the `useChrome` set to `true`.++---++### `stealthOptions`++**Type**: [`StealthOptions`](../typedefs/stealth-options)++Using this configuration, you can disable some of the hiding tricks. For these settings to take effect `stealth` must be set to true++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/RequestAsBrowserOptions.md b/website/versioned_docs/version-1.0.2/typedefs/RequestAsBrowserOptions.mdnew file mode 100644index 000000000..61900fc9c--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/RequestAsBrowserOptions.md@@ -0,0 +1,103 @@+---+id: version-1.0.2-request-as-browser-options+title: RequestAsBrowserOptions+original_id: request-as-browser-options+---++<a name="requestasbrowseroptions"></a>++## Properties++### `url`++**Type**: `string`++URL of the target endpoint. Supports both HTTP and HTTPS schemes.++---++### `method`++**Type**: `string` <code> = "\"GET\""</code>++HTTP method.++---++### `headers`++**Type**: `Object<string, string>`++Additional HTTP headers to add. It's only recommended to use this option, with headers that are typically added by websites, such as cookies.+Overriding default browser headers will remove the masking this function provides.++---++### `proxyUrl`++**Type**: `string`++An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.++---++### `languageCode`++**Type**: `string` <code> = "en"</code>++Two-letter ISO 639 language code.++---++### `countryCode`++**Type**: `string` <code> = "US"</code>++Two-letter ISO 3166 country code.++---++### `useMobileVersion`++**Type**: `boolean`++If `true`, the function uses User-Agent of a mobile browser.++---++### `ignoreSslErrors`++**Type**: `boolean` <code> = true</code>++If set to true, SSL/TLS certificate errors will be ignored.++---++### `useInsecureHttpParser`++**Type**: `boolean` <code> = true</code>++Node.js' HTTP parser is stricter than parsers used by web browsers, which prevents scraping of websites whose servers do not comply with HTTP specs,+either by accident or due to some anti-scraping protections, causing e.g. the `invalid header value char` error. The `useInsecureHttpParser` option+forces the HTTP parser to ignore certain errors which lets you scrape such websites. However, it will also open your application to some security+vulnerabilities, although the risk should be negligible as these vulnerabilities mainly relate to server applications, not clients. Learn more in this+[blog post](https://snyk.io/blog/node-js-release-fixes-a-critical-http-security-vulnerability/).++---++### `abortFunction`++**Type**: [`AbortFunction`](../typedefs/abort-function)++Function accepts `response` object as a single parameter and should return true or false. If function returns true request gets aborted. This function+is passed to the [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.++---++### `useHttp2`++**Type**: `boolean` <code> = false</code>++If set to true, it will additionally accept HTTP2 requests. It will choose either HTTP/1.1 or HTTP/2 depending on the ALPN protocol.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/RequestAsBrowserResult.md b/website/versioned_docs/version-1.0.2/typedefs/RequestAsBrowserResult.mdnew file mode 100644index 000000000..b3d981a4a--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/RequestAsBrowserResult.md@@ -0,0 +1,9 @@+---+id: version-1.0.2-request-as-browser-result+title: RequestAsBrowserResult+original_id: request-as-browser-result+---++<a name="requestasbrowserresult"></a>++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/RequestListOptions.md b/website/versioned_docs/version-1.0.2/typedefs/RequestListOptions.mdnew file mode 100644index 000000000..0fa74e2dd--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/RequestListOptions.md@@ -0,0 +1,152 @@+---+id: version-1.0.2-request-list-options+title: RequestListOptions+original_id: request-list-options+---++<a name="requestlistoptions"></a>++## Properties++### `sources`++**Type**: [`Array<(RequestOptions|Request|string)>`](../typedefs/request-options)++An array of sources of URLs for the [`RequestList`](../api/request-list). It can be either an array of strings, plain objects that define at least the+`url` property, or an array of [`Request`](../api/request) instances.++**IMPORTANT:** The `sources` array will be consumed (left empty) after `RequestList` initializes. This is a measure to prevent memory leaks in+situations when millions of sources are added.++Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct `RequestList` to download the source URLs from a given+remote location. The URLs will be parsed from the received response.++```+[+ // A single URL+ 'http://example.com/a/b',++ // Modify Request options+ { method: PUT, 'https://example.com/put, payload: { foo: 'bar' }}++ // Batch import of URLs from a file hosted on the web,+ // where the URLs should be requested using the HTTP POST request+ { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },++ // Batch import from remote file, using a specific regular expression to extract the URLs.+ { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },++ // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.+ // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document+ { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }+]+```++---++### `sourcesFunction`++**Type**: [`RequestListSourcesFunction`](../typedefs/request-list-sources-function)++A function that will be called to get the sources for the `RequestList`, but only if `RequestList` was not able to fetch their persisted version (see+[`RequestListOptions.persistRequestsKey`](../typedefs/request-list-options#persistrequestskey)). It must return an `Array` of+[`Request`](../api/request) or [`RequestOptions`](../typedefs/request-options).++This is very useful in a scenario when getting the sources is a resource intensive or time consuming task, such as fetching URLs from multiple+sitemaps or parsing URLs from large datasets. Using the `sourcesFunction` in combination with `persistStateKey` and `persistRequestsKey` will allow+you to fetch and parse those URLs only once, saving valuable time when your actor migrates or restarts.++If both [`RequestListOptions.sources`](../typedefs/request-list-options#sources) and+[`RequestListOptions.sourcesFunction`](../typedefs/request-list-options#sourcesfunction) are provided, the sources returned by the function will be+added after the `sources`.++**Example:**++```javascript+// Let's say we want to scrape URLs extracted from sitemaps.++const sourcesFunction = async () => {+ // With super large sitemaps, this operation could take very long+ // and big websites typically have multiple sitemaps.+ const sitemaps = await downloadHugeSitemaps();+ return parseUrlsFromSitemaps(sitemaps);+};++// Sitemaps can change in real-time, so it's important to persist+// the URLs we collected. Otherwise we might lose our scraping+// state in case of an actor migration / failure / time-out.+const requestList = new RequestList({+ sourcesFunction,+ persistStateKey: 'state-key',+ persistRequestsKey: 'requests-key',+});++// The sourcesFunction is called now and the Requests are persisted.+// If something goes wrong and we need to start again, RequestList+// will load the persisted Requests from storage and will NOT+// call the sourcesFunction again, saving time and resources.+await requestList.initialize();+```++---++### `persistStateKey`++**Type**: `string`++Identifies the key in the default key-value store under which `RequestList` periodically stores its state (i.e. which URLs were crawled and which+not). If the actor is restarted, `RequestList` will read the state and continue where it left off.++If `persistStateKey` is not set, `RequestList` will always start from the beginning, and all the source URLs will be crawled again.++---++### `persistRequestsKey`++**Type**: `string`++Identifies the key in the default key-value store under which the `RequestList` persists its Requests during the+[`RequestList.initialize()`](../api/request-list#initialize) call. This is necessary if `persistStateKey` is set and the source URLs might potentially+change, to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.++If `persistRequestsKey` is not set, [`RequestList.initialize()`](../api/request-list#initialize) will always fetch the sources from their origin,+check that they are consistent with the restored state (if any) and throw an error if they are not.++---++### `state`++**Type**: [`RequestListState`](../typedefs/request-list-state)++The state object that the `RequestList` will be initialized from. It is in the form as returned by `RequestList.getState()`, such as follows:++```+{+ nextIndex: 5,+ nextUniqueKey: 'unique-key-5'+ inProgress: {+ 'unique-key-1': true,+ 'unique-key-4': true,+ },+}+```++Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList` is to use the `stateKeyPrefix` parameter instead.++---++### `keepDuplicateUrls`++**Type**: `boolean` <code> = false</code>++By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based on the `uniqueKey` property of passed source+[`Request`](../api/request) objects.++If the property is not present, it is generated by normalizing the URL. If present, it is kept intact. In any case, only one request per `uniqueKey`+is added to the `RequestList` resulting in removal of duplicate URLs / unique keys.++Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey` of each request that does not already include a+`uniqueKey`. Therefore, duplicate URLs will be kept in the list. It does not protect the user from having duplicates in user set `uniqueKey`s however.+It is the user's responsibility to ensure uniqueness of their unique keys if they wish to keep more than just a single copy in the `RequestList`.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/RequestListSourcesFunction.md b/website/versioned_docs/version-1.0.2/typedefs/RequestListSourcesFunction.mdnew file mode 100644index 000000000..2ac506003--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/RequestListSourcesFunction.md@@ -0,0 +1,13 @@+---+id: version-1.0.2-request-list-sources-function+title: RequestListSourcesFunction+original_id: request-list-sources-function+---++<a name="requestlistsourcesfunction"></a>++**Returns**:++[`Promise<Array<(RequestOptions|Request|string)>>`](../typedefs/request-options)++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/RequestOptions.md b/website/versioned_docs/version-1.0.2/typedefs/RequestOptions.mdnew file mode 100644index 000000000..35cd99b97--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/RequestOptions.md@@ -0,0 +1,95 @@+---+id: version-1.0.2-request-options+title: RequestOptions+original_id: request-options+---++<a name="requestoptions"></a>++Specifies required and optional fields for constructing a [`Request`](../api/request).++## Properties++### `url`++**Type**: `string`++URL of the web page to crawl. It must be a non-empty string.++---++### `uniqueKey`++**Type**: `string`++A unique key identifying the request. Two requests with the same `uniqueKey` are considered as pointing to the same URL.++If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL. For example, the URL of `HTTP://www.EXAMPLE.com/something/`+will produce the `uniqueKey` of `http://www.example.com/something`.++The `keepUrlFragment` option determines whether URL hash fragment is included in the `uniqueKey` or not.++The `useExtendedUniqueKey` options determines whether method and payload are included in the `uniqueKey`, producing a `uniqueKey` in the following+format: `METHOD(payloadHash):normalizedUrl`. This is useful when requests point to the same URL, but with different methods and payloads. For example:+form submits.++Pass an arbitrary non-empty text value to the `uniqueKey` property to override the default behavior and specify which URLs shall be considered equal.++---++### `method`++**Type**: `string` <code> = "'GET'"</code>++---++### `payload`++**Type**: `string` | `Buffer`++HTTP request payload, e.g. for POST requests.++---++### `headers`++**Type**: `Object<string, string>`++HTTP headers in the following format:++```+{+ Accept: 'text/html',+ 'Content-Type': 'application/json'+}+```++---++### `userData`++**Type**: `Object<string, *>`++Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries,+failures etc.++---++### `keepUrlFragment`++**Type**: `boolean` <code> = false</code>++If `false` then the hash part of a URL is removed when computing the `uniqueKey` property. For example, this causes the `http://www.example.com#foo`+and `http://www.example.com#bar` URLs to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal. Note that this+option only has an effect if `uniqueKey` is not set.++---++### `useExtendedUniqueKey`++**Type**: `boolean` <code> = false</code>++If `true` then the `uniqueKey` is computed not only from the URL, but also from the method and payload properties. This is useful when making requests+to the same URL that are differentiated by method or payload, such as form submit navigations in browsers.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/SessionOptions.md b/website/versioned_docs/version-1.0.2/typedefs/SessionOptions.mdnew file mode 100644index 000000000..2914211b9--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/SessionOptions.md@@ -0,0 +1,100 @@+---+id: version-1.0.2-session-options+title: SessionOptions+original_id: session-options+---++<a name="sessionoptions"></a>++## Properties++### `id`++**Type**: `string`++Id of session used for generating fingerprints. It is used as proxy session name.++---++### `maxAgeSecs`++**Type**: `number` <code> = 3000</code>++Number of seconds after which the session is considered as expired.++---++### `userData`++**Type**: `Object<string, *>`++Object where custom user data can be stored. For example custom headers.++---++### `maxErrorScore`++**Type**: `number` <code> = 3</code>++Maximum number of marking session as blocked usage. If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.+It starts at 0. Calling the `markBad` function increases the `errorScore` by 1. Calling the `markGood` will decrease the `errorScore` by+`errorScoreDecrement`++---++### `errorScoreDecrement`++**Type**: `number` <code> = 0.5</code>++It is used for healing the session. For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is+decremented by this number.++---++### `createdAt`++**Type**: `Date`++Date of creation.++---++### `expiresAt`++**Type**: `Date`++Date of expiration.++---++### `usageCount`++**Type**: `number` <code> = 0</code>++Indicates how many times the session has been used.++---++### `errorCount`++**Type**: `number` <code> = 0</code>++Indicates how many times the session is marked bad.++---++### `maxUsageCount`++**Type**: `number` <code> = 50</code>++Session should be used only a limited amount of times. This number indicates how many times the session is going to be used, before it is thrown away.++---++### `sessionPool`++**Type**: [`SessionPool`](../api/session-pool)++SessionPool instance. Session will emit the `sessionRetired` event on this instance.++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/SystemInfo.md b/website/versioned_docs/version-1.0.2/typedefs/SystemInfo.mdnew file mode 100644index 000000000..7fae6fc23--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/SystemInfo.md@@ -0,0 +1,39 @@+---+id: version-1.0.2-system-info+title: SystemInfo+original_id: system-info+---++<a name="systeminfo"></a>++Represents the current status of the system.++## Properties++### `isSystemIdle`++**Type**: `boolean`++If true, system is being overloaded.++---++### `memInfo`++**Type**: `Object<string, *>`++Memory++---++### `eventLoopInfo`++**Type**: `Object<string, *>`++---++### `cpuInfo`++**Type**: `Object<string, *>`++---diff --git a/website/versioned_docs/version-1.0.2/typedefs/WebhookRun.md b/website/versioned_docs/version-1.0.2/typedefs/WebhookRun.mdnew file mode 100644index 000000000..3bf7d98dd--- /dev/null+++ b/website/versioned_docs/version-1.0.2/typedefs/WebhookRun.md@@ -0,0 +1,87 @@+---+id: version-1.0.2-webhook-run+title: WebhookRun+original_id: webhook-run+---++<a name="webhookrun"></a>++## Properties++### `id`++**Type**: `string`++---++### `createdAt`++**Type**: `string`++---++### `modifiedAt`++**Type**: `string`++---++### `userId`++**Type**: `string`++---++### `isAdHoc`++**Type**: `boolean`++---++### `eventTypes`++**Type**: [`EventTypes`](../typedefs/event-types)++---++### `condition`++**Type**: `*`++---++### `ignoreSslErrors`++**Type**: `boolean`++---++### `doNotRetry`++**Type**: `boolean`++---++### `requestUrl`++**Type**: `string`++---++### `payloadTemplate`++**Type**: `string`++---++### `lastDispatch`++**Type**: `*`++---++### `stats`++**Type**: `*`++---diff --git a/website/versions.json b/website/versions.jsonindex d67491c79..dece73bfd 100644--- a/website/versions.json+++ b/website/versions.json@@ -1,4 +1,5 @@[+ "1.0.2","1.0.1","1.0.0","0.22.4"