Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Feature: got scraping (#980)

petrpatek3a0aa4753661
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 75173124d..bf271b2eb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,16 @@
+1.2.0 / BETA
+====================
+This release brings the long awaited HTTP2 capabilities to `requestAsBrowser`. It could make HTTP2 requests even before, but it was not very helpful in making browser-like ones. This is very important for disguising as a browser and reduction in the number of blocked requests. `requestAsBrowser` now uses [`got-scraping`](https://github.com/apify/got-scraping).
+
+The most important new feature is that the full set of headers `requestAsBrowser` uses will now be generated using live data about browser headers that we collect. This means that the "header fingeprint" will always match existing browsers and should be indistinguishable from a real browser request. The header sets will be automatically rotated for you to further reduce the chances of blocking.
+
+We also switched the default HTTP version from 1 to 2 in `requestAsBrowser`. We don't expect this change to be breaking, and we took precautions, but we're aware that there are always some edge cases, so please let us know if it causes trouble for you.
+
+## Full list of changes:
+- Replace the underlying HTTP client of `utils.requestAsBrowser()` with `got-scraping`.
+- Make `useHttp2` `true` by default with `utils.requestAsBrowser()`.
+- Update `puppeteer` to `8.0.0` and `playwright` to `1.10.0` with Chromium 90 in Docker images.
+
1.1.2 / 2021/04/10
====================
- **DEPRECATED:** `utils.waitForRunToFinish` please use the `apify-client` package and its `waitForFinish` functions. Sorry, forgot to deprecate this with v1 release.
diff --git a/package.json b/package.json
index e9da062a8..2ba8701a0 100644
--- a/package.json
+++ b/package.json
@@ -67,6 +67,7 @@
"content-type": "^1.0.4",
"express": "^4.17.1",
"fs-extra": "^9.1.0",
+ "got-scraping": "^1.0.1",
"htmlparser2": "^6.0.0",
"iconv-lite": "^0.6.2",
"jquery": "^3.5.1",
@@ -106,11 +107,11 @@
"fpscanner": "^0.1.5",
"jest": "^26.6.3",
"jest-circus": "^26.6.3",
- "playwright": "1.7.1",
+ "playwright": "1.10.0",
"portastic": "^1.0.1",
"prettier": "^2.1.2",
"proxy": "^1.0.2",
- "puppeteer": "6.0.0",
+ "puppeteer": "8.0.0",
"sinon": "^9.2.0",
"sinon-stub-promise": "^4.0.0",
"socket.io-client": "^3.0.3",
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 533efc748..ed6f308d7 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -7,8 +7,8 @@ import { WritableStream } from 'htmlparser2/lib/WritableStream';
import * as iconv from 'iconv-lite';
import ow from 'ow';
import * as util from 'util';
+import { TimeoutError } from 'got-scraping';
import { BASIC_CRAWLER_TIMEOUT_BUFFER_SECS } from '../constants';
-import { TimeoutError } from '../errors';
import { addTimeoutToPromise, parseContentTypeFromResponse } from '../utils';
import * as utilsRequest from '../utils_request'; // eslint-disable-line import/no-duplicates
import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
@@ -554,10 +554,10 @@ class CheerioCrawler extends BasicCrawler {
}
const opts = this._getRequestOptions(request, session, proxyUrl);
- let responseStream;
+ let responseWithStream;
try {
- responseStream = await utilsRequest.requestAsBrowser(opts);
+ responseWithStream = await utilsRequest.requestAsBrowser(opts);
} catch (e) {
if (e instanceof TimeoutError) {
this._handleRequestTimeout(session);
@@ -566,7 +566,7 @@ class CheerioCrawler extends BasicCrawler {
}
}
- return responseStream;
+ return responseWithStream;
}
/**
@@ -705,7 +705,9 @@ class CheerioCrawler extends BasicCrawler {
});
const parser = new WritableStream(domHandler, { decodeEntities: true });
parser.on('error', reject);
- response.on('error', reject).pipe(parser);
+ response
+ .on('error', reject)
+ .pipe(parser);
});
}
diff --git a/src/utils_request.js b/src/utils_request.js
index 64d692bc5..82e944ad1 100644
--- a/src/utils_request.js
+++ b/src/utils_request.js
@@ -1,22 +1,11 @@
-import * as httpRequest from '@apify/http-request';
-import * as errors from '@apify/http-request/src/errors';
+import * as gotScraping from 'got-scraping';
+import ow from 'ow';
+import log from './utils_log';
+
/* eslint-disable no-unused-vars,import/named,import/order */
-import { TimeoutError } from './errors';
import { IncomingMessage } from 'http';
-import { Readable } from 'stream';
/* eslint-enable no-unused-vars,import/named,import/order */
-export const FIREFOX_MOBILE_USER_AGENT = 'Mozilla/5.0 (Android; Mobile; rv:14.0) Gecko/14.0 Firefox/14.0';
-export const FIREFOX_DESKTOP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0';
-
-const DEFAULT_HTTP_REQUEST_OPTIONS = {
- useBrotli: true,
- json: false,
- useCaseSensitiveHeaders: true,
- stream: false,
- timeoutSecs: 30,
-};
-
/**
* @typedef {(IncomingMessage & Readable & { body: string })} RequestAsBrowserResult
*/
@@ -33,6 +22,9 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
* default browser headers will remove the masking this function provides.
* @property {string} [proxyUrl]
* An HTTP proxy to be passed down to the HTTP request. Supports proxy authentication with Basic Auth.
+ * @property {object} [headerGeneratorOptions]
+ * Configuration to be used for generating correct browser headers.
+ * See the [`header-generator`](https://github.com/apify/header-generator) library.
* @property {string} [languageCode=en]
* Two-letter ISO 639 language code.
* @property {string} [countryCode=US]
@@ -50,12 +42,13 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
* although the risk should be negligible as these vulnerabilities mainly relate to server applications, not clients.
* Learn more in this [blog post](https://snyk.io/blog/node-js-release-fixes-a-critical-http-security-vulnerability/).
* @property {AbortFunction} [abortFunction]
- * Function accepts `response` object as a single parameter and should return true or false.
- * If function returns true request gets aborted. This function is passed to the
- * [@apify/http-request](https://www.npmjs.com/package/@apify/http-request) NPM package.
- * @property {boolean} [useHttp2=false]
- * If set to true, it will additionally accept HTTP2 requests.
- * It will choose either HTTP/1.1 or HTTP/2 depending on the ALPN protocol.
+ * Function accepts `response` object as a single parameter and should return `true` or `false`.
+ * If function returns true, request gets aborted.
+ * @property {boolean} [useHttp2=true]
+ * If set to false, it will prevent use of HTTP2 requests. This is strongly discouraged. Websites
+ * expect HTTP2 connections, because browsers use HTTP2 by default. It will automatically downgrade
+ * to HTTP/1.1 for websites that do not support HTTP2. For Node 10 this option is always set to `false`
+ * because Node 10 does not support HTTP2 very well. Upgrade to Node 12 for better performance.
*/
/**
@@ -71,22 +64,19 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
* scenarios, please set `useInsecureHttpParser: false` and `ignoreSslErrors: false`.
*
* Sends a HTTP request that looks like a request sent by a web browser,
- * fully emulating browser's HTTP headers.
+ * fully emulating browser's HTTP headers. It uses HTTP2 by default for Node 12+.
*
* This function is useful for web scraping of websites that send the full HTML in the first response.
* Thanks to this function, the target web server has no simple way to find out the request
- * hasn't been sent by a full web browser. Using a headless browser for such requests
+ * hasn't been sent by a human's web browser. Using a headless browser for such requests
* is an order of magnitude more resource-intensive than this function.
- * By default it aborts all requests that returns 406 status codes or non-HTML content-types.
- * You can override this behavior by passing custom `abortFunction`.
*
- * Currently, the function sends requests the same way as Firefox web browser does.
- * In the future, it might add support for other browsers too.
+ * The function emulates the Chrome and Firefox web browsers. If you want more control
+ * over the browsers and their versions, use the `headerGeneratorOptions` property.
+ * You can find more info in the readme of the [`header-generator`](https://github.com/apify/header-generator) library.
*
- * Internally, the function uses `httpRequest` function from the [@apify/http-request](https://github.com/apify/http-request)
- * NPM package to perform the request.
- * All `options` not recognized by this function are passed to it,
- * so see it for more details.
+ * Internally, the function uses the [`got-scraping`](https://github.com/apify/got-scraping) library to perform the request.
+ * All `options` not recognized by this function are passed to it so see it for more details.
*
* **Example usage:**
* ```js
@@ -105,73 +95,301 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {
*
* @param {RequestAsBrowserOptions} options All `requestAsBrowser` configuration options.
*
- * @return {Promise<RequestAsBrowserResult>} This will typically be a
- * [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage),
- * however, if returned from the cache it will be a [response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.
+ * @return {Promise<RequestAsBrowserResult>} The result can be various objects, but it will always be like a
+ * [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage)
+ * with a 'body' property for the parsed response body, unless the 'stream' option is used.
* @memberOf utils
* @name requestAsBrowser
* @function
*/
-export const requestAsBrowser = async (options) => {
+export const requestAsBrowser = async (options = {}) => {
+ logDeprecatedOptions(options);
+ ow(options, 'RequestAsBrowserOptions', ow.object.partialShape({
+ payload: ow.optional.any(ow.string, ow.buffer),
+ proxyUrl: ow.optional.string.url,
+ languageCode: ow.optional.string.length(2),
+ countryCode: ow.optional.string.length(2),
+ useMobileVersion: ow.optional.boolean,
+ abortFunction: ow.optional.function,
+ ignoreSslErrors: ow.optional.boolean,
+ useInsecureHttpParser: ow.optional.boolean,
+ useHttp2: ow.optional.boolean,
+ timeoutSecs: ow.optional.number,
+ throwOnHttpErrors: ow.optional.boolean,
+ headerGeneratorOptions: ow.optional.object,
+ stream: ow.optional.boolean,
+ decodeBody: ow.optional.boolean,
+ }));
+
+ ow(options, 'RequestAsBrowserOptions', ow.object.validate((opts) => ({
+ validator: areBodyOptionsCompatible(opts),
+ message: (label) => `The 'payload', 'body', 'json' and 'form' options of ${label} are mutually exclusive.`,
+ })));
+
+ // We created the `got-scraping` package which replaced underlying @apify/http-request.
+ // At the same time, we want users to be able to use requestAsBrowser without breaking changes.
+ // So we do a lot of property mapping here, to make sure that everything works as expected.
+ // TODO Update this with SDK v2 and use `got-scraping` API directly.
const {
- url,
- method = 'GET',
- headers = {},
+ payload, // alias for body to allow direct passing of our Request objects
proxyUrl,
+ json,
+ headerGeneratorOptions,
languageCode = 'en',
countryCode = 'US',
useMobileVersion = false,
- abortFunction,
+ abortFunction = () => false,
ignoreSslErrors = true,
useInsecureHttpParser = true,
- useHttp2 = false,
- ...otherParams
+ useHttp2 = true,
+ timeoutSecs = 30,
+ throwOnHttpErrors = false,
+ stream = false,
+ decodeBody = true,
+ ...gotParams
} = options;
- const defaultHeaders = {
- 'User-Agent': useMobileVersion ? FIREFOX_MOBILE_USER_AGENT : FIREFOX_DESKTOP_USER_AGENT,
- Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
- 'Accept-Language': `${languageCode}-${countryCode},${languageCode};q=0.5`,
- 'Accept-Encoding': 'gzip, deflate, br',
- Connection: useHttp2 ? undefined : 'keep-alive',
- };
-
- const requestOpts = {
- ...DEFAULT_HTTP_REQUEST_OPTIONS,
- ...otherParams,
- url,
- method,
- // Users can provide headers in lowercase so we need to make sure
- // that their values are applied, but names are kept upper-case.
- headers: mergeHeaders(headers, defaultHeaders),
+ const gotScrapingOptions = {
proxyUrl,
- abortFunction,
- ignoreSslErrors,
insecureHTTPParser: useInsecureHttpParser,
- useHttp2,
+ http2: useHttp2,
+ timeout: timeoutSecs * 1000,
+ throwHttpErrors: throwOnHttpErrors,
+ isStream: stream,
+ decompress: decodeBody,
+ // We overwrite the above arguments because we want to give the official
+ // got interface a priority over our requestAsBrowser one.
+ // E.g. { isStream: false, stream: true } should produce { isStream: false }.
+ ...gotParams,
};
- try {
- return await httpRequest(requestOpts);
- } catch (e) {
- if (e instanceof errors.TimeoutError) {
- throw new TimeoutError(`Request Timed-out after ${requestOpts.timeoutSecs} seconds.`);
- }
+ // TODO remove when we drop support for Node 10
+ ensureValidConfigForNode10(gotScrapingOptions);
- throw e;
+ // Order is important for payload and json.
+ normalizePayloadOption(payload, gotScrapingOptions);
+ normalizeJsonOption(json, gotScrapingOptions);
+ normalizeSslErrorHandling(ignoreSslErrors, gotScrapingOptions);
+ ensureCorrectHttp2Headers(gotScrapingOptions);
+ maybeAddAbortHook(abortFunction, gotScrapingOptions);
+ if (!headerGeneratorOptions) {
+ // Values that respect old requestAsBrowser user-agents and settings
+ gotScrapingOptions.headerGeneratorOptions = {
+ devices: useMobileVersion ? ['mobile'] : ['desktop'],
+ locales: [`${languageCode}-${countryCode}`],
+ };
}
+
+ if (!gotScrapingOptions.isStream) return gotScraping(gotScrapingOptions);
+
+ // abortFunction must be handled separately for streams :(
+ const duplexStream = await gotScraping(gotScrapingOptions);
+ ensureRequestIsDispatched(duplexStream, gotScrapingOptions);
+ return new Promise((resolve, reject) => {
+ duplexStream
+ .on('error', reject)
+ .on('response', (res) => {
+ try {
+ const shouldAbort = abortFunction(res);
+ if (shouldAbort) {
+ const err = new Error(`Request for ${gotScrapingOptions.url} aborted due to abortFunction.`);
+ duplexStream.destroy(err);
+ return reject(err);
+ }
+ } catch (e) {
+ duplexStream.destroy(e);
+ return reject(e);
+ }
+
+ addResponsePropertiesToStream(duplexStream, res);
+
+ return resolve(duplexStream);
+ });
+ });
};
-function mergeHeaders(userHeaders, defaultHeaders) {
- const headers = { ...defaultHeaders, ...userHeaders };
- Object.keys(headers).forEach((key) => {
- const lowerCaseKey = key.toLowerCase();
- const keyIsNotLowerCase = key !== lowerCaseKey;
- // eslint-disable-next-line
- if (keyIsNotLowerCase && headers.hasOwnProperty(lowerCaseKey)) {
- headers[key] = headers[lowerCaseKey];
- delete headers[lowerCaseKey];
+function ensureValidConfigForNode10(gotScrapingOptions) {
+ if (process.version.startsWith('v10')) {
+ if (gotScrapingOptions.http2 === true) {
+ // Using log.deprecated to log only once.
+ log.deprecated('utils.requestAsBrowser does not support HTTP2 on Node 10. Please upgrade to Node 12+ or set useHttp2 to false.');
+ }
+ gotScrapingOptions.http2 = false;
+ gotScrapingOptions.ciphers = undefined;
+ }
+}
+
+/**
+ * `got` has a `body` option and 2 helpers, `json` and `form`, to provide specific bodies.
+ * Those options are mutually exclusive. `requestAsBrowser` also supports `payload` as
+ * an alias of `body`. It must be exclusive as well.
+ * @param {RequestAsBrowserOptions} requestAsBrowserOptions
+ * @return {boolean}
+ * @private
+ * @ignore
+ */
+function areBodyOptionsCompatible(requestAsBrowserOptions) {
+ const { payload, json, body, form } = requestAsBrowserOptions;
+ // A boolean is old requestAsBrowser interface and not a real "body"
+ // See the normalizeJsonOption function.
+ const jsonBody = typeof json !== 'boolean' ? undefined : json;
+
+ const possibleOpts = [payload, jsonBody, body, form];
+ const usedOpts = possibleOpts.filter((opt) => opt !== undefined);
+
+ // Only a single option out of the 4 can be used.
+ return usedOpts.length <= 1;
+}
+
+/**
+ * got-scraping uses 'body', but we also support 'payload' from {@link Request}.
+ * @param {string|Buffer} payload
+ * @param {GotScrapingOptions} gotScrapingOptions
+ * @ignore
+ * @private
+ */
+function normalizePayloadOption(payload, gotScrapingOptions) {
+ if (payload !== undefined) gotScrapingOptions.body = payload;
+}
+
+/**
+ * `json` is a boolean flag in `requestAsBrowser`, but a `body` alias that
+ * adds a 'content-type: application/json' header in got. To stay backwards
+ * compatible we need to figure out which option the user provided.
+ * @param {*} json
+ * @param {GotScrapingOptions} gotScrapingOptions
+ * @ignore
+ * @private
+ */
+function normalizeJsonOption(json, gotScrapingOptions) {
+ // If it's a boolean, then it's the old requestAsBrowser API.
+ if (typeof json === 'boolean') {
+ gotScrapingOptions.responseType = 'json';
+ gotScrapingOptions.ciphers = undefined;
+ // If there is a body, we move it under `json` to get the automatic
+ // 'content-type' header injection.
+ if (gotScrapingOptions.body !== undefined) gotScrapingOptions.json = gotScrapingOptions.body;
+ } else {
+ // If it's something else, we let `got` handle it.
+ gotScrapingOptions.json = json;
+ }
+}
+
+/**
+ * @param {boolean} ignoreSslErrors
+ * @param {GotScrapingOptions} gotScrapingOptions
+ * @ignore
+ * @private
+ */
+function normalizeSslErrorHandling(ignoreSslErrors, gotScrapingOptions) {
+ if (gotScrapingOptions.https) {
+ gotScrapingOptions.https.rejectUnauthorized = !ignoreSslErrors;
+ } else {
+ gotScrapingOptions.https = { rejectUnauthorized: !ignoreSslErrors };
+ }
+}
+
+/**
+ * 'connection' and 'host' headers are forbidden when using HTTP2. We delete
+ * them from user-provided headers because we switched the default from HTTP1 to 2.
+ * @param {GotScrapingOptions} gotScrapingOptions
+ * @ignore
+ * @private
+ */
+function ensureCorrectHttp2Headers(gotScrapingOptions) {
+ if (gotScrapingOptions.http2) {
+ delete gotScrapingOptions.headers?.connection;
+ delete gotScrapingOptions.headers?.Connection;
+ delete gotScrapingOptions.headers?.host;
+ delete gotScrapingOptions.headers?.Host;
+ }
+}
+
+/**
+ * `abortFunction` is an old `requestAsBrowser` interface for aborting requests before
+ * the response body is read to save bandwidth.
+ * @param {function} abortFunction
+ * @param {GotScrapingOptions} gotScrapingOptions
+ * @ignore
+ * @private
+ */
+function maybeAddAbortHook(abortFunction, gotScrapingOptions) {
+ // Stream aborting must be handled on the response object because `got`
+ // does not execute `afterResponse` hooks for streams :(
+ if (gotScrapingOptions.isStream) return;
+
+ const abortHook = (response) => {
+ const shouldAbort = abortFunction(response);
+ if (shouldAbort) {
+ throw new Error(`Request for ${gotScrapingOptions.url} aborted due to abortFunction.`);
+ }
+ return response;
+ };
+
+ const abortRequestOptions = {
+ hooks: {
+ afterResponse: [abortHook],
+ },
+ };
+ gotScrapingOptions = gotScraping.mergeOptions(gotScraping.defaults.options, gotScrapingOptions, abortRequestOptions);
+}
+
+/**
+ * 'got' will not dispatch non-GET request stream until a body is provided.
+ * @param {stream.Duplex} duplexStream
+ * @param {GotScrapingOptions} gotScrapingOptions
+ */
+function ensureRequestIsDispatched(duplexStream, gotScrapingOptions) {
+ const { method } = gotScrapingOptions;
+ const bodyIsEmpty = gotScrapingOptions.body === undefined
+ && gotScrapingOptions.json === undefined
+ && gotScrapingOptions.form === undefined;
+
+ if (method && method.toLowerCase() !== 'get' && bodyIsEmpty) {
+ duplexStream.end();
+ }
+}
+
+/**
+ * @param {RequestAsBrowserOptions} options
+ * @ignore
+ * @private
+ */
+function logDeprecatedOptions(options) {
+ const deprecatedOptions = ['languageCode', 'countryCode', 'useMobileVersion'];
+
+ for (const deprecatedOption of deprecatedOptions) {
+ if (options.hasOwnProperty(deprecatedOption)) {
+ log.deprecated(`"options.${deprecatedOption}" is deprecated. Use "options.headerGeneratorOptions" instead.`);
+ }
+ }
+}
+
+/**
+ * The stream object returned from got does not have the below properties.
+ * At the same time, you can't read data directly from the response stream,
+ * because they won't get emitted unless you also read from the primary
+ * got stream. To be able to work with only one stream, we move the expected props
+ * from the response stream to the got stream.
+ * @param {GotStream} stream
+ * @param {http.IncomingMessage} response
+ * @return {GotStream}
+ * @ignore
+ * @private
+ */
+function addResponsePropertiesToStream(stream, response) {
+ const properties = [
+ 'statusCode', 'statusMessage', 'headers',
+ 'complete', 'httpVersion', 'rawHeaders',
+ 'rawTrailers', 'trailers', 'url',
+ 'request',
+ ];
+
+ properties.forEach((prop) => {
+ if (stream[prop] === undefined) {
+ stream[prop] = response[prop];
}
});
- return headers;
+
+ return stream;
}
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 733d8ae22..0dbaabd06 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -340,7 +340,11 @@ describe('CheerioCrawler', () => {
await crawler.run();
expect(headers).toHaveLength(4);
- headers.forEach((h) => expect(h.Accept).toBe('text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'));
+ headers.forEach((h) => {
+ const acceptHeader = h.accept || h.Accept;
+ expect(acceptHeader.includes('text/html')).toBe(true);
+ expect(acceptHeader.includes('application/xhtml+xml')).toBe(true);
+ });
});
describe('by throwing', () => {
diff --git a/test/utils_request.test.js b/test/utils_request.test.js
index 8412bb8f4..5dc4e45c8 100644
--- a/test/utils_request.test.js
+++ b/test/utils_request.test.js
@@ -1,8 +1,6 @@
import express from 'express';
import {
requestAsBrowser,
- FIREFOX_MOBILE_USER_AGENT,
- FIREFOX_DESKTOP_USER_AGENT,
} from '../build/utils_request';
import { startExpressAppPromise } from './_helper';
@@ -50,6 +48,11 @@ describe('Apify.utils_request', () => {
res.send();
});
+ app.post('/echo-body', async (req, res) => {
+ res.setHeader('Content-Type', 'text/html; charset=utf-8');
+ req.pipe(res);
+ });
+
app.get('/invalidHeaderChar', (req) => {
const headers = {
'Invalid Header With Space': 'some\value',
@@ -91,7 +94,7 @@ describe('Apify.utils_request', () => {
};
const response = await requestAsBrowser(data);
expect(response.statusCode).toBe(200);
- expect(JSON.parse(response.body)['user-agent']).toEqual(FIREFOX_MOBILE_USER_AGENT);
+ expect(response.request.options.context.headerGeneratorOptions.devices).toEqual(['mobile']);
},
);
@@ -101,7 +104,7 @@ describe('Apify.utils_request', () => {
};
const response = await requestAsBrowser(data);
expect(response.statusCode).toBe(200);
- expect(JSON.parse(response.body)['user-agent']).toEqual(FIREFOX_DESKTOP_USER_AGENT);
+ expect(response.request.options.context.headerGeneratorOptions.devices).toEqual(['desktop']);
});
test('sets correct hosts', async () => {
@@ -127,7 +130,7 @@ describe('Apify.utils_request', () => {
const response = await requestAsBrowser(options);
expect(response.statusCode).toBe(200);
- expect(JSON.parse(response.body)['accept-language']).toEqual(`${languageCode}-${countryCode},${languageCode};q=0.5`);
+ expect(response.request.options.context.headerGeneratorOptions.locales).toEqual([`${languageCode}-${countryCode}`]);
});
test('does not throw for empty response body', async () => {
@@ -159,47 +162,20 @@ describe('Apify.utils_request', () => {
expect(JSON.parse(response.body)['user-agent']).toEqual(options.headers['User-Agent']);
});
- test('headers has same format as in firefox', async () => {
- const host = `${HOSTNAME}:${port}`;
- const options = {
- url: `http://${host}/rawHeaders`,
- };
-
- const response = await requestAsBrowser(options);
- const headersArray = JSON.parse(response.body);
- expect(response.statusCode).toBe(200);
-
- expect(headersArray[0]).toBe('Host');
- expect(headersArray[1]).toEqual(host);
- expect(headersArray[2]).toBe('User-Agent');
- expect(headersArray[3]).toEqual(FIREFOX_DESKTOP_USER_AGENT);
- expect(headersArray[4]).toBe('Accept');
- expect(headersArray[5]).toBe('text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8');
- expect(headersArray[6]).toBe('Accept-Language');
- expect(headersArray[7]).toBe('en-US,en;q=0.5');
- expect(headersArray[8]).toBe('Accept-Encoding');
- expect(headersArray[9]).toBe('gzip, deflate, br');
- expect(headersArray[10]).toBe('Connection');
- expect(headersArray[11]).toBe('keep-alive');
- });
-
- test('custom headers in lowercase override uppercase defaults', async () => {
+ test('custom headers in uppercase for HTTP1', async () => {
const host = `${HOSTNAME}:${port}`;
const options = {
url: `http://${host}/rawHeaders`,
headers: {
- accept: 'foo',
+ Accept: 'foo',
bar: 'BAZ',
},
+ useHttp2: false,
};
const response = await requestAsBrowser(options);
- const headersArray = JSON.parse(response.body);
expect(response.statusCode).toBe(200);
- expect(headersArray[4]).toBe('Accept');
- expect(headersArray[5]).toBe('foo');
- expect(headersArray[12]).toBe('bar');
- expect(headersArray[13]).toBe('BAZ');
+ expect(response.request.options.headers).toMatchObject(options.headers);
});
test('correctly handles invalid header characters', async () => {
@@ -236,9 +212,49 @@ describe('Apify.utils_request', () => {
}
});
+ // TODO we need to do this better, it will be flaky. The test above is not flaky,
+ // because it only checks for a very specific error so it won't fail on network errors.
test('works with useHttp2', async () => {
- const url = 'https://www.amazon.com/s?k=iphone';
- await requestAsBrowser({ url, useHttp2: true });
+ const url = 'https://apify.com';
+ const response = await requestAsBrowser({ url, useHttp2: true });
+ // TODO Node v10 does not support HTTP2 well, remove when we drop support.
+ if (process.version.startsWith('v10')) {
+ expect(response.request.options.http2).toBe(false);
+ } else {
+ expect(response.request.options.http2).toBe(true);
+ }
+ expect(response.body.length).toBeGreaterThan(10000);
+ });
+
+ // TODO same here
+ test('get works with streams', async () => {
+ const response = await requestAsBrowser({
+ url: 'https://apify.com/',
+ stream: true,
+ });
+ expect(response.options.isStream).toBe(true);
+ const chunks = [];
+ for await (const chunk of response) {
+ chunks.push(chunk);
+ }
+ const body = chunks.join();
+ expect(body.length).toBeGreaterThan(10000);
+ });
+
+ test('post works with streams', async () => {
+ const response = await requestAsBrowser({
+ method: 'POST',
+ url: `http://${HOSTNAME}:${port}/echo-body`,
+ stream: true,
+ payload: 'TEST',
+ });
+ expect(response.options.isStream).toBe(true);
+ const chunks = [];
+ for await (const chunk of response) {
+ chunks.push(chunk);
+ }
+ const body = chunks.join();
+ expect(body).toBe('TEST');
});
});
});
diff --git a/website/versioned_docs/version-1.1.2/api/KeyValueStore.md b/website/versioned_docs/version-1.1.2/api/KeyValueStore.md
new file mode 100644
index 000000000..d70bc5cf8
--- /dev/null
+++ b/website/versioned_docs/version-1.1.2/api/KeyValueStore.md
@@ -0,0 +1,211 @@
+---
+id: version-1.1.2-key-value-store
+title: KeyValueStore
+original_id: key-value-store
+---
+
+<a name="keyvaluestore"></a>
+
+The `KeyValueStore` class represents a key-value store, a simple data storage that is used for saving and reading data records or files. Each data
+record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots, actor inputs and
+outputs, web pages, PDFs or to persist the state of crawlers.
+
+Do not instantiate this class directly, use the [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) function instead.
+
+Each actor run is associated with a default key-value store, which is created exclusively for the run. By convention, the actor input and output are
+stored into the default key-value store under the `INPUT` and `OUTPUT` key, respectively. Typically, input and output are JSON files, although it can
+be any other format. To access the default key-value store directly, you can use the [`Apify.getValue()`](../api/apify#getvalue) and
+[`Apify.setValue()`](../api/apify#setvalue) convenience functions.
+
+To access the input, you can also use the [`Apify.getInput()`](../api/apify#getinput) convenience function.
+
+`KeyValueStore` stores its data either on local disk or in the Apify cloud, depending on whether the
+[`APIFY_LOCAL_STORAGE_DIR`](../guides/environment-variables#apify_local_storage_dir) or [`APIFY_TOKEN`](../guides/environment-variables#apify_token)
+environment variables are set.
+
+If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in the local directory in the following files:
+
+```
+{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}
+```
+
+Note that `{STORE_ID}` is the name or ID of the key-value store. The default key-value store has ID: `default`, unless you override it by setting the
+`APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the
+data value.
+
+If the [`APIFY_TOKEN`](../guides/environment-variables#apify_token) environment variable is set but
+[`APIFY_LOCAL_STORAGE_DIR`](../guides/environment-variables#apify_local_storage_dir) not, the data is stored in the
+[Apify Key-value store](https://docs.apify.com/storage/key-value-store) cloud storage. Note that you can force usage of the cloud storage also by
+passing the `forceCloud` option to [`Apify.openKeyValueStore()`](../api/apify#openkeyvaluestore) function, even if the
+[`APIFY_LOCAL_STORAGE_DIR`](../guides/environment-variables#apify_local_storage_dir) variable is set.
+
+**Example usage:**
+
+```javascript
+// Get actor input from the default key-value store.
+const input = await Apify.getInput();
+// Get some value from the default key-value store.
+const otherValue = await Apify.getValue('my-key');
+
+// Write actor output to the default key-value store.
+await Apify.setValue('OUTPUT', { myResult: 123 });
+
+// Open a named key-value store
+const store = await Apify.openKeyValueStore('some-name');
+
+// Write a record. JavaScript object is automatically converted to JSON,
+// strings and binary buffers are stored as they are
+await store.setValue('some-key', { foo: 'bar' });
+
+// Read a record. Note that JSON is automatically parsed to a JavaScript object,
+// text data returned as a string and other data is returned as binary buffer
+const value = await store.getValue('some-key');
+
+// Drop (delete) the store
+await store.drop();
+```
+
+---
+
+<a name="getvalue"></a>
+
+## `keyValueStore.getValue(key)`
+
+Gets a value from the key-value store.
+
+The function returns a `Promise` that resolves to the record value, whose JavaScript type depends on the MIME content type of the record. Records with
+the `application/json` content type are automatically parsed and returned as a JavaScript object. Similarly, records with `text/plain` content types
+are returned as a string. For all other content types, the value is returned as a raw [`Buffer`](https://nodejs.org/api/buffer.html) instance.
+
+If the record does not exist, the function resolves to `null`.
+
+To save or delete a value in the key-value store, use the [`KeyValueStore.setValue()`](../api/key-value-store#setvalue) function.
+
+**Example usage:**
+
+```javascript
+const store = await Apify.openKeyValueStore();
+const buffer = await store.getValue('screenshot1.png');
+```
+
+**Parameters**:
+
+- **`key`**: `string` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,
+ `A`-`Z`, `0`-`9` and `!-_.'()`
+
+**Returns**:
+
+[`Promise<KeyValueStoreValueTypes>`](../typedefs/key-value-store-value-types) - Returns a promise that resolves to an object, string or
+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record.
+
+---
+
+<a name="setvalue"></a>
+
+## `keyValueStore.setValue(key, value, [options])`
+
+Saves or deletes a record in the key-value store. The function returns a promise that resolves once the record has been saved or deleted.
+
+**Example usage:**
+
+```javascript
+const store = await Apify.openKeyValueStore();
+await store.setValue('OUTPUT', { foo: 'bar' });
+```
+
+Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`
+
+By default, `value` is converted to JSON and stored with the `application/json; charset=utf-8` MIME content type. To store the value with another
+content type, pass it in the options as follows:
+
+```javascript
+const store = await Apify.openKeyValueStore('my-text-store');
+await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });
+```
+
+If you set custom content type, `value` must be either a string or [`Buffer`](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.
+
+If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds regardless whether the record existed or not.
+
+To retrieve a value from the key-value store, use the [`KeyValueStore.getValue()`](../api/key-value-store#getvalue) function.
+
+**IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`, otherwise the actor process might finish before the value is
+stored!
+
+**Parameters**:
+
+- **`key`**: `string` - Unique key of the record. It can be at most 256 characters long and only consist of the following characters: `a`-`z`,
+ `A`-`Z`, `0`-`9` and `!-_.'()`
+- **`value`**: [`KeyValueStoreValueTypes`](../typedefs/key-value-store-value-types) - Record data, which can be one of the following values:
+ - If `null`, the record in the key-value store is deleted.
+ - If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.
+ - If `options.contentType` is set, `value` is taken as is and it must be a `String` or [`Buffer`](https://nodejs.org/api/buffer.html). For any
+ other value an error will be thrown.
+- **`[options]`**: `object`
+ - **`[contentType]`**: `string` - Specifies a custom MIME content type of the record.
+
+**Returns**:
+
+`Promise<void>`
+
+---
+
+<a name="drop"></a>
+
+## `keyValueStore.drop()`
+
+Removes the key-value store either from the Apify cloud storage or from the local directory, depending on the mode of operation.
+
+**Returns**:
+
+`Promise<void>`
+
+---
+
+<a name="getpublicurl"></a>
+
+## `keyValueStore.getPublicUrl(key)`
+
+Returns a URL for the given key that may be used to publicly access the value in the remote key-value store.
+
+**Parameters**:
+
+- **`key`**: `string`
+
+**Returns**:
+
+`string`
+
+---
+
+<a name="foreachkey"></a>
+
+## `keyValueStore.forEachKey(iteratee, [options])`
+
+Iterates over key-value store keys, yielding each in turn to an `iteratee` function. Each invocation of `iteratee` is called with three arguments:
+`(key, index, info)`, where `key` is the record key, `index` is a zero-based index of the key in the current iteration (regardless of
+`options.exclusiveStartKey`) and `info` is an object that contains a single property `size` indicating size of the record in bytes.
+
+If the `iteratee` function returns a Promise then it is awaited before the next call. If it throws an error, the iteration is aborted and the
+`forEachKey` function throws the error.
+
+**Example usage**
+
+```javascript
+const keyValueStore = await Apify.openKeyValueStore();
+await keyValueStore.forEachKey(async (key, index, info) => {
+ console.log(`Key at ${index}: ${key} has size ${info.size}`);
+});
+```
+
+**Parameters**:
+
+- **`iteratee`**: [`KeyConsumer`](../typedefs/key-consumer) - A function that is called for every key in the key-value store.
+- **`[options]`**: `object` - All `forEachKey()` parameters are passed via an options object with the following keys:
+ - **`[exclusiveStartKey]`**: `string` - All keys up to this one (including) are skipped from the result.
+
+**Returns**:
+
+`Promise<void>`
+
+---
diff --git a/website/versioned_docs/version-1.1.2/api/utils.md b/website/versioned_docs/version-1.1.2/api/utils.md
new file mode 100644
index 000000000..6520b8f09
--- /dev/null
+++ b/website/versioned_docs/version-1.1.2/api/utils.md
@@ -0,0 +1,280 @@
+---
+id: version-1.1.2-utils
+title: utils
+original_id: utils
+---
+
+<a name="utils"></a>
+
+A namespace that contains various utilities.
+
+**Example usage:**
+
+```javascript
+const Apify = require('apify');
+
+...
+
+// Sleep 1.5 seconds
+await Apify.utils.sleep(1500);
+```
+
+---
+
+<a name="url_no_commas_regex"></a>
+
+## `utils.URL_NO_COMMAS_REGEX`
+
+Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters and does not
+support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
+
+---
+
+<a name="url_with_commas_regex"></a>
+
+## `utils.URL_WITH_COMMAS_REGEX`
+
+Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query. Note,
+however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
+
+---
+
+<a name="enqueuelinks"></a>
+
+## `utils.enqueueLinks(options)`
+
+The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default) either in a Puppeteer page, or in a Cheerio object
+(parsed HTML), and enqueues the URLs in their `href` attributes to the provided [`RequestQueue`](../api/request-queue). If you're looking to find URLs
+in JavaScript heavy pages where links are not available in `href` elements, but rather navigations are triggered in click handlers see
+[`puppeteer.enqueueLinksByClickingElements()`](../api/puppeteer#enqueuelinksbyclickingelements).
+
+Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](../api/pseudo-url) objects and override settings
+of the enqueued [`Request`](../api/request) objects.
+
+**Example usage**
+
+```javascript
+await Apify.utils.enqueueLinks({
+ page,
+ requestQueue,
+ selector: 'a.product-detail',
+ pseudoUrls: ['https://www.example.com/handbags/[.*]', 'https://www.example.com/purses/[.*]'],
+});
+```
+
+**Parameters**:
+
+- **`options`**: `object` - All `enqueueLinks()` parameters are passed via an options object with the following keys:
+
+ - **`[page]`**: `PuppeteerPage` | `PlaywrightPage` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object. Either
+ `page` or `$` option must be provided.
+ - **`[limit]`**: `Number` - Limit the count of actually enqueued URLs to this number. Useful for testing across the entire crawling scope.
+ - **`[$]`**: `cheerio.Root` | `cheerio.Selector` - [`Cheerio`](https://github.com/cheeriojs/cheerio) function with loaded HTML. Either `page` or
+ `$` option must be provided.
+ - **`requestQueue`**: [`RequestQueue`](../api/request-queue) - A request queue to which the URLs will be enqueued.
+ - **`[selector]`**: `string` <code> = &quot;&#x27;a&#x27;&quot;</code> - A CSS selector matching links to be enqueued.
+ - **`[baseUrl]`**: `string` - A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer, since the
+ relative URL resolution is done inside the browser automatically.
+ - **`[pseudoUrls]`**: `Array<Object<string, *&gt;>` | `Array.&lt;string>` - An array of [`PseudoUrl`](../api/pseudo-url)s matching the URLs to
+ be enqueued, or an array of strings or RegExps or plain Objects from which the [`PseudoUrl`](../api/pseudo-url)s can be constructed.
+
+ The plain objects must include at least the `purl` property, which holds the pseudo-URL string or RegExp. All remaining keys will be used as the
+ `requestTemplate` argument of the [`PseudoUrl`](../api/pseudo-url) constructor, which lets you specify special properties for the enqueued
+ [`Request`](../api/request) objects.
+
+ If `pseudoUrls` is an empty array, `null` or `undefined`, then the function enqueues all links found on the page.
+
+ - **`[transformRequestFunction]`**: [`RequestTransform`](../typedefs/request-transform) - Just before a new [`Request`](../api/request) is
+ constructed and enqueued to the [`RequestQueue`](../api/request-queue), this function can be used to remove it or modify its contents such as
+ `userData`, `payload` or, most importantly `uniqueKey`. This is useful when you need to enqueue multiple `Requests` to the queue that share
+ the same URL, but differ in methods or payloads, or to dynamically update or create `userData`.
+
+ For example: by adding `keepUrlFragment: true` to the `request` object, URL fragments will not be removed when `uniqueKey` is computed.
+
+ **Example:**
+
+ ```javascript
+ {
+ transformRequestFunction: request => {
+ request.userData.foo = 'bar';
+ request.keepUrlFragment = true;
+ return request;
+ };
+ }
+ ```
+
+**Returns**:
+
+[`Promise<Array<QueueOperationInfo>>`](../typedefs/queue-operation-info) - Promise that resolves to an array of
+[`QueueOperationInfo`](../typedefs/queue-operation-info) objects.
+
+---
+
+<a name="requestasbrowser"></a>
+
+## `utils.requestAsBrowser(options)`
+
+**IMPORTANT:** This function uses an insecure version of HTTP parser by default and also ignores SSL/TLS errors. This is very useful in scraping,
+because it allows bypassing certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping scenarios, please set
+`useInsecureHttpParser: false` and `ignoreSslErrors: false`.
+
+Sends a HTTP request that looks like a request sent by a web browser, fully emulating browser's HTTP headers.
+
+This function is useful for web scraping of websites that send the full HTML in the first response. Thanks to this function, the target web server has
+no simple way to find out the request hasn't been sent by a full web browser. Using a headless browser for such requests is an order of magnitude more
+resource-intensive than this function. By default it aborts all requests that returns 406 status codes or non-HTML content-types. You can override
+this behavior by passing custom `abortFunction`.
+
+Currently, the function sends requests the same way as Firefox web browser does. In the future, it might add support for other browsers too.
+
+Internally, the function uses `httpRequest` function from the [@apify/http-request](https://github.com/apify/http-request) NPM package to perform the
+request. All `options` not recognized by this function are passed to it, so see it for more details.
+
+**Example usage:**
+
+```js
+const Apify = require('apify');
+
+const { utils: { requestAsBrowser } } = Apify;
+
+...
+
+const response = await requestAsBrowser({ url: 'https://www.example.com/' });
+
+const html = response.body;
+const status = response.statusCode;
+const contentType = response.headers['content-type'];
+```
+
+**Parameters**:
+
+- **`options`**: [`RequestAsBrowserOptions`](../typedefs/request-as-browser-options) - All `requestAsBrowser` configuration options.
+
+**Returns**:
+
+[`Promise<RequestAsBrowserResult>`](../typedefs/request-as-browser-result) - This will typically be a
+[Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage), however, if returned from the cache it will be a
+[response-like object](https://github.com/lukechilds/responselike) which behaves in the same way.
+
+---
+
+<a name="isdocker"></a>
+
+## `utils.isDocker(forceReset)`
+
+Returns a `Promise` that resolves to true if the code is running in a Docker container.
+
+**Parameters**:
+
+- **`forceReset`**: `boolean`
+
+**Returns**:
+
+`Promise<boolean>`
+
+---
+
+<a name="sleep"></a>
+
+## `utils.sleep(millis)`
+
+Returns a `Promise` that resolves after a specific period of time. This is useful to implement waiting in your code, e.g. to prevent overloading of
+target website or to avoid bot detection.
+
+**Example usage:**
+
+```
+const Apify = require('apify');
+
+...
+
+// Sleep 1.5 seconds
+await Apify.utils.sleep(1500);
+```
+
+**Parameters**:
+
+- **`millis`**: `number` - Period of time to sleep, in milliseconds. If not a positive number, the returned promise resolves immediately.
+
+**Returns**:
+
+`Promise<void>`
+
+---
+
+<a name="downloadlistofurls"></a>
+
+## `utils.downloadListOfUrls(options)`
+
+Returns a promise that resolves to an array of urls parsed from the resource available at the provided url. Optionally, custom regular expression and
+encoding may be provided.
+
+**Parameters**:
+
+- **`options`**: `object`
+ - **`url`**: `string` - URL to the file
+ - **`[encoding]`**: `string` <code> = &quot;&#x27;utf8&#x27;&quot;</code> - The encoding of the file.
+ - **`[urlRegExp]`**: `RegExp` <code> = URL_NO_COMMAS_REGEX</code> - Custom regular expression to identify the URLs in the file to extract. The
+ regular expression should be case-insensitive and have global flag set (i.e. `/something/gi`).
+
+**Returns**:
+
+`Promise<Array<string>>`
+
+---
+
+<a name="extracturls"></a>
+
+## `utils.extractUrls(options)`
+
+Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.
+
+**Parameters**:
+
+- **`options`**: `object`
+ - **`string`**: `string`
+ - **`[urlRegExp]`**: `RegExp` <code> = Apify.utils.URL_NO_COMMAS_REGEX</code>
+
+**Returns**:
+
+`Array<string>`
+
+---
+
+<a name="htmltotext"></a>
+
+## `utils.htmlToText(html)`
+
+The function converts a HTML document to a plain text.
+
+The plain text generated by the function is similar to a text captured by pressing Ctrl+A and Ctrl+C on a page when loaded in a web browser. The
+function doesn't aspire to preserve the formatting or to be perfectly correct with respect to HTML specifications. However, it attempts to generate
+newlines and whitespaces in and around HTML elements to avoid merging distinct parts of text and thus enable extraction of data from the text (e.g.
+phone numbers).
+
+**Example usage**
+
+```javascript
+const text = htmlToText('<html><body>Some text</body></html>');
+console.log(text);
+```
+
+Note that the function uses [cheerio](https://www.npmjs.com/package/cheerio) to parse the HTML. Optionally, to avoid duplicate parsing of HTML and
+thus improve performance, you can pass an existing Cheerio object to the function instead of the HTML text. The HTML should be parsed with the
+`decodeEntities` option set to `true`. For example:
+
+```javascript
+const cheerio = require('cheerio');
+const html = '<html><body>Some text</body></html>';
+const text = htmlToText(cheerio.load(html, { decodeEntities: true }));
+```
+
+**Parameters**:
+
+- **`html`**: `string` | `cheerio.Root` - HTML text or parsed HTML represented using a [cheerio](https://www.npmjs.com/package/cheerio) function.
+
+**Returns**:
+
+`string` - Plain text
+
+---
diff --git a/website/versioned_docs/version-1.1.2/typedefs/PlaywrightCrawlerOptions.md b/website/versioned_docs/version-1.1.2/typedefs/PlaywrightCrawlerOptions.md
new file mode 100644
index 000000000..9519f51d1
--- /dev/null
+++ b/website/versioned_docs/version-1.1.2/typedefs/PlaywrightCrawlerOptions.md
@@ -0,0 +1,234 @@
+---
+id: version-1.1.2-playwright-crawler-options
+title: PlaywrightCrawlerOptions
+original_id: playwright-crawler-options
+---
+
+<a name="playwrightcrawleroptions"></a>
+
+## Properties
+
+### `handlePageFunction`
+
+**Type**: [`PlaywrightHandlePageFunction`](../typedefs/playwright-handle-page-function)
+
+Function that is called to process each request. It is passed an object with the following fields:
+
+```
+{
+ request: Request,
+ response: Response,
+ page: Page,
+ session: Session,
+ browserController: BrowserController,
+ proxyInfo: ProxyInfo,
+ crawler: PlaywrightCrawler,
+}
+```
+
+`request` is an instance of the [`Request`](../api/request) object with details about the URL to open, HTTP method etc. `page` is an instance of the
+`Playwright` [`Page`](https://playwright.dev/docs/api/class-page) `browserController` is an instance of the
+[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller), `response` is an instance of the `Playwright`
+[`Response`](https://playwright.dev/docs/api/class-response), which is the main resource response as returned by `page.goto(request.url)`. The
+function must return a promise, which is then awaited by the crawler.
+
+If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries
+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your
+function throw exceptions rather than catch them. The exceptions are logged to the request using the
+[`Request.pushErrorMessage()`](../api/request#pusherrormessage) function.
+
+---
+
+### `navigationTimeoutSecs`
+
+**Type**: `number` <code> = 60</code>
+
+Timeout in which page navigation needs to finish, in seconds.
+
+---
+
+### `handleFailedRequestFunction`
+
+**Type**: [`HandleFailedRequest`](../typedefs/handle-failed-request)
+
+A function to handle requests that failed more than `option.maxRequestRetries` times.
+
+The function receives the following object as an argument:
+
+```
+{
+ request: Request,
+ response: Response,
+ page: Page,
+ session: Session,
+ browserController: BrowserController,
+ proxyInfo: ProxyInfo,
+ crawler: PlaywrightCrawler,
+}
+```
+
+Where the [`Request`](../api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during
+processing of the request.
+
+---
+
+### `preNavigationHooks`
+
+**Type**: [`Array<PlaywrightHook>`](../typedefs/playwright-hook)
+
+Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation.
+The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `page.goto()` function the crawler calls to
+navigate. Example:
+
+```
+preNavigationHooks: [
+ async (crawlingContext, gotoOptions) => {
+ await page.evaluate((attr) => { window.foo = attr; }, 'bar');
+ }
+]
+```
+
+---
+
+### `postNavigationHooks`
+
+**Type**: [`Array<PlaywrightHook>`](../typedefs/playwright-hook)
+
+Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts
+`crawlingContext` as an only parameter. Example:
+
+```
+postNavigationHooks: [
+ async (crawlingContext) => {
+ const { page } = crawlingContext;
+ if (hasCaptcha(page)) {
+ await solveCaptcha (page);
+ }
+ };
+]
+```
+
+---
+
+### `launchContext`
+
+**Type**: [`PlaywrightLaunchContext`](../typedefs/playwright-launch-context)
+
+The same options as used by [`Apify.launchPlaywright()`](../api/apify#launchplaywright).
+
+---
+
+### `handlePageTimeoutSecs`
+
+**Type**: `number` <code> = 60</code>
+
+Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+
+---
+
+### `browserPoolOptions`
+
+**Type**: `BrowserPoolOptions`
+
+Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor. You can tweak those to
+fine-tune browser management.
+
+---
+
+### `persistCookiesPerSession`
+
+**Type**: `boolean` <code> = true</code>
+
+Automatically saves cookies to Session. Works only if Session Pool is used.
+
+---
+
+### `proxyConfiguration`
+
+**Type**: [`ProxyConfiguration`](../api/proxy-configuration)
+
+If set, `PlaywrightCrawler` will be configured for all connections to use [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided
+and rotated according to the configuration. For more information, see the [documentation](https://docs.apify.com/proxy).
+
+---
+
+### `requestList`
+
+**Type**: [`RequestList`](../api/request-list)
+
+Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).
+
+---
+
+### `requestQueue`
+
+**Type**: [`RequestQueue`](../api/request-queue)
+
+Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be
+provided (or both).
+
+---
+
+### `maxRequestRetries`
+
+**Type**: `number` <code> = 3</code>
+
+Indicates how many times the request is retried if
+[`PlaywrightCrawlerOptions.handlePageFunction`](../typedefs/playwright-crawler-options#handlepagefunction) fails.
+
+---
+
+### `maxRequestsPerCrawl`
+
+**Type**: `number`
+
+Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite
+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+
+---
+
+### `autoscaledPoolOptions`
+
+**Type**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options)
+
+Custom options passed to the underlying [`AutoscaledPool`](../api/autoscaled-pool) constructor. Note that the `runTaskFunction` and
+`isTaskReadyFunction` options are provided by `BasicCrawler` and cannot be overridden. However, you can provide a custom implementation of
+`isFinishedFunction`.
+
+---
+
+### `minConcurrency`
+
+**Type**: `number` <code> = 1</code>
+
+Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.
+
+_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If
+you're not sure, just keep the default value and the concurrency will scale up automatically.
+
+---
+
+### `maxConcurrency`
+
+**Type**: `number` <code> = 1000</code>
+
+Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.
+
+---
+
+### `useSessionPool`
+
+**Type**: `boolean` <code> = true</code>
+
+Playwright crawler will initialize the [`SessionPool`](../api/session-pool) with the corresponding `sessionPoolOptions`. The session instance will be
+than available in the `handleRequestFunction`.
+
+---
+
+### `sessionPoolOptions`
+
+**Type**: [`SessionPoolOptions`](../typedefs/session-pool-options)
+
+The configuration options for [`SessionPool`](../api/session-pool) to use.
+
+---
diff --git a/website/versioned_docs/version-1.1.2/typedefs/PuppeteerCrawlerOptions.md b/website/versioned_docs/version-1.1.2/typedefs/PuppeteerCrawlerOptions.md
new file mode 100644
index 000000000..4e640e61e
--- /dev/null
+++ b/website/versioned_docs/version-1.1.2/typedefs/PuppeteerCrawlerOptions.md
@@ -0,0 +1,236 @@
+---
+id: version-1.1.2-puppeteer-crawler-options
+title: PuppeteerCrawlerOptions
+original_id: puppeteer-crawler-options
+---
+
+<a name="puppeteercrawleroptions"></a>
+
+## Properties
+
+### `handlePageFunction`
+
+**Type**: [`PuppeteerHandlePage`](../typedefs/puppeteer-handle-page)
+
+Function that is called to process each request. It is passed an object with the following fields:
+
+```
+{
+ request: Request,
+ response: Response,
+ page: Page,
+ session: Session,
+ browserController: BrowserController,
+ proxyInfo: ProxyInfo,
+ crawler: PuppeteerCrawler,
+}
+```
+
+`request` is an instance of the [`Request`](../api/request) object with details about the URL to open, HTTP method etc. `page` is an instance of the
+`Puppeteer` [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) `browserPool` is an instance of the
+[`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool), `browserController` is an instance of the
+[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller), `response` is an instance of the `Puppeteer`
+[`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response), which is the main resource response as returned by
+`page.goto(request.url)`. The function must return a promise, which is then awaited by the crawler.
+
+If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries
+fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your
+function throw exceptions rather than catch them. The exceptions are logged to the request using the
+[`Request.pushErrorMessage()`](../api/request#pusherrormessage) function.
+
+---
+
+### `navigationTimeoutSecs`
+
+**Type**: `number` <code> = 60</code>
+
+Timeout in which page navigation needs to finish, in seconds.
+
+---
+
+### `handleFailedRequestFunction`
+
+**Type**: [`HandleFailedRequest`](../typedefs/handle-failed-request)
+
+A function to handle requests that failed more than `option.maxRequestRetries` times.
+
+The function receives the following object as an argument:
+
+```
+{
+ request: Request,
+ response: Response,
+ page: Page,
+ session: Session,
+ browserController: BrowserController,
+ proxyInfo: ProxyInfo,
+ crawler: PuppeteerCrawler,
+}
+```
+
+Where the [`Request`](../api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during
+processing of the request.
+
+---
+
+### `launchContext`
+
+**Type**: [`PuppeteerLaunchContext`](../typedefs/puppeteer-launch-context)
+
+Options used by [`Apify.launchPuppeteer()`](../api/apify#launchpuppeteer) to start new Puppeteer instances.
+
+---
+
+### `handlePageTimeoutSecs`
+
+**Type**: `number` <code> = 60</code>
+
+Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+
+---
+
+### `browserPoolOptions`
+
+**Type**: `BrowserPoolOptions`
+
+Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor. You can tweak those to
+fine-tune browser management.
+
+---
+
+### `persistCookiesPerSession`
+
+**Type**: `boolean` <code> = true</code>
+
+Automatically saves cookies to Session. Works only if Session Pool is used.
+
+---
+
+### `proxyConfiguration`
+
+**Type**: [`ProxyConfiguration`](../api/proxy-configuration)
+
+If set, `PuppeteerCrawler` will be configured for all connections to use [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and
+rotated according to the configuration. For more information, see the [documentation](https://docs.apify.com/proxy).
+
+---
+
+### `preNavigationHooks`
+
+**Type**: [`Array<PuppeteerHook>`](../typedefs/puppeteer-hook)
+
+Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation.
+The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `page.goto()` function the crawler calls to
+navigate. Example:
+
+```
+preNavigationHooks: [
+ async (crawlingContext, gotoOptions) => {
+ const { page } = crawlingContext;
+ await page.evaluate((attr) => { window.foo = attr; }, 'bar');
+ }
+]
+```
+
+---
+
+### `postNavigationHooks`
+
+**Type**: [`Array<PuppeteerHook>`](../typedefs/puppeteer-hook)
+
+Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts
+`crawlingContext` as an only parameter. Example:
+
+```
+postNavigationHooks: [
+ async (crawlingContext) => {
+ const { page } = crawlingContext;
+ if (hasCaptcha(page)) {
+ await solveCaptcha (page);
+ }
+ };
+]
+```
+
+---
+
+### `requestList`
+
+**Type**: [`RequestList`](../api/request-list)
+
+Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).
+
+---
+
+### `requestQueue`
+
+**Type**: [`RequestQueue`](../api/request-queue)
+
+Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be
+provided (or both).
+
+---
+
+### `maxRequestRetries`
+
+**Type**: `number` <code> = 3</code>
+
+Indicates how many times the request is retried if
+[`PuppeteerCrawlerOptions.handlePageFunction`](../typedefs/puppeteer-crawler-options#handlepagefunction) fails.
+
+---
+
+### `maxRequestsPerCrawl`
+
+**Type**: `number`
+
+Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite
+loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+
+---
+
+### `autoscaledPoolOptions`
+
+**Type**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options)
+
+Custom options passed to the underlying [`AutoscaledPool`](../api/autoscaled-pool) constructor. Note that the `runTaskFunction` and
+`isTaskReadyFunction` options are provided by the crawler and cannot be overridden. However, you can provide a custom implementation of
+`isFinishedFunction`.
+
+---
+
+### `minConcurrency`
+
+**Type**: `number` <code> = 1</code>
+
+Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.
+
+_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If
+you're not sure, just keep the default value and the concurrency will scale up automatically.
+
+---
+
+### `maxConcurrency`
+
+**Type**: `number` <code> = 1000</code>
+
+Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.
+
+---
+
+### `useSessionPool`
+
+**Type**: `boolean` <code> = true</code>
+
+Puppeteer crawler will initialize the [`SessionPool`](../api/session-pool) with the corresponding `sessionPoolOptions`. The session instance will be
+than available in the `handleRequestFunction`.
+
+---
+
+### `sessionPoolOptions`
+
+**Type**: [`SessionPoolOptions`](../typedefs/session-pool-options)
+
+The configuration options for [`SessionPool`](../api/session-pool) to use.
+
+---