Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat: add `forceUrlEncoding` option to `requestAsBrowser` (#1086)

Martin Adámek442a7ac7dbd5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e0c86f045..1a231c604 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,29 @@
1.3.0 / BETA
====================
+
+## Navigation hooks in `CheerioCrawler`
+
+`CheerioCrawler` downloads the web pages using the `requestAsBrowser` utility function.
+As opposed to the browser based crawlers that are automatically encoding the URLs, the
+`requestAsBrowser` function will not do so. We either need to manually encode the URLs
+via `encodeURI()` function, or set `forceUrlEncoding: true` in the `requestAsBrowserOptions`,
+which will automatically encode all the URLs before accessing them.
+
+> We can either use `forceUrlEncoding` or encode manually, but not both - it would
+> result in double encoding and therefore lead to invalid URLs.
+
+We can use the `preNavigationHooks` to adjust `requestAsBrowserOptions`:
+
+```
+preNavigationHooks: [
+ (crawlingContext, requestAsBrowserOptions) => {
+ requestAsBrowserOptions.forceUrlEncoding = true;
+ }
+]
+```
+
+## `Apify` class and `Configuration`
+
Adds two new named exports:
- `Configuration` class that serves as the main configuration holder, replacing explicit usage of
@@ -48,6 +72,9 @@ await dataset.pushData({ myValue: 123 });
- Update `puppeteer` and `playwright` to match stable Chrome (90).
- Fix support for building TypeScript projects that depend on the SDK.
- add `taskTimeoutSecs` to allow control over timeout of `AutoscaledPool` tasks
+- add `forceUrlEncoding` to `requestAsBrowser` options
+- add `preNavigationHooks` and `postNavigationHooks` to `CheerioCrawler`
+- deprecated `prepareRequestFunction` and `postResponseFunction` methods of `CheerioCrawler`
- Added new event `aborting` for handling gracefully aborted run from Apify platform.
1.2.1 / 2021/05/14
diff --git a/jest.config.js b/jest.config.js
index d5f8a6379..12573a060 100644
--- a/jest.config.js
+++ b/jest.config.js
@@ -5,7 +5,7 @@ module.exports = {
testRunner: 'jest-circus/runner',
verbose: false,
rootDir: path.join(__dirname, './'),
- testTimeout: 30e3,
+ testTimeout: 60e3,
maxWorkers: 3,
testMatch: ['**/?(*.)+(spec|test).[tj]s?(x)'],
transform: {
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index 6f272d14e..0a6205761 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -634,6 +634,21 @@ export class BasicCrawler {
}
}
+ /**
+ * @param {Array<Hook>} hooks
+ * @param {*} args
+ * @ignore
+ * @protected
+ * @internal
+ */
+ async _executeHooks(hooks, ...args) {
+ if (Array.isArray(hooks) && hooks.length) {
+ for (const hook of hooks) {
+ await hook(...args);
+ }
+ }
+ }
+
/**
* Function for cleaning up after all request are processed.
* @ignore
diff --git a/src/crawlers/browser_crawler.js b/src/crawlers/browser_crawler.js
index d6f496f6f..30db8d2b6 100644
--- a/src/crawlers/browser_crawler.js
+++ b/src/crawlers/browser_crawler.js
@@ -130,7 +130,7 @@ import { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool';
* ```
* @property {Array<Hook>} [postNavigationHooks]
* Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
- * The function accepts `crawlingContext` as an only parameter.
+ * The function accepts `crawlingContext` as the only parameter.
* Example:
* ```
* postNavigationHooks: [
@@ -535,21 +535,6 @@ export default class BrowserCrawler extends BasicCrawler {
}
}
- /**
- * @param {Array<Hook>} hooks
- * @param {*} args
- * @ignore
- * @protected
- * @internal
- */
- async _executeHooks(hooks, ...args) {
- if (Array.isArray(hooks) && hooks.length) {
- for (const hook of hooks) {
- await hook(...args);
- }
- }
- }
-
/**
* Function for cleaning up after all request are processed.
* @ignore
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 39abef271..5384209b8 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -112,6 +112,8 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
* Either `requestList` or `requestQueue` option must be provided (or both).
* @property {PrepareRequest} [prepareRequestFunction]
+ * > This option is deprecated, use `preNavigationHooks` instead.
+ *
* A function that executes before the HTTP request is made to the target resource.
* This function is suitable for setting dynamic properties such as cookies to the {@link Request}.
*
@@ -133,6 +135,8 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* different parts of SDK would have access to a different {@link Request} instance.
*
* @property {PostResponse} [postResponseFunction]
+ * > This option is deprecated, use `postNavigationHooks` instead.
+ *
* A function that executes right after the HTTP request is made to the target resource and response is returned.
* This function is suitable for overriding custom properties of response e.g. setting headers because of response parsing.
*
@@ -159,6 +163,7 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* }
* ```
* The response is an instance of Node's http.IncomingMessage object.
+ *
* @property {number} [handlePageTimeoutSecs=60]
* Timeout in which the function passed as `handlePageFunction` needs to finish, given in seconds.
* @property {number} [requestTimeoutSecs=30]
@@ -171,7 +176,6 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* For more information, see the [documentation](https://docs.apify.com/proxy).
* @property {HandleFailedRequest} [handleFailedRequestFunction]
* A function to handle requests that failed more than `option.maxRequestRetries` times.
- *
* The function receives the following object as an argument:
* ```
* {
@@ -192,6 +196,29 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
*
* See [source code](https://github.com/apify/apify-js/blob/master/src/crawlers/cheerio_crawler.js#L13)
* for the default implementation of this function.
+ * @property {Array<Hook>} [preNavigationHooks]
+ * Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
+ * or browser properties before navigation. The function accepts two parameters, `crawlingContext` and `requestAsBrowserOptions`,
+ * which are passed to the `requestAsBrowser()` function the crawler calls to navigate.
+ * Example:
+ * ```
+ * preNavigationHooks: [
+ * async (crawlingContext, requestAsBrowserOptions) => {
+ * requestAsBrowserOptions.forceUrlEncoding = true;
+ * }
+ * ]
+ * ```
+ * @property {Array<Hook>} [postNavigationHooks]
+ * Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
+ * The function accepts `crawlingContext` as the only parameter.
+ * Example:
+ * ```
+ * postNavigationHooks: [
+ * async (crawlingContext) => {
+ * // ...
+ * };
+ * ]
+ * ```
* @property {string[]} [additionalMimeTypes]
* An array of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types"
* target="_blank">MIME types</a> you want the crawler to load and process.
@@ -271,7 +298,24 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
*
* The crawler finishes when there are no more {@link Request} objects to crawl.
*
- * `CheerioCrawler` downloads the web pages using the {@link utils#requestAsBrowser} utility function.
+ * `CheerioCrawler` downloads the web pages using the `{@link utils#requestAsBrowser}` utility function.
+ * As opposed to the browser based crawlers that are automatically encoding the URLs, the
+ * `{@link utils#requestAsBrowser}` function will not do so. We either need to manually encode the URLs
+ * via `encodeURI()` function, or set `forceUrlEncoding: true` in the `requestAsBrowserOptions`,
+ * which will automatically encode all the URLs before accessing them.
+ *
+ * > We can either use `forceUrlEncoding` or encode manually, but not both - it would
+ * > result in double encoding and therefore lead to invalid URLs.
+ *
+ * We can use the `preNavigationHooks` to adjust `requestAsBrowserOptions`:
+ *
+ * ```
+ * preNavigationHooks: [
+ * (crawlingContext, requestAsBrowserOptions) => {
+ * requestAsBrowserOptions.forceUrlEncoding = true;
+ * }
+ * ]
+ * ```
*
* By default, `CheerioCrawler` only processes web pages with the `text/html`
* and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),
@@ -358,6 +402,9 @@ class CheerioCrawler extends BasicCrawler {
prepareRequestFunction: ow.optional.function,
postResponseFunction: ow.optional.function,
persistCookiesPerSession: ow.optional.boolean,
+
+ preNavigationHooks: ow.optional.array,
+ postNavigationHooks: ow.optional.array,
}
/**
@@ -379,6 +426,8 @@ class CheerioCrawler extends BasicCrawler {
prepareRequestFunction,
postResponseFunction,
persistCookiesPerSession,
+ preNavigationHooks = [],
+ postNavigationHooks = [],
// BasicCrawler
autoscaledPoolOptions = CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS,
@@ -415,6 +464,12 @@ class CheerioCrawler extends BasicCrawler {
this.prepareRequestFunction = prepareRequestFunction;
this.postResponseFunction = postResponseFunction;
this.proxyConfiguration = proxyConfiguration;
+ /** @type {Array<Hook>} */
+ this.preNavigationHooks = preNavigationHooks;
+ /** @type {Array<Hook>} */
+ this.postNavigationHooks = postNavigationHooks;
+ /** @type {RequestAsBrowserOptions} */
+ this.defaultRequestAsBrowserOptions = {};
if (this.useSessionPool) {
this.persistCookiesPerSession = persistCookiesPerSession !== undefined ? persistCookiesPerSession : true;
@@ -475,17 +530,7 @@ class CheerioCrawler extends BasicCrawler {
crawlingContext.proxyInfo = this.proxyConfiguration.newProxyInfo(sessionId);
}
- if (this.prepareRequestFunction) await this.prepareRequestFunction(crawlingContext);
-
- const proxyUrl = crawlingContext.proxyInfo && crawlingContext.proxyInfo.url;
-
- crawlingContext.response = await addTimeoutToPromise(
- this._requestFunction({ request, session, proxyUrl }),
- this.requestTimeoutMillis,
- `request timed out after ${this.requestTimeoutMillis / 1000} seconds.`,
- );
-
- if (this.postResponseFunction) await this.postResponseFunction(crawlingContext);
+ await this._handleNavigation(crawlingContext);
const { dom, isXml, body, contentType, response } = await this._parseResponse(request, crawlingContext.response);
@@ -530,6 +575,37 @@ class CheerioCrawler extends BasicCrawler {
);
}
+ /**
+ * @param {CrawlingContext} crawlingContext
+ * @ignore
+ * @protected
+ * @internal
+ */
+ async _handleNavigation(crawlingContext) {
+ if (this.prepareRequestFunction) {
+ this.log.deprecated('Option "prepareRequestFunction" is deprecated. Use "preNavigationHooks" instead.');
+ await this.prepareRequestFunction(crawlingContext);
+ }
+
+ const requestAsBrowserOptions = { ...this.defaultRequestAsBrowserOptions };
+ await this._executeHooks(this.preNavigationHooks, crawlingContext, requestAsBrowserOptions);
+ const { request, session } = crawlingContext;
+ const proxyUrl = crawlingContext.proxyInfo && crawlingContext.proxyInfo.url;
+
+ crawlingContext.response = await addTimeoutToPromise(
+ this._requestFunction({ request, session, proxyUrl, requestAsBrowserOptions }),
+ this.requestTimeoutMillis,
+ `request timed out after ${this.requestTimeoutMillis / 1000} seconds.`,
+ );
+
+ await this._executeHooks(this.postNavigationHooks, crawlingContext, requestAsBrowserOptions);
+
+ if (this.postResponseFunction) {
+ this.log.deprecated('Option "postResponseFunction" is deprecated. Use "postNavigationHooks" instead.');
+ await this.postResponseFunction(crawlingContext);
+ }
+ }
+
/**
* Function to make the HTTP request. It performs optimizations
* on the request such as only downloading the request body if the
@@ -539,21 +615,22 @@ class CheerioCrawler extends BasicCrawler {
* @param {Request} options.request
* @param {Session} options.session
* @param {string} options.proxyUrl
+ * @param {RequestAsBrowserOptions} options.requestAsBrowserOptions
* @returns {Promise<IncomingMessage|Readable>}
* @ignore
* @protected
* @internal
*/
- async _requestFunction({ request, session, proxyUrl }) {
+ async _requestFunction({ request, session, proxyUrl, requestAsBrowserOptions }) {
// Using the streaming API of Request to be able to
- // handle the response based on headers receieved.
+ // handle the response based on headers received.
if (this.useSessionPool) {
const { headers } = request;
headers.Cookie = session.getCookieString(request.url);
}
- const opts = this._getRequestOptions(request, session, proxyUrl);
+ const opts = this._getRequestOptions(request, session, proxyUrl, requestAsBrowserOptions);
let responseWithStream;
try {
@@ -612,19 +689,22 @@ class CheerioCrawler extends BasicCrawler {
* @param {Request} request
* @param {Session} [session]
* @param {string} [proxyUrl]
+ * @param {RequestAsBrowserOptions} [requestAsBrowserOptions]
* @ignore
* @protected
* @internal
*/
- _getRequestOptions(request, session, proxyUrl) {
+ _getRequestOptions(request, session, proxyUrl, requestAsBrowserOptions) {
const mandatoryRequestOptions = {
url: request.url,
method: request.method,
- headers: { ...request.headers },
+ headers: { ...request.headers, ...requestAsBrowserOptions.headers },
ignoreSslErrors: this.ignoreSslErrors,
proxyUrl,
- stream: true,
useCaseSensitiveHeaders: true,
+ timeoutSecs: this.requestTimeoutMillis / 1000,
+ ...requestAsBrowserOptions,
+ stream: true,
abortFunction: (res) => {
const { statusCode } = res;
const { type } = parseContentTypeFromResponse(res);
@@ -642,7 +722,6 @@ class CheerioCrawler extends BasicCrawler {
return false;
},
- timeoutSecs: this.requestTimeoutMillis / 1000,
};
if (/PATCH|POST|PUT/.test(request.method)) mandatoryRequestOptions.payload = request.payload;
diff --git a/src/crawlers/playwright_crawler.js b/src/crawlers/playwright_crawler.js
index 6a9c37581..09bf87b3a 100644
--- a/src/crawlers/playwright_crawler.js
+++ b/src/crawlers/playwright_crawler.js
@@ -117,7 +117,7 @@ import { gotoExtended } from '../playwright_utils';
* ```
* @property {Array<PlaywrightHook>} [postNavigationHooks]
* Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
- * The function accepts `crawlingContext` as an only parameter.
+ * The function accepts `crawlingContext` as the only parameter.
* Example:
* ```
* postNavigationHooks: [
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 48b692d3b..a08990458 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -118,7 +118,7 @@ import { BrowserPoolOptions } from 'browser-pool';
* ```
* @property {Array<PuppeteerHook>} [postNavigationHooks]
* Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
- * The function accepts `crawlingContext` as an only parameter.
+ * The function accepts `crawlingContext` as the only parameter.
* Example:
* ```
* postNavigationHooks: [
diff --git a/src/utils_request.js b/src/utils_request.js
index 8af6121fe..978954855 100644
--- a/src/utils_request.js
+++ b/src/utils_request.js
@@ -49,6 +49,8 @@ import { IncomingMessage } from 'http';
* expect HTTP2 connections, because browsers use HTTP2 by default. It will automatically downgrade
* to HTTP/1.1 for websites that do not support HTTP2. For Node 10 this option is always set to `false`
* because Node 10 does not support HTTP2 very well. Upgrade to Node 12 for better performance.
+ * @property {boolean} [forceUrlEncoding]
+ * Automatically encode URLs via `encodeURI()` before resolving them.
*/
/**
@@ -119,6 +121,7 @@ export const requestAsBrowser = async (options = {}) => {
headerGeneratorOptions: ow.optional.object,
stream: ow.optional.boolean,
decodeBody: ow.optional.boolean,
+ forceUrlEncoding: ow.optional.boolean,
}));
ow(options, 'RequestAsBrowserOptions', ow.object.validate((opts) => ({
@@ -146,6 +149,7 @@ export const requestAsBrowser = async (options = {}) => {
throwOnHttpErrors = false,
stream = false,
decodeBody = true,
+ forceUrlEncoding,
...gotParams
} = options;
@@ -180,7 +184,13 @@ export const requestAsBrowser = async (options = {}) => {
};
}
- if (!gotScrapingOptions.isStream) return gotScraping(gotScrapingOptions);
+ if (forceUrlEncoding) {
+ gotScrapingOptions.url = encodeURI(gotScrapingOptions.url);
+ }
+
+ if (!gotScrapingOptions.isStream) {
+ return gotScraping(gotScrapingOptions);
+ }
// abortFunction must be handled separately for streams :(
const duplexStream = await gotScraping(gotScrapingOptions);
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 4e92ecc1c..22f730f84 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -190,6 +190,45 @@ describe('CheerioCrawler', () => {
expect(success.url).toEqual(MODIFIED_URL);
});
+ test('should work with not encoded urls', async () => {
+ const sources = [
+ { url: `http://${HOST}:${port}/mirror?q=abc` },
+ { url: `http://${HOST}:${port}/mirror?q=%` },
+ { url: `http://${HOST}:${port}/mirror?q=%cf` },
+ ];
+ const requestList = new Apify.RequestList({ sources });
+ await requestList.initialize();
+ const processed = [];
+ const failed = [];
+ const handlePageFunction = async ({ $, body, request }) => {
+ request.userData.title = $('title').text();
+ request.userData.body = body;
+ processed.push(request);
+ };
+
+ const cheerioCrawler = new Apify.CheerioCrawler({
+ requestList,
+ minConcurrency: 2,
+ maxConcurrency: 2,
+ handlePageFunction,
+ handleFailedRequestFunction: ({ request }) => failed.push(request),
+ preNavigationHooks: [
+ (_, gotoOptions) => {
+ gotoOptions.forceUrlEncoding = true;
+ },
+ ],
+ });
+
+ await cheerioCrawler.run();
+
+ expect(processed).toHaveLength(3);
+ expect(failed).toHaveLength(0);
+
+ expect(processed[0].loadedUrl).toBe(`http://${HOST}:${port}/mirror?q=abc`);
+ expect(processed[1].loadedUrl).toBe(`http://${HOST}:${port}/mirror?q=%25`);
+ expect(processed[2].loadedUrl).toBe(`http://${HOST}:${port}/mirror?q=%25cf`);
+ });
+
test('postResponseFunction should work', async () => {
const sources = ['http://example.com/'];
const requestList = await Apify.openRequestList(null, sources.slice());
diff --git a/test/utils_request.test.js b/test/utils_request.test.js
index 79ad67a9d..9bf43e62b 100644
--- a/test/utils_request.test.js
+++ b/test/utils_request.test.js
@@ -1,7 +1,5 @@
import express from 'express';
-import {
- requestAsBrowser,
-} from '../build/utils_request';
+import { requestAsBrowser } from '../build/utils_request';
import { startExpressAppPromise } from './_helper';
const CONTENT = 'CONTENT';
@@ -43,6 +41,10 @@ describe('Apify.utils_request', () => {
res.send(CONTENT);
});
+ app.get('/test-encoding', (req, res) => {
+ res.json(req.query);
+ });
+
app.get('/invalidBody', async (req, res) => {
res.setHeader('content-encoding', 'deflate');
res.status(500);
@@ -91,18 +93,50 @@ describe('Apify.utils_request', () => {
});
describe('Apify.requestAsBrowser', () => {
- test(
- 'it uses mobile user-agent when mobile property is set to true ',
- async () => {
- const data = {
- url: `http://${HOSTNAME}:${port}/echo`,
- useMobileVersion: true,
- };
- const response = await requestAsBrowser(data);
- expect(response.statusCode).toBe(200);
- expect(response.request.options.context.headerGeneratorOptions.devices).toEqual(['mobile']);
- },
- );
+ test('it uses mobile user-agent when mobile property is set to true', async () => {
+ const data = {
+ url: `http://${HOSTNAME}:${port}/echo`,
+ useMobileVersion: true,
+ };
+ const response = await requestAsBrowser(data);
+ expect(response.statusCode).toBe(200);
+ expect(response.request.options.context.headerGeneratorOptions.devices).toEqual(['mobile']);
+ });
+
+ const testQueries = [
+ ['abc', 'abc', true],
+ ['abc', 'abc', false],
+ ['%20', ' ', false],
+ ['%cf', '%cf', true],
+ ['helios-–-the-primordial-sun', 'helios-–-the-primordial-sun', true],
+ ['helios-%E2%80%93-the-primordial-sun', 'helios-–-the-primordial-sun', false],
+ ];
+
+ test.each(testQueries)(`it works with not encoded urls: '%s' (regular)`, async (query, decoded, forceEncode) => {
+ const response = await requestAsBrowser({
+ url: `http://${HOSTNAME}:${port}/test-encoding/?q=${query}`,
+ forceUrlEncoding: forceEncode,
+ });
+ expect(response.statusCode).toBe(200);
+ expect(JSON.parse(response.body).q).toBe(decoded);
+ });
+
+ test.each(testQueries)(`it works with not encoded urls: '%s' (stream)`, async (query, decoded, forceEncode) => {
+ const response = await requestAsBrowser({
+ url: `http://${HOSTNAME}:${port}/test-encoding/?q=${query}`,
+ stream: true,
+ forceUrlEncoding: forceEncode,
+ });
+
+ const chunks = [];
+ for await (const chunk of response) {
+ chunks.push(chunk);
+ }
+ const body = JSON.parse(chunks.join());
+
+ expect(response.statusCode).toBe(200);
+ expect(body.q).toBe(decoded);
+ });
test('uses desktop user-agent by default ', async () => {
const data = {