Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

launchPuppeteer() uses more intelligent default User-Agent (#52)

Jan Čurn119df61b0a31
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dc0a6c103..f489449b1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,7 @@
+xxxxxx / xxxxxxxxxxx
+- `launchPuppeteer()` changes `User-Agent` only when in headless mode or if not using full Google Chrome,
+ to reduce chance of detection of the crawler
+
0.5.17 / 2018-03-27
===================
- Added `useChrome` flag to `launchPuppeteer()` function
diff --git a/src/puppeteer.js b/src/puppeteer.js
index 06175d642..72dee2133 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -37,11 +37,13 @@ import { newPromise, getTypicalChromeExecutablePath } from './utils';
* @param {String} [opts.proxyUrl] URL to a HTTP proxy server.
* Port number must be specified. Proxy username and password might also be provided.
* For example, `http://bob:[email protected]:1234`.
- * @param {String} [opts.userAgent] Default User-Agent for the browser.
- * If not provided, the function sets it to a reasonable default.
- * @param {String} [opts.useChrome=false] If true-ish value and `opts.executablePath` is not set,
- * Puppeteer will launch full Chrome available on the machine rather than the bundled Chromium.
- * The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
+ * @param {String} [opts.userAgent] HTTP `User-Agent` header used by the browser.
+ * If not provided, the function sets `User-Agent` to a reasonable default
+ * to reduce the chance of detection of the crawler.
+ * @param {String} [opts.useChrome=false] If `true` and `opts.executablePath` is not set,
+ * Puppeteer will launch full Google Chrome browser available on the machine
+ * rather than the bundled Chromium. The path to Chrome executable
+ * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
* or defaults to the typical Google Chrome executable location specific for the operating system.
* By default, this option is `false`.
* @returns {Promise} Promise object that resolves to Puppeteer's `Browser` instance.
@@ -70,7 +72,6 @@ export const launchPuppeteer = (opts) => {
opts.args = opts.args || [];
opts.args.push('--no-sandbox');
- opts.args.push(`--user-agent=${opts.userAgent || DEFAULT_USER_AGENT}`);
if (opts.headless === undefined || opts.headless === null) {
opts.headless = process.env[ENV_VARS.HEADLESS] === '1' && process.env[ENV_VARS.XVFB] !== '1';
}
@@ -78,6 +79,16 @@ export const launchPuppeteer = (opts) => {
opts.executablePath = process.env[ENV_VARS.CHROME_EXECUTABLE_PATH] || getTypicalChromeExecutablePath();
}
+ // When User-Agent is not set and we're using Chromium or headless mode,
+ // it is better to use DEFAULT_USER_AGENT to reduce chance of detection
+ let { userAgent } = opts;
+ if (!userAgent && (!opts.executablePath || opts.headless)) {
+ userAgent = DEFAULT_USER_AGENT;
+ }
+ if (userAgent) {
+ opts.args.push(`--user-agent=${userAgent}`);
+ }
+
let anonymizedProxyUrl;
let promise;