Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Make the usage smoother and improve test performance by adding static html.

Petr Patek4e8c2ddd0063
diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.md
index dc6b92a42..829ec469d 100644
--- a/docs/typedefs/LaunchPuppeteerOptions.md
+++ b/docs/typedefs/LaunchPuppeteerOptions.md
@@ -79,10 +79,16 @@ function and in addition, all the options available below.
<td colspan="3"><p>Require path to a module to be used instead of default <code>puppeteer</code>. This enables usage
of various Puppeteer wrappers such as <code>puppeteer-extra</code>.</p>
</td></tr><tr>
-<td><code>[stealthOptions]</code></td><td><code><a href="../typedefs/stealthoptions">StealthOptions</a></code></td><td></td>
+<td><code>[stealth]</code></td><td><code>boolean</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>Using this configuration, you can make a headless chrome nearly undetectable.
+<td colspan="3"><p>This setting hides most of the know properties that identifies the headless Chrome and makes it nearly undetectable.
It is recommended to use it together with the <code>useChrome</code> set to <code>true</code>.</p>
+</td></tr><tr>
+<td><code>[stealthOptions]</code></td><td><code><a href="../typedefs/stealthoptions">StealthOptions</a></code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Using this configuration, you can disable some of the hiding tricks.
+ For these settings to take effect <code>stealth</code> must be set to true</p>
</td></tr></tbody>
</table>
diff --git a/docs/typedefs/stealthOptions.md b/docs/typedefs/stealthOptions.md
index 3a0e6480e..071ce698c 100644
--- a/docs/typedefs/stealthOptions.md
+++ b/docs/typedefs/stealthOptions.md
@@ -4,7 +4,8 @@ title: StealthOptions
---
<a name="StealthOptions"></a>
-Configuration of stealth tricks for a proper hiding effect all of them should be set to true
+Configuration of stealth tricks for a proper hiding effect all of them should be set to true.
+These tricks are applied only when the `stealth` option is set to `true`.
**Properties**
<table>
@@ -15,54 +16,54 @@ Configuration of stealth tricks for a proper hiding effect all of them should be
</thead>
<tbody>
<tr>
-<td><code>[addPlugins]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[addPlugins]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>if plugins should be added to the navigator</p>
+<td colspan="3"><p>If plugins should be added to the navigator.</p>
</td></tr><tr>
-<td><code>[emulateWindowFrame]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[emulateWindowFrame]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Emulates window Iframe</p>
+<td colspan="3"><p>Emulates window Iframe.</p>
</td></tr><tr>
-<td><code>[emulateWebGL]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[emulateWebGL]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Emulates graphic card</p>
+<td colspan="3"><p>Emulates graphic card.</p>
</td></tr><tr>
-<td><code>[emulateConsoleDebug]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[emulateConsoleDebug]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Emulates console.debug to return null</p>
+<td colspan="3"><p>Emulates console.debug to return null.</p>
</td></tr><tr>
-<td><code>[addLanguage]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[addLanguage]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Adds languages to the navigator</p>
+<td colspan="3"><p>Adds languages to the navigator.</p>
</td></tr><tr>
-<td><code>[hideWebDriver]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[hideWebDriver]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Hides the webdriver by changing the navigator proto</p>
+<td colspan="3"><p>Hides the webdriver by changing the navigator proto.</p>
</td></tr><tr>
-<td><code>[hackPermissions]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[hackPermissions]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Fakes interaction with permissions</p>
+<td colspan="3"><p>Fakes interaction with permissions.</p>
</td></tr><tr>
-<td><code>[mockChrome]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[mockChrome]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Adds the chrome runtime properties</p>
+<td colspan="3"><p>Adds the chrome runtime properties.</p>
</td></tr><tr>
-<td><code>[mocksChromeInIframe]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[mocksChromeInIframe]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Adds the chrome runtime properties inside the every newly created iframe</p>
+<td colspan="3"><p>Adds the chrome runtime properties inside the every newly created iframe.</p>
</td></tr><tr>
-<td><code>[mockDeviceMemory]</code></td><td><code>boolean</code></td><td><code>false</code></td>
+<td><code>[mockDeviceMemory]</code></td><td><code>boolean</code></td><td><code>true</code></td>
</tr>
<tr>
-<td colspan="3"><p>Sets device memory to other value than 0</p>
+<td colspan="3"><p>Sets device memory to other value than 0.</p>
</td></tr></tbody>
</table>
diff --git a/src/puppeteer.js b/src/puppeteer.js
index 923853860..e51f213ec 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -65,9 +65,12 @@ const LAUNCH_PUPPETEER_DEFAULT_VIEWPORT = {
* @property {String} [puppeteerModule]
* Require path to a module to be used instead of default `puppeteer`. This enables usage
* of various Puppeteer wrappers such as `puppeteer-extra`.
- * @property {StealthOptions} [stealthOptions]
- * Using this configuration, you can make a headless chrome nearly undetectable.
+ * @property {boolean} [stealth]
+ * This setting hides most of the know properties that identifies the headless Chrome and makes it nearly undetectable.
* It is recommended to use it together with the `useChrome` set to `true`.
+ * @property {StealthOptions} [stealthOptions]
+ * Using this configuration, you can disable some of the hiding tricks.
+ * For these settings to take effect `stealth` must be set to true
*/
/**
@@ -193,7 +196,8 @@ export const launchPuppeteer = async (options = {}) => {
checkParamOrThrow(options.proxyUrl, 'options.proxyUrl', 'Maybe String');
checkParamOrThrow(options.useApifyProxy, 'options.useApifyProxy', 'Maybe Boolean');
checkParamOrThrow(options.puppeteerModule, 'options.puppeteerModule', 'Maybe String');
- checkParamOrThrow(options.stealth, 'options.stealth', 'Maybe Object');
+ checkParamOrThrow(options.stealth, 'options.stealth', 'Maybe Boolean');
+ checkParamOrThrow(options.stealthOptions, 'options.stealthOptions', 'Maybe Object');
if (options.useApifyProxy && options.proxyUrl) throw new Error('Cannot combine "options.useApifyProxy" with "options.proxyUrl"!');
if (options.liveView || options.liveViewOptions) {
log.deprecated('Live view is no longer available in Apify.launchPuppeteer() and launchPuppeteerOptions. '
@@ -245,7 +249,7 @@ export const launchPuppeteer = async (options = {}) => {
}
// Add stealth
- if (optsCopy.stealthOptions) {
+ if (optsCopy.stealth) {
browser = applyStealthFromBrowser(browser, optsCopy.stealthOptions);
}
log.info('Launching Puppeteer', _.omit(optsCopy, LAUNCH_PUPPETEER_LOG_OMIT_OPTS));
diff --git a/src/stealth/stealth.js b/src/stealth/stealth.js
index 55395115b..0890a1fe8 100644
--- a/src/stealth/stealth.js
+++ b/src/stealth/stealth.js
@@ -2,31 +2,32 @@ import _ from 'underscore';
import hidingTricks from './hiding_tricks';
/**
- * Configuration of stealth tricks for a proper hiding effect all of them should be set to true
+ * Configuration of stealth tricks for a proper hiding effect all of them should be set to true.
+ * These tricks are applied only when the `stealth` option is set to `true`.
* @typedef {Object} StealthOptions
- * @property {boolean} [addPlugins=false] - If plugins should be added to the navigator.
- * @property {boolean} [emulateWindowFrame=false] - Emulates window Iframe.
- * @property {boolean} [emulateWebGL=false] - Emulates graphic card.
- * @property {boolean} [emulateConsoleDebug=false] - Emulates console.debug to return null.
- * @property {boolean} [addLanguage=false] - Adds languages to the navigator.
- * @property {boolean} [hideWebDriver=false] - Hides the webdriver by changing the navigator proto.
- * @property {boolean} [hackPermissions=false] - Fakes interaction with permissions.
- * @property {boolean} [mockChrome=false] - Adds the chrome runtime properties.
- * @property {boolean} [mocksChromeInIframe=false] - Adds the chrome runtime properties inside the every newly created iframe.
- * @property {boolean} [mockDeviceMemory=false] - Sets device memory to other value than 0.
+ * @property {boolean} [addPlugins=true] - If plugins should be added to the navigator.
+ * @property {boolean} [emulateWindowFrame=true] - Emulates window Iframe.
+ * @property {boolean} [emulateWebGL=true] - Emulates graphic card.
+ * @property {boolean} [emulateConsoleDebug=true] - Emulates console.debug to return null.
+ * @property {boolean} [addLanguage=true] - Adds languages to the navigator.
+ * @property {boolean} [hideWebDriver=true] - Hides the webdriver by changing the navigator proto.
+ * @property {boolean} [hackPermissions=true] - Fakes interaction with permissions.
+ * @property {boolean} [mockChrome=true] - Adds the chrome runtime properties.
+ * @property {boolean} [mocksChromeInIframe=true] - Adds the chrome runtime properties inside the every newly created iframe.
+ * @property {boolean} [mockDeviceMemory=true] - Sets device memory to other value than 0.
*/
const DEFAULT_STEALTH_OPTIONS = {
- addPlugins: false,
- emulateWindowFrame: false,
- emulateWebGL: false,
- emulateConsoleDebug: false,
- addLanguage: false,
- hideWebDriver: false,
- hackPermissions: false,
- mockChrome: false,
- mocksChromeInIframe: false,
- mockDeviceMemory: false,
+ addPlugins: true,
+ emulateWindowFrame: true,
+ emulateWebGL: true,
+ emulateConsoleDebug: true,
+ addLanguage: true,
+ hideWebDriver: true,
+ hackPermissions: true,
+ mockChrome: true,
+ mocksChromeInIframe: true,
+ mockDeviceMemory: true,
};
/**
diff --git a/test/stealth/stealth.js b/test/stealth/stealth.js
index 914bfb32b..f4ea8b6f6 100644
--- a/test/stealth/stealth.js
+++ b/test/stealth/stealth.js
@@ -15,16 +15,28 @@ const getFingerPrint = async (page) => {
describe('Stealth - testing headless chrome hiding tricks', () => {
it('it adds plugins, mimeTypes and passes', async () => {
const browser = await Apify.launchPuppeteer({
+ stealth: true,
stealthOptions: {
- addPlugins: true,
+ emulateWindowFrame: false,
+ emulateWebGL: false,
+ emulateConsoleDebug: false,
+ addLanguage: false,
+ hideWebDriver: false,
+ hackPermissions: false,
+ mockChrome: false,
+ mocksChromeInIframe: false,
+ mockDeviceMemory: false,
},
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
+ const { webDriver } = await getFingerPrint(page);
+ // check if disabling works
+ expect(webDriver).to.be.eql(true);
const { plugins, mimeTypes } = await getFingerPrint(page);
expect(plugins.length).to.be.eql(3);
@@ -35,15 +47,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it hides webDriver', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- hideWebDriver: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { webDriver } = await getFingerPrint(page);
expect(webDriver).to.be.eql(false);
@@ -53,15 +63,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it hacks permissions', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- hackPermissions: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { permissions } = await getFingerPrint(page);
expect(permissions.state).to.be.eql('denied');
@@ -71,15 +79,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it adds language to navigator', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- addLanguage: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { languages } = await getFingerPrint(page);
expect(languages).to.be.an('array');
@@ -90,15 +96,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it emulates WebGL', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- emulateWebGL: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { videoCard } = await getFingerPrint(page);
expect(videoCard[0]).to.be.eql('Intel Inc.');
@@ -109,15 +113,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it emulates windowFrame', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- emulateWindowFrame: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { screen } = await getFingerPrint(page);
expect(screen.wOuterHeight > 0).to.be.eql(true);
@@ -128,15 +130,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it emulates console.debug', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- emulateConsoleDebug: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const returnValue = await page.evaluate(() => console.debug('TEST'));
expect(returnValue).to.be.eql(null);
@@ -145,15 +145,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
});
it('it should mock window.chrome to plain object', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- mockChrome: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { hasChrome } = await getFingerPrint(page);
const chrome = await page.evaluate(() => window.chrome); //eslint-disable-line
expect(chrome).to.be.an('object');
@@ -165,15 +163,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it should mock chrome when iframe is created', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- mocksChromeInIframe: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { iframeChrome } = await getFingerPrint(page);
expect(iframeChrome).to.be.eql('object');
@@ -183,16 +179,14 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it should not break iframe ', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- mocksChromeInIframe: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
const testFuncReturnValue = 'TESTSTRING';
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
await page.evaluate((returnValue) => {
const { document } = window; //eslint-disable-line
const body = document.querySelector('body');
@@ -210,15 +204,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it should mock device memory', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- mockDeviceMemory: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const { deviceMemory } = await getFingerPrint(page);
expect(deviceMemory).not.to.be.eql(0);
@@ -228,24 +220,13 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
it('it should bypass all of the known tests for browser fingerprinting', async () => {
const browser = await Apify.launchPuppeteer({
- stealthOptions: {
- addPlugins: true,
- emulateWindowFrame: true,
- hideWebDriver: true,
- emulateWebGL: true,
- hackPermissions: true,
- addLanguage: true,
- emulateConsoleDebug: true,
- mockChrome: true,
- mocksChromeInIframe: true,
- mockDeviceMemory: true,
- },
+ stealth: true,
headless: true,
useChrome: true,
});
const page = await browser.newPage();
- await page.goto('http://example.com');
+ await page.goto('file://test_html.html');
const fingerPrint = await getFingerPrint(page);
const testedFingerprint = scanner.analyseFingerprint(fingerPrint);
const failedChecks = Object.values(testedFingerprint).filter(val => val.consistent < 3);
diff --git a/test/stealth/test_html.html b/test/stealth/test_html.html
new file mode 100644
index 000000000..e149a3943
--- /dev/null
+++ b/test/stealth/test_html.html
@@ -0,0 +1,10 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>Title</title>
+</head>
+<body>
+
+</body>
+</html>