Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Add useIncognitoPages to PuppeteerPool

Ondra Urbana93b075b298f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5d5bb2074..44684b5d5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,8 @@
NEXT
====================
- Update `CheerioCrawler` to use `requestAsBrowser()` to better disguise as a real browser.
+- Add `useIncognitoPages` option to `PuppeteerPool` to enable opening new pages in incognito
+ browser contexts. This is useful to keep cookies and cache unique for each page.
0.15.5 / 2019-08-15
====================
diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.md
index 595ba2542..faebf0e93 100644
--- a/docs/api/PuppeteerPool.md
+++ b/docs/api/PuppeteerPool.md
@@ -71,31 +71,31 @@ await puppeteerPool.destroy();
just before a page would be recycled by <code>PuppeteerPool</code>. If there are no clients
connected, it has close to zero impact on performance.</p>
</td></tr><tr>
-<td><code>[options.maxOpenPagesPerInstance]</code></td><td><code>Number</code></td><td><code>50</code></td>
+<td><code>[options.maxOpenPagesPerInstance]</code></td><td><code>number</code></td><td><code>50</code></td>
</tr>
<tr>
<td colspan="3"><p>Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.</p>
</td></tr><tr>
-<td><code>[options.retireInstanceAfterRequestCount]</code></td><td><code>Number</code></td><td><code>100</code></td>
+<td><code>[options.retireInstanceAfterRequestCount]</code></td><td><code>number</code></td><td><code>100</code></td>
</tr>
<tr>
<td colspan="3"><p>Maximum number of requests that can be processed by a single browser instance.
After the limit is reached, the browser is retired and new requests are
handled by a new browser instance.</p>
</td></tr><tr>
-<td><code>[options.puppeteerOperationTimeoutSecs]</code></td><td><code>Number</code></td><td><code>15</code></td>
+<td><code>[options.puppeteerOperationTimeoutSecs]</code></td><td><code>number</code></td><td><code>15</code></td>
</tr>
<tr>
<td colspan="3"><p>All browser management operations such as launching a new browser, opening a new page
or closing a page will timeout after the set number of seconds and the connected
browser will be retired.</p>
</td></tr><tr>
-<td><code>[options.instanceKillerIntervalSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
+<td><code>[options.instanceKillerIntervalSecs]</code></td><td><code>number</code></td><td><code>60</code></td>
</tr>
<tr>
<td colspan="3"><p>Indicates how often are the open Puppeteer instances checked whether they can be closed.</p>
</td></tr><tr>
-<td><code>[options.killInstanceAfterSecs]</code></td><td><code>Number</code></td><td><code>300</code></td>
+<td><code>[options.killInstanceAfterSecs]</code></td><td><code>number</code></td><td><code>300</code></td>
</tr>
<tr>
<td colspan="3"><p>When Puppeteer instance reaches the <code>options.retireInstanceAfterRequestCount</code> limit then
@@ -119,7 +119,7 @@ await puppeteerPool.destroy();
<td colspan="3"><p>Options used by <code>Apify.launchPuppeteer()</code> to start new Puppeteer instances.
See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
</td></tr><tr>
-<td><code>[options.recycleDiskCache]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+<td><code>[options.recycleDiskCache]</code></td><td><code>boolean</code></td><td><code>false</code></td>
</tr>
<tr>
<td colspan="3"><p>Enables recycling of disk cache directories by Chrome instances.
@@ -132,7 +132,13 @@ await puppeteerPool.destroy();
<p> Do not use the <code>options.recycleDiskCache</code> setting together with <code>--disk-cache-dir</code>
argument in <code>options.launchPuppeteerOptions.args</code>, the behavior is undefined.</p>
</td></tr><tr>
-<td><code>[options.proxyUrls]</code></td><td><code>Array<String></code></td><td></td>
+<td><code>[options.useIncognitoPages]</code></td><td><code>boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>With this option selected, all pages will be opened in a new incognito browser context, which means
+ that they will not share cookies or cache and their resources will not be throttled by one another.</p>
+</td></tr><tr>
+<td><code>[options.proxyUrls]</code></td><td><code>Array<string></code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>PuppeteerPool</code> instance.
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index a132be6fa..0b4d4c064 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -33,6 +33,7 @@ const DEFAULT_OPTIONS = {
launchPuppeteerFunction: launchPuppeteerOptions => launchPuppeteer(launchPuppeteerOptions),
recycleDiskCache: false,
+ useIncognitoPages: false,
useLiveView: false,
};
@@ -114,19 +115,19 @@ class PuppeteerInstance {
* Enables the use of a preconfigured {@link LiveViewServer} that serves snapshots
* just before a page would be recycled by `PuppeteerPool`. If there are no clients
* connected, it has close to zero impact on performance.
- * @param {Number} [options.maxOpenPagesPerInstance=50]
+ * @param {number} [options.maxOpenPagesPerInstance=50]
* Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.
- * @param {Number} [options.retireInstanceAfterRequestCount=100]
+ * @param {number} [options.retireInstanceAfterRequestCount=100]
* Maximum number of requests that can be processed by a single browser instance.
* After the limit is reached, the browser is retired and new requests are
* handled by a new browser instance.
- * @param {Number} [options.puppeteerOperationTimeoutSecs=15]
+ * @param {number} [options.puppeteerOperationTimeoutSecs=15]
* All browser management operations such as launching a new browser, opening a new page
* or closing a page will timeout after the set number of seconds and the connected
* browser will be retired.
- * @param {Number} [options.instanceKillerIntervalSecs=60]
+ * @param {number} [options.instanceKillerIntervalSecs=60]
* Indicates how often are the open Puppeteer instances checked whether they can be closed.
- * @param {Number} [options.killInstanceAfterSecs=300]
+ * @param {number} [options.killInstanceAfterSecs=300]
* When Puppeteer instance reaches the `options.retireInstanceAfterRequestCount` limit then
* it is considered retired and no more tabs will be opened. After the last tab is closed the
* whole browser is closed too. This parameter defines a time limit between the last tab was opened and
@@ -141,7 +142,7 @@ class PuppeteerInstance {
* @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]
* Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.
* See [`LaunchPuppeteerOptions`](../typedefs/launchpuppeteeroptions).
- * @param {Boolean} [options.recycleDiskCache=false]
+ * @param {boolean} [options.recycleDiskCache=false]
* Enables recycling of disk cache directories by Chrome instances.
* When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.
* This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
@@ -153,7 +154,10 @@ class PuppeteerInstance {
*
* Do not use the `options.recycleDiskCache` setting together with `--disk-cache-dir`
* argument in `options.launchPuppeteerOptions.args`, the behavior is undefined.
- * @param {String[]} [options.proxyUrls]
+ * @param {boolean} [options.useIncognitoPages]
+ * With this option selected, all pages will be opened in a new incognito browser context, which means
+ * that they will not share cookies or cache and their resources will not be throttled by one another.
+ * @param {string[]} [options.proxyUrls]
* An array of custom proxy URLs to be used by the `PuppeteerPool` instance.
* The provided custom proxies' order will be randomized and the resulting list rotated.
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
@@ -172,6 +176,7 @@ class PuppeteerPool {
killInstanceAfterSecs,
launchPuppeteerOptions,
recycleDiskCache,
+ useIncognitoPages,
proxyUrls,
useLiveView,
} = _.defaults({}, options, DEFAULT_OPTIONS);
@@ -196,6 +201,7 @@ class PuppeteerPool {
checkParamOrThrow(killInstanceAfterSecs, 'options.killInstanceAfterSecs', 'Number');
checkParamOrThrow(launchPuppeteerOptions, 'options.launchPuppeteerOptions', 'Maybe Object');
checkParamOrThrow(recycleDiskCache, 'options.recycleDiskCache', 'Maybe Boolean');
+ checkParamOrThrow(useIncognitoPages, 'options.useIncognitoPages', 'Maybe Boolean');
checkParamOrThrow(proxyUrls, 'options.proxyUrls', 'Maybe Array');
// Enforce non-empty proxyUrls array
if (proxyUrls && !proxyUrls.length) throw new Error('Parameter "options.proxyUrls" of type Array must not be empty');
@@ -208,6 +214,7 @@ class PuppeteerPool {
this.puppeteerOperationTimeoutMillis = puppeteerOperationTimeoutSecs * 1000;
this.killInstanceAfterMillis = killInstanceAfterMillis || killInstanceAfterSecs * 1000;
this.recycledDiskCacheDirs = recycleDiskCache ? new LinkedList() : null;
+ this.useIncognitoPages = useIncognitoPages;
this.proxyUrls = proxyUrls ? _.shuffle(proxyUrls) : null;
this.liveViewServer = useLiveView ? new LiveViewServer() : null;
this.launchPuppeteerFunction = async () => {
@@ -526,8 +533,11 @@ class PuppeteerPool {
try {
const browser = await instance.browserPromise;
+ const context = this.useIncognitoPages
+ ? await browser.createIncognitoBrowserContext()
+ : browser.defaultBrowserContext();
const page = await addTimeoutToPromise(
- browser.newPage(),
+ context.newPage(),
this.puppeteerOperationTimeoutMillis,
'PuppeteerPool: browser.newPage() timed out.',
);
@@ -545,18 +555,25 @@ class PuppeteerPool {
/**
* Adds the necessary boilerplate to allow page reuse and also
* captures page.close() errors to prevent meaningless log clutter.
- * @param page
+ * @param {Page} page
* @ignore
*/
_decoratePage(page) {
const originalPageClose = page.close;
page.close = async (...args) => {
this.closedPages.add(page);
- return originalPageClose.apply(page, args)
+ await originalPageClose.apply(page, args)
.catch((err) => {
const instance = this.pagesToInstancesMap.get(page);
log.debug('PuppeteerPool: Page.close() failed', { errorMessage: err.message, id: instance.id });
});
+ const context = page.browserContext();
+ if (context.isIncognito()) {
+ await context.close().catch((err) => {
+ const instance = this.pagesToInstancesMap.get(page);
+ log.debug('PuppeteerPool: Context.close() failed', { errorMessage: err.message, id: instance.id });
+ });
+ }
};
page.once('error', (error) => {
diff --git a/test/puppeteer_pool.js b/test/puppeteer_pool.js
index 0b64981f0..25c8c5c8f 100644
--- a/test/puppeteer_pool.js
+++ b/test/puppeteer_pool.js
@@ -472,6 +472,23 @@ describe('PuppeteerPool', () => {
});
});
+ describe('useIncognitoPages', () => {
+ it('opens a page in incognito browser context', async () => {
+ const pool = new Apify.PuppeteerPool({
+ useIncognitoPages: true,
+ });
+ const page = await pool.newPage();
+ const browser = page.browser();
+ const context = page.browserContext();
+ expect(context.isIncognito()).to.be.eql(true);
+ await page.close();
+ const contexts = browser.browserContexts();
+ expect(contexts).to.have.lengthOf(1);
+ expect(contexts[0].isIncognito()).to.be.eql(false);
+ await pool.destroy();
+ });
+ });
+
describe('the proxyUrls parameter', () => {
it('supports rotation of custom proxies', async () => {
const optionsLog = [];