Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Browser crawler (#858)

petrpatek32bf3ed860f4
diff --git a/.gitignore b/.gitignore
index 5d163aa37..b16e17eab 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,3 +20,4 @@ types
docs/api
docs/typedefs
docs/examples
+.history
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d9d87c90d..decac0dbe 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,67 @@
+1.0.0 / 2020/01/XX
+====================
+After 3.5 years of rapid development and a lot of breaking changes and deprecations,
+here comes the result - **Apify SDK v1**. There were two goals for this release. **Stability**
+and **adding support for more browsers** - Firefox and Webkit (Safari).
+
+The SDK has grown quite popular over the years, powering thousands of web scraping
+and automation projects. We think our developers deserve a stable environment to work
+in and by releasing SDK v1, **we commit to only make breaking changes once a year,
+with a new major release**.
+
+We added support for more browsers by replacing `PuppeteerPool` with
+[`browser-pool`](https://github.com/apify/browser-pool). A new library that we created
+specifically for this purpose. It builds on the ideas from `PuppeteerPool` and extends
+them to support [Playwright](https://github.com/microsoft/playwright). Playwright is
+a browser automation library similar to Puppeteer. It works with all well known browsers
+and uses almost the same interface as Puppeteer, while adding useful features and simplifying
+common tasks. Don't worry, you can still use Puppeteer with the new `BrowserPool`.
+
+A large breaking change is that neither `puppeteer` nor `playwright` are bundled with
+the SDK v1. To make the choice of a library easier and installs faster, users will
+have to install the selected modules and versions themselves. This allows us to add
+support for even more libraries in the future.
+
+Thanks to the addition of Playwright we now have a `PlaywrightCrawler`. It is very similar
+to `PuppeteerCrawler` and you can pick the one you prefer. It also means we needed to make
+some interface changes. The `launchPuppeteerFunction` option of `PuppeteerCrawler` is gone
+and `launchPuppeteerOptions` were replaced by `launchContext`. We also moved things around
+in the `handlePageFunction` arguments. See the
+[migration guide](https://github.com/apify/apify-js/blob/master/MIGRATIONS.md)
+for more detailed explanation and migration examples.
+
+What's in store for SDK v2? We want to split the SDK into smaller libraries,
+so that everyone can install only the things they need. We plan a TypeScript migration
+to make crawler development faster and safer. Finally, we will take a good look
+at the interface of the whole SDK and update it to improve the developer experience.
+Bug fixes and scraping features will of course keep landing in versions 1.X as well.
+
+Full list of changes:
+
+- **BREAKING:** Removed `puppeteer` from dependencies. If you want to use Puppeteer,
+ you must install it yourself.
+- **BREAKING:** Removed `PuppeteerPool`. Use [`browser-pool`](https://github.com/apify/browser-pool).
+- **BREAKING:** Removed `PuppeteerCrawlerOptions.launchPuppeteerOptions`. Use `launchContext`.
+- **BREAKING:** Removed `PuppeteerCrawlerOptions.launchPuppeteerFunction`.
+ Use `PuppeteerCrawlerOptions.preLaunchHooks` and `postLaunchHooks`.
+- **BREAKING:** Removed `PuppeteerCrawlerOptions.gotoFunction`.
+ Use `PuppeteerCrawlerOptions.preNavigationHooks` and `postNavigationHooks`.
+- **BREAKING:** Removed `args.autoscaledPool` and `args.puppeteerPool` from `handle(Page/Request)Function`
+ arguments. Use `args.crawler.autoscaledPool` and `args.crawler.browserPool`.
+- **BREAKING:** The `useSessionPool` and `persistCookiesPerSession` options of crawlers
+ are now `true` by default. Explicitly set them to `false` to override the behavior.
+- **BREAKING:** `Apify.launchPuppeteer()` no longer accepts `LaunchPuppeteerOptions`.
+ It now accepts `LaunchContext`.
+
+- Added `Apify.PlaywrightCrawler` which is almost identical to `PuppeteerCrawler`,
+ but it crawls with the `playwright` library.
+- Added `Apify.launchPlaywright(launchContext)` helper function.
+- Added `browserPoolOptions` to `PuppeteerCrawler` to configure `BrowserPool`.
+- Added `crawler` to `handle(Request/Page)Function` arguments.
+- Added `browserController` to `handlePageFunction` arguments.
+- Added `crawler.crawlingContexts` `Map` which includes all running `crawlingContext`s.
+
+
0.22.2 / 2020/12/22
====================
- Pinned `cheerio` to `1.0.0-rc.3` to avoid install problems in some builds.
diff --git a/MIGRATIONS.md b/MIGRATIONS.md
index d0a622c07..a2b68ed58 100644
--- a/MIGRATIONS.md
+++ b/MIGRATIONS.md
@@ -1,172 +1,368 @@
-# Migration from 0.20.x to 0.21.0
-There are 2 key changes to watch out for in version 0.21.0. We redesigned the way proxies
-are used throughout the SDK, and we removed `Apify.utils.getRandomUserAgent()`.
+# Migration from 0.2x.x to 1.0.0
+There are a lot of breaking changes in the v1.0.0 release, but we're confident that
+updating your code will be a matter of minutes. Below, you'll find examples how to do it
+and also short tutorials how to use many of the new features.
-## Proxy Configuration
-Configuring proxies the way you were used to will no longer work in SDK 0.21.0.
-To improve the proxy experience, we removed all the various proxy configuration
-options scattered throughout the SDK and replaced them with a single
-`ProxyConfiguration` class.
+> Many of the new features are made with power users in mind,
+> so don't worry if something looks complicated. You don't need to use it.
-### Removal of `Apify.getApifyProxyUrl()`
-This function has been fully replaced by the new `ProxyConfiguration` class.
-Make sure to remove any references to it before upgrading.
+<!-- toc -->
-### Usage with Apify Proxy:
-The usage of Apify Proxy in SDK was inconsistent. Now, everything is pretty simple
-and as a bonus, we validate your configuration at creation, so if there's something
-wrong, you'll know right away, instead of seeing your crawlers fail with cryptic errors.
+- [Installation](#installation)
+- [Handler arguments are now Crawling Context](#handler-arguments-are-now-crawling-context)
+ * [`Map` of crawling contexts and their IDs](#map-of-crawling-contexts-and-their-ids)
+ * [`autoscaledPool` was moved under `crawlingContext.crawler`](#autoscaledpool-was-moved-under-crawlingcontextcrawler)
+- [Replacement of `PuppeteerPool` with `BrowserPool`](#replacement-of-puppeteerpool-with-browserpool)
+ * [Access to running `BrowserPool`](#access-to-running-browserpool)
+ * [Pages now have IDs](#pages-now-have-ids)
+ * [Configuration and lifecycle hooks](#configuration-and-lifecycle-hooks)
+ * [Introduction of `BrowserController`](#introduction-of-browsercontroller)
+ * [`BrowserPool` methods vs `PuppeteerPool`](#browserpool-methods-vs-puppeteerpool)
+- [Updated `PuppeteerCrawlerOptions`](#updated-puppeteercrawleroptions)
+ * [`launchPuppeteerOptions` => `launchContext`](#launchpuppeteeroptions--launchcontext)
+ * [Removal of `launchPuppeteerFunction`](#removal-of-launchpuppeteerfunction)
+- [Launch functions](#launch-functions)
+ * [Updated arguments](#updated-arguments)
+ * [Custom modules](#custom-modules)
-1. You create a `proxyConfiguration` instance.
-2. Are you using `CheerioCrawler` or `PuppeteerCrawler`?
- - YES: You plug it into the crawler, and you're done.
- - NO: You call `proxyConfiguration.newUrl([sessionId])` to get your URL.
+<!-- tocstop -->
-> `BasicCrawler` does not automatically use `ProxyConfiguration` because it does not make
-any network requests (automatically).
+## Installation
+Previous versions of the SDK bundled the `puppeteer` package, so you did not have to install
+it. SDK v1 supports also `playwright` and we don't want to force users to install both.
+To install SDK v1 with Pupppeteer (same as previous versions), run:
-That's it. See the examples.
+```bash
+npm install apify puppeteer
+```
+
+To install SDK v1 with Playwright run:
+```bash
+npm install apify playwright
+```
+
+## Handler arguments are now Crawling Context
+Previously, arguments of user provided handler functions were provided in separate
+objects. This made it difficult to track values across function invocations.
```js
-// before
-const crawler = new Apify.PuppeteerCrawler({
- launchPuppeteerOptions: {
- useApifyProxy: true,
- apifyProxyGroups: ['GROUP1', 'GROUP2'],
- }
- // ...
-})
+const handlePageFunction = async (args1) => {
+ args1.hasOwnProperty('proxyInfo') // true
+}
-// now
-const proxyConfiguration = await Apify.createProxyConfiguration({
- groups: ['GROUP1', 'GROUP2'],
-})
+const handleFailedRequestFunction = async (args2) => {
+ args2.hasOwnProperty('proxyInfo') // false
+}
-const crawler = new Apify.PuppeteerCrawler({
- proxyConfiguration,
- // ...
-})
+args1 === args2 // false
```
+This happened because a new arguments object was created for each function.
+With SDK v1 we now have a single object called Crawling Context.
+
```js
-// before
-const browser = await Apify.launchPuppeteer({
- useApifyProxy: true,
- apifyProxyGroups: ['GROUP1', 'GROUP2'],
- apifyProxySession: 'session-123'
-})
+const gotoFunction = async (gotoContext) => {
+ // navigate somewhere
+}
-// now
-const proxyConfiguration = await Apify.createProxyConfiguration({
- groups: ['GROUP1', 'GROUP2'],
-});
+const handlePageFunction = async (handlePageContext) => {
+ handlePageContext.hasOwnProperty('proxyInfo') // true
+}
-const browser = await Apify.launchPuppeteer({
- proxyUrl: proxyConfiguration.newUrl('session-123')
-})
+const handleFailedRequestFunction = async (handleFailedContext) => {
+ handleFailedContext.hasOwnProperty('proxyInfo') // true
+}
+
+// All contexts are the same object.
+gotoContext === handlePageContext // true
+handlePageContext === handleFailedContext // true
```
-#### PuppeteerCrawler warning
-`ProxyConfiguration` integrates seamlessly with `PuppeteerCrawler`, but beware
-wrong usage of `launchPuppeteerFunction`. It receives an options parameter
-that should not be ignored. This is not new, it has always been like that,
-but until now, only non-critical features were dependent on it, so you
-might have been using it wrongly and never really noticed.
+### `Map` of crawling contexts and their IDs
+Now that all the objects are the same, we can keep track of all running crawling contexts.
+We can do that by working with the new `id` property of `crawlingContext`
+This is useful when you need cross-context access.
-**Correct:**
-```javascript
-async function launchPuppeteerFunction(options) {
- const newOpts = {
- ...options,
- foo: 'bar',
+```js
+let masterContextId;
+const handlePageFunction = async ({ id, page, request, crawler }) => {
+ if (request.userData.masterPage) {
+ masterContextId = id;
+ // Prepare the master page.
+ } else {
+ const masterContext = crawler.crawlingContexts.get(masterContextId);
+ const masterPage = masterContext.page;
+ const masterRequest = masterContext.request;
+ // Now we can manipulate the master data from another handlePageFunction.
}
- // do some other things
- return Apify.launchPuppeteer(newOpts);
}
```
-**Incorrect:**
-```javascript
-async function launchPuppeteerFunction() {
- const opts = {
- foo: 'bar',
- }
- // Because we ignored the options, correct parameters
- // will not make it to the browser. This prevents
- // proxyConfiguration from working correctly.
- return Apify.launchPuppeteer(opts);
+
+### `autoscaledPool` was moved under `crawlingContext.crawler`
+To prevent bloat and to make access to certain key objects easier, we exposed a `crawler`
+property on the handle page arguments.
+
+```js
+const handePageFunction = async ({ request, page, crawler }) => {
+ await crawler.requestQueue.addRequest({ url: 'https://example.com' });
+ await crawler.autoscaledPool.pause();
+}
+```
+
+This also means that some shorthands like `puppeteerPool` or `autoscaledPool` were
+no longer necessary.
+
+```js
+const handePageFunction = async (crawlingContext) => {
+ crawlingContext.autoscaledPool // does NOT exist anymore
+ crawlingContext.crawler.autoscaledPool // <= this is correct usage
}
```
-### Usage with your own proxies:
-Using your own proxies was possible, but it was difficult to find where to
-enter your URLs to make use of them. Now it's the same as with Apify Proxy,
-you just add them to the `ProxyConfiguration`. As a bonus, your custom proxies
-can now also be managed by `SessionPool` which was not possible before.
-So if one of your proxies goes bad, `SessionPool` will automatically
-retire it from use.
+## Replacement of `PuppeteerPool` with `BrowserPool`
+`BrowserPool` was created to extend `PuppeteerPool` with the ability to manage other
+browser automation libraries. The API is similar, but not the same.
+
+### Access to running `BrowserPool`
+Only `PuppeteerCrawler` and `PlaywrightCrawler` use `BrowserPool`. You can access it
+on the `crawler` object.
+
+```js
+const crawler = new Apify.PlaywrightCrawler({
+ handlePageFunction: async ({ page, crawler }) => {
+ crawler.browserPool // <-----
+ }
+});
+
+crawler.browserPool // <-----
+```
+
+### Pages now have IDs
+And they're equal to `crawlingContext.id` which gives you access to full `crawlingContext`
+in hooks. See [Lifecycle hooks](#configuration-and-lifecycle-hooks) below.
+
+```js
+const pageId = browserPool.getPageId
+```
+
+### Configuration and lifecycle hooks
+The most important addition with `BrowserPool` are the
+[lifecycle hooks](https://github.com/apify/browser-pool#browserpool).
+You can access them via `browserPoolOptions` in both crawlers. A full list of `browserPoolOptions`
+can be found in [`browser-pool` readme](https://github.com/apify/browser-pool#new-browserpooloptions).
```js
-// before
const crawler = new Apify.PuppeteerCrawler({
- puppeteerPoolOptions: {
- proxyUrls: [
- 'http://proxy1.com',
- 'http://proxy2.com',
+ browserPoolOptions: {
+ retireBrowserAfterPageCount: 10,
+ preLaunchHooks: [
+ async (pageId, launchContext) => {
+ const { request } = crawler.crawlingContexts.get(pageId);
+ if (request.userData.useHeadful === true) {
+ launchContext.launchOptions.headless = false;
+ }
+ }
]
}
- // ...
})
+```
+
+### Introduction of `BrowserController`
+[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller)
+is a class of `browser-pool` that's responsible for browser management.
+Its purpose is to provide a single API for working with both Puppeteer and Playwright browsers.
+It works automatically in the background, but if you ever wanted to close a browser properly,
+you should use a `browserController` to do it. You can find it in the handle page arguments.
+
+```js
+const handlePageFunction = async ({ page, browserController }) => {
+ // Wrong usage. Could backfire because it bypasses BrowserPool.
+ page.browser().close();
+
+ // Correct usage. Allows graceful shutdown.
+ browserController.close();
+
+ const cookies = [/* some cookie objects */];
+ // Wrong usage. Will only work in Puppeteer and not Playwright.
+ page.setCookies(...cookies);
-// now
-const proxyConfiguration = await Apify.createProxyConfiguration({
- proxyUrls: [
- 'http://proxy1.com',
- 'http://proxy2.com',
- ]
+ // Correct usage. Will work in both.
+ browserController.setCookies(page, cookies);
+}
+```
+
+The `BrowserController` also includes important information about the browser, such as
+the context it was launched with. This was difficult to do before SDK v1.
+
+```js
+const handlePageFunction = async ({ browserController }) => {
+ // Information about the proxy used by the browser
+ browserController.launchContext.proxyInfo
+
+ // Session used by the browser
+ browserController.launchContext.session
+}
+```
+
+### `BrowserPool` methods vs `PuppeteerPool`
+Some functions were removed (in line with earlier deprecations), and some were changed a bit:
+
+```js
+// OLD
+puppeteerPool.recyclePage(page);
+
+// NEW
+page.close();
+```
+
+```js
+// OLD
+puppeteerPool.retire(page.browser());
+
+// NEW
+browserPool.retireBrowserByPage(page);
+```
+
+```js
+// OLD
+puppeteerPool.serveLiveViewSnapshot();
+
+// NEW
+// There's no LiveView in BrowserPool
+```
+
+## Updated `PuppeteerCrawlerOptions`
+To keep `PuppeteerCrawler` and `PlaywrightCrawler` consistent, we updated the options.
+
+### `launchPuppeteerOptions` => `launchContext`
+Those were always a point of confusion because they merged custom Apify options with
+`launchOptions` of Puppeteer.
+
+```js
+const launchPuppeteerOptions = {
+ useChrome: true, // Apify option
+ headless: false, // Puppeteer option
+}
+```
+
+Use the new `launchContext` object, which explicitly defines `launchOptions`.
+`launchPuppeteerOptions` were removed.
+
+```js
+const crawler = new Apify.PuppeteerCrawler({
+ launchContext: {
+ useChrome: true, // Apify option
+ launchOptions: {
+ headless: false // Puppeteer option
+ }
+ }
})
+```
+
+> LaunchContext is also a type of [`browser-pool`](https://github.com/apify/browser-pool) and
+> the structure is exactly the same there. SDK only adds extra options.
+
+### Removal of `launchPuppeteerFunction`
+`browser-pool` introduces the idea of [lifecycle hooks](https://github.com/apify/browser-pool#browserpool),
+which are functions that are executed when a certain event in the browser lifecycle happens.
+
+```js
+const launchPuppeteerFunction = async (launchPuppeteerOptions) => {
+ if (someVariable === 'chrome') {
+ launchPuppeteerOptions.useChrome = true;
+ }
+ return Apify.launchPuppeteer(launchPuppeteerOptions);
+}
const crawler = new Apify.PuppeteerCrawler({
- proxyConfiguration,
+ launchPuppeteerFunction,
// ...
})
```
+Now you can recreate the same functionality with a `preLaunchHook`:
+
```js
-// before
-const browser1 = await Apify.launchPuppeteer({
- proxyUrl: 'http://proxy1:com',
+const maybeLaunchChrome = (pageId, launchContext) => {
+ if (someVariable === 'chrome') {
+ launchContext.useChrome = true;
+ }
+}
+
+const crawler = new Apify.PuppeteerCrawler({
+ browserPoolOptions: {
+ preLaunchHooks: [maybeLaunchChrome]
+ },
+ // ...
})
+```
-// now
-const proxyConfiguration = await Apify.createProxyConfiguration({
- proxyUrls: [
- 'http://proxy1.com',
- 'http://proxy2.com',
- ]
-});
+This is better in multiple ways. It is consistent across both Puppeteer and Playwright.
+It allows you to easily construct your browsers with pre-defined behavior:
-const browser = await Apify.launchPuppeteer({
- proxyUrl: proxyConfiguration.newUrl('session-123'),
-})
+```js
+const preLaunchHooks = [
+ maybeLaunchChrome,
+ useHeadfulIfNeeded,
+ injectNewFingerprint,
+]
+```
+
+And thanks to the addition of [`crawler.crawlingContexts`](#handler-arguments-are-now-crawling-context)
+the functions also have access to the `crawlingContext` of the `request` that triggered the launch.
+
+```js
+const preLaunchHooks = [
+ async function maybeLaunchChrome(pageId, launchContext) {
+ const { request } = crawler.crawlingContexts.get(pageId);
+ if (request.userData.useHeadful === true) {
+ launchContext.launchOptions.headless = false;
+ }
+ }
+]
```
-## Removal of random user agents
+## Launch functions
+In addition to `Apify.launchPuppeteer()` we now also have `Apify.launchPlaywright()`.
+
+### Updated arguments
+We [updated the launch options object](#launchpuppeteeroptions--launchcontext) because
+it was a frequent source of confusion.
+
+```js
+// OLD
+await Apify.launchPuppeteer({
+ useChrome: true,
+ headless: true,
+})
+
+// NEW
+await Apify.launchPuppeteer({
+ useChrome: true,
+ launchOptions: {
+ headless: true,
+ }
+})
+```
-Nowadays, bot walls cannot be bypassed simply by rotating user agents.
-Quite the opposite is true, using a user agent that does not match
-the used browser / network stack, will most likely lead to
-red flags and bans due to detected inconsistencies. For that reason,
-we've decided to remove the `Apify.utils.getRandomUserAgent()` function,
-effective immediately. Leaving a deprecation period would only have
-your scrapers get blocked for longer. Please make sure you remove all
-references to the function from your code.
+### Custom modules
+`Apify.launchPuppeteer` already supported the `puppeteerModule` option. With Playwright,
+we normalized the name to `launcher` because the `playwright` module itself does not
+launch browsers.
```js
-// before
-const browser = await Apify.launchPuppeteer({
- userAgent: Apify.utils.getRandomUserAgent(),
+const puppeteer = require('puppeteer');
+const playwright = require('playwright');
+
+await Apify.launchPuppeteer();
+// Is the same as:
+await Apify.launchPuppeteer({
+ launcher: puppeteer
})
-// now, keep the default
-const browser = await Apify.launchPuppeteer();
+await Apify.launchPlaywright();
+// Is the same as:
+await Apify.launchPlaywright({
+ launcher: playwright.chromium
+})
```
diff --git a/README.md b/README.md
index 1a8c9862e..718bffd8c 100644
--- a/README.md
+++ b/README.md
@@ -6,7 +6,7 @@
[![Build Status](https://travis-ci.com/apify/apify-js.svg?branch=master)](https://travis-ci.com/apify/apify-js)
Apify SDK simplifies the development of web crawlers, scrapers, data extractors and web automation jobs.
-It provides tools to manage and automatically scale a pool of headless Chrome / Puppeteer instances,
+It provides tools to manage and automatically scale a pool of headless browsers,
to maintain queues of URLs to crawl, store crawling results to a local filesystem or into the cloud,
rotate proxies and much more.
The SDK is available as the [`apify`](https://www.npmjs.com/package/apify) NPM package.
@@ -18,7 +18,7 @@ running on the [Apify Cloud](https://apify.com/).
## Motivation
-Thanks to tools like [Puppeteer](https://github.com/puppeteer/puppeteer) or
+Thanks to tools like [Puppeteer](https://github.com/puppeteer/puppeteer), [Playwright](https://github.com/microsoft/playwright) or
[Cheerio](https://www.npmjs.com/package/cheerio), it is easy to write Node.js code to extract data from web pages. But
eventually things will get complicated. For example, when you try to:
@@ -51,9 +51,9 @@ efficient web crawler, but it does not work on websites that require JavaScript.
a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/puppeteer/puppeteer).
The pool of Chrome browsers is automatically scaled up and down based on available system resources.
-- [`PuppeteerPool`](https://sdk.apify.com/docs/api/puppeteer-pool) - Provides web browser tabs for user jobs
-from an automatically-managed pool of Chrome browser instances, with configurable browser recycling
-and retirement policies. Supports reuse of the disk cache to speed up the crawling of websites and reduce proxy bandwidth.
+- [`PlaywrightCrawler`](https://sdk.apify.com/docs/api/playwright-crawler) - Unlike `PuppeteerCrawler`
+you can use [Playwright](https://github.com/microsoft/playwright) to manage almost any headless browser.
+It also provides a cleaner and more mature interface while keeping the ease of use and advanced features.
- [`RequestList`](https://sdk.apify.com/docs/api/request-list) - Represents a list of URLs to crawl.
The URLs can be passed in code or in a text file hosted on the web. The list persists its state so that crawling
@@ -93,34 +93,38 @@ Apify SDK requires [Node.js](https://nodejs.org/en/) 10.17 or later, with the ex
Add Apify SDK to any Node.js project by running:
```bash
-npm install apify --save
+npm install apify playwright
```
-Run the following example to perform a recursive crawl of a website using Puppeteer. For more examples showcasing various features of the Apify SDK,
-[see the Examples section of the documentation](https://sdk.apify.com/docs/examples/basic-crawler).
+> Neither `playwright` nor `puppeteer` are bundled with the SDK to reduce install size and allow greater
+> flexibility. That's why we install it with NPM. You can choose one, both, or neither.
+
+Run the following example to perform a recursive crawl of a website using Playwright. For more examples showcasing various features of the Apify SDK,
+[see the Examples section of the documentation](https://sdk.apify.com/docs/examples/crawl-multiple-urls).
```javascript
const Apify = require('apify');
+// Apify.main is a helper function, you don't need to use it.
Apify.main(async () => {
const requestQueue = await Apify.openRequestQueue();
+ // Choose the first URL to open.
await requestQueue.addRequest({ url: 'https://www.iana.org/' });
- const pseudoUrls = [new Apify.PseudoUrl('https://www.iana.org/[.*]')];
- const crawler = new Apify.PuppeteerCrawler({
+ const crawler = new Apify.PlaywrightCrawler({
requestQueue,
handlePageFunction: async ({ request, page }) => {
+ // Extract HTML title of the page.
const title = await page.title();
console.log(`Title of ${request.url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
await Apify.utils.enqueueLinks({
page,
- selector: 'a',
- pseudoUrls,
requestQueue,
+ pseudoUrls: ['https://www.iana.org/[.*]'],
});
},
- maxRequestsPerCrawl: 100,
- maxConcurrency: 10,
});
await crawler.run();
@@ -146,8 +150,6 @@ Install the CLI by running:
npm -g install apify-cli
```
-You might need to run the above command with `sudo`, depending on how crazy your configuration is.
-
Now create a boilerplate of your new web crawling project by running:
```bash
diff --git a/package.json b/package.json
index f91b2ae4d..4321e69ae 100644
--- a/package.json
+++ b/package.json
@@ -62,6 +62,7 @@
"@types/tough-cookie": "^4.0.0",
"apify-client": "1.0.1",
"apify-shared": "^0.5.0",
+ "browser-pool": "1.0.2",
"cheerio": "1.0.0-rc.3",
"content-type": "^1.0.4",
"express": "^4.17.1",
@@ -72,6 +73,7 @@
"mime-types": "^2.1.27",
"ow": "^0.20.0",
"proxy-chain": "^0.4.5",
+ "puppeteer": "^5.5.0",
"rimraf": "^3.0.2",
"semver": "^7.3.2",
"socket.io": "^3.0.3",
@@ -84,10 +86,10 @@
"@apify/eslint-config": "^0.1.3",
"@babel/cli": "^7.11.6",
"@babel/core": "^7.11.6",
- "@babel/preset-env": "^7.11.5",
- "@babel/register": "^7.11.5",
"@babel/eslint-parser": "^7.12.1",
"@babel/plugin-syntax-class-properties": "^7.12.1",
+ "@babel/preset-env": "^7.11.5",
+ "@babel/register": "^7.11.5",
"@types/content-type": "^1.1.3",
"@types/fs-extra": "^9.0.2",
"@types/htmlparser2": "^3.10.2",
@@ -108,6 +110,7 @@
"jest-circus": "^26.5.3",
"jsdoc": "^3.6.6",
"jsdoc-export-default-interop": "^0.3.1",
+ "playwright": "^1.7.1",
"portastic": "^1.0.1",
"prettier": "^2.1.2",
"proxy": "^1.0.2",
@@ -118,8 +121,5 @@
"typescript": "3.8.3",
"ua-parser-js": "^0.7.22",
"why-is-node-running": "^2.2.0"
- },
- "optionalDependencies": {
- "puppeteer": "5.5.0"
}
}
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index 9d5ba2b92..1ed54d8c8 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -44,8 +44,8 @@ const SAFE_MIGRATION_WAIT_MILLIS = 20000;
* ```
* {
* request: Request,
- * autoscaledPool: AutoscaledPool,
* session: Session,
+ * crawler: BasicCrawler,
* }
* ```
* where the {@link Request} instance represents the URL to crawl.
@@ -76,6 +76,8 @@ const SAFE_MIGRATION_WAIT_MILLIS = 20000;
* {
* request: Request,
* error: Error,
+ * session: Session,
+ * crawler: BasicCrawler,
* }
* ```
* where the {@link Request} instance corresponds to the failed request, and the `Error` instance
@@ -105,7 +107,7 @@ const SAFE_MIGRATION_WAIT_MILLIS = 20000;
* @property {boolean} [useSessionPool=false]
* If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
* The session instance will be than available in the `handleRequestFunction`.
- * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {SessionPool} to use.
+ * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {@link SessionPool} to use.
*/
/**
@@ -165,8 +167,16 @@ const SAFE_MIGRATION_WAIT_MILLIS = 20000;
* await crawler.run();
* ```
* @property {Statistics} stats
- * Contains statistics about the current run
- *
+ * Contains statistics about the current run.
+ * @property {?RequestList} requestList
+ * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?RequestQueue} requestQueue
+ * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?SessionPool} sessionPool
+ * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session}s.
+ * Only available if used by the crawler.
* @property {AutoscaledPool} autoscaledPool
* A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
* Note that this property is only initialized after calling the {@link BasicCrawler#run} function.
@@ -315,10 +325,7 @@ class BasicCrawler {
try {
await this.isRunningPromise;
} finally {
- if (this.useSessionPool) {
- await this.sessionPool.teardown();
- }
-
+ await this.teardown();
await this.stats.stopCapturing();
const finalStats = this.stats.calculate();
const { requestsFailed, requestsFinished } = this.stats.state;
@@ -451,8 +458,8 @@ class BasicCrawler {
// Shared crawling context
const crawlingContext = {
id: cryptoRandomObjectId(10),
+ crawler: this,
request,
- autoscaledPool: this.autoscaledPool,
session,
};
this.crawlingContexts.set(crawlingContext.id, crawlingContext);
@@ -588,6 +595,16 @@ class BasicCrawler {
this.handledRequestsCount = this.requestList.handledCount();
}
}
+
+ /**
+ * Function for cleaning up after all request are processed.
+ * @ignore
+ */
+ async teardown() {
+ if (this.useSessionPool) {
+ await this.sessionPool.teardown();
+ }
+ }
}
export default BasicCrawler;
@@ -600,13 +617,13 @@ export default BasicCrawler;
/**
* @typedef HandleRequestInputs
* @property {Request} request The original {Request} object.
- * @property {AutoscaledPool} autoscaledPool
* A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
* Note that this property is only initialized after calling the {@link BasicCrawler#run} function.
* You can use it to change the concurrency settings on the fly,
* to pause the crawler by calling {@link AutoscaledPool#pause}
* or to abort it by calling {@link AutoscaledPool#abort}.
* @property {Session} [session]
+ * @property {BasicCrawler} [crawler]
*/
/**
@@ -619,7 +636,6 @@ export default BasicCrawler;
* @typedef HandleFailedRequestInput
* @property {Error} error The Error thrown by `handleRequestFunction`.
* @property {Request} request The original {Request} object.
- * @property {AutoscaledPool} autoscaledPool
* @property {Session} [session]
* @property {ProxyInfo} [proxyInfo]
*/
diff --git a/src/crawlers/browser_crawler.js b/src/crawlers/browser_crawler.js
new file mode 100644
index 000000000..61c318c53
--- /dev/null
+++ b/src/crawlers/browser_crawler.js
@@ -0,0 +1,522 @@
+import ow from 'ow';
+import _ from 'underscore';
+import { BrowserPool } from 'browser-pool'; // eslint-disable-line import/no-duplicates
+import { BASIC_CRAWLER_TIMEOUT_MULTIPLIER } from '../constants';
+import { SessionPool } from '../session_pool/session_pool'; // eslint-disable-line import/no-duplicates
+import EVENTS from '../session_pool/events'; // eslint-disable-line import/no-duplicates
+import { addTimeoutToPromise } from '../utils';
+import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
+import { validators } from '../validators';
+import {
+ throwOnBlockedRequest,
+} from './crawler_utils';
+
+// eslint-enable-line import/no-duplicates
+
+/**
+ * @typedef BrowserCrawlerOptions
+ * @property {function} handlePageFunction
+ * Function that is called to process each request.
+ * It is passed an object with the following fields:
+ *
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * session: Session,
+ * browserController: BrowserController,
+ * proxyInfo: ProxyInfo,
+ * crawler: BrowserCrawler,
+ * }
+ * ```
+ *
+ * `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * `page` is an instance of the `Puppeteer`
+ * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) or `Playwright`
+ * [`Page`](https://playwright.dev/docs/api/class-page)
+ * `browserPool` is an instance of the
+ * [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool),
+ * `browserController` is an instance of the
+ * [`BrowserController`](https://github.com/apify/browser-pool#browsercontroller),
+ * `response` is an instance of the `Puppeteer`
+ * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response) or `Playwright`
+ * [`Response`](https://playwright.dev/docs/api/class-response),
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * {@link Request#pushErrorMessage} function.
+ * @property {number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+ * @property {function} [gotoFunction]
+ * Navigation function for corresponding library. `page.goto(url)` is supported by both `playwright` and `puppeteer`.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * browserPool: BrowserPool,
+ * autoscaledPool: AutoscaledPool,
+ * session: Session,
+ * browserController: BrowserController,
+ * proxyInfo: ProxyInfo,
+ * }
+ * ```
+ * Where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+* @property {BrowserPoolOptions} [browserPoolOptions]
+ * Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor.
+ * You can tweak those to fine-tune browser management.
+ * @property {boolean} [persistCookiesPerSession=false]
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ * @property {ProxyConfiguration} [proxyConfiguration]
+ * If set, `PuppeteerCrawler` will be configured for all connections to use
+ * [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and rotated according to the configuration.
+ * For more information, see the [documentation](https://docs.apify.com/proxy).
+ * @property {Array<function>} [preNavigationHooks]
+ * Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
+ * or browser properties before navigation. The function accepts two parameters, `crawlingContext` and `gotoOptions`,
+ * which are passed to the `gotoFunction` the crawler calls to navigate.
+ * Example:
+ * ```
+ * preNavigationHooks: [
+ * async (crawlingContext, gotoOptions) => {
+ * await page.evaluate((attr) => { window.foo = attr; }, 'bar');
+ * }
+ * ]
+ * ```
+ * @property {Array<function>} [postNavigationHooks]
+ * Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
+ * The function accepts `crawlingContext` as an only parameter.
+ * Example:
+ * ```
+ * postNavigationHooks: [
+ * async (crawlingContext) => {
+ * const { page } = crawlingContext;
+ * if (hasCaptcha(page)) {
+ * await solveCaptcha (page);
+ * }
+ * };
+ * ]
+ * ```
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {number} [handleRequestTimeoutSecs=60]
+ * Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * error: Error,
+ * session: Session,
+ * crawler: BrowserCrawler,
+ * }
+ * ```
+ * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ *
+ * See
+ * [source code](https://github.com/apify/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)
+ * for the default implementation of this function.
+ * @property {number} [maxRequestRetries=3]
+ * Indicates how many times the request is retried if {@link BasicCrawlerOptions.handleRequestFunction} fails.
+ * @property {number} [maxRequestsPerCrawl]
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
+ * @property {number} [minConcurrency=1]
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {number} [maxConcurrency=1000]
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @property {boolean} [useSessionPool=false]
+ * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
+ * The session instance will be than available in the `handleRequestFunction`.
+ * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {@link SessionPool} to use.
+ */
+
+/**
+ * Provides a simple framework for parallel crawling of web pages
+ * using headless browsers with [Puppeteer](https://github.com/puppeteer/puppeteer)
+ * and [Playwright](https://github.com/microsoft/playwright).
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `BrowserCrawler` uses headless or even headfull browsers to download web pages and extract data,
+ * it is useful for crawling of websites that require to execute JavaScript.
+ * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * which downloads the pages using raw HTTP requests and is about 10x faster. @TODO: more?
+ *
+ * The source URLs are represented using {@link Request} objects that are fed from
+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link BrowserCrawlerOptions.requestList}
+ * or {@link BrowserCrawlerOptions.requestQueue} constructor options, respectively.
+ *
+ * If both {@link BrowserCrawlerOptions.requestList} and {@link BrowserCrawlerOptions.requestQueue} are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes when there are no more {@link Request} objects to crawl.
+ *
+ * `BrowserCrawler` opens a new browser page (i.e. tab or window) for each {@link Request} object to crawl
+ * and then calls the function provided by user as the {@link BrowserCrawlerOptions.handlePageFunction} option.
+ *
+ * New pages are only opened when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the {@link BrowserCrawlerOptions.autoscaledPoolOptions}
+ * parameter of the `BrowserCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPoolOptions} are available directly in the `BrowserCrawler` constructor.
+ *
+ * Note that the pool of browser instances is internally managed by the {@link BrowserPool} class.
+ *
+ * **Example usage:**
+ * @TODO:
+ * ```javascript
+ * const crawler = new Apify.PuppeteerCrawler({
+ * requestList,
+ * handlePageFunction: async ({ page, request }) => {
+ * // This function is called to extract data from a single web page
+ * // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
+ * // 'request' is an instance of Request class with information about the page to load
+ * await Apify.pushData({
+ * title: await page.title(),
+ * url: request.url,
+ * succeeded: true,
+ * })
+ * },
+ * handleFailedRequestFunction: async ({ request }) => {
+ * // This function is called when the crawling of a request failed too many times
+ * await Apify.pushData({
+ * url: request.url,
+ * succeeded: false,
+ * errors: request.errorMessages,
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ * @property {Statistics} stats
+ * Contains statistics about the current run.
+ * @property {?RequestList} requestList
+ * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?RequestQueue} requestQueue
+ * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?SessionPool} sessionPool
+ * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session}s.
+ * Only available if used by the crawler.
+ * @property {?ProxyConfiguration} proxyConfiguration
+ * A reference to the underlying {@link ProxyConfiguration} class that manages the crawler's proxies.
+ * Only available if used by the crawler.
+ * @property {BrowserPool} browserPool
+ * A reference to the underlying `BrowserPool` class that manages the crawler's browsers.
+ * For more information about it, see the [`browser-pool` module](https://github.com/apify/browser-pool).
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
+ */
+class BrowserCrawler extends BasicCrawler {
+ static optionsShape = {
+ ...BasicCrawler.optionsShape,
+ // TODO temporary until the API is unified in V2
+ handleRequestFunction: ow.undefined,
+
+ handlePageFunction: ow.function,
+ gotoFunction: ow.optional.function,
+
+ handlePageTimeoutSecs: ow.optional.number,
+ preNavigationHooks: ow.optional.array,
+ postNavigationHooks: ow.optional.array,
+
+ browserPoolOptions: ow.object,
+ sessionPoolOptions: ow.optional.object,
+ persistCookiesPerSession: ow.optional.boolean,
+ useSessionPool: ow.optional.boolean,
+ proxyConfiguration: ow.optional.object.validate(validators.proxyConfiguration),
+ };
+
+ /**
+ * @param {BrowserCrawlerOptions} options
+ * All `BrowserCrawler` parameters are passed via an options object.
+ */
+ constructor(options) {
+ ow(options, 'BrowserCrawlerOptions', ow.object.exactShape(BrowserCrawler.optionsShape));
+ const {
+ handlePageFunction,
+ handlePageTimeoutSecs = 60,
+ gotoFunction,
+ persistCookiesPerSession = true,
+ useSessionPool = true,
+ sessionPoolOptions,
+ proxyConfiguration,
+ browserPoolOptions,
+ preNavigationHooks = [],
+ postNavigationHooks = [],
+ ...basicCrawlerOptions
+ } = options;
+
+ if (!useSessionPool && persistCookiesPerSession) {
+ // @TODO: Maybe we could also automatically set persistCookiesPerSession to false when useSessionPool is false and log warning
+ throw new Error('You cannot use "persistCookiesPerSession" without "useSessionPool" set to true.');
+ }
+
+ super({
+ ...basicCrawlerOptions,
+ useSessionPool: false,
+ handleRequestFunction: (...args) => this._handleRequestFunction(...args),
+ handleRequestTimeoutSecs: handlePageTimeoutSecs * BASIC_CRAWLER_TIMEOUT_MULTIPLIER,
+ });
+
+ this.handlePageFunction = handlePageFunction;
+ this.handlePageTimeoutSecs = handlePageTimeoutSecs;
+ this.handlePageTimeoutMillis = this.handlePageTimeoutSecs * 1000;
+
+ this.gotoFunction = gotoFunction;
+ this.gotoOptions = {};
+
+ this.persistCookiesPerSession = persistCookiesPerSession;
+ this.proxyConfiguration = proxyConfiguration;
+
+ this.preNavigationHooks = preNavigationHooks;
+ this.postNavigationHooks = postNavigationHooks;
+
+ if (useSessionPool) {
+ this.sessionPool = new SessionPool({
+ ...sessionPoolOptions,
+ log: this.log,
+ });
+
+ // Assuming there are not more thant 20 browsers running at once;
+ this.sessionPool.setMaxListeners(20);
+ }
+
+ const { preLaunchHooks = [], postLaunchHooks = [], ...rest } = browserPoolOptions;
+ this.browserPool = new BrowserPool({
+ ...rest,
+ preLaunchHooks: [
+ this._extendLaunchContext.bind(this),
+ ...preLaunchHooks,
+ ],
+ postLaunchHooks: [
+ this._maybeAddSessionRetiredListener.bind(this),
+ ...postLaunchHooks,
+ ],
+ });
+ }
+
+ /**
+ * Wrapper around handlePageFunction that opens and closes pages etc.
+ *
+ * @param {Object} crawlingContext
+ * @param {Request} crawlingContext.request
+ * @param {BrowserController} crawlingContext.browserController
+ * @param {Session} [crawlingContext.session]
+ * @private
+ */
+ async _handleRequestFunction(crawlingContext) {
+ const { id } = crawlingContext;
+ const page = await this.browserPool.newPage({ id });
+ this._enhanceCrawlingContextWithPageInfo(crawlingContext, page);
+
+ const { request, session } = crawlingContext;
+
+ if (this.persistCookiesPerSession) {
+ const cookies = crawlingContext.session.getPuppeteerCookies(request.url);
+ await crawlingContext.browserController.setCookies(page, cookies);
+ }
+
+ try {
+ await this._handleNavigation(crawlingContext);
+
+ await this._responseHandler(crawlingContext);
+
+ // save cookies
+ // @TODO: Should we save the cookies also after/only the handle page?
+ if (this.persistCookiesPerSession) {
+ const cookies = await crawlingContext.browserController.getCookies(page);
+ session.setPuppeteerCookies(cookies, request.loadedUrl);
+ }
+
+ await addTimeoutToPromise(
+ this.handlePageFunction(crawlingContext),
+ this.handlePageTimeoutMillis,
+ `handlePageFunction timed out after ${this.handlePageTimeoutMillis / 1000} seconds.`,
+ );
+
+ if (session) session.markGood();
+ } finally {
+ try {
+ await page.close();
+ } catch (error) {
+ // Only log error in page close.
+ this.log.debug('Error while closing page', { error });
+ }
+ }
+ }
+
+ /**
+ *
+ * @param {object} crawlingContext
+ * @param {Page} page
+ * @private
+ */
+ _enhanceCrawlingContextWithPageInfo(crawlingContext, page) {
+ crawlingContext.page = page;
+
+ // This is the wierd spam because of browser to proxy not page to proxy.
+ const browserControllerInstance = this.browserPool.getBrowserControllerByPage(page);
+ crawlingContext.browserController = browserControllerInstance;
+
+ crawlingContext.session = browserControllerInstance.launchContext.session;
+ crawlingContext.proxyInfo = browserControllerInstance.launchContext.proxyInfo;
+
+ crawlingContext.crawler = this;
+ }
+
+ /**
+ *
+ * @param {object} crawlingContext
+ * @private
+ */
+ async _handleNavigation(crawlingContext) {
+ // @TODO: consider deep clone
+ const gotoOptions = _.clone(this.gotoOptions);
+ await this._executeHooks(this.preNavigationHooks, crawlingContext, gotoOptions);
+ crawlingContext.response = await this._navigationHandler(crawlingContext, gotoOptions);
+
+ await this._executeHooks(this.postNavigationHooks, crawlingContext, gotoOptions);
+ }
+
+ /**
+ *
+ * @param {object} crawlingContext
+ * @param {object} gotoOptions
+ * @private
+ */
+ async _navigationHandler(crawlingContext, gotoOptions) {
+ if (!this.gotoFunction) {
+ // @TODO: although it is optional in the validation,
+ // because when you make automation library specific you can override this handler.
+ throw new Error('BrowserCrawler: You must specify a gotoFunction!');
+ }
+ return this.gotoFunction(crawlingContext, gotoOptions);
+ }
+
+ /**
+ * Should be overriden in case of different automation library that does not support this response API.
+ * // @TODO: This can be also done as a postNavigation hook except the loadedUrl marking.
+ * @param crawlingContext
+ * @return {Promise<void>}
+ * @private
+ */
+ async _responseHandler(crawlingContext) {
+ const { response, session, request, page } = crawlingContext;
+
+ if (this.sessionPool && response) {
+ if (typeof response === 'object' && typeof response.status === 'function') {
+ throwOnBlockedRequest(session, response.status());
+ } else {
+ this.log.debug('Got a malformed Browser response.', { request, response });
+ }
+ }
+
+ request.loadedUrl = await page.url();
+ }
+
+ /**
+ *
+ * @param {string} pageId
+ * @param {object} launchContext
+ * @private
+ */
+ async _extendLaunchContext(pageId, launchContext) {
+ const launchContextExtends = {};
+
+ if (this.sessionPool) {
+ launchContextExtends.session = await this.sessionPool.getSession();
+ }
+
+ if (this.proxyConfiguration) {
+ const proxyInfo = await this.proxyConfiguration.newProxyInfo(launchContextExtends.session && launchContextExtends.session.id);
+ launchContext.proxyUrl = proxyInfo.url;
+ launchContextExtends.proxyInfo = proxyInfo;
+ }
+
+ launchContext.extend(launchContextExtends);
+ }
+
+ /**
+ *
+ * @param {string} pageId
+ * @param {BrowserController} browserController
+ * @private
+ */
+ _maybeAddSessionRetiredListener(pageId, browserController) {
+ if (this.sessionPool) {
+ const listener = (session) => {
+ const { launchContext } = browserController;
+ if (session.id === launchContext.session.id) {
+ launchContext.extend({ sessionRetired: true }); // @TODO: kind of dirty trick done mainly for testing this important feature.
+ this.browserPool.retireBrowserController(browserController); //eslint-disable-line
+ }
+ };
+
+ this.sessionPool.on(EVENTS.SESSION_RETIRED, listener);
+ browserController.on('browserClosed', () => this.sessionPool.removeListener(EVENTS.SESSION_RETIRED, listener));
+ }
+ }
+
+ /**
+ *
+ * @param {array} hooks
+ * @param {...any} args
+ * @private
+ */
+ async _executeHooks(hooks, ...args) {
+ if (Array.isArray(hooks) && hooks.length) {
+ for (const hook of hooks) {
+ await hook(...args);
+ }
+ }
+ }
+
+ /**
+ * Function for cleaning up after all request are processed.
+ * @ignore
+ */
+ async teardown() {
+ await this.browserPool.destroy();
+ super.teardown();
+ }
+}
+
+export default BrowserCrawler;
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index c9206d753..de2cf6070 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -70,14 +70,14 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* // An instance of Node's http.IncomingMessage object,
* response: Object,
*
- * // Underlying AutoscaledPool instance used to manage the concurrency of crawler
- * autoscaledPool: AutoscaledPool,
- *
* // Session object, useful to work around anti-scraping protections
* session: Session
*
* // ProxyInfo object with information about currently used proxy
* proxyInfo: ProxyInfo
+ *
+ * // The running cheerio crawler instance.
+ * crawler: CheerioCrawler
* }
* ```
*
@@ -117,9 +117,9 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* ```
* {
* request: Request,
- * autoscaledPool: AutoscaledPool,
* session: Session,
* proxyInfo: ProxyInfo,
+ * crawler: CheerioCrawler,
* }
* ```
* where the {@link Request} instance corresponds to the initialized request
@@ -151,9 +151,9 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* {
* response: Object,
* request: Request,
- * autoscaledPool: AutoscaledPool,
* session: Session,
* proxyInfo: ProxyInfo,
+ * crawler: CheerioCrawler,
* }
* ```
* The response is an instance of Node's http.IncomingMessage object.
@@ -175,15 +175,14 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
* {
* error: Error,
* request: Request,
- * autoscaledPool: AutoscaledPool,
* session: Session,
* $: Cheerio,
* body: String|Buffer,
* json: Object,
- * request: Request,
* contentType: Object,
* response: Object,
- * proxyInfo: ProxyInfo
+ * proxyInfo: ProxyInfo,
+ * crawler: CheerioCrawler,
* }
* ```
* where the {@link Request} instance corresponds to the failed request, and the `Error` instance
@@ -319,6 +318,20 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {
*
* await crawler.run();
* ```
+ * @property {Statistics} stats
+ * Contains statistics about the current run.
+ * @property {?RequestList} requestList
+ * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?RequestQueue} requestQueue
+ * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?SessionPool} sessionPool
+ * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session}s.
+ * Only available if used by the crawler.
+ * @property {?ProxyConfiguration} proxyConfiguration
+ * A reference to the underlying {@link ProxyConfiguration} class that manages the crawler's proxies.
+ * Only available if used by the crawler.
* @property {AutoscaledPool} autoscaledPool
* A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
* Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
@@ -455,7 +468,6 @@ class CheerioCrawler extends BasicCrawler {
*
* @param {Object} crawlingContext
* @param {Request} crawlingContext.request
- * @param {AutoscaledPool} crawlingContext.autoscaledPool
* @param {Session} [crawlingContext.session]
* @ignore
*/
@@ -730,10 +742,10 @@ export default CheerioCrawler;
* Original instance fo the {Request} object. Must be modified in-place.
* @property {Session} [session]
* The current session
- * @property {AutoscaledPool} autoscaledPool
* @property {ProxyInfo} [proxyInfo]
* An object with information about currently used proxy by the crawler
* and configured by the {@link ProxyConfiguration} class.
+ * @property {CheerioCrawler} [crawler]
*/
/**
@@ -747,12 +759,12 @@ export default CheerioCrawler;
* @property {IncomingMessage|Readable} response stream
* @property {Request} request
* Original instance fo the {Request} object. Must be modified in-place.
- * @property {AutoscaledPool} autoscaledPool
* @property {Session} [session]
* The current session
* @property {ProxyInfo} [proxyInfo]
* An object with information about currently used proxy by the crawler
* and configured by the {@link ProxyConfiguration} class.
+ * @property {CheerioCrawler} [crawler]
*/
/**
@@ -775,16 +787,11 @@ export default CheerioCrawler;
* Parsed `Content-Type header: { type, encoding }`.
* @property {IncomingMessage} response
* An instance of Node's http.IncomingMessage object,
- * @property {AutoscaledPool} autoscaledPool
- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
- * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
- * You can use it to change the concurrency settings on the fly,
- * to pause the crawler by calling {@link AutoscaledPool#pause}
- * or to abort it by calling {@link AutoscaledPool#abort}.
* @property {Session} [session]
* @property {ProxyInfo} [proxyInfo]
* An object with information about currently used proxy by the crawler
* and configured by the {@link ProxyConfiguration} class.
+ * @property {CheerioCrawler} [crawler]
*/
/**
diff --git a/src/crawlers/crawler_utils.js b/src/crawlers/crawler_utils.js
new file mode 100644
index 000000000..0b6c31699
--- /dev/null
+++ b/src/crawlers/crawler_utils.js
@@ -0,0 +1,40 @@
+import { URL } from 'url';
+
+/**
+ * Handles timeout request
+ * @param {Session} session
+ * @param {string} errorMessage
+ * @private
+ */
+export function handleRequestTimeout(session, errorMessage) {
+ if (session) session.markBad();
+ const timeoutMillis = errorMessage.match(/(\d+)\s?ms/)[1]; // first capturing group
+ const timeoutSecs = Number(timeoutMillis) / 1000;
+ throw new Error(`gotoFunction timed out after ${timeoutSecs} seconds.`);
+}
+
+/**
+ * Handles blocked request
+ * @param {Session} session
+ * @param {number} statusCode
+ * @private
+ */
+export function throwOnBlockedRequest(session, statusCode) {
+ const isBlocked = session.retireOnBlockedStatusCodes(statusCode);
+
+ if (isBlocked) {
+ throw new Error(`Request blocked - received ${statusCode} status code.`);
+ }
+}
+
+export function getSessionIdFromProxyUrl(proxyUrl) {
+ const parsedUrl = new URL(proxyUrl);
+ const { username } = parsedUrl.username;
+ if (!username) {
+ return;
+ }
+ const parts = username.split(',');
+ const sessionPart = parts.find((part) => part.includes('session-'));
+
+ return sessionPart && sessionPart.replace('session-', '');
+}
diff --git a/src/crawlers/playwright_crawler.js b/src/crawlers/playwright_crawler.js
new file mode 100644
index 000000000..e5c6ec980
--- /dev/null
+++ b/src/crawlers/playwright_crawler.js
@@ -0,0 +1,282 @@
+import { PlaywrightPlugin } from 'browser-pool';
+import ow from 'ow';
+import BrowserCrawler from './browser_crawler';
+import { handleRequestTimeout } from './crawler_utils';
+import { gotoExtended } from '../playwright_utils';
+import { apifyOptionsToLaunchOptions, getPlaywrightLauncherOrThrow } from '../playwright';
+
+/**
+ * @typedef PlaywrightCrawlerOptions
+ * @property {object} playwrightModule
+ * Playwright browser module used for launching new browsers. For example: `require("playwright").chromium`
+ * [Playwright list of available modules](https://playwright.dev/docs/api/class-playwright)
+ * @property {function} handlePageFunction
+ * Function that is called to process each request.
+ * It is passed an object with the following fields:
+ *
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * session: Session,
+ * browserController: BrowserController,
+ * proxyInfo: ProxyInfo,
+ * crawler: PlaywrightCrawler,
+ * }
+ * ```
+ *
+ * `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * `page` is an instance of the `Playwright`
+ * [`Page`](https://playwright.dev/docs/api/class-page)
+ * `browserPool` is an instance of the
+ * [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool),
+ * `browserController` is an instance of the
+ * [`BrowserController`](https://github.com/apify/browser-pool#browsercontroller),
+ * `response` is an instance of the `Playwright`
+ * [`Response`](https://playwright.dev/docs/api/class-response),
+ * which is the main resource response as returned by `page.goto(request.url)`.
+ * The function must return a promise, which is then awaited by the crawler.
+ *
+ * If the function throws an exception, the crawler will try to re-crawl the
+ * request later, up to `option.maxRequestRetries` times.
+ * If all the retries fail, the crawler calls the function
+ * provided to the `handleFailedRequestFunction` parameter.
+ * To make this work, you should **always**
+ * let your function throw exceptions rather than catch them.
+ * The exceptions are logged to the request using the
+ * {@link Request#pushErrorMessage} function.
+ * @property {number} [gotoTimeoutSecs=60]
+ * Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
+ * function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
+ * @property {HandleFailedRequest} [handleFailedRequestFunction]
+ * A function to handle requests that failed more than `option.maxRequestRetries` times.
+ *
+ * The function receives the following object as an argument:
+ * ```
+ * {
+ * request: Request,
+ * response: Response,
+ * page: Page,
+ * session: Session,
+ * browserController: BrowserController,
+ * proxyInfo: ProxyInfo,
+ * crawler: PlaywrightCrawler,
+ * }
+ * ```
+ * Where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+ * represents the last error thrown during processing of the request.
+ * @property {Array<function>} [preNavigationHooks]
+ * Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
+ * or browser properties before navigation. The function accepts two parameters, `crawlingContext` and `gotoOptions`,
+ * which are passed to the `gotoFunction` the crawler calls to navigate.
+ * Example:
+ * ```
+ * preNavigationHooks: [
+ * async (crawlingContext, gotoOptions) => {
+ * await page.evaluate((attr) => { window.foo = attr; }, 'bar');
+ * }
+ * ]
+ * ```
+ * @property {Array<function>} [postNavigationHooks]
+ * Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
+ * The function accepts `crawlingContext` as an only parameter.
+ * Example:
+ * ```
+ * postNavigationHooks: [
+ * async (crawlingContext) => {
+ * const { page } = crawlingContext;
+ * if (hasCaptcha(page)) {
+ * await solveCaptcha (page);
+ * }
+ * };
+ * ]
+ * ```
+* @property {object} [launchContext]
+ * Options used by {@link Apify#launchPlaywright} to start new Playwright instances.
+ * * @property {number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+ * @property {BrowserPoolOptions} [browserPoolOptions]
+ * Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor.
+ * You can tweak those to fine-tune browser management.
+ * @property {boolean} [persistCookiesPerSession=false]
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ * @property {ProxyConfiguration} [proxyConfiguration]
+ * If set, `PlaywrightCrawler` will be configured for all connections to use
+ * [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and rotated according to the configuration.
+ * For more information, see the [documentation](https://docs.apify.com/proxy).
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {number} [handleRequestTimeoutSecs=60]
+ * Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.
+ * @property {number} [maxRequestRetries=3]
+ * Indicates how many times the request is retried if {@link PuppeteerCrawlerOptions.handlePageFunction} fails.
+ * @property {number} [maxRequestsPerCrawl]
+ * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+ * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+ * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+ * @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
+ * @property {number} [minConcurrency=1]
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ *
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+ * @property {number} [maxConcurrency=1000]
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @property {boolean} [useSessionPool=false]
+ * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
+ * The session instance will be than available in the `handleRequestFunction`.
+ * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {@link SessionPool} to use.
+ */
+
+/**
+ * Provides a simple framework for parallel crawling of web pages
+ * using headless Chromium, Firefox and Webkit browsers with [Playwright](https://github.com/microsoft/playwright).
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `Playwright` uses headless browser to download web pages and extract data,
+ * it is useful for crawling of websites that require to execute JavaScript.
+ * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * which downloads the pages using raw HTTP requests and is about 10x faster.
+ *
+ * The source URLs are represented using {@link Request} objects that are fed from
+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link BasicCrawlerOptions.requestList}
+ * or {@link BasicCrawlerOptions.requestQueue} constructor options, respectively.
+ *
+ * If both {@link BasicCrawlerOptions.requestList} and {@link BasicCrawlerOptions.requestQueue} are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes when there are no more {@link Request} objects to crawl.
+ *
+ * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
+ * and then calls the function provided by user as the {@link PlaywrightCrawlerOptions.handlePageFunction} option.
+ *
+ * New pages are only opened when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the {@link BasicCrawlerOptions.autoscaledPoolOptions}
+ * parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPoolOptions} are available directly in the `PuppeteerCrawler` constructor.
+ *
+ * Note that the pool of Puppeteer instances is internally managed by the {@link BrowserPool} class.
+ * Many constructor options such as {@link PuppeteerPoolOptions.maxOpenPagesPerInstance} or
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const crawler = new Apify.PuppeteerCrawler({
+ * requestList,
+ * handlePageFunction: async ({ page, request }) => {
+ * // This function is called to extract data from a single web page
+ * // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
+ * // 'request' is an instance of Request class with information about the page to load
+ * await Apify.pushData({
+ * title: await page.title(),
+ * url: request.url,
+ * succeeded: true,
+ * })
+ * },
+ * handleFailedRequestFunction: async ({ request }) => {
+ * // This function is called when the crawling of a request failed too many times
+ * await Apify.pushData({
+ * url: request.url,
+ * succeeded: false,
+ * errors: request.errorMessages,
+ * })
+ * },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ * @property {Statistics} stats
+ * Contains statistics about the current run.
+ * @property {?RequestList} requestList
+ * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?RequestQueue} requestQueue
+ * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?SessionPool} sessionPool
+ * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session}s.
+ * Only available if used by the crawler.
+ * @property {?ProxyConfiguration} proxyConfiguration
+ * A reference to the underlying {@link ProxyConfiguration} class that manages the crawler's proxies.
+ * Only available if used by the crawler.
+ * @property {BrowserPool} browserPool
+ * A reference to the underlying `BrowserPool` class that manages the crawler's browsers.
+ * For more information about it, see the [`browser-pool` module](https://github.com/apify/browser-pool).
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
+ */
+class PlaywrightCrawler extends BrowserCrawler {
+ static optionsShape = {
+ ...BrowserCrawler.optionsShape,
+ browserPoolOptions: ow.optional.object,
+ playwrightModule: ow.optional.object,
+ gotoTimeoutSecs: ow.optional.number,
+ launchOptions: ow.optional.object,
+ gotoFunction: ow.undefined,
+ }
+
+ constructor(options = {}) {
+ ow(options, 'PlaywrightCrawlerOptions', ow.object.exactShape(PlaywrightCrawler.optionsShape));
+
+ const {
+ playwrightModule,
+ launchContext = {},
+ gotoTimeoutSecs,
+ browserPoolOptions = {},
+ ...browserCrawlerOptions
+ } = options;
+
+ browserPoolOptions.browserPlugins = [
+ new PlaywrightPlugin(
+ // eslint-disable-next-line
+ getPlaywrightLauncherOrThrow(playwrightModule),
+ {
+ launchOptions: apifyOptionsToLaunchOptions(launchContext),
+ },
+ ),
+ ];
+
+ super({
+ ...browserCrawlerOptions,
+ browserPoolOptions,
+ });
+
+ this.browserPool.postLaunchHooks.push(({ error, session }) => {
+ // It would be better to compare the instances,
+ if (error && error.constructor.name === 'TimeoutError') {
+ handleRequestTimeout(session, error.message);
+ }
+ });
+
+ this.gotoTimeoutMillis = gotoTimeoutSecs * 1000;
+
+ this.launchContext = launchContext;
+ this.playwrightModule = playwrightModule;
+
+ this.gotoOptions = {
+ timeout: this.gotoTimeoutMillis,
+ };
+ }
+
+ async _navigationHandler(crawlingContext, gotoOptions) {
+ return gotoExtended(crawlingContext.page, crawlingContext.request, gotoOptions);
+ }
+}
+
+export default PlaywrightCrawler;
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 2342878dd..9c6b82da6 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -1,24 +1,11 @@
+import { PuppeteerPlugin } from 'browser-pool';
import ow from 'ow';
-import PuppeteerPool from '../puppeteer_pool'; // eslint-disable-line import/no-duplicates
-import { gotoExtended } from '../puppeteer_utils';
-import { addTimeoutToPromise } from '../utils';
-import BasicCrawler from './basic_crawler'; // eslint-disable-line import/no-duplicates
-// TYPE IMPORTS
-/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
-import { Browser, Page as PuppeteerPage, Response as PuppeteerResponse } from 'puppeteer';
-import { HandleFailedRequest } from './basic_crawler';
-import { PuppeteerPoolOptions, LaunchPuppeteerFunction } from '../puppeteer_pool';
-import Request from '../request'; // eslint-disable-line no-unused-vars
-import { RequestList } from '../request_list'; // eslint-disable-line no-unused-vars
-import { RequestQueue } from '../storages/request_queue'; // eslint-disable-line no-unused-vars
-import AutoscaledPool, { AutoscaledPoolOptions } from '../autoscaling/autoscaled_pool'; // eslint-disable-line no-unused-vars,import/named
-import { LaunchPuppeteerOptions } from '../puppeteer'; // eslint-disable-line no-unused-vars,import/named
-import { Session } from '../session_pool/session'; // eslint-disable-line no-unused-vars
-import { SessionPoolOptions } from '../session_pool/session_pool';
-import { ProxyConfiguration, ProxyInfo } from '../proxy_configuration';
-import { validators } from '../validators';
-// eslint-enable-line import/no-duplicates
+import BrowserCrawler from './browser_crawler';
+import { handleRequestTimeout } from './crawler_utils';
+import { gotoExtended } from '../puppeteer_utils';
+import { apifyOptionsToLaunchOptions, getPuppeteerOrThrow } from '../puppeteer';
+import applyStealthToBrowser from '../stealth/stealth';
/**
* @typedef PuppeteerCrawlerOptions
@@ -31,21 +18,23 @@ import { validators } from '../validators';
* request: Request,
* response: Response,
* page: Page,
- * puppeteerPool: PuppeteerPool,
- * autoscaledPool: AutoscaledPool,
* session: Session,
+ * browserController: BrowserController,
* proxyInfo: ProxyInfo,
+ * crawler: PuppeteerCrawler,
* }
* ```
*
* `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
* `page` is an instance of the `Puppeteer`
* [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)
+ * `browserPool` is an instance of the
+ * [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool),
+ * `browserController` is an instance of the
+ * [`BrowserController`](https://github.com/apify/browser-pool#browsercontroller),
* `response` is an instance of the `Puppeteer`
* [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),
* which is the main resource response as returned by `page.goto(request.url)`.
- * `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
- *
* The function must return a promise, which is then awaited by the crawler.
*
* If the function throws an exception, the crawler will try to re-crawl the
@@ -56,25 +45,6 @@ import { validators } from '../validators';
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
* {@link Request#pushErrorMessage} function.
- * @property {RequestList} [requestList]
- * Static list of URLs to be processed.
- * Either `requestList` or `requestQueue` option must be provided (or both).
- * @property {RequestQueue} [requestQueue]
- * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * Either `requestList` or `requestQueue` option must be provided (or both).
- * @property {number} [handlePageTimeoutSecs=60]
- * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
- * @property {PuppeteerGoto} [gotoFunction]
- * Overrides the function that opens the page in Puppeteer. The function should return the result of Puppeteer's
- * [page.goto()](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options) function,
- * i.e. a `Promise` resolving to the [Response](https://pptr.dev/#?product=Puppeteer&show=api-class-httpresponse) object.
- *
- * This is useful if you need to select different criteria to determine navigation success and also to do any
- * pre or post processing such as injecting cookies into the page.
- *
- * Note that a single page object is only used to process a single request and it is closed afterwards.
- *
- * By default, the function invokes {@link puppeteer#gotoExtended} with a timeout of 60 seconds.
* @property {number} [gotoTimeoutSecs=60]
* Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default
* function is overridden, this timeout will not be used and needs to be configured in the new `gotoFunction()`.
@@ -83,62 +53,87 @@ import { validators } from '../validators';
*
* The function receives the following object as an argument:
* ```
- * {
- * error: Error,
+ * {
* request: Request,
* response: Response,
* page: Page,
- * puppeteerPool: PuppeteerPool,
- * autoscaledPool: AutoscaledPool,
* session: Session,
+ * browserController: BrowserController,
* proxyInfo: ProxyInfo,
+ * crawler: PuppeteerCrawler,
* }
* ```
* Where the {@link Request} instance corresponds to the failed request, and the `Error` instance
* represents the last error thrown during processing of the request.
+ * @property {object} [launchContext]
+ * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.
+ * @property {number} [handlePageTimeoutSecs=60]
+ * Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.
+ * @property {BrowserPoolOptions} [browserPoolOptions]
+ * Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor.
+ * You can tweak those to fine-tune browser management.
+ * @property {boolean} [persistCookiesPerSession=false]
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ * @property {ProxyConfiguration} [proxyConfiguration]
+ * If set, `PuppeteerCrawler` will be configured for all connections to use
+ * [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and rotated according to the configuration.
+ * For more information, see the [documentation](https://docs.apify.com/proxy).
+ * @property {Array<function>} [preNavigationHooks]
+ * Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
+ * or browser properties before navigation. The function accepts two parameters, `crawlingContext` and `gotoOptions`,
+ * which are passed to the `gotoFunction` the crawler calls to navigate.
+ * Example:
+ * ```
+ * preNavigationHooks: [
+ * async (crawlingContext, gotoOptions) => {
+ * await page.evaluate((attr) => { window.foo = attr; }, 'bar');
+ * }
+ * ]
+ * ```
+ * @property {Array<function>} [postNavigationHooks]
+ * Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
+ * The function accepts `crawlingContext` as an only parameter.
+ * Example:
+ * ```
+ * postNavigationHooks: [
+ * async (crawlingContext) => {
+ * const { page } = crawlingContext;
+ * if (hasCaptcha(page)) {
+ * await solveCaptcha (page);
+ * }
+ * };
+ * ]
+ * ```
+ * @property {RequestList} [requestList]
+ * Static list of URLs to be processed.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {RequestQueue} [requestQueue]
+ * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
+ * @property {number} [handleRequestTimeoutSecs=60]
+ * Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.
* @property {number} [maxRequestRetries=3]
- * Indicates how many times the request is retried if either `handlePageFunction()` or `gotoFunction()` fails.
+ * Indicates how many times the request is retried if {@link PuppeteerCrawlerOptions.handlePageFunction} fails.
* @property {number} [maxRequestsPerCrawl]
* Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
- * @property {PuppeteerPoolOptions} [puppeteerPoolOptions]
- * Custom options passed to the underlying {@link PuppeteerPool} constructor.
- * You can tweak those to fine-tune browser management.
- * @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
- * Overrides the default function to launch a new Puppeteer instance.
- * Shortcut to the corresponding {@link PuppeteerPool} option.
- * See source code on
- * [GitHub](https://github.com/apify/apify-js/blob/master/src/puppeteer_pool.js#L28)
- * for default behavior.
- * @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]
- * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.
- * Shortcut to the corresponding {@link PuppeteerPool} option.
* @property {AutoscaledPoolOptions} [autoscaledPoolOptions]
- * Custom options passed to the underlying {@link AutoscaledPool} instance constructor.
- * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
- * are provided by `PuppeteerCrawler` and should not be overridden.
+ * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+ * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * However, you can provide a custom implementation of `isFinishedFunction`.
* @property {number} [minConcurrency=1]
- * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the
- * corresponding {@link AutoscaledPoolOptions.minConcurrency} option.
+ * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
- * *WARNING:* If you set this value too high with respect to the available system memory and CPU,
- * your crawler will run extremely slow or crash. If you're not sure, just keep the default value
- * and the concurrency will scale up automatically.
+ * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+ * If you're not sure, just keep the default value and the concurrency will scale up automatically.
* @property {number} [maxConcurrency=1000]
- * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the
- * corresponding {@link AutoscaledPoolOptions.maxConcurrency} option.
+ * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
* @property {boolean} [useSessionPool=false]
- * If set to true Crawler will automatically use Session Pool. It will automatically retire
- * sessions on 403, 401 and 429 status codes. It also marks Session as bad after a request timeout.
- * @property {SessionPoolOptions} [sessionPoolOptions]
- * Custom options passed to the underlying {@link SessionPool} constructor.
- * @property {boolean} [persistCookiesPerSession=false]
- * Automatically saves cookies to Session. Works only if Session Pool is used.
- * @property {ProxyConfiguration} [proxyConfiguration]
- * If set, `PuppeteerCrawler` will be configured for all connections to use
- * [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and rotated according to the configuration.
- * For more information, see the [documentation](https://docs.apify.com/proxy).
+ * If set to true. Basic crawler will initialize the {@link SessionPool} with the corresponding `sessionPoolOptions`.
+ * The session instance will be than available in the `handleRequestFunction`.
+ * @property {SessionPoolOptions} [sessionPoolOptions] The configuration options for {@link SessionPool} to use.
*/
/**
@@ -153,27 +148,25 @@ import { validators } from '../validators';
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
* The source URLs are represented using {@link Request} objects that are fed from
- * {@link RequestList} or {@link RequestQueue} instances provided by the {@link PuppeteerCrawlerOptions.requestList}
- * or {@link PuppeteerCrawlerOptions.requestQueue} constructor options, respectively.
+ * {@link RequestList} or {@link RequestQueue} instances provided by the {@link BasicCrawlerOptions.requestList}
+ * or {@link BasicCrawlerOptions.requestQueue} constructor options, respectively.
*
- * If both {@link PuppeteerCrawlerOptions.requestList} and {@link PuppeteerCrawlerOptions.requestQueue} are used,
+ * If both {@link BasicCrawlerOptions.requestList} and {@link BasicCrawlerOptions.requestQueue} are used,
* the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
* to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
* The crawler finishes when there are no more {@link Request} objects to crawl.
*
* `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
- * and then calls the function provided by user as the {@link PuppeteerCrawlerOptions.handlePageFunction} option.
+ * and then calls the function provided by user as the {@link BrowserCrawlerOptions.handlePageFunction} option.
*
* New pages are only opened when there is enough free CPU and memory available,
* using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the {@link PuppeteerCrawlerOptions.autoscaledPoolOptions}
+ * All {@link AutoscaledPool} configuration options can be passed to the {@link BasicCrawlerOptions.autoscaledPoolOptions}
* parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
* {@link AutoscaledPoolOptions} are available directly in the `PuppeteerCrawler` constructor.
*
- * Note that the pool of Puppeteer instances is internally managed by the {@link PuppeteerPool} class.
- * Many constructor options such as {@link PuppeteerPoolOptions.maxOpenPagesPerInstance} or
- * {@link PuppeteerPoolOptions.launchPuppeteerFunction} are passed directly to the {@link PuppeteerPool} constructor.
+ * Note that the pool of Puppeteer instances is internally managed by the {@link BrowserPool} class.
*
* **Example usage:**
*
@@ -202,292 +195,114 @@ import { validators } from '../validators';
*
* await crawler.run();
* ```
+ * @property {Statistics} stats
+ * Contains statistics about the current run.
+ * @property {?RequestList} requestList
+ * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?RequestQueue} requestQueue
+ * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request}s.
+ * Only available if used by the crawler.
+ * @property {?SessionPool} sessionPool
+ * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session}s.
+ * Only available if used by the crawler.
+ * @property {?ProxyConfiguration} proxyConfiguration
+ * A reference to the underlying {@link ProxyConfiguration} class that manages the crawler's proxies.
+ * Only available if used by the crawler.
+ * @property {BrowserPool} browserPool
+ * A reference to the underlying `BrowserPool` class that manages the crawler's browsers.
+ * For more information about it, see the [`browser-pool` module](https://github.com/apify/browser-pool).
* @property {AutoscaledPool} autoscaledPool
* A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
- * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
* You can use it to change the concurrency settings on the fly,
* to pause the crawler by calling {@link AutoscaledPool#pause}
* or to abort it by calling {@link AutoscaledPool#abort}.
- *
*/
-class PuppeteerCrawler extends BasicCrawler {
+class PuppeteerCrawler extends BrowserCrawler {
static optionsShape = {
- ...BasicCrawler.optionsShape,
- // TODO temporary until the API is unified in V2
- handleRequestFunction: ow.undefined,
-
- handlePageFunction: ow.function,
- gotoFunction: ow.optional.function,
- handlePageTimeoutSecs: ow.optional.number,
+ ...BrowserCrawler.optionsShape,
+ browserPoolOptions: ow.optional.object,
gotoTimeoutSecs: ow.optional.number,
- proxyConfiguration: ow.optional.object.validate(validators.proxyConfiguration),
- persistCookiesPerSession: ow.optional.boolean,
+ launchContext: ow.optional.object,
+ gotoFunction: ow.undefined,
- // PuppeteerPool options and shorthands
- puppeteerPoolOptions: ow.optional.object,
- launchPuppeteerFunction: ow.optional.function,
- launchPuppeteerOptions: ow.optional.object,
}
/**
- * @param {PuppeteerCrawlerOptions} options
- * All `PuppeteerCrawler` parameters are passed via an options object.
- */
- constructor(options) {
- ow(options, ow.object.exactShape(PuppeteerCrawler.optionsShape));
+ * @param {PuppeteerCrawlerOptions} options
+ * All `PuppeteerCrawler` parameters are passed via an options object.
+ */
+ constructor(options = {}) {
+ ow(options, 'PuppeteerCrawlerOptions', ow.object.exactShape(PuppeteerCrawler.optionsShape));
const {
- handlePageFunction,
- gotoFunction,
- handlePageTimeoutSecs = 60,
- gotoTimeoutSecs = 60,
+ puppeteerModule, // eslint-disable-line
+ launchContext = {},
+ gotoTimeoutSecs,
+ browserPoolOptions = {},
proxyConfiguration,
- persistCookiesPerSession = false,
-
- // PuppeteerPool options and shorthands
- puppeteerPoolOptions,
- launchPuppeteerFunction,
- launchPuppeteerOptions,
-
- ...basicCrawlerOptions
+ ...browserCrawlerOptions
} = options;
- super({
- ...basicCrawlerOptions,
- // TODO temporary until the API is unified in V2
- handleRequestFunction: handlePageFunction,
- // We need to add some time for internal functions to finish,
- // but not too much so that we would stall the crawler.
- handleRequestTimeoutSecs: (gotoTimeoutSecs + handlePageTimeoutSecs * 2) + 5,
- });
+ const {
+ stealth,
+ stealthOptions,
+ proxyUrl,
+ } = launchContext;
- if (proxyConfiguration && (launchPuppeteerOptions && launchPuppeteerOptions.proxyUrl)) {
+ if (proxyUrl && proxyConfiguration) {
throw new Error('It is not possible to combine "options.proxyConfiguration" together with '
+ 'custom "proxyUrl" option from "options.launchPuppeteerOptions".');
}
+ browserPoolOptions.browserPlugins = [
+ new PuppeteerPlugin(
+ getPuppeteerOrThrow(puppeteerModule),
+ {
+ proxyUrl,
+ launchOptions: apifyOptionsToLaunchOptions(launchContext),
+ },
+ ),
+ ];
- if (persistCookiesPerSession && !this.useSessionPool) {
- throw new Error('Cannot use "options.persistCookiesPerSession" without "options.useSessionPool"');
- }
-
- if (options.gotoTimeoutSecs && options.gotoFunction) {
- this.log.warning('You are using gotoTimeoutSecs with a custom gotoFunction. '
- + 'The timeout value will not be used. With a custom gotoFunction, you need to set the timeout in the function itself.');
- }
-
- this.gotoFunction = gotoFunction;
-
- this.handlePageTimeoutMillis = handlePageTimeoutSecs * 1000;
- this.gotoTimeoutMillis = gotoTimeoutSecs * 1000;
-
- this.puppeteerPoolOptions = {
- ...puppeteerPoolOptions,
- launchPuppeteerFunction,
- launchPuppeteerOptions,
+ super({
+ ...browserCrawlerOptions,
proxyConfiguration,
- log: this.log,
- };
-
- this.puppeteerPool = null; // Constructed when .run()
- this.persistCookiesPerSession = persistCookiesPerSession;
- this.proxyConfiguration = proxyConfiguration;
- }
-
- /**
- * Runs the crawler. Returns promise that gets resolved once all the requests got processed.
- *
- * @return {Promise<void>}
- */
- async run() {
- if (this.isRunningPromise) return this.isRunningPromise;
-
- await this._init();
-
- if (this.sessionPool) {
- this.puppeteerPoolOptions.sessionPool = this.sessionPool;
- }
- this.puppeteerPool = new PuppeteerPool(this.puppeteerPoolOptions);
-
- try {
- await super.run();
- } finally {
- await this.puppeteerPool.destroy();
- }
- }
-
- /**
- * Wrapper around handlePageFunction that opens and closes pages etc.
- *
- * @param {Object} crawlingContext
- * @param {Request} crawlingContext.request
- * @param {AutoscaledPool} crawlingContext.autoscaledPool
- * @param {Session} [crawlingContext.session]
- * @ignore
- */
- async _handleRequestFunction(crawlingContext) {
- crawlingContext.page = await this.puppeteerPool.newPage();
-
- const { page, request } = crawlingContext;
- // eslint-disable-next-line no-underscore-dangle
- const browserInstance = this.puppeteerPool._getBrowserInstance(page);
- if (this.sessionPool) {
- crawlingContext.session = browserInstance.session;
+ browserPoolOptions,
+ });
- // setting cookies to page
- if (this.persistCookiesPerSession) {
- await page.setCookie(...crawlingContext.session.getPuppeteerCookies(request.url));
+ this.browserPool.postLaunchHooks.push(({ error, session }) => {
+ // It would be better to compare the instances,
+ // but we don't have access to puppeteer.errors here.
+ if (error && error.constructor.name === 'TimeoutError') {
+ handleRequestTimeout(session, error.message);
}
- }
-
- const { session } = crawlingContext;
+ });
- if (this.proxyConfiguration) {
- crawlingContext.proxyInfo = browserInstance.proxyInfo;
+ if (proxyUrl) {
+ this.log.deprecated('options.launchPuppeteerOptions.proxyUrl is deprecated use the options.proxyConfiguration instead');
}
- try {
- let response;
- try {
- response = await this._navigationHandler(crawlingContext);
- } catch (err) {
- // It would be better to compare the instances,
- // but we don't have access to puppeteer.errors here.
- if (err.constructor.name === 'TimeoutError') {
- this._handleRequestTimeout(session, err.message);
- }
- }
-
- if (this.useSessionPool && response) {
- if (typeof response === 'object' && typeof response.status === 'function') {
- this._throwOnBlockedRequest(session, response.status());
- } else {
- this.log.debug('Got a malformed Puppeteer response.', { request, response });
- }
- }
-
- await this.puppeteerPool.serveLiveViewSnapshot(page);
- request.loadedUrl = page.url();
-
- // save cookies
- if (this.persistCookiesPerSession) {
- const cookies = await page.cookies(request.loadedUrl);
- session.setPuppeteerCookies(cookies, request.loadedUrl);
- }
-
- crawlingContext.response = response;
- crawlingContext.puppeteerPool = this.puppeteerPool;
-
- await addTimeoutToPromise(
- this.userProvidedHandler(crawlingContext),
- this.handlePageTimeoutMillis,
- `handlePageFunction timed out after ${this.handlePageTimeoutMillis / 1000} seconds.`,
- );
-
- if (session) session.markGood();
- } finally {
- await this.puppeteerPool.recyclePage(page);
+ if (stealth) {
+ this.browserPool.postLaunchHooks.push(async (pageId, browserController) => {
+ // @TODO: We can do this better now. It is not necessary to override the page.
+ // we can modify the page in the postPageCreateHook
+ await applyStealthToBrowser(browserController.browser, stealthOptions);
+ });
}
- }
- /**
- * @param {Object} crawlingContext
- * @param {PuppeteerPage} crawlingContext.page
- * @param {Request} crawlingContext.request
- * @property {AutoscaledPool} autoscaledPool
- * @property {PuppeteerPool} puppeteerPool
- * @property {Session} [session]
- * @property {ProxyInfo} [proxyInfo]
- * @return {Promise<PuppeteerResponse>}
- * @ignore
- */
- async _navigationHandler(crawlingContext) {
- if (this.gotoFunction) return this.gotoFunction(crawlingContext);
- return gotoExtended(crawlingContext.page, crawlingContext.request, { timeout: this.gotoTimeoutMillis });
- }
+ this.gotoTimeoutMillis = gotoTimeoutSecs * 1000;
+ this.launchContext = launchContext;
+ this.puppeteerModule = puppeteerModule;
- /**
- * Handles timeout request
- * @param {Session} session
- * @param {string} errorMessage
- * @private
- */
- _handleRequestTimeout(session, errorMessage) {
- if (session) session.markBad();
- const timeoutMillis = errorMessage.match(/(\d+) ms/)[1]; // first capturing group
- const timeoutSecs = Number(timeoutMillis) / 1000;
- throw new Error(`gotoFunction timed out after ${timeoutSecs} seconds.`);
+ this.gotoOptions = {
+ timeout: this.gotoTimeoutMillis,
+ };
}
- /**
- * Handles blocked request
- * @param {Session} session
- * @param {number} statusCode
- * @private
- */
- _throwOnBlockedRequest(session, statusCode) {
- const isBlocked = session.retireOnBlockedStatusCodes(statusCode);
-
- if (isBlocked) {
- throw new Error(`Request blocked - received ${statusCode} status code.`);
- }
+ async _navigationHandler(crawlingContext, gotoOptions) {
+ return gotoExtended(crawlingContext.page, crawlingContext.request, gotoOptions);
}
}
export default PuppeteerCrawler;
-
-/**
- * @typedef PuppeteerHandlePageInputs
- * @property {Request} request
- * An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
- * @property {PuppeteerResponse} response An instance of the Puppeteer
- * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),
- * which is the main resource response as returned by `page.goto(request.url)`.
- * @property {PuppeteerPage} page is an instance of the Puppeteer
- * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)
- * @property {PuppeteerPool} puppeteerPool
- * An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
- * @property {AutoscaledPool} autoscaledPool
- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
- * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.
- * You can use it to change the concurrency settings on the fly,
- * to pause the crawler by calling {@link AutoscaledPool#pause}
- * or to abort it by calling {@link AutoscaledPool#abort}.
- * @property {Session} [session]
- * @property {ProxyInfo} [proxyInfo]
- */
-
-/**
- * @callback PuppeteerHandlePage
- * @param {PuppeteerHandlePageInputs} inputs Arguments passed to this callback.
- * @return {Promise<void>}
- */
-
-/**
- * @typedef PuppeteerGotoInputs
- * @property {PuppeteerPage} page is an instance of the Puppeteer
- * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)
- * @property {Request} request
- * An instance of the {@link Request} object with details about the URL to open, HTTP method etc.
- * @property {AutoscaledPool} autoscaledPool An instance of the `AutoscaledPool`.
- * @property {PuppeteerPool} puppeteerPool An instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
- * @property {Session} [session] `Session` object for this request.
- * @property {ProxyInfo} [proxyInfo] Proxy info object
- */
-
-/**
- * @callback PuppeteerGoto
- * @param {PuppeteerGotoInputs} inputs Arguments passed to this callback.
- * @return {Promise<(PuppeteerResponse | null)>} An instance of the Puppeteer
- * [`Response`](https://pptr.dev/#?product=Puppeteer&show=api-class-response),
- * which is the main resource response as returned by `page.goto(request.url)`.
- */
-
-/**
- * @callback LaunchPuppeteer
- * @param {LaunchPuppeteerOptions} inputs Arguments passed to this callback.
- * @return {Promise<Browser>} Promise that resolves to Puppeteer's `Browser` instance.
- * This might be obtained by calling
- * [puppeteer.launch()](https://pptr.dev/#?product=Puppeteer&version=v2.0.0&show=api-puppeteerlaunchoptions)
- * directly, or by delegating to
- * {@link Apify#launchPuppeteer}.
- */
diff --git a/src/main.js b/src/main.js
index 2b522ab41..2446d49c4 100644
--- a/src/main.js
+++ b/src/main.js
@@ -8,14 +8,17 @@ import { pushData, openDataset } from './storages/dataset';
import events, { initializeEvents, stopEvents } from './events';
import { getValue, setValue, getInput, openKeyValueStore } from './storages/key_value_store';
import { launchPuppeteer } from './puppeteer';
+import { launchPlaywright } from './playwright';
+import BrowserCrawler from './crawlers/browser_crawler';
import PuppeteerCrawler from './crawlers/puppeteer_crawler';
-import PuppeteerPool from './puppeteer_pool';
+import PlaywrightCrawler from './crawlers/playwright_crawler';
import Request from './request';
import { RequestList, openRequestList } from './request_list';
import { createProxyConfiguration } from './proxy_configuration';
import { openRequestQueue } from './storages/request_queue';
import { newClient, getMemoryInfo, isAtHome, publicUtils } from './utils';
import { puppeteerUtils } from './puppeteer_utils';
+import { playwrightUtils } from './playwright_utils';
import { socialUtils } from './utils_social';
import { enqueueLinks } from './enqueue_links/enqueue_links';
import PseudoUrl from './pseudo_url';
@@ -29,6 +32,7 @@ EventEmitter.defaultMaxListeners = 50;
const exportedUtils = Object.assign(publicUtils, {
puppeteer: puppeteerUtils,
+ playwright: playwrightUtils,
social: socialUtils,
log,
enqueueLinks,
@@ -78,8 +82,10 @@ export {
openKeyValueStore,
launchPuppeteer,
- PuppeteerPool,
+ launchPlaywright,
+ BrowserCrawler,
PuppeteerCrawler,
+ PlaywrightCrawler,
PseudoUrl,
diff --git a/src/playwright.js b/src/playwright.js
new file mode 100644
index 000000000..1181f6344
--- /dev/null
+++ b/src/playwright.js
@@ -0,0 +1,163 @@
+import { URL } from 'url';
+import ow from 'ow';
+
+import { ENV_VARS } from 'apify-shared/consts';
+import { PlaywrightPlugin } from 'browser-pool';
+import { Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
+import { getTypicalChromeExecutablePath, isAtHome } from './utils';
+import applyStealthToBrowser, { StealthOptions } from './stealth/stealth'; // eslint-disable-line no-unused-vars,import/named
+
+/**
+ * Requires `playwright` browserType, uses a replacement or throws meaningful error if not installed.
+ *
+ * @param {(string|Object)} launcher
+ * @ignore
+ */
+export function getPlaywrightLauncherOrThrow(launcher = require('playwright').chromium) { // eslint-disable-line
+ if (typeof launcher === 'object') return launcher;
+ try {
+ // This is an optional dependency because it is quite large, only require it when used (ie. image with Chrome)
+ return require(launcher); // eslint-disable-line
+ } catch (err) {
+ if (err.code === 'MODULE_NOT_FOUND') {
+ const msg = `Cannot find module '${launcher}'. Did you you install the '${launcher}' package?`;
+ err.message = isAtHome()
+ ? `${msg} The 'playwright' package is automatically bundled when using apify/actor-node-chrome-* Base image.`
+ : msg;
+ }
+
+ throw err;
+ }
+}
+
+/**
+ *@ignore
+ */
+export function getDefaultHeadlessOption() {
+ return process.env[ENV_VARS.HEADLESS] === '1' && process.env[ENV_VARS.XVFB] !== '1';
+}
+
+/**
+ *@ignore
+ */
+export function getChromeExecutablePath() {
+ return process.env[ENV_VARS.CHROME_EXECUTABLE_PATH] || getTypicalChromeExecutablePath();
+}
+
+export function apifyOptionsToLaunchOptions(launchContext) {
+ const { launchOptions = {}, useChrome } = launchContext;
+
+ if (launchOptions.headless == null) {
+ launchOptions.headless = getDefaultHeadlessOption();
+ }
+
+ if (useChrome && !launchOptions.executablePath) {
+ launchOptions.executablePath = getChromeExecutablePath();
+ }
+
+ return launchOptions;
+}
+
+// TODO yin: `@property ...` didn't work. Extend Puppeteer's `LaunchOptions` didn't work. There is a GitHub issue for that:
+// https://github.com/Microsoft/TypeScript/issues/20077
+/**
+ * Apify extends the launch options of Puppeteer.
+ * You can use any of the Puppeteer compatible
+ * [`LaunchOptions`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions)
+ * options in the {@link Apify#launchPuppeteer}
+ * function and in addition, all the options available below.
+ *
+ * @typedef LaunchPlaywrightOptions
+ * @property {string} [proxyUrl]
+ * URL to a HTTP proxy server. It must define the port number,
+ * and it may also contain proxy username and password.
+ *
+ * Example: `http://bob:[email protected]:1234`.
+ * @property {boolean} [useChrome=false]
+ * If `true` and `executablePath` is not set,
+ * Puppeteer will launch full Google Chrome browser available on the machine
+ * rather than the bundled Chromium. The path to Chrome executable
+ * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
+ * or defaults to the typical Google Chrome executable location specific for the operating system.
+ * By default, this option is `false`.
+ * @property {(string|Object)} [launcher]
+ * By default this function uses require("playwright").chromium`.
+ * If you want to use a different browser you can pass it by this property as `require("playwright").firefox
+ *
+ * @property {object} [launchOptions]
+ * `browserType.launch` [options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions)
+ */
+
+/**
+ * Launches headless browsers using Playwright pre-configured to work within the Apify platform.
+ * The function has the same return value as `browserType.launch()`.
+ * See <a href="https://playwright.dev/docs/api/class-browsertype" target="_blank">
+ * Playwright documentation</a> for more details.
+ *
+ * The `launchPlaywright()` function alters the following Playwright options:
+ *
+ * - Passes the setting from the `APIFY_HEADLESS` environment variable to the `headless` option,
+ * unless it was already defined by the caller or `APIFY_XVFB` environment variable is set to `1`.
+ * Note that Apify Actor cloud platform automatically sets `APIFY_HEADLESS=1` to all running actors.
+ * - Takes the `proxyUrl` option, validates it and adds it to `args` as `--proxy-server=XXX`.
+ * The proxy URL must define a port number and have one of the following schemes: `http://`,
+ * `https://`, `socks4://` or `socks5://`.
+ * If the proxy is HTTP (i.e. has the `http://` scheme) and contains username or password,
+ * the `launchPlaywright` functions sets up an anonymous proxy HTTP
+ * to make the proxy work with headless Chrome. For more information, read the
+ * <a href="https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212"
+ * target="_blank">blog post about proxy-chain library</a>.
+ *
+ * To use this function, you need to have the [Playwright](https://www.npmjs.com/package/playwright)
+ * NPM package installed in your project.
+ * When running on the Apify cloud, you can achieve that simply
+ * by using the `apify/actor-node-chrome` base Docker image for your actor - see @TODO:
+ * [Apify Actor documentation](https://docs.apify.com/actor/build#base-images)
+ * for details.
+ *
+ *
+ * @param {LaunchPlaywrightOptions} [options]
+ * Optional settings passed to `browserType.launch()`. In addition to
+ * [Playwright's options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions)
+ * the object may contain our own {@link LaunchPlaywrightOptions} that enable additional features.
+ * @returns {Promise<Browser>}
+ * Promise that resolves to Playwright's `Browser` instance.
+ * @memberof module:Apify
+ * @name launchPlaywright
+ * @function
+ */
+export const launchPlaywright = async (launchContext = {}) => {
+ ow(launchContext, ow.object.partialShape({
+ proxyUrl: ow.optional.string.url,
+ launcher: ow.optional.any(ow.string, ow.object),
+ useChrome: ow.optional.boolean,
+
+ }));
+
+ const {
+ proxyUrl,
+ launcher,
+ } = launchContext;
+
+ if (proxyUrl) {
+ const parsedProxyUrl = new URL(proxyUrl);
+ if (!parsedProxyUrl.host || !parsedProxyUrl.port) {
+ throw new Error('Invalid "proxyUrl" option: both hostname and port must be provided.');
+ }
+ if (!/^(http|https|socks4|socks5)$/.test(parsedProxyUrl.protocol.replace(':', ''))) {
+ throw new Error(`Invalid "proxyUrl" option: Unsupported scheme (${parsedProxyUrl.protocol.replace(':', '')}).`);
+ }
+ }
+ const plugin = new PlaywrightPlugin(
+ getPlaywrightLauncherOrThrow(launcher),
+ {
+ proxyUrl,
+ launchOptions: apifyOptionsToLaunchOptions(launchContext),
+ },
+ );
+ const context = await plugin.createLaunchContext();
+
+ const browser = await plugin.launch(context);
+
+ return browser;
+};
diff --git a/src/playwright_utils.js b/src/playwright_utils.js
new file mode 100644
index 000000000..6f2083477
--- /dev/null
+++ b/src/playwright_utils.js
@@ -0,0 +1,64 @@
+import ow from 'ow';
+import _ from 'underscore';
+import { Page, Response, DirectNavigationOptions } from 'playwright'; // eslint-disable-line no-unused-vars
+import log from './utils_log';
+import { validators } from './validators';
+/**
+ * Extended version of Playwright's `page.goto()` allowing to perform requests with HTTP method other than GET,
+ * with custom headers and POST payload. URL, method, headers and payload are taken from
+ * request parameter that must be an instance of Apify.Request class.
+ *
+ * *NOTE:* In recent versions of Playwright using requests other than GET, overriding headers and adding payloads disables
+ * browser cache which degrades performance.
+ *
+ * @param {Page} page
+ * Puppeteer [`Page`](https://playwright.dev/docs/api/class-page) object.
+ * @param {Request} request
+ * @param {DirectNavigationOptions} [gotoOptions] Custom options for `page.goto()`.
+ * @return {Promise<(Response | null)>}
+ *
+ * @memberOf playwright
+ * @name gotoExtended
+ */
+export const gotoExtended = async (page, request, gotoOptions = {}) => {
+ ow(page, ow.object.validate(validators.browserPage));
+ ow(request, ow.object.partialShape({
+ url: ow.string.url,
+ method: ow.optional.string,
+ headers: ow.optional.object,
+ payload: ow.optional.any(ow.string, ow.buffer),
+ }));
+ ow(gotoOptions, ow.object);
+
+ const { url, method, headers, payload } = request;
+
+ if (method !== 'GET' || payload || !_.isEmpty(headers)) {
+ // This is not deprecated, we use it to log only once.
+ log.deprecated('Using other request methods than GET, rewriting headers and adding payloads has a high impact on performance '
+ + 'in recent versions of Playwright. Use only when necessary.');
+ let wasCalled = false;
+ const interceptRequestHandler = (route) => {
+ // We want to ensure that this won't get executed again in a case that there is a subsequent request
+ // for example for some asset file link from main HTML.
+ if (wasCalled) {
+ return route.continue();
+ }
+
+ wasCalled = true;
+ const overrides = {};
+
+ if (method !== 'GET') overrides.method = method;
+ if (payload) overrides.postData = payload;
+ if (!_.isEmpty(headers)) overrides.headers = headers;
+ route.continue(overrides);
+ };
+
+ await page.route('**/*', interceptRequestHandler);
+ }
+
+ return page.goto(url, gotoOptions);
+};
+
+export const playwrightUtils = {
+ gotoExtended,
+};
diff --git a/src/puppeteer.js b/src/puppeteer.js
index 444746301..88ebcb0e9 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -1,94 +1,31 @@
import ow from 'ow';
-import * as _ from 'underscore';
-import { anonymizeProxy, closeAnonymizedProxy, redactUrl, parseUrl } from 'proxy-chain';
+
import { ENV_VARS } from 'apify-shared/consts';
+import { PuppeteerPlugin } from 'browser-pool';
import { Browser } from 'puppeteer'; // eslint-disable-line no-unused-vars
import { DEFAULT_USER_AGENT } from './constants';
-import log from './utils_log';
import { getTypicalChromeExecutablePath, isAtHome } from './utils';
import applyStealthToBrowser, { StealthOptions } from './stealth/stealth'; // eslint-disable-line no-unused-vars,import/named
-const LAUNCH_PUPPETEER_LOG_OMIT_OPTS = [
- 'proxyUrl', 'userAgent', 'puppeteerModule', 'stealthOptions',
-];
-
const LAUNCH_PUPPETEER_DEFAULT_VIEWPORT = {
width: 1366,
height: 768,
};
-const LAUNCH_PUPPETEER_APIFY_OPTIONS = [
- ...LAUNCH_PUPPETEER_LOG_OMIT_OPTS,
- 'useChrome', 'stealth',
-];
-
-/**
- * Launches Puppeteer with proxy used via `proxy-chain` package.
- *
- * @ignore
- */
-const launchPuppeteerWithProxy = async (puppeteer, opts) => {
- // Parse and validate proxy URL
- const parsedProxyUrl = parseUrl(opts.proxyUrl);
- if (!parsedProxyUrl.host || !parsedProxyUrl.port) {
- throw new Error('Invalid "proxyUrl" option: both hostname and port must be provided.');
- }
- if (!/^(http|https|socks4|socks5)$/.test(parsedProxyUrl.scheme)) {
- throw new Error(`Invalid "proxyUrl" option: Unsupported scheme (${parsedProxyUrl.scheme}).`);
- }
-
- // Anonymize proxy URL if it has username or password
- let anonymizedProxyUrl = null;
- if (parsedProxyUrl.username || parsedProxyUrl.password) {
- if (parsedProxyUrl.scheme !== 'http') {
- throw new Error('Invalid "proxyUrl" option: authentication is only supported for HTTP proxy type.');
- }
- anonymizedProxyUrl = await anonymizeProxy(opts.proxyUrl);
- }
-
- opts.args.push(`--proxy-server=${anonymizedProxyUrl || opts.proxyUrl}`);
- const optsForLog = _.omit(opts, LAUNCH_PUPPETEER_LOG_OMIT_OPTS);
- optsForLog.proxyUrl = redactUrl(opts.proxyUrl);
- optsForLog.args = opts.args.slice(0, opts.args.length - 1);
-
- log.info('Launching Puppeteer', optsForLog);
- const onlyPuppeteerOptions = _.omit(opts, LAUNCH_PUPPETEER_APIFY_OPTIONS);
- const browser = await puppeteer.launch(onlyPuppeteerOptions);
-
- // Close anonymization proxy server when Puppeteer finishes
- if (anonymizedProxyUrl) {
- const cleanUp = () => {
- // Don't wait for finish, only log errors
- closeAnonymizedProxy(anonymizedProxyUrl, true)
- .catch((err) => log.exception(err, 'closeAnonymizedProxy() failed.'));
- };
-
- browser.on('disconnected', cleanUp);
-
- const prevClose = browser.close.bind(browser);
- browser.close = () => {
- cleanUp();
- return prevClose();
- };
- }
-
- return browser;
-};
-
/**
* Requires `puppeteer` package, uses a replacement or throws meaningful error if not installed.
*
- * @param {(string|Object)} puppeteerModule
+ * @param {(string|Object)} launcher
* @ignore
*/
-const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
- if (typeof puppeteerModule === 'object') return puppeteerModule;
+export function getPuppeteerOrThrow(launcher = 'puppeteer') {
+ if (typeof puppeteerModule === 'object') return launcher;
try {
// This is an optional dependency because it is quite large, only require it when used (ie. image with Chrome)
- return require(puppeteerModule); // eslint-disable-line
+ return require(launcher); // eslint-disable-line
} catch (err) {
if (err.code === 'MODULE_NOT_FOUND') {
- const msg = `Cannot find module '${puppeteerModule}'. Did you you install the '${puppeteerModule}' package?`;
+ const msg = `Cannot find module '${launcher}'. Did you you install the '${launcher}' package?`;
err.message = isAtHome()
? `${msg} The 'puppeteer' package is automatically bundled when using apify/actor-node-chrome-* Base image.`
: msg;
@@ -96,7 +33,57 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
throw err;
}
-};
+}
+
+/**
+ *@ignore
+ */
+export function getDefaultHeadlessOption() {
+ return process.env[ENV_VARS.HEADLESS] === '1' && process.env[ENV_VARS.XVFB] !== '1';
+}
+
+/**
+ *@ignore
+ */
+export function getChromeExecutablePath() {
+ return process.env[ENV_VARS.CHROME_EXECUTABLE_PATH] || getTypicalChromeExecutablePath();
+}
+
+export function apifyOptionsToLaunchOptions(launchContext) {
+ const { launchOptions = {}, useChrome } = launchContext;
+
+ launchOptions.args = launchOptions.args || [];
+ // Add --no-sandbox for Platform, because running Chrome in Docker
+ // is a very complex problem and most likely requires sys admin privileges,
+ // which is a larger security concern than --no-sandbox itself.
+ // TODO Find if the arg has any impact on browser detection.
+ if (isAtHome()) launchOptions.args.push('--no-sandbox');
+
+ if (launchOptions.headless == null) {
+ launchOptions.headless = getDefaultHeadlessOption();
+ }
+
+ if (useChrome && !launchOptions.executablePath) {
+ launchOptions.executablePath = getChromeExecutablePath();
+ }
+
+ if (launchOptions.defaultViewport === undefined) {
+ launchOptions.defaultViewport = LAUNCH_PUPPETEER_DEFAULT_VIEWPORT;
+ }
+
+ // When User-Agent is not set and we're using Chromium or headless mode,
+ // it is better to use DEFAULT_USER_AGENT to reduce chance of detection
+ let { userAgent } = launchContext;
+ if (!userAgent && (!launchOptions.executablePath || launchOptions.headless)) {
+ userAgent = DEFAULT_USER_AGENT;
+ }
+
+ if (userAgent) {
+ launchOptions.args.push(`--user-agent=${userAgent}`);
+ }
+
+ return launchOptions;
+}
// TODO yin: `@property ...` didn't work. Extend Puppeteer's `LaunchOptions` didn't work. There is a GitHub issue for that:
// https://github.com/Microsoft/TypeScript/issues/20077
@@ -124,7 +111,7 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
* is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,
* or defaults to the typical Google Chrome executable location specific for the operating system.
* By default, this option is `false`.
- * @property {(string|Object)} [puppeteerModule]
+ * @property {(string|Object)} [launcher]
* Either a require path (`string`) to a package to be used instead of default `puppeteer`,
* or an already required module (`Object`). This enables usage of various Puppeteer
* wrappers such as `puppeteer-extra`.
@@ -137,6 +124,8 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
* @property {StealthOptions} [stealthOptions]
* Using this configuration, you can disable some of the hiding tricks.
* For these settings to take effect `stealth` must be set to true
+ * @property {object} [launchOptions]
+ * `puppeteer.launch` [options](https://pptr.dev/#?product=Puppeteer&version=v5.5.0&show=api-puppeteerlaunchoptions)
*/
/**
@@ -188,60 +177,47 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
* @name launchPuppeteer
* @function
*/
-export const launchPuppeteer = async (options = {}) => {
- ow(options, ow.object.partialShape({
+export const launchPuppeteer = async (launchContext = {}) => {
+ ow(launchContext, ow.object.partialShape({
args: ow.optional.array.ofType(ow.string),
proxyUrl: ow.optional.string.url,
- puppeteerModule: ow.optional.any(ow.string, ow.object),
+ launcher: ow.optional.any(ow.string, ow.object),
stealth: ow.optional.boolean,
stealthOptions: ow.optional.object,
+ useChrome: ow.optional.boolean,
+ userAgent: ow.optional.string,
}));
- const puppeteer = getPuppeteerOrThrow(options.puppeteerModule);
-
- const optsCopy = { ...options };
-
- optsCopy.args = optsCopy.args || [];
- // Add --no-sandbox for Platform, because running Chrome in Docker
- // is a very complex problem and most likely requires sys admin privileges,
- // which is a larger security concern than --no-sandbox itself.
- // TODO Find if the arg has any impact on browser detection.
- if (isAtHome()) optsCopy.args.push('--no-sandbox');
-
- if (optsCopy.headless == null) {
- optsCopy.headless = process.env[ENV_VARS.HEADLESS] === '1' && process.env[ENV_VARS.XVFB] !== '1';
- }
- if (optsCopy.useChrome && (optsCopy.executablePath === undefined || optsCopy.executablePath === null)) {
- optsCopy.executablePath = process.env[ENV_VARS.CHROME_EXECUTABLE_PATH] || getTypicalChromeExecutablePath();
- }
-
- if (optsCopy.defaultViewport === undefined) {
- optsCopy.defaultViewport = LAUNCH_PUPPETEER_DEFAULT_VIEWPORT;
+ const {
+ stealth,
+ stealthOptions,
+ proxyUrl,
+ launcher,
+ } = launchContext;
+
+ if (proxyUrl) {
+ const parsedProxyUrl = new URL(proxyUrl);
+ if (!parsedProxyUrl.host || !parsedProxyUrl.port) {
+ throw new Error('Invalid "proxyUrl" option: both hostname and port must be provided.');
+ }
+ if (!/^(http|https|socks4|socks5)$/.test(parsedProxyUrl.protocol.replace(':', ''))) {
+ throw new Error(`Invalid "proxyUrl" option: Unsupported scheme (${parsedProxyUrl.protocol.replace(':', '')}).`);
+ }
}
- // When User-Agent is not set and we're using Chromium or headless mode,
- // it is better to use DEFAULT_USER_AGENT to reduce chance of detection
- let { userAgent } = optsCopy;
- if (!userAgent && (!optsCopy.executablePath || optsCopy.headless)) {
- userAgent = DEFAULT_USER_AGENT;
- }
- if (userAgent) {
- optsCopy.args.push(`--user-agent=${userAgent}`);
- }
+ const plugin = new PuppeteerPlugin(
+ getPuppeteerOrThrow(launcher),
+ {
+ proxyUrl,
+ launchOptions: apifyOptionsToLaunchOptions(launchContext),
+ },
+ );
+ const context = await plugin.createLaunchContext();
- let browser;
- if (optsCopy.proxyUrl) {
- // The log for launching with proxyUrl is inside launchPuppeteerWithProxy
- browser = await launchPuppeteerWithProxy(puppeteer, optsCopy);
- } else {
- log.info('Launching Puppeteer', _.omit(optsCopy, LAUNCH_PUPPETEER_LOG_OMIT_OPTS));
- const onlyPuppeteerOptions = _.omit(optsCopy, LAUNCH_PUPPETEER_APIFY_OPTIONS);
- browser = await puppeteer.launch(onlyPuppeteerOptions);
- }
+ const browser = await plugin.launch(context);
- // Add stealth
- if (optsCopy.stealth) {
- browser = applyStealthToBrowser(browser, optsCopy.stealthOptions);
+ if (stealth) {
+ applyStealthToBrowser(browser, stealthOptions);
}
return browser;
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
deleted file mode 100644
index 91a3f5e49..000000000
--- a/src/puppeteer_pool.js
+++ /dev/null
@@ -1,817 +0,0 @@
-/* eslint-disable max-classes-per-file */
-import * as _ from 'underscore';
-import * as fs from 'fs';
-import * as os from 'os';
-import ow from 'ow';
-import * as path from 'path';
-import * as util from 'util';
-import * as LinkedList from 'apify-shared/linked_list';
-import * as rimraf from 'rimraf';
-import defaultLog from './utils_log';
-import { addTimeoutToPromise } from './utils';
-import LiveViewServer from './live_view/live_view_server';
-import EVENTS from './session_pool/events';
-// TYPE IMPORTS
-/* eslint-disable no-unused-vars,import/named,import/order */
-import { Browser, Page } from 'puppeteer';
-import { launchPuppeteer, LaunchPuppeteerOptions } from './puppeteer';
-import { SessionPool } from './session_pool/session_pool';
-import { validators } from './validators';
-/* eslint-enable no-unused-vars */
-
-const PROCESS_KILL_TIMEOUT_MILLIS = 5000;
-const PAGE_CLOSE_KILL_TIMEOUT_MILLIS = 1000;
-
-const mkdtempAsync = util.promisify(fs.mkdtemp);
-const rimrafAsync = util.promisify(rimraf);
-const DISK_CACHE_DIR = path.join(os.tmpdir(), 'puppeteer_disk_cache-');
-
-/**
- * Internal representation of Puppeteer instance.
- *
- * @ignore
- */
-class PuppeteerInstance { // TODO: this is in progress and it will be refactored later
- constructor(options) {
- const {
- id,
- sessionPool,
- proxyConfiguration,
- launchPuppeteerFunction,
- } = options;
-
- this.id = id;
- this.proxyInfo = null;
- this.session = null;
- this.activePages = 0;
- this.totalPages = 0;
- this.browserPromise = null;
- this.lastPageOpenedAt = Date.now();
- this.killed = false;
- this.childProcess = null;
- this.recycleDiskCacheDir = null;
- this.sessionPool = sessionPool;
- this.proxyConfiguration = proxyConfiguration;
- this.launchPuppeteerFunction = launchPuppeteerFunction;
- }
-
- async launch() {
- this.browserPromise = new Promise(async (resolve, reject) => { // eslint-disable-line no-async-promise-executor
- try {
- if (this.sessionPool) {
- this.session = await this.sessionPool.getSession();
- }
-
- if (this.proxyConfiguration) {
- this.proxyInfo = this.proxyConfiguration.newProxyInfo(this.session ? this.session.id : undefined);
- }
- const proxyUrl = this.proxyInfo ? this.proxyInfo.url : null;
- resolve(this.launchPuppeteerFunction({ proxyUrl }));
- } catch (err) {
- reject(err);
- }
- });
- }
-}
-
-/**
- * @callback LaunchPuppeteerFunction
- * @param {LaunchPuppeteerOptions} options
- * @returns {Promise<Browser>}
- */
-
-/**
- * @typedef PuppeteerPoolOptions
- * @property {number} [maxOpenPagesPerInstance=50]
- * Maximum number of open pages (i.e. tabs) per browser. When this limit is reached, new pages are loaded in a new browser instance.
- * @property {number} [retireInstanceAfterRequestCount=100]
- * Maximum number of requests that can be processed by a single browser instance.
- * After the limit is reached, the browser is retired and new requests are
- * handled by a new browser instance.
- * @property {number} [puppeteerOperationTimeoutSecs=15]
- * All browser management operations such as launching a new browser, opening a new page
- * or closing a page will timeout after the set number of seconds and the connected
- * browser will be retired.
- * @property {number} [instanceKillerIntervalSecs=60]
- * Indicates how often are the open Puppeteer instances checked whether they can be closed.
- * @property {number} [killInstanceAfterSecs=300]
- * When Puppeteer instance reaches the `retireInstanceAfterRequestCount` limit then
- * it is considered retired and no more tabs will be opened. After the last tab is closed the
- * whole browser is closed too. This parameter defines a time limit between the last tab was opened and
- * before the browser is closed even if there are pending open tabs.
- * @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
- * Overrides the default function to launch a new Puppeteer instance. It must return a promise resolving to
- * [`Browser`](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#class-browser) instance.
- *
- * The function receives one parameter that includes launch options, such as `proxyUrl` and other options
- * generated by the `PuppeteerPool` instance. You can extend or update this options object, but you must
- * not ignore it.
- *
- * **Correct:**
- * ```javascript
- * async function launchPuppeteerFunction(options) {
- * const newOpts = {
- * ...options,
- * foo: 'bar',
- * }
- *
- * // do some other things
- *
- * return Apify.launchPuppeteer(newOpts);
- * }
- * ```
- * **Incorrect:**
- * ```javascript
- * async function launchPuppeteerFunction() {
- * const opts = {
- * foo: 'bar',
- * }
- * // Because we ignored the options, correct parameters
- * // will not make it to the browser. Eg. this prevents
- * // proxyConfiguration from working correctly.
- * return Apify.launchPuppeteer(opts);
- * }
- * ```
- *
- * @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]
- * Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.
- * @property {boolean} [recycleDiskCache=false]
- * Enables recycling of disk cache directories by Chrome instances.
- * When a browser instance is closed, its disk cache directory is not deleted but it's used by a newly opened browser instance.
- * This is useful to reduce amount of data that needs to be downloaded to speed up crawling and reduce proxy usage.
- * Note that the new browser starts with empty cookies, local storage etc. so this setting doesn't affect anonymity of your crawler.
- *
- * Beware that the disk cache directories can consume a lot of disk space.
- * To limit the space consumed, you can pass the `--disk-cache-size=X` argument to `launchPuppeteer` `args`,
- * where `X` is the approximate maximum number of bytes for disk cache.
- *
- * Do not use the `recycleDiskCache` setting together with `--disk-cache-dir`
- * argument in `launchPuppeteer` `args`, the behavior is undefined.
- * @property {boolean} [useIncognitoPages]
- * With this option selected, all pages will be opened in a new incognito browser context, which means
- * that they will not share cookies or cache and their resources will not be throttled by one another.
- * @property {SessionPool} [sessionPool]
- * A pool of Session instances.
- * @property {ProxyConfiguration} [proxyConfiguration]
- * If set, `PuppeteerPool` will be configured for all connections to use
- * [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided and rotated according to the configuration.
- * For more information, see the [documentation](https://docs.apify.com/proxy).
- */
-
-/**
- * Manages a pool of Chrome browser instances controlled using
- * [Puppeteer](https://github.com/puppeteer/puppeteer).
- *
- * `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.
- * This is useful in order to facilitate rotation of proxies, cookies
- * or other settings in order to prevent detection of your web scraping bot,
- * access web pages from various countries etc.
- *
- * Additionally, the reuse of browser instances instances speeds up crawling,
- * and the retirement of instances helps mitigate effects of memory leaks in Chrome.
- *
- * `PuppeteerPool` is internally used by the {@link PuppeteerCrawler} class.
- *
- * **Example usage:**
- *
- * ```javascript
- * const puppeteerPool = new PuppeteerPool({
- * launchPuppeteerOptions: {
- * // Use Chrome instead of Chromium
- * useChrome: true,
- * // Hide from bot detection
- * stealth: true,
- * },
- * });
- *
- * const page1 = await puppeteerPool.newPage();
- * const page2 = await puppeteerPool.newPage();
- *
- * // ... do something with the pages ...
- *
- * // Close all browsers.
- * await puppeteerPool.destroy();
- * ```
- */
-class PuppeteerPool {
- /**
- * @param {PuppeteerPoolOptions} [options]
- * All `PuppeteerPool` parameters are passed
- * via an options object.
- */
- constructor(options = {}) {
- ow(options, ow.object.exactShape({
- maxOpenPagesPerInstance: ow.optional.number,
- retireInstanceAfterRequestCount: ow.optional.number,
- launchPuppeteerFunction: ow.optional.function,
- puppeteerOperationTimeoutSecs: ow.optional.number,
- instanceKillerIntervalSecs: ow.optional.number,
- killInstanceAfterSecs: ow.optional.number,
- launchPuppeteerOptions: ow.optional.object,
- recycleDiskCache: ow.optional.boolean,
- useIncognitoPages: ow.optional.boolean,
- proxyConfiguration: ow.optional.object.validate(validators.proxyConfiguration),
- useLiveView: ow.optional.boolean,
- sessionPool: ow.optional.object,
- log: ow.optional.object,
- }));
-
- const {
- // Don't make these too large, otherwise Puppeteer might start crashing weirdly,
- // and the default settings should just work
- maxOpenPagesPerInstance = 50,
- retireInstanceAfterRequestCount = 100,
- launchPuppeteerFunction = (launchPuppeteerOptions) => launchPuppeteer(launchPuppeteerOptions),
- puppeteerOperationTimeoutSecs = 15,
- instanceKillerIntervalSecs = 60,
- killInstanceAfterSecs = 300,
- launchPuppeteerOptions,
- recycleDiskCache = false,
- useIncognitoPages = false,
- proxyConfiguration,
- useLiveView = false,
- sessionPool,
- log = defaultLog,
- } = options;
-
- // Disabling due to memory leak.
- const reusePages = false;
-
- if (proxyConfiguration && (launchPuppeteerOptions && launchPuppeteerOptions.proxyUrl)) {
- throw new Error('It is not possible to combine "options.proxyConfiguration" together with '
- + 'custom "proxyUrl" option from "options.launchPuppeteerOptions".');
- }
- this.log = log.child({ prefix: 'PuppeteerPool' });
-
- if (options.useLiveView) {
- this.log.deprecated('options.useLiveView is deprecated without replacement. Use devtools-server NPM package for similar functionality.');
- }
-
- // Config.
- this.sessionPool = sessionPool;
- this.reusePages = reusePages;
- this.maxOpenPagesPerInstance = maxOpenPagesPerInstance;
- this.retireInstanceAfterRequestCount = retireInstanceAfterRequestCount;
- this.puppeteerOperationTimeoutMillis = puppeteerOperationTimeoutSecs * 1000;
- this.killInstanceAfterMillis = killInstanceAfterSecs * 1000;
- // this is needed to avoid TS typings trying to link to the .d.ts
- /**
- * @type {*}
- * @ignore
- */
- this.recycledDiskCacheDirs = recycleDiskCache ? new LinkedList() : null;
- this.useIncognitoPages = useIncognitoPages;
- this.proxyConfiguration = proxyConfiguration;
- this.liveViewServer = useLiveView ? new LiveViewServer() : null;
- this.launchPuppeteerOptions = launchPuppeteerOptions;
- this.launchPuppeteerFunction = async ({ proxyUrl }) => {
- // Do not modify passed launchPuppeteerOptions!
- const opts = _.clone(launchPuppeteerOptions) || {};
- opts.args = _.clone(opts.args || []);
-
- // If requested, use recycled disk cache directory
- let diskCacheDir = null;
- if (recycleDiskCache) {
- diskCacheDir = this.recycledDiskCacheDirs.length > 0
- ? this.recycledDiskCacheDirs.removeFirst()
- : await mkdtempAsync(DISK_CACHE_DIR);
- opts.args.push(`--disk-cache-dir=${diskCacheDir}`);
- }
-
- // Set timeout for browser launch.
- if (opts.timeout == null) opts.timeout = this.puppeteerOperationTimeoutMillis;
-
- if (proxyConfiguration) {
- opts.proxyUrl = proxyUrl;
- }
-
- const browser = await launchPuppeteerFunction(opts);
- if (!browser || typeof browser.newPage !== 'function') {
- // eslint-disable-next-line max-len
- throw new Error("The custom 'launchPuppeteerFunction' passed to PuppeteerPool must return a promise resolving to Puppeteer's Browser instance.");
- }
- browser.recycleDiskCacheDir = diskCacheDir;
-
- return browser;
- };
-
- // State.
- this.browserCounter = 0;
- this.activeInstances = {};
- this.retiredInstances = {};
- this.lastUsedProxyUrlIndex = 0;
- this.instanceKillerInterval = setInterval(() => this._killRetiredInstances(), instanceKillerIntervalSecs * 1000);
- this.idlePages = [];
- // WeakSet/Map items do not prevent garbage collection,
- // and thus no management of the collections is needed.
- // They will automatically empty themselves once there
- // are no references to the stored pages.
- this.closedPages = new WeakSet();
- this.pagesToInstancesMap = new WeakMap();
- this.browsersToInstancesMap = new WeakMap();
-
- this.liveViewSnapshotsInProgress = new WeakMap();
-
- // ensure termination on SIGINT
- this.sigintListener = () => this._killAllInstances();
- process.on('SIGINT', this.sigintListener);
-
- if (sessionPool) {
- this._retireBrowserWithSession = this._retireBrowserWithSession.bind(this);
- sessionPool.on(EVENTS.SESSION_RETIRED, this._retireBrowserWithSession);
- }
- }
-
- /**
- * Launches new browser instance.
- *
- * @ignore
- */
- _launchInstance() {
- const id = this.browserCounter++;
- this.log.debug('Launching new browser', { id });
-
- const instance = new PuppeteerInstance({
- id,
- sessionPool: this.sessionPool,
- proxyConfiguration: this.proxyConfiguration,
- launchPuppeteerOptions: this.launchPuppeteerOptions,
- launchPuppeteerFunction: this.launchPuppeteerFunction,
- });
- this.activeInstances[id] = instance;
-
- // Handle the async stuff elsewhere.
- this._initBrowser(instance);
-
- return instance;
- }
-
- /**
- * Takes care of async processes in PuppeteerInstance construction with a Browser.
- * @param {PuppeteerInstance} instance
- * @returns {Promise<void>}
- * @ignore
- */
- async _initBrowser(instance) {
- const { id } = instance;
- let browser;
- try {
- await instance.launch();
- browser = await instance.browserPromise;
- this.browsersToInstancesMap.set(browser, instance);
- } catch (err) {
- this.log.exception(err, 'Browser launch failed', { id });
- delete this.activeInstances[id];
- return;
- }
-
- instance.childProcess = browser.process();
- instance.recycleDiskCacheDir = browser.recycleDiskCacheDir;
-
- browser.on('disconnected', () => {
- // If instance.killed === true then we killed the instance so don't log it.
- if (!instance.killed) this.log.error('Puppeteer sent "disconnect" event. Maybe it crashed???', { id });
- this._retireInstance(instance);
- });
- }
-
- /**
- * Retires some of the instances for example due to many uses.
- *
- * @ignore
- */
- _retireInstance(instance) {
- const { id } = instance;
-
- if (!this.activeInstances[id]) return this.log.debug('browser is retired already', { id });
-
- this.log.debug('Retiring browser', { id });
-
- this.retiredInstances[id] = instance;
- delete this.activeInstances[id];
- }
-
- /**
- * Kills all the retired instances that:
- * - have all tabs closed
- * - or are inactive for more then killInstanceAfterMillis.
- *
- * @ignore
- */
- _killRetiredInstances() {
- this.log.debug('Retired browsers count', { count: _.values(this.retiredInstances).length });
-
- _.mapObject(this.retiredInstances, async (instance) => {
- // Kill instances that are more than this.killInstanceAfterMillis from last opened page
- if (Date.now() - instance.lastPageOpenedAt > this.killInstanceAfterMillis) {
- this.log.debug('killing retired browser after period of inactivity', { id: instance.id, killInstanceAfterMillis: this.killInstanceAfterMillis }); // eslint-disable-line max-len
- this._killInstance(instance);
- return;
- }
-
- try {
- const browser = await instance.browserPromise;
- const pages = await browser.pages();
- // NOTE: we are killing instance when the number of pages is less or equal to 1 because there is always about:blank page.
- if (pages.length <= 1) {
- this.log.debug('Killing retired browser because it has no open tabs', { id: instance.id });
- this._killInstance(instance);
- }
- } catch (err) {
- this.log.exception(err, 'Browser.pages() failed', { id: instance.id });
- this._killInstance(instance);
- }
- });
- }
-
- /**
- * Kills given browser instance.
- *
- * @ignore
- */
- async _killInstance(instance) {
- const { id, childProcess, killed, browserPromise } = instance;
- if (killed) return;
-
- this.log.debug('Killing browser', { id });
-
- delete this.retiredInstances[id];
-
- const recycleDiskCache = () => {
- if (!instance.recycleDiskCacheDir) return;
- this.log.debug('Recycling disk cache dir', { id, diskCacheDir: instance.recycleDiskCacheDir });
- this.recycledDiskCacheDirs.add(instance.recycleDiskCacheDir);
- instance.recycleDiskCacheDir = null;
- };
-
- // Ensure that Chrome process will be really killed.
- setTimeout(() => {
- // This is here because users reported that it happened
- // that error `TypeError: Cannot read property 'kill' of null` was thrown.
- // Likely Chrome process wasn't started due to some error ...
- if (childProcess) childProcess.kill('SIGKILL');
-
- recycleDiskCache();
- }, PROCESS_KILL_TIMEOUT_MILLIS);
-
- try {
- const browser = await browserPromise;
- instance.killed = true;
- await browser.close();
- recycleDiskCache();
- } catch (err) {
- this.log.exception(err, 'Cannot close the browser instance, it will be killed forcibly.', { id });
- }
- }
-
- /**
- * Kills all running PuppeteerInstances.
- * @ignore
- */
- _killAllInstances() {
- // TODO: delete all dirs
- const allInstances = Object.values(this.activeInstances).concat(Object.values(this.retiredInstances));
- allInstances.forEach((instance) => {
- try {
- instance.childProcess.kill('SIGKILL');
- } catch (e) {
- // do nothing, it's dead
- }
- });
- }
-
- /**
- * Updates the instance metadata when a new page is opened.
- *
- * @param {PuppeteerInstance} instance
- * @ignore
- */
- _incrementPageCount(instance) {
- instance.lastPageOpenedAt = Date.now();
- instance.totalPages++;
- if (instance.totalPages >= this.retireInstanceAfterRequestCount) this._retireInstance(instance);
- }
-
- /**
- * Produces a new page instance either by reusing an idle page that currently isn't processing
- * any request or by spawning a new page (new browser tab) in one of the available
- * browsers when no idle pages are available.
- *
- * To spawn a new browser tab for each page, set the `reusePages` constructor option to false.
- *
- * @return {Promise<Page>}
- */
- async newPage() {
- if (this.liveViewServer && !this.liveViewServer.isRunning()) await this.liveViewServer.start();
- let idlePage;
- // We don't need to check whether options.reusePages is true,
- // because if it's false, the array will be empty and the loop will never start.
- while (idlePage = this.idlePages.shift()) { // eslint-disable-line no-cond-assign
- // Since pages can close for various reasons that we have no control over,
- // we need to make sure that we're only using live pages, so we go
- // through the queue until we get a page that's live, which means
- // that it's not closed and its browser is not retired.
- const pageIsNotClosed = !this.closedPages.has(idlePage);
- const instance = this.pagesToInstancesMap.get(idlePage);
- const instanceIsActive = !!this.activeInstances[instance.id];
- if (pageIsNotClosed && instanceIsActive) {
- this._incrementPageCount(instance);
- return idlePage;
- }
- // Close pages of retired instances so they don't keep hanging there forever.
- if (pageIsNotClosed && !instanceIsActive) {
- await idlePage.close();
- }
- }
- // If there are no live pages to be reused, we spawn a new tab.
- return this._openNewTab();
- }
-
- /**
- * Opens new tab in one of the browsers in the pool and returns a `Promise`
- * that resolves to an instance of a Puppeteer
- * [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page).
- *
- * @return {Promise<Page>}
- * @ignore
- */
- async _openNewTab() {
- let instance = Object
- .values(this.activeInstances)
- .find((inst) => inst.activePages < this.maxOpenPagesPerInstance);
-
- if (!instance) instance = this._launchInstance();
- this._incrementPageCount(instance);
- instance.activePages++;
-
- try {
- const browser = await instance.browserPromise;
- const context = this.useIncognitoPages
- ? await browser.createIncognitoBrowserContext()
- : browser.defaultBrowserContext();
- const page = await addTimeoutToPromise(
- context.newPage(),
- this.puppeteerOperationTimeoutMillis,
- 'browser.newPage() timed out.',
- );
- await this._focusOldestTab(browser).catch(() => this.log.debug('Could not focus oldest tab.'));
- this.pagesToInstancesMap.set(page, instance);
- return this._decoratePage(page);
- } catch (err) {
- this._retireInstance(instance);
- const betterError = new Error(`browser.newPage() failed: ${instance.id}.`);
- betterError.stack = err.stack;
- throw betterError;
- }
- }
-
- /**
- * Adds the necessary boilerplate to allow page reuse and also
- * captures page.close() errors to prevent meaningless log clutter.
- * @param {Page} page
- * @ignore
- */
- _decoratePage(page) {
- const instance = this.pagesToInstancesMap.get(page);
-
- const originalPageClose = page.close;
- page.close = async (...args) => {
- this.closedPages.add(page);
- await originalPageClose.apply(page, args)
- .catch((err) => {
- this.log.debug('Page.close() failed', { errorMessage: err.message, id: instance.id });
- });
- const context = page.browserContext();
- if (context.isIncognito()) {
- await context.close().catch((err) => {
- this.log.debug('Context.close() failed', { errorMessage: err.message, id: instance.id });
- });
- }
- };
-
- page.once('error', (error) => {
- this.log.exception(error, 'Page crashed.');
- page.close();
- });
-
- page.once('close', () => {
- instance.activePages--;
- this._killInstanceWithNoPages(instance);
- });
-
- return page;
- }
-
- /**
- * Tells Chromium to focus oldest tab. This is to work around Chromium
- * throttling CPU and network in inactive tabs.
- *
- * @param {Browser} browser
- * @ignore
- */
- async _focusOldestTab(browser) { // eslint-disable-line class-methods-use-this
- const pages = await browser.pages();
- if (pages.length > 1) return pages[1].bringToFront();
- }
-
- /**
- * Deletes Chrome's user data directory
- * @param {string} diskCacheDir
- * @private
- * @ignore
- */
- async _deleteDiskCacheDir(diskCacheDir) {
- this.log.debug('Deleting disk cache directory', { diskCacheDir });
- return rimrafAsync(diskCacheDir)
- .catch((err) => {
- this.log.warning('Cannot delete Chrome disk cache directory', { diskCacheDir, errorMessage: err.message });
- });
- }
-
- /**
- * Closes all open browsers.
- * @return {Promise<void>}
- */
- async destroy() {
- clearInterval(this.instanceKillerInterval);
- process.removeListener('SIGINT', this.sigintListener);
-
- // TODO: delete of dir doesn't seem to work!
-
- const browserPromises = _
- .values(this.activeInstances)
- .concat(_.values(this.retiredInstances))
- .map((instance) => {
- // This is needed so that "Puppeteer disconnected" errors are not printed.
- instance.killed = true;
-
- return instance.browserPromise;
- });
-
- const closePromises = browserPromises.map(async (browserPromise) => {
- const browser = await browserPromise;
- await browser.close();
- if (browser.recycleDiskCacheDir) await this._deleteDiskCacheDir(browser.recycleDiskCacheDir);
- });
-
- try {
- await Promise.all(closePromises);
- // Delete all cache directories
- const dirDeletionPromises = [];
- while (this.recycledDiskCacheDirs && this.recycledDiskCacheDirs.length > 0) {
- dirDeletionPromises.push(this._deleteDiskCacheDir(this.recycledDiskCacheDirs.removeFirst()));
- }
- await Promise.all(dirDeletionPromises);
- } catch (err) {
- this.log.exception(err, 'Cannot close the browsers.');
- }
- if (this.liveViewServer) {
- await this.liveViewServer.stop().catch((err) => this.log.exception(err, 'Cannot close LiveViewServer.'));
- }
- }
-
- /**
- * Finds a PuppeteerInstance given a Puppeteer Browser running in the instance.
- * @param {Browser} browser
- * @return {Promise<*>}
- * @ignore
- */
- async _findInstanceByBrowser(browser) {
- const instances = Object.values(this.activeInstances);
- const resultPromises = instances.map(async (instance) => {
- const savedBrowser = await instance.browserPromise;
- return browser === savedBrowser ? instance : null;
- });
- const results = (await Promise.all(resultPromises)).filter((i) => i);
- switch (results.length) {
- case 0:
- return null;
- case 1:
- return results[0];
- default:
- throw new Error('Multiple instances of PuppeteerPool found using a single browser instance.');
- }
- }
-
- /**
- * Manually retires a Puppeteer
- * [`Browser`](https://pptr.dev/#?product=Puppeteer&show=api-class-browser)
- * instance from the pool. The browser will continue to process open pages so that they may gracefully finish.
- * This is unlike `browser.close()` which will forcibly terminate the browser and all open pages will be closed.
- * @param {Browser} browser
- * @return {Promise<void>}
- */
- async retire(browser) {
- const instance = await this._findInstanceByBrowser(browser);
- if (instance) return this._retireInstance(instance);
- this.log.debug('Browser is retired already');
- }
-
- /**
- * Closes the page, unless the `reuseTabs` option is set to true.
- * Then it would only flag the page for a future reuse, without actually closing it.
- *
- * NOTE: LiveView snapshotting is tied to this function. When `useLiveView` option
- * is set to true, a snapshot of the page will be taken just before closing the page
- * or flagging it for reuse.
- *
- * @param {Page} page
- * @return {Promise<void>}
- */
- async recyclePage(page) {
- const snapshotPromise = this.liveViewSnapshotsInProgress.get(page);
- if (snapshotPromise) await snapshotPromise;
-
- if (this.reusePages) {
- page.removeAllListeners();
- this.idlePages.push(page);
- } else {
- try {
- await addTimeoutToPromise(
- page.close(),
- this.puppeteerOperationTimeoutMillis,
- 'page.close() timed out.',
- );
- } catch (err) {
- this.log.debug('page.close() failed.', { reason: err && err.message });
- }
- }
- }
-
- /**
- * Tells the connected LiveViewServer to serve a snapshot when available.
- *
- * @param {Page} page
- * @return {Promise<void>}
- */
- async serveLiveViewSnapshot(page) {
- const isLiveViewConnected = this.liveViewServer && this.liveViewServer.hasClients();
- if (!isLiveViewConnected) return;
-
- const browser = page.browser();
- const pages = await browser.pages();
-
- // We only serve the second page of the browser because it's in focus,
- // which is necessary for screenshots. First page is about:blank.
- if (pages[1] !== page) return;
-
- const instance = await this._findInstanceByBrowser(browser);
- // Sometimes the browser gets killed and there's no instance.
- if (!instance) return;
-
- // Only take snapshots in the most recently opened browser.
- if (instance.id !== this.browserCounter - 1) return;
-
- const snapshotPromise = this.liveViewServer.serve(page)
- .catch((err) => this.log.debug('Live View failed to be served.', { message: err.message }));
- this.liveViewSnapshotsInProgress.set(page, snapshotPromise);
- }
-
- _findInstancesBySession(session) {
- const instances = Object.values(this.activeInstances);
- return instances.filter((instance) => instance.session.id === session.id);
- }
-
- async _retireBrowserWithSession(session) {
- const instances = this._findInstancesBySession(session);
-
- const isInstanceRunning = instances.length >= 1;
-
- if (isInstanceRunning) {
- const retireInstances = instances.map((instance) => this._retireInstance(instance));
- try {
- await Promise.all(retireInstances);
- } catch (e) {
- // ignore the error, since the instance is either retired already, being retired or cannot be retired
- // ( we cant do nothing about this at this point)
- this.log.debug('Could not retire instances ', e);
- }
- }
- }
-
- _killInstanceWithNoPages(instance) {
- const { id } = instance;
- if (instance.activePages === 0 && this.retiredInstances[id]) {
- // Run this with a delay, otherwise page.close()
- // might fail with "Protocol error (Target.closeTarget): Target closed."
- setTimeout(() => {
- this.log.debug('Killing retired browser because it has no active pages', { id });
- this._killInstance(instance);
- }, PAGE_CLOSE_KILL_TIMEOUT_MILLIS);
- }
- }
-
- /**
- * Gets Puppeteer Instance by the page.
- *
- * @param {Page} page
- * @return {PuppeteerInstance}
- * @ignore
- */
- _getBrowserInstance(page) {
- const browser = page.browser();
- return this.browsersToInstancesMap.get(browser);
- }
-}
-
-export default PuppeteerPool;
diff --git a/src/utils.js b/src/utils.js
index 85451f424..a22a8841b 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -17,16 +17,17 @@ import * as semver from 'semver';
import * as _ from 'underscore';
import { URL } from 'url';
import * as util from 'util';
-import * as requestUtils from './utils_request';
-import log from './utils_log';
-import { version as apifyVersion } from '../package.json';
// TYPE IMPORTS
/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */
import { IncomingMessage } from 'http';
import { Response as PuppeteerResponse } from 'puppeteer';
+import { version as apifyVersion } from '../package.json';
+import log from './utils_log';
+import * as requestUtils from './utils_request';
import Request, { RequestOptions } from './request';
import { ActorRun } from './typedefs';
+
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
/**
@@ -361,7 +362,7 @@ export const getTypicalChromeExecutablePath = () => {
switch (os.platform()) {
case 'darwin': return '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome';
case 'win32': return 'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe';
- default: return 'google-chrome';
+ default: return '/usr/bin/google-chrome';
}
};
diff --git a/test/crawlers/browser_crawler.test.js b/test/crawlers/browser_crawler.test.js
new file mode 100644
index 000000000..94c7c8a45
--- /dev/null
+++ b/test/crawlers/browser_crawler.test.js
@@ -0,0 +1,705 @@
+import { URL } from 'url';
+import { ENV_VARS } from 'apify-shared/consts';
+import sinon from 'sinon';
+import { BrowserPool, PuppeteerPlugin } from 'browser-pool';
+import puppeteer from 'puppeteer';
+import log from '../../build/utils_log';
+import * as Apify from '../../build';
+import { STATUS_CODES_BLOCKED } from '../../build/constants';
+import LocalStorageDirEmulator from '../local_storage_dir_emulator';
+import * as utilsRequest from '../../build/utils_request';
+import * as utils from '../../build/utils';
+import Request from '../../build/request';
+import AutoscaledPool from '../../build/autoscaling/autoscaled_pool';
+import { Session } from '../../build/session_pool/session';
+import EVENTS from '../../build/session_pool/events';
+
+describe('BrowserCrawler', () => {
+ let prevEnvHeadless;
+ let logLevel;
+ let localStorageEmulator;
+ let puppeteerPlugin;
+
+ beforeAll(async () => {
+ prevEnvHeadless = process.env[ENV_VARS.HEADLESS];
+ process.env[ENV_VARS.HEADLESS] = '1';
+ logLevel = log.getLevel();
+ log.setLevel(log.LEVELS.ERROR);
+ localStorageEmulator = new LocalStorageDirEmulator();
+ });
+ beforeEach(async () => {
+ const storageDir = await localStorageEmulator.init();
+ utils.apifyStorageLocal = utils.newStorageLocal({ storageDir });
+ puppeteerPlugin = new PuppeteerPlugin(puppeteer);
+ });
+ afterEach(() => {
+ puppeteerPlugin = null;
+ });
+ afterAll(async () => {
+ log.setLevel(logLevel);
+ process.env[ENV_VARS.HEADLESS] = prevEnvHeadless;
+ await localStorageEmulator.destroy();
+ });
+
+ test('should work', async () => {
+ const sources = [
+ { url: 'http://example.com/?q=1' },
+ { url: 'http://example.com/?q=2' },
+ { url: 'http://example.com/?q=3' },
+ { url: 'http://example.com/?q=4' },
+ { url: 'http://example.com/?q=5' },
+ { url: 'http://example.com/?q=6' },
+ ];
+ const sourcesCopy = JSON.parse(JSON.stringify(sources));
+ const processed = [];
+ const failed = [];
+ const requestList = new Apify.RequestList({ sources });
+ const handlePageFunction = async ({ page, request, response }) => {
+ await page.waitForSelector('title');
+
+ expect(await response.status()).toBe(200);
+ request.userData.title = await page.title();
+ processed.push(request);
+ };
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ gotoFunction: ({ page, request }) => page.goto(request.url),
+ minConcurrency: 1,
+ maxConcurrency: 1,
+ handlePageFunction,
+ handleFailedRequestFunction: ({ request }) => failed.push(request),
+ });
+
+ await requestList.initialize();
+ await browserCrawler.run();
+
+ expect(browserCrawler.autoscaledPool.minConcurrency).toBe(1);
+ expect(processed).toHaveLength(6);
+ expect(failed).toHaveLength(0);
+
+ processed.forEach((request, id) => {
+ expect(request.url).toEqual(sourcesCopy[id].url);
+ expect(request.userData.title).toBe('Example Domain');
+ });
+ });
+ test('should teardown browser pool', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async () => {
+ return Promise.resolve();
+ },
+ maxRequestRetries: 1,
+ gotoFunction: async () => {
+ },
+ });
+ jest.spyOn(browserCrawler.browserPool, 'destroy');
+
+ await requestList.initialize();
+ await browserCrawler.run();
+ expect(browserCrawler.browserPool.destroy).toBeCalled();
+ });
+
+ test('should evaluate preNavigationHooks', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ let isEvaluated = false;
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async () => {
+ return Promise.resolve();
+ },
+ maxRequestRetries: 0,
+ gotoFunction: async ({ hookFinished }) => {
+ isEvaluated = hookFinished;
+ },
+ preNavigationHooks: [
+ async (crawlingContext) => {
+ await Apify.utils.sleep(10);
+ crawlingContext.hookFinished = true;
+ },
+ ],
+ });
+
+ await requestList.initialize();
+ await browserCrawler.run();
+
+ expect(isEvaluated).toBeTruthy();
+ });
+
+ test('should evaluate postNavigationHooks', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ let isEvaluated = false;
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async ({ hookFinished }) => {
+ isEvaluated = hookFinished;
+ },
+ maxRequestRetries: 0,
+ gotoFunction: ({ page, request }) => page.goto(request.url),
+ postNavigationHooks: [
+ async (crawlingContext) => {
+ await Apify.utils.sleep(10);
+ crawlingContext.hookFinished = true;
+ },
+ ],
+ });
+
+ await requestList.initialize();
+ await browserCrawler.run();
+
+ expect(isEvaluated).toBeTruthy();
+ });
+
+ test('should allow modifying gotoOptions by pre navigation hooks', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ let optionsGoto;
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async () => {},
+ maxRequestRetries: 0,
+ gotoFunction: ({ page, request }, gotoOptions) => {
+ optionsGoto = gotoOptions;
+ return page.goto(request.url, gotoOptions);
+ },
+ preNavigationHooks: [
+ async (crawlingContext, gotoOptions) => {
+ gotoOptions.timeout = 60000;
+ },
+ ],
+ });
+
+ await requestList.initialize();
+ await browserCrawler.run();
+
+ expect(optionsGoto.timeout).toEqual(60000);
+ });
+
+ test('should ignore errors in Page.close()', async () => {
+ for (let i = 0; i < 2; i++) {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ let failedCalled = false;
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ gotoFunction: ({ page, request }) => page.goto(request.url),
+ handlePageFunction: ({ page }) => {
+ page.close = async () => {
+ if (i === 0) {
+ throw new Error();
+ } else {
+ return Promise.reject(new Error());
+ }
+ };
+ return Promise.resolve();
+ },
+ handleFailedRequestFunction: async () => {
+ failedCalled = true;
+ },
+ });
+ await requestList.initialize();
+ await browserCrawler.run();
+ expect(failedCalled).toBe(false);
+ }
+ });
+
+ test('should use SessionPool', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ const handlePageSessions = [];
+ const goToPageSessions = [];
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async ({ session }) => {
+ handlePageSessions.push(session);
+ return Promise.resolve();
+ },
+
+ });
+
+ browserCrawler.gotoFunction = async ({ session, page, request }) => {
+ goToPageSessions.push(session);
+ return page.goto(request.url);
+ };
+
+ await requestList.initialize();
+ await browserCrawler.run();
+
+ expect(browserCrawler.sessionPool.constructor.name).toEqual('SessionPool');
+ expect(handlePageSessions).toHaveLength(1);
+ expect(goToPageSessions).toHaveLength(1);
+ handlePageSessions.forEach((session) => expect(session.constructor.name).toEqual('Session'));
+ goToPageSessions.forEach((session) => expect(session.constructor.name).toEqual('Session'));
+ });
+
+ test('should persist cookies per session', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ { url: 'http://example.com/?q=2' },
+ { url: 'http://example.com/?q=3' },
+ { url: 'http://example.com/?q=4' },
+ ],
+ });
+ const goToPageSessions = [];
+ const loadedCookies = [];
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ persistCookiesPerSession: true,
+ handlePageFunction: async ({ session, request }) => {
+ loadedCookies.push(session.getCookieString(request.url));
+ return Promise.resolve();
+ },
+ });
+
+ browserCrawler.gotoFunction = async ({ session, page, request }) => {
+ await page.setCookie({ name: 'TEST', value: '12321312312', domain: 'example.com', expires: Date.now() + 100000 });
+ goToPageSessions.push(session);
+ return page.goto(request.url);
+ };
+
+ await requestList.initialize();
+ await browserCrawler.run();
+ expect(loadedCookies).toHaveLength(4);
+ loadedCookies.forEach((cookie) => expect(cookie).toEqual('TEST=12321312312'));
+ });
+
+ test('should throw on "blocked" status codes', async () => {
+ const baseUrl = 'https://example.com/';
+ const sources = STATUS_CODES_BLOCKED.map((statusCode) => {
+ return {
+ url: baseUrl + statusCode,
+ userData: { statusCode },
+ };
+ });
+ const requestList = await Apify.openRequestList(null, sources);
+
+ let called = false;
+ const failedRequests = [];
+ const crawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ persistCookiesPerSession: false,
+ maxRequestRetries: 0,
+ handlePageFunction: async () => { // eslint-disable-line no-loop-func
+ called = true;
+ },
+ handleFailedRequestFunction: async ({ request }) => {
+ failedRequests.push(request);
+ },
+ });
+
+ crawler.gotoFunction = async ({ request }) => {
+ return { status: () => request.userData.statusCode };
+ };
+
+ await crawler.run();
+
+ expect(failedRequests.length).toBe(STATUS_CODES_BLOCKED.length);
+ failedRequests.forEach((fr) => {
+ const [msg] = fr.errorMessages;
+ expect(msg).toContain(`Request blocked - received ${fr.userData.statusCode} status code.`);
+ });
+ expect(called).toBe(false);
+ });
+
+ test('should throw on "blocked" status codes (retire session)', async () => {
+ const baseUrl = 'https://example.com/';
+ const sources = STATUS_CODES_BLOCKED.map((statusCode) => {
+ return {
+ url: baseUrl + statusCode,
+ userData: { statusCode },
+ };
+ });
+ const requestList = await Apify.openRequestList(null, sources);
+
+ let called = false;
+ const failedRequests = [];
+ const crawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ persistCookiesPerSession: false,
+ maxRequestRetries: 0,
+ handlePageFunction: async () => { // eslint-disable-line no-loop-func
+ called = true;
+ },
+ gotoFunction: ({ page, request }) => page.goto(request.url),
+ handleFailedRequestFunction: async ({ request }) => {
+ failedRequests.push(request);
+ },
+ });
+
+ crawler.gotoFunction = async ({ request }) => {
+ return { status: () => request.userData.statusCode };
+ };
+
+ await crawler.run();
+
+ expect(failedRequests.length).toBe(STATUS_CODES_BLOCKED.length);
+ failedRequests.forEach((fr) => {
+ const [msg] = fr.errorMessages;
+ expect(msg).toContain(`Request blocked - received ${fr.userData.statusCode} status code.`);
+ });
+ expect(called).toBe(false);
+ });
+
+ test('should retire browser with session', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ ],
+ });
+ let controller;
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async () => {
+ return Promise.resolve();
+ },
+ maxRequestRetries: 1,
+ gotoFunction: async ({ session, browserController }) => {
+ controller = browserController;
+ session.retire();
+ },
+ });
+
+ await requestList.initialize();
+ await browserCrawler.run();
+ expect(controller.launchContext.sessionRetired).toBeTruthy();
+ });
+
+ test('should remove browser listener on session pool', async () => {
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ { url: 'http://example.com/?q=2' },
+ ],
+ });
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ maxOpenPagesPerBrowser: 1,
+ },
+ requestList,
+ useSessionPool: true,
+ handlePageFunction: async () => {
+ return Promise.resolve();
+ },
+ maxRequestRetries: 1,
+ gotoFunction: async ({ session }) => {
+ expect(session.sessionPool.listeners(EVENTS.SESSION_RETIRED)).toHaveLength(1);
+ session.retire();
+ },
+ });
+ // prevent browser to auto close the browsers
+ const teardown = browserCrawler._teardown // eslint-disable-line
+ browserCrawler._teardown = () => {// eslint-disable-line
+
+ };
+
+ await requestList.initialize();
+ await browserCrawler.run();
+ await Apify.utils.sleep(5000);
+ expect(browserCrawler.sessionPool.listeners(EVENTS.SESSION_RETIRED)).toHaveLength(0);
+ await browserCrawler.browserPool.destroy();
+ });
+
+ describe('proxy', () => {
+ let requestList;
+ beforeEach(async () => {
+ requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://example.com/?q=1' },
+ { url: 'http://example.com/?q=2' },
+ { url: 'http://example.com/?q=3' },
+ { url: 'http://example.com/?q=4' },
+ ],
+ });
+ await requestList.initialize();
+ });
+
+ afterEach(() => {
+ requestList = null;
+ });
+
+ test('browser should launch with correct proxyUrl', async () => {
+ process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
+ const status = { connected: true };
+ const fakeCall = async () => {
+ return { body: status };
+ };
+
+ const stub = sinon.stub(utilsRequest, 'requestAsBrowser').callsFake(fakeCall);
+ const proxyConfiguration = await Apify.createProxyConfiguration();
+ const generatedProxyUrl = new URL(await proxyConfiguration.newUrl()).href;
+ let browserProxy;
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ postLaunchHooks: [(pageId, browserController) => {
+ browserProxy = browserController.launchContext.proxyUrl;
+ }],
+ },
+ useSessionPool: false,
+ persistCookiesPerSession: false,
+ requestList,
+ maxRequestsPerCrawl: 1,
+ maxRequestRetries: 0,
+ gotoFunction: ({ page, request }) => page.goto(request.url, { timeout: 1000 }),
+ handlePageFunction: async () => {
+ },
+ proxyConfiguration,
+ });
+ await browserCrawler.run();
+ delete process.env[ENV_VARS.PROXY_PASSWORD];
+
+ expect(browserProxy).toEqual(generatedProxyUrl);
+
+ stub.restore();
+ });
+
+ test('handlePageFunction should expose the proxyInfo object with sessions correctly', async () => {
+ process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
+ const status = { connected: true };
+ const fakeCall = async () => {
+ return { body: status };
+ };
+
+ const stub = sinon.stub(utilsRequest, 'requestAsBrowser').callsFake(fakeCall);
+
+ const proxyConfiguration = await Apify.createProxyConfiguration();
+ const proxies = [];
+ const sessions = [];
+ const handlePageFunction = async ({ session, proxyInfo }) => {
+ proxies.push(proxyInfo);
+ sessions.push(session);
+ };
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ handlePageFunction,
+ gotoFunction: ({ page, request }) => page.goto(request.url),
+ proxyConfiguration,
+ useSessionPool: true,
+ sessionPoolOptions: {
+ maxPoolSize: 1,
+ },
+ });
+
+ await browserCrawler.run();
+
+ expect(proxies[0].sessionId).toEqual(sessions[0].id);
+ expect(proxies[1].sessionId).toEqual(sessions[1].id);
+ expect(proxies[2].sessionId).toEqual(sessions[2].id);
+ expect(proxies[3].sessionId).toEqual(sessions[3].id);
+
+ delete process.env[ENV_VARS.PROXY_PASSWORD];
+ stub.restore();
+ });
+
+ test('browser should launch with rotated custom proxy', async () => {
+ process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
+
+ const proxyConfiguration = await Apify.createProxyConfiguration({
+ proxyUrls: ['http://proxy.com:1111', 'http://proxy.com:2222', 'http://proxy.com:3333'],
+ });
+
+ const browserProxies = [];
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ maxOpenPagesPerBrowser: 1,
+ retireBrowserAfterPageCount: 1,
+ },
+ requestList,
+ handlePageFunction: async () => {
+ },
+ proxyConfiguration,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ });
+
+ browserCrawler.gotoFunction = async () => {
+ };
+
+ browserCrawler.browserPool.postLaunchHooks.push((pageId, browserController) => {
+ browserProxies.push(browserController.launchContext.proxyUrl);
+ });
+
+ await browserCrawler.run();
+
+ const proxiesToUse = proxyConfiguration.proxyUrls;
+ for (const proxyUrl of proxiesToUse) {
+ expect(browserProxies.includes(new URL(proxyUrl).href)).toBeTruthy();
+ }
+
+ delete process.env[ENV_VARS.PROXY_PASSWORD];
+ });
+ });
+ describe('Crawling context', () => {
+ const sources = ['http://example.com/'];
+ let requestList;
+ let actualLogLevel;
+ beforeEach(async () => {
+ actualLogLevel = log.getLevel();
+ log.setLevel(log.LEVELS.OFF);
+ requestList = await Apify.openRequestList(null, sources.slice());
+ });
+
+ afterAll(() => {
+ log.setLevel(actualLogLevel);
+ });
+
+ test('uses correct crawling context', async () => {
+ let prepareCrawlingContext;
+
+ const gotoFunction = async (crawlingContext) => {
+ prepareCrawlingContext = crawlingContext;
+ expect(crawlingContext.request).toBeInstanceOf(Request);
+ expect(crawlingContext.crawler.autoscaledPool).toBeInstanceOf(AutoscaledPool);
+ expect(crawlingContext.session).toBeInstanceOf(Session);
+ expect(typeof crawlingContext.page).toBe('object');
+ };
+
+ const handlePageFunction = async (crawlingContext) => {
+ expect(crawlingContext === prepareCrawlingContext).toEqual(true);
+ expect(crawlingContext.request).toBeInstanceOf(Request);
+ expect(crawlingContext.crawler.autoscaledPool).toBeInstanceOf(AutoscaledPool);
+ expect(crawlingContext.session).toBeInstanceOf(Session);
+ expect(typeof crawlingContext.page).toBe('object');
+ expect(crawlingContext.crawler).toBeInstanceOf(Apify.BrowserCrawler);
+ expect(crawlingContext.hasOwnProperty('response')).toBe(true);
+
+ throw new Error('some error');
+ };
+
+ const handleFailedRequestFunction = async (crawlingContext) => {
+ expect(crawlingContext === prepareCrawlingContext).toEqual(true);
+ expect(crawlingContext.request).toBeInstanceOf(Request);
+ expect(crawlingContext.crawler.autoscaledPool).toBeInstanceOf(AutoscaledPool);
+ expect(crawlingContext.session).toBeInstanceOf(Session);
+ expect(typeof crawlingContext.page).toBe('object');
+ expect(crawlingContext.crawler).toBeInstanceOf(Apify.BrowserCrawler);
+ expect(crawlingContext.crawler.browserPool).toBeInstanceOf(BrowserPool);
+ expect(crawlingContext.hasOwnProperty('response')).toBe(true);
+
+ expect(crawlingContext.error).toBeInstanceOf(Error);
+ expect(crawlingContext.error.message).toEqual('some error');
+ };
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ useSessionPool: true,
+ handlePageFunction,
+ handleFailedRequestFunction,
+ });
+ browserCrawler.gotoFunction = gotoFunction;
+
+ await browserCrawler.run();
+ });
+
+ test('handleFailedRequestFunction contains proxyInfo', async () => {
+ process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
+ const stub = sinon.stub(utilsRequest, 'requestAsBrowser').resolves({ body: { connected: true } });
+
+ const proxyConfiguration = await Apify.createProxyConfiguration();
+
+ const browserCrawler = new Apify.BrowserCrawler({
+ browserPoolOptions: {
+ browserPlugins: [puppeteerPlugin],
+ },
+ requestList,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ useSessionPool: true,
+ proxyConfiguration,
+ handlePageFunction: async () => {
+ throw new Error('some error');
+ },
+ handleFailedRequestFunction: async (crawlingContext) => {
+ expect(typeof crawlingContext.proxyInfo).toEqual('object');
+ expect(crawlingContext.proxyInfo.hasOwnProperty('url')).toEqual(true);
+ },
+ });
+
+ await browserCrawler.run();
+
+ delete process.env[ENV_VARS.PROXY_PASSWORD];
+ stub.restore();
+ });
+ });
+});
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 5943015d0..117abb3a4 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -848,14 +848,14 @@ describe('CheerioCrawler', () => {
const prepareRequestFunction = async (crawlingContext) => {
prepareCrawlingContext = crawlingContext;
expect(crawlingContext.request).toBeInstanceOf(Request);
- expect(crawlingContext.autoscaledPool).toBeInstanceOf(AutoscaledPool);
+ expect(crawlingContext.crawler.autoscaledPool).toBeInstanceOf(AutoscaledPool);
expect(crawlingContext.session).toBeInstanceOf(Session);
};
const handlePageFunction = async (crawlingContext) => {
expect(crawlingContext === prepareCrawlingContext).toEqual(true);
expect(crawlingContext.request).toBeInstanceOf(Request);
- expect(crawlingContext.autoscaledPool).toBeInstanceOf(AutoscaledPool);
+ expect(crawlingContext.crawler.autoscaledPool).toBeInstanceOf(AutoscaledPool);
expect(crawlingContext.session).toBeInstanceOf(Session);
expect(typeof crawlingContext.$).toBe('function');
expect(typeof crawlingContext.response).toBe('object');
@@ -867,7 +867,7 @@ describe('CheerioCrawler', () => {
const handleFailedRequestFunction = async (crawlingContext) => {
expect(crawlingContext === prepareCrawlingContext).toEqual(true);
expect(crawlingContext.request).toBeInstanceOf(Request);
- expect(crawlingContext.autoscaledPool).toBeInstanceOf(AutoscaledPool);
+ expect(crawlingContext.crawler.autoscaledPool).toBeInstanceOf(AutoscaledPool);
expect(crawlingContext.session).toBeInstanceOf(Session);
expect(typeof crawlingContext.$).toBe('function');
expect(typeof crawlingContext.response).toBe('object');
diff --git a/test/crawlers/playwright_crawler.test.js b/test/crawlers/playwright_crawler.test.js
new file mode 100644
index 000000000..20c7c643e
--- /dev/null
+++ b/test/crawlers/playwright_crawler.test.js
@@ -0,0 +1,132 @@
+import { ENV_VARS } from 'apify-shared/consts';
+import playwright from 'playwright';
+import log from '../../build/utils_log';
+import * as Apify from '../../build';
+import LocalStorageDirEmulator from '../local_storage_dir_emulator';
+import * as utils from '../../build/utils';
+
+describe('PlaywrightCrawler', () => {
+ let prevEnvHeadless;
+ let logLevel;
+ let localStorageEmulator;
+ let requestList;
+
+ beforeAll(async () => {
+ prevEnvHeadless = process.env[ENV_VARS.HEADLESS];
+ process.env[ENV_VARS.HEADLESS] = '1';
+ logLevel = log.getLevel();
+ log.setLevel(log.LEVELS.ERROR);
+ localStorageEmulator = new LocalStorageDirEmulator();
+ });
+ beforeEach(async () => {
+ const storageDir = await localStorageEmulator.init();
+ utils.apifyStorageLocal = utils.newStorageLocal({ storageDir });
+ const sources = ['http://example.com/'];
+ requestList = await Apify.openRequestList(`sources-${Math.random * 10000}`, sources);
+ });
+ afterAll(async () => {
+ log.setLevel(logLevel);
+ process.env[ENV_VARS.HEADLESS] = prevEnvHeadless;
+ await localStorageEmulator.destroy();
+ });
+
+ describe('should work', () => {
+ // @TODO: add webkit and solve te timeout issue on github actions.
+ test.each(['chromium', 'firefox'])('with %s', async (browser) => {
+ const sourcesLarge = [
+ { url: 'http://example.com/?q=1' },
+ { url: 'http://example.com/?q=2' },
+ { url: 'http://example.com/?q=3' },
+ { url: 'http://example.com/?q=4' },
+ { url: 'http://example.com/?q=5' },
+ { url: 'http://example.com/?q=6' },
+ ];
+ const sourcesCopy = JSON.parse(JSON.stringify(sourcesLarge));
+ const processed = [];
+ const failed = [];
+ const requestListLarge = new Apify.RequestList({ sources: sourcesLarge });
+ const handlePageFunction = async ({ page, request, response }) => {
+ expect(await response.status()).toBe(200);
+ request.userData.title = await page.title();
+ processed.push(request);
+ };
+
+ const playwrightCrawler = new Apify.PlaywrightCrawler({
+ playwrightModule: playwright[browser],
+ requestList: requestListLarge,
+ minConcurrency: 1,
+ maxConcurrency: 1,
+ handlePageFunction,
+ handleFailedRequestFunction: ({ request }) => failed.push(request),
+ });
+
+ await requestListLarge.initialize();
+ await playwrightCrawler.run();
+
+ expect(playwrightCrawler.autoscaledPool.minConcurrency).toBe(1);
+ expect(processed).toHaveLength(6);
+ expect(failed).toHaveLength(0);
+
+ processed.forEach((request, id) => {
+ expect(request.url).toEqual(sourcesCopy[id].url);
+ expect(request.userData.title).toBe('Example Domain');
+ });
+ });
+ });
+ test('should throw on gotoFunction', async () => {
+ try {
+ const playwrightCrawler = new Apify.PlaywrightCrawler({ //eslint-disable-line
+ requestList,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ handlePageFunction: async () => {
+ },
+ gotoFunction: () => { },
+ });
+ } catch (e) {
+ expect(e.message.includes('Expected property `gotoFunction` to be of type `undefined`')).toBeTruthy();
+ }
+
+ expect.hasAssertions();
+ });
+
+ test('should launch with webkit browser', async () => {
+ try {
+ const playwrightCrawler = new Apify.PlaywrightCrawler({ //eslint-disable-line
+ requestList,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ handlePageFunction: async () => {
+ },
+ gotoFunction: () => { },
+ });
+ } catch (e) {
+ expect(e.message.includes('Expected property `gotoFunction` to be of type `undefined`')).toBeTruthy();
+ }
+
+ expect.hasAssertions();
+ });
+
+ test('should override goto timeout ', async () => {
+ const timeoutSecs = 10;
+ let options;
+ const playwrightCrawler = new Apify.PlaywrightCrawler({ //eslint-disable-line
+ requestList,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ handlePageFunction: async () => {
+ },
+ preNavigationHooks: [(context, gotoOptions) => {
+ options = gotoOptions;
+ }],
+ gotoTimeoutSecs: timeoutSecs,
+ });
+
+ expect(playwrightCrawler.gotoOptions.timeout).toEqual(timeoutSecs * 1000);
+ await playwrightCrawler.run();
+
+ expect(options.timeout).toEqual(timeoutSecs * 1000);
+
+ expect.hasAssertions();
+ });
+});
diff --git a/test/crawlers/puppeteer_crawler.test.js b/test/crawlers/puppeteer_crawler.test.js
index 91db373d3..42c2d8b55 100644
--- a/test/crawlers/puppeteer_crawler.test.js
+++ b/test/crawlers/puppeteer_crawler.test.js
@@ -2,19 +2,14 @@ import { ENV_VARS } from 'apify-shared/consts';
import sinon from 'sinon';
import log from '../../build/utils_log';
import * as Apify from '../../build';
-import { STATUS_CODES_BLOCKED } from '../../build/constants';
import LocalStorageDirEmulator from '../local_storage_dir_emulator';
-import * as utilsRequest from '../../build/utils_request';
import * as utils from '../../build/utils';
-import Request from '../../build/request';
-import AutoscaledPool from '../../build/autoscaling/autoscaled_pool';
-import { Session } from '../../build/session_pool/session';
-import PuppeteerPool from '../../build/puppeteer_pool';
describe('PuppeteerCrawler', () => {
let prevEnvHeadless;
let logLevel;
let localStorageEmulator;
+ let requestList;
beforeAll(async () => {
prevEnvHeadless = process.env[ENV_VARS.HEADLESS];
@@ -26,6 +21,8 @@ describe('PuppeteerCrawler', () => {
beforeEach(async () => {
const storageDir = await localStorageEmulator.init();
utils.apifyStorageLocal = utils.newStorageLocal({ storageDir });
+ const sources = ['http://example.com/'];
+ requestList = await Apify.openRequestList(`sources-${Math.random * 10000}`, sources);
});
afterAll(async () => {
log.setLevel(logLevel);
@@ -34,7 +31,7 @@ describe('PuppeteerCrawler', () => {
});
test('should work', async () => {
- const sources = [
+ const sourcesLarge = [
{ url: 'http://example.com/?q=1' },
{ url: 'http://example.com/?q=2' },
{ url: 'http://example.com/?q=3' },
@@ -42,10 +39,10 @@ describe('PuppeteerCrawler', () => {
{ url: 'http://example.com/?q=5' },
{ url: 'http://example.com/?q=6' },
];
- const sourcesCopy = JSON.parse(JSON.stringify(sources));
+ const sourcesCopy = JSON.parse(JSON.stringify(sourcesLarge));
const processed = [];
const failed = [];
- const requestList = new Apify.RequestList({ sources });
+ const requestListLarge = new Apify.RequestList({ sources: sourcesLarge });
const handlePageFunction = async ({ page, request, response }) => {
await page.waitForSelector('title');
@@ -55,14 +52,14 @@ describe('PuppeteerCrawler', () => {
};
const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
+ requestList: requestListLarge,
minConcurrency: 1,
maxConcurrency: 1,
handlePageFunction,
handleFailedRequestFunction: ({ request }) => failed.push(request),
});
- await requestList.initialize();
+ await requestListLarge.initialize();
await puppeteerCrawler.run();
expect(puppeteerCrawler.autoscaledPool.minConcurrency).toBe(1);
@@ -75,450 +72,108 @@ describe('PuppeteerCrawler', () => {
});
});
- test('should ignore errors in Page.close()', async () => {
- for (let i = 0; i < 2; i++) {
- const requestList = new Apify.RequestList({
- sources: [
- { url: 'http://example.com/?q=1' },
- ],
- });
- let failedCalled = false;
-
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
+ test('should throw on gotoFunction', async () => {
+ try {
+ const puppeteerCrawler = new Apify.PuppeteerCrawler({ //eslint-disable-line
requestList,
- handlePageFunction: ({ page }) => {
- page.close = () => {
- if (i === 0) {
- throw new Error();
- } else {
- return Promise.reject(new Error());
- }
- };
- return Promise.resolve();
- },
- handleFailedRequestFunction: async () => {
- failedCalled = true;
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ handlePageFunction: async () => {
},
+ gotoFunction: () => {},
});
-
- await requestList.initialize();
- await puppeteerCrawler.run();
-
- expect(failedCalled).toBe(false);
+ } catch (e) {
+ expect(e.message.includes('Expected property `gotoFunction` to be of type `undefined`')).toBeTruthy();
}
- });
-
- test('should use SessionPool', async () => {
- const requestList = new Apify.RequestList({
- sources: [
- { url: 'http://example.com/?q=1' },
- ],
- });
- const handlePageSessions = [];
- const goToPageSessions = [];
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- useSessionPool: true,
- handlePageFunction: async ({ session }) => {
- handlePageSessions.push(session);
- return Promise.resolve();
- },
- gotoFunction: async ({ session, page, request }) => {
- goToPageSessions.push(session);
- return page.goto(request.url);
- },
- });
-
- await requestList.initialize();
- await puppeteerCrawler.run();
- expect(puppeteerCrawler.sessionPool.constructor.name).toEqual('SessionPool');
- expect(handlePageSessions).toHaveLength(1);
- expect(goToPageSessions).toHaveLength(1);
- handlePageSessions.forEach((session) => expect(session.constructor.name).toEqual('Session'));
- goToPageSessions.forEach((session) => expect(session.constructor.name).toEqual('Session'));
+ expect.hasAssertions();
});
- test('should persist cookies per session', async () => {
- const requestList = new Apify.RequestList({
- sources: [
- { url: 'http://example.com/?q=1' },
- { url: 'http://example.com/?q=2' },
- { url: 'http://example.com/?q=3' },
- { url: 'http://example.com/?q=4' },
- ],
- });
- const goToPageSessions = [];
- const loadedCookies = [];
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
+ test('should override goto timeout ', async () => {
+ const timeoutSecs = 10;
+ let options;
+ const puppeteerCrawler = new Apify.PuppeteerCrawler({ //eslint-disable-line
requestList,
- useSessionPool: true,
- persistCookiesPerSession: true,
- handlePageFunction: async ({ session, request }) => {
- loadedCookies.push(session.getCookieString(request.url));
- return Promise.resolve();
- },
- gotoFunction: async ({ session, page, request }) => {
- await page.setCookie({ name: 'TEST', value: '12321312312', domain: 'example.com', expires: Date.now() + 100000 });
- goToPageSessions.push(session);
- return page.goto(request.url);
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ handlePageFunction: async () => {
},
+ preNavigationHooks: [(context, gotoOptions) => {
+ options = gotoOptions;
+ }],
+ gotoTimeoutSecs: timeoutSecs,
});
- await requestList.initialize();
+ expect(puppeteerCrawler.gotoOptions.timeout).toEqual(timeoutSecs * 1000);
await puppeteerCrawler.run();
- expect(loadedCookies).toHaveLength(4);
- loadedCookies.forEach((cookie) => expect(cookie).toEqual('TEST=12321312312'));
- });
- test('should throw on "blocked" status codes', async () => {
- const baseUrl = 'https://example.com/';
- const sources = STATUS_CODES_BLOCKED.map((statusCode) => {
- return {
- url: baseUrl + statusCode,
- userData: { statusCode },
- };
- });
- const requestList = await Apify.openRequestList(null, sources);
+ expect(options.timeout).toEqual(timeoutSecs * 1000);
- let called = false;
- const failedRequests = [];
- const crawler = new Apify.PuppeteerCrawler({
- requestList,
- useSessionPool: true,
- persistCookiesPerSession: false,
- maxRequestRetries: 0,
- gotoFunction: async ({ request }) => {
- return { status: () => request.userData.statusCode };
- },
- handlePageFunction: async () => { // eslint-disable-line no-loop-func
- called = true;
- },
- handleFailedRequestFunction: async ({ request }) => {
- failedRequests.push(request);
- },
- });
+ expect.hasAssertions();
+ });
- await crawler.run();
+ test('should throw if launchOptions.proxyUrl and proxyConfiguration is suplied', async () => {
+ try {
+ const puppeteerCrawler = new Apify.PuppeteerCrawler({ //eslint-disable-line
+ requestList,
+ maxRequestRetries: 0,
+ maxConcurrency: 1,
+ launchContext: {
+ proxyUrl: 'http://localhost@1234',
+ },
+ proxyConfiguration: await Apify.createProxyConfiguration({ proxyUrls: ['http://localhost@1234'] }),
+ handlePageFunction: async () => {},
+ });
+ } catch (e) {
+ expect(e.message).toEqual('It is not possible to combine "options.proxyConfiguration" together with '
+ + 'custom "proxyUrl" option from "options.launchPuppeteerOptions".');
+ }
- expect(failedRequests.length).toBe(STATUS_CODES_BLOCKED.length);
- failedRequests.forEach((fr) => {
- const [msg] = fr.errorMessages;
- expect(msg).toContain(`Request blocked - received ${fr.userData.statusCode} status code.`);
- });
- expect(called).toBe(false);
+ expect.hasAssertions();
});
+ test('supports useChrome option', async () => {
+ const spy = sinon.spy(utils, 'getTypicalChromeExecutablePath');
- test('should throw on goto timeouts (markBad session)', async () => {
- const baseUrl = 'https://example.com/';
- const requestList = await Apify.openRequestList(null, [baseUrl]);
- const gotoTimeoutSecs = 0.001;
-
- let called = false;
- const failedRequests = [];
- const crawler = new Apify.PuppeteerCrawler({
+ const puppeteerCrawler = new Apify.PuppeteerCrawler({ //eslint-disable-line
requestList,
- useSessionPool: true,
- persistCookiesPerSession: false,
maxRequestRetries: 0,
- gotoTimeoutSecs,
- handlePageFunction: async () => { // eslint-disable-line no-loop-func
- called = true;
+ maxConcurrency: 1,
+ launchContext: {
+ useChrome: true,
+ launchOptions: {
+ headless: true,
+ },
},
- handleFailedRequestFunction: async ({ request }) => {
- failedRequests.push(request);
+ handlePageFunction: async () => {
},
});
- await crawler.run();
-
- expect(failedRequests.length).toBe(1);
- failedRequests.forEach((fr) => {
- const [msg] = fr.errorMessages;
- expect(msg).toContain(`gotoFunction timed out after ${gotoTimeoutSecs} seconds.`);
- });
- expect(called).toBe(false);
+ expect(spy.calledOnce).toBe(true);
+ spy.restore();
});
- test('should throw on "blocked" status codes (retire session)', async () => {
- const baseUrl = 'https://example.com/';
- const sources = STATUS_CODES_BLOCKED.map((statusCode) => {
- return {
- url: baseUrl + statusCode,
- userData: { statusCode },
- };
- });
- const requestList = await Apify.openRequestList(null, sources);
+ test('supports userAgent option', async () => {
+ const opts = {
+ // Have space in user-agent to test passing of params
+ userAgent: 'MyUserAgent/1234 AnotherString/456',
+ launchOptions: {
+ headless: true,
+ },
+ };
+ let loadedUserAgent;
- let called = false;
- const failedRequests = [];
- const crawler = new Apify.PuppeteerCrawler({
+ const puppeteerCrawler = new Apify.PuppeteerCrawler({
requestList,
- useSessionPool: true,
- persistCookiesPerSession: false,
maxRequestRetries: 0,
- gotoFunction: async ({ request }) => {
- return { status: () => request.userData.statusCode };
- },
- handlePageFunction: async () => { // eslint-disable-line no-loop-func
- called = true;
- },
- handleFailedRequestFunction: async ({ request }) => {
- failedRequests.push(request);
+ maxConcurrency: 1,
+ launchContext: opts,
+ handlePageFunction: async ({ page }) => {
+ loadedUserAgent = await page.evaluate(() => window.navigator.userAgent);
},
});
- await crawler.run();
-
- expect(failedRequests.length).toBe(STATUS_CODES_BLOCKED.length);
- failedRequests.forEach((fr) => {
- const [msg] = fr.errorMessages;
- expect(msg).toContain(`Request blocked - received ${fr.userData.statusCode} status code.`);
- });
- expect(called).toBe(false);
- });
-
- describe('proxy', () => {
- let requestList;
- beforeEach(async () => {
- requestList = new Apify.RequestList({
- sources: [
- { url: 'http://example.com/?q=1' },
- { url: 'http://example.com/?q=2' },
- { url: 'http://example.com/?q=3' },
- { url: 'http://example.com/?q=4' },
- ],
- });
- await requestList.initialize();
- });
-
- afterEach(() => {
- requestList = null;
- });
-
- test('browser should launch with correct proxyUrl', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- const status = { connected: true };
- const fakeCall = async () => {
- return { body: status };
- };
-
- const stub = sinon.stub(utilsRequest, 'requestAsBrowser').callsFake(fakeCall);
- const proxyConfiguration = await Apify.createProxyConfiguration();
- const generatedProxyUrl = proxyConfiguration.newUrl();
- let browserProxy;
- const launchPuppeteerFunction = async (opts) => {
- browserProxy = opts.proxyUrl;
- const browser = await Apify.launchPuppeteer(opts);
-
- return browser;
- };
-
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- handlePageFunction: async () => {},
- launchPuppeteerFunction,
- proxyConfiguration,
- });
-
- await puppeteerCrawler.run();
- delete process.env[ENV_VARS.PROXY_PASSWORD];
-
- expect(browserProxy).toEqual(generatedProxyUrl);
-
- stub.restore();
- });
-
- test('handlePageFunction should expose the proxyInfo object with sessions correctly', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- const status = { connected: true };
- const fakeCall = async () => {
- return { body: status };
- };
-
- const stub = sinon.stub(utilsRequest, 'requestAsBrowser').callsFake(fakeCall);
-
- const proxyConfiguration = await Apify.createProxyConfiguration();
- const proxies = [];
- const sessions = [];
- const handlePageFunction = async ({ session, proxyInfo }) => {
- proxies.push(proxyInfo);
- sessions.push(session);
- };
-
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- handlePageFunction,
- proxyConfiguration,
- useSessionPool: true,
- sessionPoolOptions: {
- maxPoolSize: 1,
- },
- });
-
- await puppeteerCrawler.run();
-
- expect(proxies[0].sessionId).toEqual(sessions[0].id);
- expect(proxies[1].sessionId).toEqual(sessions[1].id);
- expect(proxies[2].sessionId).toEqual(sessions[2].id);
- expect(proxies[3].sessionId).toEqual(sessions[3].id);
-
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- stub.restore();
- });
-
- test('browser should launch with rotated custom proxy', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
-
- const proxyConfiguration = await Apify.createProxyConfiguration({
- proxyUrls: ['http://proxy.com:1111', 'http://proxy.com:2222', 'http://proxy.com:3333'],
- });
-
- const browserProxies = [];
- const launchPuppeteerFunction = async (opts) => {
- browserProxies.push(opts.proxyUrl);
- const browser = await Apify.launchPuppeteer(opts);
-
- return browser;
- };
-
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- handlePageFunction: async () => {},
- gotoFunction: async () => {},
- launchPuppeteerFunction,
- puppeteerPoolOptions: {
- retireInstanceAfterRequestCount: 1,
- },
- proxyConfiguration,
- });
-
- await puppeteerCrawler.run();
-
- const proxiesToUse = proxyConfiguration.proxyUrls;
- expect(browserProxies[0]).toEqual(proxiesToUse[0]);
- expect(browserProxies[1]).toEqual(proxiesToUse[1]);
- expect(browserProxies[2]).toEqual(proxiesToUse[2]);
- expect(browserProxies[3]).toEqual(proxiesToUse[0]);
-
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- });
-
- test('should throw on proxyConfiguration together with proxyUrl from launchPuppeteerOptions', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
-
- const proxyConfiguration = await Apify.createProxyConfiguration({
- proxyUrls: ['http://proxy.com:1111', 'http://proxy.com:2222', 'http://proxy.com:3333'],
- });
-
- try {
- // eslint-disable-next-line no-unused-vars
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- handlePageFunction: async () => {},
- gotoFunction: async () => {},
- proxyConfiguration,
- launchPuppeteerOptions: {
- proxyUrl: 'http://proxy.com:1111',
- },
- });
- throw new Error('wrong error');
- } catch (err) {
- expect(err.message).toMatch('It is not possible to combine "options.proxyConfiguration"');
- }
-
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- });
- });
-
- describe('Crawling context', () => {
- const sources = ['http://example.com/'];
- let requestList;
- let actualLogLevel;
- beforeEach(async () => {
- actualLogLevel = log.getLevel();
- log.setLevel(log.LEVELS.OFF);
- requestList = await Apify.openRequestList(null, sources.slice());
- });
-
- afterAll(() => {
- log.setLevel(actualLogLevel);
- });
-
- test('uses correct crawling context', async () => {
- let prepareCrawlingContext;
-
- const gotoFunction = async (crawlingContext) => {
- prepareCrawlingContext = crawlingContext;
- expect(crawlingContext.request).toBeInstanceOf(Request);
- expect(crawlingContext.autoscaledPool).toBeInstanceOf(AutoscaledPool);
- expect(crawlingContext.session).toBeInstanceOf(Session);
- expect(typeof crawlingContext.page).toBe('object');
- };
-
- const handlePageFunction = async (crawlingContext) => {
- expect(crawlingContext === prepareCrawlingContext).toEqual(true);
- expect(crawlingContext.request).toBeInstanceOf(Request);
- expect(crawlingContext.autoscaledPool).toBeInstanceOf(AutoscaledPool);
- expect(crawlingContext.session).toBeInstanceOf(Session);
- expect(typeof crawlingContext.page).toBe('object');
- expect(crawlingContext.puppeteerPool).toBeInstanceOf(PuppeteerPool);
- expect(crawlingContext.hasOwnProperty('response')).toBe(true);
-
- throw new Error('some error');
- };
-
- const handleFailedRequestFunction = async (crawlingContext) => {
- expect(crawlingContext === prepareCrawlingContext).toEqual(true);
- expect(crawlingContext.request).toBeInstanceOf(Request);
- expect(crawlingContext.autoscaledPool).toBeInstanceOf(AutoscaledPool);
- expect(crawlingContext.session).toBeInstanceOf(Session);
- expect(typeof crawlingContext.page).toBe('object');
- expect(crawlingContext.puppeteerPool).toBeInstanceOf(PuppeteerPool);
- expect(crawlingContext.hasOwnProperty('response')).toBe(true);
-
- expect(crawlingContext.error).toBeInstanceOf(Error);
- expect(crawlingContext.error.message).toEqual('some error');
- };
-
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- maxRequestRetries: 0,
- maxConcurrency: 1,
- useSessionPool: true,
- gotoFunction,
- handlePageFunction,
- handleFailedRequestFunction,
- });
-
- await puppeteerCrawler.run();
- });
-
- test('handleFailedRequestFunction contains proxyInfo', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- const stub = sinon.stub(utilsRequest, 'requestAsBrowser').resolves({ body: { connected: true } });
-
- const proxyConfiguration = await Apify.createProxyConfiguration();
-
- const puppeteerCrawler = new Apify.PuppeteerCrawler({
- requestList,
- maxRequestRetries: 0,
- maxConcurrency: 1,
- useSessionPool: true,
- proxyConfiguration,
- handlePageFunction: async () => {
- throw new Error('some error');
- },
- handleFailedRequestFunction: async (crawlingContext) => {
- expect(typeof crawlingContext.proxyInfo).toEqual('object');
- expect(crawlingContext.proxyInfo.hasOwnProperty('url')).toEqual(true);
- },
- });
-
- await puppeteerCrawler.run();
+ await puppeteerCrawler.run();
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- stub.restore();
- });
+ expect(loadedUserAgent).toEqual(opts.userAgent);
});
});
diff --git a/test/enqueue_links/click_elements.test.js b/test/enqueue_links/click_elements.test.js
index fad730d46..fdf33e7c8 100644
--- a/test/enqueue_links/click_elements.test.js
+++ b/test/enqueue_links/click_elements.test.js
@@ -7,7 +7,7 @@ describe('enqueueLinksByClickingElements()', () => {
let page;
beforeAll(async () => {
- browser = await Apify.launchPuppeteer({ headless: true });
+ browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
});
afterAll(async () => {
diff --git a/test/enqueue_links/enqueue_links.test.js b/test/enqueue_links/enqueue_links.test.js
index a6f2605b0..828d4109a 100644
--- a/test/enqueue_links/enqueue_links.test.js
+++ b/test/enqueue_links/enqueue_links.test.js
@@ -42,12 +42,15 @@ describe('enqueueLinks()', () => {
log.setLevel(ll);
});
- describe('using Puppeteer', () => {
+ describe.each([
+ ['launchPuppeteer'],
+ ['launchPlaywright'],
+ ])('using %s', (launchName) => {
let browser;
let page;
beforeEach(async () => {
- browser = await Apify.launchPuppeteer({ headless: true });
+ browser = await Apify[launchName]({ launchOptions: { headless: true } });
page = await browser.newPage();
await page.setContent(HTML);
});
diff --git a/test/playwright.test.js b/test/playwright.test.js
new file mode 100644
index 000000000..0c2682fd3
--- /dev/null
+++ b/test/playwright.test.js
@@ -0,0 +1,159 @@
+import proxy from 'proxy';
+import http from 'http';
+import util from 'util';
+import portastic from 'portastic';
+import basicAuthParser from 'basic-auth-parser';
+import _ from 'underscore';
+import sinon from 'sinon';
+import { ENV_VARS } from 'apify-shared/consts';
+import Apify from '../build/index';
+import * as utils from '../build/utils';
+
+let prevEnvHeadless;
+let proxyServer;
+let proxyPort; // eslint-disable-line no-unused-vars
+const proxyAuth = { scheme: 'Basic', username: 'username', password: 'password' };
+let wasProxyCalled = false; // eslint-disable-line no-unused-vars
+
+// Setup local proxy server for the tests
+beforeAll(() => {
+ prevEnvHeadless = process.env[ENV_VARS.HEADLESS];
+ process.env[ENV_VARS.HEADLESS] = '1';
+
+ // Find free port for the proxy
+ return portastic.find({ min: 50000, max: 50100 }).then((ports) => {
+ return new Promise((resolve, reject) => {
+ const httpServer = http.createServer();
+
+ // Setup proxy authorization
+ httpServer.authenticate = function (req, fn) {
+ // parse the "Proxy-Authorization" header
+ const auth = req.headers['proxy-authorization'];
+ if (!auth) {
+ // optimization: don't invoke the child process if no
+ // "Proxy-Authorization" header was given
+ // console.log('not Proxy-Authorization');
+ return fn(null, false);
+ }
+ const parsed = basicAuthParser(auth);
+ const isEqual = _.isEqual(parsed, proxyAuth);
+ // console.log('Parsed "Proxy-Authorization": parsed: %j expected: %j : %s', parsed, proxyAuth, isEqual);
+ if (isEqual) wasProxyCalled = true;
+ fn(null, isEqual);
+ };
+
+ httpServer.on('error', reject);
+
+ proxyServer = proxy(httpServer);
+ proxyServer.listen(ports[0], () => {
+ proxyPort = proxyServer.address().port;
+ resolve();
+ });
+ });
+ });
+});
+
+afterAll(() => {
+ process.env[ENV_VARS.HEADLESS] = prevEnvHeadless;
+
+ if (proxyServer) return util.promisify(proxyServer.close).bind(proxyServer)();
+}, 5000);
+
+describe('Apify.launchPlaywright()', () => {
+ test('throws on invalid args', () => {
+ expect(Apify.launchPlaywright('some non-object')).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright(1234)).rejects.toThrow(Error);
+
+ expect(Apify.launchPlaywright({ proxyUrl: 234 })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: {} })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: 'invalidurl' })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: 'http://host-without-port' })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: 'invalid://somehost:1234' })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: 'https://user:[email protected]:1234' })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: 'socks4://user:[email protected]:1234' })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: 'socks5://user:[email protected]:1234' })).rejects.toThrow(Error);
+ expect(Apify.launchPlaywright({ proxyUrl: ' something really bad' })).rejects.toThrow(Error);
+ });
+
+ test('opens supports non-HTTP proxies without authentication', async () => {
+ const browser1 = await Apify.launchPlaywright({ proxyUrl: 'socks4://example.com:1234' });
+ browser1.close();
+
+ const browser2 = await Apify.launchPlaywright({ proxyUrl: 'socks5://example.com:1234' });
+ browser2.close();
+
+ const browser3 = await Apify.launchPlaywright({ proxyUrl: 'https://example.com:1234' });
+ browser3.close();
+
+ const browser4 = await Apify.launchPlaywright({ proxyUrl: 'HTTP://example.com:1234' });
+ browser4.close();
+ });
+
+ test('opens https://www.example.com', async () => {
+ const browser = await Apify.launchPlaywright();
+ const page = await browser.newPage();
+
+ await page.goto('https://www.example.com');
+ const html = await page.content();
+ expect(html).toMatch('<h1>Example Domain</h1>');
+ browser.close();
+ });
+
+ test('opens https://www.example.com via proxy with authentication', () => {
+ let browser;
+ let page;
+
+ // Test headless parameter
+ process.env[ENV_VARS.HEADLESS] = false;
+
+ return Apify.launchPlaywright({
+ launchOptions: { headless: true },
+ proxyUrl: `http://username:[email protected]:${proxyPort}`,
+ })
+ .then((createdBrowser) => {
+ browser = createdBrowser;
+
+ return browser.newPage();
+ })
+ .then((openedPage) => {
+ page = openedPage;
+
+ return page.goto('https://example.com');
+ })
+ .then(() => {
+ expect(wasProxyCalled).toBe(true);
+
+ return page.content();
+ })
+ .then((html) => expect(html).toMatch('<h1>Example Domain</h1>'))
+ .then(() => browser.close());
+ });
+
+ test('supports useChrome option', async () => {
+ const spy = sinon.spy(utils, 'getTypicalChromeExecutablePath');
+
+ let browser;
+ const opts = {
+ useChrome: true,
+ launchOptions: { headless: true },
+ };
+
+ try {
+ browser = await Apify.launchPlaywright(opts);
+ const page = await browser.newPage();
+
+ // Add a test to go to an actual domain because we've seen issues
+ // where pages would not load at all with Chrome.
+ await page.goto('https://example.com');
+ const title = await page.title();
+ const version = await browser.version();
+
+ expect(title).toBe('Example Domain');
+ expect(version).not.toMatch('Chromium');
+ expect(spy.calledOnce).toBe(true);
+ } finally {
+ spy.restore();
+ if (browser) await browser.close();
+ }
+ });
+});
diff --git a/test/playwright_utils.test.js b/test/playwright_utils.test.js
new file mode 100644
index 000000000..87ec8143e
--- /dev/null
+++ b/test/playwright_utils.test.js
@@ -0,0 +1,52 @@
+import playwright from 'playwright';
+import Apify from '../build/index';
+import LocalStorageDirEmulator from './local_storage_dir_emulator';
+import * as utils from '../build/utils';
+
+const { utils: { log } } = Apify;
+
+describe('Apify.utils.playwright', () => {
+ let ll;
+ let localStorageEmulator;
+
+ beforeAll(async () => {
+ ll = log.getLevel();
+ log.setLevel(log.LEVELS.ERROR);
+ localStorageEmulator = new LocalStorageDirEmulator();
+ });
+
+ beforeEach(async () => {
+ const storageDir = await localStorageEmulator.init();
+ utils.apifyStorageLocal = utils.newStorageLocal({ storageDir });
+ });
+
+ afterAll(async () => {
+ log.setLevel(ll);
+ await localStorageEmulator.destroy();
+ });
+
+ test('gotoExtended() works', async () => {
+ const browser = await playwright.chromium.launch({ headless: true });
+
+ try {
+ const page = await browser.newPage();
+ const request = new Apify.Request({
+ url: 'https://api.apify.com/v2/browser-info',
+ method: 'POST',
+ headers: {
+ 'Content-Type': 'application/json; charset=utf-8',
+ },
+ payload: '{ "foo": "bar" }',
+ });
+
+ const response = await Apify.utils.playwright.gotoExtended(page, request);
+
+ const { method, headers, bodyLength } = JSON.parse(await response.text());
+ expect(method).toBe('POST');
+ expect(bodyLength).toBe(16);
+ expect(headers['content-type']).toBe('application/json; charset=utf-8');
+ } finally {
+ await browser.close();
+ }
+ }, 60000);
+});
diff --git a/test/puppeteer.test.js b/test/puppeteer.test.js
index 7990f9dc0..37acec910 100644
--- a/test/puppeteer.test.js
+++ b/test/puppeteer.test.js
@@ -121,7 +121,7 @@ describe('Apify.launchPuppeteer()', () => {
process.env[ENV_VARS.HEADLESS] = false;
return Apify.launchPuppeteer({
- headless: true,
+ launchOptions: { headless: true },
proxyUrl: `http://username:[email protected]:${proxyPort}`,
})
.then((createdBrowser) => {
@@ -149,7 +149,7 @@ describe('Apify.launchPuppeteer()', () => {
const opts = {
// Have space in user-agent to test passing of params
userAgent: 'MyUserAgent/1234 AnotherString/456',
- headless: true,
+ launchOptions: { headless: true },
};
return Apify.launchPuppeteer(opts)
.then((result) => {
@@ -177,7 +177,7 @@ describe('Apify.launchPuppeteer()', () => {
let browser;
const opts = {
useChrome: true,
- headless: true,
+ launchOptions: { headless: true },
};
try {
@@ -205,7 +205,7 @@ describe('Apify.launchPuppeteer()', () => {
try {
browser = await Apify.launchPuppeteer({
puppeteerModule: 'puppeteer',
- headless: true,
+ launchOptions: { headless: true },
});
} finally {
if (browser) await browser.close();
@@ -223,7 +223,7 @@ describe('Apify.launchPuppeteer()', () => {
launch: async () => {},
someProps,
},
- headless: true,
+ launchOptions: { headless: true },
});
} finally {
if (browser) await browser.close();
diff --git a/test/puppeteer_pool.test.js b/test/puppeteer_pool.test.js
deleted file mode 100644
index 2028a8ff2..000000000
--- a/test/puppeteer_pool.test.js
+++ /dev/null
@@ -1,728 +0,0 @@
-import fs from 'fs';
-import _ from 'underscore';
-import { ENV_VARS } from 'apify-shared/consts';
-import sinon from 'sinon';
-import log from '../build/utils_log';
-import * as Apify from '../build/index';
-import { launchPuppeteer } from '../build/puppeteer';
-import { SessionPool } from '../build/session_pool/session_pool';
-
-import LocalStorageDirEmulator from './local_storage_dir_emulator';
-import * as utilsRequest from '../build/utils_request';
-import * as utils from '../build/utils';
-
-const shortSleep = (millis = 25) => new Promise((resolve) => setTimeout(resolve, millis));
-
-describe('PuppeteerPool', () => {
- let prevEnvHeadless;
- let logLevel;
-
- beforeAll(() => {
- logLevel = log.getLevel();
- log.setLevel(log.LEVELS.ERROR);
- prevEnvHeadless = process.env[ENV_VARS.HEADLESS];
- process.env[ENV_VARS.HEADLESS] = '1';
- });
-
- afterAll(() => {
- log.setLevel(logLevel);
- process.env[ENV_VARS.HEADLESS] = prevEnvHeadless;
- });
-
- test('should work', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- process.env[ENV_VARS.PROXY_HOSTNAME] = 'my.host.com';
- process.env[ENV_VARS.PROXY_PORT] = 123;
-
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 3,
- retireInstanceAfterRequestCount: 5,
- });
- const browsers = [];
-
- // Open 6 pages 3 in both browsers.
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- await Promise.all(browsers);
-
- expect(_.values(pool.activeInstances).length).toBe(2);
- expect(pool.activeInstances[0].activePages).toBe(3);
- expect(pool.activeInstances[0].totalPages).toBe(3);
- expect(pool.activeInstances[1].activePages).toBe(3);
- expect(pool.activeInstances[1].totalPages).toBe(3);
-
- // Close 2 pages in a first browser.
- await (await browsers[0]).close();
- await (await browsers[1]).close();
- await shortSleep();
-
- expect(_.values(pool.activeInstances).length).toBe(2);
- expect(pool.activeInstances[0].activePages).toBe(1);
- expect(pool.activeInstances[0].totalPages).toBe(3);
- expect(pool.activeInstances[1].activePages).toBe(3);
- expect(pool.activeInstances[1].totalPages).toBe(3);
-
- // Open two more pages in first browser so it reaches 5 and gets retired.
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- await Promise.all(browsers);
- expect(_.values(pool.activeInstances).length).toBe(1);
- expect(_.values(pool.retiredInstances).length).toBe(1);
- expect(pool.activeInstances[1].activePages).toBe(3);
- expect(pool.activeInstances[1].totalPages).toBe(3);
- expect(pool.retiredInstances[0].activePages).toBe(3);
- expect(pool.retiredInstances[0].totalPages).toBe(5);
-
- // Open one more page to see that 3rd browser gets started.
- browsers.push(pool.newPage());
- await Promise.all(browsers);
-
- expect(_.values(pool.activeInstances).length).toBe(2);
- expect(_.values(pool.retiredInstances).length).toBe(1);
- expect(pool.activeInstances[1].activePages).toBe(3);
- expect(pool.activeInstances[1].totalPages).toBe(3);
- expect(pool.activeInstances[2].activePages).toBe(1);
- expect(pool.activeInstances[2].totalPages).toBe(1);
- expect(pool.retiredInstances[0].activePages).toBe(3);
- expect(pool.retiredInstances[0].totalPages).toBe(5);
-
- // Kill the remaining 3 pages from the 1st browser to see that it gets closed.
- await (await browsers[2]).close();
- await (await browsers[6]).close();
-
- await (await browsers[7]).close();
- await shortSleep(2000);
- expect(_.values(pool.retiredInstances).length).toBe(0);
-
- // Cleanup everything.
- await pool.destroy();
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- delete process.env[ENV_VARS.PROXY_HOSTNAME];
- delete process.env[ENV_VARS.PROXY_PORT];
- });
-
- test('kills hanging retired instances', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- process.env[ENV_VARS.PROXY_HOSTNAME] = 'my.host.com';
- process.env[ENV_VARS.PROXY_PORT] = 123;
-
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 3,
- retireInstanceAfterRequestCount: 5,
- instanceKillerIntervalSecs: 1,
- killInstanceAfterSecs: 0.5,
- });
- const browsers = [];
-
- // Open 3 pages.
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- await Promise.all(browsers);
- expect(_.values(pool.activeInstances).length).toBe(1);
- expect(_.values(pool.retiredInstances).length).toBe(0);
-
- // Close 2.
- await (await browsers[0]).close();
- await (await browsers[1]).close();
- await shortSleep();
- expect(_.values(pool.activeInstances).length).toBe(1);
- expect(_.values(pool.retiredInstances).length).toBe(0);
-
- // Open 2 more so that the browser gets retired.
- browsers.push(pool.newPage());
- browsers.push(pool.newPage());
- await Promise.all(browsers);
-
- // Check that it's retired.
- expect(_.values(pool.activeInstances).length).toBe(0);
- expect(_.values(pool.retiredInstances).length).toBe(1);
- expect(pool.retiredInstances[0].activePages).toBe(3);
- expect(pool.retiredInstances[0].totalPages).toBe(5);
-
- // Sleep and check that it has been killed.
- await shortSleep(2100);
- expect(_.values(pool.activeInstances).length).toBe(0);
- expect(_.values(pool.retiredInstances).length).toBe(0);
-
- // Cleanup everything.
- await pool.destroy();
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- delete process.env[ENV_VARS.PROXY_HOSTNAME];
- delete process.env[ENV_VARS.PROXY_PORT];
- });
-
- test('retire manually', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- process.env[ENV_VARS.PROXY_HOSTNAME] = 'my.host.com';
- process.env[ENV_VARS.PROXY_PORT] = 123;
-
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 1,
- instanceKillerIntervalSecs: 1,
- killInstanceAfterSecs: 0.5,
- });
- const pages = [];
-
- // Open 3 pages.
- pages.push(pool.newPage());
- pages.push(pool.newPage());
- pages.push(pool.newPage());
- await Promise.all(pages);
- expect(_.values(pool.activeInstances).length).toBe(3);
- expect(_.values(pool.retiredInstances).length).toBe(0);
-
- // Retire 1.
- await pool.retire((await pages[0]).browser());
- expect(_.values(pool.activeInstances).length).toBe(2);
- expect(_.values(pool.retiredInstances).length).toBe(1);
-
- // Retire 2.
- await pool.retire((await pages[1]).browser());
- expect(_.values(pool.activeInstances).length).toBe(1);
- expect(_.values(pool.retiredInstances).length).toBe(2);
-
- // Sleep and check that two have been killed.
- await shortSleep(2100);
- expect(_.values(pool.activeInstances).length).toBe(1);
- expect(_.values(pool.retiredInstances).length).toBe(0);
-
- // Cleanup everything.
- await pool.destroy();
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- delete process.env[ENV_VARS.PROXY_HOSTNAME];
- delete process.env[ENV_VARS.PROXY_PORT];
- });
-
- test('works with one page per instance', async () => {
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 1,
- retireInstanceAfterRequestCount: 1,
- });
-
- const page1 = await pool.newPage();
- await page1.goto('about:blank');
- const pid1 = page1.browser().process().pid;
-
- const page2 = await pool.newPage();
- await page2.goto('about:blank');
- const pid2 = page2.browser().process().pid;
-
- await page1.close();
- await page2.close();
-
- const page3 = await pool.newPage();
- await page3.goto('about:blank');
- const pid3 = page3.browser().process().pid;
-
- await page3.close();
-
- // Ensure we spawned 3 different processes
- expect(pid1).not.toEqual(pid2);
- expect(pid2).not.toEqual(pid3);
- expect(pid3).not.toEqual(pid1);
-
- // Cleanup everything.
- await pool.destroy();
- });
-
- test('does not create more than maxOpenPagesPerInstance', async () => {
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 2,
- retireInstanceAfterRequestCount: 100,
- });
- const opennedPages = [];
-
- for (let i = 0; i < 12; i++) {
- opennedPages.push(pool.newPage());
- }
- await Promise.all(opennedPages);
-
- const instances = Object.values(pool.activeInstances);
-
- expect(instances.length).toEqual(6);
-
- instances.forEach((instance) => {
- expect(instance.activePages).toEqual(2);
- });
-
- // Cleanup everything.
- await pool.destroy();
- });
-
- test('supports recycleDiskCache option', async () => {
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 1,
- retireInstanceAfterRequestCount: 1,
- recycleDiskCache: true,
- launchPuppeteerOptions: { headless: true },
- });
-
- // log.setLevel(log.LEVELS.DEBUG);
-
- const url = 'https://www.wikipedia.org';
-
- const page1 = await pool.newPage();
- const dir1 = page1.browser().recycleDiskCacheDir;
-
- console.log(dir1);
-
- expect(fs.existsSync(dir1)).toBe(true);
-
- // First time, nothing can come from disk cache
- let fromDiskCache1 = 0;
- page1.on('response', (response) => {
- if (response._fromDiskCache) fromDiskCache1++; // eslint-disable-line no-underscore-dangle
- // console.log(response._url + ": " + response.fromCache() + "/" + response._fromDiskCache);
- });
-
- const cookies1before = await page1.cookies(url);
- expect(cookies1before.length).toBe(0);
-
- await page1.goto(url);
-
- expect(fromDiskCache1).toBe(0);
-
- await Apify.utils.sleep(1000);
-
- const cookies1after = await page1.cookies();
- expect(cookies1after.length).toBeGreaterThanOrEqual(1);
-
- await page1.close();
-
- // Wait for browser to close
- await Apify.utils.sleep(5000);
-
- // User directory must be the same
- const page2 = await pool.newPage();
- const dir2 = page2.browser().recycleDiskCacheDir;
- expect(dir1).toEqual(dir2);
- expect(fs.existsSync(dir2)).toBe(true);
-
- // Ensure at least few assets are loaded from disk cache
- let fromDiskCache2 = 0;
- page2.on('response', (response) => {
- if (response._fromDiskCache) fromDiskCache2++; // eslint-disable-line no-underscore-dangle
- });
-
- const cookies2before = await page2.cookies(url);
- expect(cookies2before.length).toBe(0);
-
- await page2.goto(url);
-
- expect(fromDiskCache2).toBeGreaterThanOrEqual(1);
-
- // Open third browser while second is still open, it should use a new cache directory
- const page3 = await pool.newPage();
- const dir3 = page3.browser().recycleDiskCacheDir;
- expect(dir3).not.toEqual(dir2);
- expect(fs.existsSync(dir3)).toBe(true);
-
- await page2.close();
-
- // Cleanup everything.
- await pool.destroy();
-
- // Check cache dirs were deleted
- expect(fs.existsSync(dir1)).toBe(false);
- expect(fs.existsSync(dir3)).toBe(false);
- }, 20000);
-
- describe('reuse of browser tabs', () => {
- xit('should work', async () => {
- const pool = new Apify.PuppeteerPool({
- reusePages: true,
- });
- const firstPage = await pool.newPage();
- const secondPage = await pool.newPage();
- await pool.recyclePage(firstPage);
- const recycledFirstPage = await pool.newPage();
- const thirdPage = await pool.newPage();
- await pool.recyclePage(thirdPage);
- await pool.recyclePage(secondPage);
- const recycledThirdPage = await pool.newPage();
- const recycledSecondPage = await pool.newPage();
-
- expect(recycledFirstPage === firstPage).toBe(true);
- expect(recycledSecondPage === secondPage).toBe(true);
- expect(recycledThirdPage === thirdPage).toBe(true);
-
- expect(firstPage === secondPage).toBe(false);
- expect(firstPage === thirdPage).toBe(false);
- expect(firstPage === recycledSecondPage).toBe(false);
- expect(firstPage === recycledThirdPage).toBe(false);
- expect(secondPage === thirdPage).toBe(false);
- expect(secondPage === recycledFirstPage).toBe(false);
- expect(secondPage === recycledThirdPage).toBe(false);
- expect(thirdPage === recycledFirstPage).toBe(false);
- expect(thirdPage === recycledSecondPage).toBe(false);
-
- await pool.destroy();
- });
-
- xit('should not open new browsers when idle pages are available', async () => {
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 1,
- reusePages: true,
- });
-
- const pageOne = await pool.newPage();
- await pool.recyclePage(pageOne);
- expect(Object.keys(pool.activeInstances).length).toBe(1);
-
- const pageTwo = await pool.newPage();
- expect(pageOne === pageTwo).toBe(true);
- await pool.recyclePage(pageTwo);
- expect(Object.keys(pool.activeInstances).length).toBe(1);
-
- const pageThree = await pool.newPage();
- expect(pageTwo === pageThree).toBe(true);
- expect(Object.keys(pool.activeInstances).length).toBe(1);
-
- const pageFour = await pool.newPage();
- expect(pageFour === pageThree).toBe(false);
- expect(Object.keys(pool.activeInstances).length).toBe(2);
-
- await pool.recyclePage(pageThree);
- await pool.recyclePage(pageFour);
- await pool.newPage();
- await pool.newPage();
- expect(Object.keys(pool.activeInstances).length).toBe(2);
-
- await pool.destroy();
- });
-
- xit('should count towards retireInstanceAfterRequestCount option', async () => {
- const pool = new Apify.PuppeteerPool({
- retireInstanceAfterRequestCount: 2,
- reusePages: true,
- });
-
- const len = (obj) => Object.keys(obj).length;
-
- let page = await pool.newPage();
- await pool.recyclePage(page);
- expect(len(pool.activeInstances)).toBe(1);
-
- page = await pool.newPage();
- await pool.recyclePage(page);
- expect(len(pool.activeInstances)).toBe(0);
-
- page = await pool.newPage();
- expect(len(pool.activeInstances)).toBe(1);
-
- const pageTwo = await pool.newPage();
- expect(len(pool.activeInstances)).toBe(0);
-
- await pool.recyclePage(page);
- await pool.recyclePage(pageTwo);
- expect(len(pool.activeInstances)).toBe(0);
- await pool.newPage();
- expect(len(pool.activeInstances)).toBe(1);
- await pool.newPage();
- expect(len(pool.activeInstances)).toBe(0);
-
- await pool.destroy();
- });
-
- xit('should close pages in retired instances', async () => {
- const pool = new Apify.PuppeteerPool({
- retireInstanceAfterRequestCount: 1,
- reusePages: true,
- });
-
- // Open 3 pages, should have 3 retired instances later.
- const pages = [
- await pool.newPage(),
- await pool.newPage(),
- await pool.newPage(),
- ];
-
- let closeCounter = 0;
-
- // Recycle all pages, this should just make them idle.
- for (const p of pages) {
- const { close } = p;
- p.close = (...args) => { // eslint-disable-line no-loop-func
- closeCounter++;
- close.apply(p, ...args);
- };
- await pool.recyclePage(p);
- }
-
- // Get one new page. This should flush all the idle ones because
- // they are in retired instances.
- await pool.newPage();
-
- expect(closeCounter).toBe(3);
-
- await pool.destroy();
- });
-
- xit('should skip closed pages', async () => {
- const pool = new Apify.PuppeteerPool({ reusePages: true });
-
- const firstPage = await pool.newPage();
- const secondPage = await pool.newPage();
- await pool.recyclePage(firstPage);
- await pool.recyclePage(secondPage);
- await firstPage.close();
- const recycledSecondPage = await pool.newPage();
- const thirdPage = await pool.newPage();
- await thirdPage.close();
- const fourthPage = await pool.newPage();
- await secondPage.close();
- const fifthPage = await pool.newPage();
- await pool.recyclePage(recycledSecondPage);
- await pool.recyclePage(thirdPage);
- await pool.recyclePage(fourthPage);
- await pool.recyclePage(fifthPage);
- await fourthPage.close();
- const recycledFifthPage = await pool.newPage();
-
- expect(recycledSecondPage === secondPage).toBe(true);
- expect(recycledFifthPage === fifthPage).toBe(true);
-
- const all = [firstPage, secondPage, thirdPage, fourthPage, fifthPage, recycledSecondPage, recycledFifthPage];
- const matches = (page) => all.filter((x) => page === x).length;
-
- expect(matches(firstPage)).toBe(1);
- expect(matches(secondPage)).toBe(2);
- expect(matches(thirdPage)).toBe(1);
- expect(matches(fourthPage)).toBe(1);
- expect(matches(fifthPage)).toBe(2);
-
- await pool.destroy();
- });
- });
-
- describe('useIncognitoPages', () => {
- test('opens a page in incognito browser context', async () => {
- const pool = new Apify.PuppeteerPool({
- useIncognitoPages: true,
- });
- const page = await pool.newPage();
- const browser = page.browser();
- const context = page.browserContext();
- expect(context.isIncognito()).toBe(true);
- await page.close();
- const contexts = browser.browserContexts();
- expect(contexts).toHaveLength(1);
- expect(contexts[0].isIncognito()).toBe(false);
- await pool.destroy();
- });
- });
-
- describe('the proxyConfiguration', () => {
- test('correctly creates the proxyUrl', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- const status = { connected: true };
- const fakeCall = async () => {
- return { body: status };
- };
-
- const stub = sinon.stub(utilsRequest, 'requestAsBrowser').callsFake(fakeCall);
- const proxyConfiguration = await Apify.createProxyConfiguration({
- groups: ['G1', 'G2'],
- });
-
- const optionsLog = [];
- const proxyUrls = [];
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 1,
- proxyConfiguration,
- launchPuppeteerFunction: async (launchOpts) => {
- optionsLog.push(launchOpts);
- return launchPuppeteer(launchOpts);
- },
- });
-
- // Open 4 browsers to do rotation cycle
- const page1 = await pool.newPage();
- const page2 = await pool.newPage();
- const page3 = await pool.newPage();
- const page4 = await pool.newPage();
- /* eslint-disable no-underscore-dangle */
- proxyUrls.push(pool._getBrowserInstance(page1).proxyInfo.url);
- proxyUrls.push(pool._getBrowserInstance(page2).proxyInfo.url);
- proxyUrls.push(pool._getBrowserInstance(page3).proxyInfo.url);
- proxyUrls.push(pool._getBrowserInstance(page4).proxyInfo.url);
-
- await pool.destroy();
-
- expect(optionsLog).toHaveLength(4);
- expect(optionsLog[0].proxyUrl).toEqual(proxyUrls[0]);
- expect(optionsLog[1].proxyUrl).toEqual(proxyUrls[1]);
- expect(optionsLog[2].proxyUrl).toEqual(proxyUrls[2]);
- expect(optionsLog[3].proxyUrl).toEqual(proxyUrls[3]);
-
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- stub.restore();
- });
-
- test('supports rotation of custom proxies', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
- const optionsLog = [];
-
- const proxyConfiguration = await Apify.createProxyConfiguration({
- proxyUrls: ['http://proxy.com:1111', 'http://proxy.com:2222', 'http://proxy.com:3333'],
- });
- const pool = new Apify.PuppeteerPool({
- maxOpenPagesPerInstance: 1,
- launchPuppeteerFunction: async (launchOpts) => {
- optionsLog.push(launchOpts);
- return launchPuppeteer(launchOpts);
- },
- proxyConfiguration,
- });
- const proxies = proxyConfiguration.proxyUrls;
- // Open 4 browsers to do full rotation cycle
- await pool.newPage();
- await pool.newPage();
- await pool.newPage();
- await pool.newPage();
- await pool.destroy();
-
- expect(optionsLog).toHaveLength(4);
- expect(optionsLog[0].proxyUrl).toEqual(proxies[0]);
- expect(optionsLog[1].proxyUrl).toEqual(proxies[1]);
- expect(optionsLog[2].proxyUrl).toEqual(proxies[2]);
- expect(optionsLog[3].proxyUrl).toEqual(proxies[0]);
-
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- });
-
- test('should throw on proxyConfiguration together with proxyUrl from launchPuppeteerOptions', async () => {
- process.env[ENV_VARS.PROXY_PASSWORD] = 'abc123';
-
- const proxyConfiguration = await Apify.createProxyConfiguration({
- proxyUrls: ['http://proxy.com:1111', 'http://proxy.com:2222', 'http://proxy.com:3333'],
- });
-
- try {
- // eslint-disable-next-line no-unused-vars
- const pool = new Apify.PuppeteerPool({
- proxyConfiguration,
- launchPuppeteerOptions: {
- proxyUrl: 'http://proxy.com:1111',
- },
- });
- throw new Error('wrong error');
- } catch (err) {
- expect(err.message).toMatch('It is not possible to combine "options.proxyConfiguration"');
- }
-
- delete process.env[ENV_VARS.PROXY_PASSWORD];
- });
- });
-
- describe('prevents hanging of puppeteer operations', () => {
- let pool;
- beforeEach(() => {
- log.setLevel(log.LEVELS.OFF);
- pool = new Apify.PuppeteerPool({
- launchPuppeteerOptions: {
- headless: true,
- },
- });
- });
- afterEach(async () => {
- log.setLevel(log.LEVELS.ERROR);
- await pool.destroy();
- });
-
- test('should work', async () => {
- // Start browser;
- await pool._openNewTab(); // eslint-disable-line no-underscore-dangle
- pool.puppeteerOperationTimeoutMillis = 0.005;
- try {
- await pool._openNewTab(); // eslint-disable-line no-underscore-dangle
- throw new Error('invalid error');
- } catch (err) {
- expect(err.stack).toMatch('browser.newPage() timed out.');
- }
- });
- });
-
- describe('enables LiveView', () => {
- beforeAll(() => {
- process.env[ENV_VARS.IS_AT_HOME] = '1';
- });
- afterAll(() => {
- delete process.env[ENV_VARS.IS_AT_HOME];
- });
- test('should work', async () => {
- const serveCalledWith = [];
- let started = 0;
- let stopped = 0;
- const pool = new Apify.PuppeteerPool({
- launchPuppeteerOptions: { headless: true },
- useLiveView: true,
- });
- pool.liveViewServer.start = async () => {
- started++;
- pool.liveViewServer._isRunning = true; // eslint-disable-line no-underscore-dangle
- pool.liveViewServer.clientCount++;
- };
- pool.liveViewServer.serve = async (arg) => serveCalledWith.push(arg);
- pool.liveViewServer.stop = async () => stopped++;
-
- for (let i = 0; i < 3; i++) {
- const page = await pool.newPage();
- await pool.serveLiveViewSnapshot(page);
- await pool.recyclePage(page);
- }
-
- expect(started).toBe(1);
- expect(serveCalledWith).toHaveLength(3);
- serveCalledWith.forEach((item) => expect(item.constructor.name === 'Page'));
-
- await pool.destroy();
- expect(stopped).toBe(1);
- });
- });
-
- describe('uses sessionPool', () => {
- let localStorageEmulator;
-
- beforeAll(async () => {
- localStorageEmulator = new LocalStorageDirEmulator();
- });
-
- beforeEach(async () => {
- const storageDir = await localStorageEmulator.init();
- utils.apifyStorageLocal = utils.newStorageLocal({ storageDir });
- });
-
- afterAll(async () => {
- await localStorageEmulator.destroy();
- });
-
- test('should work', async () => {
- const sessionPool = new SessionPool();
- await sessionPool.initialize();
- const pool = new Apify.PuppeteerPool({
- launchPuppeteerOptions: { headless: true },
- sessionPool,
- });
- expect(pool.sessionPool.constructor.name).toEqual('SessionPool');
- const page = await pool.newPage();
- // eslint-disable-next-line no-underscore-dangle
- const browserSession = pool._getBrowserInstance(page).session;
-
- expect(browserSession.id).toEqual(sessionPool.sessions[0].id);
- expect(
- Object.values(pool.activeInstances).filter((instance) => instance.session.id === browserSession.id),
- ).toHaveLength(1);
-
- sessionPool.sessions[0].retire();
-
- await utils.sleep(2000);
- expect(Object.values(pool.activeInstances)).toHaveLength(0);
- });
- });
-});
diff --git a/test/puppeteer_request_interception.test.js b/test/puppeteer_request_interception.test.js
index 5b6114c1f..19dd9ad19 100644
--- a/test/puppeteer_request_interception.test.js
+++ b/test/puppeteer_request_interception.test.js
@@ -13,7 +13,7 @@ const HTML_PAGE = `<html><body>
describe('Apify.utils.puppeteer.addInterceptRequestHandler|removeInterceptRequestHandler()', () => {
test('should allow multiple handlers', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
const allUrls = [];
const loadedUrls = [];
@@ -61,7 +61,7 @@ describe('Apify.utils.puppeteer.addInterceptRequestHandler|removeInterceptReques
test(
'should not propagate aborted/responded requests to following handlers',
async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
const propagatedUrls = [];
try {
@@ -100,7 +100,7 @@ describe('Apify.utils.puppeteer.addInterceptRequestHandler|removeInterceptReques
);
test('should allow to modify request', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
try {
const page = await browser.newPage();
@@ -137,7 +137,7 @@ describe('Apify.utils.puppeteer.addInterceptRequestHandler|removeInterceptReques
});
test('should allow async handler', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
try {
const page = await browser.newPage();
@@ -178,7 +178,7 @@ describe('Apify.utils.puppeteer.addInterceptRequestHandler|removeInterceptReques
});
test('should correctly capitalize headers', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
try {
const page = await browser.newPage();
@@ -222,7 +222,7 @@ describe('Apify.utils.puppeteer.addInterceptRequestHandler|removeInterceptReques
describe('Apify.utils.puppeteer.removeInterceptRequestHandler()', () => {
test('works', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
const loadedUrls = [];
diff --git a/test/puppeteer_utils.test.js b/test/puppeteer_utils.test.js
index 03c4352c6..552064428 100644
--- a/test/puppeteer_utils.test.js
+++ b/test/puppeteer_utils.test.js
@@ -26,374 +26,378 @@ describe('Apify.utils.puppeteer', () => {
log.setLevel(ll);
await localStorageEmulator.destroy();
});
-
- test('injectFile()', async () => {
+ describe.each([
+ ['launchPuppeteer', { launchOptions: { headless: true } }],
+ // ['launchPlaywright', { launchOptions: { headless: true } }],
+ ])('with %s', (launchName, launchContext) => {
+ test('injectFile()', async () => {
/* eslint-disable no-shadow */
- const browser = await Apify.launchPuppeteer({ headless: true });
- const survive = async (browser) => {
+ const browser = await Apify[launchName](launchContext);
+ const survive = async (browser) => {
// Survive navigations
- const page = await browser.newPage();
- let result = await page.evaluate(() => window.injectedVariable === 42);
- expect(result).toBe(false);
- await Apify.utils.puppeteer.injectFile(page, path.join(__dirname, 'data', 'inject_file.txt'), { surviveNavigations: true });
- result = await page.evaluate(() => window.injectedVariable);
- expect(result).toBe(42);
- await page.goto('about:chrome');
- result = await page.evaluate(() => window.injectedVariable);
- expect(result).toBe(42);
- await page.goto('https://www.example.com');
- result = await page.evaluate(() => window.injectedVariable);
- expect(result).toBe(42);
- };
- const remove = async (browser) => {
+ const page = await browser.newPage();
+ let result = await page.evaluate(() => window.injectedVariable === 42);
+ expect(result).toBe(false);
+ await Apify.utils.puppeteer.injectFile(page, path.join(__dirname, 'data', 'inject_file.txt'), { surviveNavigations: true });
+ result = await page.evaluate(() => window.injectedVariable);
+ expect(result).toBe(42);
+ await page.goto('about:chrome');
+ result = await page.evaluate(() => window.injectedVariable);
+ expect(result).toBe(42);
+ await page.goto('https://www.example.com');
+ result = await page.evaluate(() => window.injectedVariable);
+ expect(result).toBe(42);
+ };
+ const remove = async (browser) => {
// Remove with navigations
- const page = await browser.newPage();
- let result = await page.evaluate(() => window.injectedVariable === 42);
- expect(result).toBe(false);
- await page.goto('about:chrome');
- result = await page.evaluate(() => window.injectedVariable === 42);
- expect(result).toBe(false);
- await Apify.utils.puppeteer.injectFile(page, path.join(__dirname, 'data', 'inject_file.txt'));
- result = await page.evaluate(() => window.injectedVariable);
- expect(result).toBe(42);
- await page.goto('https://www.example.com');
- result = await page.evaluate(() => window.injectedVariable === 42);
- expect(result).toBe(false);
- };
- try {
- await Promise.all([survive(browser), remove(browser)]);
- } finally {
- browser.close();
- }
- });
-
- test('injectJQuery()', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
-
- try {
- const page = await browser.newPage();
- await page.goto('about:blank');
-
- // NOTE: Chrome already defines window.$ as alias to document.querySelector(),
- // (https://developers.google.com/web/tools/chrome-devtools/console/command-line-reference#queryselector)
- const result1 = await page.evaluate(() => {
- return {
- isDefined: window.jQuery !== undefined,
- };
- });
- expect(result1).toEqual({
- isDefined: false,
- });
+ const page = await browser.newPage();
+ let result = await page.evaluate(() => window.injectedVariable === 42);
+ expect(result).toBe(false);
+ await page.goto('about:chrome');
+ result = await page.evaluate(() => window.injectedVariable === 42);
+ expect(result).toBe(false);
+ await Apify.utils.puppeteer.injectFile(page, path.join(__dirname, 'data', 'inject_file.txt'));
+ result = await page.evaluate(() => window.injectedVariable);
+ expect(result).toBe(42);
+ await page.goto('https://www.example.com');
+ result = await page.evaluate(() => window.injectedVariable === 42);
+ expect(result).toBe(false);
+ };
+ try {
+ await Promise.all([survive(browser), remove(browser)]);
+ } finally {
+ browser.close();
+ }
+ });
- await Apify.utils.puppeteer.injectJQuery(page);
- const result2 = await page.evaluate(() => {
+ test('injectJQuery()', async () => {
+ const browser = await Apify[launchName](launchContext);
+
+ try {
+ const page = await browser.newPage();
+ await page.goto('about:blank');
+
+ // NOTE: Chrome already defines window.$ as alias to document.querySelector(),
+ // (https://developers.google.com/web/tools/chrome-devtools/console/command-line-reference#queryselector)
+ const result1 = await page.evaluate(() => {
+ return {
+ isDefined: window.jQuery !== undefined,
+ };
+ });
+ expect(result1).toEqual({
+ isDefined: false,
+ });
+
+ await Apify.utils.puppeteer.injectJQuery(page);
+ const result2 = await page.evaluate(() => {
/* global $ */
- return {
- isDefined: window.jQuery === window.$,
- text: $('h1').text(),
- };
- });
- expect(result2).toEqual({
- isDefined: true,
- text: '',
- });
- } finally {
- browser.close();
- }
- });
+ return {
+ isDefined: window.jQuery === window.$,
+ text: $('h1').text(),
+ };
+ });
+ expect(result2).toEqual({
+ isDefined: true,
+ text: '',
+ });
+ } finally {
+ browser.close();
+ }
+ });
- test('injectUnderscore()', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
+ test('injectUnderscore()', async () => {
+ const browser = await Apify[launchName](launchContext);
- try {
- const page = await browser.newPage();
- await page.goto('about:blank');
+ try {
+ const page = await browser.newPage();
+ await page.goto('about:blank');
- const result1 = await page.evaluate(() => {
- return { isDefined: window._ !== undefined };
- });
- expect(result1).toEqual({ isDefined: false });
+ const result1 = await page.evaluate(() => {
+ return { isDefined: window._ !== undefined };
+ });
+ expect(result1).toEqual({ isDefined: false });
- await Apify.utils.puppeteer.injectUnderscore(page);
- const result2 = await page.evaluate(() => {
+ await Apify.utils.puppeteer.injectUnderscore(page);
+ const result2 = await page.evaluate(() => {
/* global _ */
- return { isDefined: _.isEmpty({}) };
- });
- expect(result2).toEqual({ isDefined: true });
- } finally {
- browser.close();
- }
- });
-
- // TODO Remove with 1.0.0. This is here as a backwards comp for moving
- // the function from utils.puppeteer to utils.
- test('enqueueLinks() exists in this namespace', async () => {
- expect(Apify.utils.puppeteer.enqueueLinks).toBeInstanceOf(Function);
- try {
- await Apify.utils.puppeteer.enqueueLinks({ page: {}, $: () => {} });
- } catch (err) {
- expect(err.message).toBe('Only one of the parameters "options.page" or "options.$" must be provided!');
- }
- });
-
- describe('blockRequests()', () => {
- let browser = null;
- beforeAll(async () => {
- browser = await Apify.launchPuppeteer({ headless: true });
+ return { isDefined: _.isEmpty({}) };
+ });
+ expect(result2).toEqual({ isDefined: true });
+ } finally {
+ browser.close();
+ }
});
- afterAll(async () => {
- await browser.close();
+
+ // TODO Remove with 1.0.0. This is here as a backwards comp for moving
+ // the function from utils.puppeteer to utils.
+ test('enqueueLinks() exists in this namespace', async () => {
+ expect(Apify.utils.puppeteer.enqueueLinks).toBeInstanceOf(Function);
+ try {
+ await Apify.utils.puppeteer.enqueueLinks({ page: {}, $: () => {} });
+ } catch (err) {
+ expect(err.message).toBe('Only one of the parameters "options.page" or "options.$" must be provided!');
+ }
});
- test('works with default values', async () => {
- const loadedUrls = [];
+ describe('blockRequests()', () => {
+ let browser = null;
+ beforeAll(async () => {
+ browser = await Apify[launchName](launchContext);
+ });
+ afterAll(async () => {
+ await browser.close();
+ });
+
+ test('works with default values', async () => {
+ const loadedUrls = [];
- const page = await browser.newPage();
- await Apify.utils.puppeteer.blockRequests(page);
- page.on('response', (response) => loadedUrls.push(response.url()));
- await page.setContent(`<html><body>
+ const page = await browser.newPage();
+ await Apify.utils.puppeteer.blockRequests(page);
+ page.on('response', (response) => loadedUrls.push(response.url()));
+ await page.setContent(`<html><body>
<link rel="stylesheet" type="text/css" href="https://example.com/style.css">
<img src="https://example.com/image.png">
<img src="https://example.com/image.gif">
<script src="https://example.com/script.js" defer="defer">></script>
</body></html>`, { waitUntil: 'load' });
- expect(loadedUrls).toEqual(['https://example.com/script.js']);
- });
+ expect(loadedUrls).toEqual(['https://example.com/script.js']);
+ });
- test('works with overridden values', async () => {
- const loadedUrls = [];
+ test('works with overridden values', async () => {
+ const loadedUrls = [];
- const page = await browser.newPage();
- await Apify.utils.puppeteer.blockRequests(page, {
- urlPatterns: ['.css'],
- });
- page.on('response', (response) => loadedUrls.push(response.url()));
- await page.setContent(`<html><body>
+ const page = await browser.newPage();
+ await Apify.utils.puppeteer.blockRequests(page, {
+ urlPatterns: ['.css'],
+ });
+ page.on('response', (response) => loadedUrls.push(response.url()));
+ await page.setContent(`<html><body>
<link rel="stylesheet" type="text/css" href="https://example.com/style.css">
<img src="https://example.com/image.png">
<img src="https://example.com/image.gif">
<script src="https://example.com/script.js" defer="defer">></script>
</body></html>`, { waitUntil: 'load' });
- expect(loadedUrls).toEqual(expect.arrayContaining([
- 'https://example.com/image.png',
- 'https://example.com/script.js',
- 'https://example.com/image.gif',
- ]));
- });
+ expect(loadedUrls).toEqual(expect.arrayContaining([
+ 'https://example.com/image.png',
+ 'https://example.com/script.js',
+ 'https://example.com/image.gif',
+ ]));
+ });
- test('blockResources() supports default values', async () => {
- const loadedUrls = [];
+ test('blockResources() supports default values', async () => {
+ const loadedUrls = [];
- const page = await browser.newPage();
- await Apify.utils.puppeteer.blockResources(page);
- page.on('response', (response) => loadedUrls.push(response.url()));
- await page.setContent(`<html><body>
+ const page = await browser.newPage();
+ await Apify.utils.puppeteer.blockResources(page);
+ page.on('response', (response) => loadedUrls.push(response.url()));
+ await page.setContent(`<html><body>
<link rel="stylesheet" type="text/css" href="https://example.com/style.css">
<img src="https://example.com/image.png" />
<script src="https://example.com/script.js" defer="defer">></script>
</body></html>`, { waitUntil: 'load' });
- expect(loadedUrls).toEqual(expect.arrayContaining([
- 'https://example.com/script.js',
- ]));
- });
+ expect(loadedUrls).toEqual(expect.arrayContaining([
+ 'https://example.com/script.js',
+ ]));
+ });
- test('blockResources() supports nondefault values', async () => {
- const loadedUrls = [];
+ test('blockResources() supports nondefault values', async () => {
+ const loadedUrls = [];
- const page = await browser.newPage();
- await Apify.utils.puppeteer.blockResources(page, ['script']);
- page.on('response', (response) => loadedUrls.push(response.url()));
- await page.setContent(`<html><body>
+ const page = await browser.newPage();
+ await Apify.utils.puppeteer.blockResources(page, ['script']);
+ page.on('response', (response) => loadedUrls.push(response.url()));
+ await page.setContent(`<html><body>
<link rel="stylesheet" type="text/css" href="https://example.com/style.css">
<img src="https://example.com/image.png" />
<script src="https://example.com/script.js" defer="defer">></script>
</body></html>`, { waitUntil: 'load' });
- expect(loadedUrls).toEqual(expect.arrayContaining([
- 'https://example.com/style.css',
- 'https://example.com/image.png',
- ]));
+ expect(loadedUrls).toEqual(expect.arrayContaining([
+ 'https://example.com/style.css',
+ 'https://example.com/image.png',
+ ]));
+ });
});
- });
- test('supports cacheResponses()', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
- const cache = {};
-
- const getResourcesLoadedFromWiki = async () => {
- let downloadedBytes = 0;
- const page = await browser.newPage();
- // Cache all javascript files, png files and svg files
- await Apify.utils.puppeteer.cacheResponses(page, cache, ['.js', /.+\.png/i, /.+\.svg/i]);
- page.on('response', async (response) => {
- if (cache[response.url()]) return;
- try {
- const buffer = await response.buffer();
- downloadedBytes += buffer.byteLength;
- } catch (e) {
+ test('supports cacheResponses()', async () => {
+ const browser = await Apify[launchName](launchContext);
+ const cache = {};
+
+ const getResourcesLoadedFromWiki = async () => {
+ let downloadedBytes = 0;
+ const page = await browser.newPage();
+ // Cache all javascript files, png files and svg files
+ await Apify.utils.puppeteer.cacheResponses(page, cache, ['.js', /.+\.png/i, /.+\.svg/i]);
+ page.on('response', async (response) => {
+ if (cache[response.url()]) return;
+ try {
+ const buffer = await response.buffer();
+ downloadedBytes += buffer.byteLength;
+ } catch (e) {
// do nothing
- }
- });
- await page.goto('https://www.wikipedia.org/', { waitUntil: 'networkidle0' });
- await page.close();
- return downloadedBytes;
- };
-
- try {
- const bytesDownloadedOnFirstRun = await getResourcesLoadedFromWiki();
- const bytesDownloadedOnSecondRun = await getResourcesLoadedFromWiki();
- expect(bytesDownloadedOnSecondRun).toBeLessThan(bytesDownloadedOnFirstRun);
- } finally {
- await browser.close();
- }
- });
-
- test(
- 'cacheResponses() throws when rule with invalid type is provided',
- async () => {
- const mockedPage = {
- setRequestInterception: () => {},
- on: () => {},
+ }
+ });
+ await page.goto('https://www.wikipedia.org/', { waitUntil: 'networkidle0' });
+ await page.close();
+ return downloadedBytes;
};
- const testRuleType = async (value) => {
- try {
- await Apify.utils.puppeteer.cacheResponses(mockedPage, {}, [value]);
- } catch (error) {
- // this is valid path for this test
- return;
- }
-
- expect(`Rule '${value}' should have thrown error`).toBe('');
- };
- await testRuleType(0);
- await testRuleType(1);
- await testRuleType(null);
- await testRuleType([]);
- await testRuleType(['']);
- await testRuleType(() => {});
- },
- );
-
- test('compileScript() works', async () => {
- const { compileScript } = Apify.utils.puppeteer;
- const scriptStringGood = 'await page.goto("about:blank"); return await page.content();';
- const scriptStringBad = 'for const while';
- const script = compileScript(scriptStringGood);
-
- expect(typeof script).toBe('function');
- expect(script.toString()).toEqual(`async ({ page, request }) => {${scriptStringGood}}`);
-
- try {
- compileScript(scriptStringBad);
- throw new Error('Should fail.');
- } catch (err) {
- // TODO figure out why the err.message comes out empty in the logs.
- expect(err.message).toMatch(/Unexpected token '?const'?/);
- }
- const browser = await Apify.launchPuppeteer({ headless: true });
- try {
- const page = await browser.newPage();
- const content = await script({ page });
- expect(typeof content).toBe('string');
- expect(content).toBe('<html><head></head><body></body></html>');
- } finally {
- browser.close();
- }
- });
-
- test('gotoExtended() works', async () => {
- const browser = await Apify.launchPuppeteer({ headless: true });
-
- try {
- const page = await browser.newPage();
- const request = new Apify.Request({
- url: 'https://api.apify.com/v2/browser-info',
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json; charset=utf-8',
- },
- payload: '{ "foo": "bar" }',
- });
-
- const response = await Apify.utils.puppeteer.gotoExtended(page, request);
+ try {
+ const bytesDownloadedOnFirstRun = await getResourcesLoadedFromWiki();
+ const bytesDownloadedOnSecondRun = await getResourcesLoadedFromWiki();
+ expect(bytesDownloadedOnSecondRun).toBeLessThan(bytesDownloadedOnFirstRun);
+ } finally {
+ await browser.close();
+ }
+ });
- const { method, headers, bodyLength } = JSON.parse(await response.text());
- expect(method).toBe('POST');
- expect(bodyLength).toBe(16);
- expect(headers['content-type']).toBe('application/json; charset=utf-8');
- } finally {
- await browser.close();
- }
- });
+ test(
+ 'cacheResponses() throws when rule with invalid type is provided',
+ async () => {
+ const mockedPage = {
+ setRequestInterception: () => {},
+ on: () => {},
+ };
- test('infiniteScroll() works', async () => {
- function isAtBottom() {
- return (window.innerHeight + window.pageYOffset) >= document.body.offsetHeight;
- }
- const browser = await Apify.launchPuppeteer({ headless: true });
- try {
- const page = await browser.newPage();
- let count = 0;
- const content = Array(1000).fill(null).map(() => {
- return `<div style="border: 1px solid black">Div number: ${count++}</div>`;
- });
- const contentHTML = `<html><body>${content}</body></html>`;
- await page.setContent(contentHTML);
+ const testRuleType = async (value) => {
+ try {
+ await Apify.utils.puppeteer.cacheResponses(mockedPage, {}, [value]);
+ } catch (error) {
+ // this is valid path for this test
+ return;
+ }
- const before = await page.evaluate(isAtBottom);
- expect(before).toBe(false);
+ expect(`Rule '${value}' should have thrown error`).toBe('');
+ };
+ await testRuleType(0);
+ await testRuleType(1);
+ await testRuleType(null);
+ await testRuleType([]);
+ await testRuleType(['']);
+ await testRuleType(() => {});
+ },
+ );
+
+ test('compileScript() works', async () => {
+ const { compileScript } = Apify.utils.puppeteer;
+ const scriptStringGood = 'await page.goto("about:blank"); return await page.content();';
+ const scriptStringBad = 'for const while';
+ const script = compileScript(scriptStringGood);
+
+ expect(typeof script).toBe('function');
+ expect(script.toString()).toEqual(`async ({ page, request }) => {${scriptStringGood}}`);
+
+ try {
+ compileScript(scriptStringBad);
+ throw new Error('Should fail.');
+ } catch (err) {
+ // TODO figure out why the err.message comes out empty in the logs.
+ expect(err.message).toMatch(/Unexpected token '?const'?/);
+ }
+ const browser = await Apify[launchName](launchContext);
+ try {
+ const page = await browser.newPage();
+ const content = await script({ page });
+ expect(typeof content).toBe('string');
+ expect(content).toBe('<html><head></head><body></body></html>');
+ } finally {
+ browser.close();
+ }
+ });
- await Apify.utils.puppeteer.infiniteScroll(page, { waitForSecs: 0 });
+ test('gotoExtended() works', async () => {
+ const browser = await Apify[launchName](launchContext);
+
+ try {
+ const page = await browser.newPage();
+ const request = new Apify.Request({
+ url: 'https://api.apify.com/v2/browser-info',
+ method: 'POST',
+ headers: {
+ 'Content-Type': 'application/json; charset=utf-8',
+ },
+ payload: '{ "foo": "bar" }',
+ });
+
+ const response = await Apify.utils.puppeteer.gotoExtended(page, request);
+
+ const { method, headers, bodyLength } = JSON.parse(await response.text());
+ expect(method).toBe('POST');
+ expect(bodyLength).toBe(16);
+ expect(headers['content-type']).toBe('application/json; charset=utf-8');
+ } finally {
+ await browser.close();
+ }
+ });
- const after = await page.evaluate(isAtBottom);
- expect(after).toBe(true);
- } finally {
- await browser.close();
- }
- });
+ test('infiniteScroll() works', async () => {
+ function isAtBottom() {
+ return (window.innerHeight + window.pageYOffset) >= document.body.offsetHeight;
+ }
+ const browser = await Apify.launchPuppeteer({ launchOptions: { headless: true } });
+ try {
+ const page = await browser.newPage();
+ let count = 0;
+ const content = Array(1000).fill(null).map(() => {
+ return `<div style="border: 1px solid black">Div number: ${count++}</div>`;
+ });
+ const contentHTML = `<html><body>${content}</body></html>`;
+ await page.setContent(contentHTML);
+
+ const before = await page.evaluate(isAtBottom);
+ expect(before).toBe(false);
+
+ await Apify.utils.puppeteer.infiniteScroll(page, { waitForSecs: 0 });
+
+ const after = await page.evaluate(isAtBottom);
+ expect(after).toBe(true);
+ } finally {
+ await browser.close();
+ }
+ });
- it('saveSnapshot() works', async () => {
- const mock = sinon.mock(keyValueStore);
- const browser = await Apify.launchPuppeteer({ headless: true });
- try {
- const page = await browser.newPage();
- const contentHTML = '<html><head></head><body><div style="border: 1px solid black">Div number: 1</div></body></html>';
- await page.setContent(contentHTML);
+ it('saveSnapshot() works', async () => {
+ const mock = sinon.mock(keyValueStore);
+ const browser = await Apify[launchName](launchContext);
+ try {
+ const page = await browser.newPage();
+ const contentHTML = '<html><head></head><body><div style="border: 1px solid black">Div number: 1</div></body></html>';
+ await page.setContent(contentHTML);
- const screenshot = await page.screenshot({ fullPage: true, type: 'jpeg', screenshotQuality: 60 });
+ const screenshot = await page.screenshot({ fullPage: true, type: 'jpeg', screenshotQuality: 60 });
- // Test saving both image and html
- const object = { setValue: async () => {} };
- const stub = sinon.stub(object, 'setValue');
+ // Test saving both image and html
+ const object = { setValue: async () => {} };
+ const stub = sinon.stub(object, 'setValue');
- mock.expects('openKeyValueStore')
- .once()
- .withExactArgs('TEST-STORE')
- .resolves(object);
+ mock.expects('openKeyValueStore')
+ .once()
+ .withExactArgs('TEST-STORE')
+ .resolves(object);
- await Apify.utils.puppeteer.saveSnapshot(page, { key: 'TEST', keyValueStoreName: 'TEST-STORE', screenshotQuality: 60 });
+ await Apify.utils.puppeteer.saveSnapshot(page, { key: 'TEST', keyValueStoreName: 'TEST-STORE', screenshotQuality: 60 });
- expect(stub.calledWithExactly('TEST.jpg', screenshot, { contentType: 'image/jpeg' })).toBe(true);
- expect(stub.calledWithExactly('TEST.html', contentHTML, { contentType: 'text/html' })).toBe(true);
+ expect(stub.calledWithExactly('TEST.jpg', screenshot, { contentType: 'image/jpeg' })).toBe(true);
+ expect(stub.calledWithExactly('TEST.html', contentHTML, { contentType: 'text/html' })).toBe(true);
- // Test saving only image
- const object2 = { setValue: async () => {} };
- const stub2 = sinon.stub(object2, 'setValue');
- mock.expects('openKeyValueStore')
- .withExactArgs(undefined)
- .resolves(object2);
+ // Test saving only image
+ const object2 = { setValue: async () => {} };
+ const stub2 = sinon.stub(object2, 'setValue');
+ mock.expects('openKeyValueStore')
+ .withExactArgs(undefined)
+ .resolves(object2);
- await Apify.utils.puppeteer.saveSnapshot(page, { saveHtml: false });
+ await Apify.utils.puppeteer.saveSnapshot(page, { saveHtml: false });
- // Default quality is 50
- const screenshot2 = await page.screenshot({ fullPage: true, type: 'jpeg', screenshotQuality: 50 });
- expect(stub2.calledOnceWithExactly('SNAPSHOT.jpg', screenshot2, { contentType: 'image/jpeg' })).toBe(true);
+ // Default quality is 50
+ const screenshot2 = await page.screenshot({ fullPage: true, type: 'jpeg', screenshotQuality: 50 });
+ expect(stub2.calledOnceWithExactly('SNAPSHOT.jpg', screenshot2, { contentType: 'image/jpeg' })).toBe(true);
- mock.verify();
- } finally {
- await browser.close();
- }
+ mock.verify();
+ } finally {
+ await browser.close();
+ }
+ });
});
});
diff --git a/test/stealth/stealth.test.js b/test/stealth/stealth.test.js
index 41b89c30a..06777eb15 100644
--- a/test/stealth/stealth.test.js
+++ b/test/stealth/stealth.test.js
@@ -30,8 +30,10 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
utils.apifyStorageLocal = utils.newStorageLocal({ storageDir });
browser = await Apify.launchPuppeteer({
stealth: true,
- headless: true,
useChrome: true,
+ launchOptions: {
+ headless: true,
+ },
});
page = await browser.newPage();
});
@@ -195,10 +197,12 @@ describe('Stealth - testing headless chrome hiding tricks', () => {
const requestList = await Apify.openRequestList(null, [testUrl]);
const puppeteerCrawler = new Apify.PuppeteerCrawler({
requestList,
- launchPuppeteerOptions: {
+ launchContext: {
stealth: true,
useChrome: true,
- headless: true,
+ launchOptions: {
+ headless: true,
+ },
},
handlePageFunction: async ({ page: activePage }) => {
const fingerprint = await getFingerPrint(activePage);
diff --git a/test/typescript/crawlers/cheerio_crawler.test.ts b/test/typescript/crawlers/cheerio_crawler.test.ts
index f52ab463f..6f431d25a 100644
--- a/test/typescript/crawlers/cheerio_crawler.test.ts
+++ b/test/typescript/crawlers/cheerio_crawler.test.ts
@@ -24,7 +24,7 @@ describe('CheerioCrawler TS', () => {
request: new Apify.Request({url: ' http://www.test1234.com'}),
contentType: {type: 'text/html', encoding: 'utf-8'},
response: null as any,
- autoscaledPool: null as any
+ crawler: null as any
};
});
diff --git a/website/i18n/en.json b/website/i18n/en.json
index 066d1bfcd..26ad38741 100644
--- a/website/i18n/en.json
+++ b/website/i18n/en.json
@@ -17,6 +17,9 @@
"api/basic-crawler": {
"title": "BasicCrawler"
},
+ "api/browser-crawler": {
+ "title": "BrowserCrawler"
+ },
"api/cache-container": {
"title": "CacheContainer"
},
@@ -32,18 +35,12 @@
"api/exports": {
"title": "exports"
},
- "api/global-cache": {
- "title": "GlobalCache"
- },
"api/job": {
"title": "Job"
},
"api/key-value-store": {
"title": "KeyValueStore"
},
- "api/live-view-server": {
- "title": "LiveViewServer"
- },
"api/log": {
"title": "utils.log"
},
@@ -59,9 +56,6 @@
"api/puppeteer-crawler": {
"title": "PuppeteerCrawler"
},
- "api/puppeteer-pool": {
- "title": "PuppeteerPool"
- },
"api/request": {
"title": "Request"
},
@@ -209,6 +203,9 @@
"typedefs/basic-crawler-options": {
"title": "BasicCrawlerOptions"
},
+ "typedefs/browser-crawler-options": {
+ "title": "BrowserCrawlerOptions"
+ },
"typedefs/cheerio-crawler-options": {
"title": "CheerioCrawlerOptions"
},
@@ -251,12 +248,6 @@
"typedefs/key-consumer": {
"title": "KeyConsumer"
},
- "typedefs/launch-puppeteer": {
- "title": "LaunchPuppeteer"
- },
- "typedefs/launch-puppeteer-function": {
- "title": "LaunchPuppeteerFunction"
- },
"typedefs/launch-puppeteer-options": {
"title": "LaunchPuppeteerOptions"
},
@@ -287,21 +278,6 @@
"typedefs/puppeteer-crawler-options": {
"title": "PuppeteerCrawlerOptions"
},
- "typedefs/puppeteer-goto": {
- "title": "PuppeteerGoto"
- },
- "typedefs/puppeteer-goto-inputs": {
- "title": "PuppeteerGotoInputs"
- },
- "typedefs/puppeteer-handle-page": {
- "title": "PuppeteerHandlePage"
- },
- "typedefs/puppeteer-handle-page-inputs": {
- "title": "PuppeteerHandlePageInputs"
- },
- "typedefs/puppeteer-pool-options": {
- "title": "PuppeteerPoolOptions"
- },
"typedefs/queue-operation-info": {
"title": "QueueOperationInfo"
},