Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
feat: support proxy in connect/connectOverCDP (#35389)
Dmitry Gozman471a28e0d51a
diff --git a/docs/src/api/class-browsertype.md b/docs/src/api/class-browsertype.mdindex f970d4aef3..1480d567c5 100644--- a/docs/src/api/class-browsertype.md+++ b/docs/src/api/class-browsertype.md@@ -144,6 +144,16 @@ Some common examples:1. `"<loopback>"` to expose localhost network.1. `"*.test.internal-domain,*.staging.internal-domain,<loopback>"` to expose test/staging deployments and localhost.+### option: BrowserType.connect.proxy+* since: v1.52+- `proxy` <[Object]>+ - `server` <[string]> Proxy to be used for the remote connection. HTTP and SOCKS proxies are supported, for example `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP proxy.+ - `bypass` ?<[string]> Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.+ - `username` ?<[string]> Optional username to use if HTTP proxy requires authentication.+ - `password` ?<[string]> Optional password to use if HTTP proxy requires authentication.++Proxy settings to use for the connection between the client and the remote browser. Note this proxy **is not** used by the browser to load web pages.+## async method: BrowserType.connectOverCDP* since: v1.9- returns: <[Browser]>@@ -232,6 +242,16 @@ Logger sink for Playwright logging. Optional.Maximum time in milliseconds to wait for the connection to be established. Defaults to`30000` (30 seconds). Pass `0` to disable timeout.+### option: BrowserType.connectOverCDP.proxy+* since: v1.52+- `proxy` <[Object]>+ - `server` <[string]> Proxy to be used for the remote connection. HTTP and SOCKS proxies are supported, for example `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP proxy.+ - `bypass` ?<[string]> Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.+ - `username` ?<[string]> Optional username to use if HTTP proxy requires authentication.+ - `password` ?<[string]> Optional password to use if HTTP proxy requires authentication.++Proxy settings to use for the connection between the client and the remote browser. Note this proxy **is not** used by the browser to load web pages.+## method: BrowserType.executablePath* since: v1.8- returns: <[string]>diff --git a/docs/src/api/params.md b/docs/src/api/params.mdindex 4c49370044..aeb8f7fde3 100644--- a/docs/src/api/params.md+++ b/docs/src/api/params.md@@ -226,10 +226,8 @@ Dangerous option; use with care. Defaults to `false`.## browser-option-proxy- `proxy` <[Object]>- `server` <[string]> Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example- `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP- proxy.- - `bypass` ?<[string]> Optional comma-separated domains to bypass proxy, for example `".com, chromium.org,- .domain.com"`.+ `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP proxy.+ - `bypass` ?<[string]> Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.- `username` ?<[string]> Optional username to use if HTTP proxy requires authentication.- `password` ?<[string]> Optional password to use if HTTP proxy requires authentication.diff --git a/packages/playwright-client/types/types.d.ts b/packages/playwright-client/types/types.d.tsindex aff26e8942..8ad9c6e348 100644--- a/packages/playwright-client/types/types.d.ts+++ b/packages/playwright-client/types/types.d.ts@@ -21793,6 +21793,34 @@ export interface ConnectOverCDPOptions {*/logger?: Logger;+ /**+ * Proxy settings to use for the connection between the client and the remote browser. Note this proxy **is not** used+ * by the browser to load web pages.+ */+ proxy?: {+ /**+ * Proxy to be used for the remote connection. HTTP and SOCKS proxies are supported, for example+ * `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP+ * proxy.+ */+ server: string;++ /**+ * Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.+ */+ bypass?: string;++ /**+ * Optional username to use if HTTP proxy requires authentication.+ */+ username?: string;++ /**+ * Optional password to use if HTTP proxy requires authentication.+ */+ password?: string;+ };+/*** Slows down Playwright operations by the specified amount of milliseconds. Useful so that you can see what is going* on. Defaults to 0.@@ -21834,6 +21862,34 @@ export interface ConnectOptions {*/logger?: Logger;+ /**+ * Proxy settings to use for the connection between the client and the remote browser. Note this proxy **is not** used+ * by the browser to load web pages.+ */+ proxy?: {+ /**+ * Proxy to be used for the remote connection. HTTP and SOCKS proxies are supported, for example+ * `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP+ * proxy.+ */+ server: string;++ /**+ * Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.+ */+ bypass?: string;++ /**+ * Optional username to use if HTTP proxy requires authentication.+ */+ username?: string;++ /**+ * Optional password to use if HTTP proxy requires authentication.+ */+ password?: string;+ };+/*** Slows down Playwright operations by the specified amount of milliseconds. Useful so that you can see what is going* on. Defaults to 0.diff --git a/packages/playwright-core/src/client/browserType.ts b/packages/playwright-core/src/client/browserType.tsindex f67c97ced8..47e5722335 100644--- a/packages/playwright-core/src/client/browserType.ts+++ b/packages/playwright-core/src/client/browserType.ts@@ -129,6 +129,7 @@ export class BrowserType extends ChannelOwner<channels.BrowserTypeChannel> impleexposeNetwork: params.exposeNetwork ?? params._exposeNetwork,slowMo: params.slowMo,timeout: params.timeout,+ proxy: params.proxy,};if ((params as any).__testHookRedirectPortForwarding)connectParams.socksProxyRedirectPortForTest = (params as any).__testHookRedirectPortForwarding;@@ -188,7 +189,8 @@ export class BrowserType extends ChannelOwner<channels.BrowserTypeChannel> impleendpointURL,headers,slowMo: params.slowMo,- timeout: params.timeout+ timeout: params.timeout,+ proxy: params.proxy,});const browser = Browser.from(result.browser);this._didLaunchBrowser(browser, {}, params.logger);diff --git a/packages/playwright-core/src/client/types.ts b/packages/playwright-core/src/client/types.tsindex 5235fef8b4..0a7d93c8a4 100644--- a/packages/playwright-core/src/client/types.ts+++ b/packages/playwright-core/src/client/types.ts@@ -103,6 +103,12 @@ export type ConnectOptions = {slowMo?: number,timeout?: number,logger?: Logger,+ proxy?: {+ server: string,+ bypass?: string,+ username?: string,+ password?: string+ },};export type LaunchServerOptions = {channel?: channels.BrowserTypeLaunchOptions['channel'],diff --git a/packages/playwright-core/src/protocol/validator.ts b/packages/playwright-core/src/protocol/validator.tsindex 0484fd1c11..96b31310b6 100644--- a/packages/playwright-core/src/protocol/validator.ts+++ b/packages/playwright-core/src/protocol/validator.ts@@ -327,6 +327,12 @@ scheme.LocalUtilsConnectParams = tObject({exposeNetwork: tOptional(tString),slowMo: tOptional(tNumber),timeout: tOptional(tNumber),+ proxy: tOptional(tObject({+ server: tString,+ bypass: tOptional(tString),+ username: tOptional(tString),+ password: tOptional(tString),+ })),socksProxyRedirectPortForTest: tOptional(tNumber),});scheme.LocalUtilsConnectResult = tObject({@@ -650,6 +656,12 @@ scheme.BrowserTypeConnectOverCDPParams = tObject({headers: tOptional(tArray(tType('NameValue'))),slowMo: tOptional(tNumber),timeout: tOptional(tNumber),+ proxy: tOptional(tObject({+ server: tString,+ bypass: tOptional(tString),+ username: tOptional(tString),+ password: tOptional(tString),+ })),});scheme.BrowserTypeConnectOverCDPResult = tObject({browser: tChannel(['Browser']),diff --git a/packages/playwright-core/src/server/browserType.ts b/packages/playwright-core/src/server/browserType.tsindex c4c44d83de..f0f3c53a58 100644--- a/packages/playwright-core/src/server/browserType.ts+++ b/packages/playwright-core/src/server/browserType.ts@@ -292,7 +292,7 @@ export abstract class BrowserType extends SdkObject {await fs.promises.mkdir(options.tracesDir, { recursive: true });}- async connectOverCDP(metadata: CallMetadata, endpointURL: string, options: { slowMo?: number }, timeout?: number): Promise<Browser> {+ async connectOverCDP(metadata: CallMetadata, endpointURL: string, options: { slowMo?: number, proxy?: types.ProxySettings, timeout?: number, headers?: types.HeadersArray }): Promise<Browser> {throw new Error('CDP connections are only supported by Chromium');}diff --git a/packages/playwright-core/src/server/chromium/chromium.ts b/packages/playwright-core/src/server/chromium/chromium.tsindex 35acc621e2..0f478c9d50 100644--- a/packages/playwright-core/src/server/chromium/chromium.ts+++ b/packages/playwright-core/src/server/chromium/chromium.ts@@ -64,15 +64,15 @@ export class Chromium extends BrowserType {this._devtools = this._createDevTools();}- override async connectOverCDP(metadata: CallMetadata, endpointURL: string, options: { slowMo?: number, headers?: types.HeadersArray }, timeout?: number) {+ override async connectOverCDP(metadata: CallMetadata, endpointURL: string, options: { slowMo?: number, proxy?: types.ProxySettings, timeout?: number, headers?: types.HeadersArray }) {const controller = new ProgressController(metadata, this);controller.setLogName('browser');return controller.run(async progress => {return await this._connectOverCDPInternal(progress, endpointURL, options);- }, TimeoutSettings.timeout({ timeout }));+ }, TimeoutSettings.timeout(options));}- async _connectOverCDPInternal(progress: Progress, endpointURL: string, options: types.LaunchOptions & { headers?: types.HeadersArray }, onClose?: () => Promise<void>) {+ async _connectOverCDPInternal(progress: Progress, endpointURL: string, options: types.LaunchOptions & { headers?: types.HeadersArray, proxy?: types.ProxySettings }, onClose?: () => Promise<void>) {let headersMap: { [key: string]: string; } | undefined;if (options.headers)headersMap = headersArrayToObject(options.headers, false);@@ -84,10 +84,10 @@ export class Chromium extends BrowserType {const artifactsDir = await fs.promises.mkdtemp(ARTIFACTS_FOLDER);- const wsEndpoint = await urlToWSEndpoint(progress, endpointURL, headersMap);+ const wsEndpoint = await urlToWSEndpoint(progress, endpointURL, headersMap, options.proxy);progress.throwIfAborted();- const chromeTransport = await WebSocketTransport.connect(progress, wsEndpoint, headersMap);+ const chromeTransport = await WebSocketTransport.connect(progress, wsEndpoint, { headers: headersMap, proxy: options.proxy });const cleanedUp = new ManualPromise<void>();const doCleanup = async () => {await removeFolders([artifactsDir]);@@ -365,18 +365,35 @@ class ChromiumReadyState extends BrowserReadyState {}}-async function urlToWSEndpoint(progress: Progress, endpointURL: string, headers: { [key: string]: string; }) {+async function urlToWSEndpoint(progress: Progress, endpointURL: string, headers: { [key: string]: string; }, proxy?: types.ProxySettings) {if (endpointURL.startsWith('ws'))return endpointURL;progress.log(`<ws preparing> retrieving websocket url from ${endpointURL}`);const httpURL = endpointURL.endsWith('/') ? `${endpointURL}json/version/` : `${endpointURL}/json/version/`;+ // Chromium insists on localhost "Host" header for security reasons, and in the case of proxy+ // we end up with the remote host instead of localhost.+ const extraHeaders = proxy ? { Host: `localhost:9222` } : {};const json = await fetchData({url: httpURL,- headers,+ headers: { ...headers, ...extraHeaders },+ proxy,}, async (_, resp) => new Error(`Unexpected status ${resp.statusCode} when connecting to ${httpURL}.\n` +`This does not look like a DevTools server, try connecting via ws://.`));- return JSON.parse(json).webSocketDebuggerUrl;+ const wsUrl = JSON.parse(json).webSocketDebuggerUrl;+ if (proxy) {+ // webSocketDebuggerUrl will be a localhost URL, accessible from the browser's computer.+ // When using a proxy, assume we need to connect through the original endpointURL.+ // Example:+ // connecting to http://example.com/+ // making request to http://example.com/json/version/+ // webSocketDebuggerUrl ends up as ws://localhost:9222/devtools/page/<guid>+ // we construct ws://example.com/devtools/page/<guid> by appending the pathname to the original URL+ const url = new URL(endpointURL);+ url.pathname += (url.pathname.endsWith('/') ? '' : '/') + new URL(wsUrl).pathname.substring(1);+ return url.toString();+ }+ return wsUrl;}async function seleniumErrorHandler(params: HTTPRequestParams, response: http.IncomingMessage) {diff --git a/packages/playwright-core/src/server/dispatchers/browserTypeDispatcher.ts b/packages/playwright-core/src/server/dispatchers/browserTypeDispatcher.tsindex bf21e726bc..40bf3eb9d8 100644--- a/packages/playwright-core/src/server/dispatchers/browserTypeDispatcher.ts+++ b/packages/playwright-core/src/server/dispatchers/browserTypeDispatcher.ts@@ -43,7 +43,7 @@ export class BrowserTypeDispatcher extends Dispatcher<BrowserType, channels.Brow}async connectOverCDP(params: channels.BrowserTypeConnectOverCDPParams, metadata: CallMetadata): Promise<channels.BrowserTypeConnectOverCDPResult> {- const browser = await this._object.connectOverCDP(metadata, params.endpointURL, params, params.timeout);+ const browser = await this._object.connectOverCDP(metadata, params.endpointURL, params);const browserDispatcher = new BrowserDispatcher(this, browser);return {browser: browserDispatcher,diff --git a/packages/playwright-core/src/server/dispatchers/localUtilsDispatcher.ts b/packages/playwright-core/src/server/dispatchers/localUtilsDispatcher.tsindex a681b349d3..2a66d0a788 100644--- a/packages/playwright-core/src/server/dispatchers/localUtilsDispatcher.ts+++ b/packages/playwright-core/src/server/dispatchers/localUtilsDispatcher.ts@@ -33,6 +33,7 @@ import type { RootDispatcher } from './dispatcher';import type * as channels from '@protocol/channels';import type * as http from 'http';import type { HTTPRequestParams } from '../utils/network';+import type { ProxySettings } from '../types';export class LocalUtilsDispatcher extends Dispatcher<{ guid: string }, channels.LocalUtilsChannel, RootDispatcher> implements channels.LocalUtilsChannel {_type_LocalUtils: boolean;@@ -90,9 +91,9 @@ export class LocalUtilsDispatcher extends Dispatcher<{ guid: string }, channels.'x-playwright-proxy': params.exposeNetwork ?? '',...params.headers,};- const wsEndpoint = await urlToWSEndpoint(progress, params.wsEndpoint);+ const wsEndpoint = await urlToWSEndpoint(progress, params.wsEndpoint, params.proxy);- const transport = await WebSocketTransport.connect(progress, wsEndpoint, wsHeaders, true, 'x-playwright-debug-log');+ const transport = await WebSocketTransport.connect(progress, wsEndpoint, { headers: wsHeaders, followRedirects: true, debugLogHeader: 'x-playwright-debug-log', proxy: params.proxy });const socksInterceptor = new SocksInterceptor(transport, params.exposeNetwork, params.socksProxyRedirectPortForTest);const pipe = new JsonPipeDispatcher(this);transport.onmessage = json => {@@ -128,7 +129,7 @@ export class LocalUtilsDispatcher extends Dispatcher<{ guid: string }, channels.}}-async function urlToWSEndpoint(progress: Progress | undefined, endpointURL: string): Promise<string> {+async function urlToWSEndpoint(progress: Progress | undefined, endpointURL: string, proxy?: ProxySettings): Promise<string> {if (endpointURL.startsWith('ws'))return endpointURL;@@ -142,6 +143,7 @@ async function urlToWSEndpoint(progress: Progress | undefined, endpointURL: strimethod: 'GET',timeout: progress?.timeUntilDeadline() ?? 30_000,headers: { 'User-Agent': getUserAgent() },+ proxy,}, async (params: HTTPRequestParams, response: http.IncomingMessage) => {return new Error(`Unexpected status ${response.statusCode} when connecting to ${fetchUrl.toString()}.\n` +`This does not look like a Playwright server, try connecting via ws://.`);diff --git a/packages/playwright-core/src/server/fetch.ts b/packages/playwright-core/src/server/fetch.tsindex 9760ea89a1..71f9de0833 100644--- a/packages/playwright-core/src/server/fetch.ts+++ b/packages/playwright-core/src/server/fetch.ts@@ -18,14 +18,12 @@ import http from 'http';import https from 'https';import { Transform, pipeline } from 'stream';import { TLSSocket } from 'tls';-import url from 'url';import * as zlib from 'zlib';import { TimeoutSettings } from './timeoutSettings';-import { assert, constructURLBasedOnBaseURL, eventsHelper, monotonicTime } from '../utils';+import { assert, constructURLBasedOnBaseURL, createProxyAgent, eventsHelper, monotonicTime } from '../utils';import { createGuid } from './utils/crypto';import { getUserAgent } from './utils/userAgent';-import { HttpsProxyAgent, SocksProxyAgent } from '../utilsBundle';import { BrowserContext, verifyClientCertificates } from './browserContext';import { CookieStore, domainMatches, parseRawCookie } from './cookieStore';import { MultipartFormData } from './formData';@@ -183,8 +181,8 @@ export abstract class APIRequestContext extends SdkObject {let agent;// We skip 'per-context' in order to not break existing users. 'per-context' was previously used to// workaround an upstream Chromium bug. Can be removed in the future.- if (proxy && proxy.server !== 'per-context' && !shouldBypassProxy(requestUrl, proxy.bypass))- agent = createProxyAgent(proxy);+ if (proxy?.server !== 'per-context')+ agent = createProxyAgent(proxy, requestUrl);let maxRedirects = params.maxRedirects ?? (defaults.maxRedirects ?? 20);maxRedirects = maxRedirects === 0 ? -1 : maxRedirects;@@ -647,13 +645,6 @@ export class GlobalAPIRequestContext extends APIRequestContext {const timeoutSettings = new TimeoutSettings();if (options.timeout !== undefined)timeoutSettings.setDefaultTimeout(options.timeout);- const proxy = options.proxy;- if (proxy?.server) {- let url = proxy?.server.trim();- if (!/^\w+:\/\//.test(url))- url = 'http://' + url;- proxy.server = url;- }if (options.storageState) {this._origins = options.storageState.origins?.map(origin => ({ indexedDB: [], ...origin }));this._cookieStore.addCookies(options.storageState.cookies || []);@@ -668,7 +659,7 @@ export class GlobalAPIRequestContext extends APIRequestContext {maxRedirects: options.maxRedirects,httpCredentials: options.httpCredentials,clientCertificates: options.clientCertificates,- proxy,+ proxy: options.proxy,timeoutSettings,};this._tracing = new Tracing(this, options.tracesDir);@@ -705,20 +696,6 @@ export class GlobalAPIRequestContext extends APIRequestContext {}}-export function createProxyAgent(proxy: types.ProxySettings) {- const proxyOpts = url.parse(proxy.server);- if (proxyOpts.protocol?.startsWith('socks')) {- return new SocksProxyAgent({- host: proxyOpts.hostname,- port: proxyOpts.port || undefined,- });- }- if (proxy.username)- proxyOpts.auth = `${proxy.username}:${proxy.password || ''}`;- // TODO: We should use HttpProxyAgent conditional on proxyOpts.protocol instead of always using CONNECT method.- return new HttpsProxyAgent(proxyOpts);-}-function toHeadersArray(rawHeaders: string[]): types.HeadersArray {const result: types.HeadersArray = [];for (let i = 0; i < rawHeaders.length; i += 2)@@ -791,19 +768,6 @@ function removeHeader(headers: { [name: string]: string }, name: string) {delete headers[name];}-function shouldBypassProxy(url: URL, bypass?: string): boolean {- if (!bypass)- return false;- const domains = bypass.split(',').map(s => {- s = s.trim();- if (!s.startsWith('.'))- s = '.' + s;- return s;- });- const domain = '.' + url.hostname;- return domains.some(d => domain.endsWith(d));-}-function setBasicAuthorizationHeader(headers: { [name: string]: string }, credentials: HTTPCredentials) {const { username, password } = credentials;const encoded = Buffer.from(`${username || ''}:${password || ''}`).toString('base64');diff --git a/packages/playwright-core/src/server/socksClientCertificatesInterceptor.ts b/packages/playwright-core/src/server/socksClientCertificatesInterceptor.tsindex a68cb88abb..2cb5883782 100644--- a/packages/playwright-core/src/server/socksClientCertificatesInterceptor.ts+++ b/packages/playwright-core/src/server/socksClientCertificatesInterceptor.ts@@ -23,7 +23,7 @@ import tls from 'tls';import { SocksProxy } from './utils/socksProxy';import { ManualPromise, escapeHTML, generateSelfSignedCertificate, rewriteErrorMessage } from '../utils';import { verifyClientCertificates } from './browserContext';-import { createProxyAgent } from './fetch';+import { createProxyAgent } from './utils/network';import { debugLogger } from './utils/debugLogger';import { createSocket, createTLSSocket } from './utils/happyEyeballs';@@ -242,7 +242,7 @@ export class ClientCertificatesProxy {ignoreHTTPSErrors: boolean | undefined;secureContextMap: Map<string, tls.SecureContext> = new Map();alpnCache: ALPNCache;- proxyAgentFromOptions: ReturnType<typeof createProxyAgent> | undefined;+ proxyAgentFromOptions: ReturnType<typeof createProxyAgent>;constructor(contextOptions: Pick<types.BrowserContextOptions, 'clientCertificates' | 'ignoreHTTPSErrors' | 'proxy'>@@ -250,7 +250,7 @@ export class ClientCertificatesProxy {verifyClientCertificates(contextOptions.clientCertificates);this.alpnCache = new ALPNCache();this.ignoreHTTPSErrors = contextOptions.ignoreHTTPSErrors;- this.proxyAgentFromOptions = contextOptions.proxy ? createProxyAgent(contextOptions.proxy) : undefined;+ this.proxyAgentFromOptions = createProxyAgent(contextOptions.proxy);this._initSecureContexts(contextOptions.clientCertificates);this._socksProxy = new SocksProxy();this._socksProxy.setPattern('*');diff --git a/packages/playwright-core/src/server/transport.ts b/packages/playwright-core/src/server/transport.tsindex ec2caec0c5..a4e092ef6e 100644--- a/packages/playwright-core/src/server/transport.ts+++ b/packages/playwright-core/src/server/transport.ts@@ -15,13 +15,13 @@* limitations under the License.*/-import { makeWaitForNextTask } from '../utils';+import { createProxyAgent, makeWaitForNextTask } from '../utils';import { httpHappyEyeballsAgent, httpsHappyEyeballsAgent } from './utils/happyEyeballs';import { ws } from '../utilsBundle';import type { WebSocket } from '../utilsBundle';import type { Progress } from './progress';-import type { HeadersArray } from './types';+import type { HeadersArray, ProxySettings } from './types';import type { ClientRequest, IncomingMessage } from 'http';export const perMessageDeflate = {@@ -60,6 +60,13 @@ export interface ConnectionTransport {onclose?: (reason?: string) => void,}+type WebSocketTransportOptions = {+ headers?: { [key: string]: string; };+ followRedirects?: boolean;+ debugLogHeader?: string;+ proxy?: ProxySettings;+};+export class WebSocketTransport implements ConnectionTransport {private _ws: WebSocket;private _progress?: Progress;@@ -70,14 +77,14 @@ export class WebSocketTransport implements ConnectionTransport {readonly wsEndpoint: string;readonly headers: HeadersArray = [];- static async connect(progress: (Progress|undefined), url: string, headers?: { [key: string]: string; }, followRedirects?: boolean, debugLogHeader?: string): Promise<WebSocketTransport> {- return await WebSocketTransport._connect(progress, url, headers || {}, { follow: !!followRedirects, hadRedirects: false }, debugLogHeader);+ static async connect(progress: (Progress|undefined), url: string, options: WebSocketTransportOptions = {}): Promise<WebSocketTransport> {+ return await WebSocketTransport._connect(progress, url, options, false /* hadRedirects */);}- static async _connect(progress: (Progress|undefined), url: string, headers: { [key: string]: string; }, redirect: { follow: boolean, hadRedirects: boolean }, debugLogHeader?: string): Promise<WebSocketTransport> {+ static async _connect(progress: (Progress|undefined), url: string, options: WebSocketTransportOptions, hadRedirects: boolean): Promise<WebSocketTransport> {const logUrl = stripQueryParams(url);progress?.log(`<ws connecting> ${logUrl}`);- const transport = new WebSocketTransport(progress, url, logUrl, headers, redirect.follow && redirect.hadRedirects, debugLogHeader);+ const transport = new WebSocketTransport(progress, url, logUrl, { ...options, followRedirects: !!options.followRedirects && hadRedirects });let success = false;progress?.cleanupWhenAborted(async () => {if (!success)@@ -94,13 +101,13 @@ export class WebSocketTransport implements ConnectionTransport {transport._ws.close();});transport._ws.on('unexpected-response', (request: ClientRequest, response: IncomingMessage) => {- if (redirect.follow && !redirect.hadRedirects && (response.statusCode === 301 || response.statusCode === 302 || response.statusCode === 307 || response.statusCode === 308)) {+ if (options.followRedirects && !hadRedirects && (response.statusCode === 301 || response.statusCode === 302 || response.statusCode === 307 || response.statusCode === 308)) {fulfill({ redirect: response });transport._ws.close();return;}for (let i = 0; i < response.rawHeaders.length; i += 2) {- if (debugLogHeader && response.rawHeaders[i] === debugLogHeader)+ if (options.debugLogHeader && response.rawHeaders[i] === options.debugLogHeader)progress?.log(response.rawHeaders[i + 1]);}const chunks: Buffer[] = [];@@ -117,32 +124,34 @@ export class WebSocketTransport implements ConnectionTransport {if (result.redirect) {// Strip authorization headers from the redirected request.- const newHeaders = Object.fromEntries(Object.entries(headers || {}).filter(([name]) => {+ const newHeaders = Object.fromEntries(Object.entries(options.headers || {}).filter(([name]) => {return !name.includes('access-key') && name.toLowerCase() !== 'authorization';}));- return WebSocketTransport._connect(progress, result.redirect.headers.location!, newHeaders, { follow: true, hadRedirects: true }, debugLogHeader);+ return WebSocketTransport._connect(progress, result.redirect.headers.location!, { ...options, headers: newHeaders }, true /* hadRedirects */);}success = true;return transport;}- constructor(progress: Progress|undefined, url: string, logUrl: string, headers?: { [key: string]: string; }, followRedirects?: boolean, debugLogHeader?: string) {+ constructor(progress: Progress|undefined, url: string, logUrl: string, options: WebSocketTransportOptions) {this.wsEndpoint = url;this._logUrl = logUrl;+ const proxyAgent = createProxyAgent(options.proxy, new URL(url));+ const happyEyeballsAgent = (/^(https|wss):\/\//.test(url)) ? httpsHappyEyeballsAgent : httpHappyEyeballsAgent;this._ws = new ws(url, [], {maxPayload: 256 * 1024 * 1024, // 256Mb,// Prevent internal http client error when passing negative timeout.handshakeTimeout: Math.max(progress?.timeUntilDeadline() ?? 30_000, 1),- headers,- followRedirects,- agent: (/^(https|wss):\/\//.test(url)) ? httpsHappyEyeballsAgent : httpHappyEyeballsAgent,+ headers: options.headers,+ followRedirects: options.followRedirects,+ agent: proxyAgent || happyEyeballsAgent,perMessageDeflate,});this._ws.on('upgrade', response => {for (let i = 0; i < response.rawHeaders.length; i += 2) {this.headers.push({ name: response.rawHeaders[i], value: response.rawHeaders[i + 1] });- if (debugLogHeader && response.rawHeaders[i] === debugLogHeader)+ if (options.debugLogHeader && response.rawHeaders[i] === options.debugLogHeader)progress?.log(response.rawHeaders[i + 1]);}});diff --git a/packages/playwright-core/src/server/utils/network.ts b/packages/playwright-core/src/server/utils/network.tsindex 8e0357f958..d954a69bdf 100644--- a/packages/playwright-core/src/server/utils/network.ts+++ b/packages/playwright-core/src/server/utils/network.ts@@ -20,10 +20,11 @@ import http2 from 'http2';import https from 'https';import url from 'url';-import { HttpsProxyAgent, getProxyForUrl } from '../../utilsBundle';+import { HttpsProxyAgent, SocksProxyAgent, getProxyForUrl } from '../../utilsBundle';import { httpHappyEyeballsAgent, httpsHappyEyeballsAgent } from './happyEyeballs';import type net from 'net';+import type { ProxySettings } from '../types';export type HTTPRequestParams = {url: string,@@ -32,6 +33,7 @@ export type HTTPRequestParams = {data?: string | Buffer,timeout?: number,rejectUnauthorized?: boolean,+ proxy?: ProxySettings,};export const NET_DEFAULT_TIMEOUT = 30_000;@@ -49,22 +51,26 @@ export function httpRequest(params: HTTPRequestParams, onResponse: (r: http.Incoconst timeout = params.timeout ?? NET_DEFAULT_TIMEOUT;- const proxyURL = getProxyForUrl(params.url);- if (proxyURL) {- const parsedProxyURL = url.parse(proxyURL);- if (params.url.startsWith('http:')) {- options = {- path: parsedUrl.href,- host: parsedProxyURL.hostname,- port: parsedProxyURL.port,- headers: options.headers,- method: options.method- };- } else {- (parsedProxyURL as any).secureProxy = parsedProxyURL.protocol === 'https:';-- options.agent = new HttpsProxyAgent(parsedProxyURL);- options.rejectUnauthorized = false;+ if (params.proxy) {+ options.agent = createProxyAgent(params.proxy, new URL(params.url));+ } else {+ const proxyURL = getProxyForUrl(params.url);+ if (proxyURL) {+ const parsedProxyURL = url.parse(proxyURL);+ if (params.url.startsWith('http:')) {+ options = {+ path: parsedUrl.href,+ host: parsedProxyURL.hostname,+ port: parsedProxyURL.port,+ headers: options.headers,+ method: options.method+ };+ } else {+ (parsedProxyURL as any).secureProxy = parsedProxyURL.protocol === 'https:';++ options.agent = new HttpsProxyAgent(parsedProxyURL);+ options.rejectUnauthorized = false;+ }}}@@ -109,6 +115,47 @@ export function fetchData(params: HTTPRequestParams, onError?: (params: HTTPRequ});}+function shouldBypassProxy(url: URL, bypass?: string): boolean {+ if (!bypass)+ return false;+ const domains = bypass.split(',').map(s => {+ s = s.trim();+ if (!s.startsWith('.'))+ s = '.' + s;+ return s;+ });+ const domain = '.' + url.hostname;+ return domains.some(d => domain.endsWith(d));+}++export function createProxyAgent(proxy?: ProxySettings, forUrl?: URL) {+ if (!proxy)+ return;+ if (forUrl && proxy.bypass && shouldBypassProxy(forUrl, proxy.bypass))+ return;++ // Browsers allow to specify proxy without a protocol, defaulting to http.+ let proxyServer = proxy.server.trim();+ if (!/^\w+:\/\//.test(proxyServer))+ proxyServer = 'http://' + proxyServer;++ const proxyOpts = url.parse(proxyServer);+ if (proxyOpts.protocol?.startsWith('socks')) {+ return new SocksProxyAgent({+ host: proxyOpts.hostname,+ port: proxyOpts.port || undefined,+ });+ }+ if (proxy.username)+ proxyOpts.auth = `${proxy.username}:${proxy.password || ''}`;+ if (forUrl && ['ws:', 'wss:'].includes(forUrl.protocol)) {+ // Force CONNECT method for WebSockets.+ return new HttpsProxyAgent(proxyOpts);+ }+ // TODO: We should use HttpProxyAgent conditional on proxyOpts.protocol instead of always using CONNECT method.+ return new HttpsProxyAgent(proxyOpts);+}+export function createHttpServer(requestListener?: (req: http.IncomingMessage, res: http.ServerResponse) => void): http.Server;export function createHttpServer(options: http.ServerOptions, requestListener?: (req: http.IncomingMessage, res: http.ServerResponse) => void): http.Server;export function createHttpServer(...args: any[]): http.Server {diff --git a/packages/playwright-core/types/types.d.ts b/packages/playwright-core/types/types.d.tsindex aff26e8942..8ad9c6e348 100644--- a/packages/playwright-core/types/types.d.ts+++ b/packages/playwright-core/types/types.d.ts@@ -21793,6 +21793,34 @@ export interface ConnectOverCDPOptions {*/logger?: Logger;+ /**+ * Proxy settings to use for the connection between the client and the remote browser. Note this proxy **is not** used+ * by the browser to load web pages.+ */+ proxy?: {+ /**+ * Proxy to be used for the remote connection. HTTP and SOCKS proxies are supported, for example+ * `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP+ * proxy.+ */+ server: string;++ /**+ * Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.+ */+ bypass?: string;++ /**+ * Optional username to use if HTTP proxy requires authentication.+ */+ username?: string;++ /**+ * Optional password to use if HTTP proxy requires authentication.+ */+ password?: string;+ };+/*** Slows down Playwright operations by the specified amount of milliseconds. Useful so that you can see what is going* on. Defaults to 0.@@ -21834,6 +21862,34 @@ export interface ConnectOptions {*/logger?: Logger;+ /**+ * Proxy settings to use for the connection between the client and the remote browser. Note this proxy **is not** used+ * by the browser to load web pages.+ */+ proxy?: {+ /**+ * Proxy to be used for the remote connection. HTTP and SOCKS proxies are supported, for example+ * `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP+ * proxy.+ */+ server: string;++ /**+ * Optional comma-separated domains to bypass proxy, for example `".com, chromium.org, .domain.com"`.+ */+ bypass?: string;++ /**+ * Optional username to use if HTTP proxy requires authentication.+ */+ username?: string;++ /**+ * Optional password to use if HTTP proxy requires authentication.+ */+ password?: string;+ };+/*** Slows down Playwright operations by the specified amount of milliseconds. Useful so that you can see what is going* on. Defaults to 0.diff --git a/packages/protocol/src/channels.d.ts b/packages/protocol/src/channels.d.tsindex 2e1c447b88..5a9c5687b6 100644--- a/packages/protocol/src/channels.d.ts+++ b/packages/protocol/src/channels.d.ts@@ -540,6 +540,12 @@ export type LocalUtilsConnectParams = {exposeNetwork?: string,slowMo?: number,timeout?: number,+ proxy?: {+ server: string,+ bypass?: string,+ username?: string,+ password?: string,+ },socksProxyRedirectPortForTest?: number,};export type LocalUtilsConnectOptions = {@@ -547,6 +553,12 @@ export type LocalUtilsConnectOptions = {exposeNetwork?: string,slowMo?: number,timeout?: number,+ proxy?: {+ server: string,+ bypass?: string,+ username?: string,+ password?: string,+ },socksProxyRedirectPortForTest?: number,};export type LocalUtilsConnectResult = {@@ -1165,11 +1177,23 @@ export type BrowserTypeConnectOverCDPParams = {headers?: NameValue[],slowMo?: number,timeout?: number,+ proxy?: {+ server: string,+ bypass?: string,+ username?: string,+ password?: string,+ },};export type BrowserTypeConnectOverCDPOptions = {headers?: NameValue[],slowMo?: number,timeout?: number,+ proxy?: {+ server: string,+ bypass?: string,+ username?: string,+ password?: string,+ },};export type BrowserTypeConnectOverCDPResult = {browser: BrowserChannel,diff --git a/packages/protocol/src/protocol.yml b/packages/protocol/src/protocol.ymlindex c146e995da..a169f1a54c 100644--- a/packages/protocol/src/protocol.yml+++ b/packages/protocol/src/protocol.yml@@ -702,6 +702,13 @@ LocalUtils:exposeNetwork: string?slowMo: number?timeout: number?+ proxy:+ type: object?+ properties:+ server: string+ bypass: string?+ username: string?+ password: string?socksProxyRedirectPortForTest: number?returns:pipe: JsonPipe@@ -1010,6 +1017,13 @@ BrowserType:items: NameValueslowMo: number?timeout: number?+ proxy:+ type: object?+ properties:+ server: string+ bypass: string?+ username: string?+ password: string?returns:browser: BrowserdefaultContext: BrowserContext?diff --git a/tests/library/browsertype-connect.spec.ts b/tests/library/browsertype-connect.spec.tsindex 1298e38b5e..df182c32ae 100644--- a/tests/library/browsertype-connect.spec.ts+++ b/tests/library/browsertype-connect.spec.ts@@ -765,6 +765,24 @@ for (const kind of ['launchServer', 'run-server'] as const) {await browser.close();});+ test('should connect over http proxy', {+ annotation: { type: 'issue', description: 'https://github.com/microsoft/playwright/issues/33894' },+ }, async ({ connect, startRemoteServer, proxyServer }) => {+ const remoteServer = await startRemoteServer(kind);+ const url = new URL(remoteServer.wsEndpoint());+ proxyServer.forwardTo(+url.port, { allowConnectRequests: true });+ const browser = await connect(`http://some.random.host.does.not.exist:1337`, {+ proxy: { server: `localhost:${proxyServer.PORT}` },+ });+ const page = await browser.newPage();+ expect(await page.evaluate('11 * 11')).toBe(121);+ await browser.close();+ // We should "CONNECT" twice:+ // - to convert http url into ws url+ // - actually connect to the ws endpoint+ expect(proxyServer.connectHosts).toEqual(['some.random.host.does.not.exist:1337', 'some.random.host.does.not.exist:1337']);+ });+test.describe('socks proxy', () => {test.skip(({ mode }) => mode !== 'default');test.skip(kind === 'launchServer', 'not supported yet');diff --git a/tests/library/chromium/connect-over-cdp.spec.ts b/tests/library/chromium/connect-over-cdp.spec.tsindex 3191403be8..71ceefbe45 100644--- a/tests/library/chromium/connect-over-cdp.spec.ts+++ b/tests/library/chromium/connect-over-cdp.spec.ts@@ -446,6 +446,32 @@ test('should be able to connect via localhost', async ({ browserType }, testInfo}});+test('should be able to connect over http proxy', {+ annotation: { type: 'issue', description: 'https://github.com/microsoft/playwright/issues/35206' },+}, async ({ browserType, proxyServer }, testInfo) => {+ const port = 9339 + testInfo.workerIndex;+ const browserServer = await browserType.launch({+ args: ['--remote-debugging-port=' + port]+ });+ proxyServer.forwardTo(port, { allowConnectRequests: true });+ try {+ const cdpBrowser = await browserType.connectOverCDP(`http://some.random.host.does.not.exist:1337`, {+ proxy: { server: `localhost:${proxyServer.PORT}` },+ });+ const contexts = cdpBrowser.contexts();+ expect(contexts.length).toBe(1);+ const page = await contexts[0].newPage();+ expect(await page.evaluate('11 * 11')).toBe(121);+ await cdpBrowser.close();+ // We should "CONNECT" twice:+ // - to convert http url into ws url+ // - actually connect to the ws endpoint+ expect(proxyServer.connectHosts).toEqual(['some.random.host.does.not.exist:1337', 'some.random.host.does.not.exist:1337']);+ } finally {+ await browserServer.close();+ }+});+test('emulate media should not be affected by second connectOverCDP', async ({ browserType }, testInfo) => {test.info().annotations.push({ type: 'issue', description: 'https://github.com/microsoft/playwright/issues/24109' });test.fixme();diff --git a/tests/library/proxy.spec.ts b/tests/library/proxy.spec.tsindex b7df2b66f7..71233106b5 100644--- a/tests/library/proxy.spec.ts+++ b/tests/library/proxy.spec.ts@@ -323,7 +323,7 @@ it('should use SOCKS proxy for websocket requests', async ({ browserType, serverawait closeProxyServer();});-it('should use http proxy for websocket requests', async ({ browserName, browserType, server, proxyServer }) => {+it('should use http proxy for websocket requests', async ({ isWindows, browserName, browserType, server, proxyServer }) => {proxyServer.forwardTo(server.PORT, { allowConnectRequests: true });const browser = await browserType.launch({proxy: { server: `localhost:${proxyServer.PORT}` }@@ -350,7 +350,7 @@ it('should use http proxy for websocket requests', async ({ browserName, browser// WebKit does not use CONNECT for websockets, but other browsers do.if (browserName === 'webkit')- expect(proxyServer.wsUrls).toContain('ws://fake-localhost-127-0-0-1.nip.io:1337/ws');+ expect(proxyServer.wsUrls).toContain(isWindows ? '/ws' : 'ws://fake-localhost-127-0-0-1.nip.io:1337/ws');elseexpect(proxyServer.connectHosts).toContain('fake-localhost-127-0-0-1.nip.io:1337');