Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
Revert "feat(test-runner): add httpCache config option" (#41855)
Pavel Feldmanf5ae97cb5d31
diff --git a/docs/src/test-api/class-testconfig.md b/docs/src/test-api/class-testconfig.mdindex ffe92beda8..848882fe2e 100644--- a/docs/src/test-api/class-testconfig.md+++ b/docs/src/test-api/class-testconfig.md@@ -68,115 +68,6 @@ The structure of the git commit metadata is subject to change.:::-## property: TestConfig.httpCache-* since: v1.62-- type: ?<[Object]>- - `dir` <[string]> Directory where the cache is stored, resolved relative to the configuration file.- - `match` ?<[string]|[RegExp]|[HttpCachePolicy]> Limits or customizes what is cached. A glob pattern or regular expression restricts caching to requests whose URL matches; a callback returns a per-request decision (see [HttpCachePolicy]). When omitted, every request is considered with the default behavior.- - `proxy` ?<[Object]> Upstream proxy for cache misses.- - `server` <[string]> Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP proxy.- - `bypass` ?<[string]> Optional comma-separated domains to bypass proxy.- - `username` ?<[string]> Optional username to use if HTTP proxy requires authentication.- - `password` ?<[string]> Optional password to use if HTTP proxy requires authentication.--Records network responses to disk and replays them on later runs, so large static-dependencies are downloaded from a remote server once instead of on every run. This is-most useful against a slow or remote environment such as staging.--When `httpCache` is set, Playwright starts a caching proxy for the run and routes all-browser traffic through it. On the first run, eligible responses are recorded under `dir`;-on subsequent runs they are served from disk without reaching the network. A single proxy-is shared by all workers, so a resource fetched by one worker is a cache hit for the rest,-and the cache persists across runs until you delete `dir`.--Loopback traffic (`localhost`, `127.0.0.1`) is never cached — a local dev server already-serves from disk, so there is nothing to optimize. The cache targets remote origins.--**What is cached by default**--With no `match`, the cache stores only **shared static assets**: successful `GET` requests-the browser makes for a static subresource — a script, stylesheet, image, font, or media-element — as reported by the request's `Sec-Fetch-Dest` metadata. These bytes do not depend-on who is signed in, so replaying them into a fresh browser context is always safe, which-keeps tests that each create their own context isolated by construction.--The following are therefore **not** cached by default:--* `fetch`/`XMLHttpRequest` (API) requests and top-level documents — their request- destination is not a static subresource. This is the dynamic, per-user surface.-* Any response marked `Cache-Control: no-store`.-* Any response carrying a personalization signal: `Cache-Control: private`, a `Set-Cookie`- header, or `Vary: Cookie`/`Vary: Authorization`.--The `Authorization` and `Cookie` request headers are deliberately ignored when deciding-what to cache. On a gated staging environment these are a shared environment credential-attached to every request, not a per-user identity, so caching on their presence would be-wrong.--Freshness directives (`max-age`, `no-cache`, `Expires`) are ignored: once a response is-recorded it is replayed until `dir` is deleted, keeping runs deterministic. `Vary` is-honored — responses are keyed by the request-header values they vary on, and `Vary: *` is-never stored.--**Customizing with `match`**--A string or [RegExp] restricts caching to requests whose URL matches; other requests pass-straight through to the network. For full control, pass a callback that returns a decision-object per request:--* `disposition` — `'cache'` force-stores the response and serves it back, `'no-cache'`- bypasses the cache entirely, and `'default'` (or an empty object) applies the rules above.-* `identity` — a stable principal id (such as a session token) that partitions the cache.- Entries recorded under one identity are never served to a request with a different one,- so per-user content can be cached without leaking across contexts. The value is hashed- into the cache key and never written to disk.--Set `proxy` to fetch cache misses through an upstream proxy — for example, to reach a-staging environment that is only accessible behind one. Browsers connect to the caching-proxy, which chains to `proxy` for anything not served from disk.--**Usage**--Cache shared static assets from a staging server with zero configuration:--```js title="playwright.config.ts"-import { defineConfig } from '@playwright/test';--export default defineConfig({- httpCache: { dir: './.network-cache' },-});-```--Fetch cache misses through an upstream proxy:--```js title="playwright.config.ts"-import { defineConfig } from '@playwright/test';--export default defineConfig({- httpCache: { dir: './.network-cache', proxy: { server: 'http://myproxy.com:3128' } },-});-```--Take control per request — force-cache a per-user API response with session isolation, and-bypass the cache for others:--```js title="playwright.config.ts"-import { defineConfig } from '@playwright/test';--export default defineConfig({- httpCache: {- dir: './.network-cache',- match: request => {- if (request.url.includes('/api/config'))- return { disposition: 'cache', identity: request.headers.get('authorization') };- if (request.url.includes('/telemetry'))- return { disposition: 'no-cache' };- return {};- },- },-});-```-## property: TestConfig.expect* since: v1.10- type: ?<[Object]>diff --git a/packages/playwright/src/common/config.ts b/packages/playwright/src/common/config.tsindex c971e8b8b1..0287e87d3e 100644--- a/packages/playwright/src/common/config.ts+++ b/packages/playwright/src/common/config.ts@@ -48,7 +48,6 @@ export class FullConfigInternal {readonly projects: FullProjectInternal[] = [];readonly singleTSConfigPath?: string;readonly captureGitInfo: Config['captureGitInfo'];- readonly httpCache: Config['httpCache'];readonly retryStrategy: 'immediate' | 'isolated';defineConfigWasUsed = false;@@ -69,7 +68,6 @@ export class FullConfigInternal {this.plugins = (privateConfiguration?.plugins || []).map((p: any) => ({ factory: p }));this.singleTSConfigPath = pathResolve(configDir, userConfig.tsconfig);this.captureGitInfo = userConfig.captureGitInfo;- this.httpCache = resolveHttpCache(userConfig.httpCache, configDir);this.retryStrategy = takeFirst(userConfig.retryStrategy, 'immediate');this.globalSetups = (Array.isArray(userConfig.globalSetup) ? userConfig.globalSetup : [userConfig.globalSetup]).map(s => resolveScript(s, configDir)).filter(script => script !== undefined);@@ -220,12 +218,6 @@ function resolveReporters(reporters: Config['reporter'], rootDir: string): Repor});}-function resolveHttpCache(httpCache: Config['httpCache'], configDir: string): Config['httpCache'] {- if (!httpCache)- return undefined;- return { ...httpCache, dir: path.resolve(configDir, httpCache.dir) };-}-function resolveWorkers(workers: string | number): number {if (typeof workers === 'string') {if (workers.endsWith('%')) {diff --git a/packages/playwright/src/index.ts b/packages/playwright/src/index.tsindex a8b7cd2d5f..34f9d6deda 100644--- a/packages/playwright/src/index.ts+++ b/packages/playwright/src/index.ts@@ -226,7 +226,7 @@ const playwrightFixtures: Fixtures<TestFixtures, WorkerFixtures, UtilityTestFixt}, { scope: 'worker', option: true, box: true }],video: ['off', { scope: 'worker', option: true, box: true }],- _browserOptions: [async ({ playwright, headless, channel, launchOptions }, use, workerInfo) => {+ _browserOptions: [async ({ playwright, headless, channel, launchOptions }, use) => {const options: LaunchOptions = {handleSIGINT: false,...launchOptions,@@ -237,9 +237,6 @@ const playwrightFixtures: Fixtures<TestFixtures, WorkerFixtures, UtilityTestFixtoptions.headless = headless;if (channel !== undefined)options.channel = channel;- const cacheProxy = cacheProxySettings();- if (cacheProxy)- options.proxy = cacheProxy;playwright._defaultLaunchOptions = options;await use(options);@@ -370,12 +367,6 @@ const playwrightFixtures: Fixtures<TestFixtures, WorkerFixtures, UtilityTestFixtif (serviceWorkers !== undefined)options.serviceWorkers = serviceWorkers;- const cacheProxy = cacheProxySettings();- if (cacheProxy) {- options.proxy = cacheProxy;- options.ignoreHTTPSErrors = true;- }-await use({...contextOptions,...options,@@ -584,14 +575,6 @@ function resolveClientCerticates(clientCertificates: ClientCertificates): Clientconst kTracingStarted = Symbol('kTracingStarted');-function cacheProxySettings(): { server: string, bypass: string } | undefined {- const server = process.env.PLAYWRIGHT_TEST_CACHE_PROXY;- // The cache targets remote environments (e.g. staging). Loopback is bypassed- // so a local dev server - which already serves from disk - is never cached,- // consistently across browsers.- return server ? { server, bypass: 'localhost, 127.0.0.1, ::1' } : undefined;-}-function connectOptionsFromEnv() {const wsEndpoint = process.env.PW_TEST_CONNECT_WS_ENDPOINT;if (!wsEndpoint)diff --git a/packages/playwright/src/plugins/DEPS.list b/packages/playwright/src/plugins/DEPS.listindex 1d04d3acb1..af3c8c1c20 100644--- a/packages/playwright/src/plugins/DEPS.list+++ b/packages/playwright/src/plugins/DEPS.list@@ -1,7 +1,5 @@[*]@isomorphic/**@utils/**-cacheProxy/**-../common/node_modules/colors/safenode_modules/debugdiff --git a/packages/playwright/src/plugins/cacheProxy/cache.ts b/packages/playwright/src/plugins/cacheProxy/cache.tsdeleted file mode 100644index 49840f0a66..0000000000--- a/packages/playwright/src/plugins/cacheProxy/cache.ts+++ /dev/null@@ -1,192 +0,0 @@-/**- * Copyright (c) Microsoft Corporation.- *- * Licensed under the Apache License, Version 2.0 (the "License");- * you may not use this file except in compliance with the License.- * You may obtain a copy of the License at- *- * http://www.apache.org/licenses/LICENSE-2.0- *- * Unless required by applicable law or agreed to in writing, software- * distributed under the License is distributed on an "AS IS" BASIS,- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- * See the License for the specific language governing permissions and- * limitations under the License.- */--import fs from 'fs';-import path from 'path';--import { calculateSha1, createGuid } from '@utils/crypto';-import { existsAsync } from '@utils/fileUtils';--import { computeVaryFields, varyMatches } from './cacheSemantics';--import type { ProxyHeaders } from './cacheSemantics';--const CACHE_VERSION = 1;-const INLINE_THRESHOLD = 8 * 1024;--export type CachedResponse = {- status: number;- statusText: string;- headers: [string, string][];- body: Buffer;-};--type CacheRecord = {- k: string; // key: sha1(method + '\n' + url)- u: string; // url, verbatim so the index stays greppable- s: number; // status- st: string; // statusText- hh: [string, string][]; // headers, verbatim (incl. content-encoding)- ts: number; // record time, epoch seconds- v?: [string, string][]; // Vary fields: request header values this variant is keyed on- c?: string; // inline body (base64), or...- f?: string; // ...content-addressed blob (sha1)-};--export class ResponseCache {- private _dir: string;- private _blobsDir: string;- private _indexFile: string;- private _metaFile: string;- private _index = new Map<string, CacheRecord[]>();- private _writeChain: Promise<void> = Promise.resolve();- private _initialized = false;-- constructor(dir: string) {- this._dir = path.resolve(dir);- this._blobsDir = path.join(this._dir, 'blobs');- this._indexFile = path.join(this._dir, 'index.jsonl');- this._metaFile = path.join(this._dir, 'meta.json');- }-- async load() {- let content: string;- try {- content = await fs.promises.readFile(this._indexFile, 'utf8');- } catch {- return; // No cache yet.- }- // Discard an incompatible cache instead of mis-parsing it.- if (await this._onDiskVersion() !== CACHE_VERSION) {- await fs.promises.rm(this._indexFile, { force: true }).catch(() => {});- await fs.promises.rm(this._blobsDir, { recursive: true, force: true }).catch(() => {});- return;- }- for (const line of content.split('\n')) {- if (!line.trim())- continue;- try {- const record = JSON.parse(line) as CacheRecord;- this._variants(record.k).push(record);- } catch {- // Ignore a malformed line rather than failing the whole cache.- }- }- }-- static key(method: string, url: string, identity: string = ''): string {- return calculateSha1(`${method}\n${url}\n${identity}`);- }-- async get(key: string, requestHeaders: ProxyHeaders): Promise<CachedResponse | undefined> {- const record = this._index.get(key)?.find(candidate => varyMatches(candidate.v, requestHeaders));- if (!record)- return undefined;- try {- const body = record.c !== undefined ? Buffer.from(record.c, 'base64') : await fs.promises.readFile(this._blobPath(record.f!));- return { status: record.s, statusText: record.st, headers: record.hh, body };- } catch {- return undefined; // Corrupt/missing blob - treat as a miss.- }- }-- async set(key: string, url: string, requestHeaders: ProxyHeaders, response: CachedResponse) {- const vary = computeVaryFields(response.headers, requestHeaders);- const variants = this._variants(key);- if (variants.some(existing => sameVary(existing.v, vary)))- return;- const record: CacheRecord = {- k: key,- u: url,- s: response.status,- st: response.statusText,- hh: response.headers,- ts: Math.floor(Date.now() / 1000),- };- if (vary)- record.v = vary;- if (response.body.length >= INLINE_THRESHOLD) {- const hash = calculateSha1(response.body);- await this._writeBlob(hash, response.body);- record.f = hash;- } else {- record.c = response.body.toString('base64');- }- variants.push(record);- await this._append(record);- }-- async flush() {- await this._writeChain;- }-- private _variants(key: string): CacheRecord[] {- let variants = this._index.get(key);- if (!variants) {- variants = [];- this._index.set(key, variants);- }- return variants;- }-- private _blobPath(hash: string): string {- return path.join(this._blobsDir, hash.slice(0, 2), hash);- }-- private async _writeBlob(hash: string, body: Buffer) {- const dest = this._blobPath(hash);- if (await existsAsync(dest))- return;- await fs.promises.mkdir(path.dirname(dest), { recursive: true });- const tmp = `${dest}.tmp-${createGuid()}`;- await fs.promises.writeFile(tmp, body);- try {- await fs.promises.rename(tmp, dest);- } catch {- await fs.promises.rm(tmp, { force: true }).catch(() => {});- }- }-- private _append(record: CacheRecord): Promise<void> {- const line = JSON.stringify(record) + '\n';- const run = async () => {- if (!this._initialized) {- await fs.promises.mkdir(this._dir, { recursive: true });- await fs.promises.writeFile(this._metaFile, JSON.stringify({ version: CACHE_VERSION }, null, 2), { flag: 'w' });- this._initialized = true;- }- await fs.promises.appendFile(this._indexFile, line);- };- const result = this._writeChain.then(run, run);- this._writeChain = result.catch(() => {});- return result;- }-- private async _onDiskVersion(): Promise<number | undefined> {- try {- const meta = JSON.parse(await fs.promises.readFile(this._metaFile, 'utf8'));- return typeof meta.version === 'number' ? meta.version : undefined;- } catch {- return undefined;- }- }-}--function sameVary(a: [string, string][] | undefined, b: [string, string][] | undefined): boolean {- if (!a || !b)- return !a && !b;- return a.length === b.length && a.every(([name, value], i) => b[i][0] === name && b[i][1] === value);-}diff --git a/packages/playwright/src/plugins/cacheProxy/cacheSemantics.ts b/packages/playwright/src/plugins/cacheProxy/cacheSemantics.tsdeleted file mode 100644index c3beb8f7da..0000000000--- a/packages/playwright/src/plugins/cacheProxy/cacheSemantics.ts+++ /dev/null@@ -1,108 +0,0 @@-/**- * Copyright (c) Microsoft Corporation.- *- * Licensed under the Apache License, Version 2.0 (the "License");- * you may not use this file except in compliance with the License.- * You may obtain a copy of the License at- *- * http://www.apache.org/licenses/LICENSE-2.0- *- * Unless required by applicable law or agreed to in writing, software- * distributed under the License is distributed on an "AS IS" BASIS,- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- * See the License for the specific language governing permissions and- * limitations under the License.- */--import type http from 'http';--// RFC 9111 6.1 heuristically cacheable status codes, minus 206 (partial) and-// temporary redirects (302/307), which only cache with explicit freshness that-// this cache intentionally ignores.-const CACHEABLE_STATUS = new Set([200, 203, 204, 300, 301, 308, 404, 405, 410, 414, 501]);-const SEC_FETCH_DEST_STATIC = new Set(['audio', 'audioworklet', 'embed', 'font', 'image', 'manifest', 'object', 'paintworklet', 'script', 'serviceworker', 'sharedworker', 'style', 'track', 'video', 'worker', 'xslt']);--export type ProxyHeaders = [string, string][] | http.IncomingHttpHeaders;--export function isCacheableStatus(status: number): boolean {- return CACHEABLE_STATUS.has(status);-}--export function isHardBlocked(requestHeaders: ProxyHeaders, status: number, responseHeaders: ProxyHeaders): boolean {- if (!isCacheableStatus(status))- return true;- if (parseCacheControl(requestHeaders).has('no-store'))- return true;- if (parseCacheControl(responseHeaders).has('no-store'))- return true;- if (varyNames(responseHeaders).includes('*'))- return true;- return false;-}--export function isDefaultStorable(requestHeaders: ProxyHeaders, status: number, responseHeaders: ProxyHeaders): boolean {- if (isHardBlocked(requestHeaders, status, responseHeaders))- return false;- if (isPersonalized(responseHeaders))- return false;- if (status === 301 || status === 308)- return true;- return SEC_FETCH_DEST_STATIC.has(headerValue(requestHeaders, 'sec-fetch-dest'));-}--function isPersonalized(responseHeaders: ProxyHeaders): boolean {- if (parseCacheControl(responseHeaders).has('private'))- return true;- if (headerValues(responseHeaders, 'set-cookie').length)- return true;- const vary = varyNames(responseHeaders);- return vary.includes('cookie') || vary.includes('authorization');-}--export function parseCacheControl(headers: ProxyHeaders): Set<string> {- const directives = new Set<string>();- for (const value of headerValues(headers, 'cache-control')) {- for (const part of value.split(',')) {- const name = part.split('=')[0].trim().toLowerCase();- if (name)- directives.add(name);- }- }- return directives;-}--export function computeVaryFields(responseHeaders: ProxyHeaders, requestHeaders: ProxyHeaders): [string, string][] | undefined {- const names = varyNames(responseHeaders).filter(name => name !== '*');- if (!names.length)- return undefined;- return names.map(name => [name, headerValue(requestHeaders, name)]);-}--export function varyMatches(fields: [string, string][] | undefined, requestHeaders: ProxyHeaders): boolean {- if (!fields)- return true;- return fields.every(([name, value]) => headerValue(requestHeaders, name) === value);-}--function varyNames(headers: ProxyHeaders): string[] {- return headerValues(headers, 'vary').flatMap(value => value.split(',').map(name => name.trim().toLowerCase()).filter(Boolean));-}--function headerValue(headers: ProxyHeaders, name: string): string {- return headerValues(headers, name).join(', ');-}--function headerValues(headers: ProxyHeaders, name: string): string[] {- if (Array.isArray(headers)) {- const result: string[] = [];- for (const [key, value] of headers) {- if (key.toLowerCase() === name)- result.push(value);- }- return result;- }- const value = headers[name];- if (value === undefined)- return [];- return Array.isArray(value) ? value : [value];-}diff --git a/packages/playwright/src/plugins/cacheProxy/server.ts b/packages/playwright/src/plugins/cacheProxy/server.tsdeleted file mode 100644index 9a1d25d92b..0000000000--- a/packages/playwright/src/plugins/cacheProxy/server.ts+++ /dev/null@@ -1,356 +0,0 @@-/**- * Copyright (c) Microsoft Corporation.- *- * Licensed under the Apache License, Version 2.0 (the "License");- * you may not use this file except in compliance with the License.- * You may obtain a copy of the License at- *- * http://www.apache.org/licenses/LICENSE-2.0- *- * Unless required by applicable law or agreed to in writing, software- * distributed under the License is distributed on an "AS IS" BASIS,- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- * See the License for the specific language governing permissions and- * limitations under the License.- */--import http from 'http';-import https from 'https';-import net from 'net';-import tls from 'tls';--import { generateSelfSignedCertificate } from '@utils/crypto';-import { httpHappyEyeballsAgent, httpsHappyEyeballsAgent } from '@utils/happyEyeballs';-import { createHttpServer, createHttpsServer, createProxyAgent, shouldBypassProxy, startHttpServer } from '@utils/network';-import { urlMatches } from '@isomorphic/urlMatch';--import { ResponseCache } from './cache';-import { isDefaultStorable, isHardBlocked } from './cacheSemantics';--import type { CachedResponse } from './cache';-import type { ProxySettings } from '@utils/network';--const HOP_BY_HOP = new Set(['connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization', 'te', 'trailer', 'transfer-encoding', 'upgrade', 'proxy-connection']);-const LOOPBACK_BYPASS = 'localhost, 127.0.0.1, ::1, [::1]';--export type HttpCacheDecision = {- disposition?: 'cache' | 'no-cache' | 'default';- identity?: string | null;-};-export type HttpCachePolicy = (request: Request) => HttpCacheDecision;-export type HttpCacheMatch = string | RegExp | HttpCachePolicy | undefined;--export type CacheEntry = { cache: ResponseCache, match: HttpCacheMatch };--type Selection = { cache: ResponseCache, read: boolean, write: 'force' | 'never' | 'default', identity: string };--export class CacheProxy {- private _entry: CacheEntry;- private _proxy: ProxySettings | undefined;- private _proxyAgent: http.Agent | undefined;- private _httpServer: http.Server;- private _httpsServer: https.Server;- private _inflight = new Map<string, Promise<CachedResponse | undefined>>();-- constructor(entry: CacheEntry, proxy?: ProxySettings) {- this._entry = entry;- this._proxy = proxy;- this._proxyAgent = createProxyAgent(proxy, undefined, { keepAlive: true });- const { cert, key } = generateSelfSignedCertificate();-- this._httpServer = createHttpServer((req, res) => this._handleRequest(req, res, false));- this._httpServer.on('connect', (req, socket, head) => this._onConnect(req, socket as net.Socket, head));- this._httpServer.on('upgrade', (req, socket, head) => this._onUpgrade(req, socket as net.Socket, head, false));- this._httpsServer = createHttpsServer({ cert, key }, (req, res) => this._handleRequest(req, res, true));- this._httpsServer.on('upgrade', (req, socket, head) => this._onUpgrade(req, socket as net.Socket, head, true));- }-- async start(): Promise<string> {- await startHttpServer(this._httpServer, { host: '127.0.0.1', port: 0 });- const address = this._httpServer.address() as net.AddressInfo;- return `http://127.0.0.1:${address.port}`;- }-- async stop() {- await Promise.all([- new Promise<void>(resolve => this._httpServer.close(() => resolve())),- new Promise<void>(resolve => this._httpsServer.close(() => resolve())),- ]);- this._proxyAgent?.destroy();- }-- private _onConnect(req: http.IncomingMessage, socket: net.Socket, head: Buffer) {- socket.on('error', () => {});- socket.write('HTTP/1.1 200 Connection Established\r\n\r\n');- // A CONNECT tunnel carries either TLS (https / wss) or a plaintext HTTP- // request (http / ws). Peek the first byte - 0x16 is the TLS handshake -- // and hand the socket to the matching server instead of assuming TLS.- // Pause first so no data is lost between the sniff and the server attaching.- const route = (first: Buffer) => {- socket.pause();- socket.unshift(first);- if (first[0] === 0x16) {- this._httpsServer.emit('connection', socket);- } else {- // The http server does not resume a socket handed over while paused- // (the TLS server does), so the tunnelled request would never parse.- this._httpServer.emit('connection', socket);- socket.resume();- }- };- if (head && head.length)- route(head);- else- socket.once('data', route);- }-- // A throwing match callback or any handler error must never take down the- // runner process the proxy lives in.- private _handleRequest(req: http.IncomingMessage, res: http.ServerResponse, isTls: boolean) {- req.on('error', () => {});- Promise.resolve().then(() => this._onRequest(req, res, isTls)).catch(() => {- if (!res.headersSent)- res.writeHead(502);- res.end();- });- }-- private async _onRequest(req: http.IncomingMessage, res: http.ServerResponse, isTls: boolean) {- req.url = requestUrl(req, isTls);- const url = req.url;- const method = req.method || 'GET';-- const selected = method === 'GET' ? this._select(req, url) : undefined;- if (!selected || (!selected.read && selected.write === 'never')) {- this._passThrough(req, res, url, method);- return;- }-- const key = ResponseCache.key('GET', url, selected.identity);- if (selected.read) {- const cached = await selected.cache.get(key, req.headers);- if (cached) {- writeResponse(res, cached);- return;- }- }- this._serveMiss(selected, key, url, req, res);- }-- private _serveMiss(selected: Selection, key: string, url: string, req: http.IncomingMessage, res: http.ServerResponse) {- // Coalesce concurrent misses onto a single upstream fetch, but only while- // that fetch is being buffered for storage - a streamed (non-storable)- // response can't be shared, so waiters on it fetch their own.- const inflight = this._inflight.get(key);- if (inflight) {- void inflight.then(response => {- if (response)- writeResponse(res, response);- else- this._fetchAndServe(selected, key, url, req, res, false);- });- return;- }- this._fetchAndServe(selected, key, url, req, res, true);- }-- private _fetchAndServe(selected: Selection, key: string, url: string, req: http.IncomingMessage, res: http.ServerResponse, coalesce: boolean) {- // Waiters receive the buffered response, or undefined when this fetch- // ended up streaming (or failing) and cannot be shared.- let finish: (response?: CachedResponse) => void = () => {};- if (coalesce) {- const buffered = new Promise<CachedResponse | undefined>(resolve => finish = response => {- this._inflight.delete(key);- resolve(response);- });- this._inflight.set(key, buffered);- }- const fail = () => {- finish();- failResponse(res);- };-- const upstream = this._requestUpstream(url, 'GET', forwardHeaders(req.headers));- upstream.on('error', fail);- res.on('close', () => upstream.destroy());- upstream.on('response', proxyRes => {- proxyRes.on('error', fail);- const status = proxyRes.statusCode || 502;- const store = selected.write === 'force'- ? !isHardBlocked(req.headers, status, proxyRes.headers)- : selected.write === 'default' && isDefaultStorable(req.headers, status, proxyRes.headers);- // Stream anything we won't store straight through, so SSE, long-poll, and- // large dynamic responses are never buffered in the runner process.- if (!store) {- finish();- res.writeHead(status, filterHeadersFlat(proxyRes.rawHeaders));- proxyRes.pipe(res);- return;- }- const chunks: Buffer[] = [];- proxyRes.on('data', chunk => chunks.push(chunk));- proxyRes.on('end', () => {- const response: CachedResponse = {- status,- statusText: proxyRes.statusMessage || '',- headers: pairsFromRaw(proxyRes.rawHeaders).filter(([name]) => !HOP_BY_HOP.has(name.toLowerCase())),- body: Buffer.concat(chunks),- };- finish(response);- selected.cache.set(key, url, req.headers, response).catch(() => {});- writeResponse(res, response);- });- });- upstream.end();- }-- private _select(req: http.IncomingMessage, url: string): Selection | undefined {- // The cache targets remote environments; loopback always passes through,- // regardless of how the browser or context was pointed at the proxy.- let parsed: URL;- try {- parsed = new URL(url);- } catch {- return undefined;- }- if (shouldBypassProxy(parsed, LOOPBACK_BYPASS))- return undefined;- const entry = this._entry;- let decision: HttpCacheDecision = {};- if (typeof entry.match === 'function')- decision = entry.match(toWebRequest(req, url)) || {};- else if (entry.match !== undefined && !urlMatches(undefined, url, entry.match))- return undefined;- return { cache: entry.cache, identity: decision.identity ?? '', ...resolveDisposition(decision) };- }-- private _passThrough(req: http.IncomingMessage, res: http.ServerResponse, url: string, method: string) {- const fail = () => failResponse(res);- const upstream = this._requestUpstream(url, method, forwardHeaders(req.headers));- upstream.on('response', proxyRes => {- proxyRes.on('error', fail);- res.writeHead(proxyRes.statusCode || 502, filterHeadersFlat(proxyRes.rawHeaders));- proxyRes.pipe(res);- });- upstream.on('error', fail);- req.on('error', () => upstream.destroy());- req.pipe(upstream);- }-- private _requestUpstream(url: string, method: string, headers: http.OutgoingHttpHeaders): http.ClientRequest {- const parsed = new URL(url);- const isHttps = parsed.protocol === 'https:';- const mod = isHttps ? https : http;- const direct = this._proxyAgent === undefined || shouldBypassProxy(parsed, this._proxy?.bypass);- const agent = direct ? (isHttps ? httpsHappyEyeballsAgent : httpHappyEyeballsAgent) : this._proxyAgent;- return mod.request(parsed, {- method,- headers,- agent,- rejectUnauthorized: false,- });- }-- private _onUpgrade(req: http.IncomingMessage, socket: net.Socket, head: Buffer, isTls: boolean) {- socket.on('error', () => {});- const target = new URL(requestUrl(req, isTls));- const host = target.hostname;- const port = Number(target.port) || (isTls ? 443 : 80);- const onConnect = () => {- const lines = [`${req.method} ${target.pathname}${target.search} HTTP/1.1`];- for (const [name, value] of Object.entries(req.headers))- lines.push(`${name}: ${Array.isArray(value) ? value.join(', ') : value}`);- upstream.write(lines.join('\r\n') + '\r\n\r\n');- if (head && head.length)- upstream.write(head);- socket.pipe(upstream);- upstream.pipe(socket);- };- const upstream = isTls- ? tls.connect({ host, port, rejectUnauthorized: false, servername: net.isIP(host) ? undefined : host }, onConnect)- : net.connect({ host, port }, onConnect);- upstream.on('error', () => socket.destroy());- }-}--// Match callbacks receive a standard WHATWG Request. Unlike a browser's,-// Node's Request keeps "forbidden" headers (host, cookie, sec-fetch-*), so-// no information is lost; the copy also keeps callbacks from mutating the-// live incoming request.-function toWebRequest(req: http.IncomingMessage, url: string): Request {- const headers: [string, string][] = [];- for (const [name, value] of Object.entries(req.headers)) {- if (value === undefined)- continue;- if (Array.isArray(value))- headers.push(...value.map(item => [name, item] as [string, string]));- else- headers.push([name, value]);- }- return new Request(url, { method: req.method, headers });-}--// Tunnelled requests arrive in origin form (`GET /path`), direct proxy-// requests in absolute form; reconstruct via the Host header when needed.-function requestUrl(req: http.IncomingMessage, isTls: boolean): string {- const raw = req.url || '/';- if (/^\w+:\/\//.test(raw))- return raw;- return `${isTls ? 'https' : 'http'}://${req.headers.host}${raw}`;-}--function resolveDisposition(decision: HttpCacheDecision): { read: boolean, write: 'force' | 'never' | 'default' } {- if (decision.disposition === 'no-cache')- return { read: false, write: 'never' };- if (decision.disposition === 'cache')- return { read: true, write: 'force' };- return { read: true, write: 'default' };-}--function failResponse(res: http.ServerResponse) {- if (!res.headersSent) {- res.writeHead(502);- res.end();- } else {- res.destroy();- }-}---function forwardHeaders(headers: http.IncomingHttpHeaders): http.OutgoingHttpHeaders {- const result: http.OutgoingHttpHeaders = {};- for (const [name, value] of Object.entries(headers)) {- if (value !== undefined && !HOP_BY_HOP.has(name.toLowerCase()))- result[name] = value;- }- return result;-}--function writeResponse(res: http.ServerResponse, cached: CachedResponse) {- const raw: string[] = [];- for (const [name, value] of cached.headers) {- const lower = name.toLowerCase();- if (HOP_BY_HOP.has(lower) || lower === 'content-length')- continue;- raw.push(name, value);- }- raw.push('Content-Length', String(cached.body.length));- res.writeHead(cached.status, cached.statusText || undefined, raw);- res.end(cached.body);-}--function filterHeadersFlat(rawHeaders: string[]): string[] {- const result: string[] = [];- for (let i = 0; i + 1 < rawHeaders.length; i += 2) {- if (!HOP_BY_HOP.has(rawHeaders[i].toLowerCase()))- result.push(rawHeaders[i], rawHeaders[i + 1]);- }- return result;-}--function pairsFromRaw(rawHeaders: string[]): [string, string][] {- const pairs: [string, string][] = [];- for (let i = 0; i + 1 < rawHeaders.length; i += 2)- pairs.push([rawHeaders[i], rawHeaders[i + 1]]);- return pairs;-}diff --git a/packages/playwright/src/plugins/cacheProxyPlugin.ts b/packages/playwright/src/plugins/cacheProxyPlugin.tsdeleted file mode 100644index 23ce518f81..0000000000--- a/packages/playwright/src/plugins/cacheProxyPlugin.ts+++ /dev/null@@ -1,46 +0,0 @@-/**- * Copyright (c) Microsoft Corporation.- *- * Licensed under the Apache License, Version 2.0 (the "License");- * you may not use this file except in compliance with the License.- * You may obtain a copy of the License at- *- * http://www.apache.org/licenses/LICENSE-2.0- *- * Unless required by applicable law or agreed to in writing, software- * distributed under the License is distributed on an "AS IS" BASIS,- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- * See the License for the specific language governing permissions and- * limitations under the License.- */--import { CacheProxy } from './cacheProxy/server';-import { ResponseCache } from './cacheProxy/cache';--import type { TestRunnerPlugin } from '.';-import type { FullConfigInternal } from '../common';--export const cacheProxyPluginForConfig = (config: FullConfigInternal): TestRunnerPlugin[] => {- const httpCache = config.httpCache;- if (!httpCache)- return [];-- let cache: ResponseCache | undefined;- let proxy: CacheProxy | undefined;- return [{- name: 'playwright:cache-proxy',- setup: async () => {- cache = new ResponseCache(httpCache.dir);- await cache.load();- proxy = new CacheProxy({ cache, match: httpCache.match }, httpCache.proxy);- process.env.PLAYWRIGHT_TEST_CACHE_PROXY = await proxy.start();- },- teardown: async () => {- delete process.env.PLAYWRIGHT_TEST_CACHE_PROXY;- await proxy?.stop();- await cache?.flush();- proxy = undefined;- cache = undefined;- },- }];-};diff --git a/packages/playwright/src/runner/testRunner.ts b/packages/playwright/src/runner/testRunner.tsindex 65fc0cbde0..9f8b9c37d5 100644--- a/packages/playwright/src/runner/testRunner.ts+++ b/packages/playwright/src/runner/testRunner.ts@@ -29,7 +29,6 @@ import { FSWatcher } from './fsWatcher';import { baseFullConfig } from '../isomorphic/teleReceiver';import { addGitCommitInfoPlugin } from '../plugins/gitCommitInfoPlugin';import { webServerPluginsForConfig } from '../plugins/webServerPlugin';-import { cacheProxyPluginForConfig } from '../plugins/cacheProxyPlugin';import { internalScreen } from '../reporters/base';import { InternalReporter } from '../reporters/internalReporter';import { serializeError } from '../util';@@ -395,7 +394,6 @@ export class TestRunner extends EventEmitter<TestRunnerEventMap> {// Preserve plugin instances between setup and build.if (!this._plugins) {webServerPluginsForConfig(config).forEach(p => config.plugins.push({ factory: p }));- cacheProxyPluginForConfig(config).forEach(p => config.plugins.push({ factory: p }));addGitCommitInfoPlugin(config);this._plugins = config.plugins || [];} else {@@ -451,7 +449,6 @@ export async function runAllTestsWithConfig(config: FullConfigInternal, options:// Legacy webServer support.webServerPluginsForConfig(config).forEach(p => config.plugins.push({ factory: p }));- cacheProxyPluginForConfig(config).forEach(p => config.plugins.push({ factory: p }));const filteredProjects = filterProjects(config.projects, options.projectFilter);const reporters = await createReporters(config, options.listMode ? 'list' : 'test', undefined, options);diff --git a/packages/playwright/types/test.d.ts b/packages/playwright/types/test.d.tsindex 46005f1248..44d6742a8c 100644--- a/packages/playwright/types/test.d.ts+++ b/packages/playwright/types/test.d.ts@@ -1415,141 +1415,6 @@ interface TestConfig<TestArgs = {}, WorkerArgs = {}> {*/grepInvert?: RegExp|Array<RegExp>;- /**- * Records network responses to disk and replays them on later runs, so large static dependencies are downloaded from- * a remote server once instead of on every run. This is most useful against a slow or remote environment such as- * staging.- *- * When `httpCache` is set, Playwright starts a caching proxy for the run and routes all browser traffic through it.- * On the first run, eligible responses are recorded under `dir`; on subsequent runs they are served from disk without- * reaching the network. A single proxy is shared by all workers, so a resource fetched by one worker is a cache hit- * for the rest, and the cache persists across runs until you delete `dir`.- *- * Loopback traffic (`localhost`, `127.0.0.1`) is never cached — a local dev server already serves from disk, so there- * is nothing to optimize. The cache targets remote origins.- *- * **What is cached by default**- *- * With no `match`, the cache stores only **shared static assets**: successful `GET` requests the browser makes for a- * static subresource — a script, stylesheet, image, font, or media element — as reported by the request's- * `Sec-Fetch-Dest` metadata. These bytes do not depend on who is signed in, so replaying them into a fresh browser- * context is always safe, which keeps tests that each create their own context isolated by construction.- *- * The following are therefore **not** cached by default:- * - `fetch`/`XMLHttpRequest` (API) requests and top-level documents — their request destination is not a static- * subresource. This is the dynamic, per-user surface.- * - Any response marked `Cache-Control: no-store`.- * - Any response carrying a personalization signal: `Cache-Control: private`, a `Set-Cookie` header, or `Vary:- * Cookie`/`Vary: Authorization`.- *- * The `Authorization` and `Cookie` request headers are deliberately ignored when deciding what to cache. On a gated- * staging environment these are a shared environment credential attached to every request, not a per-user identity,- * so caching on their presence would be wrong.- *- * Freshness directives (`max-age`, `no-cache`, `Expires`) are ignored: once a response is recorded it is replayed- * until `dir` is deleted, keeping runs deterministic. `Vary` is honored — responses are keyed by the request-header- * values they vary on, and `Vary: *` is never stored.- *- * **Customizing with `match`**- *- * A string or [RegExp] restricts caching to requests whose URL matches; other requests pass straight through to the- * network. For full control, pass a callback that returns a decision object per request:- * - `disposition` — `'cache'` force-stores the response and serves it back, `'no-cache'` bypasses the cache- * entirely, and `'default'` (or an empty object) applies the rules above.- * - `identity` — a stable principal id (such as a session token) that partitions the cache. Entries recorded under- * one identity are never served to a request with a different one, so per-user content can be cached without- * leaking across contexts. The value is hashed into the cache key and never written to disk.- *- * Set `proxy` to fetch cache misses through an upstream proxy — for example, to reach a staging environment that is- * only accessible behind one. Browsers connect to the caching proxy, which chains to `proxy` for anything not served- * from disk.- *- * **Usage**- *- * Cache shared static assets from a staging server with zero configuration:- *- * ```js- * // playwright.config.ts- * import { defineConfig } from '@playwright/test';- *- * export default defineConfig({- * httpCache: { dir: './.network-cache' },- * });- * ```- *- * Fetch cache misses through an upstream proxy:- *- * ```js- * // playwright.config.ts- * import { defineConfig } from '@playwright/test';- *- * export default defineConfig({- * httpCache: { dir: './.network-cache', proxy: { server: 'http://myproxy.com:3128' } },- * });- * ```- *- * Take control per request — force-cache a per-user API response with session isolation, and bypass the cache for- * others:- *- * ```js- * // playwright.config.ts- * import { defineConfig } from '@playwright/test';- *- * export default defineConfig({- * httpCache: {- * dir: './.network-cache',- * match: request => {- * if (request.url.includes('/api/config'))- * return { disposition: 'cache', identity: request.headers.get('authorization') };- * if (request.url.includes('/telemetry'))- * return { disposition: 'no-cache' };- * return {};- * },- * },- * });- * ```- *- */- httpCache?: {- /**- * Directory where the cache is stored, resolved relative to the configuration file.- */- dir: string;-- /**- * Limits or customizes what is cached. A glob pattern or regular expression restricts caching to requests whose URL- * matches; a callback returns a per-request decision (see [HttpCachePolicy]). When omitted, every request is- * considered with the default behavior.- */- match?: string|RegExp|HttpCachePolicy;-- /**- * Upstream proxy for cache misses.- */- proxy?: {- /**- * Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example `http://myproxy.com:3128` or- * `socks5://myproxy.com:3128`. Short form `myproxy.com:3128` is considered an HTTP proxy.- */- server: string;-- /**- * Optional comma-separated domains to bypass proxy.- */- bypass?: string;-- /**- * Optional username to use if HTTP proxy requires authentication.- */- username?: string;-- /**- * Optional password to use if HTTP proxy requires authentication.- */- password?: string;- };- };-/*** Whether to skip snapshot expectations, such as `expect(value).toMatchSnapshot()` and `await* expect(page).toHaveScreenshot()`.@@ -2160,32 +2025,6 @@ export interface Config<TestArgs = {}, WorkerArgs = {}> extends TestConfig<TestAexport type Metadata = { [key: string]: any };-/**- * A per-request caching decision. Every field is optional; an empty object applies the- * default behavior.- */-export type HttpCacheDecision = {- /**- * `'cache'` serves a stored response and force-stores on a miss, `'no-cache'` bypasses the cache (fetch fresh, do- * not store), and `'default'` applies the default rules (cache shared static assets only). Defaults to `'default'`.- */- disposition?: 'cache' | 'no-cache' | 'default';-- /**- * A stable principal id, such as a session token, that partitions the cache. Entries recorded under one identity- * are never served to a request with a different one, so per-user content can be cached without leaking across- * contexts. The value is hashed into the cache key and never written to disk. `null` is treated as anonymous, so- * `request.headers.get()` results can be passed directly.- */- identity?: string | null;-};--/**- * Called for each request to decide how it is cached. Receives a standard- * [Request](https://developer.mozilla.org/en-US/docs/Web/API/Request). See [`property: TestConfig.httpCache`].- */-export type HttpCachePolicy = (request: Request) => HttpCacheDecision;-/*** Resolved configuration which is accessible via* [testInfo.config](https://playwright.dev/docs/api/class-testinfo#test-info-config) and is passed to the testdiff --git a/packages/utils/network.ts b/packages/utils/network.tsindex 1929ab56f9..3b3790754c 100644--- a/packages/utils/network.ts+++ b/packages/utils/network.ts@@ -103,7 +103,7 @@ export function httpRequest(params: HTTPRequestParams, onResponse: (r: http.Incoreturn { cancel: e => cancelRequest(e) };}-export function shouldBypassProxy(url: URL, bypass?: string): boolean {+function shouldBypassProxy(url: URL, bypass?: string): boolean {if (!bypass)return false;const domains = bypass.split(',').map(s => {@@ -124,7 +124,7 @@ function normalizeProxyURL(proxy: string): URL {return new URL(proxy);}-export function createProxyAgent(proxy?: ProxySettings, forUrl?: URL, agentOptions?: http.AgentOptions) {+export function createProxyAgent(proxy?: ProxySettings, forUrl?: URL) {if (!proxy)return;if (forUrl && proxy.bypass && shouldBypassProxy(forUrl, proxy.bypass))@@ -140,7 +140,7 @@ export function createProxyAgent(proxy?: ProxySettings, forUrl?: URL, agentOptioelse if (proxyURL.protocol === 'socks4:')proxyURL.protocol = 'socks4a:';- return new SocksProxyAgent(proxyURL, agentOptions);+ return new SocksProxyAgent(proxyURL);}if (proxy.username) {proxyURL.username = proxy.username;@@ -149,11 +149,11 @@ export function createProxyAgent(proxy?: ProxySettings, forUrl?: URL, agentOptioif (forUrl && ['ws:', 'wss:'].includes(forUrl.protocol)) {// Force CONNECT method for WebSockets.- return new HttpsProxyAgent(proxyURL, agentOptions);+ return new HttpsProxyAgent(proxyURL);}// TODO: This branch should be different from above. We should use HttpProxyAgent conditional on proxyURL.protocol instead of always using CONNECT method.- return new HttpsProxyAgent(proxyURL, agentOptions);+ return new HttpsProxyAgent(proxyURL);}export function createHttpServer(requestListener?: (req: http.IncomingMessage, res: http.ServerResponse) => void): http.Server;diff --git a/tests/library/unit/cache-proxy.spec.ts b/tests/library/unit/cache-proxy.spec.tsdeleted file mode 100644index 89060af57a..0000000000--- a/tests/library/unit/cache-proxy.spec.ts+++ /dev/null@@ -1,598 +0,0 @@-/**- * Copyright (c) Microsoft Corporation.- *- * Licensed under the Apache License, Version 2.0 (the "License");- * you may not use this file except in compliance with the License.- * You may obtain a copy of the License at- *- * http://www.apache.org/licenses/LICENSE-2.0- *- * Unless required by applicable law or agreed to in writing, software- * distributed under the License is distributed on an "AS IS" BASIS,- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- * See the License for the specific language governing permissions and- * limitations under the License.- */--import { test as base, expect } from '@playwright/test';-import http from 'http';-import https from 'https';-import net from 'net';-import fs from 'fs';-import path from 'path';-import { WebSocket, WebSocketServer } from 'ws';-import { HttpsProxyAgent } from 'https-proxy-agent';-import { generateSelfSignedCertificate } from '@utils/crypto';-import { CacheProxy } from '../../../packages/playwright/src/plugins/cacheProxy/server';-import { ResponseCache } from '../../../packages/playwright/src/plugins/cacheProxy/cache';-import type { CacheEntry } from '../../../packages/playwright/src/plugins/cacheProxy/server';-import type { ProxySettings } from '@utils/network';--// A non-loopback name that resolves to 127.0.0.1, so the proxy's loopback-// bypass does not skip our local origin.-const HOST = 'fake-localhost-127-0-0-1.nip.io';--type RouteHandler = (req: http.IncomingMessage, res: http.ServerResponse) => void;-type Origin = {- port: number;- url: (p: string) => string;- loopbackUrl: (p: string) => string;- wsUrl: (p: string) => string;- setRoute: (p: string, h: RouteHandler) => void;- hits: (p: string) => number;- onWebSocket: (h: (ws: WebSocket) => void) => void;- close: () => Promise<void>;-};--async function startOrigin(tls?: https.ServerOptions): Promise<Origin> {- const hits = new Map<string, number>();- const routes = new Map<string, RouteHandler>();- const handler: RouteHandler = (req, res) => {- const p = new URL(req.url || '/', 'http://x').pathname;- hits.set(p, (hits.get(p) || 0) + 1);- const route = routes.get(p);- if (route) {- route(req, res);- } else {- res.writeHead(404);- res.end('not found');- }- };- const server = tls ? https.createServer(tls, handler) : http.createServer(handler);- const wss = new WebSocketServer({ server });- let wsHandler: ((ws: WebSocket) => void) | undefined;- wss.on('connection', ws => wsHandler?.(ws));- await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve));- const port = (server.address() as net.AddressInfo).port;- const scheme = tls ? 'https' : 'http';- return {- port,- url: p => `${scheme}://${HOST}:${port}${p}`,- loopbackUrl: p => `${scheme}://127.0.0.1:${port}${p}`,- wsUrl: p => `${tls ? 'wss' : 'ws'}://${HOST}:${port}${p}`,- setRoute: (p, h) => routes.set(p, h),- hits: p => hits.get(p) || 0,- onWebSocket: h => { wsHandler = h; },- close: () => new Promise<void>(resolve => {- for (const client of wss.clients)- client.terminate();- wss.close();- server.closeAllConnections();- server.close(() => resolve());- }),- };-}--type Result = { status: number, headers: http.IncomingHttpHeaders, body: string };--// Drive a forward-proxy GET (absolute request target) through the proxy and-// buffer the whole response.-function drive(proxyAddress: string, targetUrl: string, headers: Record<string, string> = {}, method = 'GET'): Promise<Result> {- const proxy = new URL(proxyAddress);- const target = new URL(targetUrl);- return new Promise((resolve, reject) => {- const req = http.request({ host: proxy.hostname, port: Number(proxy.port), method, path: targetUrl, headers: { host: target.host, ...headers } }, res => {- const chunks: Buffer[] = [];- res.on('data', chunk => chunks.push(chunk));- res.on('end', () => resolve({ status: res.statusCode || 0, headers: res.headers, body: Buffer.concat(chunks).toString() }));- res.on('error', reject);- });- req.on('error', reject);- req.end();- });-}--// Resolve as soon as the first body chunk arrives, without waiting for the-// response to end - used to prove streaming responses are not buffered.-function driveFirstChunk(proxyAddress: string, targetUrl: string, headers: Record<string, string> = {}): Promise<{ status: number, chunk: string }> {- const proxy = new URL(proxyAddress);- const target = new URL(targetUrl);- return new Promise((resolve, reject) => {- const req = http.request({ host: proxy.hostname, port: Number(proxy.port), path: targetUrl, headers: { host: target.host, ...headers } }, res => {- res.once('data', chunk => {- resolve({ status: res.statusCode || 0, chunk: chunk.toString() });- req.destroy();- });- res.on('error', () => {});- });- req.on('error', reject);- req.end();- });-}--// Drive an https (MITM) GET through the proxy via CONNECT.-function driveTls(proxyAddress: string, targetUrl: string, headers: Record<string, string> = {}): Promise<Result> {- const agent = new HttpsProxyAgent(proxyAddress);- return new Promise((resolve, reject) => {- const req = https.request(targetUrl, { agent, rejectUnauthorized: false, headers } as https.RequestOptions, res => {- const chunks: Buffer[] = [];- res.on('data', chunk => chunks.push(chunk));- res.on('end', () => resolve({ status: res.statusCode || 0, headers: res.headers, body: Buffer.concat(chunks).toString() }));- res.on('error', reject);- });- req.on('error', reject);- req.end();- });-}--function wsEcho(proxyAddress: string, wsUrl: string): Promise<string> {- return new Promise((resolve, reject) => {- const ws = new WebSocket(wsUrl, { agent: new HttpsProxyAgent(proxyAddress), rejectUnauthorized: false } as any);- ws.on('open', () => ws.send('ping'));- ws.on('message', data => { resolve(data.toString()); ws.close(); });- ws.on('error', reject);- });-}--const response = (body: string, headers: [string, string][] = []): { status: number, statusText: string, headers: [string, string][], body: Buffer } =>- ({ status: 200, statusText: 'OK', headers, body: Buffer.from(body) });--type Fixtures = {- origin: Origin;- httpsOrigin: Origin;- cacheDir: () => string;- startProxy: (entry: CacheEntry, upstream?: ProxySettings) => Promise<string>;-};--const it = base.extend<Fixtures>({- origin: async ({}, use) => {- const origin = await startOrigin();- await use(origin);- await origin.close();- },- httpsOrigin: async ({}, use) => {- const origin = await startOrigin(generateSelfSignedCertificate());- await use(origin);- await origin.close();- },- cacheDir: async ({}, use, testInfo) => {- let index = 0;- await use(() => testInfo.outputPath('cache-' + (index++)));- },- startProxy: async ({}, use) => {- const started: CacheProxy[] = [];- await use(async (entry, upstream) => {- const proxy = new CacheProxy(entry, upstream);- const address = await proxy.start();- started.push(proxy);- return address;- });- await Promise.all(started.map(proxy => proxy.stop()));- },-});--// Convenience: a single-cache proxy over a fresh directory.-async function cachingProxy(startProxy: Fixtures['startProxy'], cacheDir: Fixtures['cacheDir'], match: CacheEntry['match'] = undefined, upstream?: ProxySettings) {- const dir = cacheDir();- const cache = new ResponseCache(dir);- const address = await startProxy({ cache, match }, upstream);- return { address, dir, cache };-}--it.describe('ResponseCache', () => {- it('key is deterministic and identity-sensitive', () => {- const a = ResponseCache.key('GET', 'http://x/a');- expect(ResponseCache.key('GET', 'http://x/a')).toBe(a);- expect(ResponseCache.key('GET', 'http://x/a', '')).toBe(a);- expect(ResponseCache.key('GET', 'http://x/b')).not.toBe(a);- expect(ResponseCache.key('GET', 'http://x/a', 'alice')).not.toBe(ResponseCache.key('GET', 'http://x/a', 'bob'));- });-- it('roundtrips status, headers and body', async ({ cacheDir }) => {- const cache = new ResponseCache(cacheDir());- await cache.load();- const key = ResponseCache.key('GET', 'http://x/a');- expect(await cache.get(key, {})).toBeUndefined();- await cache.set(key, 'http://x/a', {}, response('hello', [['content-type', 'text/plain']]));- const got = await cache.get(key, {});- expect(got!.status).toBe(200);- expect(got!.headers).toEqual([['content-type', 'text/plain']]);- expect(got!.body.toString()).toBe('hello');- });-- it('stores small bodies inline and large bodies as blobs', async ({ cacheDir }) => {- const dir = cacheDir();- const cache = new ResponseCache(dir);- await cache.load();- await cache.set(ResponseCache.key('GET', 'http://x/small'), 'http://x/small', {}, response('tiny'));- const big = Buffer.alloc(9000, 0x61);- await cache.set(ResponseCache.key('GET', 'http://x/big'), 'http://x/big', {}, { status: 200, statusText: 'OK', headers: [], body: big });- await cache.flush();- const index = fs.readFileSync(path.join(dir, 'index.jsonl'), 'utf8');- expect(index).toContain('"c":'); // inline- expect(index).toContain('"f":'); // blob reference- expect(fs.existsSync(path.join(dir, 'blobs'))).toBe(true);- expect((await cache.get(ResponseCache.key('GET', 'http://x/big'), {}))!.body.length).toBe(9000);- });-- it('persists across cache instances', async ({ cacheDir }) => {- const dir = cacheDir();- const key = ResponseCache.key('GET', 'http://x/a');- const first = new ResponseCache(dir);- await first.load();- await first.set(key, 'http://x/a', {}, response('persisted'));- await first.flush();- const second = new ResponseCache(dir);- await second.load();- expect((await second.get(key, {}))!.body.toString()).toBe('persisted');- });-- it('discards the cache on version mismatch', async ({ cacheDir }) => {- const dir = cacheDir();- const key = ResponseCache.key('GET', 'http://x/a');- const first = new ResponseCache(dir);- await first.load();- await first.set(key, 'http://x/a', {}, response('v'));- await first.flush();- fs.writeFileSync(path.join(dir, 'meta.json'), JSON.stringify({ version: 999 }));- const second = new ResponseCache(dir);- await second.load();- expect(await second.get(key, {})).toBeUndefined();- });-- it('load is a no-op without an index', async ({ cacheDir }) => {- const cache = new ResponseCache(cacheDir());- await cache.load();- expect(await cache.get(ResponseCache.key('GET', 'http://x/a'), {})).toBeUndefined();- });-- it('keys variants by their vary fields', async ({ cacheDir }) => {- const cache = new ResponseCache(cacheDir());- await cache.load();- const key = ResponseCache.key('GET', 'http://x/v');- await cache.set(key, 'http://x/v', { 'x-foo': 'a' }, response('A', [['vary', 'x-foo']]));- await cache.set(key, 'http://x/v', { 'x-foo': 'b' }, response('B', [['vary', 'x-foo']]));- expect((await cache.get(key, { 'x-foo': 'a' }))!.body.toString()).toBe('A');- expect((await cache.get(key, { 'x-foo': 'b' }))!.body.toString()).toBe('B');- expect(await cache.get(key, { 'x-foo': 'c' })).toBeUndefined();- });-- it('dedups identical variants', async ({ cacheDir }) => {- const dir = cacheDir();- const cache = new ResponseCache(dir);- await cache.load();- const key = ResponseCache.key('GET', 'http://x/a');- await cache.set(key, 'http://x/a', {}, response('one'));- await cache.set(key, 'http://x/a', {}, response('two'));- await cache.flush();- const lines = fs.readFileSync(path.join(dir, 'index.jsonl'), 'utf8').trim().split('\n');- expect(lines).toHaveLength(1);- expect((await cache.get(key, {}))!.body.toString()).toBe('one');- });-});--it.describe('default caching', () => {- it('caches a static subresource and replays it', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/img', (req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('pixels'); });- const { address, dir, cache } = await cachingProxy(startProxy, cacheDir);- const first = await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- expect(first.status).toBe(200);- expect(first.body).toBe('pixels');- expect(origin.hits('/img')).toBe(1);- const second = await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- expect(second.body).toBe('pixels');- expect(second.headers['content-type']).toBe('image/png');- expect(origin.hits('/img')).toBe(1);- await cache.flush();- expect(fs.existsSync(path.join(dir, 'index.jsonl'))).toBe(true);- });-- it('does not cache requests without fetch metadata', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/x', (req, res) => res.end('body'));- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/x'));- await drive(address, origin.url('/x'));- expect(origin.hits('/x')).toBe(2);- });-- it('does not cache xhr/fetch (empty destination)', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/api', (req, res) => res.end('data'));- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/api'), { 'sec-fetch-dest': 'empty' });- await drive(address, origin.url('/api'), { 'sec-fetch-dest': 'empty' });- expect(origin.hits('/api')).toBe(2);- });-- for (const directive of ['no-store', 'private']) {- it(`does not cache Cache-Control: ${directive}`, async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/r', (req, res) => { res.writeHead(200, { 'cache-control': directive }); res.end('x'); });- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/r')).toBe(2);- });- }-- it('does not cache responses with Set-Cookie', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/r', (req, res) => { res.writeHead(200, { 'set-cookie': 'sid=1' }); res.end('x'); });- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/r')).toBe(2);- });-- for (const vary of ['cookie', 'authorization', '*']) {- it(`does not cache Vary: ${vary}`, async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/r', (req, res) => { res.writeHead(200, { 'vary': vary }); res.end('x'); });- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/r')).toBe(2);- });- }-- it('keys cached variants by Vary', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/v', (req, res) => { res.writeHead(200, { 'vary': 'x-variant', 'sec-fetch-dest': 'image' }); res.end('variant:' + (req.headers['x-variant'] || '')); });- const { address } = await cachingProxy(startProxy, cacheDir);- expect((await drive(address, origin.url('/v'), { 'sec-fetch-dest': 'image', 'x-variant': 'a' })).body).toBe('variant:a');- expect((await drive(address, origin.url('/v'), { 'sec-fetch-dest': 'image', 'x-variant': 'b' })).body).toBe('variant:b');- expect(origin.hits('/v')).toBe(2);- expect((await drive(address, origin.url('/v'), { 'sec-fetch-dest': 'image', 'x-variant': 'a' })).body).toBe('variant:a');- expect(origin.hits('/v')).toBe(2);- });-- for (const status of [301, 308]) {- it(`caches ${status} redirects`, async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/old', (req, res) => { res.writeHead(status, { location: '/new' }); res.end(); });- const { address } = await cachingProxy(startProxy, cacheDir);- expect((await drive(address, origin.url('/old'))).status).toBe(status);- expect((await drive(address, origin.url('/old'))).status).toBe(status);- expect(origin.hits('/old')).toBe(1);- });- }-- it('does not cache 302 temporary redirects', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/old', (req, res) => { res.writeHead(302, { location: '/new' }); res.end(); });- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/old'));- await drive(address, origin.url('/old'));- expect(origin.hits('/old')).toBe(2);- });-- it('does not cache non-GET requests', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/p', (req, res) => res.end('x'));- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/p'), { 'sec-fetch-dest': 'image' }, 'POST');- await drive(address, origin.url('/p'), { 'sec-fetch-dest': 'image' }, 'POST');- expect(origin.hits('/p')).toBe(2);- });-- it('honors a request-side Cache-Control: no-store', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/r', (req, res) => { res.writeHead(200); res.end('x'); });- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image', 'cache-control': 'no-store' });- await drive(address, origin.url('/r'), { 'sec-fetch-dest': 'image', 'cache-control': 'no-store' });- expect(origin.hits('/r')).toBe(2);- });-});--it.describe('match callback', () => {- it('restricts caching to a URL glob', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/assets/a', (req, res) => res.end('a'));- origin.setRoute('/api/b', (req, res) => res.end('b'));- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: '**/assets/**' });- await drive(address, origin.url('/assets/a'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/assets/a'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/api/b'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/api/b'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/assets/a')).toBe(1);- expect(origin.hits('/api/b')).toBe(2);- });-- it("force-caches with disposition 'cache' regardless of metadata", async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/api', (req, res) => res.end('data'));- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: () => ({ disposition: 'cache' }) });- await drive(address, origin.url('/api'));- await drive(address, origin.url('/api'));- expect(origin.hits('/api')).toBe(1);- });-- it("bypasses the cache with disposition 'no-cache'", async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/img', (req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('x'); });- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: () => ({ disposition: 'no-cache' }) });- await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/img')).toBe(2);- });-- it('partitions the cache by identity', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/me', (req, res) => res.end('user:' + (req.headers['x-user'] || 'guest')));- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: request => ({ disposition: 'cache', identity: request.headers.get('x-user') }) });- expect((await drive(address, origin.url('/me'), { 'x-user': 'alice' })).body).toBe('user:alice');- expect((await drive(address, origin.url('/me'), { 'x-user': 'bob' })).body).toBe('user:bob');- expect(origin.hits('/me')).toBe(2);- // Alice's own entry, never bob's.- expect((await drive(address, origin.url('/me'), { 'x-user': 'alice' })).body).toBe('user:alice');- expect((await drive(address, origin.url('/me'))).body).toBe('user:guest');- expect(origin.hits('/me')).toBe(3);- });-- it('invokes the callback once per request with a WHATWG Request', async ({ origin, startProxy, cacheDir }) => {- const seen: { url: string, method: string, dest: string | null, host: string | null, cookie: string | null }[] = [];- origin.setRoute('/probe', (req, res) => res.end('x'));- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: request => {- expect(request).toBeInstanceOf(Request);- seen.push({- url: request.url,- method: request.method,- dest: request.headers.get('sec-fetch-dest'),- host: request.headers.get('host'),- cookie: request.headers.get('cookie'),- });- return {};- } });- await drive(address, origin.url('/probe'), { 'sec-fetch-dest': 'image', 'cookie': 'sid=1' });- expect(seen).toHaveLength(1);- expect(seen[0]).toEqual({- url: origin.url('/probe'),- method: 'GET',- dest: 'image',- host: `${HOST}:${origin.port}`,- cookie: 'sid=1', // Node's Request keeps "forbidden" headers.- });- });-- it('passes through a URL that does not match the glob', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/api/b', (req, res) => res.end('b'));- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: '**/assets/**' });- await drive(address, origin.url('/api/b'), { 'sec-fetch-dest': 'image' });- await drive(address, origin.url('/api/b'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/api/b')).toBe(2);- });-});--it.describe('streaming and coalescing', () => {- it('streams a non-storable response without buffering', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/stream', (req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.write('first-chunk'); });- const { address } = await cachingProxy(startProxy, cacheDir);- const { chunk } = await driveFirstChunk(address, origin.url('/stream'));- expect(chunk).toBe('first-chunk');- });-- it('coalesces concurrent identical misses into one upstream fetch', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/slow', (req, res) => setTimeout(() => res.end('slow'), 200));- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: () => ({ disposition: 'cache' }) });- const results = await Promise.all(Array.from({ length: 8 }, () => drive(address, origin.url('/slow'))));- expect(results.map(r => r.body)).toEqual(Array(8).fill('slow'));- expect(origin.hits('/slow')).toBe(1);- });-- it('does not coalesce non-storable responses', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/slow', (req, res) => setTimeout(() => res.end('slow'), 200));- const { address } = await cachingProxy(startProxy, cacheDir);- const results = await Promise.all(Array.from({ length: 5 }, () => drive(address, origin.url('/slow'))));- expect(results.map(r => r.body)).toEqual(Array(5).fill('slow'));- expect(origin.hits('/slow')).toBe(5);- });-});--it.describe('websockets', () => {- it('tunnels a plaintext ws:// connection', async ({ origin, startProxy, cacheDir }) => {- origin.onWebSocket(ws => ws.on('message', () => ws.send('pong')));- const { address } = await cachingProxy(startProxy, cacheDir);- expect(await wsEcho(address, origin.wsUrl('/ws'))).toBe('pong');- });-- it('tunnels a secure wss:// connection through MITM', async ({ httpsOrigin, startProxy, cacheDir }) => {- httpsOrigin.onWebSocket(ws => ws.on('message', () => ws.send('pong')));- const { address } = await cachingProxy(startProxy, cacheDir);- expect(await wsEcho(address, httpsOrigin.wsUrl('/ws'))).toBe('pong');- });-});--it.describe('https MITM', () => {- it('caches https responses via TLS termination', async ({ httpsOrigin, startProxy, cacheDir }) => {- httpsOrigin.setRoute('/img', (req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('secure'); });- const { address } = await cachingProxy(startProxy, cacheDir);- expect((await driveTls(address, httpsOrigin.url('/img'), { 'sec-fetch-dest': 'image' })).body).toBe('secure');- expect((await driveTls(address, httpsOrigin.url('/img'), { 'sec-fetch-dest': 'image' })).body).toBe('secure');- expect(httpsOrigin.hits('/img')).toBe(1);- });-});--it.describe('errors and edge cases', () => {- it('returns 502 when the upstream is unreachable', async ({ origin, startProxy, cacheDir }) => {- const url = origin.url('/gone');- await origin.close();- const { address } = await cachingProxy(startProxy, cacheDir);- expect((await drive(address, url, { 'sec-fetch-dest': 'image' })).status).toBe(502);- });-- it('survives a throwing match callback', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/ok', (req, res) => res.end('ok'));- let boom = true;- const dir = cacheDir();- const address = await startProxy({ cache: new ResponseCache(dir), match: () => {- if (boom)- throw new Error('boom');- return {};- } });- expect((await drive(address, origin.url('/ok'))).status).toBe(502);- boom = false;- expect((await drive(address, origin.url('/ok'))).body).toBe('ok');- });-- it('passes loopback traffic through without caching', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/img', (req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('x'); });- const { address, dir } = await cachingProxy(startProxy, cacheDir);- // Addressed over loopback (127.0.0.1) instead of the resolvable name.- expect((await drive(address, origin.loopbackUrl('/img'), { 'sec-fetch-dest': 'image' })).body).toBe('x');- await drive(address, origin.loopbackUrl('/img'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/img')).toBe(2);- expect(fs.existsSync(path.join(dir, 'index.jsonl'))).toBe(false);- });-- it('preserves response headers and body verbatim on replay', async ({ origin, startProxy, cacheDir }) => {- origin.setRoute('/img', (req, res) => {- res.writeHead(200, { 'content-type': 'image/svg+xml', 'x-custom': 'kept', 'content-encoding': 'identity' });- res.end('<svg/>');- });- const { address } = await cachingProxy(startProxy, cacheDir);- await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- const replayed = await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/img')).toBe(1);- expect(replayed.headers['content-type']).toBe('image/svg+xml');- expect(replayed.headers['x-custom']).toBe('kept');- expect(replayed.body).toBe('<svg/>');- });-});--it.describe('upstream proxy chaining', () => {- it('fetches misses through the configured upstream proxy', async ({ origin, startProxy, cacheDir }) => {- const tunnelled: string[] = [];- const upstream = http.createServer((req, res) => { res.writeHead(502); res.end(); });- upstream.on('connect', (req, socket, head) => {- tunnelled.push(req.url!);- const [host, port] = req.url!.split(':');- const target = net.connect({ host, port: Number(port), family: 4 }, () => {- socket.write('HTTP/1.1 200 Connection Established\r\n\r\n');- if (head.length)- target.write(head);- socket.pipe(target);- target.pipe(socket);- });- target.on('error', () => socket.destroy());- socket.on('error', () => target.destroy());- });- await new Promise<void>(resolve => upstream.listen(0, '127.0.0.1', resolve));- const upstreamUrl = `http://127.0.0.1:${(upstream.address() as net.AddressInfo).port}`;-- origin.setRoute('/img', (req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('x'); });- try {- const { address } = await cachingProxy(startProxy, cacheDir, undefined, { server: upstreamUrl });- expect((await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' })).body).toBe('x');- expect(tunnelled.length).toBeGreaterThan(0);- await drive(address, origin.url('/img'), { 'sec-fetch-dest': 'image' });- expect(origin.hits('/img')).toBe(1); // second is a cache hit, no new tunnel- } finally {- await new Promise<void>(resolve => upstream.close(() => resolve()));- }- });-});diff --git a/tests/playwright-test/cache-proxy.spec.ts b/tests/playwright-test/cache-proxy.spec.tsdeleted file mode 100644index b6b6f801c7..0000000000--- a/tests/playwright-test/cache-proxy.spec.ts+++ /dev/null@@ -1,536 +0,0 @@-/**- * Copyright (c) Microsoft Corporation.- *- * Licensed under the Apache License, Version 2.0 (the "License");- * you may not use this file except in compliance with the License.- * You may obtain a copy of the License at- *- * http://www.apache.org/licenses/LICENSE-2.0- *- * Unless required by applicable law or agreed to in writing, software- * distributed under the License is distributed on an "AS IS" BASIS,- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- * See the License for the specific language governing permissions and- * limitations under the License.- */--import { test, expect } from './playwright-test-fixtures';-import fs from 'fs';-import http from 'http';-import net from 'net';-import path from 'path';--async function startUpstreamProxy(): Promise<{ url: string, log: string[], stop: () => Promise<void> }> {- const log: string[] = [];- const server = http.createServer((req, res) => {- res.writeHead(502);- res.end();- });- server.on('connect', (req, socket, head) => {- log.push(req.url!);- const [host, port] = req.url!.split(':');- // 'localhost' resolves to ::1 first, but test servers listen on IPv4.- const target = net.connect({ port: Number(port), host, family: 4 }, () => {- socket.write('HTTP/1.1 200 Connection Established\r\n\r\n');- if (head && head.length)- target.write(head);- socket.pipe(target);- target.pipe(socket);- });- target.on('error', () => socket.destroy());- socket.on('error', () => target.destroy());- });- await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve));- const url = `http://127.0.0.1:${(server.address() as net.AddressInfo).port}`;- return { url, log, stop: () => new Promise(resolve => server.close(() => resolve())) };-}--const loadImage = `src => new Promise(f => { const img = document.createElement('img'); img.onload = img.onerror = f; img.src = src; })`;-const loadScript = `src => new Promise(f => { const s = document.createElement('script'); s.onload = s.onerror = f; s.src = src; document.head.appendChild(s); })`;--// httpCache bypasses loopback, so tests reach the local server through a-// non-loopback name that still resolves to 127.0.0.1 - forcing traffic through-// the caching proxy the way a remote (staging) host would.-const HOST = 'fake-localhost-127-0-0-1.nip.io';-const remote = (server: { PORT: number }, tls = false) => `${tls ? 'https' : 'http'}://${HOST}:${server.PORT}`;-const remoteHost = (server: { PORT: number }) => `${HOST}:${server.PORT}`;--test('should record and replay responses across runs', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- httpsServer.setRoute('/asset.png', (req, res) => {- ++hits;- res.writeHead(200, { 'content-type': 'image/png' });- res.end('payload');- });-- const files = {- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/asset.png');- });- `,- };-- const result1 = await runInlineTest(files, { workers: 1 });- expect(result1.exitCode).toBe(0);- expect(hits).toBe(1);- expect(fs.existsSync(path.join(cacheDir, 'index.jsonl'))).toBe(true);- expect(JSON.parse(fs.readFileSync(path.join(cacheDir, 'meta.json'), 'utf8')).version).toBe(1);-- const hitsAfterRecord = hits;- const result2 = await runInlineTest(files, { workers: 1 });- expect(result2.exitCode).toBe(0);- expect(hits).toBe(hitsAfterRecord); // Served entirely from cache, no new httpsServer hit.-- fs.writeFileSync(path.join(cacheDir, 'meta.json'), JSON.stringify({ version: 999 }));- const result3 = await runInlineTest(files, { workers: 1 });- expect(result3.exitCode).toBe(0);- expect(hits).toBe(hitsAfterRecord + 1); // Went back to the httpsServer.-});--test('should cache https responses via MITM', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- httpsServer.setRoute('/secure.png', (req, res) => {- ++hits;- res.writeHead(200, { 'content-type': 'image/png' });- res.end('secure payload');- });-- const files = {- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/secure.png');- });- `,- };-- const result1 = await runInlineTest(files, { workers: 1 });- expect(result1.exitCode).toBe(0);- expect(hits).toBe(1); // Proxy terminated TLS, fetched once, cached.-- const hitsAfterRecord = hits;- const result2 = await runInlineTest(files, { workers: 1 });- expect(result2.exitCode).toBe(0);- expect(hits).toBe(hitsAfterRecord); // Replayed from cache over HTTPS, no new hit.-});--test('should tunnel secure WebSockets through the MITM proxy', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- httpsServer.onceWebSocketConnection(ws => ws.on('message', () => ws.send('pong')));-- const result = await runInlineTest({- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('wss', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- const echoed = await page.evaluate(host => new Promise(resolve => {- const ws = new WebSocket('wss://' + host + '/ws');- ws.onopen = () => ws.send('ping');- ws.onmessage = e => resolve(e.data);- ws.onerror = () => resolve('error');- }), '${remoteHost(httpsServer)}');- expect(echoed).toBe('pong');- });- `,- }, { workers: 1 });-- expect(result.exitCode).toBe(0);- expect(result.passed).toBe(1);-});--test('should tunnel plaintext WebSockets', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- server.onceWebSocketConnection(ws => ws.on('message', () => ws.send('pong')));-- const result = await runInlineTest({- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('ws', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- const echoed = await page.evaluate(host => new Promise(resolve => {- const ws = new WebSocket('ws://' + host + '/ws');- ws.onopen = () => ws.send('ping');- ws.onmessage = e => resolve(e.data);- ws.onerror = () => resolve('error');- }), '${remoteHost(server)}');- expect(echoed).toBe('pong');- });- `,- }, { workers: 1 });-- expect(result.exitCode).toBe(0);- expect(result.passed).toBe(1);-});--test('should stream non-storable responses without buffering', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- // A response that emits a chunk and then never ends; buffering would hang.- server.setRoute('/stream', (req, res) => {- res.writeHead(200, { 'content-type': 'text/plain' });- res.write('first-chunk');- });-- const result = await runInlineTest({- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('stream', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- const chunk = await page.evaluate(async () => {- const res = await fetch('/stream');- const reader = res.body.getReader();- const { value } = await reader.read();- await reader.cancel();- return new TextDecoder().decode(value);- });- expect(chunk).toBe('first-chunk');- });- `,- }, { workers: 1, timeout: 15000 });-- expect(result.exitCode).toBe(0);- expect(result.passed).toBe(1);-});--test('should cache only shared static assets by default', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- const counts: Record<string, number> = { asset: 0, xhr: 0, private: 0, cookie: 0, noStore: 0 };- httpsServer.setRoute('/asset', (req, res) => { ++counts.asset; res.end('a'); });- httpsServer.setRoute('/api/data', (req, res) => { ++counts.xhr; res.end('data'); });- httpsServer.setRoute('/private', (req, res) => { ++counts.private; res.writeHead(200, { 'cache-control': 'private' }); res.end('a'); });- httpsServer.setRoute('/cookie', (req, res) => { ++counts.cookie; res.writeHead(200, { 'set-cookie': 'sid=1' }); res.end('a'); });- httpsServer.setRoute('/no-store', (req, res) => { ++counts.noStore; res.writeHead(200, { 'cache-control': 'no-store' }); res.end('a'); });-- const files = {- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- // Loaded as image subresources (Sec-Fetch-Dest: image).- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/asset');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/private');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/cookie');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/no-store');- // Loaded as an XHR/fetch (Sec-Fetch-Dest: empty).- await page.evaluate(() => fetch('/api/data').then(r => r.text()));- });- `,- };-- await runInlineTest(files, { workers: 1 });- await runInlineTest(files, { workers: 1 });- expect(counts.asset).toBe(1); // Static subresource -> cached, replayed.- expect(counts.xhr).toBe(2); // fetch/XHR -> not cached by default.- expect(counts.private).toBe(2); // Cache-Control: private -> personalized, not cached.- expect(counts.cookie).toBe(2); // Set-Cookie -> personalized, not cached.- expect(counts.noStore).toBe(2); // Cache-Control: no-store -> never cached.-});--test('should isolate cached entries by identity', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- server.setRoute('/me', (req, res) => { ++hits; res.end('user:' + (req.headers['x-user'] || 'guest')); });-- const files = {- 'playwright.config.ts': `export default { httpCache: {- dir: ${JSON.stringify(cacheDir)},- match: request => ({ disposition: 'cache', identity: request.headers.get('x-user') }),- } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- const fetchAs = user => page.evaluate(u =>- fetch('/me', { headers: { 'x-user': u } }).then(r => r.text()), user);- expect(await fetchAs('alice')).toBe('user:alice');- expect(await fetchAs('bob')).toBe('user:bob');- expect(await fetchAs('alice')).toBe('user:alice'); // Alice's own entry, not bob's.- expect(await fetchAs('')).toBe('user:guest'); // Guest identity -> its own namespace.- });- `,- };-- await runInlineTest(files, { workers: 1 });- expect(hits).toBe(3); // alice, bob, guest each fetched once; the repeat 'alice' was a hit.-- await runInlineTest(files, { workers: 1 });- expect(hits).toBe(3); // All three identities replayed from cache.-});--test('should key cached variants by Vary', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- server.setRoute('/variant', (req, res) => {- ++hits;- res.writeHead(200, { 'vary': 'x-variant' });- res.end('variant:' + (req.headers['x-variant'] || 'none'));- });-- const files = {- // Force-cache so the fetch is stored; this isolates the Vary keying.- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)}, match: () => ({ disposition: 'cache' }) } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- const fetchVariant = v => page.evaluate(variant =>- fetch('/variant', { headers: { 'x-variant': variant } }).then(r => r.text()), v);- expect(await fetchVariant('a')).toBe('variant:a');- expect(await fetchVariant('b')).toBe('variant:b');- expect(await fetchVariant('a')).toBe('variant:a'); // From cache, still the 'a' body.- });- `,- };-- await runInlineTest(files, { workers: 1 });- expect(hits).toBe(2); // Two distinct variants fetched once each; the repeat 'a' was a hit.-- await runInlineTest(files, { workers: 1 });- expect(hits).toBe(2); // Both variants replayed from cache.-});--test('should not cache Vary: * responses', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- server.setRoute('/star', (req, res) => {- ++hits;- res.writeHead(200, { 'vary': '*' });- res.end('a');- });-- const files = {- // Even when caching is forced, Vary: * is hard-blocked and never stored.- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)}, match: () => ({ disposition: 'cache' }) } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- await page.evaluate(() => fetch('/star').then(r => r.text()));- });- `,- };-- await runInlineTest(files, { workers: 1 });- await runInlineTest(files, { workers: 1 });- expect(hits).toBe(2); // Vary: * is not storable -> network every run.-});--test('should cache redirects', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let redirects = 0;- server.setRoute('/old', (req, res) => {- ++redirects;- res.writeHead(301, { 'location': '/new' });- res.end();- });- server.setRoute('/new', (req, res) => { res.end('arrived'); });-- const files = {- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- expect(await page.evaluate(() => fetch('/old').then(r => r.text()))).toBe('arrived');- });- `,- };-- await runInlineTest(files, { workers: 1 });- expect(redirects).toBe(1);-- await runInlineTest(files, { workers: 1 });- expect(redirects).toBe(1); // 301 replayed from cache.-});--test('should not cache loopback traffic', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- server.setRoute('/asset.png', (req, res) => {- ++hits;- res.writeHead(200, { 'content-type': 'image/png' });- res.end('payload');- });-- const files = {- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)} } };`,- // Reached over loopback (localhost), which the cache proxy bypasses.- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${server.EMPTY_PAGE}');- await page.evaluate(${loadImage}, '${server.PREFIX}/asset.png');- });- `,- };-- await runInlineTest(files, { workers: 1 });- await runInlineTest(files, { workers: 1 });- expect(hits).toBe(2); // Loopback is not proxied -> never cached.- expect(fs.existsSync(path.join(cacheDir, 'index.jsonl'))).toBe(false);-});--test('should respect the match callback disposition', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let pinned = 0;- let volatileScript = 0;- let normalScript = 0;- httpsServer.setRoute('/api/pinned', (req, res) => { ++pinned; res.end('pinned'); });- httpsServer.setRoute('/volatile.js', (req, res) => {- ++volatileScript;- res.writeHead(200, { 'content-type': 'application/javascript' });- res.end(';');- });- httpsServer.setRoute('/normal.js', (req, res) => {- ++normalScript;- res.writeHead(200, { 'content-type': 'application/javascript' });- res.end(';');- });-- const files = {- 'playwright.config.ts': `export default { httpCache: {- dir: ${JSON.stringify(cacheDir)},- match: request => {- if (request.url.includes('/api/pinned'))- return { disposition: 'cache' };- if (request.url.includes('/volatile'))- return { disposition: 'no-cache' };- return {};- },- } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- await page.evaluate(() => fetch('/api/pinned').then(r => r.text()));- await page.evaluate(${loadScript}, '${remote(httpsServer, true)}/volatile.js');- await page.evaluate(${loadScript}, '${remote(httpsServer, true)}/normal.js');- });- `,- };-- await runInlineTest(files, { workers: 1 });- await runInlineTest(files, { workers: 1 });- expect(pinned).toBe(1); // XHR force-cached by the policy.- expect(volatileScript).toBe(2); // Static resource forced to the network.- expect(normalScript).toBe(1); // Default behavior - static, cached.-});--test('should coalesce concurrent identical requests into one upstream fetch', async ({ runInlineTest, server }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let hits = 0;- server.setRoute('/slow', (req, res) => {- ++hits;- setTimeout(() => {- res.writeHead(200, { 'content-type': 'text/plain' });- res.end('slow payload');- }, 300);- });-- const result = await runInlineTest({- // Force-cache everything so that concurrent fetch() misses coalesce.- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)}, match: () => ({ disposition: 'cache' }) } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('parallel', async ({ page }) => {- await page.goto('${remote(server)}/empty.html');- const bodies = await page.evaluate(() => Promise.all(- Array.from({ length: 8 }, () => fetch('${remote(server)}/slow').then(r => r.text()))));- expect(bodies).toEqual(Array(8).fill('slow payload'));- });- `,- }, { workers: 1 });-- expect(result.exitCode).toBe(0);- expect(hits).toBe(1); // 8 concurrent misses coalesced into a single upstream fetch.-});--test('should only cache requests matching the filter', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- let assets = 0;- let api = 0;- httpsServer.setRoute('/assets/logo.png', (req, res) => { ++assets; res.end('logo'); });- httpsServer.setRoute('/api/data.png', (req, res) => { ++api; res.end('data'); });-- const files = {- 'playwright.config.ts': `export default { httpCache: { dir: ${JSON.stringify(cacheDir)}, match: '**/assets/**' } };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/assets/logo.png');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/api/data.png');- });- `,- };-- await runInlineTest(files, { workers: 1 });- expect(assets).toBe(1);- expect(api).toBe(1);-- await runInlineTest(files, { workers: 1 });- expect(assets).toBe(1); // Matched the filter -> served from cache.- expect(api).toBe(2); // Not matched -> always hits the network.-});--test('should chain to the configured proxy', async ({ runInlineTest, httpsServer }, testInfo) => {- const cacheDir = testInfo.outputPath('.network-cache');- const upstreamProxy = await startUpstreamProxy();- let hits = 0;- httpsServer.setRoute('/asset.png', (req, res) => {- ++hits;- res.writeHead(200, { 'content-type': 'image/png' });- res.end('payload');- });-- const files = {- 'playwright.config.ts': `export default {- httpCache: { dir: ${JSON.stringify(cacheDir)}, proxy: { server: ${JSON.stringify(upstreamProxy.url)} } },- };`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('load', async ({ page }) => {- await page.goto('${remote(httpsServer, true)}/empty.html');- await page.evaluate(${loadImage}, '${remote(httpsServer, true)}/asset.png');- });- `,- };-- try {- const result1 = await runInlineTest(files, { workers: 1 });- expect(result1.exitCode).toBe(0);- expect(hits).toBe(1);- expect(upstreamProxy.log.length).toBeGreaterThan(0); // Cache misses went through the configured proxy.-- const result2 = await runInlineTest(files, { workers: 1 });- expect(result2.exitCode).toBe(0);- expect(hits).toBe(1); // Still served from cache when chaining through a proxy.- } finally {- await upstreamProxy.stop();- }-});--test('should not start the proxy without httpCache', async ({ runInlineTest }) => {- const result = await runInlineTest({- 'playwright.config.ts': `export default {};`,- 'a.test.ts': `- import { test, expect } from '@playwright/test';- test('no proxy', async ({ page }) => {- expect(process.env.PLAYWRIGHT_TEST_CACHE_PROXY).toBeUndefined();- });- `,- }, { workers: 1 });- expect(result.exitCode).toBe(0);- expect(result.passed).toBe(1);-});diff --git a/utils/generate_types/overrides-test.d.ts b/utils/generate_types/overrides-test.d.tsindex ccb62d8e7b..713979b3bf 100644--- a/utils/generate_types/overrides-test.d.ts+++ b/utils/generate_types/overrides-test.d.ts@@ -74,32 +74,6 @@ export interface Config<TestArgs = {}, WorkerArgs = {}> extends TestConfig<TestAexport type Metadata = { [key: string]: any };-/**- * A per-request caching decision. Every field is optional; an empty object applies the- * default behavior.- */-export type HttpCacheDecision = {- /**- * `'cache'` serves a stored response and force-stores on a miss, `'no-cache'` bypasses the cache (fetch fresh, do- * not store), and `'default'` applies the default rules (cache shared static assets only). Defaults to `'default'`.- */- disposition?: 'cache' | 'no-cache' | 'default';-- /**- * A stable principal id, such as a session token, that partitions the cache. Entries recorded under one identity- * are never served to a request with a different one, so per-user content can be cached without leaking across- * contexts. The value is hashed into the cache key and never written to disk. `null` is treated as anonymous, so- * `request.headers.get()` results can be passed directly.- */- identity?: string | null;-};--/**- * Called for each request to decide how it is cached. Receives a standard- * [Request](https://developer.mozilla.org/en-US/docs/Web/API/Request). See [`property: TestConfig.httpCache`].- */-export type HttpCachePolicy = (request: Request) => HttpCacheDecision;-export interface FullConfig<TestArgs = {}, WorkerArgs = {}> {projects: FullProject<TestArgs, WorkerArgs>[];reporter: ReporterDescription[];