Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
refactor!: remove `tieredProxyUrls` support (#3599)
Jindřich Bär99b751f1f81d
diff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy_management.mdxindex 70dce2f72..bc7253aa6 100644--- a/docs/guides/proxy_management.mdx+++ b/docs/guides/proxy_management.mdx@@ -83,7 +83,7 @@ The `ProxyConfiguration` class allows you to provide a custom function to pick a```javascriptconst proxyConfiguration = new ProxyConfiguration({- newUrlFunction: (sessionId, { request }) => {+ newUrlFunction: ({ request } = {}) => {if (request?.url.includes('crawlee.dev')) {return null; // for crawlee.dev, we don't use a proxy}@@ -93,39 +93,10 @@ const proxyConfiguration = new ProxyConfiguration({});```-The `newUrlFunction` receives two parameters - `sessionId` and `options` - and returns a string containing the proxy URL.--The `sessionId` parameter is always provided and allows us to differentiate between different sessions - e.g. when Crawlee recognizes your crawlers are being blocked, it will automatically create a new session with a different id.+The `newUrlFunction` receives a single optional `options` parameter and returns a string with the proxy URL (or `null` to skip the proxy for the current request).The `options` parameter is an object containing a <ApiLink to="core/class/Request">`Request`</ApiLink>, which is the request that will be made. Note that this object is not always available, for example when we are using the `newUrl` function directly. Your custom function should therefore not rely on the `request` object being present and provide a default behavior when it is not.-### Tiered proxies--You can also provide a list of proxy tiers to the `ProxyConfiguration` class. This is useful when you want to switch between different proxies automatically based on the blocking behavior of the website.--:::warning--Note that the `tieredProxyUrls` option requires `ProxyConfiguration` to be used from a crawler instance ([see below](#crawler-integration)).--Using this configuration through the `newUrl` calls will not yield the expected results.--:::--```javascript-const proxyConfiguration = new ProxyConfiguration({- tieredProxyUrls: [- [null], // At first, we try to connect without a proxy- ['http://okay-proxy.com'],- ['http://slightly-better-proxy.com', 'http://slightly-better-proxy-2.com'],- ['http://very-good-and-expensive-proxy.com'],- ]-});-```--This configuration will start with no proxy, then switch to `http://okay-proxy.com` if Crawlee recognizes we're getting blocked by the target website. If that proxy is also blocked, we will switch to one of the `slightly-better-proxy` URLs. If those are blocked, we will switch to the `very-good-and-expensive-proxy.com` URL.--Crawlee also periodically probes lower tier proxies to see if they are unblocked, and if they are, it will switch back to them.-## Crawler integration`ProxyConfiguration` integrates seamlessly into <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>, <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink>, <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex a5ff92026..be076fda4 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -127,6 +127,46 @@ const crawler = new BasicCrawler({});```+## `tieredProxyUrls` is removed from `ProxyConfiguration`++The `tieredProxyUrls` option has been removed, together with the `proxyTier` field on `ProxyInfo` and the `proxyTier` plumbing in `BrowserPool`. In v4 the `Session` is the main rotation unit - a session already carries its own proxy, cookies and error score, so the pool rotates the whole fingerprint when a session gets retired on a block.++If you used tiers to escalate from a cheap proxy pool to a pricier one on blocks, you can achieve the same behavior by pre-populating a `SessionPool` with named sessions - one per proxy tier - and flipping `request.sessionId` in an `errorHandler` to reassign the retry to the next tier. Skip the `proxyConfiguration` option on the crawler - the session already carries its own proxy.++```typescript+import { BasicCrawler, SessionPool } from '@crawlee/core';++const proxyInfoFromUrl = (proxyUrl: string) => {+ const { username, password, hostname, port } = new URL(proxyUrl);+ return {+ url: proxyUrl,+ username: decodeURIComponent(username),+ password: decodeURIComponent(password),+ hostname,+ port,+ };+};++const sessionPool = new SessionPool();+await sessionPool.addSession({ id: 'basic', proxyInfo: proxyInfoFromUrl('http://cheap-proxy.com') });+await sessionPool.addSession({ id: 'premium', proxyInfo: proxyInfoFromUrl('http://expensive-proxy.com') });++const crawler = new BasicCrawler({+ sessionPool,+ retryOnBlocked: true,+ requestHandler: async ({ request, sendRequest }) => {+ await sendRequest({ url: request.url });+ },+ errorHandler: async ({ request }) => {+ request.sessionId = 'premium';+ },+});++await crawler.run([{ url: 'https://example.com', sessionId: 'basic' }]);+```++More complex routing (more tiers, weighted draws, sticky assignment, cooldowns) can be expressed with additional named sessions and custom `errorHandler` logic.+## Remove `experimentalContainers` optionThis experimental option relied on an outdated manifest version for browser extensions, it is not possible to achieve this with the currently supported versions.diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.tsindex daf45afc7..f28dda752 100644--- a/packages/browser-crawler/src/internals/browser-crawler.ts+++ b/packages/browser-crawler/src/internals/browser-crawler.ts@@ -474,7 +474,6 @@ export abstract class BrowserCrawler<const proxyInfo = crawlingContext.session.proxyInfo;newPageOptions.proxyUrl = proxyInfo?.url;- newPageOptions.proxyTier = proxyInfo?.proxyTier;newPageOptions.ignoreTlsErrors = proxyInfo?.ignoreTlsErrors;}diff --git a/packages/browser-pool/src/abstract-classes/browser-controller.ts b/packages/browser-pool/src/abstract-classes/browser-controller.tsindex 3f88c23f0..cb9435085 100644--- a/packages/browser-pool/src/abstract-classes/browser-controller.ts+++ b/packages/browser-pool/src/abstract-classes/browser-controller.ts@@ -58,13 +58,7 @@ export abstract class BrowserController<launchContext: LaunchContext<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult> = undefined!;/**- * The proxy tier tied to this browser controller.- * `undefined` if no tiered proxy is used.- */- proxyTier?: number;-- /**- * The proxy URL used by the browser controller. This is set every time the browser controller uses proxy (even the tiered one).+ * The proxy URL used by the browser controller.* `undefined` if no proxy is used*/proxyUrl?: string;diff --git a/packages/browser-pool/src/abstract-classes/browser-plugin.ts b/packages/browser-pool/src/abstract-classes/browser-plugin.tsindex 2addad9b9..f24027e0c 100644--- a/packages/browser-pool/src/abstract-classes/browser-plugin.ts+++ b/packages/browser-pool/src/abstract-classes/browser-plugin.ts@@ -155,7 +155,6 @@ export abstract class BrowserPlugin<userDataDir = this.userDataDir,browserPerProxy = this.browserPerProxy,ignoreProxyCertificate = this.ignoreProxyCertificate,- proxyTier,} = options;return new LaunchContext({@@ -167,7 +166,6 @@ export abstract class BrowserPlugin<userDataDir,browserPerProxy,ignoreProxyCertificate,- proxyTier,});}diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.tsindex bee488f7b..be9572b72 100644--- a/packages/browser-pool/src/browser-pool.ts+++ b/packages/browser-pool/src/browser-pool.ts@@ -1,6 +1,6 @@import { AsyncResource } from 'node:async_hooks';-import { type CrawleeLogger, serviceLocator, type TieredProxy } from '@crawlee/core';+import { type CrawleeLogger, serviceLocator } from '@crawlee/core';import type { BrowserFingerprintWithHeaders } from 'fingerprint-generator';import { FingerprintGenerator } from 'fingerprint-generator';import { FingerprintInjector } from 'fingerprint-injector';@@ -443,7 +443,6 @@ export class BrowserPool<pageOptions,browserPlugin = this._pickBrowserPlugin(),proxyUrl,- proxyTier,ignoreTlsErrors,} = options;@@ -465,12 +464,11 @@ export class BrowserPool<// Limiter is necessary - https://github.com/apify/crawlee/issues/1126return this.limiter(AsyncResource.bind(async () => {- let browserController = this._pickBrowserWithFreeCapacity(browserPlugin, { proxyTier, proxyUrl });+ let browserController = this._pickBrowserWithFreeCapacity(browserPlugin, { proxyUrl });if (!browserController)browserController = await this._launchBrowser(id, {browserPlugin,- proxyTier,proxyUrl,ignoreTlsErrors,});@@ -715,7 +713,7 @@ export class BrowserPool<}private async _launchBrowser(pageId: string, options: InternalLaunchBrowserOptions<BrowserPlugins[number]>) {- const { browserPlugin, launchOptions, proxyTier, proxyUrl, ignoreTlsErrors } = options;+ const { browserPlugin, launchOptions, proxyUrl, ignoreTlsErrors } = options;const browserController = browserPlugin.createController() as BrowserControllerReturn;this.startingBrowserControllers.add(browserController);@@ -723,7 +721,6 @@ export class BrowserPool<const launchContext = browserPlugin.createLaunchContext({id: pageId,launchOptions,- proxyTier,proxyUrl,});@@ -751,7 +748,6 @@ export class BrowserPool<}this.log.debug('Launched new browser.', { id: browserController.id });- browserController.proxyTier = proxyTier;browserController.proxyUrl = proxyUrl;try {@@ -788,20 +784,18 @@ export class BrowserPool<return this.browserPlugins[pluginIndex];}- private _pickBrowserWithFreeCapacity(browserPlugin: BrowserPlugin, options?: Partial<TieredProxy>) {+ private _pickBrowserWithFreeCapacity(browserPlugin: BrowserPlugin, options?: { proxyUrl?: string }) {return [...this.activeBrowserControllers].find((controller) => {const hasCapacity = controller.activePages < this.maxOpenPagesPerBrowser;const isCorrectPlugin = controller.browserPlugin === browserPlugin;const isSameProxyUrl = controller.proxyUrl === options?.proxyUrl;- const isCorrectProxyTier = controller.proxyTier === options?.proxyTier;return (isCorrectPlugin &&hasCapacity &&- ((!controller.launchContext.browserPerProxy && !options?.proxyTier) ||- (options?.proxyTier && isCorrectProxyTier) ||+ (!controller.launchContext.browserPerProxy ||(options?.proxyUrl && isSameProxyUrl) ||- (!options?.proxyUrl && !options?.proxyTier && !controller.proxyUrl && !controller.proxyTier))+ (!options?.proxyUrl && !controller.proxyUrl)));});}@@ -919,10 +913,6 @@ export interface BrowserPoolNewPageOptions<PageOptions, BP extends BrowserPlugin* Proxy URL.*/proxyUrl?: string;- /**- * Proxy tier.- */- proxyTier?: number;/*** Disable TLS certificate verification for MITM proxies.* Applied both when launching a new browser and when creating a page in an existing one.@@ -963,7 +953,6 @@ export interface BrowserPoolNewPageInNewBrowserOptions<PageOptions, BP extends Binterface InternalLaunchBrowserOptions<BP extends BrowserPlugin> {browserPlugin: BP;launchOptions?: BP['launchOptions'];- proxyTier?: number;proxyUrl?: string;ignoreTlsErrors?: boolean;}diff --git a/packages/browser-pool/src/launch-context.ts b/packages/browser-pool/src/launch-context.tsindex 9ae4f3343..8248da3e9 100644--- a/packages/browser-pool/src/launch-context.ts+++ b/packages/browser-pool/src/launch-context.ts@@ -51,7 +51,6 @@ export interface LaunchContextOptions<*/userDataDir?: string;proxyUrl?: string;- proxyTier?: number;/*** If set to `true`, TLS certificate errors from the upstream proxy will be ignored.* This is useful when using HTTPS proxies with self-signed certificates.@@ -72,7 +71,6 @@ export class LaunchContext<useIncognitoPages: boolean;browserPerProxy?: boolean;userDataDir: string;- proxyTier?: number;ignoreProxyCertificate?: boolean;private _proxyUrl?: string;@@ -90,7 +88,6 @@ export class LaunchContext<useIncognitoPages,browserPerProxy,userDataDir = '',- proxyTier,ignoreProxyCertificate,} = options;@@ -100,7 +97,6 @@ export class LaunchContext<this.browserPerProxy = browserPerProxy ?? false;this.useIncognitoPages = useIncognitoPages ?? false;this.userDataDir = userDataDir;- this.proxyTier = proxyTier;this.ignoreProxyCertificate = ignoreProxyCertificate ?? false;this._proxyUrl = proxyUrl;diff --git a/packages/core/src/proxy_configuration.ts b/packages/core/src/proxy_configuration.tsindex 154b59404..2c6f65078 100644--- a/packages/core/src/proxy_configuration.ts+++ b/packages/core/src/proxy_configuration.ts@@ -25,82 +25,10 @@ export interface ProxyConfigurationOptions {* This function is used to generate the URL when {@apilink ProxyConfiguration.newUrl} or {@apilink ProxyConfiguration.newProxyInfo} is called.*/newUrlFunction?: ProxyConfigurationFunction;-- /**- * An array of custom proxy URLs to be rotated stratified in tiers.- * This is a more advanced version of `proxyUrls` that allows you to define a hierarchy of proxy URLs- * If everything goes well, all the requests will be sent through the first proxy URL in the list.- * Whenever the crawler encounters a problem with the current proxy on the given domain, it will switch to the higher tier for this domain.- * The crawler probes lower-level proxies at intervals to check if it can make the tier downshift.- *- * This feature is useful when you have a set of proxies with different performance characteristics (speed, price, antibot performance etc.) and you want to use the best one for each domain.- *- * Use `null` as a proxy URL to disable the proxy for the given tier.- */- tieredProxyUrls?: UrlList[];}-export interface TieredProxy {- proxyUrl: string | null;- proxyTier?: number;-}--interface TieredProxyOptions {+interface NewUrlOptions {request?: Request;- proxyTier?: number;-}--/**- * Internal class for tracking the proxy tier history for a specific domain.- *- * Predicts the best proxy tier for the next request based on the error history for different proxy tiers.- */-class ProxyTierTracker {- private histogram: number[];- private currentTier: number;-- constructor(tieredProxyUrls: (string | null)[][]) {- this.histogram = tieredProxyUrls.map(() => 0);- this.currentTier = 0;- }-- /**- * Processes a single step of the algorithm and updates the current tier prediction based on the error history.- */- private processStep(): void {- this.histogram.forEach((x, i) => {- if (this.currentTier === i) return;- if (x > 0) this.histogram[i]--;- });-- const left = this.currentTier > 0 ? this.histogram[this.currentTier - 1] : Infinity;- const right = this.currentTier < this.histogram.length - 1 ? this.histogram[this.currentTier + 1] : Infinity;-- if (this.histogram[this.currentTier] > Math.min(left, right)) {- this.currentTier = left <= right ? this.currentTier - 1 : this.currentTier + 1;- } else if (this.histogram[this.currentTier] === left) {- this.currentTier--;- }- }-- /**- * Increases the error score for the given proxy tier. This raises the chance of picking a different proxy tier for the subsequent requests.- *- * The error score is increased by 10 for the given tier. This means that this tier will be disadvantaged for the next 10 requests (every new request prediction decreases the error score by 1).- * @param tier The proxy tier to mark as problematic.- */- addError(tier: number) {- this.histogram[tier] += 10;- }-- /**- * Returns the best proxy tier for the next request based on the error history for different proxy tiers.- * @returns The proxy tier prediction- */- predictTier() {- this.processStep();- return this.currentTier;- }}/**@@ -135,11 +63,9 @@ export class ProxyConfiguration {isManInTheMiddle = false;protected nextCustomUrlIndex = 0;protected proxyUrls?: UrlList;- protected tieredProxyUrls?: UrlList[];protected usedProxyUrls = new Map<string, string | null>();protected newUrlFunction?: ProxyConfigurationFunction;protected log = serviceLocator.getLogger().child({ prefix: 'ProxyConfiguration' });- protected domainTiers = new Map<string, ProxyTierTracker>();/*** Creates a {@apilink ProxyConfiguration} instance based on the provided options. Proxy servers are used to prevent target websites from@@ -163,26 +89,29 @@ export class ProxyConfiguration {*/constructor(options: ProxyConfigurationOptions = {}) {const { validateRequired, ...rest } = options as Dictionary;++ if ('tieredProxyUrls' in rest) {+ throw new Error(+ 'The `tieredProxyUrls` option has been removed in Crawlee v4. ' ++ 'See the v4 upgrading guide for the recommended migration to named sessions.',+ );+ }+ow(rest,ow.object.exactShape({proxyUrls: ow.optional.array.nonEmpty.ofType(ow.any(ow.string.url, ow.null)),newUrlFunction: ow.optional.function,- tieredProxyUrls: ow.optional.array.nonEmpty.ofType(- ow.array.nonEmpty.ofType(ow.any(ow.string.url, ow.null)),- ),}),);- const { proxyUrls, newUrlFunction, tieredProxyUrls } = options;+ const { proxyUrls, newUrlFunction } = options;- if ([proxyUrls, newUrlFunction, tieredProxyUrls].filter((x) => x).length > 1)- this._throwCannotCombineCustomMethods();+ if (proxyUrls && newUrlFunction) this._throwCannotCombineCustomMethods();if (!proxyUrls && !newUrlFunction && validateRequired) this._throwNoOptionsProvided();this.proxyUrls = proxyUrls;this.newUrlFunction = newUrlFunction;- this.tieredProxyUrls = tieredProxyUrls;}/**@@ -194,17 +123,8 @@ export class ProxyConfiguration {** @return Represents information about used proxy and its configuration.*/- async newProxyInfo(options?: TieredProxyOptions): Promise<ProxyInfo | undefined> {- let url: string | undefined;- let tier: number | undefined;- if (this.tieredProxyUrls) {- const { proxyUrl, proxyTier } = this._handleTieredUrl(options);- url = proxyUrl ?? undefined;- tier = proxyTier;- } else {- url = await this.newUrl(options);- }-+ async newProxyInfo(options?: NewUrlOptions): Promise<ProxyInfo | undefined> {+ const url = await this.newUrl(options);if (!url) return undefined;const { username, password, port, hostname } = new URL(url);@@ -215,92 +135,20 @@ export class ProxyConfiguration {password: decodeURIComponent(password),hostname,port: port!,- proxyTier: tier,};}- /**- * Given a request / proxy tier, this function returns a new proxy URL based on the provided configuration options.- * @param options Options for the tiered proxy rotation- * @returns An object with the proxy URL and the proxy tier used.- */- protected _handleTieredUrl(options?: TieredProxyOptions): TieredProxy {- if (!this.tieredProxyUrls) throw new Error('Tiered proxy URLs are not set');-- if (!options || (!options?.request && options?.proxyTier === undefined)) {- const allProxyUrls = this.tieredProxyUrls.flat();- return {- proxyUrl: allProxyUrls[this.nextCustomUrlIndex++ % allProxyUrls.length],- };- }-- let tierPrediction = options.proxyTier!;-- if (typeof tierPrediction !== 'number') {- tierPrediction = this.predictProxyTier(options.request!)!;- }-- const proxyTier = this.tieredProxyUrls![tierPrediction];-- return {- proxyUrl: proxyTier[this.nextCustomUrlIndex++ % proxyTier.length],- proxyTier: tierPrediction,- };- }-- /**- * Given a `Request` object, this function returns the tier of the proxy that should be used for the request.- *- * This returns `null` if `tieredProxyUrls` option is not set.- */- protected predictProxyTier(request: Request): number | null {- if (!this.tieredProxyUrls) return null;-- const domain = new URL(request.url).hostname;- if (!this.domainTiers.has(domain)) {- this.domainTiers.set(domain, new ProxyTierTracker(this.tieredProxyUrls));- }-- request.userData.__crawlee ??= {};-- const tracker = this.domainTiers.get(domain)!;-- if (typeof request.userData.__crawlee.lastProxyTier === 'number') {- tracker.addError(request.userData.__crawlee.lastProxyTier);- }-- const tierPrediction = tracker.predictTier();-- if (- typeof request.userData.__crawlee.lastProxyTier === 'number' &&- request.userData.__crawlee.lastProxyTier !== tierPrediction- ) {- this.log.debug(- `Changing proxy tier for domain "${domain}" from ${request.userData.__crawlee.lastProxyTier} to ${tierPrediction}.`,- );- }-- request.userData.__crawlee.lastProxyTier = tierPrediction;- request.userData.__crawlee.forefront = true;-- return tierPrediction;- }-/*** Returns a new proxy URL based on provided configuration options.** @return A string with a proxy URL, including authentication credentials and port number.* For example, `http://bob:[email protected]:8000`*/- async newUrl(options?: TieredProxyOptions): Promise<string | undefined> {+ async newUrl(options?: NewUrlOptions): Promise<string | undefined> {if (this.newUrlFunction) {return (await this._callNewUrlFunction({ request: options?.request })) ?? undefined;}- if (this.tieredProxyUrls) {- return this._handleTieredUrl(options).proxyUrl ?? undefined;- }-return this._handleProxyUrlsList() ?? undefined;}diff --git a/packages/types/src/session.ts b/packages/types/src/session.tsindex 5427db68b..9ff549ba2 100644--- a/packages/types/src/session.ts+++ b/packages/types/src/session.ts@@ -56,11 +56,6 @@ export interface ProxyInfo {*/port: number | string;- /**- * Proxy tier for the current proxy, if applicable (only for `tieredProxyUrls`).- */- proxyTier?: number;-/*** When `true`, the proxy is likely intercepting HTTPS traffic and is able to view and modify its content.*diff --git a/test/core/proxy_configuration.test.ts b/test/core/proxy_configuration.test.tsindex ed16bb93c..bd42904f6 100644--- a/test/core/proxy_configuration.test.ts+++ b/test/core/proxy_configuration.test.ts@@ -1,4 +1,4 @@-import { ProxyConfiguration, Request } from '@crawlee/core';+import { ProxyConfiguration } from '@crawlee/core';describe('ProxyConfiguration', () => {test('newUrl() should return proxy URL', async () => {@@ -174,121 +174,4 @@ describe('ProxyConfiguration', () => {}});});-- describe('with tieredProxyUrls', () => {- test('without Request rotates the urls uniformly', async () => {- const proxyConfiguration = new ProxyConfiguration({- tieredProxyUrls: [- ['http://proxy.com:1111', 'http://proxy.com:2222'],- ['http://proxy.com:3333', 'http://proxy.com:4444'],- ],- });-- // @ts-expect-error protected property- const tieredProxyUrls = proxyConfiguration.tieredProxyUrls!;- expect(await proxyConfiguration.newUrl()).toEqual(tieredProxyUrls[0][0]);- expect(await proxyConfiguration.newUrl()).toEqual(tieredProxyUrls[0][1]);- expect(await proxyConfiguration.newUrl()).toEqual(tieredProxyUrls[1][0]);- expect(await proxyConfiguration.newUrl()).toEqual(tieredProxyUrls[1][1]);- expect(await proxyConfiguration.newUrl()).toEqual(tieredProxyUrls[0][0]);- });-- test('rotating a request results in higher-level proxies', async () => {- const proxyConfiguration = new ProxyConfiguration({- tieredProxyUrls: [['http://proxy.com:1111'], ['http://proxy.com:2222'], ['http://proxy.com:3333']],- });-- const request = new Request({- url: 'http://example.com',- });-- // @ts-expect-error protected property- const tieredProxyUrls = proxyConfiguration.tieredProxyUrls!;- expect(await proxyConfiguration.newUrl({ request })).toEqual(tieredProxyUrls[0][0]);- expect(await proxyConfiguration.newUrl({ request })).toEqual(tieredProxyUrls[1][0]);- expect(await proxyConfiguration.newUrl({ request })).toEqual(tieredProxyUrls[2][0]);-- // we still get the same (higher) proxy tier even with a new request- const request2 = new Request({- url: 'http://example.com/another-resource',- });-- expect(await proxyConfiguration.newUrl({ request: request2 })).toEqual(tieredProxyUrls[2][0]);- });-- test('upshifts and downshifts properly', async () => {- const tieredProxyUrls = [['http://proxy.com:1111'], ['http://proxy.com:2222'], ['http://proxy.com:3333']];-- const proxyConfiguration = new ProxyConfiguration({- tieredProxyUrls,- });-- const request = new Request({- url: 'http://example.com',- });-- let gotToTheHighestProxy = false;- for (let i = 0; i < 10; i++) {- const lastProxyUrl = await proxyConfiguration.newUrl({ request });- if (lastProxyUrl === tieredProxyUrls[2][0]) {- gotToTheHighestProxy = true;- break;- }- }-- expect(gotToTheHighestProxy).toBe(true);-- // Even the highest-tier proxies didn't help - we should try going down- let gotToTheLowestProxy = false;-- for (let i = 0; i < 20; i++) {- const lastProxyUrl = await proxyConfiguration.newUrl({ request });- if (lastProxyUrl === tieredProxyUrls[0][0]) {- gotToTheLowestProxy = true;- break;- }- }-- expect(gotToTheLowestProxy).toBe(true);- });-- test('successful requests make the proxy tier drop eventually', async () => {- const tieredProxyUrls = [['http://proxy.com:1111'], ['http://proxy.com:2222'], ['http://proxy.com:3333']];-- const proxyConfiguration = new ProxyConfiguration({- tieredProxyUrls,- });-- const failingRequest = new Request({- url: 'http://example.com',- });- let gotToTheHighestProxy = false;-- for (let i = 0; i < 10; i++) {- const lastProxyUrl = await proxyConfiguration.newUrl({ request: failingRequest });-- if (lastProxyUrl === tieredProxyUrls[2][0]) {- gotToTheHighestProxy = true;- break;- }- }-- expect(gotToTheHighestProxy).toBe(true);-- let gotToTheLowestProxy = false;-- for (let i = 0; i < 100; i++) {- const lastProxyUrl = await proxyConfiguration.newUrl({- request: new Request({ url: `http://example.com/${i}` }),- });-- if (lastProxyUrl === tieredProxyUrls[0][0]) {- gotToTheLowestProxy = true;- break;- }- }-- expect(gotToTheLowestProxy).toBe(true);- });- });});