Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
refactor!: drop the unused `request` option from `ProxyConfiguration` (#4153)
Jindřich Bärb6684527fa90
diff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy_management.mdxindex a3178d7ba..65d07094b 100644--- a/docs/guides/proxy_management.mdx+++ b/docs/guides/proxy_management.mdx@@ -80,19 +80,14 @@ The `ProxyConfiguration` class allows you to provide a custom function to pick a```javascriptconst proxyConfiguration = new ProxyConfiguration({- newUrlFunction: ({ request } = {}) => {- if (request?.url.includes('crawlee.dev')) {- return null; // for crawlee.dev, we don't use a proxy- }-- return 'http://proxy-1.com'; // for all other URLs, we use this proxy+ newUrlFunction: async () => {+ const { proxyUrl } = await myProxyProvider.getProxy();+ return proxyUrl; // or return null to skip the proxy}});```-The `newUrlFunction` receives a single optional `options` parameter and returns a string with the proxy URL (or `null` to skip the proxy for the current request).--The `options` parameter is an object containing a <ApiLink to="core/class/Request">`Request`</ApiLink>, which is the request that will be made. Note that this object is not always available, for example when we are using the `newUrl` function directly. Your custom function should therefore not rely on the `request` object being present and provide a default behavior when it is not.+The `newUrlFunction` takes no arguments and returns a string with the proxy URL (or `null` to skip the proxy). Crawlers call it once for every new session, so it cannot route individual requests. To send specific requests through specific proxies, create named sessions with their own proxies and pin requests to them, as shown in the [session management guide](../guides/session-management#pinning-a-request-to-a-specific-session).## Crawler integrationdiff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.mdindex 75687aa64..1e7295c88 100644--- a/docs/public-api/crawlee-core.api.md+++ b/docs/public-api/crawlee-core.api.md@@ -406,7 +406,7 @@ interface Intervals {// @publicexport interface IProxyConfiguration {- newProxyInfo(options?: NewUrlOptions): Promise<ProxyInfo | undefined>;+ newProxyInfo(): Promise<ProxyInfo | undefined>;}// @public@@ -599,13 +599,6 @@ export interface MemoryStorageOptions {logger?: CrawleeLogger;}-// Not exported by the entry point; reachable only as a referenced type.-// @public (undocumented)-interface NewUrlOptions {- // (undocumented)- request?: Request_2;-}-// @publicexport class NonRetryableError extends Error {}@@ -638,16 +631,14 @@ export class ProxyConfiguration implements IProxyConfiguration {constructor(options?: ProxyConfigurationOptions);// (undocumented)readonly isManInTheMiddle = false;- newProxyInfo(options?: NewUrlOptions): Promise<ProxyInfo | undefined>;- newUrl(options?: NewUrlOptions): Promise<string | undefined>;+ newProxyInfo(): Promise<ProxyInfo | undefined>;+ newUrl(): Promise<string | undefined>;}// @public (undocumented)export interface ProxyConfigurationFunction {// (undocumented)- (options?: {- request?: Request_2;- }): string | null | Promise<string | null>;+ (): string | null | Promise<string | null>;}// @public (undocumented)diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex b0d7b3f3f..dfd08c622 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -1167,7 +1167,7 @@ More complex routing (more tiers, weighted draws, sticky assignment, cooldowns)#### `ProxyConfiguration.newUrl` / `newProxyInfo` signatures changed-Because proxy tiers are gone, the leading `sessionId` positional argument was dropped from `ProxyConfiguration.newUrl()` and `ProxyConfiguration.newProxyInfo()`. Both now take a single optional options object instead of `(sessionId?, options?)`. The `ProxyConfigurationFunction` callback (the `newUrlFunction` option) was likewise simplified — it no longer receives a `sessionId` as its first argument; it now receives an optional `{ request }` object. The `TieredProxy` interface and the `TieredProxyOptions` type have been removed.+Because proxy tiers are gone, the leading `sessionId` positional argument was dropped from `ProxyConfiguration.newUrl()` and `ProxyConfiguration.newProxyInfo()`. Both now take no arguments instead of `(sessionId?, options?)`. The `ProxyConfigurationFunction` callback (the `newUrlFunction` option) was likewise simplified — it no longer receives any arguments, neither a `sessionId` nor a `{ request }` object. The proxy is resolved once per session, so to route specific requests through specific proxies, pin them to named sessions (see [Pinning a request to a specific session](../guides/session-management#pinning-a-request-to-a-specific-session)). The `TieredProxy` interface and the `TieredProxyOptions` type have been removed.**Before:**```typescript@@ -1180,7 +1180,7 @@ const url = await proxyConfiguration.newUrl(sessionId);**After:**```typescriptconst proxyConfiguration = new ProxyConfiguration({- newUrlFunction: ({ request } = {}) => pickProxyFor(request),+ newUrlFunction: () => pickProxy(),});const url = await proxyConfiguration.newUrl();```diff --git a/packages/core/src/proxy_configuration.ts b/packages/core/src/proxy_configuration.tsindex fd1c2ac7c..ee3bcc38d 100644--- a/packages/core/src/proxy_configuration.ts+++ b/packages/core/src/proxy_configuration.ts@@ -1,7 +1,6 @@import type { Dictionary, ProxyInfo } from '@crawlee/types';import { z } from 'zod';-import type { Request } from './request.js';import { parseArgument, schemas } from './validators.js';const proxyConfigurationOptionsSchema = z.strictObject({@@ -13,7 +12,7 @@ const proxyConfigurationOptionsSchema = z.strictObject({});export interface ProxyConfigurationFunction {- (options?: { request?: Request }): string | null | Promise<string | null>;+ (): string | null | Promise<string | null>;}type UrlList = (string | null)[];@@ -27,7 +26,7 @@ export interface ProxyConfigurationOptions {proxyUrls?: UrlList;/**- * Custom function that allows you to generate the new proxy URL dynamically. It gets an optional parameter with the `Request` object when applicable.+ * Custom function that allows you to generate the new proxy URL dynamically.* Can return either stringified proxy URL or `null` if the proxy should not be used. Can be asynchronous.** This function is used to generate the URL when {@apilink ProxyConfiguration.newUrl} or {@apilink ProxyConfiguration.newProxyInfo} is called.@@ -35,10 +34,6 @@ export interface ProxyConfigurationOptions {newUrlFunction?: ProxyConfigurationFunction;}-interface NewUrlOptions {- request?: Request;-}-/*** Minimal contract that any object passed to a crawler as its `proxyConfiguration`* option must satisfy.@@ -51,10 +46,10 @@ interface NewUrlOptions {*/export interface IProxyConfiguration {/**- * Creates a new {@apilink ProxyInfo} object describing the proxy to use for the given- * request. Returns `undefined` when no proxy should be used.+ * Creates a new {@apilink ProxyInfo} object describing the proxy to use.+ * Returns `undefined` when no proxy should be used.*/- newProxyInfo(options?: NewUrlOptions): Promise<ProxyInfo | undefined>;+ newProxyInfo(): Promise<ProxyInfo | undefined>;}/**@@ -142,8 +137,8 @@ export class ProxyConfiguration implements IProxyConfiguration {** @return Represents information about used proxy and its configuration.*/- async newProxyInfo(options?: NewUrlOptions): Promise<ProxyInfo | undefined> {- const url = await this.newUrl(options);+ async newProxyInfo(): Promise<ProxyInfo | undefined> {+ const url = await this.newUrl();if (!url) return undefined;const { username, password, port, hostname } = new URL(url);@@ -163,9 +158,9 @@ export class ProxyConfiguration implements IProxyConfiguration {* @return A string with a proxy URL, including authentication credentials and port number.* For example, `http://bob:[email protected]:8000`*/- async newUrl(options?: NewUrlOptions): Promise<string | undefined> {+ async newUrl(): Promise<string | undefined> {if (this.#newUrlFunction) {- return (await this.callNewUrlFunction({ request: options?.request })) ?? undefined;+ return (await this.callNewUrlFunction()) ?? undefined;}return this.handleProxyUrlsList() ?? undefined;@@ -178,8 +173,8 @@ export class ProxyConfiguration implements IProxyConfiguration {/*** Calls the custom newUrlFunction and checks format of its return value*/- private async callNewUrlFunction(options?: { request?: Request }) {- const proxyUrl = await this.#newUrlFunction!(options);+ private async callNewUrlFunction() {+ const proxyUrl = await this.#newUrlFunction!();try {if (proxyUrl) {new URL(proxyUrl); // eslint-disable-line no-new