Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
---
id: proxy-management
title: Proxy Management
description: Using proxies to get around those annoying IP-blocks
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
import HttpSource from '!!raw-loader!./proxy-management-integration-http.ts';
import CheerioSource from '!!raw-loader!./proxy-management-integration-cheerio.ts';
import PlaywrightSource from '!!raw-loader!./proxy-management-integration-playwright.ts';
import PuppeteerSource from '!!raw-loader!./proxy-management-integration-puppeteer.ts';
import SessionStandaloneSource from '!!raw-loader!./proxy-management-session-standalone.ts';
import SessionHttpSource from '!!raw-loader!./proxy-management-session-http.ts';
import SessionCheerioSource from '!!raw-loader!./proxy-management-session-cheerio.ts';
import SessionPlaywrightSource from '!!raw-loader!./proxy-management-session-playwright.ts';
import SessionPuppeteerSource from '!!raw-loader!./proxy-management-session-puppeteer.ts';
import InspectionHttpSource from '!!raw-loader!./proxy-management-inspection-http.ts';
import InspectionCheerioSource from '!!raw-loader!./proxy-management-inspection-cheerio.ts';
import InspectionPlaywrightSource from '!!raw-loader!./proxy-management-inspection-playwright.ts';
import InspectionPuppeteerSource from '!!raw-loader!./proxy-management-inspection-puppeteer.ts';
[IP address blocking](https://en.wikipedia.org/wiki/IP_address_blocking) is one of the oldest
and most effective ways of preventing access to a website. It is therefore paramount for
a good web scraping library to provide easy to use but powerful tools which can work around
IP blocking. The most powerful weapon in our anti IP blocking arsenal is a
[proxy server](https://en.wikipedia.org/wiki/Proxy_server).
With Crawlee we can use our own proxy servers or proxy servers acquired from
third-party providers.
Check out the [avoid blocking guide](./avoid-blocking) for more information about blocking.
## Quick start
If we already have proxy URLs of our own, we can start using
them immediately in only a few lines of code.
```javascript
import { ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://proxy-1.com',
'http://proxy-2.com',
]
});
const proxyUrl = await proxyConfiguration.newUrl();
```
Examples of how to use our proxy URLs with crawlers are shown below in [Crawler integration](#crawler-integration) section.
## Proxy Configuration
All our proxy needs are managed by the <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> class. We create an instance using the `ProxyConfiguration` <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided options. See the <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink> for all the possible constructor options.
### Static proxy list
You can provide a static list of proxy URLs to the `proxyUrls` option. The `ProxyConfiguration` will then rotate through the provided proxies.
```javascript
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://proxy-1.com',
'http://proxy-2.com',
null // null means no proxy is used
]
});
```
This is the simplest way to use a list of proxies. Crawlee will rotate through the list of proxies in a round-robin fashion.
### Custom proxy function
The `ProxyConfiguration` class allows you to provide a custom function to pick a proxy URL. This is useful when you want to implement your own logic for selecting a proxy.
```javascript
const proxyConfiguration = new ProxyConfiguration({
newUrlFunction: async () => {
const { proxyUrl } = await myProxyProvider.getProxy();
return proxyUrl; // or return null to skip the proxy
}
});
```
The `newUrlFunction` takes no arguments and returns a string with the proxy URL (or `null` to skip the proxy). Crawlers call it once for every new session, so it cannot route individual requests. To send specific requests through specific proxies, create named sessions with their own proxies and pin requests to them, as shown in the [session management guide](../guides/session-management#pinning-a-request-to-a-specific-session).
## Crawler integration
`ProxyConfiguration` integrates seamlessly into <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>, <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.
<Tabs groupId="proxy_session_management">
<TabItem value="http" label="HttpCrawler">
<CodeBlock language="js">
{HttpSource}
</CodeBlock>
</TabItem>
<TabItem value="cheerio" label="CheerioCrawler" default>
<CodeBlock language="js">
{CheerioSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
<CodeBlock language="js">
{PlaywrightSource}
</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
<CodeBlock language="js">
{PuppeteerSource}
</CodeBlock>
</TabItem>
</Tabs>
Our crawlers will now use the selected proxies for all connections.
### Proxies in browser crawlers
A browser is launched with a single proxy, so in `PlaywrightCrawler` and `PuppeteerCrawler` the session's proxy only applies to a page if the browser it lands in was launched with the same one. By default, the pool opens the page in any browser with free capacity, which can ignore the session's proxy. Two `launchContext` options fix that:
- `useIncognitoPages: true` - every page gets its own browser context with the session's proxy applied to it. Pages share no cookies or cache with each other; the session's cookies are still carried over.
- `browserPerProxy: true` - the pool only reuses a browser that was launched with the page's proxy, and launches a new one otherwise. Useful when you need a shared context per proxy, at the cost of more browser instances.
```js
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
proxyConfiguration,
launchContext: {
browserPerProxy: true,
},
// ...
});
```
## IP Rotation and session management
Each call to <ApiLink to="core/class/ProxyConfiguration#newUrl">`proxyConfiguration.newUrl()`</ApiLink> generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and <ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> class for more information on how keeping a real session helps us avoid blocking.
<Tabs groupId="proxy_session_management">
<TabItem value="http" label="HttpCrawler">
<CodeBlock language="js">
{SessionHttpSource}
</CodeBlock>
</TabItem>
<TabItem value="cheerio" label="CheerioCrawler" default>
<CodeBlock language="js">
{SessionCheerioSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
<CodeBlock language="js">
{SessionPlaywrightSource}
</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
<CodeBlock language="js">
{SessionPuppeteerSource}
</CodeBlock>
</TabItem>
<TabItem value="standalone" label="Standalone">
<CodeBlock language="js">
{SessionStandaloneSource}
</CodeBlock>
</TabItem>
</Tabs>
## Inspecting current proxy in Crawlers
`HttpCrawler`, `CheerioCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler` grant access to information about the currently used proxy
in their `requestHandler` using a <ApiLink to="types/interface/ProxyInfo">`proxyInfo`</ApiLink> object.
With the `proxyInfo` object, we can easily access the proxy URL.
<Tabs groupId="proxy_session_management">
<TabItem value="http" label="HttpCrawler">
<CodeBlock language="js">
{InspectionHttpSource}
</CodeBlock>
</TabItem>
<TabItem value="cheerio" label="CheerioCrawler" default>
<CodeBlock language="js">
{InspectionCheerioSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
<CodeBlock language="js">
{InspectionPlaywrightSource}
</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
<CodeBlock language="js">
{InspectionPuppeteerSource}
</CodeBlock>
</TabItem>
</Tabs>