Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

18.6 KB
Raw Blame
---
id: session-management
title: Session Management
description: How to manage your cookies, proxy IP rotations and more
---

import ApiLink from '@site/src/components/ApiLink';

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';

import BasicSource from '!!raw-loader!./session-management-basic.ts';
import HttpSource from '!!raw-loader!./session-management-http.ts';
import CheerioSource from '!!raw-loader!./session-management-cheerio.ts';
import PlaywrightSource from '!!raw-loader!./session-management-playwright.ts';
import PuppeteerSource from '!!raw-loader!./session-management-puppeteer.ts';
import StandaloneSource from '!!raw-loader!./session-management-standalone.ts';

&#8203;<ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> manages the rotation of proxy IP addresses, cookies, and browser fingerprints in Crawlee. A single <ApiLink to="basic-crawler/class/Session">`Session`</ApiLink> bundles all the identifying state of one "virtual user" — its cookie jar, its proxy (and therefore its IP), and a fingerprint hint — so that everything that makes a series of requests look like it comes from one person rotates together. When a session gets blocked, the whole bundle is thrown away at once and a fresh identity takes over, rather than reusing a burnt IP with new cookies (or vice versa).

The main benefits of the session pool are that it filters out blocked or non-working proxies so the crawler does not keep retrying over them, it keeps identity-bound state (cookies, auth tokens, headers) tied to the IP that obtained it, and it spreads requests across IPs to avoid burning a small pool. The selection strategy is configurable — see [Choosing a rotation strategy](#choosing-a-rotation-strategy) below.

All crawler instances now require a `SessionPool`. In most cases you do not create one yourself: you just read the `session` from the request handler and let the crawler mark it good or bad for you. You only construct a `SessionPool` explicitly when you want to override its defaults or share one instance across several crawlers.

Check out the [avoid blocking guide](./avoid-blocking) for the bigger picture on why blocking happens and how fingerprints fit in.

Now let's take a look at the examples of how to use the session pool:
- with <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink>;
- with <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>;
- with <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>;
- with <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>;
- with <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>;
- without a crawler (standalone usage to manage sessions manually).

<Tabs groupId="session_pool">
    <TabItem value="basic" label="BasicCrawler">
        <CodeBlock language="js">
            {BasicSource}
        </CodeBlock>
    </TabItem>
    <TabItem value="http" label="HttpCrawler">
        <CodeBlock language="js">
            {HttpSource}
        </CodeBlock>
    </TabItem>
    <TabItem value="cheerio" label="CheerioCrawler" default>
        <CodeBlock language="js">
            {CheerioSource}
        </CodeBlock>
    </TabItem>
    <TabItem value="playwright" label="PlaywrightCrawler">
        <CodeBlock language="js">
            {PlaywrightSource}
        </CodeBlock>
    </TabItem>
    <TabItem value="puppeteer" label="PuppeteerCrawler">
        <CodeBlock language="js">
            {PuppeteerSource}
        </CodeBlock>
    </TabItem>
    <TabItem value="standalone" label="Standalone">
        <CodeBlock language="js">
            {StandaloneSource}
        </CodeBlock>
    </TabItem>
</Tabs>

These are the basics of configuring the session pool. The rest of this guide covers how to control which session is used, what state it carries, and when it is thrown away.

## How a session is retired

A session stays in the pool and keeps being handed out as long as <ApiLink to="basic-crawler/class/Session#isUsable">`isUsable()`</ApiLink> returns `true`. It stops being usable — and is dropped from rotation — as soon as any of the following happens:

- its **error score** reaches `maxErrorScore` (default `3`),
- its **usage count** reaches `maxUsageCount` (default `50`),
- it is older than `maxAgeSecs` (default `3000` seconds), or
- it has been explicitly **retired**.

You influence this with three methods on the session. <ApiLink to="basic-crawler/class/Session#markGood">`markGood()`</ApiLink> records a successful use — it increments the usage count and heals the error score a little (by `errorScoreDecrement`, default `0.5`). <ApiLink to="basic-crawler/class/Session#markBad">`markBad()`</ApiLink> records a failure that *might* be the session's fault and *might* just be bad luck — it raises the error score by one, so a session needs to fail repeatedly before it is dropped. <ApiLink to="basic-crawler/class/Session#retire">`retire()`</ApiLink> drops the session immediately and permanently; this is what you call when you are certain the identity itself is burnt (for example, a `403` response).

The distinction between `markBad()` and `retire()` matters. Use `markBad()` for transient, external problems such as a timeout or a `5XX` response — the IP is probably fine and a couple of retries should not throw it away. Use `retire()` for problems that prove the session is blocked, where reusing it is pointless. Retirement is terminal: once a session is retired, a later `markGood()` will not bring it back.

When using a crawler you rarely call `markGood()` yourself — the crawler calls it automatically after a successful request handler run. You only need to reach for `markBad()` / `retire()` (or let blocked status codes do it for you, see [below](#letting-blocked-responses-retire-sessions)) when you detect a problem the crawler cannot see, such as a "you are blocked" message inside an otherwise `200` response.

## Managing cookies

Every session owns a [`tough-cookie`](https://github.com/salesforce/tough-cookie) cookie jar, reachable as `session.cookieJar`. Cookies arriving in `Set-Cookie` response headers are stored in it automatically — this is controlled by the `saveResponseCookies` crawler option (default `true`) — so they are replayed on every later request that reuses the same session. Set `saveResponseCookies: false` to keep response cookies out of the session jar.

You can also seed or read cookies yourself. <ApiLink to="basic-crawler/class/Session#setCookie">`session.setCookie('name=value', url)`</ApiLink> adds a single cookie, <ApiLink to="basic-crawler/class/Session#getCookieString">`session.getCookieString(url)`</ApiLink> returns the `Cookie` header value the session would send for that URL, and `session.cookieJar` gives you the full jar for anything more involved.

```js
const crawler = new CheerioCrawler({
    requestHandler: async ({ session, request }) => {
        await session.setCookie('consent=yes', request.url);
    },
});
```

### Cookie precedence and overrides

When an HTTP-based crawler (or a direct `sendRequest` call) builds the outgoing `Cookie` header, it starts from a **base jar** and then overlays any cookies set on the request:

- The base jar is the explicit `cookieJar` passed to `sendRequest` if you provide one, otherwise the session's own cookie jar.
- A `Cookie` header on the request (`request.headers.Cookie`) is merged on top of that base. A cookie set this way wins over a base-jar cookie of the same name, but it is *not* persisted back into the session.

So a `Cookie` request header always beats the stored cookie of the same name regardless of which jar is the base, while passing an explicit `cookieJar` swaps out the whole base for that single call. To override a single cookie for one request, set it on the request header:

```js
import { HttpCrawler } from 'crawlee';
import { CookieJar } from 'tough-cookie';

const crawler = new HttpCrawler({
    preNavigationHooks: [
        async ({ request }) => {
            // wins over any same-named cookie in the session jar, for this request only
            request.headers = { ...request.headers, Cookie: 'token=override' };
        },
    ],
    requestHandler: async ({ sendRequest }) => {
        // ...or to fully replace the jar for a single call:
        const jar = new CookieJar();
        await jar.setCookie('token=override', 'https://example.com');
        await sendRequest({ url: 'https://example.com' }, { cookieJar: jar });
    },
});
```

A `Cookie` header you set on a request is always honored — it is never silently overwritten by the session jar.

## Choosing a rotation strategy

The `sessionReuseStrategy` option decides *which* session `getSession()` hands out, and it is the main lever for matching the pool's behavior to a target site. Three strategies are available, each suited to a different use case.

**Maximise IP and fingerprint diversity** — use `'random'` (the default). The pool creates a brand-new session for every request until it reaches `maxPoolSize`, then picks a usable session at random. This spreads traffic as widely as possible across IPs and fingerprints and is the right default for most large crawls.

**Distribute load evenly across sessions** — use `'round-robin'`. Like `random`, the pool fills up to `maxPoolSize` first, but then cycles through sessions in order instead of picking randomly. This is useful when you want every session to do roughly the same amount of work — for example, combined with `maxUsageCount` so all sessions reach their limit and rotate out at about the same time.

**Use a single IP until it breaks** — use `'use-until-failure'`. The pool returns the *same* session on every call and only moves to the next one once the current session is retired. This is the strategy for sites that reward consistency: where switching IP mid-flow looks suspicious, where you have logged in and want to stay logged in, or where you simply want to squeeze a working proxy for as long as it lasts before paying for another.

<Tabs>
    <TabItem value="diversity" label="Maximise diversity (default)" default>

```js
import { SessionPool } from 'crawlee';

const sessionPool = new SessionPool({
    sessionReuseStrategy: 'random',
});
```

    </TabItem>
    <TabItem value="even" label="Even load">

```js
import { SessionPool } from 'crawlee';

const sessionPool = new SessionPool({
    sessionReuseStrategy: 'round-robin',
    // make every session retire after the same amount of work
    sessionOptions: { maxUsageCount: 100 },
});
```

    </TabItem>
    <TabItem value="sticky" label="One IP until it breaks">

```js
import { SessionPool } from 'crawlee';

const sessionPool = new SessionPool({
    sessionReuseStrategy: 'use-until-failure',
});
```

    </TabItem>
</Tabs>

Whichever strategy you pick, you can cap how hard each session works through `sessionOptions`. Set `maxUsageCount` when you know a site starts blocking after roughly _N_ requests from one IP, `maxAgeSecs` when sessions should be cycled on a time basis, and `maxErrorScore` to control how forgiving the pool is about intermittent failures before dropping a session.

```js
const sessionPool = new SessionPool({
    maxPoolSize: 25,
    sessionOptions: {
        maxAgeSecs: 600,
        maxUsageCount: 150, // e.g. when you know the site blocks after ~150 requests
    },
});
```

## Letting blocked responses retire sessions

You do not have to inspect every response by hand. Crawlers treat a configurable set of HTTP status codes as proof that a session is blocked and retire it automatically, retrying the request with a fresh session. This is controlled by the `blockedStatusCodes` crawler option (default `[401, 403, 429]`).

```js
import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    // a 403 or 429 will retire the current session and retry on a new one
    blockedStatusCodes: [403, 429],
    requestHandler: async ({ session, request }) => {
        // session is already a working, non-blocked one
    },
});
```

For sites that respond with a `200` page that is actually a bot wall (Cloudflare challenges, Google's rate-limit page), set `retryOnBlocked: true` to have the crawler detect those by content and retry as well. For deeper anti-blocking measures see the [avoid blocking guide](./avoid-blocking).

:::tip A 429 is a rate limit, not a block

Retiring a session on HTTP 429 burns proxies without slowing anything down — the site is asking you to wait, not telling you the session is unwelcome. Wrap your request manager in a <ApiLink to="basic-crawler/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> to back off per domain instead of rotating; see [per-domain throttling](./request-loaders#per-domain-throttling). Sessions are left untouched for the domains it covers.

Those domains are handled as rate limits before `blockedStatusCodes` is consulted, so leave 429 in the list — removing it only changes what happens for domains the manager does not cover.

:::

## Sharing a session pool between crawlers

A `SessionPool` instance can be shared across multiple crawlers by passing the same object to each crawler's `sessionPool` option. This is useful in multi-stage scrapers — for example a fast `CheerioCrawler` that discovers links and a `PlaywrightCrawler` that renders detail pages — where you want both stages to reuse the same proven, non-blocked identities and their cookies instead of each warming up its own pool from scratch.

```js
import { CheerioCrawler, PlaywrightCrawler, SessionPool } from 'crawlee';

const sessionPool = new SessionPool({ maxPoolSize: 100 });

const listingCrawler = new CheerioCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
const detailCrawler = new PlaywrightCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
```

A pool you construct yourself is owned by you, not the crawler — the crawler will never tear it down or reset it between runs. Dispose of it with `await using` when you are done, which persists its final state and stops it listening for persistence events:

```js
import { CheerioCrawler, SessionPool } from 'crawlee';

await using sessionPool = new SessionPool({ maxPoolSize: 100 });

const crawler = new CheerioCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
await crawler.run(['https://crawlee.dev']);
// sessionPool is torn down here, even if the crawl throws
```

:::note Requires Node.js 24

The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="basic-crawler/class/SessionPool#teardown">`teardown()`</ApiLink> yourself instead — it is what the disposal hook calls anyway.

:::

## Custom session pools

A crawler accepts any object implementing the <ApiLink to="types/interface/ISessionPool">`ISessionPool`</ApiLink> interface as its `sessionPool` option, not just the built-in `SessionPool`. The contract is intentionally tiny — a single `getSession()` / `getSession(id)` method that hands out an <ApiLink to="types/interface/ISession">`ISession`</ApiLink> for a request. This lets you plug in a remote, shared, or database-backed session strategy without subclassing `SessionPool` or copying its internals.

`ISessionPool` and `ISession` are owned by `@crawlee/types`, which the `crawlee` meta-package does not re-export. Add it to your dependencies to import them.

```ts
import { BasicCrawler, Session } from 'crawlee';
import type { ISessionPool } from '@crawlee/types';

class MySessionPool implements ISessionPool {
    private readonly sessions = new Map<string, Session>();

    async getSession(sessionId?: string): Promise<Session | undefined> {
        if (sessionId) {
            const existing = this.sessions.get(sessionId);
            return existing?.isUsable() ? existing : undefined;
        }

        const usable = [...this.sessions.values()].find((s) => s.isUsable());
        if (usable) return usable;

        const fresh = new Session();
        this.sessions.set(fresh.id, fresh);
        return fresh;
    }
}

const crawler = new BasicCrawler({
    sessionPool: new MySessionPool(),
    requestHandler: async ({ session }) => {
        // session is a Session instance, use it as usual
    },
});
```

The returned objects just need to implement <ApiLink to="types/interface/ISession">`ISession`</ApiLink> — the crawler only calls `markGood()`, `markBad()`, `retire()`, and reads `cookieJar`, `proxyInfo`, and `fingerprint`, all of which are part of that interface.

## Pinning a request to a specific session

By default the pool decides which session a request gets. Setting `request.sessionId` overrides that and forces the request — and all of its retries — onto the session with that id. You can create a custom named session with <ApiLink to="basic-crawler/class/SessionPool#addSession">`addSession()`</ApiLink>, giving each its own proxy, cookies, or fingerprint. Because a session bundles a proxy, this is how you bind specific requests to specific proxies.

One important consequence: if a named session is retired — whether through accumulated `markBad()` calls, hitting `maxUsageCount`, or an explicit `retire()` — any subsequent `getSession(id)` call for that id returns `undefined`.
The crawler treats that as a `MissingSessionError`, counts it as a regular request error, and retries the request with the same `sessionId`. If the session stays retired, retries keep failing and the request eventually exhausts `maxRequestRetries`.
When a named session can be retired, handle this in your `errorHandler`: either recreate the session via `addSession()` with the same id, or clear `request.sessionId` to let the pool assign a fresh one.

A common usage pattern is escalating between proxy "tiers": add a cheap session and a premium one, start requests on the cheap session, and reassign `request.sessionId` to the premium one in an `errorHandler` so the retry goes out over the better proxy.

```ts
import { BasicCrawler, SessionPool } from 'crawlee';

const proxyInfoFromUrl = (proxyUrl: string) => {
    const { username, password, hostname, port } = new URL(proxyUrl);
    return { url: proxyUrl, username, password, hostname, port };
};

const sessionPool = new SessionPool();
await sessionPool.addSession({ id: 'cheap', proxyInfo: proxyInfoFromUrl('http://cheap-proxy.com') });
await sessionPool.addSession({ id: 'premium', proxyInfo: proxyInfoFromUrl('http://expensive-proxy.com') });

const crawler = new BasicCrawler({
    sessionPool,
    retryOnBlocked: true,
    requestHandler: async ({ sendRequest, request }) => {
        await sendRequest({ url: request.url });
    },
    errorHandler: async ({ request }) => {
        request.sessionId = 'premium'; // escalate the retry to the premium proxy
    },
});

await crawler.run([{ url: 'https://example.com', sessionId: 'cheap' }]);
```