Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

1007 B
Raw Blame
import { HttpCrawler, ProxyConfiguration, SessionPool } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({/* opts */});

const crawler = new HttpCrawler({
    // To use the proxy IP session rotation logic, you must turn the proxy usage on.
    proxyConfiguration,
    // Overrides default Session pool configuration.
    sessionPool: new SessionPool({ maxPoolSize: 100 }),
    // Set to true if you want the crawler to save cookies per session,
    // and set the cookie header to request automatically (default is true).
    saveResponseCookies: true,
    async requestHandler({ session, body }) {
        const title = /<title(?:.*?)>(.*?)<\/title>/.exec(body as string)?.[1];

        if (title === 'Blocked') {
            session?.retire();
        } else if (title === 'Not sure if blocked, might also be a connection error') {
            session?.markBad();
        } else {
            // session.markGood() - this step is done automatically in BasicCrawler.
        }
    },
});