Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
import { SessionPool } from 'crawlee';
// Override the default Session pool configuration.
const sessionPoolOptions = {
maxPoolSize: 100,
};
const sessionPool = new SessionPool(sessionPoolOptions);
// Get session.
const session = await sessionPool.getSession();
// Increase the errorScore.
session?.markBad();
// Throw away the session.
session?.retire();
// Lower the errorScore and mark the session good.
session?.markGood();