Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

534 B
Raw Blame
import { RequestList } from 'crawlee';

// Open a request list with a static set of URLs.
// The name is used to persist the list's state in the default key-value store.
const requestList = await RequestList.open('my-list', [
    'https://crawlee.dev/',
    'https://crawlee.dev/docs',
    'https://crawlee.dev/api',
]);

// Iterate over the requests manually (a crawler does this for you under the hood).
for await (const request of requestList) {
    console.log(request.url);
    await requestList.markRequestAsHandled(request);
}