Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: log original error message on session rotation (#2022)
8a11ffb Jindřich Bär committed 3 years ago
chore: fix `sessionRotationCount` when running on apify platform
c4fb5e2 Martin Adámek committed 3 years ago
feat: add `closeCookieModals` context helper for Playwright and Puppeteer (#1927)
98d93bb Jindřich Bär committed 3 years ago
fix(core): support relative links in `enqueueLinks` explicitly provided via `urls` option (#2014)
cbd9d08 Martin Adámek committed 3 years ago
fix: cleanup worker stuff from memory storage to fix `vitest` (#2004)
d2e098c Vlad Frangu committed 3 years ago
chore(deps): update typescript-eslint monorepo to v6 (#1967)
d0810d3 renovate[bot] committed 3 years ago
chore(deps): update dependency @apify/eslint-config-ts to ^0.3.0 (#1924)
39a7dbc renovate[bot] committed 3 years ago
feat(basic-crawler): allow configuring the automatic status message (#2001)
3eb4e4c Martin Adámek committed 3 years ago
feat(core): use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper
4d61ca9 Martin Adámek committed 3 years ago
chore: use relative paths for `test/shared` import from the `packages` folder
05af44d Martin Adámek committed 3 years ago
chore: update eslint and improve the linter setup (#1997)
37145fa Martin Adámek committed 3 years ago
fix(core): add requests from URL list (`requestsFromUrl`) to the queue in batches
418fbf8 Martin Adámek committed 3 years ago