Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: migrate to pnpm and enable minimum release age (#3581)
fa06fab Martin Adámek committed 7 months ago
feat: allow binding a Session instance to a Request (#3518)
52bc273 Jindřich Bär committed 7 months ago
feat: share `SessionPool` instance between crawlers (#3499)
cb1b984 Jindřich Bär committed 7 months ago
chore: migrate from eslint/biome to oxlint/oxfmt (#3569)
1640c40 Martin Adámek committed 7 months ago
fix(puppeteer-crawler): migrate cookie handling to browser-context API (#3572)
0174b81 Martin Adámek committed 7 months ago
fix: Fix `skipNavigation` behavior in `HttpCrawler` subclasses (#3381)
d9b5fe0 Jan Buchar committed 7 months ago
refactor: replace service locator forwarding object with Proxy (#3515)
1508ab9 Martin Adámek committed 8 months ago
feat: lazy-initialize `SessionPool`, drop `SessionPool.open()` (#3513)
a23e79b Jindřich Bär committed 8 months ago
refactor: move blocked HTTP status code management to `BasicCrawler` (#3496)
8bf2407 Jindřich Bär committed 8 months ago
refactor: use `BasicCrawler` session in browser launch context (#3489)
9c0bd50 Jindřich Bär committed 8 months ago
feat: drop `useSessionPool` option, make `SessionPool` required in all crawlers (#3478)
2e1e450 Jindřich Bär committed 8 months ago
fix: improve ServiceConflictError message readability (#3480)
6456a64 Richard Solar committed 8 months ago
chore(memory-storage): sort dependencies and update yarn.lock (#3493)
0b5b3f8 Richard Solar committed 8 months ago
feat: add `AbortSignal` and rename `timeout` → `timeoutMillis` in `SendRequestOptions` (#3457)
c3e5697 Copilot committed 8 months ago
docs: sync 4.0 version with the latest changes in `master`
6653fb4 Jindřich Bär committed 8 months ago
refactor: use `ContextPipeline` to initialize `BasicCrawler`'s context idiomatically (#3388)
d33ab5f Jindřich Bär committed 8 months ago