Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
refactor!: Introduce IBrowserPool interface and utilize it in BrowserCrawler (#3669)
97701bd Jan Buchar committed 5 months ago
refactor!: Finalize transition to IRequestManager and IRequestLoader (#3719)
4041322 Jan Buchar committed 5 months ago
feat: allow navigation hooks to override crawling context members (#3672)
a54e308 Jindřich Bär committed 5 months ago
feat: store browser / HTTP client fingerprint in Session (#3646)
342dd44 Jindřich Bär committed 5 months ago
refactor!: Align `DatasetClient` and `KeyValueStoreClient` interfaces with their Python counterparts (#3627)
59ec997 Jan Buchar committed 5 months ago
chore: bump puppeteer to 25 and playwright to 1.60 (#3703)
49e417d Jindřich Bär committed 5 months ago
feat: export `SessionPool` interface, minimize `ISession`, allow custom implementations (#3644)
dcf8d6b Jindřich Bär committed 5 months ago
refactor!: remove the `EventEmitter` interface from `SessionPool` (#3643)
37a0c57 Jindřich Bär committed 6 months ago
refactor: revert Configuration.reset() static method (#3651)
412ad09 Martin Adámek committed 6 months ago
ci: disable pnpm 11's verifyDepsBeforeRun to unbreak parallel turbo copy (#3650)
a65778a Martin Adámek committed 6 months ago
refactor: add `Configuration.reset()` static method (#3649)
63ef023 Martin Adámek committed 6 months ago
refactor!: Rename StorageManager to StorageInstanceManager and use it to centralize storage instance caching (#3584)
b6e5dd6 Jan Buchar committed 6 months ago
refactor!: treat session errors as regular request retries (#3631)
1a8de68 Jindřich Bär committed 6 months ago
feat: rename `persistCookiesPerSession` to `saveResponseCookies`, various fixes (#3606)
337b54a Jindřich Bär committed 6 months ago
feat: reuse `Sessions` correctly in `BasicCrawler` (#3605)
1cf1060 Jindřich Bär committed 6 months ago
fix(linkedom-crawler): declare cheerio as a dependency (#3620)
9ffa53c Martin Adámek committed 6 months ago
chore(ci): restore Docker image dispatch, gated to master
ac41130 Martin Adámek committed 6 months ago
chore(ci): treat any non-prod dist-tag as canary in publish workflow
1093e31 Martin Adámek committed 6 months ago
chore(ci): pin lerna's minimatch to v3 and fail step on lerna error
69157f5 Martin Adámek committed 6 months ago
chore: pass --no-error-on-unmatched-pattern to oxlint in lint-staged (#3611)
cfb6dde Martin Adámek committed 6 months ago
chore(deps-dev): bump oxlint to 1.62 and oxfmt to 0.46 (#3608)
03e343a Martin Adámek committed 6 months ago
chore: use `@apify/oxlint-config` as shared lint preset (#3607)
b8cea5a Martin Adámek committed 6 months ago
refactor: move `describe` helper to `@crawlee/utils` as `inspectValue` (#3586)
2fad6be Richard Solar committed 6 months ago
feat(`session-pool`): add `sessionReuseStrategy` option (#3587)
9047dc2 Jindřich Bär committed 7 months ago