Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

chore(deps): update got-scraping package (#3452)
Richard Solar committed 8 months ago
98be258
b663940
fix: Fix file download hang (#3411)
Jan Buchar committed 9 months ago
6b507b8
chore: add `discoverValidSitemaps` utility (#3392)
Jindřich Bär committed 9 months ago
c6ab237
feat: `fetch` API for custom http clients (#3326)
Jindřich Bär committed 10 months ago
e44f7f3
f57ff3d
chore: Fix HttpCrawler context types (#3291)
Jan Buchar committed 11 months ago
28468a3
8c5bc0f
chore: fix broken types in docs examples (#3287)
Jindřich Bär committed 11 months ago
6361d06
chore: fix build errors from `master` rebase (#3285)
Jindřich Bär committed 11 months ago
52fe3d1
feat!: use native `Request` / `Response` interface (#3163)
Jindřich Bär committed 11 months ago
32caa0c
feat: store `ProxyInfo` inside `Session` instances (#3199)
Jindřich Bär committed 11 months ago
14ed530
70aa8c9
7b5ffbe
fix: remove old system info implementation
Martin Adámek committed 1 year ago
db404a1
fix CLI 2
Martin Adámek committed 1 year ago
e6b7dc0
fix CLI
Martin Adámek committed 1 year ago
8952c78
chore: bump a few more dependencies
Martin Adámek committed 1 year ago
7d01334
chore: remove some deadcode
Martin Adámek committed 1 year ago
6b317b7
chore: improve types to get rid of some `as any`
Martin Adámek committed 1 year ago
b312bf7
chore: use node 24 in e2e tests and project templates
Martin Adámek committed 1 year ago
cf93415