Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Kill browser process on pipe errors
Ondra Urban committed 7 years ago
cc14a12
Add gotoExtended back with better message
Ondra Urban committed 7 years ago
bfa4c84
Use pipes instead of web sockets
Ondra Urban committed 7 years ago
6345aee
Remove interception from code base
Ondra Urban committed 7 years ago
4db6a08
Build docs
Ondra Urban committed 7 years ago
561f305
Fix flaky test, update changelog, bump version
Ondra Urban committed 7 years ago
c0e5762
Fix wrong CPU autoscaling check
Ondra Urban committed 7 years ago
590f901
Update changelog
Ondra Urban committed 7 years ago
ba33e42
Fix failing test, update Puppeteer to 1.17
Ondra Urban committed 7 years ago
f37ea8b
Add idempotencyKey to addWebhook
Ondra Urban committed 7 years ago
3daa5c2
Initial commit
Ondra Urban committed 7 years ago
d2d5caf
Fix doc generation and build docs
Ondra Urban committed 7 years ago
71dbe9f
Bump version, update Changelog
Ondra Urban committed 7 years ago
da241f3
Build docs
Ondra Urban committed 7 years ago
f365ea7
c5c3585
Refactor Job#run
Ondra Urban committed 7 years ago
58a9015
Fix typo
Ondra Urban committed 7 years ago
7c58276
Refactor _ensureCorrectMaxMemory
Ondra Urban committed 7 years ago
913be97
Remove delayPromise (bluebird)
Ondra Urban committed 7 years ago
a1e433a
Add statistics logMessage to BasicCrawler
Ondra Urban committed 7 years ago
08b9e78
0f6ffb6
Better logs from AutoscaledPool class
Jan Curn committed 7 years ago
3b0091a
Implement PR comments
Ondra Urban committed 7 years ago
2f61d60
Move crawlers to their own folder
Ondra Urban committed 7 years ago
de44764
Fix request list doc links
Ondra Urban committed 7 years ago
e51527f