Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Updated publish.sh
Jan Curn committed 7 years ago
4f0ecef
4c2cb7c
17ff5b9
64f6cd6
Fix lint issue
Ondra Urban committed 7 years ago
52b6a80
Update docs
Ondra Urban committed 7 years ago
f70a359
Update autoscaling config for sync jobs
Ondra Urban committed 7 years ago
86ad83e
Fix memory leak
Ondra Urban committed 7 years ago
3010caf
Fix docs
Ondra Urban committed 7 years ago
052758b
Add test
Ondra Urban committed 7 years ago
a360fbc
Add autoscaled pool test
Ondra Urban committed 7 years ago
16fe787
Implement feature
Ondra Urban committed 7 years ago
4c6d780
Add openRequestList tests
Ondra Urban committed 7 years ago
cb2d80f
Fix test
Ondra Urban committed 7 years ago
f5f1b98
Add test
Ondra Urban committed 7 years ago
ec2b091
Better log messages
Jan Curn committed 7 years ago
1c9d1dd
Development of utils.htmlToText()
Jan Curn committed 7 years ago
626b37e
Implement solution, ensure original tests pass
Ondra Urban committed 7 years ago
ae3b5d9
Build docs
Ondra Urban committed 7 years ago
82d5c47
Increment version, update changelog
Ondra Urban committed 7 years ago
f6e6dc5
06687f6
7655846
Merge branch 'develop' into feature/reuse-tabs
Ondra Urban committed 7 years ago
1c8b6c1
1ff419e