Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Merge branch 'develop' into feature/stop-crawler
Ondra Urban committed 8 years ago
b91a654
Tests pass
Ondra Urban committed 8 years ago
b114474
1ca3a86
Improving documentation
Jan Curn committed 8 years ago
6e6a2ae
Merge branch 'develop' into feature/cheerio-crawler
Ondra Urban committed 8 years ago
8cbd698
initial commit
Ondra Urban committed 8 years ago
8dbafe2
Improving documentation
Jan Curn committed 8 years ago
1e758f1
Improving documentation
Jan Curn committed 8 years ago
e026e9d
Updating docs and README.md
Jan Curn committed 8 years ago
616696f
Update changelog; bump version
Ondra Urban committed 8 years ago
f930bd1
Add more tests
Ondra Urban committed 8 years ago
2ee93bf
Merge branch 'develop' into feature/cheerio-crawler
Ondra Urban committed 8 years ago
09dd5e7
Add getInfo() test
Ondra Urban committed 8 years ago
89ab84d
Add getInfo() to dataset
Ondra Urban committed 8 years ago
3d2c3a5
Add test
Ondra Urban committed 8 years ago
8801c01
Add cheerio
Ondra Urban committed 8 years ago
c24d781
Initial commit
Ondra Urban committed 8 years ago
a8774a5
Merge branch 'develop' into feature/sdk-review
Jan Curn committed 8 years ago
a46dbb2
Renaming "act" to "actor"
Jan Curn committed 8 years ago
437a7a6
Writing README.md
Jan Curn committed 8 years ago
3329df7
Writing README.md
Jan Curn committed 8 years ago
52a51f8
Writing README.md
Jan Curn committed 8 years ago
1278bd6
Writing README.md
Jan Curn committed 8 years ago
a58a5e3
Merge branch 'develop' into feature/sdk-review
Jan Curn committed 8 years ago
825deb1
Update changelog; bump version
Ondra Urban committed 8 years ago
b97e9f1
Sanitize regex
Ondra Urban committed 8 years ago
5bac009