Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

fd85c57
0689b91
Merge branch 'develop' into fix/pool-reject
Ondra Urban committed 7 years ago
df65d03
Add missing comment
Ondra Urban committed 7 years ago
3ea7387
Unflake some tests
Ondra Urban committed 7 years ago
7860882
Fix some flaky tests
Ondra Urban committed 7 years ago
50ee221
Fix almost all hanging tests
Ondra Urban committed 7 years ago
2b4d635
Rework timeouts, fix some hanging tests
Ondra Urban committed 7 years ago
637237a
Better message in ApifyCallError
Jan Curn committed 7 years ago
7faa209
Implement comments
Ondra Urban committed 7 years ago
de601dd
Merge branch 'develop' into feature/reuse-tabs
Ondra Urban committed 7 years ago
a7c3f04
Implement comments
Ondra Urban committed 7 years ago
3155f5e
95ffdc4
Implement, add tests
Ondra Urban committed 7 years ago
3eb34b0
Small fix in CheerioCrawler arguments
Jan Curn committed 7 years ago
5ca6ba5
Improved docs
Jan Curn committed 7 years ago
478a018
Improved docs
Jan Curn committed 7 years ago
bd492e1
Improved docs
Jan Curn committed 7 years ago
b219994
Improved docs
Jan Curn committed 7 years ago
877ae04
Improve example docs
Ondra Urban committed 7 years ago
49b723b
Improve docs
Ondra Urban committed 7 years ago
1d3a591
Update changelog
Ondra Urban committed 7 years ago
31d8147
Implement, add tests
Ondra Urban committed 7 years ago
de5b915
Update changelog
Ondra Urban committed 7 years ago
a0c8cc5
Add isFinishedFunction logs
Ondra Urban committed 7 years ago
b3b4438
Update changelog
Ondra Urban committed 7 years ago
8c081f1
Fix lint issue
Ondra Urban committed 7 years ago
e9c0c46
Implement local support
Ondra Urban committed 7 years ago
c67354c
Fix tests
Ondra Urban committed 7 years ago
a4af916