Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Build readme
Ondra Urban committed 7 years ago
c30b59f
Improve logging in a test
Ondra Urban committed 7 years ago
50522a2
Update changelog, bump version, build docs
Ondra Urban committed 7 years ago
0f574d0
Publish createRequestDebugInfo in docs
Ondra Urban committed 7 years ago
561ebed
Update nyc to resolve vulnerability
Ondra Urban committed 7 years ago
06a2e4a
Add cache to injectFile
Ondra Urban committed 7 years ago
66ad521
Adding requestId to logs
Marek Trunkat committed 7 years ago
c41b4ac
1df2d44
Updated README
Jan Curn committed 7 years ago
e02165f
Fix RequestList persistence error message
Ondra Urban committed 7 years ago
9cafc15
Update changelog
Ondra Urban committed 7 years ago
f7b41f4
Evaluate concurrently
Ondra Urban committed 7 years ago
fe39606
Fixes from PR and broken tests
Marek Trunkat committed 7 years ago
b57b061
14dfd64
Build docs
Ondra Urban committed 7 years ago
70b42c0
Improve forEach examples in docs
Ondra Urban committed 7 years ago
50d554c
5320362
apify-shared update
Marek Trunkat committed 7 years ago
05b943e
Linting
Marek Trunkat committed 7 years ago
3c8d3ee
Adding getInput()
Marek Trunkat committed 7 years ago
6fe30c6
Improving getEnv()
Marek Trunkat committed 7 years ago
620b610
Update runTaskFunction error message
Ondra Urban committed 7 years ago
155b526
Fix misleading docs
Ondra Urban committed 7 years ago
359080b
Improve docs
Ondra Urban committed 7 years ago
1ad2e69
Make survival optional
Ondra Urban committed 7 years ago
4125182
3edc5b3