Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Improve injectFile, add test
Ondra Urban committed 7 years ago
a3853e2
c786c12
Updated log messages
Jan Curn committed 7 years ago
71813ab
Updated log message
Jan Curn committed 7 years ago
0bfce1f
e96432e
Build docs
Ondra Urban committed 7 years ago
987b8dc
Fix pseudo URL docs
Ondra Urban committed 7 years ago
c2897a3
Fixed broken tests
Jan Curn committed 7 years ago
ebcc7d9
Improved pseudo-URL docs
Jan Curn committed 7 years ago
cd0c219
Improved pseudo-URL docs
Jan Curn committed 7 years ago
ded6f94
Fixed lint error
Jan Curn committed 7 years ago
9b37b55
Updated DEFAULT_USER_AGENT and USER_AGENT_LIST
Jan Curn committed 7 years ago
d457222
Updated comments
Jan Curn committed 7 years ago
9e23bec
Upgraded to [email protected]
Jan Curn committed 7 years ago
242216f
Upgraded to Puppeteer version 1.12.2
Jan Curn committed 7 years ago
873edce
Updated CHANGELOG.md
Jan Curn committed 7 years ago
def6d11
56a8504
Add missing ignore tag
Ondra Urban committed 7 years ago
f4c9af1
Remove unneeded package and extra line in CHANGELOG
Ondra Urban committed 7 years ago
e7dfe5b
Make changes
Ondra Urban committed 7 years ago
f77e34f
Merge branch 'develop' into AaronTJDev-master
Ondra Urban committed 7 years ago
de268da
Fix merge conflicts
Ondra Urban committed 7 years ago
9627b2f