Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Remove critical memory warning
Ondra Urban committed 7 years ago
183554f
Update changelog
Ondra Urban committed 7 years ago
1efcf4a
Bump version, update Changelog, build docs
Ondra Urban committed 7 years ago
dc56eda
4ec9646
9785c47
Decrease live view snapshot frequency
Ondra Urban committed 7 years ago
130e62d
Merge branch 'develop' into feature/live-view-ui
Ondra Urban committed 7 years ago
c9d3564
fixed window emulation test.
Petr Patek committed 7 years ago
e189765
d371f07
Increase live view screenshot timeout
Ondra Urban committed 7 years ago
0fc5750
Improved documentation.
Petr Patek committed 7 years ago
c30e327
Added iframe test.
Petr Patek committed 7 years ago
550df1a
Better log message
Jan Curn committed 7 years ago
1cbcd74
Remove full page screenshots
Ondra Urban committed 7 years ago
7dc46db
Fix lint issue
Ondra Urban committed 7 years ago
391de40
Move live view timeout to LiveViewServer
Ondra Urban committed 7 years ago
add6c74
357423f
Change live view snapshotting to time based
Ondra Urban committed 7 years ago
a368384
Improve messages
Ondra Urban committed 7 years ago
3cbc987
Fix socket connection
Ondra Urban committed 7 years ago
fe87d7d
Fix socket connection
Ondra Urban committed 7 years ago
2767933
Fix socket connection
Ondra Urban committed 7 years ago
d0b6ac4
Improve path
Ondra Urban committed 7 years ago
b710b06
improved comments and puppeteer initialization.
Petr Patek committed 7 years ago
76da5ff
Add path to socket connections
Ondra Urban committed 7 years ago
33a8be9