Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

PR fixes
Marek Trunkat committed 7 years ago
093ce7e
f79f13e
added default user agent check.
Petr Patek committed 7 years ago
2a3fba9
Fix latest release condition in Travis
Ondra Urban committed 7 years ago
db971b1
Merging remote
Marek Trunkat committed 7 years ago
47c8a25
updated userAgent.
Petr Patek committed 7 years ago
09e270b
Update docs
Ondra Urban committed 7 years ago
b2d152a
builded docs.
Petr Patek committed 7 years ago
35bbe32
Build docs [skip ci]
Ondra Urban committed 7 years ago
fe83db1
Update travis configuration
Ondra Urban committed 7 years ago
576a86b
builded docs.
Petr Patek committed 7 years ago
d62e184
Added httpRequest v1.0.1 support.
Petr Patek committed 7 years ago
617d5fc
Update changelog [skip ci]
Ondra Urban committed 7 years ago
be3a42e
Update docs [skip ci]
Ondra Urban committed 7 years ago
4780452
d63637e
Merge branch 'master' into fix/anonymize-proxy
Ondra Urban committed 7 years ago
9e33d6d
3f1ab4f
Build docs
Ondra Urban committed 7 years ago
f6b031a
Merge branch 'master' into fix/anonymize-proxy
Ondra Urban committed 7 years ago
5be4919
Build docs
Ondra Urban committed 7 years ago
46e3566
Merge branch 'master' into feature/desired-conc
Ondra Urban committed 7 years ago
a8c7f8b
b81c4ed
Add local warnings to travis scripts
Ondra Urban committed 7 years ago
69aeebe
Update docs
Travis CI committed 7 years ago
4a313e2
Fix some selectors in tutorial
Travis CI committed 7 years ago
69a96c7
added docs for abort function
Petr Patek committed 7 years ago
19fb93f
commited docs.
Petr Patek committed 7 years ago
09887e5
builded docs
Petr Patek committed 7 years ago
b410cf3
renamed browserHeaders to defaultHeaders.
Petr Patek committed 7 years ago
bab497d