Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

rename quality => screenshotQuality
metalwarrior665 committed 7 years ago
3d4bfba
fix retryHistogram for RequestList when URL is the same
metalwarrior665 committed 7 years ago
e33cd25
Updating docs
Marek Trunkat committed 7 years ago
943c419
added quality option and changed from png to jpeg
metalwarrior665 committed 7 years ago
edf3406
renamed storeName option to keyValueStoreName
metalwarrior665 committed 7 years ago
c38f4fb
8fbea91
Update package.json
Ondra Urban committed 7 years ago
1d5b4f9
Lint fix
Ondra Urban committed 7 years ago
fa40be1
Migrate all tests to Jest
Ondra Urban committed 7 years ago
1027a34
added documentation for new country parameter.
Petr Patek committed 7 years ago
e8a0393
a840d08
enhanced options for utils.puppeteer.saveSnapshot
metalwarrior665 committed 7 years ago
b25c869
Added SessionPool teardown. Fixed tests.
Petr Patek committed 7 years ago
813bea9
Remove chai.use from tests
Ondra Urban committed 7 years ago
866cfc5
Restructure test folder
Ondra Urban committed 7 years ago
9a51af4
Codemod tests
Ondra Urban committed 7 years ago
ecdfe4c
Install Jest
Ondra Urban committed 7 years ago
a188ee7
Bump version to release new beta
Ondra Urban committed 7 years ago
5e25dd6
Update changelog [skip ci]
Ondra Urban committed 7 years ago
0bd6135
Fix minor issues
Ondra Urban committed 7 years ago
21ac525
75d0050
Merge branch 'master' into feature/call-task-458
Ondra Urban committed 7 years ago
d2583a4
be371ca
67f4db9
45a420d
Update docs [skip ci]
Ondra Urban committed 7 years ago
ac93361