Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Improve docs [skip ci]
Ondra Urban committed 6 years ago
9ca2019
added defaults.
Petr Patek committed 6 years ago
6b3c9a1
fixed the (o)n problem.
Petr Patek committed 6 years ago
98f8461
Update changelog [skip ci]
Ondra Urban committed 6 years ago
64bf3f2
Add desc option to local dataset getData
Ondra Urban committed 6 years ago
689e642
f7fea14
addded todo.
Petr Patek committed 6 years ago
234c33d
implementation proposal.
Petr Patek committed 6 years ago
2327f35
Build readme
Ondra Urban committed 6 years ago
c055469
Add object serialization to Request constructor
Ondra Urban committed 6 years ago
04f4a2f
Update Node 8 to 10 in docs [skip-ci]
Ondra Urban committed 6 years ago
066a082
fixed test
Petr Patek committed 6 years ago
f5d6432
fixed lint
Petr Patek committed 6 years ago
981ddaf
improved naming.
Petr Patek committed 6 years ago
5890bcd
Cheerio ignores ssl errors by default now.
Petr Patek committed 6 years ago
2cf99c1
improved session docs.
Petr Patek committed 6 years ago
4b3be97
fixed docs.
Petr Patek committed 6 years ago
641c4e2
faebf6c
fixed lint
Petr Patek committed 6 years ago
1b48b89
Improved code.
Petr Patek committed 6 years ago
f34b84d
Update changelog, bump version
Ondra Urban committed 6 years ago
b22fee9
Remove Puppeteer pipes, fix docs
Ondra Urban committed 7 years ago
b322203
fixed scroller
Petr Patek committed 7 years ago
caf09a6
d9fd1a4
added docs.
Petr Patek committed 7 years ago
71ba633
added timeout session retire.
Petr Patek committed 7 years ago
0a6d102
fixed lint.
Petr Patek committed 7 years ago
a63b40b