Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Update changelog, bump version
Ondra Urban committed 7 years ago
f35091c
Fix request interception for POST requests
Ondra Urban committed 7 years ago
0c33b19
Update changelog, bump version, build docs
Ondra Urban committed 7 years ago
a59afd4
Add throttling to critical memory overload warnings
Ondra Urban committed 7 years ago
cc02840
Update changelog
Ondra Urban committed 7 years ago
97923b7
Add throttling to critical memory overload warnings
Ondra Urban committed 7 years ago
d6ef87c
Fix teardown in Snapshotter
Ondra Urban committed 7 years ago
04aad84
Improve Statistics a bit
Ondra Urban committed 7 years ago
4db41dc
Fix flaky test
Ondra Urban committed 7 years ago
fcd599e
Serve live view on finished navigation
Ondra Urban committed 7 years ago
8519962
Fix docs
Ondra Urban committed 7 years ago
febb4dc
Fix docs
Ondra Urban committed 7 years ago
3bc0c8d
Update changelog, bump version, build docs
Ondra Urban committed 7 years ago
1693b77
be9f13e
e482739
Fix typo in docs
Ondra Urban committed 7 years ago
1b66a39
Build docs
Ondra Urban committed 7 years ago
98ffe25
Updated docs
Jan Curn committed 7 years ago
96d8971
Add statistics logging test
Ondra Urban committed 7 years ago
e121aca
Add basic docs
Ondra Urban committed 7 years ago
b61e45e
Add Statistics tests
Ondra Urban committed 7 years ago
aa3ecf5
Fixed doc, unwind should be string
drobnikj committed 7 years ago
6016662
Merge branch 'develop' into feature/crawler-stats
Ondra Urban committed 7 years ago
696def8