Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

8ab5f8d
Merge develop, resolve conflicts
Ondra Urban committed 7 years ago
03cfe4c
Merge branch 'develop'
Ondra Urban committed 7 years ago
1743c3c
Bump version, update Changelog, build docs
Ondra Urban committed 7 years ago
52742bc
Build docs
Ondra Urban committed 7 years ago
f1bc610
Fix lint issues
Ondra Urban committed 7 years ago
c590881
Clean up code, lazy load HTML
Ondra Urban committed 7 years ago
d268300
renamed function
Petr Patek committed 7 years ago
6514251
Fixed API docs link
Jan Curn committed 7 years ago
49ce93a
Try without saving original html
Ondra Urban committed 7 years ago
320301c
First stab at streaming cheerio
Ondra Urban committed 7 years ago
8c609a2
Rename misleading function
Ondra Urban committed 7 years ago
e12c65b
Fix dataset.reduce
Ondra Urban committed 7 years ago
081a097
Remove bluebird dependency
Ondra Urban committed 7 years ago
14de969
Fixed docs for Request.noRetry
Jan Curn committed 7 years ago
065f9aa
Updated log
Jan Curn committed 7 years ago
8674568
Removing outdated comment
Marek Trunkát committed 7 years ago
35a12f4
Added CHECKLIST.md
Jan Curn committed 7 years ago
312594e
Request queue - minor code improvements
Jan Curn committed 7 years ago
a23bb12
Comments
Marek Trunkat committed 7 years ago
a052f03
Fixes from PR
Marek Trunkat committed 7 years ago
95994e8
19bc32e
Better logging of errors in Apify.main()
Jan Curn committed 7 years ago
c789090
Bump apify-client
Ondra Urban committed 7 years ago
971705e
Fix invalid type check
Ondra Urban committed 7 years ago
34e1c3e