Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Update index.js
Ondra Urban committed 8 years ago
112c709
Remove request-promise specific options
Ondra Urban committed 8 years ago
f369215
Fix unnecessary response reads
Ondra Urban committed 8 years ago
858dd67
Build docs
Ondra Urban committed 8 years ago
5eef3fc
Finalize tests
Ondra Urban committed 8 years ago
dda38ee
8fd5fcb
Add more tests
Ondra Urban committed 8 years ago
d52f4f9
a4bf69b
Last few changes of callTask() and call()
Marek Trunkat committed 8 years ago
45ffc3c
Rewriting call() and adding callTask()
Marek Trunkat committed 8 years ago
eab0bb5
Adding setMin/MaxConcurrency()
Marek Trunkat committed 8 years ago
284f5a8
Adding switch enforcing remote store to be used
Marek Trunkat committed 8 years ago
12b18de
Merge branch 'develop' into feature/copywriting
Marek Trunkát committed 8 years ago
adb012e
Typo
Marek Trunkat committed 8 years ago
3d466a8
Fixing error message when puppeteer is not found
Marek Trunkat committed 8 years ago
317d0c0
Adding handledCount() method to RequestList
Marek Trunkat committed 8 years ago
9478a09
Merge branch 'develop' into feature/renamed-req-op-info
Marek Trunkát committed 8 years ago
538accf
Documentation links fix
Vratislav Bartonicek committed 8 years ago
54e056d
Fix current tests
Ondra Urban committed 8 years ago
40772ab
Fix requestFunction
Ondra Urban committed 8 years ago
f51fb7b
Progress commit
Ondra Urban committed 8 years ago
e68cc2c
Broken link fix
Vratislav Bartonicek committed 8 years ago
6e2d0c8
Fix frontpage
Ondra Urban committed 8 years ago
b1a26f6
a39e307
Merge branch 'develop'
Ondra Urban committed 8 years ago
45a30a9
Build docs
Ondra Urban committed 8 years ago
a9e42f3
Initial commit
Ondra Urban committed 8 years ago
43fba73
Docs homepage broken links fix
Vratislav Bartonicek committed 8 years ago
6573116
e647d04