Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Fix clicking with POST requests
Ondra Urban committed 7 years ago
9438770
Fix lint and flaky test
Ondra Urban committed 7 years ago
18ca613
Implement and test clicking that opens new tabs
Ondra Urban committed 7 years ago
b857fc0
expanded comments
vdrmota committed 7 years ago
5b1ccf4
fixed lint
vdrmota committed 7 years ago
e436df5
fixed tests
vdrmota committed 7 years ago
058d576
fixed tests
vdrmota committed 7 years ago
2e958a1
options object
vdrmota committed 7 years ago
e5d65fa
b18fd9d
Add forgotten StealthOptions.md
Ondra Urban committed 7 years ago
e6ca6eb
Fix StealthOptions caps that broke the build
Ondra Urban committed 7 years ago
fa9c9e0
Try to see git status in docs
Ondra Urban committed 7 years ago
86e2ef5
Make stages conditional
Ondra Urban committed 7 years ago
ca5a3e8
Add error message to docs build
Ondra Urban committed 7 years ago
87c42cc
Run docs build on Node 10
Ondra Urban committed 7 years ago
fc84ff0
Fix docusaurus install command
Ondra Urban committed 7 years ago
26c7655
lint fixes
vdrmota committed 7 years ago
4cff0ce
Add docusaurus install
Ondra Urban committed 7 years ago
6f902f0
Parallelize lint and tests
Ondra Urban committed 7 years ago
20eb4db
test errors
vdrmota committed 7 years ago
8a5986a
Add a lot of travis configuration
Ondra Urban committed 7 years ago
07a33e1
took out only test constraint
vdrmota committed 7 years ago
769598b
fixed tests and eslint
vdrmota committed 7 years ago
48b5b1e
7ddaf33
added infiniteScroll util
vdrmota committed 7 years ago
d24e7cf
Add publish and tagging to release script
Ondra Urban committed 7 years ago
e1f63f3
Add validation to release script
Ondra Urban committed 7 years ago
c6f7d12
Add update-images script
Ondra Urban committed 7 years ago
e607d71
Better docs
Jan Curn committed 7 years ago
65b15ce