Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

0a29d4d
Fix lint issue
Ondra Urban committed 8 years ago
8301ee3
Optimize isTaskReadyFunction
Ondra Urban committed 8 years ago
b5d55bd
Reduce clutter in test logs
Ondra Urban committed 8 years ago
53216e6
Remove wrongly commited task
Ondra Urban committed 8 years ago
4374c9d
Make test stable
Ondra Urban committed 8 years ago
bc3fc35
Speed up CheerioCrawler tests
Ondra Urban committed 8 years ago
8bb698b
Fix typo
Ondra Urban committed 8 years ago
daab9d3
Refactor .stop() to .abort()
Ondra Urban committed 8 years ago
eccd8aa
Add deprecation warning to pageOpsTimeoutMillis
Ondra Urban committed 8 years ago
6463b53
Fix merge conflicts
Ondra Urban committed 8 years ago
b735347
Update changelog
Ondra Urban committed 8 years ago
ce8cc84
Add log to Apify.utils
Ondra Urban committed 8 years ago
eb8c58a
Fix tests
Ondra Urban committed 8 years ago
4349f65
Refactoring to improve reliability of tests
Ondra Urban committed 8 years ago
ea319d0
Fix merge conflicts
Ondra Urban committed 8 years ago
e24c86f
Update changelog
Ondra Urban committed 8 years ago
109642a
Improve link in comment
Ondra Urban committed 8 years ago
139f8fd
Fix lint issue
Ondra Urban committed 8 years ago
b02cf06
Improve docs
Ondra Urban committed 8 years ago
9cb7863
Fix merge conflicts
Ondra Urban committed 8 years ago
3ac7a59
Add keepDuplicateUrls option
Ondra Urban committed 8 years ago
fc59f98
Fix lint issue
Ondra Urban committed 8 years ago
a64d3d7
Fix merge conflicts
Ondra Urban committed 8 years ago
123308b
BasicCrawler done
Ondra Urban committed 8 years ago
72b4c1b
Fixed lint errors
Jan Curn committed 8 years ago
57e764e