Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Bumped package.json version to 0.6.1
Jan Curn committed 8 years ago
1c84a5f
5c18e1b
Bumped package.json version to 0.6.0
Jan Curn committed 8 years ago
b65ba15
Dropped support for Node 7
Jan Curn committed 8 years ago
5f7aefa
Upgraded NPM dependencies
Jan Curn committed 8 years ago
975a558
779f5cb
71181a9
Removed leftover console.dir()
Jan Curn committed 8 years ago
52e8f90
Fixed usage of ps-tree COMMAND
Jan Curn committed 8 years ago
e241cac
Progress commit
Ondra Urban committed 8 years ago
47858f9
Temporary test
Jan Curn committed 8 years ago
67225b8
Added few comments
Jan Curn committed 8 years ago
441fe09
Fixed lint errors
Jan Curn committed 8 years ago
833d761
Fixed tests for getMemoryInfo() function
Jan Curn committed 8 years ago
c0b0ab0
Improvements in getMemoryInfo() function
Jan Curn committed 8 years ago
03f1b95
Refactor to async await progress commit
Ondra Urban committed 8 years ago
16e894d
update comment
Ondra Urban committed 8 years ago
d83ac04
Move to Node 7.6
Ondra Urban committed 8 years ago
459da8f
Merge branch 'develop' into feature/stop-crawler
Ondra Urban committed 8 years ago
7a394d6
Clean up code
Ondra Urban committed 8 years ago
8c93330
Updated CHANGELOG
Jan Curn committed 8 years ago
60e1a37
Fixed live view port and URL logging
Jan Curn committed 8 years ago
2f3f302
Applied feedback from PR comments
Jan Curn committed 8 years ago
a65753a
Better logging of live view URLs and ports
Jan Curn committed 8 years ago
6c2eb2d
Rename promise
Ondra Urban committed 8 years ago
e08c123