Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

7a4117e
Updated README.md
Jan Curn committed 9 years ago
b57222d
Updated README.md
Jan Curn committed 9 years ago
1e86c41
Updated README.md
Jan Curn committed 9 years ago
08a623b
Updated URLs to GitHub repo
Jan Curn committed 9 years ago
cebce6b
Improvements in logging
Jan Curn committed 9 years ago
58353ea
Bumped package.json version
Jan Curn committed 9 years ago
0172e76
Temporaroly enabled selenium logging
Jan Curn committed 9 years ago
75a0438
Replaced ProxyChainManager with ProxyChain
Jan Curn committed 9 years ago
7b91390
361d31c
d010d57
192d200
Updated Squid install in .travis.yml
Jan Curn committed 9 years ago
d9d3c5e
Added Squid install to .travis.yml
Jan Curn committed 9 years ago
0ddfa11
Developmed of Squid proxy chain and unit tests
Jan Curn committed 9 years ago
3feed04
Developmed of Squid proxy chain and unit tests
Jan Curn committed 9 years ago
88c5d35
Developmed of Squid proxy chain
Jan Curn committed 9 years ago
c15e1bf
Merge branch 'master' into feature/proxy
Jan Curn committed 9 years ago
de11caf
Developmed of Squid proxy chain
Jan Curn committed 9 years ago
af7b1cc