Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Fixing tests
Marek Trunkat committed 8 years ago
b759fe1
Version bump
Marek Trunkat committed 8 years ago
87690f1
Merge pull request #203 from apifytech/fix/pup-pool
Marek Trunkát committed 8 years ago
f0f26f7
Merge branch 'develop' into fix/pup-pool
Marek Trunkát committed 8 years ago
fbe46cc
8b7fec7
Merge branch 'develop' into feature/request_queue_fix
Marek Trunkát committed 8 years ago
6f70a3a
85977b7
cf6cd3e
Upgraded Puppeteer to 1.9.0
Jan Curn committed 8 years ago
2242819
Updated docs
Jan Curn committed 8 years ago
143195f
Updated docs
Jan Curn committed 8 years ago
f431092
Updated docs
Jan Curn committed 8 years ago
4845509
Updated docs
Jan Curn committed 8 years ago
8d3ccbb
Updated docs
Jan Curn committed 8 years ago
8708767
Bump version, update changelog
Ondra Urban committed 8 years ago
b36c8e9
cacheResponses function small improvements
Jaroslav Hejlek committed 8 years ago
fd6b878
6c0fc28
Puppeteer utils: Cache responses
Jaroslav Hejlek committed 8 years ago
d96bcad
Add RequestQueue overload for adding requests
Michael Huynh committed 8 years ago
69e48de
Add regression test to autoscaled pool
Ondra Urban committed 8 years ago
6f89520
Improve test stability
Ondra Urban committed 8 years ago
9d14d9e
Bump version, update changelog
Ondra Urban committed 8 years ago
fa72418
Improve test stability
Ondra Urban committed 8 years ago
41006ba
655b637
5edd1fe