Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Tweaking autoscaled pool values
Marek Trunkat committed 8 years ago
3c10dce
Changing maxEventLoopOverloadedRatio to 20%
Marek Trunkat committed 8 years ago
a4b300e
Merge pull request #178 from apifytech/fix/crawlers
Marek Trunkát committed 8 years ago
82f736b
Update basic_crawler.js
Marek Trunkát committed 8 years ago
42e6ede
Update cheerio_crawler.js
davidjohnbarton committed 8 years ago
49b0bc6
Updated docs and README
Jan Curn committed 8 years ago
3f8adb0
Updated docs
Jan Curn committed 8 years ago
88dfd2e
Update basic_crawler.js
davidjohnbarton committed 8 years ago
610c80c
Updated docs
Jan Curn committed 8 years ago
e60a6d5
Update actor.js
davidjohnbarton committed 8 years ago
cc7974d
Fix unstable test
Ondra Urban committed 8 years ago
a8caed4
Updated docs
Jan Curn committed 8 years ago
4c1a4ce
Updated README
Jan Curn committed 8 years ago
5a47bfe
Merge branch 'master' into develop
Jan Curn committed 8 years ago
c20f872
Update synchronous_run.js
davidjohnbarton committed 8 years ago
ee71fa5
Update screenshots.js
davidjohnbarton committed 8 years ago
03c56be
Update puppeteer_proxy.js
davidjohnbarton committed 8 years ago
af6900b
Update cheerio_crawler.js
davidjohnbarton committed 8 years ago
25bc9b2
Update puppeteer_crawler.js
davidjohnbarton committed 8 years ago
3c5f23b
Update call_actor.js
davidjohnbarton committed 8 years ago
ef6f7fe
Update basic_crawler.js
davidjohnbarton committed 8 years ago
0a8140c
Update CHANGELOG.md
davidjohnbarton committed 8 years ago
ab04a1a
Update README.md
davidjohnbarton committed 8 years ago
4b33656
Update README.md
davidjohnbarton committed 8 years ago
81943b5
Update README.md
davidjohnbarton committed 8 years ago
0a8ec21
Update changelog
Ondra Urban committed 8 years ago
5ac8a0b
Feature/better docs2 (#175)
Jan Čurn committed 8 years ago
1bbd732
Fix autoscaled pool (#165)
Ondra Urban committed 8 years ago
65ff48e
Update CHANGELOG.md
Marek Trunkát committed 8 years ago
3133a4a