Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Adding missing await
Marek Trunkat committed 8 years ago
7311797
53e17a5
Rewriting fetchNextRequest() to async/await
Marek Trunkat committed 8 years ago
4b2e97d
Merge pull request #184 from apifytech/fix/max-memory
Marek Trunkát committed 8 years ago
e08cb78
Linting
Marek Trunkat committed 8 years ago
1fb5914
Set null instead of NaN for invalid memory env var
Ondra Urban committed 8 years ago
d833f8d
Update synchronous_run.js
davidjohnbarton committed 8 years ago
5117925
Update puppeteer_sitemap.js
davidjohnbarton committed 8 years ago
dd5db74
Updated hello_world.js and README
Jan Curn committed 8 years ago
d5b7332
Using atomic rename instead of delete+create
Marek Trunkat committed 8 years ago
85f7c0c
Update puppeteer_crawler.js
davidjohnbarton committed 8 years ago
dc9ddc7
Updated README
Jan Curn committed 8 years ago
76c970a
Updated hello_world.js and README
Jan Curn committed 8 years ago
39d66d5
Update call_actor.js
davidjohnbarton committed 8 years ago
eedff7b
Update basic_crawler.js
davidjohnbarton committed 8 years ago
841f091
Update key_value_store.js
davidjohnbarton committed 8 years ago
4e1e286
Fix NaN condition
Ondra Urban committed 8 years ago
3ec02c3
Merge branch 'develop' into fix/max-memory
Ondra Urban committed 8 years ago
94a9a14
Updated README
Jan Curn committed 8 years ago
b52b77d
Updated README
Jan Curn committed 8 years ago
c82845b
Fixing race condition in request queue
Marek Trunkat committed 8 years ago
5320a94
Update README
Ondra Urban committed 8 years ago
1719f67
Update readme
Ondra Urban committed 8 years ago
52f03bc
Merge branch 'develop' into fix/max-memory
Ondra Urban committed 8 years ago
d9b6832
Add memory env var to snapshotter
Ondra Urban committed 8 years ago
e6b5ac0
Fixing readme and docs
Marek Trunkat committed 8 years ago
ace6cb1
Updated README
Jan Curn committed 8 years ago
10b7ec8