Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

824e7b6
Update Node restrictions to 10.17.0
Ondra Urban committed 7 years ago
2f57964
Remove Node 8 from Travis
Ondra Urban committed 7 years ago
3470d23
Update changelog
Ondra Urban committed 7 years ago
b8ff17a
Bump http-request version
Ondra Urban committed 7 years ago
056d4d6
Attempt to fix flaky tests
Ondra Urban committed 7 years ago
491079f
Rework merging of headers in requestAsBrowser
Ondra Urban committed 7 years ago
b01a15f
8f34d32
99d7c29
fix infinite scroll
Lukáš Křivka committed 7 years ago
c2d6569
builded docs.
Petr Patek committed 7 years ago
3583b56
fixed tests.
Petr Patek committed 7 years ago
7e5f2ea
fixed docs and logs.
Petr Patek committed 7 years ago
2ad497d
a2dc378
75dc619
Update tests
Ondra Urban committed 7 years ago
8945309
13a5bf8
Merge branch 'master' into fix/dataset-limits
Ondra Urban committed 7 years ago
00d5717
281ed78
Build docs
Ondra Urban committed 7 years ago
e2a2ff1
Fix tests
Ondra Urban committed 7 years ago
8dfad23