Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

chore(deps): lock file maintenance (#1579)
renovate[bot] committed 4 years ago
745b650
88674d6
fix: add retry attempts to methods in CLI (#1588)
Vlad Frangu committed 4 years ago
9142e59
chore(deps): update patch/minor dependencies (#1569)
renovate[bot] committed 4 years ago
976ced9
docs: remove facebook link (#1585)
Jindřich Bär committed 4 years ago
d39146a
chore: fix missing step (#1584)
Vlad Frangu committed 4 years ago
03201c0
334e72d
test: fix PLATFORM e2e tests (#1558)
Szymon Marczak committed 4 years ago
73e105a
736b99b
docs: update homepage features icons (#1574)
Jindřich Bär committed 4 years ago
1540b8f
docs: explain session pool allocation flow (#1572)
Lukáš Křivka committed 4 years ago
1f022f5
refactor: store raw values via `Dataset.exportToValue`
Martin Adámek committed 4 years ago
0c1d1a8
feat: add static `Dataset.exportToValue` (#1564)
Matt Stephens committed 4 years ago
a7c17d4
532a983
chore(deps): lock file maintenance (#1566)
renovate[bot] committed 4 years ago
438bfed
chore: refactor the `enqueueLinks` options type
Martin Adámek committed 4 years ago
26a1a96
chore(deps): update dependency turbo to v1.5.5
renovate[bot] committed 4 years ago
076215e
chore(deps): update patch/minor dependencies
renovate[bot] committed 4 years ago
81af8cb
b32fe04
chore(deps): update dependency turbo to v1.5.3
renovate[bot] committed 4 years ago
5f84642
feat: add `Dataset.exportToValue` (#1553)
Matt Stephens committed 4 years ago
acc6344
chore(deps): update dependency turbo to v1.5.2
renovate[bot] committed 4 years ago
3899301
feat: add `utils.playwright.compileScript` (#1559)
Matt Stephens committed 4 years ago
2e14162
chore(deps): update dependency turbo to v1.5.1
renovate[bot] committed 4 years ago
fbea90e