Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

chore(release): update internal dependencies [skip ci]
Apify Release Bot committed 2 years ago
a93da6c
chore(release): v3.7.1
Apify Release Bot committed 2 years ago
608c884
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
dbb7aec
f78137d
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
20191d9
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
5d40f13
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
418b04d
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
7ebd0bd
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
5bee948
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
57f1671
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
2570388
c2eeb94
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
668283e
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
d43124a
docs: add missing files for previous commit
Martin Adámek committed 2 years ago
0f783b3
docs: add v3.7 docs snapshot
Martin Adámek committed 2 years ago
745f0d9
chore(docker): update docker state [skip ci]
Apify Release Bot committed 2 years ago
a9dc7fe
chore(release): update internal dependencies [skip ci]
Apify Release Bot committed 2 years ago
22fb053
chore(release): v3.7.0
Apify Release Bot committed 2 years ago
3641c87
feat: log cause with `retryOnBlocked` (#2252)
Jindřich Bär committed 2 years ago
e19a773
3c5f9d6
feat: allow configuring crawler statistics (#2213)
Sviatozar Petrenko committed 2 years ago
9fd60e4
83f3179
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
4918f26
feat: robots.txt and sitemap.xml utils (#2214)
Jan Buchar committed 2 years ago
fdfec4f
chore(deps): lock file maintenance
renovate[bot] committed 2 years ago
c1cfd45
8dfaf95