Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

feat: add `utils.parseOpenGraph()` (#1555)
Matt Stephens committed 4 years ago
059f85e
feat: enqueue links by clicking for Playwright (#1545)
Jindřich Bär committed 4 years ago
3d25ade
feat: add global `useState` helper (#1551)
Matt Stephens committed 4 years ago
2b03177
chore(deps): update dependency playwright to v1.26.0
renovate[bot] committed 4 years ago
be701ea
chore: small tweaks to templates [skip ci] (#1554)
Vlad Frangu committed 4 years ago
8724a2f
docs: new GitHub issue forms (#1552)
Vlad Frangu committed 4 years ago
0aeca50
5d37cf1
1c8cd82
feat: add `utils.playwright.saveSnapshot` (#1544)
Jindřich Bär committed 4 years ago
a4ceef0
feat: add `utils.playwright.infiniteScroll` (#1543)
Jindřich Bär committed 4 years ago
60c8289
8c87390
19cdf13
chore(deps): update dependency turbo to v1.4.7
renovate[bot] committed 4 years ago
0ba204d
chore(deps): lock file maintenance
renovate[bot] committed 4 years ago
bd37cf9
feat: allow disabling storage persistence (#1539)
Vlad Frangu committed 4 years ago
f65e3c6
3ff398a
feat: use JSON5 for INPUT.json (#1538)
Szymon Marczak committed 4 years ago
09133ff
chore(deps): update npm to v8.19.2
renovate[bot] committed 4 years ago
8225597
chore(deps): lock file maintenance
renovate[bot] committed 4 years ago
0acf6b1
87e05ab
docs: add playwright + firefox example (#1535)
Andrey Bykov committed 4 years ago
3f6fc92
chore(deps): lock file maintenance
renovate[bot] committed 4 years ago
776c161
7b33b0a
fix: export `Cookie` from `crawlee` metapackage
Martin Adámek committed 4 years ago
7b02ceb
refactor: error analysis message (#1534)
Szymon Marczak committed 4 years ago
c0b6ee3
9e96af8