Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

chore: add log about starting the crawl
Martin Adámek committed 4 years ago
cb85fec
chore(deps): update actions/cache action to v3 (#1386)
renovate[bot] committed 4 years ago
62fd6bf
fix: improve file saving on memory storage (#1421)
Vlad Frangu committed 4 years ago
75dc4bd
chore(deps): update dependency playwright to v1.24.1
renovate[bot] committed 4 years ago
5e6b183
237c188
chore: update changelog
Martin Adámek committed 4 years ago
da67d57
chore: bump version to 3.0.2
Martin Adámek committed 4 years ago
4c9e5c3
fix: declare dependency on `ow` in cheerio package
Martin Adámek committed 4 years ago
be59f99
test: increase timeout before we try to read the stats
Martin Adámek committed 4 years ago
b9d1757
chore: do not log initial state of `AutoscaledPool`
Martin Adámek committed 4 years ago
5c4df89
chore: disable worker for memory storage (#1420)
Vlad Frangu committed 4 years ago
8fbb3e8
ad2dbef
docs: improve upgrading guide
Martin Adámek committed 4 years ago
24b6d2c
docs: improve upgrading guide
Martin Adámek committed 4 years ago
9f5d326
chore: use `^3.0.0` in the templates
Martin Adámek committed 4 years ago
6426f22
chore: try to fix canary builds of templates package
Martin Adámek committed 4 years ago
c2d65a3
57c10df
chore(deps): lock file maintenance
renovate[bot] committed 4 years ago
b4e285c
chore: add `utils.sleep` shortcut
Martin Adámek committed 4 years ago
3a6957d
chore: remove `PlaywrightCookie` type
Martin Adámek committed 4 years ago
7ca4397
docs: update quick start (#1401)
Ondra Urban committed 4 years ago
58ac551
fix: cookie monster is no more (#1414)
Vlad Frangu committed 4 years ago
a361147
chore(deps): update dependency playwright to v1.24.0
renovate[bot] committed 4 years ago
dfa6f1f
chore(deps): update dependency turbo to v1.3.4
renovate[bot] committed 4 years ago
6be9caf