Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(release): update internal dependencies [skip ci]
6141fa5 Apify Release Bot committed 2 years ago
fix: decode special characters in proxy `username` and `password` (#2696)
0f0fcc5 Martin Adámek committed 2 years ago
fix: check `.isFinished()` before `RequestList` reads (#2695)
6fa170f Jindřich Bär committed 2 years ago
fix: `prolong-` and `deleteRequestLock` `forefront` option (#2690)
cba8da3 Jindřich Bär committed 2 years ago
fix: respect `forefront` option in `MemoryStorage`'s `RequestQueue` (#2681)
b0527f9 Jindřich Bär committed 2 years ago
fix(http-crawler): avoid crashing when gotOptions.cache is on (#2686)
1106d3a Kaffi Y. committed 2 years ago
chore: fix dependency updates stuck because of `stream-chain`
a453e58 Martin Adámek committed 2 years ago
fix(core): accept `UInt8Array` in `KVS.setValue()` (#2682)
8ef0e60 Martin Adámek committed 2 years ago
fix(core): trigger `errorHandler` for session errors (#2683)
7d72bcb Martin Adámek committed 2 years ago
fix(puppeteer): rename `ignoreHTTPSErrors` to `acceptInsecureCerts` to support v23 (#2684)
f3927e6 Martin Adámek committed 2 years ago
chore(release): update internal dependencies [skip ci]
11efcc2 Apify Release Bot committed 2 years ago
chore(deps): update dependency express to v4.20.0 [security]
2c09ace renovate[bot] committed 2 years ago
chore(deps): update dependency body-parser to v1.20.3 [security]
6dc9780 renovate[bot] committed 2 years ago