Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(release): update internal dependencies [skip ci]
20387a5 Apify Release Bot committed 2 years ago
fix: declare missing peer dependencies in `@crawlee/browser` package (#2532)
3357c7f Benjamin D. Brodie committed 2 years ago
fix: allow creating new adaptive crawler instance without any parameters
9b7f595 Martin Adámek committed 2 years ago
fix: mark `context.request.loadedUrl` and `id` as required inside the request handler (#2531)
2b54660 Martin Adámek committed 2 years ago
fix: fix detection of HTTP site when using the `useState` in adaptive crawler (#2530)
7e195c1 Martin Adámek committed 2 years ago
chore(release): update internal dependencies [skip ci]
b48e0bb Apify Release Bot committed 2 years ago
fix: add `waitForAllRequestsToBeAdded` option to `enqueueLinks` helper
925546b Martin Adámek committed 2 years ago
fix: respect `crawler.log` when creating child logger for `Statistics`
0a0d75d Martin Adámek committed 2 years ago
fix: add missing `useState` implementation into crawling context
eec4a71 Martin Adámek committed 2 years ago
fix(playwright): allow passing new context options in `launchOptions` on type level
0519d40 Martin Adámek committed 2 years ago