Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(puppeteer): replace `page.waitForTimeout()` with `sleep()`
52d7219 Martin Adámek committed 2 years ago
chore(deps): update typescript-eslint monorepo to v7 (#2333)
a8cd603 renovate[bot] committed 2 years ago
fix(puppeteer): add 'process' to the browser bound methods (#2329)
2750ba6 Marc Plouhinec committed 2 years ago
fix: declare missing dependencies on `csv-stringify` and `fs-extra` (#2326)
718959d reda committed 2 years ago
feat: add Sitemap.tryCommonNames to check well known sitemap locations (#2311)
85589f1 Jan Buchar committed 2 years ago
feat: Support plain-text sitemap files (sitemap.txt) (#2315)
0bee7da Jan Buchar committed 2 years ago
feat: accessing crawler state, key-value store and named datasets via crawling context (#2283)
58dd5fc Jan Buchar committed 2 years ago
chore(release): update internal dependencies [skip ci]
57e7063 Apify Release Bot committed 2 years ago