Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
docs: try to implement redirects for URLs with trailing slash so they dont flash 404
1ba5587 Martin Adámek committed 4 years ago
fix: key value stores emitting an error when multiple write promises ran in parallel (#1460)
f201cca Vlad Frangu committed 4 years ago
fix: respect `headless` option in browser crawlers (#1455)
46f6e4c Martin Adámek committed 4 years ago
chore: use `npm start` in log after template gets created
89afbbb Martin Adámek committed 4 years ago