Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: regression in resolving the base url for enqueue link filtering (#1422)
8a00cf6 Vlad Frangu committed 4 years ago
chore: do not enable debug logging in cheerio-ts template
237c188 Martin Adámek committed 4 years ago
feat: add `parseWithCheerio` for puppeteer & playwright (#1418)
02b0aeb Andrey Bykov committed 4 years ago
fix: fix building projects with TS when puppeteer and playwright are not installed (#1404)
3e1218b Martin Adámek committed 4 years ago
fix: enqueue link not finding relative links if the checked page is redirected (#1416)
b78b37b Vlad Frangu committed 4 years ago
test: use `Dataset.pushData` instead of `Actor.pushData` in e2e tests
db50f36 Martin Adámek committed 4 years ago
chore: prefer interfaces over types where possible for better API docs
aad900a Martin Adámek committed 4 years ago