Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(contributing): add loopback instructions for macOS
b818033 Szymon Marczak committed 4 years ago
feat: add `set` and `useStorageClient` shortcut to `Configuration`
2e66fa2 Martin Adámek committed 4 years ago
chore: re-export package.json from the core module (#1654)
74a1948 Vlad Frangu committed 4 years ago
ci: trigger docker image builds after publishing new versions (#1653)
f7ad886 Vlad Frangu committed 4 years ago
chore: do not write files to disk when running tests (#1645)
4d96b09 Vlad Frangu committed 4 years ago
fix: `utils.playwright.blockRequests` warning message (#1632)
76549eb Jindřich Bär committed 4 years ago
docs: fix gif leaking desktop, being too compressed, and showing a random `;` (#1625)
a6d8086 Vlad Frangu committed 4 years ago
fix: type errors when `playwright` is not installed (#1637)
de9db0c Szymon Marczak committed 4 years ago