Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat: add `RetryRequestError` + add error to the context for BC (#1443)
1a54171 Vlad Frangu committed 4 years ago
docs: add dom crawler to core section of api docs + add links to core readme
19c9779 Martin Adámek committed 4 years ago
feat: prefer `/INPUT.json` files for `KeyValueStore.getInput()` (#1453)
7ea0552 Vlad Frangu committed 4 years ago
docs: make giscus color mode based on the current one from docusaurus context
bc1ddec Martin Adámek committed 4 years ago
docs: fix giscus resolution being cached (switch to pathname based)
120fad9 Martin Adámek committed 4 years ago
docs: fix word wraps for the tagline on smaller laptop devices
47591ff Martin Adámek committed 4 years ago
docs: use `@apilink` + render package readmes on overview page
61e3964 Martin Adámek committed 4 years ago
docs: add mention about `Request.label` shortcut to the upgrading guide
897cdcf Martin Adámek committed 4 years ago
docs: update ConfigurationOptions description + more (#1437)
f45758d Andrey Bykov committed 4 years ago