Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update styfle/cancel-workflow-action action to v0.11.0 (#1591)
28d21d5 renovate[bot] committed 4 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5 (#1590)
54105af renovate[bot] committed 4 years ago
test: run tests in series to get around current flakyness
88674d6 Martin Adámek committed 4 years ago
docs: fix circular Examples category, guide description (#1578)
736b99b Jindřich Bär committed 4 years ago
chore: update dependencies, bump puppeteer support to v18 (#1568)
532a983 Martin Adámek committed 4 years ago
chore(deps): update styfle/cancel-workflow-action action to v0.10.1 (#1565)
1fda447 renovate[bot] committed 4 years ago
chore: make RQ option optional for the context bound `enqueueLinks`
c5a1022 Martin Adámek committed 4 years ago
chore(deps): update amannn/action-semantic-pull-request action to v4.6.0
d52e7fe renovate[bot] committed 4 years ago
docs: refactor crawling context helper methods to have them properly documented (#1550)
35b717b Martin Adámek committed 4 years ago