Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: try to detect stuck request queue and fix its state (#1837)
95a9f94 Martin Adámek committed 3 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5.2.0
19ad4d7 renovate[bot] committed 3 years ago
docs: add documentation for playwright and puppeteer-extra (#1823)
c96ea11 Vlad Frangu committed 3 years ago
chore(release): update internal dependencies [skip ci]
c1123ba Apify Release Bot committed 3 years ago
feat(core): add `exclude` option to `enqueueLinks` (#1786)
2e833dc Jack Hedaya committed 3 years ago
fix(MemoryStorage): RequestQueue#handledRequestCount should update (#1817)
a775e4a Vlad Frangu committed 3 years ago