Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
test: fix e2e tests after yet another store redesign (#2402)
3559436 Martin Adámek committed 2 years ago
feat: expand #shadow-root elements automatically in `parseWithCheerio` helper (#2396)
a05b3a9 Martin Adámek committed 2 years ago
feat: better `newUrlFunction` for ProxyConfiguration (#2392)
330598b Jindřich Bär committed 2 years ago
chore(deps): update actions/configure-pages action to v5 (#2397)
be7dec4 renovate[bot] committed 2 years ago
fix: notify autoscaled pool about newly added requests (#2400)
a90177d Martin Adámek committed 2 years ago
fix(puppeteer): allow passing `networkidle` to `waitUntil` in `gotoExtended` (#2399)
5d0030d Martin Adámek committed 2 years ago