Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(MemoryStorage): always resolve to a new path when batchRemoveFiles is called (#1818)
ef83f4b Vlad Frangu committed 3 years ago
fix(CheerioCrawler): pass ixXml down to response parser (#1807)
af7a5c4 Vlad Frangu committed 3 years ago
fix(MemoryStorage): RequestQueue should respect `forefront` (#1816)
b68e86a Vlad Frangu committed 3 years ago
fix: ignore invalid URLs in `enqueueLinks` in browser crawlers (#1803)
5ac336c robertm-97 committed 3 years ago
feat: move the status message implementation to Crawlee, noop in storage (#1808)
99c3fdc Jindřich Bär committed 3 years ago
chore(MemoryStorage): remove extra `@crawlee/utils` dependency (#1809)
d57580d František Nesveda committed 3 years ago
fix(MemoryStorage): request queues race conditions causing crashes (#1806)
083a9db Vlad Frangu committed 3 years ago
docs: add missing dependency to typescript guide package.json (#1791)
38fa7be Shivam Dua committed 3 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5.1.0
58b1b65 renovate[bot] committed 3 years ago
chore(release): update internal dependencies [skip ci]
9388ab3 Apify Release Bot committed 3 years ago
fix(MemoryStorage): request queues saved in the wrong place (#1779)
19409db Vlad Frangu committed 3 years ago
chore(deps): update patch/minor dependencies to v5.51.0 (#1775)
d82cb19 renovate[bot] committed 3 years ago
chore(deps): update dependency @types/jsdom to v21 (#1770)
cf48373 renovate[bot] committed 3 years ago