Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
ci: replace `playwright-github-action` with `playwright install --with-deps` (#2812)
b4894a4 Martin Adámek committed 1 year ago
chore(release): update internal dependencies [skip ci]
0b55335 Apify Release Bot committed 1 year ago
docs: Fix syntax and wording in google maps scraping blog (#2822)
16403da Jan Buchar committed 1 year ago
feat: stopping the crawlers gracefully with `BasicCrawler.stop()` (#2792)
af2966f Jindřich Bär committed 1 year ago
fix: graceful `BasicCrawler` tidy-up on `CriticalError` (#2817)
53331e8 Jindřich Bär committed 1 year ago
chore(ci): pin Ubuntu 22.04 because of Playwright dependencies (#2816)
7877cc6 Jindřich Bär committed 1 year ago
docs: more details about different `ProxyConfiguration` options (#2793)
8dc3368 Jindřich Bär committed 1 year ago
chore(deps): update dependency eslint-config-prettier to v10 (#2809)
02a317a renovate[bot] committed 1 year ago
docs: Add Crawlee for Python v0.5 release blog post (#2804)
dff07b9 Vlada Dusek committed 1 year ago
fix: destructure `CrawlerRunOptions` before passing them to `addRequests` (#2803)
02a598c Jindřich Bär committed 1 year ago