Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: don't retire browsers with long-running `pre|postLaunchHooks` prematurely (#3062)
681660e Jindřich Bär committed 1 year ago
docs: fix automatic documentation versioning on `minor|major` releases (#3065)
804ee59 Jindřich Bär committed 1 year ago
fix: respect `exclude` option in `enqueueLinksByClickingElements` (#3058)
013eb02 Jindřich Bär committed 1 year ago
fix: retry on blocked status codes in `HttpCrawler` (#3060)
b5fcd79 Jindřich Bär committed 1 year ago
chore(ci): move ZenHub cards for draft PRs to the "In Progress" column (#3059)
e857c17 Jindřich Bär committed 1 year ago
docs: fix `enqueueLinks` documentation on option interactions (#3057)
136e3ad Jindřich Bär committed 1 year ago
chore(release): update internal dependencies [skip ci]
c81d2b4 Apify Release Bot committed 1 year ago
fix: call `onSkippedRequest` for `AdaptivePlaywrightCrawler.enqueueLinks` (#3043)
fc23d34 Jindřich Bär committed 1 year ago
chore(e2e): make `playwright-multi-run` more stable (#3042)
7ed3167 Jindřich Bär committed 1 year ago
chore(e2e): fix indefinite wait in `puppeteer-store-pagination` (#3041)
25a6b60 Jindřich Bär committed 1 year ago
chore(deps): update dependency camoufox-js to ^0.6.0 (#3032)
f1b1f68 renovate[bot] committed 1 year ago