Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: crawler instances with different StorageClients do not affect each other (#2056)
3f4c863 Jindřich Bär committed 3 years ago
fix: respect current config when creating implicit `RequestQueue` instance
845141d Martin Adámek committed 3 years ago
refactor: make `purgeDefaultStorages` accept `options` with `onlyPurgeOnce` (#2044)
8d6946c Sviatozar Petrenko committed 3 years ago
fix: clean up `inProgress` cache when delaying requests via `sameDomainDelaySecs` (#2045)
f63ccc0 Martin Adámek committed 3 years ago
fix: pin `@crawlee/*` packages versions in `crawlee` metapackage (#2040)
61f91c7 Martin Adámek committed 3 years ago
chore(deps): update dependency lint-staged to v14 (#2033)
7f6893c renovate[bot] committed 3 years ago
fix: make the `Request` constructor options typesafe (#2034)
75e7d65 Martin Adámek committed 3 years ago
feat: Add options for custom HTTP error status codes (#2035)
b50ef1a Hamza Alwan committed 3 years ago
feat: exceeding maxSessionRotations calls failedRequestHandler (#2029)
b1cb108 Jindřich Bär committed 3 years ago
fix: add `Request.maxRetries` to the `RequestOptions` interface (#2024)
6433821 Martin Adámek committed 3 years ago