Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: treat `CookieJar` write methods as potentially throwing (#3426)
4738869 Copilot committed 9 months ago
refactor!: Extract service management from Configuration into ServiceLocator class (#3325)
bcde2b0 Jan Buchar committed 9 months ago
fix: rectify transformRequestFunction precedence rules (#3372)
b663940 Richard Solar committed 9 months ago
chore: drop `got-scraping` as the direct `crawlee` dependency (#3358)
f886e46 Jindřich Bär committed 10 months ago
fix: `sendRequest` uses correct `Session` instance and allows for overrides (#3313)
db31875 Jindřich Bär committed 11 months ago
fix: log warning on sharing `useState()`, add `id` option to `BasicCrawler` (#3309)
58eb9fa Jindřich Bär committed 11 months ago
feat: custom `httpClient` implementation for `@crawlee/utils` and `Request` storages (#3306)
b96749d Jindřich Bär committed 11 months ago
fix: patch `GotScrapingHttpClient` header handling and proxies (#3308)
57d1f43 Jindřich Bär committed 11 months ago
feat: drop `gotOptions` params from the public interface (#3300)
f57ff3d Jindřich Bär committed 11 months ago
feat: replace `got`-specific `HttpRequest` with native `Request` interfaces (#3295)
55b4983 Jindřich Bär committed 11 months ago
fix: `KVS.getPublicUrl()` reads the public URL directly from storage client (#3286)
34d579b Jindřich Bär committed 11 months ago
feat!: use native `Request` / `Response` interface (#3163)
32caa0c Jindřich Bär committed 11 months ago
feat: store `ProxyInfo` inside `Session` instances (#3199)
14ed530 Jindřich Bär committed 11 months ago
refactor!: Introduce the `ContextPipeline` abstraction (#3119)
70aa8c9 Jan Buchar committed 12 months ago
chore: replace `lodash.isequal` with `util.isDeepStrictEqual`
7b5ffbe Martin Adámek committed 1 year ago