Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat: redesign request timeouts and allow per-route overrides (#3860)
f01b620 Martin Adámek committed 3 months ago
fix(browser-crawler): persist session cookies after requestHandler (#3939)
fb39222 Felix-Ayush committed 3 months ago
fix(core): preserve subclass overrides when binding methods to a scoped service locator (#3951)
9667ab4 Martin Adámek committed 3 months ago
refactor: move async iterable helpers to `@crawlee/core` (#3930)
6bf2dc5 Jindřich Bär committed 3 months ago
fix(core): log a warning on malformed cookie fragments in `mergeCookies` (#3850)
4631399 Jindřich Bär committed 3 months ago
refactor: move buffer and stream helpers to `@crawlee/core` (#3928)
3ac7fc4 Jindřich Bär committed 3 months ago
refactor: drop re-exported types from `@crawlee/utils` (#3931)
fc2930f Jindřich Bär committed 3 months ago
refactor: Introduce IRenderingTypePredictor and unify the ownership pattern (#3944)
9bde1e6 Jan Buchar committed 3 months ago
refactor!: Split off ConcurrencySystem from AutoscaledPool (#3917)
2e70010 Jan Buchar committed 3 months ago
refactor!: remove unused exports from `@crawlee/utils` (#3929)
6be354a Jindřich Bär committed 3 months ago
refactor!: unify `config` and `configuration` naming (#3926)
3471064 Martin Adámek committed 3 months ago
feat: infer router route map on the crawler & type request inputs (#3748)
2070c10 Martin Adámek committed 3 months ago
refactor: Introduce OwnedOrInjected dependency tracking helper (#3891)
ec5ff9b Jan Buchar committed 3 months ago
refactor!: Privatize unintentionally exposed internals (#3893)
a78527d Jan Buchar committed 3 months ago
feat: use crawler id for `Statistics` and non-default `RequestQueue`s (#3834)
bf20a15 Jindřich Bär committed 4 months ago
chore: Exclude internal symbols from public API docs (#3882)
83003fb Jan Buchar committed 4 months ago
refactor!: Remove unnecessary parameters to storage frontend constructors (#3880)
f83b6f0 Jan Buchar committed 4 months ago
test: fix storage pollution in browser crawler tests (#3861)
92a356f Martin Adámek committed 4 months ago
fix: remove noisy implicit configuration warning from service locator (#3837)
ffd1dab Jindřich Bär committed 4 months ago
feat: move autoscaling helpers to `@crawlee/core` (#3835)
e614738 Jindřich Bär committed 4 months ago
chore(ci): fix failing e2e and docs actions after rebase (#3874)
f745f87 Jindřich Bär committed 4 months ago