Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat: newUrlFunction receives Request and can return false
7bfc15e Jindřich Bär committed 3 years ago
fix: session pool leaks memory on multiple crawler runs (#2083)
b96582a Jindřich Bär committed 3 years ago
docs: add `asrc=run_on_apify` to the runnable example links for tracking
0ef6bb1 Martin Adámek committed 3 years ago
docs: add start:fast script for faster docs development (#2080)
34a6d94 Jindřich Bär committed 3 years ago
chore(release): update internal dependencies [skip ci]
ea7e979 Apify Release Bot committed 3 years ago
fix: various helpers opening KVS now respect Configuration (#2071)
59dbb16 Jindřich Bär committed 3 years ago
feat: remove side effect from the deprecated error context augmentation (#2069)
f9fb5c4 Jindřich Bär committed 3 years ago
fix(core): allow explicit calls to `purgeDefaultStorage` to wipe the storage on each call (#2060)
4831f07 Sviatozar Petrenko committed 3 years ago
chore(deps): update actions/checkout action to v4 (#2064)
e726036 renovate[bot] committed 3 years ago
fix(browser-pool): improve error handling when browser is not found (#2050)
282527f Sviatozar Petrenko committed 3 years ago
feat(core): add default dataset helpers to `BasicCrawler` (#2057)
e2a7544 Jindřich Bär committed 3 years ago