Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): dependency sweep — lockfile freshening (#4324)
6525441 apify-deps[bot] committed 20 days ago
chore(deps): update dependency header-generator>ow to v3 (#4206)
0bd662a renovate[bot] committed 20 days ago
fix(browser-pool): use protocol+host instead of url.origin for SOCKS proxy server (#4290)
6911e6b SwastikTripathi committed 21 days ago
fix: keep the adaptive crawler's detection rerun from seeing afterStorageCommit writes (#4310)
79ae888 Nikita Chapovskii committed 22 days ago
fix(core): read crawlee.json saved with a UTF-8 BOM (#4307)
2369a31 Wei-Lun Li committed 23 days ago
fix: Track modifications to crawlingContext.request in adaptive crawler (#4296)
6c336ab Jan Buchar committed 24 days ago
fix(core): keep descending memory dataset pages from repeating newer items (#4301)
912e7e3 Raphael Fakhri committed 25 days ago
fix: keep the impit client from timing out before the crawler's deadline (#4215)
6655629 Muhammad Haadhee Sheeraz Mian committed 28 days ago
fix: let the adaptive crawler's navigation window be set and extended (#4219)
7038553 Muhammad Haadhee Sheeraz Mian committed 28 days ago
fix: fetch robots.txt through the crawler's HTTP client like a browser (#4218)
e4c8964 Muhammad Haadhee Sheeraz Mian committed 29 days ago
fix(basic): settle teardown during crawler startup (#4094)
51a74ca Michael Bujnovský committed 29 days ago
docs: add Browser Use to remote browser services guide (#4292)
94abff8 Gregor Žunič committed 1 month ago
ci: call the apify-factory review through apify/workflows (#4287)
45e7e89 Martin Forejt committed 1 month ago
chore(deps): update opentelemetry-js monorepo to ^0.223.0 (#4227)
11b4618 renovate[bot] committed 1 month ago
fix: Make response and enqueueLinks in adaptive crawler context consistent (#4279)
a2029cd Jan Buchar committed 1 month ago
chore(deps): update dependency camoufox-js to ^0.13.0 (#4285)
a839e38 renovate[bot] committed 1 month ago
docs: add Jooble mention to Gradcracker scraper article (#4286)
7cc71c3 Saurav Jain committed 1 month ago
feat: adaptive crawler result checks get request context, rejections feed the predictor (#4272)
0289924 Jan Buchar committed 1 month ago
feat: unify querySelectorAll/querySelectorOne across crawlers (#4271)
eaec7ae Jan Buchar committed 1 month ago
fix: prefer ImpitHttpClient browser over session fingerprint (#4262)
ea99e2f Jindřich Bär committed 1 month ago
docs: Document running multiple crawlers in the same process (#4257)
346601a Jan Buchar committed 1 month ago