Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

6525441
chore(deps): update dependency ubuntu to v26 (#4288)
renovate[bot] committed 20 days ago
774e26e
13e3822
6d8dea6
fe60af7
2369a31
6655629
fix: let the adaptive crawler's navigation window be set and extended (#4219)
Muhammad Haadhee Sheeraz Mian committed 28 days ago
7038553
fix: fetch robots.txt through the crawler's HTTP client like a browser (#4218)
Muhammad Haadhee Sheeraz Mian committed 29 days ago
e4c8964
fix(basic): settle teardown during crawler startup (#4094)
Michael Bujnovský committed 29 days ago
51a74ca
94abff8
ea75707
docs: Add an Adaptive Crawling guide (#4273)
Jan Buchar committed 1 month ago
b564bf9
a839e38
chore(deps): update patch/minor dependencies (#4274)
renovate[bot] committed 1 month ago
700fb95
chore: Re-delete maybeRunIntervalSecs (#4269)
Jan Buchar committed 1 month ago
c69d289
docs: Add a guide on ContextPipeline (#4232)
Jan Buchar committed 1 month ago
4507750