Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Change Log
All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.
3.18.1 (2026-08-12)
Note: Version bump only for package @crawlee/browser
3.18.0 (2026-08-04)
Bug Fixes
- core: ignore explicitly undefined baseUrl in enqueueLinks helpers (#3924) (a58a9c1)
- core: keep crawler enqueue limits when options carry explicit undefined (#3927) (5f1b0ab), closes #3924 #3924
Features
- opt-in schema validation of request userData per router label (#3851) (c7347f9), closes #3747 #3748 #3747 #3082
- support
puppeteer@25(#3869) (1430062), closes #3772
3.17.0 (2026-06-04)
Bug Fixes
Features
- add
ignoreProxyCertificateoption for the internalproxy-chaininstance (#3418) (02eec66), closes #3369
3.16.0 (2026-02-06)
Performance Improvements
3.15.3 (2025-11-10)
Note: Version bump only for package @crawlee/browser
3.15.2 (2025-10-23)
Note: Version bump only for package @crawlee/browser
3.15.1 (2025-09-26)
Note: Version bump only for package @crawlee/browser
3.15.0 (2025-09-17)
Note: Version bump only for package @crawlee/browser
3.14.1 (2025-08-05)
Note: Version bump only for package @crawlee/browser
3.14.0 (2025-07-25)
Features
3.13.10 (2025-07-09)
Note: Version bump only for package @crawlee/browser
3.13.9 (2025-06-27)
Features
3.13.8 (2025-06-16)
Bug Fixes
- Do not enqueue more links than what the crawler is capable of processing (#2990) (ea094c8), closes #2728
3.13.7 (2025-06-06)
Note: Version bump only for package @crawlee/browser
3.13.6 (2025-06-05)
Note: Version bump only for package @crawlee/browser
3.13.5 (2025-05-20)
Note: Version bump only for package @crawlee/browser
3.13.4 (2025-05-14)
Note: Version bump only for package @crawlee/browser
3.13.3 (2025-05-05)
Note: Version bump only for package @crawlee/browser
3.13.2 (2025-04-08)
Features
3.13.1 (2025-04-07)
Bug Fixes
- don't double increment session usage count in
BrowserCrawler(#2908) (3107e55), closes #2851 - rename
RobotsFiletoRobotsTxtFile(#2913) (3160f71), closes #2910
Features
3.13.0 (2025-03-04)
Note: Version bump only for package @crawlee/browser
3.12.2 (2025-01-27)
Note: Version bump only for package @crawlee/browser
3.12.1 (2024-12-04)
Note: Version bump only for package @crawlee/browser
3.12.0 (2024-11-04)
Note: Version bump only for package @crawlee/browser
3.11.5 (2024-10-04)
Bug Fixes
3.11.4 (2024-09-23)
Note: Version bump only for package @crawlee/browser
3.11.3 (2024-09-03)
Note: Version bump only for package @crawlee/browser
3.11.2 (2024-08-28)
Note: Version bump only for package @crawlee/browser
3.11.1 (2024-07-24)
Note: Version bump only for package @crawlee/browser
3.11.0 (2024-07-09)
Features
- add
iframeexpansion toparseWithCheerioin browsers (#2542) (328d085), closes #2507 - add
ignoreIframesopt-out from the Cheerio iframe expansion (#2562) (474a8dc)
3.10.5 (2024-06-12)
Bug Fixes
- declare missing peer dependencies in
@crawlee/browserpackage (#2532) (3357c7f) - mark
context.request.loadedUrlandidas required inside the request handler (#2531) (2b54660)
3.10.4 (2024-06-11)
Note: Version bump only for package @crawlee/browser
3.10.3 (2024-06-07)
Note: Version bump only for package @crawlee/browser
3.10.2 (2024-06-03)
Note: Version bump only for package @crawlee/browser
3.10.1 (2024-05-23)
Note: Version bump only for package @crawlee/browser
3.10.0 (2024-05-16)
Note: Version bump only for package @crawlee/browser
3.9.2 (2024-04-17)
Note: Version bump only for package @crawlee/browser
3.9.1 (2024-04-11)
Features
3.9.0 (2024-04-10)
Features
tieredProxyUrlsfor ProxyConfiguration (#2348) (5408c7f)- better
newUrlFunctionfor ProxyConfiguration (#2392) (330598b), closes #2348 #2065 - expand #shadow-root elements automatically in
parseWithCheeriohelper (#2396) (a05b3a9)
3.8.2 (2024-03-21)
Note: Version bump only for package @crawlee/browser
3.8.1 (2024-02-22)
Note: Version bump only for package @crawlee/browser
3.8.0 (2024-02-21)
Features
3.7.3 (2024-01-30)
Note: Version bump only for package @crawlee/browser
3.7.2 (2024-01-09)
Note: Version bump only for package @crawlee/browser
3.7.1 (2024-01-02)
Note: Version bump only for package @crawlee/browser
3.7.0 (2023-12-21)
Bug Fixes
Features
- check enqueue link strategy post redirect (#2238) (3c5f9d6), closes #2173
- log cause with
retryOnBlocked(#2252) (e19a773), closes #2249
3.6.2 (2023-11-26)
Note: Version bump only for package @crawlee/browser
3.6.1 (2023-11-15)
Features
3.6.0 (2023-11-15)
Note: Version bump only for package @crawlee/browser
3.5.8 (2023-10-17)
Note: Version bump only for package @crawlee/browser
3.5.7 (2023-10-05)
Note: Version bump only for package @crawlee/browser
3.5.6 (2023-10-04)
Note: Version bump only for package @crawlee/browser
3.5.5 (2023-10-02)
Features
3.5.4 (2023-09-11)
Note: Version bump only for package @crawlee/browser
3.5.3 (2023-08-31)
Bug Fixes
3.5.2 (2023-08-21)
Note: Version bump only for package @crawlee/browser
3.5.1 (2023-08-16)
Bug Fixes
3.5.0 (2023-07-31)
Features
3.4.2 (2023-07-19)
Features
3.4.1 (2023-07-13)
Note: Version bump only for package @crawlee/browser
3.4.0 (2023-06-12)
Bug Fixes
3.3.3 (2023-05-31)
Note: Version bump only for package @crawlee/browser
3.3.2 (2023-05-11)
Note: Version bump only for package @crawlee/browser
3.3.1 (2023-04-11)
Note: Version bump only for package @crawlee/browser
3.3.0 (2023-03-09)
Bug Fixes
3.2.2 (2023-02-08)
Note: Version bump only for package @crawlee/browser
3.2.1 (2023-02-07)
Note: Version bump only for package @crawlee/browser
3.2.0 (2023-02-07)
Bug Fixes
3.1.4 (2022-12-14)
Note: Version bump only for package @crawlee/browser
3.1.3 (2022-12-07)
Note: Version bump only for package @crawlee/browser
3.1.2 (2022-11-15)
Note: Version bump only for package @crawlee/browser
3.1.1 (2022-11-07)
Note: Version bump only for package @crawlee/browser
3.1.0 (2022-10-13)
Note: Version bump only for package @crawlee/browser