Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Change Log
All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.
3.18.1 (2026-08-12)
Note: Version bump only for package @crawlee/http
3.18.0 (2026-08-04)
Bug Fixes
Features
- opt-in schema validation of request userData per router label (#3851) (c7347f9), closes #3747 #3748 #3747 #3082
- type-safe router labels via per-label userData map (#3747) (1124aca), closes #3082
3.17.0 (2026-06-04)
Features
3.16.0 (2026-02-06)
Performance Improvements
3.15.3 (2025-11-10)
Note: Version bump only for package @crawlee/http
3.15.2 (2025-10-23)
Note: Version bump only for package @crawlee/http
3.15.1 (2025-09-26)
Note: Version bump only for package @crawlee/http
3.15.0 (2025-09-17)
Note: Version bump only for package @crawlee/http
3.14.1 (2025-08-05)
Note: Version bump only for package @crawlee/http
3.14.0 (2025-07-25)
Bug Fixes
- retry on blocked status codes in
HttpCrawler(#3060) (b5fcd79), closes /github.com/apify/crawlee/blob/f68d2a95d67cc6230122dc1a5226c57ca23d0ae7/packages/browser-crawler/src/internals/browser-crawler.ts#L481-L486 #3029
3.13.10 (2025-07-09)
Note: Version bump only for package @crawlee/http
3.13.9 (2025-06-27)
Features
3.13.8 (2025-06-16)
Note: Version bump only for package @crawlee/http
3.13.7 (2025-06-06)
Note: Version bump only for package @crawlee/http
3.13.6 (2025-06-05)
Bug Fixes
3.13.5 (2025-05-20)
Features
3.13.4 (2025-05-14)
Note: Version bump only for package @crawlee/http
3.13.3 (2025-05-05)
Features
3.13.2 (2025-04-08)
Note: Version bump only for package @crawlee/http
3.13.1 (2025-04-07)
Bug Fixes
Features
3.13.0 (2025-03-04)
Note: Version bump only for package @crawlee/http
3.12.2 (2025-01-27)
Note: Version bump only for package @crawlee/http
3.12.1 (2024-12-04)
Note: Version bump only for package @crawlee/http
3.12.0 (2024-11-04)
Features
3.11.5 (2024-10-04)
Bug Fixes
3.11.4 (2024-09-23)
Note: Version bump only for package @crawlee/http
3.11.3 (2024-09-03)
Note: Version bump only for package @crawlee/http
3.11.2 (2024-08-28)
Note: Version bump only for package @crawlee/http
3.11.1 (2024-07-24)
Note: Version bump only for package @crawlee/http
3.11.0 (2024-07-09)
Note: Version bump only for package @crawlee/http
3.10.5 (2024-06-12)
Bug Fixes
3.10.4 (2024-06-11)
Bug Fixes
3.10.3 (2024-06-07)
Features
3.10.2 (2024-06-03)
Note: Version bump only for package @crawlee/http
3.10.1 (2024-05-23)
Note: Version bump only for package @crawlee/http
3.10.0 (2024-05-16)
Features
3.9.2 (2024-04-17)
Note: Version bump only for package @crawlee/http
3.9.1 (2024-04-11)
Note: Version bump only for package @crawlee/http
3.9.0 (2024-04-10)
Features
3.8.2 (2024-03-21)
Note: Version bump only for package @crawlee/http
3.8.1 (2024-02-22)
Note: Version bump only for package @crawlee/http
3.8.0 (2024-02-21)
Note: Version bump only for package @crawlee/http
3.7.3 (2024-01-30)
Note: Version bump only for package @crawlee/http
3.7.2 (2024-01-09)
Note: Version bump only for package @crawlee/http
3.7.1 (2024-01-02)
Note: Version bump only for package @crawlee/http
3.7.0 (2023-12-21)
Features
- check enqueue link strategy post redirect (#2238) (3c5f9d6), closes #2173
- log cause with
retryOnBlocked(#2252) (e19a773), closes #2249
3.6.2 (2023-11-26)
Note: Version bump only for package @crawlee/http
3.6.1 (2023-11-15)
Note: Version bump only for package @crawlee/http
3.6.0 (2023-11-15)
Bug Fixes
Features
3.5.8 (2023-10-17)
Note: Version bump only for package @crawlee/http
3.5.7 (2023-10-05)
Note: Version bump only for package @crawlee/http
3.5.6 (2023-10-04)
Note: Version bump only for package @crawlee/http
3.5.5 (2023-10-02)
Note: Version bump only for package @crawlee/http
3.5.4 (2023-09-11)
Note: Version bump only for package @crawlee/http
3.5.3 (2023-08-31)
Bug Fixes
3.5.2 (2023-08-21)
Bug Fixes
Features
3.5.1 (2023-08-16)
Bug Fixes
3.5.0 (2023-07-31)
Features
3.4.2 (2023-07-19)
Features
3.4.1 (2023-07-13)
Bug Fixes
- http-crawler: replace
IncomingMessagewithPlainResponsefor context'sresponse(#1973) (2a1cc7f), closes #1964
3.4.0 (2023-06-12)
Note: Version bump only for package @crawlee/http
3.3.3 (2023-05-31)
Note: Version bump only for package @crawlee/http
3.3.2 (2023-05-11)
Features
- HttpCrawler: add
parseWithCheeriohelper toHttpCrawler(#1906) (ff5f76f) - router: allow inline router definition (#1877) (2d241c9)
3.3.1 (2023-04-11)
Bug Fixes
- jsdom: use no-op
enqueueLinksin http crawlers when parsing fails (fd35270)
3.3.0 (2023-03-09)
Note: Version bump only for package @crawlee/http
3.2.2 (2023-02-08)
Note: Version bump only for package @crawlee/http
3.2.1 (2023-02-07)
Note: Version bump only for package @crawlee/http
3.2.0 (2023-02-07)
Bug Fixes
3.1.4 (2022-12-14)
Note: Version bump only for package @crawlee/http
3.1.3 (2022-12-07)
Note: Version bump only for package @crawlee/http
3.1.2 (2022-11-15)
Note: Version bump only for package @crawlee/http
3.1.1 (2022-11-07)
Note: Version bump only for package @crawlee/http
3.1.0 (2022-10-13)
Note: Version bump only for package @crawlee/http
3.0.4 (2022-08-22)
Note: Version bump only for package @crawlee/http