Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Change Log
All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.
3.18.1 (2026-08-12)
Note: Version bump only for package @crawlee/utils
3.18.0 (2026-08-04)
Bug Fixes
- filter sitemap-derived URLs by enqueue strategy (#3797) (9dc3160)
- sitemap: recognize sitemap_index as a valid input sitemap url (#3909) (eacab5e)
- utils: advance the CPU tick baseline in systemInfoV2 getCurrentCpuTicks (#3824) (bb23800)
- utils: allow apostrophes and asterisks in extracted URLs (#3790) (e9baffd), closes #2755
- utils: prevent ReDoS in social email regex (#3845) (d6a2b39)
- utils: prevent sitemap metadata leak across url without loc (#3795) (895ed62)
- utils: use article:* selectors for article Open Graph properties (#3789) (b660506)
3.17.0 (2026-06-04)
Bug Fixes
- add network timeouts to discoverValidSitemaps to prevent indefinite hangs (#3429) (ee90835)
- core: apply object searchParams in HTTP request options (#3622) (d32946e)
- log warning when Sitemap.load returns empty sitemap due to loading error (#3462) (2cf9c1c)
- Prevent accidental request dropping with
maxRequestsPerCrawl(#3531) (b23319b) - prevent v3 sitemap discovery init hangs in discoverValidSitemaps (#3434) (6c04f92), closes #3412
- using GET for candidate checks - sitemaps (#3464) (f82eb04), closes #3463
Features
3.16.0 (2026-02-06)
Bug Fixes
Features
3.15.3 (2025-11-10)
Note: Version bump only for package @crawlee/utils
3.15.2 (2025-10-23)
Features
3.15.1 (2025-09-26)
Note: Version bump only for package @crawlee/utils
3.15.0 (2025-09-17)
Note: Version bump only for package @crawlee/utils
3.14.1 (2025-08-05)
Note: Version bump only for package @crawlee/utils
3.14.0 (2025-07-25)
Bug Fixes
3.13.10 (2025-07-09)
Note: Version bump only for package @crawlee/utils
3.13.9 (2025-06-27)
Bug Fixes
- Do not log 'malformed sitemap content' on network errors in
Sitemap.tryCommonNames(#3015) (64a090f), closes #2884
Features
3.13.8 (2025-06-16)
Bug Fixes
- Persist rendering type detection results in
AdaptivePlaywrightCrawler(#2987) (76431ba), closes #2899
3.13.7 (2025-06-06)
Note: Version bump only for package @crawlee/utils
3.13.6 (2025-06-05)
Note: Version bump only for package @crawlee/utils
3.13.5 (2025-05-20)
Note: Version bump only for package @crawlee/utils
3.13.4 (2025-05-14)
Bug Fixes
3.13.3 (2025-05-05)
Note: Version bump only for package @crawlee/utils
3.13.2 (2025-04-08)
Note: Version bump only for package @crawlee/utils
3.13.1 (2025-04-07)
Bug Fixes
Features
3.13.0 (2025-03-04)
Features
3.12.2 (2025-01-27)
Note: Version bump only for package @crawlee/utils
3.12.1 (2024-12-04)
Bug Fixes
3.12.0 (2024-11-04)
Bug Fixes
Features
3.11.5 (2024-10-04)
Note: Version bump only for package @crawlee/utils
3.11.4 (2024-09-23)
Bug Fixes
SitemapRequestList.teardown()doesn't breakpersistStatecalls (#2673) (fb2c5cd), closes /github.com/apify/crawlee/blob/f3eb99d9fa9a7aa0ec1dcb9773e666a9ac14fb76/packages/core/src/storages/sitemap_request_list.ts#L446 #2672
3.11.3 (2024-09-03)
Bug Fixes
3.11.2 (2024-08-28)
Bug Fixes
Features
3.11.1 (2024-07-24)
Bug Fixes
3.11.0 (2024-07-09)
Features
3.10.5 (2024-06-12)
Note: Version bump only for package @crawlee/utils
3.10.4 (2024-06-11)
Note: Version bump only for package @crawlee/utils
3.10.3 (2024-06-07)
Bug Fixes
3.10.2 (2024-06-03)
Bug Fixes
Features
3.10.1 (2024-05-23)
Bug Fixes
- adjust
URL_NO_COMMAS_REGEXregexp to allow single character hostnames (#2492) (ec802e8), closes #2487
3.10.0 (2024-05-16)
Bug Fixes
- malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
- return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
- sitemap
content-typecheck breaks oncontent-typeparameters (#2442) (db7d372)
Features
3.9.2 (2024-04-17)
Features
3.9.1 (2024-04-11)
Note: Version bump only for package @crawlee/utils
3.9.0 (2024-04-10)
Bug Fixes
Features
3.8.2 (2024-03-21)
Bug Fixes
3.8.1 (2024-02-22)
Note: Version bump only for package @crawlee/utils
3.8.0 (2024-02-21)
Features
- add Sitemap.tryCommonNames to check well known sitemap locations (#2311) (85589f1), closes #2307
- core: add
userAgentparameter toRobotsFile.isAllowed()+RobotsFile.from()helper (#2338) (343c159) - Support plain-text sitemap files (sitemap.txt) (#2315) (0bee7da)
3.7.3 (2024-01-30)
Bug Fixes
3.7.2 (2024-01-09)
Note: Version bump only for package @crawlee/utils
3.7.1 (2024-01-02)
Bug Fixes
3.7.0 (2023-12-21)
Bug Fixes
Features
3.6.2 (2023-11-26)
Note: Version bump only for package @crawlee/utils
3.6.1 (2023-11-15)
Note: Version bump only for package @crawlee/utils
3.6.0 (2023-11-15)
Features
3.5.8 (2023-10-17)
Bug Fixes
3.5.7 (2023-10-05)
Note: Version bump only for package @crawlee/utils
3.5.6 (2023-10-04)
Features
- add incapsula iframe selector to the blocked list (#2111) (2b17d8a), closes apify/store-website-content-crawler#154
3.5.5 (2023-10-02)
Note: Version bump only for package @crawlee/utils
3.5.4 (2023-09-11)
Note: Version bump only for package @crawlee/utils
3.5.3 (2023-08-31)
Bug Fixes
3.5.2 (2023-08-21)
Note: Version bump only for package @crawlee/utils
3.5.1 (2023-08-16)
Note: Version bump only for package @crawlee/utils
3.5.0 (2023-07-31)
Features
3.4.2 (2023-07-19)
Features
3.4.1 (2023-07-13)
Note: Version bump only for package @crawlee/utils
3.4.0 (2023-06-12)
Note: Version bump only for package @crawlee/utils
3.3.3 (2023-05-31)
Note: Version bump only for package @crawlee/utils
3.3.2 (2023-05-11)
Note: Version bump only for package @crawlee/utils
3.3.1 (2023-04-11)
Bug Fixes
- jsdom: delay closing of the window and add some polyfills (2e81618)
3.3.0 (2023-03-09)
Bug Fixes
3.2.2 (2023-02-08)
Note: Version bump only for package @crawlee/utils
3.2.1 (2023-02-07)
Note: Version bump only for package @crawlee/utils
3.2.0 (2023-02-07)
Bug Fixes
3.1.2 (2022-11-15)
Note: Version bump only for package @crawlee/utils
3.1.1 (2022-11-07)
Note: Version bump only for package @crawlee/utils
3.1.0 (2022-10-13)
Note: Version bump only for package @crawlee/utils
3.0.4 (2022-08-22)
Note: Version bump only for package @crawlee/utils