Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: adjust `URL_NO_COMMAS_REGEX` regexp to allow single character hostnames (#2492)

Martin Adámekec802e85f540
diff --git a/packages/utils/src/internals/general.ts b/packages/utils/src/internals/general.ts
index ab1904286..221c84d4b 100644
--- a/packages/utils/src/internals/general.ts
+++ b/packages/utils/src/internals/general.ts
@@ -5,21 +5,15 @@ import { setTimeout } from 'node:timers/promises';
* Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters
* and does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
*/
-// eslint-disable-next-line
-export const URL_NO_COMMAS_REGEX = RegExp(
- 'https?://(www\\.)?[\\p{L}0-9][-\\p{L}0-9@:%._\\+~#=]{0,254}[\\p{L}0-9]\\.[a-z]{2,63}(:\\d{1,5})?(/[-\\p{L}0-9@:%_\\+.~#?&//=\\(\\)]*)?',
- 'giu',
-);
+export const URL_NO_COMMAS_REGEX =
+ /https?:\/\/(www\.)?([\p{L}0-9]|[\p{L}0-9][-\p{L}0-9@:%._+~#=]{0,254}[\p{L}0-9])\.[a-z]{2,63}(:\d{1,5})?(\/[-\p{L}0-9@:%_+.~#?&/=()]*)?/giu;
/**
* Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.
* Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
*/
-// eslint-disable-next-line
-export const URL_WITH_COMMAS_REGEX = RegExp(
- 'https?://(www\\.)?[\\p{L}0-9][-\\p{L}0-9@:%._\\+~#=]{0,254}[\\p{L}0-9]\\.[a-z]{2,63}(:\\d{1,5})?(/[-\\p{L}0-9@:%_\\+,.~#?&//=\\(\\)]*)?',
- 'giu',
-);
+export const URL_WITH_COMMAS_REGEX =
+ /https?:\/\/(www\.)?([\p{L}0-9]|[\p{L}0-9][-\p{L}0-9@:%._+~#=]{0,254}[\p{L}0-9])\.[a-z]{2,63}(:\d{1,5})?(\/[-\p{L}0-9@:%_+,.~#?&/=()]*)?/giu;
let isDockerPromiseCache: Promise<boolean> | undefined;
diff --git a/test/shared/data/simple_url_list.txt b/test/shared/data/simple_url_list.txt
index f15af531b..383858b8d 100644
--- a/test/shared/data/simple_url_list.txt
+++ b/test/shared/data/simple_url_list.txt
@@ -98,3 +98,4 @@
https://www.example.com/
https://example.net/action/box.php
http://www.example.com/
+ https://x.com/elonmusk/status/1783231058462617962