Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: refactor `extractUrls` to split the text line by line first (#2122)

Martin Adámek7265cd7148bb
diff --git a/packages/utils/src/internals/extract-urls.ts b/packages/utils/src/internals/extract-urls.ts
index d1de7299d..8e3402186 100644
--- a/packages/utils/src/internals/extract-urls.ts
+++ b/packages/utils/src/internals/extract-urls.ts
@@ -73,6 +73,13 @@ export function extractUrls(options: ExtractUrlsOptions): string[] {
string: ow.string,
urlRegExp: ow.optional.regExp,
}));
- const { string, urlRegExp = URL_NO_COMMAS_REGEX } = options;
- return string.match(urlRegExp) || [];
+ const lines = options.string.split('\n');
+ const result: string[] = [];
+ const urlRegExp = options.urlRegExp ?? URL_NO_COMMAS_REGEX;
+
+ for (const line of lines) {
+ result.push(...(line.match(urlRegExp) ?? []));
+ }
+
+ return result;
}