Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(social): extract emails from each text node separately (#2952)

Ondrej Klinovsky799afc1dbb68
diff --git a/packages/utils/src/internals/social.ts b/packages/utils/src/internals/social.ts
index 7eccc7a2e..2c4a6a179 100644
--- a/packages/utils/src/internals/social.ts
+++ b/packages/utils/src/internals/social.ts
@@ -681,13 +681,21 @@ export function parseHandlesFromHtml(html: string, data: Record<string, unknown>
const text = htmlToText($);
if (data) data.text = text;
+ // NOTE: we need to parse each text separately, orherwise we might concatenate unrelated texts
+ // e.g. `<div>6HT<a>[email protected]</a></div>` would become [email protected]
+ const texts = $('*')
+ .contents()
+ .toArray()
+ .filter((node) => node.type === 'text')
+ .map((node) => $(node).text().trim());
+
// Find all <a> links with href tag
const linkUrls: string[] = [];
$('a[href]').each((_index, elem) => {
if (elem) linkUrls.push($(elem).attr('href')!);
});
- result.emails = emailsFromUrls(linkUrls).concat(emailsFromText(text));
+ result.emails = emailsFromUrls(linkUrls).concat(texts.flatMap(emailsFromText));
result.phones = phonesFromUrls(linkUrls);
result.phonesUncertain = phonesFromText(text);
diff --git a/test/utils/social.test.ts b/test/utils/social.test.ts
index 54f237c22..ca91ef515 100644
--- a/test/utils/social.test.ts
+++ b/test/utils/social.test.ts
@@ -465,12 +465,19 @@ describe('utils.social', () => {
<a href="https://pinterest.com/pin/10084556789">My favourite pin</a> duplicate
<a href="https://discord.com/invite/jyEM2PRvMU/">Join us on Discord</a>
<a href="discord.gg/discord-developers">Join our Discord community</a>
+ <div>+4207751111111<a href="mailto:[email protected]">[email protected]</a></div>
</body>
</html>
`),
).toEqual({
discords: ['discord.gg/discord-developers', 'https://discord.com/invite/jyEM2PRvMU/'],
+ emails: [
+ ],
phones: ['+42077533333'],
phonesUncertain: [
'+4207751111111',