Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improvements in Apify.utils.social.parseHandlesFromHtml
Jan Curn4d70d2584871
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 60e347f6b..194aba61a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,8 @@+xxx+===================+- Improvements in `Apify.utils.social.parseHandlesFromHtml` and `Apify.utils.htmlToText`+- Updated docs+0.11.3 / 2018-01-10===================- Fix `reusePages` causing Puppeteer to fail when used together with request interception.diff --git a/src/utils_social.js b/src/utils_social.jsindex c4e863c78..1d85ad19a 100644--- a/src/utils_social.js+++ b/src/utils_social.js@@ -451,6 +451,7 @@ try {* {* emails: String[],* phones: String[],+ * phonesUncertain: String[],* linkedIns: String[],* twitters: String[],* instagrams: String[],@@ -458,6 +459,11 @@ try {* }* ```*+ * Note that the `phones` field contains phone numbers extracted from the special phone links+ * such as `<a href="tel:+1234556789">call us</a>` (see {@link phonesFromUrls})+ * and potentially other sources with high certainty, while `phonesUncertain` contains phone numbers+ * extracted from the plain text, which might be very inaccurate.+ ** **Example usage:*** ```javascript* const puppeteer = await Apify.launchPuppeteer();@@ -479,13 +485,14 @@ const parseHandlesFromHtml = (html, data = null) => {const result = {emails: [],phones: [],+ phonesUncertain: [],linkedIns: [],twitters: [],instagrams: [],facebooks: [],};- // TODO: extract phone numbers from JSON+LD, only put sure phones in "phones", add "phonesGuessed" for others+ // TODO: maybe extract phone numbers from JSON+LDif (!_.isString(html)) return result;@@ -502,7 +509,8 @@ const parseHandlesFromHtml = (html, data = null) => {});result.emails = emailsFromUrls(linkUrls).concat(emailsFromText(text));- result.phones = phonesFromUrls(linkUrls).concat(phonesFromText(text));+ result.phones = phonesFromUrls(linkUrls);+ result.phonesUncertain = phonesFromText(text);// Note that these regexps extract just the base profile path. For example for// https://www.linkedin.com/in/carl-newman-123456a/detail/recent-activity/diff --git a/test/utils_social.js b/test/utils_social.jsindex fa47120b0..c2a598ec6 100644--- a/test/utils_social.js+++ b/test/utils_social.js@@ -315,6 +315,7 @@ describe('utils.social', () => {const EMPTY_RESULT = {emails: [],phones: [],+ phonesUncertain: [],linkedIns: [],twitters: [],instagrams: [],@@ -389,7 +390,8 @@ describe('utils.social', () => {</html>`)).to.eql({- phones: ['+4207751111111', '+420775222222', '+42077533333'],+ phones: ['+42077533333'],+ phonesUncertain: ['+4207751111111', '+420775222222'],linkedIns: ['http://www.linkedin.com/in/nohttps/','https://cz.linkedin.com/in/somecountry',