Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Bumped package.json version, updated CHANGELOG and docs

Jan Curn6a6d79b5dafc
diff --git a/docs/api/social.md b/docs/api/social.md
index bdd0d8aa6..01d45010f 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -15,3 +15,175 @@ const Apify = require('apify');
const emails = Apify.utils.social.emailsFromText('[email protected] [email protected]');
```
+
+* [`social`](#social) : <code>object</code>
+ * [`.EMAIL_REGEX`](#social.EMAIL_REGEX) : <code>RegExp</code>
+ * [`.EMAIL_REGEX_GLOBAL`](#social.EMAIL_REGEX_GLOBAL) : <code>RegExp</code>
+ * [`.LINKEDIN_REGEX`](#social.LINKEDIN_REGEX) : <code>RegExp</code>
+ * [`.LINKEDIN_REGEX_GLOBAL`](#social.LINKEDIN_REGEX_GLOBAL) : <code>RegExp</code>
+ * [`.INSTAGRAM_REGEX`](#social.INSTAGRAM_REGEX) : <code>RegExp</code>
+ * [`.INSTAGRAM_REGEX_GLOBAL`](#social.INSTAGRAM_REGEX_GLOBAL) : <code>RegExp</code>
+ * [`.TWITTER_REGEX`](#social.TWITTER_REGEX) : <code>RegExp</code>
+ * [`.TWITTER_REGEX_GLOBAL`](#social.TWITTER_REGEX_GLOBAL) : <code>RegExp</code>
+ * [`.FACEBOOK_REGEX`](#social.FACEBOOK_REGEX) : <code>RegExp</code>
+ * [`.FACEBOOK_REGEX_GLOBAL`](#social.FACEBOOK_REGEX_GLOBAL) : <code>RegExp</code>
+ * [`.emailsFromText(text)`](#social.emailsFromText) ⇒ <code>Array&lt;String&gt;</code>
+ * [`.emailsFromUrls(urls)`](#social.emailsFromUrls) ⇒ <code>Array&lt;String&gt;</code>
+ * [`.phonesFromText(text)`](#social.phonesFromText) ⇒ <code>Array&lt;String&gt;</code>
+ * [`.phonesFromUrls(urls)`](#social.phonesFromUrls) ⇒ <code>Array&lt;String&gt;</code>
+
+<a name="social.EMAIL_REGEX"></a>
+
+## `social.EMAIL\_REGEX` : <code>RegExp</code>
+Regular expression to exactly match a single email address.
+It has the following form: `/^...$/i`.
+
+<a name="social.EMAIL_REGEX_GLOBAL"></a>
+
+## `social.EMAIL\_REGEX\_GLOBAL` : <code>RegExp</code>
+Regular expression to find multiple email addresses in a text.
+It has the following form: `/.../ig`.
+
+<a name="social.LINKEDIN_REGEX"></a>
+
+## `social.LINKEDIN\_REGEX` : <code>RegExp</code>
+Regular expression to exactly match a single LinkedIn profile URL, without any additional
+subdirectories or query parameters. The regular expression has the following form: `/^...$/i`.
+
+Example usage:
+```
+TODO
+```
+
+<a name="social.LINKEDIN_REGEX_GLOBAL"></a>
+
+## `social.LINKEDIN\_REGEX\_GLOBAL` : <code>RegExp</code>
+Regular expression to find multiple LinkedIn profile URLs in a text.
+It has the following form: `/.../ig`.
+
+<a name="social.INSTAGRAM_REGEX"></a>
+
+## `social.INSTAGRAM\_REGEX` : <code>RegExp</code>
+Regular expression to exactly match a single Instagram profile URL.
+It has the following form: `/^...$/i`.
+
+<a name="social.INSTAGRAM_REGEX_GLOBAL"></a>
+
+## `social.INSTAGRAM\_REGEX\_GLOBAL` : <code>RegExp</code>
+Regular expression to find multiple Instagram profile URLs in a text.
+It has the following form: `/.../ig`.
+
+<a name="social.TWITTER_REGEX"></a>
+
+## `social.TWITTER\_REGEX` : <code>RegExp</code>
+Regular expression to exactly match a single Instagram profile URL.
+It has the following form: `/^...$/i`.
+
+<a name="social.TWITTER_REGEX_GLOBAL"></a>
+
+## `social.TWITTER\_REGEX\_GLOBAL` : <code>RegExp</code>
+Regular expression to find multiple Instagram profile URLs in a text.
+It has the following form: `/.../ig`.
+
+<a name="social.FACEBOOK_REGEX"></a>
+
+## `social.FACEBOOK\_REGEX` : <code>RegExp</code>
+Regular expression to exactly match a single Facebook user profile URL.
+It has the following form: `/^...$/i`.
+
+<a name="social.FACEBOOK_REGEX_GLOBAL"></a>
+
+## `social.FACEBOOK\_REGEX\_GLOBAL` : <code>RegExp</code>
+Regular expression to find multiple Instagram profile URLs in a text.
+It has the following form: `/.../ig`.
+
+<a name="social.emailsFromText"></a>
+
+## `social.emailsFromText(text)` ⇒ <code>Array&lt;String&gt;</code>
+The function extracts email addresses from a plain text.
+Note that the function preserves the order of emails and keep duplicates.
+
+**Returns**: <code>Array&lt;String&gt;</code> - Array of emails addresses found.
+If no emails are found, the function returns an empty array.
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>text</code></td><td><code>String</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Text to search in.</p>
+</td></tr></tbody>
+</table>
+<a name="social.emailsFromUrls"></a>
+
+## `social.emailsFromUrls(urls)` ⇒ <code>Array&lt;String&gt;</code>
+The function extracts email addresses from a list of URLs.
+Basically it looks for all `mailto:` URLs and returns valid email addresses from them.
+Note that the function preserves the order of emails and keep duplicates.
+
+**Returns**: <code>Array&lt;String&gt;</code> - Array of emails addresses found.
+If no emails are found, the function returns an empty array.
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>urls</code></td><td><code>Array&lt;String&gt;</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Array of URLs.</p>
+</td></tr></tbody>
+</table>
+<a name="social.phonesFromText"></a>
+
+## `social.phonesFromText(text)` ⇒ <code>Array&lt;String&gt;</code>
+The function attempts to extract phone numbers from a text. Please note that
+the results might not be accurate, since phone numbers appear in a large variety of formats and conventions.
+If you encounter some problems, please [file an issue](https://github.com/apifytech/apify-js/issues).
+
+**Returns**: <code>Array&lt;String&gt;</code> - Array of phone numbers found.
+If no phone numbers are found, the function returns an empty array.
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>text</code></td><td><code>String</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Text to search the phone numbers in.</p>
+</td></tr></tbody>
+</table>
+<a name="social.phonesFromUrls"></a>
+
+## `social.phonesFromUrls(urls)` ⇒ <code>Array&lt;String&gt;</code>
+Finds phone number links in an array of URLs and extracts the phone numbers from them.
+Note that the phone number links look like `tel://123456789`, `tel:/123456789` or `tel:123456789`.
+
+**Returns**: <code>Array&lt;String&gt;</code> - Array of phone numbers found.
+If no phone numbers are found, the function returns an empty array.
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>urls</code></td><td><code>Array&lt;String&gt;</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Array of URLs.</p>
+</td></tr></tbody>
+</table>
diff --git a/package.json b/package.json
index 8a0d5bb63..beae5424c 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apify",
- "version": "0.9.8",
+ "version": "0.9.9",
"description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=8.0.0"
diff --git a/src/utils_social.js b/src/utils_social.js
index ae44ed8d3..7baa79ede 100644
--- a/src/utils_social.js
+++ b/src/utils_social.js
@@ -13,7 +13,7 @@ const EMAIL_REGEX_STRING = '(?:[a-z0-9!#$%&\'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&\'
* Regular expression to exactly match a single email address.
* It has the following form: `/^...$/i`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const EMAIL_REGEX = new RegExp(`^${EMAIL_REGEX_STRING}$`, 'i');
@@ -21,7 +21,7 @@ const EMAIL_REGEX = new RegExp(`^${EMAIL_REGEX_STRING}$`, 'i');
* Regular expression to find multiple email addresses in a text.
* It has the following form: `/.../ig`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const EMAIL_REGEX_GLOBAL = new RegExp(EMAIL_REGEX_STRING, 'ig');
@@ -35,7 +35,7 @@ const EMAIL_URL_PREFIX_REGEX = /^mailto:/i;
* @param {String} text Text to search in.
* @return {String[]} Array of emails addresses found.
* If no emails are found, the function returns an empty array.
- * @memberOf utils.social
+ * @memberOf social
*/
const emailsFromText = (text) => {
if (!_.isString(text)) return [];
@@ -50,7 +50,7 @@ const emailsFromText = (text) => {
* @param {String[]} urls Array of URLs.
* @return {String[]} Array of emails addresses found.
* If no emails are found, the function returns an empty array.
- * @memberOf utils.social
+ * @memberOf social
*/
const emailsFromUrls = (urls) => {
if (!Array.isArray(urls)) throw new Error('The "urls" parameter must be an array');
@@ -135,7 +135,7 @@ const SKIP_PHONE_REGEX = new RegExp(`^(${SKIP_PHONE_REGEXS.join('|')})$`, 'i');
* @param {String} text Text to search the phone numbers in.
* @return {String[]} Array of phone numbers found.
* If no phone numbers are found, the function returns an empty array.
- * @memberOf utils.social
+ * @memberOf social
*/
const phonesFromText = (text) => {
if (!_.isString(text)) return [];
@@ -163,7 +163,7 @@ const phonesFromText = (text) => {
* @param {String[]} urls Array of URLs.
* @return {String[]} Array of phone numbers found.
* If no phone numbers are found, the function returns an empty array.
- * @memberOf utils.social
+ * @memberOf social
*/
const phonesFromUrls = (urls) => {
if (!Array.isArray(urls)) throw new Error('The "urls" parameter must be an array');
@@ -209,7 +209,7 @@ const FACEBOOK_REGEX_STRING = `(?<!\\w)(?:http(?:s)?:\\/\\/)?(?:www.)?(?:faceboo
* TODO
* ```
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const LINKEDIN_REGEX = new RegExp(`^${LINKEDIN_REGEX_STRING}$`, 'i');
@@ -217,7 +217,7 @@ const LINKEDIN_REGEX = new RegExp(`^${LINKEDIN_REGEX_STRING}$`, 'i');
* Regular expression to find multiple LinkedIn profile URLs in a text.
* It has the following form: `/.../ig`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const LINKEDIN_REGEX_GLOBAL = new RegExp(LINKEDIN_REGEX_STRING, 'ig');
@@ -226,7 +226,7 @@ const LINKEDIN_REGEX_GLOBAL = new RegExp(LINKEDIN_REGEX_STRING, 'ig');
* Regular expression to exactly match a single Instagram profile URL.
* It has the following form: `/^...$/i`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const INSTAGRAM_REGEX = new RegExp(`^${INSTAGRAM_REGEX_STRING}$`, 'i');
@@ -234,7 +234,7 @@ const INSTAGRAM_REGEX = new RegExp(`^${INSTAGRAM_REGEX_STRING}$`, 'i');
* Regular expression to find multiple Instagram profile URLs in a text.
* It has the following form: `/.../ig`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const INSTAGRAM_REGEX_GLOBAL = new RegExp(INSTAGRAM_REGEX_STRING, 'ig');
@@ -243,7 +243,7 @@ const INSTAGRAM_REGEX_GLOBAL = new RegExp(INSTAGRAM_REGEX_STRING, 'ig');
* Regular expression to exactly match a single Instagram profile URL.
* It has the following form: `/^...$/i`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const TWITTER_REGEX = new RegExp(`^${TWITTER_REGEX_STRING}$`, 'i');
@@ -251,7 +251,7 @@ const TWITTER_REGEX = new RegExp(`^${TWITTER_REGEX_STRING}$`, 'i');
* Regular expression to find multiple Instagram profile URLs in a text.
* It has the following form: `/.../ig`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const TWITTER_REGEX_GLOBAL = new RegExp(TWITTER_REGEX_STRING, 'ig');
@@ -259,7 +259,7 @@ const TWITTER_REGEX_GLOBAL = new RegExp(TWITTER_REGEX_STRING, 'ig');
* Regular expression to exactly match a single Facebook user profile URL.
* It has the following form: `/^...$/i`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const FACEBOOK_REGEX = new RegExp(`^${FACEBOOK_REGEX_STRING}$`, 'i');
@@ -267,7 +267,7 @@ const FACEBOOK_REGEX = new RegExp(`^${FACEBOOK_REGEX_STRING}$`, 'i');
* Regular expression to find multiple Instagram profile URLs in a text.
* It has the following form: `/.../ig`.
* @type {RegExp}
- * @memberOf utils.social
+ * @memberOf social
*/
const FACEBOOK_REGEX_GLOBAL = new RegExp(FACEBOOK_REGEX_STRING, 'ig');