Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated docs of Apify.utils.htmlToText()
Jan Curn68e59252b8f4
diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 8939800f3..cb9ca6082 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -26,6 +26,7 @@ await Apify.utils.sleep(1500);* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ <code>Promise<Array<String>></code>* [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ <code>Array<String></code>* [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ <code>String</code>+ * [`.htmlToText(html)`](#utils.htmlToText) ⇒ <code>String</code><a name="utils.URL_NO_COMMAS_REGEX"></a>@@ -137,3 +138,47 @@ Collects all URLs in an arbitrary string to an array, optionally using a custom## `utils.getRandomUserAgent()` ⇒ <code>String</code>Returns a randomly selected User-Agent header out of a list of the most common headers.+<a name="utils.htmlToText"></a>++## `utils.htmlToText(html)` ⇒ <code>String</code>+The function converts a HTML document to a plain text.++The plain text generated by the function is similar to a text captured+by pressing Ctrl+A and Ctrl+C on a page when loaded in a web browser.+The function doesn't aspire to preserve the formatting or to be perfectly correct with respect to HTML specifications.+However, it attempts to generate newlines and whitespaces in and around HTML elements+to avoid merging distinct parts of text and thus enable extraction of data from the text (e.g. phone numbers).++**Example usage**+```javascript+const text = htmlToText('<html><body>Some text</body></html>');+console.log(text);+```++Note that the function uses [cheerio](https://www.npmjs.com/package/cheerio) to parse the HTML.+Optionally, to avoid duplicate parsing of HTML and thus improve performance, you can pass+an existing Cheerio object to the function instead of the HTML text. The HTML should be parsed+with the `decodeEntities` option set to `true`. For example:++```javascript+const cheerio = require('cheerio');+const html = '<html><body>Some text</body></html>';+const text = htmlToText(cheerio.load(html, { decodeEntities: true }));+```++**Returns**: <code>String</code> - Plain text+<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>html</code></td><td><code>String</code> | <code>function</code></td>+</tr>+<tr>+<td colspan="3"><p>HTML text or parsed HTML represented using a+<a href="https://www.npmjs.com/package/cheerio">cheerio</a> function.</p>+</td></tr></tbody>+</table>diff --git a/src/utils.js b/src/utils.jsindex c46e53b71..d65f83037 100644--- a/src/utils.js+++ b/src/utils.js@@ -517,7 +517,7 @@ const BLOCK_TAGS_REGEX = /^(p|h1|h2|h3|h4|h5|h6|ol|ul|li|pre|address|blockquote|* @return {String} Plain text* @memberOf utils*/-export const htmlToText = (html) => {+const htmlToText = (html) => {if (!html) return '';const $ = typeof html === 'function' ? html : cheerio.load(html, { decodeEntities: true });