Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated comments

Jan Curn66b0712384dc
diff --git a/src/dataset.js b/src/dataset.js
index 6ce9b35aa..574b91193 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -251,7 +251,6 @@ export class Dataset {
* browser to download the file rather than to display it. By default, this header is not present.
* @param {String} [options.delimiter=',']
* A delimiter character for CSV files, only used if `format` is `csv`.
- * You might need to URL-encode the character (e.g. use `%09` for tab or `%3B` for semicolon).
* @param {Boolean} [options.bom]
* All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte
* Order Mark (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default
diff --git a/src/utils_social.js b/src/utils_social.js
index 27e613c21..c4e863c78 100644
--- a/src/utils_social.js
+++ b/src/utils_social.js
@@ -485,6 +485,8 @@ const parseHandlesFromHtml = (html, data = null) => {
facebooks: [],
};
+ // TODO: extract phone numbers from JSON+LD, only put sure phones in "phones", add "phonesGuessed" for others
+
if (!_.isString(html)) return result;
const $ = cheerio.load(html, { decodeEntities: true });